Reading his post, and some of the comments afterwards (when will I learn?!), got me to thinking about my own experiences with TDD, and I decided to share two of my own stories.
Once Upon a Time
I used to write bug-free software without TDD! (Look at me: I am so smart...)
My boss and I would play this game called "Who Wrote the Bug?" in which he would claim that a bug was in my code, and I would come back (after painstakingly debugging the whole product) and point out where it was in his code. (Hmmm...who was smarter?)
Humility Sets In
The software I was writing was a protocol stack that needed to convert between 8-bit and 9-bit bytes, or 16-bit and 36-bit words. I used the command-line/printf approach (System.out.println for the Java readers; Console.WriteLine for C# readers) to test stuff. The debugger, COFF, was okay, but testing through a debugger is such a pain.
If you haven't figured it out from the clues I've dropped, this was over 20 years ago. In retrospect, this was pretty tame stuff. I could keep track of what was going on in my head.
One can assume that software projects these days are more complex. Perhaps too complex for one person to keep track of all the little moving parts. It's enough for the team to have a grasp of the overall concepts, goals, and metaphors.
I would argue that even on a team with only three or four developers, eventually they're going to bump into each other's changes. They need a suite of microtests in order to maintain quality code.
The test frameworks like JUnit, NUnit, and RSpec essentially give you a way to let the computer check the "printf" output (figuratively speaking) for you. You record your assumptions, rather than manually checking them each time, and you can let those assumptions grow upon each other. You record them separately, rather than interspersing the following infamous abomination:
#ifdef DEBUG
printf("foo=%s", foo);
#endif
Two Years of Investment
In 2002 I worked on a team of four J2EE developers on a life-critical (i.e., "You break it, someone may die") application. After two years, we had built some pretty sophisticated reports and heuristics that helped hospitals determine who was most likely to survive an organ transplant.
We had over 10,000 tests. They all ran in about 12 minutes.
Imagine: All aspects of a complex, life-critical, team-built application could be thoroughly checked in 12 minutes. We could (we did) bring in someone new, and essentially say "today we will make whatever changes we need, even mere hours before the next release, as long as we run all the tests and see a 100% pass-rate!"
Twelve minutes, and we knew we hadn't broken any of our careful, 8-(plus-)person-year investment.
Rise to the Occasion
If you're sitting alone in your basement writing a game to sell to a VC, you may not care if it's maintainable. By all means, please continue using only manual testing techniques to verify your app. I'm not being snarky: This may be the optimal win-win approach for you and the VC. (The team that eventually maintains your app will curse your name and throw darts at your picture, but do you care?)
On the other hand, if you're part of a team, writing mission-critical software for a large or complex domain, and it needs to get beyond version 1.0; then please: Make sure you can always test everything, and quickly. TDD is the best--the only--way the industry has invented so far for doing this.
Anything else would be unprofessional.
Nice post!)
ReplyDelete"Make sure you can always test everything, and quickly." - is very true!
Nice read.
ReplyDeleteYou mention two extremes: no tests vs. 100% coverage. In my experience, the maintenance of legacy code (the stuff one may build for a VC) benefits tremendously from a couple of coarse and a couple of fine grained tests here and there. E. g. I'm currently in a project that's nowhere near 100% coverage, and we see velocity going up.
Where do you think is the right trade-off, for a regular application (not life-critical)? Could you still work without any test automation?
Hi, Tijn,
ReplyDeleteLegacy (untested) code is a reality for almost all teams I work with. The ultimate wish would be to get all code looking like it had been TDD'd, but we have to acknowledge that we need to prioritize, and we may never reach that ideal state.
The team must decide on a rule of thumb (or perhaps a definition of done) regarding when to add pinning tests around untested code, when to rely on existing functional tests, when to refactor that code, and when to just leave it alone.
So my intent was not to be dogmatic about 100% coverage (unless you're starting fresh). I would recommend that the team not tolerate anything less than 100% pass-rates for all existing tests (all tests that have passed in the past, be they unit tests or acceptance tests or...).
Be intolerant of regression. Otherwise, you'll fall into the "Broken Window" syndrome: Once a team or collection of teams tolerates "that test that fails only on Tuesdays" it doesn't take very long before people start to lose confidence in the regression suite.
Then you're back to the pain and suffering that all other undisciplined teams live with (because they've never experienced anything better).
Would I work without *any* test automation? No. I agree that you have to find the right balance, but if I were the guy writing alone in a basement for a VC, I would do TDD. I'm thoroughly "infected": Writing code without tests would seem unnatural and silly, like having one arm tied behind my back.
If I were handed someone else's crud? Whenever I had to work with some part of that legacy code, I'd use behavior-pinning tests to cover as much of the behavior as possible. Microtests would be preferable, but that's not usually readily possible with garage-band code. Untested code tends to present us with a compromise: We have to either refactor it to make it testable, or use some inappropriately fancy part of the testing framework to dig into the code. That last choice leaves the code looking like crud, and eliminates the team's motivation to embrace the crud and make it their own. (Yes, I'm referring to "testing private methods" and other cheats.)
TDD brings one more "surprise" into development process. It is hard to believe that so small amount of code may do so much. It is just because you write only code you really need. Also I agree with J.B. that TDD is about design at first place. Code coverage is just an additional "benefit".
ReplyDeleteRob, good post.
ReplyDeleteCould you have written this mission-critical software by writing tests later, without using TDD, there by ensuring the desired level of quality?
Was there any time invested in design at all or the developers were designing "on their feet" while writing unit tests ?
Is TDD really meant for mission-critical and complex software? Can you explain how/when you design in TDD?
Kind regards.
Anonymous,
ReplyDeleteThanks for the comment. To answer your questions:
1. No. Code without tests tends to be difficult to test. TDD assures you cannot write code that is difficult to unit-test.
2. Yes, there was some design time, though it was just-in-time. I recommend that a pair of developers talk about the task, use a whiteboard and UML, or (for tougher problems) use CRC cards in a small group. Once the pair has identified some behaviors/responsibilities, they can sit down and start TDD. What I like to see is that they start this as early as possible, because the act of coding some behavior is a strong feedback loop for the design of this and other behaviors. We "tell how hot or cold the water really is by taking a sip" not by deliberating for weeks.
3. I cannot recommend writing software without TDD. Recall, I said "life-critical": We were reminded each iteration that a mistake could cost someone her life. That is why we used TDD, (along with ATDD based on historically-validated results). The more risky or complex the software project, the more necessary it is to pin down existing functional parts of the system with a thorough safety-net of regression tests.
I hope that helps.