Joel Test

Joel created the “Joel Test,” which is a measure of how good a software team is. (Attaching one’s own name to a set of rules is a great PR move.) The test is simple. Just answer yes or no to each question.

1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?

Joel states that a score of 10 or lower indicates serious development problems and that most software organizations run at a score of 2 or 3. Microsoft, he claims, runs at a score of 12, which seems to be consistent with my experience, except maybe the last point. However, I have to think that #12 “hallway usability testing” is not a very common practice and #9 “best tools money can buy” is perhaps too extreme, so I would guess that ten would actually be a very good score.

This test is frequently cited, yet some large successful projects don’t actually score well with this measure. Consider Linux:

Andrew Morton, Linux kernel maintainer, indicated in an interview at LINUX Format, that the Linux operating system had been developed without source control:

LXF: Do you think it was good to have had the time with BitKeeper in kernel development, or should they have stuck with CVS?

AM: Well, we never even used CVS. Before BitKeeper we basically weren't using anything - just a bunch of patches sitting on Linus' hard drive and it uploaded occasionally. We had no tracking of what had gone in the kernel at all. Which I always thought was pretty poor form; I wish Linus had used CVS from day one. He hates CVS, he has real problems with it, but for just a simple linear evolving model I think CVS would have done the job and at least we would have had some tracking history.

I think the main thing BitKeeper brought is it got Linus using something, and even if we used one of the free revision control systems I think we would have been 90% of the way there…

A post last month in Slashdot reported that “Linus Says No to ‘Specs’.”

A 'spec' is close to useless. I have _never_ seen a spec that was both big enough to be useful _and_ accurate. And I have seen _lots_ of total crap work that was based on specs. It's _the_ single worst way to write software, because it by definition means that the software was written to match theory, not reality."

Eric Raymond bluntly criticizes Linus and his undisciplined development process in this email:

Your tendency to undervalue modularization and code-sharing is one
symptom.  Another is your refusal to use systematic version-control or
release-engineering practices.  To you, these things seem mostly like
overhead and a way of needlessly complicating your life.  And so far,
your strategy has worked; your natural if relatively undisciplined
ability has proved more than equal to the problems you have set it.
That success predisposes you to relatively sloppy tactics like
splitting drivers before you ought to and using your inbox as a patch
queue.

Lack of source control and specs are huge. I suspect a whole host of other violations of the Joel test as suggested by LinuxToday, yet magically we have a major operating system. Perhaps, despite its development practices, Linux benefited from the widespread need for an alternative, free operating systems to Windows.

I don’t have real world experience with extreme programming to know if that methodology eschews most of the points in Joel’s test.  If so, it would be another example where a development team could be highly successful despite performing miserably on the Joel test. XP doesn't do upfront design, so it may fail the specs test. Programmers work and communicate in pairs, so working conditions aren’t supposed to be quiet. XP emphasizes unit testing at the expense of independent testers.

Comments

[Drazen Dotlic]

>> I suspect a whole host of violations of the Joel test, yet magically we have a major operating system.

This is a well known paradox. The less strict things are the faster they get wider adoption.
Have a look at RSS as a standard. It's mostly open to interpretation of each and every publisher, does not have strict schema, many people publish what's not even a well-formed XML... yet it is the most widespread syndication format.
Contrast that with Atom which is a lot more precise and strict, younger and thus is some circles more cool and still Atom is distant second syndication format.

Technologies succeed often in spite of their simplicity and not thanks to their quality. Same goes with methodologies - people do/use what is simple and works.

[Geoff Webb]

Scoring well on the Joel Test is more a measure of a team's capability then an indicator of any one project's success.

Some projects succeed in spite of themselves. A team that scores well on the Joel Test is able to consistantly produce high quality software.

[Adam Keys]

I don't want to be a zealot, so I'll try to keep this brief.

"XP doesn't do upfront design, so it may fail the specs test. Programmers work and communicate in pairs, so working conditions aren’t quiet. XP emphasizes unit testing at the expense of independent testers."

XP doesn't do detailed upfront design that produces a detailed spec. There is still room for deciding which features are the most important to build first, with it the implicit need to understand what is important. To say XP doesn't do design is a very poor choice of words.

Similarly, to say XP unit testing comes at the expense of independent testing is another poor choice of words. Nothing in XP says you can't have a traditional testing/QA team. XP simply suggests that there is another class of testing that is useful and productive.

[Mike Gale]

I saw a pretty harsh piece by Paul Graham yesterday.

It basically said that really competent developers don't need a lot of the crutches of modern software development. They are good enough to "just do it". In contrast he says typical corporate developers are not much good and the tools stop them really fouling up the project.

These ideas are worth considering.

Basically there are a lot of ways to develop software. Some suit some people some suit others. The current crop of OO, Unit...-Tests, coverage, NDoc... Can swamp the process and turn a motivated high productivity programmer into a low productivity drone. Especially on slow hardware.

Joel's opinion is based on what he knows and does. It doesn't apply everywhere and to everyone. Graham's opinion doesn't apply universally either. (The test is stated too dogmatically, in my view, and for small teams is probably way off target more often than not!!)

The essence is to understand the issues, understand the team, understand the tools and do what seems right. If you push the limits it's good, if you don't you may be heading to the drone park.

[David Douglass]

If only it were this simple. While the list certainly contains some good ideas, the health of something as complex as a software development organization can't be measured by a little quiz.

[Wesner Moise]

Related posts:

The Joel Test and Agility
http://www.agileprogrammer.com/dotnetguy/archive/2005/11/27/9846.aspx

What say you Agile folks?
http://randyh.wordpress.com/2005/11/27/what-say-you-agile-folks/