The Devil’s in Both (the details and the big picture)

Dreaming in Code, a newly released book by Scott Rosenberg that explores ‘why building software is so hard’, should be read by everyone involved in the creation of software. Unfortunately, it won’t be, which is one of the problems with the software industry. But more on that later.

Dreaming in Code is really two books in one. Primarily it’s about the development of Chandler, an open-source “Outlook-Killer” that’s been in development for 3+ years and, well, as Joel Spolsky says in his review of the book:

…it’s taken them millions of dollars and several years to get to where they are now, which is, they have a pretty buggy and incomplete calendar application that’s not very impressive compared to the 58 me-too Web 2.0 calendars that came out last year, each of which was developed by two college kids in their spare time, one of whom really mostly just drew mascots.

So development hasn’t gone so well, to say the least. This post doesn’t set out to criticize or offer an opinion on that effort, as there are probably tens, or hundreds, of good reasons why after three years only a handful of people actively use Chandler. The subtitle of the book does offer a few clues, though:

• Two Dozen Programmers
• Three Years
• 4,732 Bugs
• …and One Quest for Transcendent Software

The other facet of this book is an study of the history of software development, and the ways people have approached or continue to approach creating software differently. Both parts of this book seek the same end though, to answer the oft-asked question – “why is building software so [insert adjective here] hard?”

I’ve personally been a part of a few successful software efforts, and a couple of failed ones, and a few patterns emerged from those experiences:

Good software requires a singular vision, and a singular keeper of that vision. The Chandler effort was plagued by ambiguity from the start, what were they building? Did anyone know? Did all of them have the same vision? Any successful software efforts that I’ve been a part of had either a benevolent dictator, a less-than-benevolent dictator or a team with a shared brain. Another manifestation of this is the fact that often great software is built by a team of one – where the vision is never lost in the details of communication. Even worse for the Chandler team, not only did the team not have a shared vision of what they were striving towards, it seemed that no individual did either. Rosenberg articulates this point perfectly later in the book:

…software development takes simple elegant visions and atomizes them, separating them into millions of implementation details and interface choices and compromises. The art of making software well is, in a sense, the ability to send a vision through that atomizer in such a way that it can eventually be put back together…

A good litmus test here would be to walk around and talk to your software team, asking each member what they’re building. Force them to describe the vision, not the details of a specific task. If you hear entirely different things from the team, you’re likely in trouble.

You can’t design good software by committee. Feedback is great; input is essential; people will naturally influence a design overtime. But an empty whiteboard and 13 people equals trouble. Send me the cliff notes version from that meeting, please.

Large software development teams are scary. I’ve managed teams of 15+ developers and the level of [mis]communication required to keep that many people moving in roughly the same direction is impossible. The energy and effort required drains away from productivity. The best work I’ve been involved with never included more than 5 active developers.

There’s one huge caveat to this rule though – service-orientation. Currently at Arc90 we’re building a platform for one of our clients, and the total number of people working on that effort is triple my comfort zone of 5. However, we’re really building distinct, separate, protected services that speak to each other in agreed-upon dialogues (APIs). This allows a larger team to build a composite application in a way that isn’t subject to the physics of a house of cards. So I’ll restate this as “large software teams, building a monolithic application, are scary.”

Was there a design for what is being built, before construction started? If you’re in software, you know how uncommon this (this being a design) is. If you’re not in software, you’d be shocked to know how uncommon this is. Generally, people start building before designing. And by design, I mean “prototype, architecture, API, interface”, not “fonts, colors, logos.”

Can you interact with the software after a few weeks? Or has your baby born when the project kicked-off moved on to kindergarten while you’re still waiting for beta v1? One of the goals of your project should be to get a live prototype that humans can interact with in weeks, this will influence the design decisions, shed light on problems or areas that deserve focus, allow you to remove entire elements that were thought of as important, etc.

Software development is often compared to construction, and the metaphor here is the model, where you build a model of the home before ground is broken. But software “models” are even more powerful, as you can’t sleep in the bedroom of the model of the home, but you can fully interact with a good software prototype.

Be careful of your developer’s resumes. This one’s tricky – the best technical people take what they’ve done in the past; the tools they’re experts with; the patterns that emerged from their previous work, and apply them appropriately to the new task at hand. Not-so-good developers take their bag of tricks and apply them entirely and always. You want a house made of wood? Too bad, I’m a concrete guy.

We collectively aren’t learning from the past, and are therefore doomed to repeat it. Rosenberg cites both Spolsky in the book as saying that “developers don’t read books about software development, they don’t read Web sites about software, they don’t even read Slashdot” and Richard Gabriel:

…what do people do when they’re being trained, for example, to get a masters of fine arts in poetry? They study great works of poetry. Do we do that in our software engineering disciplines? No. You don’t look at the source code for great pieces of software. Or look at the architecture of great pieces of software. You don’t look at their design. You don’t study the lives of great software designers. So you don’t study the literature of the thing you’re trying to build.

Now there’s a few reasons why this industry doesn’t learn from the past – most bad, and all good software is protected by the corporate veil, software is future-focused and run by inherent optimists (who cares why NASA spent millions and got nothing, we’re nothing like that!), software isn’t governed by immutable laws like gravity and the speed of light and there’s very little good writing on the subject. I’m a voracious reader, and can count on two hands the great books on software I’ve read in the few past years (and only two of them have a pencil drawing of an animal on the cover.)

Midway through the book, Scott Rosenberg comes to some sort of a revelation and offers us Rosenberg’s Law:

Software is easy to make, except when you want it to do something new. The only software that’s worth making is software that does something new.

That about says it all.