BDUF Waterfall vs Lightweight Agile vs the Cowboy Way

I have been working on a new whitepaper about how to become an online leader. One of the cornerstones is to use a strategic, agile process. While agile is not for every project, often it is by far the optimal process for web development. While many large development organization such as Microsoft, IBM, Google and Yahoo! use agile, very few companies that don’t do software development for a living are using it – particularly small and midsized businesses. I am tasked with how to promote agile development to companies needing a new website. So it was perfect timing that the latest edition of Information Week included “Agile Development” and “Agile Processes Go Lean” articles. The articles did not disappoint. They provided plenty of great arguments for the move to agile.

Instead of collecting, collating, and casting in concrete everyone’s 2 cents worth before launching development, agile developers take a more flexible approach. They work one-on-one with stakeholders who make decisions, provide information, and prioritize requirements throughout the process.

The end result is a much leaner, more focused requirements specification that includes only the features that stakeholders expect and developers can deliver.

Agile is perfect when you’re not sure what you’re getting into…If you run into a big road-block, you’re only derailed, on average a week’s worth of work.

I was brought up on Big Design Up Front (BDUF) waterfall methods. The Info Week articles speak to me. I have worked under some great project management organizations and have seen what it takes to deliver great projects. In the 90’s BDUF driven waterfall was the dominate method when it had to be done right. It is also very expensive and very ridged. Great for big budget, mission critical project such as online backing and government sites but it can kill the ROI of a public facing website. The web changes too often, stakeholders evolve there thinking or new stakeholders are introduced to the project, customer needs change. The web is evolving at an incredible speed and the process needs to keep up. I have read dozens of great arguments of agile over waterfall and lightweight specs over heavy. The proof is compelling for website owners to move from waterfall to more agile processes except for one thing: most websites were not development with heavyweight BDUF specs. The truth is that the vast majority of existing websites used the cowboy code methodology. In other words, no formal process, no formal specs. Those that develop software for a living know the value of rigorous planning regardless of whether it is a BDUF or lightweight. However, most small to mid-size companies have little experience with custom development. Their website might be the only custom technical project they have any experience with. As web developers, we are often a part of the problem. Most of us were never taught in school formal project management – and unless you worked under a world-class consultancy, we were never taught on the job either. Many developers are self-taught experts in graphic design, markup, JavaScript, programming, online marketing and so on – not actual managing of projects. Too many developers and clients think great project planning and management is something that should just happen when you have smart, qualified people on a project. For small, simple projects there is some truth in this thinking. However, no one became an online leader with a small, simple website. The expectations for a typical website are for a client to provide a list of high level requirements and for the vendor to provide a fixed bid on delivering those requirements. Companies that do custom software for a living know that this is a pipe dream. The Standish Group’s CHAOS Report has methodically proven that the [[iron triangle]] is a not achieved more than 80% of the time even with BDUF. It is essentially hopeless with no proven process for planning and management on all but the most trivial projects. Those experienced with development know that there are many sources of unpredictability. The web is evolving quickly. Stakeholders, customers and the competition demands evolves rapidly, adding to the uncertainty in Web projects. Some companies deal with unpredictability by gathering more requirements, the waterfall method, or planning to respond to them in iterations, the agile process – but the worst thing to do is to ignore unpredictability all together, the cowboy way. That brings me back to my search for 3rd party authorities to aid in my argument for agile. All the great articles assume you already know the value of great planning and project management, so the arguments just come down to BDUF vs lightweight. What most small business owner’s need it know is that great planning and execution does not just happen, there needs to be a rigorous process – oh, and by the way it should be agile. Or maybe Martin Fowler said it best in UML Distilled:

You should use iterative development only on projects that you want to succeed. … it is an essential technique, one you can use to expose risk early on and to obtain better control over development. It is not the same as having no management, although to be fair, I should point out that some have used it that way. It does need to be well planned.