Jim Highsmith wrote an insightful blog post about how waterfall project management is about "plan and do" whereas agile is about "envision and explore." This weekend I was working on a video statistics system for Tutr.tv. It ended up being another one of those ah-ha moments of why agile processes are so brilliantly effective and how "envision and explore" produces better results faster than the old "plan and do."
In my day job, I mostly do strategy and solution architecture work (e.g. requirements and planning) but still sometimes get to play developer. In the role of solution architect, I have always found creating detailed project requirements difficult, in large part because as a developer I am used to working through a problem by exploring. Only by exploring do I know what is possible and what can be done efficiently. Only then can I understand the detailed requirements for maximizing value.
Of course that is bass ackwards from traditional waterfall project management. In waterfall, you have to detail out the requirements by estimating (e.g. guessing) the logic and effort required with little information other than past experience.
One of the beautiful dynamics of agile is you don't sweat the planning small stuff. In a waterfall process, if I needed to specify a statistics system I would use cases, wireframes, and maybe even interface and class diagrams. Everything would be spaced out in detail before I started to do coding and before I truly knew what was possible.
In agile, I simply have a high-level need in the form of a user story: As a visitor I would like to know which videos are the most popular.
Time to go exploring
I knew what I ideally wanted: when someone pressed the play button on a video, the play would be tracked and then of course, reports galore. But first, I needed the tracking. The problem was that Tutr is embedding videos from 3rd party services such as YouTube, Vimeo, Blip.tv and Archive.org. There is no way to tell what is going on inside their players - or so I thought.
So I tried interfacing with Vimeo's API. After a few hours of frustration trying to unsuccessfully bend Drupal to get froogaloop working, I gave up for plan B (or is it C by now?).
While Googling through copious amount of Vimeo's API documentation, I wandered across documentation for their standard API. Ah-ha, it has views stats. While the stats from the provider wouldn't tell me how many views actually occurred on Tutr.tv, the stats from the providers would fulfill the user story of knowing which tutorials where the most popular.
I set up a cron to run each night to download the data points for that day. There remained a significant unknown - how much can I poll the APIs before being flagged for abuse? The API docs were unclear on this, so time to learn by experiment. I set the crons to run and went to sleep.
The next day I had my data, except for those non-data-sharing Blip.tv folks. Problem solved? Yes, but we can do better.
Looking over the backlog of user stories, I noticed the As a member I would like to track the videos I have watched. We implemented a watched flag to satisfy this story. It was a quick, cheap solution for this requirement. The problem is, most people were not using the flag buttons. They either didn't understand what it was or were used to the way video watches were automatically tracked on sites like YouTube and Vimeo.
When it was done, I picked up two bonus innovations - auto-watched flagging for members and network views stats from the providers.
If I had tried to "plan" this feature, I would have wasted time on a highly uncertain guess for the effort, spec-ed a more conservative version of the feature, and lost out on a couple of innovations. In the end, big design up-front would have meant more work and less value - and no real certainty in estimating.
The lesson: Stop "planning" what can't be planned with certainty (e.g. most website features). Quickly envision and get to exploring as soon as possible. Then document your agile wins. See how many you can rack up.