Tue Sep 29 22:49:32 BST 2009
Until I started working for my current employer, it was rare for anyone to ask me for plans and time estimations for the projects I was working on. By and large, my projects were either so young and small that serious planning would have been extraneous; or so gnarly and mature that the main thrust of the software activities were reacting to customer requests rather than breaking new ground. Similarly, the organisations I had worked for were either so dynamic and hand-to-mouth that even medium-term plans were meaningless; or so large that most of the planning and estimation burden was shouldered by a thick layer of middle-managers.
As a consequence, it has been something of a culture shock to not only be asked for plans and estimations, but also to be expected to follow through on them more or less on schedule. Ever the chap to relish a challenge, I have been doing my best to improve on my planning and estimation skills by actively critiquing each plan I make once I have completed the work. Did it come in on time ? If not, how far out was I ? Which tasks took longer than predicted ? Which tasks were less time-consuming ? If I were to do it all again, could I plan it better ? Is there any way I could have spotted the problem areas in advance ?
So far, this has borne some fruit. For the relatively small, bounded undertakings I've planned up until now it has been clear that:
These observations make suggest a trend which bodes badly for my current project, which is a relatively large undertaking in an area I have very little experience of. To make matters worse, it is a rather important project for my manager, and he is keen for us to complete our work on schedule. Over the last couple of weeks it has seemed like our progress is diverging from the plan; and although we're making good headway it isn't quite in the direction I had anticipated. As such my confidence in the final delivery date is diminishing: if we're not following the plan, why should we expect to be done when the plan says we should be ?
All of which tends to beg the question: how can one plan sensibly for the unknown? On one hand, there is what I think of as the traditional software engineering approach of making an estimate, adding twenty percent, doubling it and hoping for the best. On the other hand, there are various more sensible-sounding alternatives such as the Agile methodology or Evidence-based scheduling.
Assuming that the "traditional" approach is just a variation on the theme of guessing, it would seem worthwhile to look at one of the more considered approaches. From what I've read, these seem to have certain features in common:
These ideas all seem worthwhile, and are surely good advice, but I feel they still leave something missing from the practical experience of forging into the unknown. In The Pragmatic Programmer Andy Hunt and Dave Thomas make considerable mileage out of the benefits of prototyping to explore a problem domain before starting work for real. This is in keeping with Fred Brook's suggestion to plan to throw one away -- the prototype takes the pain out of throwing code away.
In reality, though, it is not always practical either to build a completely disposable prototype, nor to throw away man-hours. In such cases my experience suggests that the process in building a new system in an unknown environment is rather akin to the process of an oyster growing a pearl. One starts with some small nugget of code which does almost nothing, and one attempts to add some function to it. For a while all is in flux as different approaches are experimented with, but over time that code solidifies and hardens around the original nugget. After a time, the codebase is a bit bigger and mildly more shiny and it's time to push forward again, expanding into a new layer. And so the process repeats, until you have a functional system, or a pearl if you happen to be an oyster.
Perhaps the final analysis of this piece is that actually planning something unknown is not feasible (unlike planning an oyster's pearl production rate, apparently...), and the state of the art would seem to be more focused on controlling and understanding progress rather than on scheduling a firm date for project completion. Something for me to bear in mind, perhaps, the next time I prepare an update on my project's progress.