Aug 30, 2021
Getting over the Waterfall
Deconstructing sequential development to better understand more continuous development
Sequential development, as typified by the waterfall model, is the common punching bag of anyone supporting a more adaptive approach to the creation of software. In spite of the increased popularity of iterative and incremental approaches over recent decades — from the classic spiral model to agile development, from RAD to Kanban — waterfall thinking and practices still colour the way software is developed and waterfalls still represent, in practice, the most popular formalised approach to managing projects — rivalled only in popularity by the two most popular informal approaches: unmanaged and mismanaged. Neither the metaphor of a body of water crashing inexorably and forcefully onto the rocks beneath nor its poor track record appear to have removed it from popularity or consideration.
Of course, there are many businesses and teams that would claim to be flying the Agile flag (or the Lean flag or the DevOps flag or…) rather than the flag of waterfall development, but these are, in many cases, unwitting false flag operations — waterfall projects rebadged with new terminology, more urgent time scales and increased micromanagement.
It is, however, all too easy to mock or demonise the waterfall approach without understanding its motivation and attraction. Without such understanding, advocacy and adoption of a more agile approach can be superficial, dogmatic and propaganda based, which misses much of the deeper rationale and actual value of a more flexible and responsive way of doing things.
It is not my intention to promote sequential development as a general approach; it is my intention to promote understanding of development by appreciating the specifics. Any honest criticism of it in favour of something else needs to begin with a more even-handed appraisal.
Riding the waterfall
The organising principle of sequential development can be summarised quite simply: strictly align development activities with phases in development. This principle is tidy, clear and easy to explain: given a number of different activities that occur in development, combined with the recognition that a development effort follows a lifecycle that can be characterised differently at each point in time, divide time into phases, characterise each phase in terms of an activity and run those phases in sequence. That’s it.
Such a model of development is simple: it is easy to lay out, easy to explain and easy to present visually. At any given point in time it is clear what activity is being performed. The sequence of activities seems not unreasonably organised. Problem discovery and reasoning comes before problem solution and execution, which in turn precedes final confirmation that the right thing was built, just before proceeding onto deployment.
This model makes perfect sense if you make some important assumptions: the problem space is inherently stable; the problem being addressed is fully understood by all parties (and fully understood in the same way by each party); the approach to the solution is well defined and fully understood (in the same way) by developers; the technology to be used is fixed and fully understood by developers and its use is guaranteed to be free of surprises; the development can be structured as a project with an end state based on a final delivery. If you can guarantee those assumptions, you can ride the waterfall and keep your head above water. But if you can’t, the raft of assumptions quickly unravels into something far less watertight.
Walking on water and developing software from a specification are easy if both are frozen.
— Edward V Berard
Let it go
Software development is typically a multi-variable problem with few guaranteed constants. Tackling a dynamic situation with a static plan is a recipe in risk, easily upset by the slightest change. Change comes from many sources: the market or the customer; the surrounding development organisation; the team; the technology; knowledge from actually developing the software; and so on. Change is normal and real; the absence of change is either an anomaly or an idealised thought experiment.
Sequential development models contain elements of the truth we’re after, i.e., the activities and notion of breaking up time, but mixes in other constraints that work against the whole, i.e., strict allocation of activity to time period and strict sequence of time periods. Such time specialisation also leads to role and team specialisation. Instead of software development teams organised in terms of software development, they are specialised and separated by specific activities — documenting requirements, writing code, system testing, etc. — which bottleneck easily for want of communication.
The waterfall approach has the laudable aim of attempting to derisk unknowns by exploring the problem in detail at the start of the lifecycle, with a final check on things at the end. Unfortunately, in practice the effect can be the exact opposite: instead of derisking at the earliest possible opportunity, risk is pushed and accumulated towards the back end of the lifecycle. Unless the level of detail includes code, then the detail attained in the early stages is likely to be superficial, incomplete and readily contradicted. Code represents a commitment that raises questions about requirements, challenges assumptions about technology and has a habit of revealing that the devil is in the details.
By signposting progress in terms of abstract goals — documents, meetings, sign-offs, etc. — rather than executable software, waterfall can end up being an approach that avoids detail and commitment rather than embracing it. Regardless of the intent of spending a lot of up-front time planning to avoid unintended consequences, the law of unintended consequences usually supervenes to increase rather than decrease risk over time.
In a sequential approach, significant decisions are made at the start — this is the point of least knowledge of what is involved in developing a system, both in terms of tangible requirements and technical requirements. At the point of greatest knowledge — the end — the chance for effective change has all but vanished. Time delivers knowledge but reduces options and opportunity. A sequential approach locks this in by further suppressing options up front and, inadvertently, steepening the knowledge acquisition (a.k.a. learning) curve at the end. The knowledge of what to build and how to build arrives late; the opportunity to act on this knowledge is long gone.
This is not to say that there is no merit in emphasising problem discovery and architectural foundation early in development, just that these activities are not the exclusive preserve of the front end of the lifecycle — whether for a whole product or a significant release — and that the results of these activities are not set in stone. It is precisely because of the uncertainty surrounding these issues that you want to start them early and keep doing them continuously. They are never done.
A work is never truly completed, only abandoned.
— Paul Valéry
The reason you engage in a development lifecycle with repeating feedback loops is to give yourself the opportunity to clarify and converge as you go, replanning and redesigning as you learn rather than being caught off guard when you’re supposed to be done — and then being caught in a self-perpetuating cycle of promises and “almost done, just one more thing, no really, this time…” This is the wrong kind of iteration.
Ends justify the means
There is also another assumption baked in: that there is a well-defined end. Although the word project is used throughout the world of software development — from IDEs to books — only a minority of software development actually meets the bounded and goal-oriented definition of what a project is and, therefore, very little of what is called project management in software development is actually project management:
Project management is the application of processes, methods, skills, knowledge and experience to achieve specific project objectives according to the project acceptance criteria within agreed parameters. Project management has final deliverables that are constrained to a finite timescale and budget.
A key factor that distinguishes project management from just ‘management’ is that it has this final deliverable and a finite timespan, unlike management which is an ongoing process.
A software system may have a beginning and an end but, for most systems, few developers will be present at either, let alone both.
Planning the plan
It is perhaps telling that, in response to the surge of interest surrounding agile development in the 2000s, some advocates of waterfall-style development favoured a rebranding of sequential process models under the umbrella of plan-driven development. The intended implication being that agile development was unplanned, and that unplanned has enough negative connotations that we wouldn’t want that, would we?
A more accurate reading, however, is that an agile development lifecycle is not driven by a plan, and it turns out this is not wrong: agile approaches involve planning, where planning represents an activity that is ongoing and responsive to the emerging situation. They may be considered planning driven, but not plan driven — a subtle but important distinction. Being driven by a plan is fragile and risky when what are assumed to be constants are actually variables. The role of the word plan in agile processes is better understood as a verb than a noun: planning is an activity that is pervasive and continuous, not a static artefact produced at an early stage as input to later phases. The metaphorical entailment of plan-driven development is perhaps closer to the idea of planned economy than anything else.
That said, it would be disingenuous to say that plan-driven approaches exclude the possibility of modification or revision to an initial plan in the light of new information, changed circumstances and variations in progress. The attitude to such changes, however, is one of reluctance. Changes are corrections — irritations and exceptions to be tolerated rather than the normal state of affairs.
In spite of much experience and published wisdom to the contrary, estimates are still often treated as predictions or commitments rather than as forecasts or projections. Some amount of change and uncertainty is acknowledged, but it is a grudging and partial acceptance that does not inform the overall mindset or nature of the development process. Given the typical human response to change and uncertainty, this is hardly surprising: this trait is within each of us to a greater or lesser extent.
The shape of water
All this suggests that the management of incremental development — particularly processes intended to be streamlined and responsive — is not necessarily the path of least resistance. It sounds hard: it is continuous; it is constantly buffeted by change; it can come into conflict with human nature. The apparent alternative of following a plan that lays everything out in a predictable and tidy sequence — abstracting out interference from change, discovery and human nature — does indeed look simpler. But appearances can be deceptive.
What at first sight looks simple can reveal itself as simplistic under the duress of reality. Rather than being an approach for reducing development risk, applying waterfall development to most software is a methodology for increasing development risk.
A precondition for applying a sequential development model successfully is that the work is project-based, so it has a clear end state, and that pretty much everything needed to undertake the development is known, and that knowledge is stable. Anything that is unknown or not stable cannot be critical. If all these things hold true, waterfall may be the answer. The problem is that the question is rarely asked or, if asked, not answered with appropriate awareness or humility. Often without meaning to, significant considerations are abstracted away. As I noted previously:
A good abstraction is a simplification that allows us to reason about a concept or work with a design more effectively; a poor abstraction is one that misleads us either by omitting necessary detail or by including unnecessary detail, drowning signal in noise.
In this case, the appeal of the highly planned model comes from ignoring or overlooking some fairly critical details— details that if taken into account would, by necessity, change the needs and nature of the development process, details that cannot be known in advance and can only be revealed by attempting to create the system. We change the world by creating things in it, and the world changes the things we think to put in it.
So yes, the management of an agile process sounds hard, but that’s because it is. That’s a property of software development rather than specifically of agile or related approaches. But if you think that’s hard, using a mismatched model of software development makes it even harder. Where jumping a waterfall requires a leap of faith and crossed fingers, most software needs smaller steps to ensure footing and gauge the next step.
There is an irony that many current trends in development processes and practices— from continuous testing to continuous deployment, from agile processes to lean thinking — are truer to the metaphor of a waterfall than the so-called waterfall model. In the real world, a waterfall is a continuous flow, always receiving input, always delivering. Except in conditions of water scarcity, waterfalls in the real world are not batch-based. A discrete mass of water does not pour over the edge, progressing through different phases as it falls, with the next mass of water waiting until the previous one hits the bottom before it follows. Reclaiming words is not easy, but waterfall might be one to consider.
