Agile vs. waterfall methods

I’m regularly asked by my customers to advise on the benefits of agile vs. waterfall methods of software development, and which method they should adopt (or continue to use). In my response I regularly use the phrase “horses for courses” in my discussions, as there is no simple answer – it really depends on what you want to develop, who’s developing it and the expectations of (or contractual relationship with) the person/organisation funding the development.

From a technician’s point of view, the agile approach lets him get down to the bits and bytes quickly, using the technology to help him work with end users to collect requirements and build solutions to satisfy those requirements. If you have skilled technicians, either with a high level of understanding of the requirements (unlikely) or with a very good working relationship with end users who not only know the requirements, but also have the ability to spend significant time with the developers, then you may have a project for the agile approach. If those end users are also the project sponsors and are funding the development, then the case for agile development gets stronger.

If, however, the development is to be built in a client-contractor environment, normally against a fixed price or firmly set estimates and timetables, then the level of risk in using agile methods increases. Add in larger contract size, leading to larger teams, and more remote communication with end users, then going down a waterfall approach can quickly become the preferred route.

For application package developers the problems become even greater – they have no one customer, but look for a solution that can handle multiple sets of requirements. Then they have to not only maintain, but also enhance the package over many future years, in many cases without the designers and technicians who wrote the package initially. Also, and perhaps most importantly, senior management are unlikely to sign blank cheques for the development of new packages without some degree of certainty that the developed product will meet their target customers’ requirements, be marketable and delivered within an agreed budget against a realistic time schedule.

From my own experience, I’ve always believed that the coding stage is, surprisingly enough, one of the least important stages – as with any stage, if you get it wrong the project will suffer, but it will suffer less than if you get the requirements collection wrong, or adopt the wrong design. Coding is used to implement a design – no matter how good a coding team is, if it has a poor or incomplete design, it is highly likely to produce a poor system.

Ever since my early days of developing software the 50-30-20 split of work (50% of the effort spent on requirements collection and system design, 30% coding and unit testing, and 20% system/acceptance testing) has remained curiously static for new software developments over the years. (Before I get a lot of comments - I note that the development of new versions of, or replacements for, existing products means that a large proportion of the requirements capture has been completed already – however, the success of any new development is still dependent on the overall design being absolutely right).

Evangelists of agile methods will point to large projects developed using waterfall methods, where the requirements collection and analysis phase goes on forever in an attempt to capture every last requirement before the development starts. Then any new project spends so long in development, that by the time it is complete, the requirements have changed and the solution no longer meets the users’ needs.

Exponents of waterfall methods will point to agile developments where the level of rework to build in missed functionality has been many times the effort that would have been required if the requirement had been identified before coding started, to projects where the quality assurance team weren’t able to start to generate their test plans until just before the development was complete, or systems where the user and technical interfaces were inconsistent across the code built by separate individuals.

The advice I give my customers is based on the size of the project, the size and skills of the team, the expectations of the funding organisation/person, and, most importantly, the criteria for success. If I advise a waterfall approach, I encourage the use of modern software development tools to produce (normally disposable) prototypes during the analysis and design stages (one of the – waterfall - projects that I reviewed was using Microsoft Expression Blend for prototyping – it just blew my mind, and the prototypes could be used in the later stages of development). I also encourage the use of high levels of automation for documentation and testing processes.

If I advise an agile approach, I stress the importance of an overall scoping document, with clear standards for look-and-feel, navigation and interactions between modules. I also look for strong management and ownership, not only of the project but also of all common aspects of the development, be it the database or common classes. Also, no matter how good the developers and the methods adopted, I still recommend a separate testing team.

In all cases I emphasise the involvement of the main project sponsor to decide on scope, give clear direction to the team, and help to avoid requirements creep. All too often, early expectations are set far too high, and once reality sets in, when forecasts seem unattainable and budgets insufficient, there needs to be open communication between the team and the sponsor to arrive at an acceptable solution.

All too often problems get hidden – “we’re behind but we’ll recover” - and it is only later in the project that confidence crumbles and reality sets in. Unfortunately with agile developments it is difficult to regain management confidence in revised project plans due to a lack of measurable performance to date that can be extrapolated towards a final completion date for each phase. It is at this time that a good and involved project sponsor can be the saviour of a project – all too often I see sponsors returning to a ‘hands off’ position and failing to get involved – making a problem project into an even bigger problem......

P.S. As well as providing a “fire fighting” service for problem projects, I also provide consultancy to help new or existing projects proceed successfully and avoid becoming problem projects. If you would like to discuss ways of avoiding problem projects in the future, please contact me at Phil@systemsolveconsultancy.co.uk