This is fourth of part 1 in a 3 part series. Please read Agile — What isn’t told when sold, Agile is dead, again & again and Agile methods (alone) don’t scale to understand the context for this article.
Waterfall to agile — what’s the big change?
Let me get few points straight across to you. I was a project & programme manager for most part of my life, religiously following Prince 2 and other Waterfall / plan-driven / defined methodologies. I made a living out of it. I don’t have distaste towards Waterfall. It has its place — still useful under certain circumstances. In fact, none of Waterfall methodologies said “don’t deliver value, don’t work together, documentation is all you need, don’t reflect or retrospect” etc. But the way we implemented those Waterfall methodologies didn’t, for most part, deliver what we expected. In my humble opinion, I don’t see any conflict between the values and principles of agile and waterfall methodologies.
The first question we need to ask ourselves is — if there is no conflict, what’s that we are missing, why do we need agile methods, Scrum in particular? Next, why do I single out Scrum?
Let me start with the latter. Though most of my findings are true for any agile method, the reason to single out Scrum is that it is now the dominant strain. Martin F quipped[i]
In the early days of the agile virus, the dominant strain was Extreme Programming, which has a lot to say about technical practices. Now the dominant agile strains are Scrum and Lean.
As Scrum is the dominant virus, I’d like to describe how Scrum became the new Waterfall 2.0.
Exhibit 1: Team level Practices
Backlog and Business Requirements Document
Scrum Guide says:
The Product Backlog lists all features, functions, requirements, enhancements, and fixes that constitute the changes to be made to the product in future releases.
When the traditional software world hears this, they’d say “Great! That’s where our requirements document goes”.
Estimation & Project Plan
Product Backlog items have the attributes of a description, order, estimate, and value. Product Backlog items often include test descriptions that will prove its completeness when “Done”.
What a Prince 2 or PMP project manager wants more than this? You have [all] features estimated in some weird way (some call it scientific way because they use Fibonacci sequence). Lay it over a timeline. There you have your lovely Gantt chart. All you need to do is just schedule endless refinement & planning meetings, get the team to estimate and you are done. You can do this over and over again every week or two. You could have never achieved in the traditional project management. Lovely isn’t it?
By the way, I’m not judging the pros and cons of Gantt chart or a well defined plan, just providing a view on how similar they both are.
Daily Stand ups & Progress Tracking
This is the place where the project manager (PM) or scrum master (SM) asks the team what is happening and then tell them what to do for rest of the day. Every member is expected to say what they did yesterday, what’s their “commitment” today. PM gets to command why they didn’t deliver what they committed. Yes, Scrum Master is there as a sheepdog to get everyone to a place, circle around PM and take notes so that PM has track of what is committed by team every day.
That’s practices at “team” level. Let’s consider what’s happening at enterprise level.
Exhibit 2: Programme / Enterprise level realities
If you are ever close to a reasonably large project or part of one delivered using Scrum, you’d have observed this:
1. Some big-titled business executive comes up with an idea or fed one by his / her team. This will be socialised among other big-titled executives. All agree that it must be tabled at next investment board, which usually happens once a year
2. A business case is ordered, reviewed at investment board and gets approved
3. A team is set up — either internal or from an external Partner (the usual names are 3rd party, agency, vendor, systems integrator, service provider, resource provider, but “Partner” looks cool)
4. If Partner is engaged, procurement & legal teams get involved, even for an established Partner
5. From then onwards it is the usual routine
5.1. Programme Board and Programme Managers are appointed. They in turn appoint Product Owner, Scrum Master and Project Manager (yes, both SM & PM). Then development teams are constituted
5.2. By the time Product Owner assumes the office, Scope (Epic / Road map Item / Contractual Deliverable etc.) and programme plan are in place, with milestones and deliverables etched into contracts
5.3. Then Business analyst(s) start developing detailed requirements.
5.4. Architects design a solution once the requirements are handed down to them often isolated both from analysts and developers
5.5. Dev team starts sprinting. SM sprints in different directions to get meeting rooms and distribute endless velocity & burn-down charts
5.6. Dev team somehow hands over something to testers and bugs happily find their place under “tech debt” moniker
5.7. Sprints in multiple of tens are done prior to handing over to “service introduction or route to live” teams along with tens of documents & checklists. For those release teams “real requirements” are now being gathered
5.8. Then the fun of “user acceptance testing” starts and eventually finding the piece of functionality on production systems.
And we declare — we’ve gone Agile!!!
Exhibit 3: Scrum & CMM
Two main camps of Scrum Framework — Scrum, Inc. and Scrum Alliance started a cosy relationship with CMM, the dreaded process heavy framework. In fact it is dubbed as the “Magic potion for code warriors” by none other than Jeff Sutherland[ii]. There are similar sentiments expressed by Scrum Alliance camp as well[iii]. IEEE has a paper, mapping CMMI Project Management process areas to Scrum Practices[iv]. Though I didn’t read the full paper, the abstract is sufficient enough to put you off, especially if you believe in The Manifesto’s values and principles.
Though I still need to see the full analysis of how Scrum fits in with CMM, there is a clear change of heart here from the Scrum camps. In their article published in December 1998 “SCRUM: An extension pattern language for hyper productive software development”, they clearly refute the core principles of CMM. They said “The CMM defines five stages of process maturity – initial, repeatable, defined, managed and optimising, and asks its users to define the processes of 18 KPAs (key process areas). However, many of us doing work in the trenches have found over time that the “repeatable or defined” process approach makes many incorrect assumptions, such as:
1. Repeatable / defined problem. A repeatable/defined process assumes that there is a step to capture requirements, but in most cases, it is not possible to define the requirements of an application, because they are either not well defined or they keep changing.
2. Repeatable / defined solution. A repeatable/defined process assumes that an architecture can be fully specified, but in reality it is evolved, partly due to the fact of missing or changing requirements (as described above), and partly because of the creative process involved in creating it.
3. Repeatable / defined developers. The capabilities of a software developer vary widely, so a process that works for one developer may not work for another one.
4. Repeatable / defined organisational environment. The schedule pressure, priorities (e.g. quality vs. price), client behaviour, and so on; are never repeatable or defined
From that standpoint when they started to the current alignment to CMM, you can clearly see how Scrum truly became the new waterfall — Waterfall 2.0.
Exhibit 4: Going back to FW Taylor
I’m further disheartened to see the new research by Jeff Sutherland and others titled “Process Efficiency — Adapting Flow to the Agile Improvement Effort”[v]. This research promises orders magnitude improvements if you stand next to developers with a stopwatch. In my humble opinion, the entire agile movement is a backlash to that Tayloristic thinking. Consider these tables from the article:
This literally tracks the team down to the minute. Who in the right mind would stand next to each team member of the team with a stopwatch? How feasible and how cruel? Isn’t this exactly FW Taylor did in Midvale Steel Company? What happened to “Humans are not resources”? Is the thinking time to write better code is waste? Is writing code is the only valuable task, what about time spent on analysis, feedback loops?
Don’t you think Scrum already started to stink?
Scrum moved from its foundations of “complex adaptive systems, punctuated equilibrium and empirical process control thinking” to more defined ways of thinking, making predictability and productivity as end goals.
- Decision making still rests with big-titled execs, who doesn’t understand the implications of those decisions
- Scope & time are etched into contracts
- Skills required to deliver the scope are still isolated from one another, especially Architects, Release Management and down-stream Operations — making coding & test teams sandwiched between immovable brick walls
- Team members are told what to do on daily basis often without knowing the big picture — why they are doing what they are doing
- Story points became the metric to achieve, in the place of business value
This is further exacerbated by pyramid schemes of certification by both the camps. Contract Negotiations moved from time and money to story points and velocity. Measuring success moved from tracking activity (plan- driven) to tracking output (Scrum). Nothing else is changed in terms of programme / project delivery, Scrum the new Waterfall.
All this doesn’t mean that Scrum / Agile implementations are a failure. Agile thinking and methods did clean up a lot of mess in the middle that we see in any technology driven programme delivery. Scrum in fact made the dysfunctions quite visible. We should give it that credit. What we need to understand is that those dysfunctions won’t go away magically just because you are doing Scrum. Wrong patterns & frustrations I mentioned in previous articles are the result of that false hope or wrong expectation.
References
[i] https://martinfowler.com/bliki/CraftmanshipAndTheCrevasse.html
[ii] https://www.scruminc.com/wp-content/uploads/2014/05/Scrum-and-CMMI-Level-5-A-Magic-Potion-for-Code-Warriors.pdf
[iii] https://www.scrumalliance.org/community/articles/2008/july/agile-and-cmmi-better-together
[iv] https://ieeexplore.ieee.org/document/4402760/authors#authors
Enterprise Transformation 