Does Agile Suck in some Cynefin domains?

Here’s an article from Bertil Muth on freeCodeCamp on why Agile isn’t working at your company.

This contains a lot of great, level-headed advice about how to excel at Agile—by which Bertil seems to mean ‘Scrum’. The assumption seems to be that Agile (which equals Scrum) is great for everyone! But is it? Might Agile methods not be applicable for some projects?

Agile is all sorts of things. Even Scrum is all sorts of things. What I am interested in here is not so much the various artefacts and ceremonies themselves, but in how they advance the Lean principle of keeping Design In Process Inventory as low as possible; that is, don’t create something in advance of when you need it. This applies to code, but also design, estimations, detailed product requirements and so on. Scrum is pretty good at allowing this Lean principle to work provided everyone agrees to do it. If someone in management decides that all design is a waste of time and that Scrum gives them license to insist on cutting code without it, that probably won’t end well, depending on the need for design, of course (see a previous post about the dangers of headless agile, where Scrum is used without governance).

So, I’m not going to be arguing that, say, daily standups are appropriate in this situation but not in that or when the cadence of Scrum beats the flow of Kanban-like methods, but we will be talking about when and where it is appropriate to use an Agile methodology to support low Design In Process Inventory.

I have been interested in whether Dan Snowden’s Cynefin system is a good lens through which to view high-level governance problems such as this one, so let’s give it a whirl.

What even is Cynefin?

Think of a yin-yang circle. The swirl of order and chaos. Cynefin splits ‘chaos’ into two, the true Chaotic domain, where your actions do not affect outcomes in any predictable way, and the Complex domain, where the effects of your actions can be guessed at but there is uncertainty. Cynefin also splits ‘order’ into two, but on a different axis. The Simple, or Obvious domain is where appropriate actions can be taken by non-experts to have a predictable effect. The Complicated domain is where appropriate actions can be taken to have a predictable effect, but only by experts with deep understanding. There is also a fifth domain called Disorder, in which you do not know which domain you are ‘really’ in. Once you decide in which domain you are, you can determine how to act.

For me, the most important take-away from Cynefin is that everyone prefers a leadership style that is appropriate for one of the four main domains. The process-heavy style—if it went wrong the process needs changing—is applicable to the Simple domain where correct actions can be taken by non-experts. The planning style—if it went wrong we didn’t think about it hard enough—is appropriate to the Complicated domain, where the problem is difficult but similar enough to previous problems that the lessons carry over. The experimental style—if it went wrong we didn’t learn lessons as we went along quickly enough—is appropriate for the Complex domain where situations are novel but discoverable with persistence. The dictator style—if it went wrong it was because people weren’t following orders—is appropriate for the Chaotic domain where the situation’s difficulties are emergent.

Scrum and other Agile techniques enable some leadership styles and inhibit others.

In which domain is Agile appropriate?

Snowden talks of Scrum as being an excellent tool for moving from Complex to Complicated. Certainly the ideal domain of Agile techniques is the Complex domain, as you proceed by conducting repeated experiments. One sort of experiment is to implement a user story and put it in front of that user to find out if they like it. Another is to implement process changes as a result of retrospectives, checking if the changes improve the development experience in the following sprint. So yes, the Complex domain seems ideal for Agile.
But note that there are two different levels here, the governance and problem domain level and the code and solution domain level. In the ideal case just described both are Complex domain. There is another approach to Agile techniques: Just apply the process and hope! In this case we are hoping that a backlog and a set of sprint ceremonies is all the management we need, which is a Simple domain assumption.

Examples of projects being in different domains

Let us consider what it would mean for software engineering itself to be in each of the five domains, then consider the same for the governance of a software engineering project.

Chaotic Software Engineering

Software engineering is chaotic when any action has an entirely unpredictable effect. You can look at this (unverified!) post on Hacker News about working on the Oracle database, and decide whether this story represents Chaotic software development. Each bug fix requires two weeks of up-front work, and will break an unpredictable 100-1000 tests! Note that the author of this post almost describes Snowden’s ‘act-sense-respond’ prescription for working in the Chaotic domain. Would Snowden advocate cutting out the two weeks of up-front work, I wonder?

Complex Software Engineering

Software engineering is usually complex; action and effect are related but often surprising in the moment. Every software engineer is surprised by the effect of their actions on the system daily.

Complicated Software Engineering

When a client wants a system that is composed of well-understood pieces composed in well-understood ways, development is simply Complicated. Wiring together this server with that Content Management System, this authentication framework and that monitoring system is Complicated, not Complex. You just have to know how to do it.

Simple Software Engineering

User tools such as the various website builders are in this domain.

Disordered Software Engineering

If you have not seen software you need to use, update or install, you do not know which of the above is applicable, so you do not know how to develop.

Now for the governance of software projects, let us work in reverse order:

Disordered Governance

Not knowing how and why requirements might change means not knowing how to manage the project; this is disorder.

Simple Governance

If the tests pass, the client will like it. Adding small, well-understood tasks to an existing well-understood project falls into this category. Scrum works here, but then so does everything else.

Complicated Governance

A project where the client knows that they do not know what they want falls into this category. Experimenting with putting micro-solutions will not satisfy them, they need to know that the up-front work has been done to feel confident in continuing with the project.

Complex Governance

Perfect Scrum territory. Each piece of work is not only (hopefully) moving us closer to the final working product, but also discovering information both about the problem and solution domains. Rapid user and problem domain expert feedback is essential.

Chaotic Governance

Project management can be in chaos all too easily, especially if a project is a political football and be wrestled under the control of different individuals with different management styles.

In which domains is an agile approach warranted?

If the code is Chaotic domain, you shouldn’t be promising anything. Work on documentation, enforcing architecture or rewriting. Using Scrum would likely disappoint, as sprints repeatedly break, estimates are always incorrect and a velocity refuses to become apparent. Kanban (such as it is practiced in software projects) would not break, but neither would it be (on its own) a decent tool with which to undo the chaos in the code.

If the code is not in the Chaotic domain and the management is genuinely in the Simple domain, Scrum (or other Agile technique) with no oversight, architecture, design or planning (beyond sprint planning) will work fine. But then again, so will most approaches; you are not getting a lot of value from Scrum here.

If the governance is in the Complicated domain you can use a Lean approach too. However, you might want more upfront planning than that normally implies as it will help surface issues that the stakeholders might want some time to deal with in better time.

If the governance is in the Chaotic domain, do not think that using Scrum or any other Agile technique will solve your problems; what you will end up with is Acephalic Agile, as described by Andy Wilson’s post. You must solve management problems with management, not methodology.

Perhaps we can summarize this in the following table:

I cannot think of a reason why the domain of the solution domain, provided that it is known not to be Chaotic, changes the range of methodologies that should be considered. This surprised me, so please leave a comment if you disagree.

But for a ‘nice, normal’ software development project with non-Chaotic software and Complex-domain management, yes, by all means use Scrum or another Agile methodology. Just don’t expect it to solve your design, architecture or especially management problems for you.