Flaws In Scrum And Agile

March 4, 2015Giles Bowkett

Pandas are not necessarily Agile.

The Agile Manifesto was written in mid-February of 2001, and it made the tech industry a better place. But it was written by people, not gods, and its day is fading. Likewise, the Scrum development methodology helps people build stuff, but it has flaws. So let’s talk about what Panda Strike does instead of Scrum and Agile, and why.

We’ll start with the biggest flaw in the Agile Manifesto.

Remote Work And Distributed Companies

Panda Strike is a distributed team. Distributed teams were unusual fourteen years ago, but they’re very common today. The Agile Manifesto states:

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

This implies that open source projects are less efficient and effective at developing software than commercial projects. And, to quote one of the posts on my personal blog:

It’s just not true. There’s a reason we write code onto screens, rather than transmitting it as oral history like an epic poem from before the invention of the written word. Face-to-face communication has a lot of virtues, and there are certainly times when it’s necessary, but it’s not designed to facilitate extremely detailed changes in extremely large code bases, and tools which are designed for that purpose are often superior for the task.

Such tools tend to be visual, rather than verbal. This is a GitHub diff:

When the Agile Manifesto was written, waterfall development was king. Agile deposed waterfall from its status as the dominant software development paradigm. Waterfall revolved around writing tedious documents, and face-to-face conversation was indeed superior to that. In 2001, Subversion was still new, Git did not exist, Skype didn’t exist either, and the closest contemporary equivalent to GitHub was SourceForge. “Face-to-face conversation,” in the Agile Manifesto, was “the most efficient and effective method of conveying information” because the other methods were terrible.

But when you’re reviewing a pull request, a GitHub diff usually beats face-to-face conversation. And this is not a fluke. Remote work requires a ton of writing, and that’s one of the best things about it.

How many times have you sat down to email somebody a question, and found that in the process of writing the email, the answer became obvious to you? This step is built in to the workflow of every remote team. Likewise, have you ever had to tell a co-worker the same thing twice? Technical work involves lots of tradeoffs, and people don’t always remember the intricacies of every debate. But you can refer to a conversation on GitHub three years later and review every detail with perfect clarity.

In a distributed company, the written word becomes much more influential than the spoken word.

At Panda Strike, we leverage this natural advantage with a process that keeps the good parts of Agile, but augments and replaces face-to-face conversation with technologies like IRC, GitHub, email, Skype, Hangouts, and our own GlideRoom. Waterfall used too much written communication, but Agile doesn’t use enough.

Our “Process” Is Mostly Just GitHub

The GitHub diff above makes a good example, because just about every software project on Earth today has a process which starts with GitHub, and in many cases ends with GitHub too. If your process isn’t just “use GitHub,” GitHub is still going to be the foundation of whatever process you use instead, 90% of the time.

This wasn’t the case when the Agile Manifesto was written, but if you’re using Scrum today, you’re having meetings in person so you can organize people who then go back to their desks and do most of their actual work through a toolset for remote, distributed teams.

The conversations and Scrum meetings which teams have outside of GitHub are becoming mere recaps, staged for the benefit of nontechnical participants, of the real conversations, which happen on GitHub or in IRC.If you write code on an in-person team somewhere, you’ve probably had the experience of having some nontechnical person walk into your space and remark with surprise about how quiet everybody is – not realizing that everybody else in the room is in the middle of a long and involved discussion on IRC. Scrum lags behind the modern toolchain enough that there can be a Potemkin village vibe to the whole thing.

And nearly every tech business today runs on open source software (created via distributed teams) and GitHub (created via distributed teams, and optimized for distributed teams). Because of this, nearly every tech company today stakes their entire existence on the productivity of distributed teams – even when they don’t allow their own employees to do any remote work at all.

So if you hear some company say that they don’t use distributed teams, the bad news is they probably believe it, but the good news is they’re wrong. Virtually zero tech companies have been built without work from distributed teams since Linux became ubiquitous in the 1990s. If your company uses open source and GitHub, your company depends on distributed teams. If you use any Apple products at all, they’re based on open source, which means you depend on distributed teams, whether you know it or not. The same is true if you use any major web browser.

And, since a core principle of the Agile Manifesto puts it in conflict with the entire open source movement – which has a much better track record of creating good software than any commercial methodology – you have to look at the Manifesto with a skeptical eye.

Agile Development: The Bad Parts

The evolution in tools for remote work made the Agile Manifesto’s remarks about face-to-face conversation obsolete. This happened to a lesser degree with this Agile principle as well:

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

That’s right, in spirit. The only problem there is that “the shorter timescale” can now be measured in minutes and hours, not weeks and months.

But in either case, evolving tools change our interpretation of the Manifesto. So what about the Manifesto’s core idea?

Individuals and interactions over processes and tools

It’s good to prioritize “individuals and interactions over processes and tools” when the only tools that exist are terrible. But on the one hand, the Agile culture often seems like nothing more than a marketplace for processes – including Scrum – and on the other hand, you won’t be a productive developer if you’re not fluent with modern tools. It doesn’t matter how good you are with individuals and interactions if everybody’s texting and you’re still leaving voicemails. Even companies that claim not to do remote work, or asynchronous communication, will still expect their employees to answer work emails on their phones from almost anywhere. If you change the tools, you change the interactions.

Because tools shape interactions (and vice versa). This was a major theme in the work of Marshall McLuhan. We respect the Manifesto, on the whole, but if you’re trying to tell people how to define culture at Internet companies, McLuhan is required reading. Wired magazine, which was also required reading during the era when the Manifesto was written, named McLuhan their patron saint.

The Manifesto’s “interactions over tools” languaging sounds like a diplomatic way to say “don’t use tools which suck, or, if you do use them, don’t take them too seriously.” This is good advice, and we absolutely endorse it – it may even apply to Scrum – but the Manifesto’s tactful phrasing cost it the capacity to acknowledge McLuhan’s work. That was a loss. The boundary between tools and interactions is fluid. The source code host is the process, because the medium is the message.

Here’s another destructive flaw in the Agile Manifesto:

Business people and developers must work together daily throughout the project.

First, this assumes that “business people” and “developers” are distinct and separate. There’s a weird and condescending implication that developers are not in business, which they typically are, in Scrum environments. I’ve never heard of anyone doing Scrum for fun in their free time.

Second, the “daily” part is silly. There’s lots of technical work where “I’ll see you in a few days” is the only sensible thing to tell your colleagues in marketing, UX, QA, product, or what have you.

Scrum takes this “daily” thing much too literally, making a daily meeting between developers and “business people” mandatory, whether there’s any need for it or not. If you’re required to say “that thing I said would take a couple days is taking a couple days,” it’s hard to do so without sounding apologetic, defiant, or mindless. It’s just a silly thing to require a professional to say to other professionals on a daily basis.

And those are just the problems in Scrum and the Agile Manifesto. There’s a much, much bigger problem with the Agile culture as a whole. A co-author of the Agile Manifesto, Dave Thomas, wrote a blog post called Agile Is Dead, in which he said:

I haven’t participated in any Agile events, I haven’t affiliated with the Agile Alliance, and I haven’t done any “agile” consultancy. I didn’t attend the 10th anniversary celebrations.

Why? Because I didn’t think that any of these things were in the spirit of the manifesto we produced…

The word “agile” has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products.

Moore’s Law created Agile. Back when programming a gigantic, warehouse-sized machine took several days, you had to do big design up front. As machines got smaller and faster, business practices had to adapt, and Agile evolved around the machines of the late 1990s.

But my guess is that Apple, SpaceX, Tesla, and plenty of other great companies are still using at least some waterfall practices with terrific success today. It’s still good for hardware manufacturing projects.

Insofar as Panda Strike has a process, it’s informal, and based on years of remote work, but it’s also based on extracting the good stuff from various Agile ceremonies and rituals. So you could maybe call us Agile. We’ve even kept a few tiny pieces from Scrum, although we’ve thrown out its vocabulary.

Semantics Are Important, For Knowledge Workers

Quoting from the private Panda Strike wiki:

Why We Don’t Use Scrum Terminology

Scrum is a popular Agile process, so why not use its terminology?

First, there are plain English words that already mean the same things. “Iteration” is a perfectly fine word and is actually closer to the intended meaning than “sprint.”

Second, Scrum terminology directly contradicts the stated principles of Agile development. For example:

Agile processes promote sustainable development…[The team] should…maintain a constant pace indefinitely.

Contrast this with the term “sprint,” which suggests a short burst, an unsustainable pace.

Third, the Scrum terminology creates a negative frame of reference. For example, you can be on the first day of a project and have a “backlog.” “Sprints” and “spikes” suggest a state of permanent and meaningless urgency.

I reiterated these points on my personal blog, and some people complained that I was splitting hairs. But choosing good names is important when you’re defining business practices, just like it is with variables.

In a recent blog post, Gregg Caines put it well:

when you want to get people to change the way they work, and you want them to understand the completely foreign concepts you’re bringing to them, it’s absolutely crucial that you name the thing in a way that also explains what it is not.

He continues with an example:

In Scrum, it’s also common to have a “sprint commitment” where the team “commits” to a body of work to accomplish in that time frame. The commitment is meant to be a rough estimate for the sake of planning purposes, and if a team doesn’t get that work done in that time, it tries to learn from the estimate and be more realistic in the next sprint. Developers are not supposed to be chastized [sic] for not meeting the sprint commitment — it’s just an extra piece of information to improve upon and to use for future planning. Obviously naming is hugely important here too, because in every other use of the word, a “commitment” is a pledge or a binding agreement, and this misnomer really influences the way people (mis)understand the concept of sprints. Let’s face it: if people see sprints as just more frequent deadlines (including those implementing them), the fault can’t be entirely theirs.

At some companies, various Scrum practices deteriorate into dysfunctional behavior, and part of that comes from the cognitive dissonance in Scrum’s naming. If you refer to an iteration estimate as a sprint commitment, people are going to expect that you will be hurrying, and that they can hold you to your commitments. By choosing bad names for things, Scrum sets itself up to fail.

And it’s not just Scrum that fails. A blog post on “Agile fluency,” by two Agile consultants who help upgrade teams to Agile development, classifies Agile implementations as one-star, two-star, three-star, and four-star. They say that one out of twenty organizations will achieve three out of four stars, after one to five years of effort:

In our experience, it takes a team two to six months to become fluent at the one-star level. About 45% of the teams we talk to say they’re fluent at this level.

Reaching the two-star level takes another three to 24 months, depending on the amount of technical debt in the code. About 35% of teams report fluency at this level.

Three-star teams are much more rare. About 5% of teams report fluency at this level. We’ve heard reports ranging from a year to five years to reach this level of fluency.

At Panda Strike, we’re grateful for the Agile Manifesto, but we don’t regard it as gospel. In addition to Scrum’s vocabulary, we’ve also thrown away its concept of blockers, its standups, and its planning days. We prefer asynchronous communication, avoid scheduling mandatory meetings, and favor Agile over Scrum, but maintain some skepticism about both. We also reject the Scrum custom of measuring velocity, because we agree with the Agile Manifesto that “working software is the primary measure of progress.”

Caveat: We Make Exceptions All The Time

If a client’s using Scrum, we’ll often go along with it. We sometimes see early-stage startups who think Scrum might be a good fit for them, and we discourage that when we see it. But we’re not process zealots. Life is too short for that kind of thing.

(We are process hipsters, though, a little bit. We were into Agile before it was cool, and we’re still fans of Agile’s early work. So if you’re dead set on using an Agile development process, we recommend you look into Extreme Programming instead.)