Introduction

While working on our last project, we encountered an unexpected issue before launching a module into production. We needed someone with elevated access to one of our external systems, so we called a stakeholder who could help. His response, "This should have been part of the plan," while sounding a bit annoyed. We had to fix the issue ourselves.

This blog post examines the downsides of the well-intentioned, upfront, detailed software project plan. The kind used in waterfall projects and in so-called agile projects, and that middle management often relies on.

Preface

Just to be clear, I am not advocating for no plan at all. Even though the agile manifesto says it values responding to change over following a plan, it does not say there should be no plan. Also, here I am talking about the large up-front plan that is "necessary" to get board approval and funding. The Gantt diagram stretches months or even years into the future. Planning a week is, of course, totally fine.

Project plan downsides

Illusion of control

One reason the project plan, maybe even with a fair amount of estimation, is so appealing is the illusion of control. I have heard multiple times from middle management: "We need a project plan to know when we will be done and how much it will cost." Followed up with: "How else will we get it approved by the board?" The problem, which almost everyone working on something new and difficult knows, is that you cannot plan yourself into predictability in something as unpredictable as software development. With the project plan in place, maybe with dependencies between different phases(read Gantt diagram), we can be fooled into believing we have control. Any detail or question that arises during development and deviates from the plan may be seen as a loss of control. Additionally, getting domain experts' or middle management's attention later can be difficult, as they believe the project is under control based on the project plan.

Starting too many projects

With project X seemingly under control, it might be tempting for middle management to start a new project. Project managers can easily mistake creating the project plan for doing the actual work. When the board asks whether the managers can take on another project, they will, with good intentions, answer yes. The new project will give the company new capabilities and help it compete, and with project X seemingly under control and handed off for implementation, it may start a new one.

As more projects come in, the cost of context switching increases, as does the cognitive load (on both the individuals and the teams). As a consequence, the development speed goes down.

Not fixing bottlenecks

The overarching project plan does not concern itself with functionality flow or feedback loops. I have witnessed multiple bottlenecks that almost all projects/initiatives run into within a company. One example could be the lack of a production environment. With the project plan obscuring these details and indirectly preventing middle management from spending time with the teams to solve these issues, more and more projects get stuck in the same place.

Not improving software development

Maybe the most concerning downside of the up-front software plan that replaces tight collaboration is the lack of skill improvement in software development. Software development is a skill, just like playing an instrument, that needs constant practise. Project leaders and middle management seldom practice this skill if they see software development as creating a high-level plan and handing it over "to implementation".

Wasting time on planning

Eventually, as projects deviate from the plan and take longer than estimated, management needs to take action. One approach I have witnessed is to double down on planning: "This project took longer than expected because we need even more planning". This starts a non-constructive cycle that increases both time and spending on planning.

An alternative approach

Numerous books and talks have explored agile development and why the waterfall approach is less effective. A good book is A Seat at the Table, which also lays out an alternative approach that focuses on software capabilities instead of projects, features, and plans.

Without middle management embracing uncertainty and instead focusing on continuously learning about the domain and, maybe more importantly, the team, there is little chance of embracing true agile development.

A better start would be to create the smallest functionality possible and get it all the way into production. There will be many bottlenecks and issues that middle management can help resolve. All project members should work as much as possible, sitting next to each other and solving problems. The projects need backing and priority from management, and the company should prioritize continuous testing. Teams can accelerate experimentation and receive feedback by leveraging cloud providers to rapidly provision infrastructure.