Front Load All The Things
Agile methodologies often excuse poor planning, but nothing ever goes to plan anyway. Is there a better way?
"The bigger my island of knowledge gets, the longer my coastline of ignorance grows."
As a junior engineer I remember watching some colleagues present their project debriefs as part of a professional development session. Afterwards I remarked to a mentor that I thought the presentations were good, and it sounded like the engineers had learnt things they could apply next time. He responded:
"Yeah, that's the problem. All they talked about was how they learnt this thing and won’t make that particular mistake next time."
"Isn't that a good thing, that they were learning from their mistakes?"
"Well, it's fine. But it's not remarkable. The thing is, you will always make mistakes, and always have something new to learn. If your only take away is the technical detail you messed up, then you're missing the biggest lessons."
It took me a decade or two, but I've firmly arrived at the same conclusion.
When planning a project, an enthusiastic junior engineer will tend to visualise the solution, colouring it with the features they have a good handle on, inadvertently focusing on the parts they are confident about. The mark of a senior engineer is that they will focus on all the things that they do not understand well. We typically call these "risks". A senior engineer is expected to anticipate risks and make good technical decisions under uncertainty. The things that determine the success of a project are not the things that go perfectly to plan. It's how the things that were overlooked, or underestimated, or poorly understood, are handled.
And what engineers eventually learn, sooner or later, is that things never go to plan. Early on your learnings are so fast and dramatic that it is natural to declare "now I get it!" and "that won't happen next time". Stick around long enough and you realise that there's something to learn every single time. Even if your island of knowledge were somehow complete, the world is dynamic, people matter and relationships aren't set and forget. That's why you expect a senior engineer to be able to carry a project in the real world - knowledge helps, but knowing to plan for the unknown helps more.
This is why the adage that "plans are useless but planning is invaluable" is so profound. Or, if you prefer, as the great orator Mike Tyson put it:
"Everyone has a plan until they get punched in the mouth."
So how do you plan for the unknown? One popular method is to only plan within sight. To identify the next most important thing, estimate its complexity and dive into it. This is usually known as Sprint Planning. Note that the term is also part of the lore of the Scrum methodology, and I use it here broadly, without the nuance relevant to any particular domain. Note also the term is not to be confused with the potent, short-term, focus sessions often called "Design Sprints".
The point is, Sprint Planning is an effective way of ensuring that miscalculations don't cascade into project killers. By keeping exposure constrained, you provide opportunities to course correct before loss aversion kicks in, which limits the likelihood that a path will be pursued simply because it was the original expectation. As things change, and they always do, you're free to adapt continuously, using the most up to date knowledge possible. This is a particularly attractive approach if at each junction point you can deliver value to stakeholders. Ideally that's a functioning, albeit incomplete system, so stakeholders can make use of it for evaluation, training or even business use.
You'll often find variations on this approach utilised when an Agile methodology is in effect, although the term has become increasingly applied without much precision.
But Sprint Planning is a bad approach for some projects. It's difficult to apply to consultancy and other hourly-costed professional work, since a conversion layer is required to provide an interface for budgeting and deliverable dates. It's also not well suited to work that doesn't have frequent opportunities to deliver value and receive user feedback, or that require complex coordination between multiple contributing disciplines that rely on each other's outputs. But perhaps most egregiously, it can steer engineers towards developing solutions that are only locally optimal. That's because tunnel vision is an insidious force in engineering. It sneaks up on us and buries us in effort - well-meaning and skilful effort - but effort which only solves the issues in front of our nose. Do that too often and you end up with excellent solutions to problems that don't matter. It is easy to get lost working to deliver outcomes that, in hindsight, were neither the easiest nor the most effective solution.
There are various tweaks that can be applied to address these shortcomings. One extreme example is Big Design Up Front (BDUF). It emphasises detailed design work before implementation starts, arguing that serious effort prior to the implementation phase is a good investment. The payback is due to the fact that design flaws are significantly quicker to fix the earlier they are found - ideally before implementation has even started.
Again, the terminology is blurry, but this approach is often utilised when a Waterfall methodology is applied.
The insidious issue with BDUF is that action informs. Planning, prediction and experience are powerful tools. But the sheer act of building always teaches you more about how that thing works than any amount of research and on-paper design can ever provide.
It pays to be wary when an engineer exclaims "that's easy". Sometimes they are only thinking of the easy parts when they make that determination. After all, they're the parts that are both apparent to them, and align with their competence. Professionals have a desire to appear competent, and working on well-understood things is a sure way to demonstrate that. It requires vulnerability to highlight the less well-understood things. But the prudent engineer must exercise humility and do it anyway. A neat trick to remind yourself of this is to ask, "if this project were to go awry, what would the likely causes be?".
If one is willing to be so humble as to admit that things change, and learning is constant, then it is misleading to claim that freezing requirements prior to implementation results in an optimal solution. But optimal solutions aren't always the goal - sometimes predictability, at any cost, is a higher priority.
Given the outsized impact of project planning, it behoves a professional to evaluate a project according to a rubric like the following:
- Does the project lend itself to be delivered in valuable (ie. useable for some business purpose) and frequent (ie. at least fortnightly or so) updates?
- Is this project without a clear transition to a maintenance phase, in the sense that unbounded, incremental development is worthwhile, rather than being dependent on budgeted and deadlined milestones?
- Is the cost of the project a lower priority than knowing its completion date prior to completion?
- Does the project seek to predominantly reimplement proven knowledge, and does not benefit from leverage of new discoveries?
If the answer to #1 and #2 is "yes", then Sprint Planning is a suitable approach. Web development projects typically meet this criteria. Few others do, so treat the enthusiasm about its universality with caution.
If the answer to #3 and #4 is "yes", then BDUF Planning is a suitable approach. Some government and military projects meet this criteria, when significant external pressure is on "to-spec" delivery at all costs, and the job of research and development is cleanly outsourced to other departments.
In all other cases there are superior approaches. Alas, existing between extremes, proponents of the pragmatic solutions are unlikely to garner the most book deals. Nonetheless, they are the timeless truths that plot a reliable course. And that makes them special. While many are looking for 10x improvements applying idealistic, ordained rules (that worked for NASA! Or Google!) real progress can be made. Without the thrashing of "No True Scotsman" type adherence to purity, the wisdom of experience can be maximised.
That's why I've come to advocate for Front Loading.
The defining characteristic of Front Loading is to do the hard things first. It drives you to firstly identify those things that are nebulous or you'd prefer not to think about, or that "should be alright". It's about being sincere with stakeholders, and doing your best to work with eyes wide open. Typically there's a bunch of things in any project that you'll sail through and kick goal after goal on. For the sake of all involved, they're best saved for when the deadline is looming, the pressure is on and energy is waning. Lifting the lid on the unknowns at the pointy end of the project is a recipe for disaster. It's the "save the best for last" approach to project execution.
By identifying the stickier aspects of the project, you're forced to highlight those things that may or may not go according to plan, or whose outcome is difficult to predict. And the best time to deal with those aspects is up front. That's when you have the most latitude to accommodate them. If something requires some research, or has ambiguous requirements, or is otherwise not well understood, up front is when you want to address it. Not everything can be resolved without execution - but at the very least shining a light on these aspects allows preparations to be made, priming the subconscious mind to spot mitigations. Not only could these aspects have a significant impact on the most suitable system design, they may also have lasting impacts on stakeholder expectations. Bringing them into the light provides the best opportunity for getting them addressed.
"All the really important mistakes are made the first day." -- Eberhardt Rechtin
Front loading is about saying up front that there's some things that won't go according to a plan, and here's how I'm planning for that. It's about using your experience to point out those aspects that bring some doubt, as a demonstration of expertise rather than a revelation of naivety.
And it's about being professional about the nature of engineering projects. That doing something novel means accepting risk, and that it is unprofessional to claim risks can be eliminated, or indeed, that they can't be managed.
In practice, Front Loading typically means advocating for a longer Discovery and Requirements Elicitation stage than is popular, but also not pretending that those activities are finalised when subsequent phases begin. Effort up front is spent to achieve an effort reduction multiple during the long tail. It's about being firm that any early implementation activities are purely risk reduction motivated, and any direct re-use during actual implementation is a matter of coincidence. Crucially, it's about ensuring that any talk of requirements and cost is noted as contingent on up-front risk reduction activities - and then coming good on that contingency and providing a working specification and budget to guide the remainder of the project.
Ultimately, Front Loading applies the time-honoured benefits of up front project planning exemplified by the Waterfall methodology, while acknowledging the learn-by-doing truths exemplified by the Agile methodology. For projects between the extremes where those methodologies are best suited, Front Loading provides experienced professionals the best opportunity to leverage their expertise.