Categorise Unsolved Problems in Agile Development: Premature & Foreseeable

What is going wrong with Agile today?

We all feel it and debate it: Agile development feels right, developers love it, but sometimes it falls short.

Many Agile projects get less successful over time. There is procrastination, technical debt stacks up, progress slow down, morale deteriorates. Developers are working as hard as they can, yet the delivered value per sprint keeps declining.

All of a sudden, the low hanging fruit has been eaten, and the remaining backlog is full of items that are big, necessary, and of little value to stakeholders. Friction with stakeholders emerges, the project stalls, and may even die.

Some say stricter adherence to Agile’s core principles should prevent this. This is true. But to be agile and self-steering, everyone must be self-motivated to stick to the rules. The stricter the rules and the bigger the group, the quicker discipline waters down.

(further reading, a plea for restoring core values: Understanding Agile Values & Principles, free e-book)

What to do instead?

Dealing with the underlying causes

Unsolved problems belong on the backlog. In theory, the Product Owner processes all backlog items, dismisses the irrelevant and prioritizes the most important ones into sprints, until the backlog is empty and the project is done.

But in practice, that’s not what happens. The backlog just grows forever. It collects items that can wait, together with technical debt and hot potatoes which cannot simply be dismissed.

To developers, the backlog is a spillway to keep their job doable. Agile says: whatever you don't know yet, or can do without for now, park it on the backlog, and forget about it. It will reemerge when needed. For the most part, this works. It is the power of Agile.

But by the time unsolved problems reemerge, hot potatoes have become too hot to handle, and technical debt has become too expensive to repay. Implementation effort has grown far beyond the available resources.

This can be prevented by adding some core insights and making a few small but essential changes to the Agile approach.

The root is a systemic weakness: Agile declares all unsolved problems premature. But there is more. Some problems are foreseeable.

Premature / Foreseeable

There are two kinds of unsolved problems and they are fundamentally different.

Premature problems

Premature problems are unsolved problems not worth worrying about. Working on them is a waste of time. You can trust them to reemerge when the time is right.

Based on expectations, premature problems come in four flavors:

• Become irrelevant: Like a too-slow script that runs so rarely that few ever notice it. It is not solved, but not worth anyone's time either
• Stay the same: Like a UI that flashes annoyingly if you change window size; it isn't pretty, but not blocking, and regardless of when it's solved, the amount of work will be the same
• Become easier: Like complex configuration UIs. Later, you have more progressive insight and better understand what the data model looks like. You have more reusable UI widgets which were added for other reasons. Difficulty lowers in stages and stabilizes on some minimum. This is what you hope for when you let it mellow on the backlog.
• Go away: Like a difficult feature request where it's not even clear if the customer still wants it six months from now. It can be resolved with zero effort.

Premature problems reemerge if the problem changes, or the circumstances change. Calling a problem premature is a bit of a gamble on both. For many problems, laying them away for now is the right thing to do.

Foreseeable problems

Foreseeable problems are those that get harder to solve over time. Technical debt and hot potatoes are known subcategories. Agile produces them in high quantities.

The sooner they are addressed, the better. If you ignore them long enough, foreseeable problems grow beyond their "feasibility limit"; they become too big to solve.

Experience shows that difficulty rises slowly for a long time, and when circumstances change, it ramps up very fast.

Foreseeable problems come in two flavors:

• Direct: The problem itself is the root cause. Like a slow portal initialization script that ploughs through an ever growing database.
• Indirect: A root cause that is not "broken" in itself, but causes the worsening of problems everywhere else. Like an intrusive framework. The more code relies on it, the more conflicts it generates, and the harder it becomes to remove the framework later.

To determine whether a problem is foreseeable, requires predictive skills:

• A sense of apparent cause and root cause
• Understanding why certain patterns lead to certain problems later
• Thinking in scenarios and estimating their probability
• Predicting what circumstances are likely to change and how soon
• How much time you have left before the problem hits the feasibility limit

These are skills typically expected of architects, and require some distance from the everyday madness of getting code to work.

Foreseeable problems have one or more foreseeable scenarios. You can predict how things get worse if you don't act. You can predict how it crosses the feasibility limit when certain circumstances change.

Problems that worsen over time, are always foreseeable. Hence the name.

The role of the architect is to see which problems are premature or foreseeable.

Foreseeable problems in Agile

The essential flaw of Agile is that all unsolved problems are considered premature and foreseeable problems don't exist.

Developers believe they will solve backlog items when things get less busy, which never happens. Meanwhile, foreseeable problems do exist. While buried in the backlog, it’s not visible how work items change. Some stay the same, get easier, lose relevance or go away. But some get worse.

At the most inconvenient moment the difficulty level grows past the feasibility limit; to everyone's horror, foreseeable problems reemerge from their graves in herds, and have become too big to solve.

In well-funded organizations, this is solved with money. More servers, more developers. The feasibility limit is moved up and hit again later. In less wealthy organizations, the project loses its ability to innovate, becomes obsolete and dies.

What the architect should do for Agile teams

Besides their regular responsibilities, the architect should scan the backlogs frequently, and label problems as premature or foreseeable; with notes about a foreseeable scenario, possible circumstance-change risks, and possible impact.

The architect should make a selection of the most pressing foreseeable backlog items and produce designs to solve them. Then agendize these designs as regular work items. The designs should be mature enough to chop up in stickies and execute right off the bat.

Design is necessary. Just dumping an old problem back on the developer's plate with no plan causes stress, resistance and denial. The problem landed on the backlog for a reason and is foreseeable for a reason.

Ranking foreseeable problems' priority is a multi-dimensional puzzle of value, effort, damage, likelihood and time left before hitting the feasibility limit. These parameters are hard to quantify and best rated by experience.

As a rule of thumb, foreseeable problems of the most "indirect" kind should be solved first: they cause the most trouble all over the place, fixing them first frees resources to fix the next in priority.

Trust in Agile role play

Developers should trust that there is an architect who catches unsolved problems that would turn ugly when neglected. This architect will make them reemerge on the developer agenda when needed, with a design, planned with regular priority, and no stress.

The only thing to watch out for is developers relying on this "service" too much and using the backlog as a dumping ground for everything they didn't think through well enough.

Making up for the Product Owner?

Does premature / foreseeable mean the Architect is doing the work of a bad Product Owner? Not really. The method turns the backlog into clear and actionable information so the Product Owner can fill the sprints.

The Product Owner generally can’t fulfill the premature / foreseeable role, lacking the technical insight and with conflicting interests: keeping both the customer and the team happy in stressful times is priority number one. A Product Owner is not going to dig for skeletons.

So, Agile can work

With this approach, Agile can actually work as intended: a sustainable development method that keeps developer's lives bearable, without stacking up technical debt and hot potatoes.

The small price to pay is acknowledgement that self-steering teams of developers only, without anyone in an architect-like role, are not sustainable.

(further reading: Maybe Agile is the problem)
(further reading: The Agile Manifesto: A Software Architect's Perspective)

But there is more

This premature / foreseeable principle touches on more subjects.

After all, this is about concretizing the unspoken tension between the short term pressure of writing code and making things work, versus long term maintainability and evolvability.

(further reading: Building Evolutionary Architectures, especially chapters 2, 6 and 8)

Developer's mindset

All architects know that developers look at projects from a completely different angle, and it causes many misunderstandings. Using the premature / foreseeable principle can help bridge the gap.

Backlog as a distraction dumpster

Developers are craftspeople, busy making things. They want as little distraction as possible to stay in their "flow" state. Developers are good at pushing distractions out of their mind space.

They have a million problems to solve to get code to work, an extra million on top causes stress. Developers need the backlog to put stumbled-upon problems away.

Architects sometimes feel this is problem dumping. But it works for developers. What's missing is oversight, someone to filter out foreseeable problems that must be reinserted in the development cycle.

Such backlog oversight must not be made a developer's responsibility in addition to regular development. That causes conflicts of interest and stress. This stress could even trigger despair, advocating to rebuild everything from scratch, demoralizing the entire team.

Judgment-free unsolved problems

Developers trust their ability to code themselves out of whatever hole they get caught in. They do it all day.

To developers, an architect's design can feel judgmental, as if their coding capabilities fall short, while they made everything. But the intent of the architect is neutral: with design, developers' coding power can reach much more mileage.

By introducing the premature / foreseeable principle in daily conversation, it is easier to keep such discussions judgment free. It's not about who's slow, inadequate, better or smarter, it's just about what's likely to happen and how to prevent that.

It also helps everyone when developers, while posting new backlog items, add warning labels to suspected foreseeable problems, with a foreseeable scenario.

Frameworks as foreseeable problems

Developers are careless with throwing frameworks at the code base. They just ran into a problem, ouch, too big to solve, so let's see who else has had this. Find a framework, OK, seems to work, good, resume coding.

To architects, frameworks are structural decisions. To developers, they are off-the-shelf code. If developers would shut the sprint down for a thorough framework selection procedure every single time, they'd get nothing done.

To an architect, hasty choices close options that should stay open. Often, rash framework choices disappoint later. Undoing frameworks is hard once they are in use. If you wait long enough, reversal becomes infeasible.

This is a foreseeable problem. Framework selection should be done ahead of sprints. Preemptive framework selection should be seen as a solution for a foreseeable problem.

Less need for heroes

Agile wants developers to be courageous with unsolved problems: the team is autonomous, trusted, and knows what to do. In real life, it does not always work that way. Normally, stakeholder trust erodes over the course of a project, and is restored when the delivered product is a success.

The more trust erodes, the more courage team members need to convince frustrated stakeholders to work on "vague" items. Gradually it becomes too much courage for most people. They avoid bringing up hard-to-explain subjects altogether. Avoidance makes foreseeable problems creep towards their feasibility limit.

In the stakeholder discussion, expressing a problem as foreseeable, can take the courage factor out, and make it a more neutral discussion. It should be much easier if:

• The need is clear
• There is no judgment
• Results are beneficial
• Consequences of doing nothing are understood by everyone

(further reading: Importance of Courageous Communicators in Agile Organizations)

Developers resisting design

As an architect you want developers to understand and support your design, debate it with you to identify holes, and actively come up with ideas to make it better. A design worth its salt should survive a healthy debate. The premature / foreseeable principle could improve the debate.

Resisting complexity

Developers resist a design when they think it is too complex, and are convinced a simpler solution is better. If there is a simple solution, great, use it!  Often, however, the complex solution is necessary because the simple solution is inadequate.

Developers may resist openly, or just passively follow orders, refusing active contribution. This has a cost: more design holes, more rework during programming and more bug fixing downstream.

Usually, the architect's reasons revolve around vision, future and design principles. But if you explain these to developers, the picture becomes too wide and sprawling. The more you try to clarify, the more heel-digging you meet.

Design resistance is often accompanied by emotions like disappointment, fear, feeling outsmarted or robbed of autonomy. Talking it over is hard because developers debate both technical and emotional subjects with technical arguments. It's an elusive discussion where nobody wins. Architects should rather prevent design resistance altogether.

Resisting design holes

Design holes add to this resistance. Humans make mistakes. Architects are not exempt. The more complex a design, the more likely it is the architect has overlooked something, leaving a design hole.

To a developer, stumbling upon such holes means throwing away just-made code, more complexity, missing deadlines. Design holes are always inconspicuous and very hard to solve. From experience developers fear that the more complex a design is, the more holes they will run into.

Why showing the big picture does not work

When developers say "I'm missing the bigger picture" or "you are making it too beautiful", they mean that if they can't think of a valid reason to make it so complex, there cannot be a good reason.

It's tempting to explain your choices by showing the bigger picture, with adverse effects.

To developers, an ambitious design is like the architect loosely sketching the outline of the Sistine Chapel with a big paint roller, leaving the developers to deal with solving a million details.

Left: how architects see a design. Right: what developers fear they will have to make.

Showing the bigger picture makes developers freak out, because with each part they think "oh hell, how do I code this". A big design always is too much to fit in a mind, with a fear of design holes on top.

If your grand design contains unknowns, it gets worse. Things you don't know yet are a normal part of the architecture's design process. In the developer's mind, tainted by poor customer specs and having to make working code out of those, these unknowns translate to uncertainty, more problems to solve, more fear of holes, and a false idea that the architect doesn't know where he/she is going.

When developers ask for bigger pictures, investigate the real reasons. Better only share grand total designs with experienced people who can handle the vertigo.

Why debating in abstractions does not work

To architects, certain abstractions are the right things to use, without the need to overthink each step in between. It is a necessary ability of an architect. It is tempting to explain the design to developers by these abstractions.

Developers see principles and design patterns like cars: they all get you from A to B, so why waste money on the most expensive one?

Debating in abstractions with developers will run into a discussion about taste, or a competition about who's making up the "cleanest" abstraction. Both get you nowhere.

Why business reasons don't work

For an architect, several technical choices have commercial reasons. For example, a technical ability to make partnering easy can make the difference between niche and market leader. Architects should be involved in such choices.

Commercial reasons do not convince developers. Rather, they turn the debate to the irrational minds of commercial people, having irrational requirements, hence disprove the validity of your design.

Developers fear commercial stuff. Past experiences of partners often resulted in cleaning up the mess made by demanding and underqualified IT staffers. To developers, commercial success means getting pushy managers making unrealistic demands, being forced into regrettable quick & dirty solutions and being incident driven.

Many developers don't like business, sales and marketing, both the people and the subject. Three decades of Dilbert comics tell you why.

So, what does work?

It is the architect's responsibility to carve executable designs from the grand total design, devoid of bigger pictures, esoteric stuff and unknowns.

Designs should start with a list of foreseeable problems and foreseeable scenarios that you want to solve. This is how you show why simplistic solutions are insufficient or don't last long enough to justify working on them.

It is best if you write multiple foreseeable scenarios and make them interrelated. Why? With foreseeable scenarios you turn abstractions and entire categories of problems into concrete examples. If there is only one scenario, developers may be tempted to find simple hacks to solve that one scenario instead of the entire category of problems it represents, hence underestimate scope.

Related problems that are obvious but premature, should be listed under "out-of-scope".

A design listing the premature and foreseeable problems, tells developers what not to worry about and what to focus on. It gives the architect a more neutral route to debating design, well before resistance reaches a boiling point.

To developers, a premature / foreseeable driven design is much easier to follow, easier to scan for design holes and easier to come with ideas to improve it. It does so without adding the stress, uncertainty and the burden of too many problems that come with complex designs.

It also gives more room for developers to make their own decisions, as the architect’s intent is well-understood, less detailed specs are needed, and the design is on topic for the sprint.

(watch: video on design resistance, BDUF, delivering architecture during sprints)

Bringing it all together

Likelihood remains a talking point

The discussion moves from debating a problem's relevance to its probability. How likely it is to happen? How likely is it that circumstances change? How long until it hits the feasibility limit?

For simplicity, likelihood of foreseeable problems can be classified into three levels:

• Inevitable: will happen, must act
• Fire risk: unlikely but worth sprinklers
• Meteor hit risk: very unlikely and not worth a space shield

Likelihood puts challenges in the right perspective. It helps when deciding which foreseeable problems to prioritize and which are too improbable, both for developers and architects.

Empathy is required

You need to understand the developer's mindset, to see why things happen, before this new logic becomes effective. Labeling problems as "foreseeable" out of the blue can feel blunt and judgmental about the developer's intelligence, work ethic, or coding skills. Developers may keep unsolved problems a secret and take the burden onto themselves, like developers already do by nature. You would replace one problem with another.

(further reading: Empathy is a Technical Skill)

If developers feel safe to share, discussing unsolved problems doesn’t feel like incompetence, laziness or sloppiness. Unsolved problems will be just a neutral consequence of working hard and meaning well, and the architect is there to pick out the nasty ones. Then Agile works.

(further reading: architect’s tips including the "vertical cut" idea that can help agendizing foreseeable problems without sounding like a prophet of doom)

How about keeping your options open?

Keeping options open is part of what architects do: you can't foresee it, so you keep it open. It seems to be in direct contradiction with the whole premature / foreseeable principle.

In reality, keeping options open is actively arranging your unsolved problems so that you turn them into as many premature problems as possible, and be left with as few foreseeable problems as possible.

The most effective thing an architect can do is solve one foreseeable problem that turns many other foreseeable problems into premature ones, leaving your options open.

Conclusion

Introducing the premature / foreseeable distinction as a new thinking and discussing habit:

• Makes hot potatoes and technical debt discussable without developers stressing out
• Gives architects the role to deal with things that run out of control when left alone
• Helps to discuss designs in a neutral way that everyone understands
• Creates a better understanding for a more complex but necessary solution
• Prevents design holes and bugs

Distinguishing premature and foreseeable problems makes Agile the sustainable development method it was meant to be.

About the Author

Bas Groot was an Amsterdam-based entrepreneur for 19 years, developing a web application platform and content management system, back in 1996 when there was no name for such software yet. Then 7 years as employee at a financial services software vendor, as the architect for the web solutions department with the same platform. During his career, Bas lead and trained developer teams, headed product development and software design as architect, and realized customer projects with the platform in government, finance, legal, non-profit, automotive and broadcast media, using various agile methods. Bas headed development of two programming languages and a graphical low code application programming environment, a web application framework with real-time ensemble coding capabilities, a data modeling framework, a software/content distribution and deployment framework, a web development base library and many more. Bas has hands-in-the-mud experience on the creation and usage side of the framework world and a strongly conceptual approach when designing software and methodology.

"When something doesn’t work, first thing I look for is what rules people follow and why."