When you look at software development or corporate change projects, you often see some creative fiction. There is fiction in the plans, fiction in the designs and fiction in the requirements. This fiction is created by the notion that “Before we can start, we have to know everything required to get to done.”
Intuitively, we all know that this is not really true. We all know that information will “emerge” from our activities that will change how we get to done. We learn from our mistakes. We try things that don’t produce as good a result as we want. We clarify our understanding of the problem as we demonstrate portions of the solution.
The problems start when your plan to deal with emergent information is to ignore it, or to intentionally inhibit the natural change resulting from this emergence. This doesn’t sound bad, but by analogy it is like planning the route for a trip from Chicago to St. Louis, then when at Pontiac, IL we learn that the next hundred miles of I-55 are under construction, but we neither change routes or adjust our time line based on a new guess regarding our average speed. We act as if the obstacles to our progress are not there. We represent a state of affairs that does not reflect our experience, but our intention.
The fiction comes from our desire to represent the information we know at the start as complete, even when it is not. We need our requirements to appear complete, so that we can get permission to start design. We need our plan to appear complete, so we can report status a certain way. We need our design to appear complete, so that we can start coding.
This fiction arises because every activity must be “done” before the next activity can “start”. By fiction what I mean is that we represent a “certainty” that isn’t real. Detail in a plan for coding tasks that did not derive from the design component model, but from some template plan, is fiction. Fiction gives the impression that we know more about how to get to done than we actually do. Detailed requirements that are expressed from a process centric perspective without contemplating user experience tell a fiction. They lead us to believe that we know how to get to done, when we have only contemplated how to ensure the process isn’t damaged through implementation. Designs that have lists of components, but do not explain the purpose of those components tell a fiction. They lead us to believe that we have made all the necessary design decisions so that we can simply build the software.
So what can you do to eliminate fiction?
- Create planning milestones for decisions to reflect the flow of information into the decision making process – the real work in any endeavor is in the decisions that are made. The trick is to plan around decisions, not activities. Making decisions based on a schedule rather than on information flowing into decision frameworks is the end result of fiction.
- Reflect the “decision readiness” in the status of your endeavor – Since we recognize that every type of activity in a project adds information the status should reflect what decisions are informed by that information, and our status then becomes about which decisions we have enough information to make, which decisions are already made, decisions we were forced to make without adequate information (carried risk) and made decisions what have been proved through delivery (retired risk).
- Move decisions to the “Latest Responsible Moment” – Recognize that the real work in planning, requirements, design, and implementation all involve decisions. Plan so that your decisions are the inflection points of the project, not the “phases”. Risk in software or change projects is retired by making decisions and by completing “capabilities” that prove those decisions to be sound.
- Design to defer decisions – Make decisions about the software or organizational design that “keep our options open” – by reducing the risk of changing our mind. Estimate multiple options, and use the largest estimate, so that if lower cost options is preferable, our plan reflects a savings, not a increase in cost. Alter design proposal, to minimize the difference in cost between those options so that it is safe to defer the selection of the best option until any foundational capabilities have been delivered (design decisions have been proven to be sound).
- Select a deliverables framework that supports atomic capabilities – When your requirements are organized around a principle of defining and delivering in small increments of business value, it is easier to make decisions about how something needs to work. One that then envision iterating to improve a small capability over time. But you also have the choice of replacing that capability with something different, or of combining that capability with others to improve its cohesion with the application. If your requirements framework is more monolithic, the capabilities all need to be improved together and re-arranging for higher cohesion feels like a re-write.
Why do these things help remove fiction from my plan?
By waiting until the last responsible moment, I have more information available. I have allowed that information to emerge from the other activities required for delivery. By designing to defer, I evaluate options in parallel, rather than serially. This creates information that wouldn’t otherwise exist. By defining requirements for small atomic capabilities, I am by default designing in smaller increments. I have a greater ability to use reference classes to estimate complexity than if I had larger more monolithic capabilities defined.
What these lead to is a practice in planning, where we compose our plan out of many small independently valuable deliverables, that does not require us to guess. When we start with this structure, and then elaborate the details behind each deliverable, We know how to get each capability to done. Isn’t the whole thing done, when each of the capabilities are done? Isn’t that the way it works?
The key here is that I can start moving capabilities to done as soon as I have the first one elaborated. I then learn from delivering that capability and that learning can inform the definition of done for the next capability, and it can inform my understanding of how fast things can get to done. Delivering each thing informs my design and my plan. So when I make all my decisions up front, I have essentially ignored the new information arising from delivery work. If I continue down the path of my initial plan without considering the latest information, I am preventing information from flowing into decisions.
By reflecting decisions as the key factors in planning, and not activities, my plan focuses on the production of information that leads to better decision making. Activities, while costly, change from being the main focus of the plan to becoming useful drivers of information into our decision making.
Is this really important?
If you ask any experienced project manager or software architect how frequently they were required to make planning or design decisions based on insufficient information, or just plan wild ass guesses, I would wager that the answer is more often than not. If you talk to executives about corporate change programs they have lead, and ask whether they have ever felt comfortable that they had the information to make key decisions, or even felt like they had the time to figure out what that key information was – let alone gather it reliably, they would say much less often than they wanted to.
If you look at statistics on successful software delivery or corporate change initiatives – the industry literature shows that there are a significant number of failures. I believe that that number is after the kind of “lipstick on the pig” efforts to paint marginal programs as big wins.
What does this have to do with progressive elaboration?
The idea that the activities that we do within any project do not inform each other is just plain foolishness. Whether it is a building project or a software project or a corporate change project – we don’t work on one activity at a time to completion, we work in successive levels of detail across activities to ensure that the various activities remain connected and continue to inform each other. These successive levels of detail is the progression of elaboration. Each project is different, in that the information flowing out of the activities influence decisions differently. Thus defining the information delivery processes for the project teams is important.
Progressive elaboration of plan, requirements, design and metrics becomes the means of staying connected. It becomes the means of tying the decisions together. It becomes the principle of organization around which the information flow begins. If you want to run information driven projects – progressive elaboration is the place where you start.
John McDonagh
February 11, 2015 at 9:03 amRich – I don’t really disagree with anything here. The bit I struggle with is probably the same thing we struggle with moving from waterfall to agile methods – how do we give a level of certainty and manage expectations of what will be achieved when, who will be needed when etc. to manage the business.
Rich
February 11, 2015 at 9:18 pmI tend to trade “certainty” for “trust and transparency”. How fast can we go before the wheels fall off? Before we lose control and become unpredictable? That is really about understanding the “moment” of each decision, and providing a framework to enable the decisions to be made in their moment – that is, without introducing risk, cost, or delay. The faster we want to go, the faster the decisions come.
In my experience, the biggest obstacle to project certainty is not managing the activities, it is getting decisions made and making decisions stick. How often does no mean “Ask again later.” – or yes mean “I’ll probably change my mind after you show me the product.” How often do the activities stop, slow or get re-sequenced because we are uncomfortable with a decision that needs to be made.
To your last point – managing the business (or the customer) is best accomplished by focusing their attention on the decision making process and their role in that process. Letting them tell us what (information, signals, etc.) they need to become comfortable with that role and their responsibilities is a great alternative to providing them a standard but less meaningful or semi-fictional “status” that they don’t really understand anyway. Why do you think we freak out when a project status “turns red”? Is it because we are uncomfortable with the decisions that are necessary to return status to green.