Using Agile in less-than-perfect situations since Y2K

A mailing list pointed me to a recent post by Jeremy Miller on Agile design and an older one by Joe Ocampo. Both discuss how design happens in an Agile project. Both are great posts. Jeremy’s post sticks largely to what is considered the Agile “party line”: design is done all the time rather than up front. Joe’s post talks about using Domain-Driven Design to develop the “ubiquitous language” that teams need for communication. Jeremy admits in the comments that a bit of up-front establishment of a domain is useful as context, but distinguishes it from design. Presumably he is equating it with analysis. Having written a book that, in part, shows how use of consistent domain modeling patterns can inform design and implementation, I would say it’s both. But that’s not what I want to talk about….

Reading these posts and exploring a bit further, I began to relate the establishing of a domain context for stories with establishing an architectural context.

One of the main challenges I have faced and hear others describe is how to maintain agility when there is a scope of multiple Agile projects (what the PMBOK would call a program) or an enterprise portfolio that really needs consistent architecture. I am generally a believer in simple, evolutionary design. However, as Jeremy points out in his post, it requires people with the proper skills to maintain an open design/architecture and who know how to tell when they have reached the last responsible moment. What I have discovered is that as the scope of systems under development scale, the last responsible moment for certain architectural decisions tends to shift earlier.

As such, a reasonable step would be to include a (short) architectural analysis session for each release that would mirror the domain modeling exercises Joe mentions late in his post. In that way, those with architectural knowledge could get involved and discuss what architectures support the features required and which decisions need to be agreed up front. Provided this is timeboxed and the group puts appropriate emphasis on postponing that which can be, this is in line with Agile values as well.

I should also note that I would only add this practice if the team found it necessary. As a coach, I might recommend it upon seeing the following things either in a retrospective or during planning or other working sessions:

• multiple components, libraries, and/or frameworks in use that accomplish the same goals
• “not invented here” syndrome
• lack of integration testing between project teams’ output
• failure of or difficulty with coordinated deployment (usually preceded by the lack of integration testing)
• no or limited consideration of nonfunctional requirements (performance, stability, etc.)
• failure to meet nonfunctional requirements

I’m sure there are other smells that would indicate a need for architectural planning. If you think of others, let me know in the comments.

This is the second part of a long post on iterative and incremental development. In Part 1, definitions were provided to distinguish both practices from the general term “iterative development”. This part describes some pitfalls of doing one without the other and how each adds value to the other.

The pitfalls of only iterating without incrementing are simple to summarize: nothing new is added. That is, if iterating is the practice of improving what already exists, then it provides no facility to add functionality to a system. While — in theory — the product and/or process would eventually be as near to perfect as humans can make it, the overall value would not increase.

Though it seems contrived, this happens in real projects. For example, assume that a new project starts its first iteration with requirements that are not properly thought out. The product that is created will not make the customers happy. They will want changes: improvements. They iterate on the requirements themselves, resulting in iteration on the product. The project team then discovers that the new implementation could support the changed requirements better with a change in architecture. They convince the customer to allow them to make the improvements. Meanwhile, the customer has spent so much time thinking about this one requirements set that they start to “need” more bells and whistles. Soon, updates to the requirements are created, and the cycle begins again.

The pitfalls of only incrementing without iterating are a little more complex. If incrementing only adds to what exists, then it is not spending effort to improve what is already there. At an extreme, this would be like sealing each iteration result in a shared library file (DLL, jar, etc.) and requiring each subsequent iteration to only use those pre-compiled artifacts. In a more realistic example, this is simply not allowing time for refactoring. Another version is skipping the retrospective or not implementing any of the improvements suggested by the retrospective. Both result in creating or failing to remove hurdles that slow or block progress.

Incrementing without iterating is more common in my experience. It is somewhat easier to see the lack of value increasing when nothing new is being created. It is more difficult to tell when the rate of value increase is stymied by technical, project, or other debt.

If you spend a portion of each period of work improving what has been done before, you can remove those hurdles that prevent progress. By making sure that a portion of each period of work is spent on adding new features or practices, we reduce or prevent unnecessary gold-plating and particularly wasteful forms of analysis paralysis. In short, each practice fills gaps left by the other.

At some point (1994 according to Alastair Cockburn’s article), the project management and process community made the interesting decision to combine these to in the single term “iterative”.
Was it, as Alastair says, because they were just happy to be “repeating the cycle”? My hope is that it is more because they saw the need for both to work together. The term may have been a poor choice, but combining them is a great choice.

There are few people who argue against iterative development. However, it seems that while most of us consider iterative and incremental development to be part and parcel of iterations, not everyone seems to do both — at least not consciously and effectively. In a pair of posts, I will explain why both have more value together than either does separately. This post will build the ground-work, describing what each of these practices is and their separate value. Part 2 will describe how they become more valuable when used together and the pitfalls of doing one without the other.

Iterative development is a means of scheduling time to improve the process and product. Think about retrospectives. An amount of time is spent working on the product. A discussion of strengths and needed improvements is held. Then, an amount of time is spent reworking the product just built and the practices in use. The point is that you spend time working on the same pieces each work period (iteration).

This iteration on the same thing allows considered improvement. It is similar to refactoring. This allows for paying off of technical debt, enhanced security and performance, and improvements in understanding, testing, and implementation. It increases overall quality and efficiency of the system under development and the practices in use.

Incremental development is a means of growing a large product through delivery of a series of smaller pieces. Think about use cases or user stories or features. The overall product is described as a set of discrete pieces, each having value on their own (see INVEST). The pieces are related enough that when put together, they form a larger, cohesive system. In incremental development time is spent working on new pieces each work period (iteration).

This iteration on small pieces allows for each new increment to be considered for deployment to production. A release plan may set goals for timing or scoping of releases, but the actual ability to demonstrate the working portion may lead to a decision that a set of functionality that is delivered earlier may have enough value to warrant an early release. Delivery of properly prioritized increments also helps to avoid getting to the end of a schedule without something valuable to deploy. The most important (and risky) parts of the system are guaranteed to be there early.

While writing this post, I noticed that Alistair Cockburn wrote something similar in August of last year. His ending statement says in part that iterative development improves the product, while incremental improves the process. While Alistair is an authority and my experience and understanding is consistent with everything else he writes on that page, I disagree with this summation. It is more appropriate to say instead that iterative improves both product and process, while incremental builds upon both product and process.