Agile's Iterative Development Incremental Release Rhythm

The Agile Methodology Isn't

Before we get too far along, it's important to underscore one particular point: Agile Software Development isn't a specific methodology. Rather, Agile Software Development refers to a general class of software development approaches. In 2001 a group of thought leaders using a variety of light weight, low formality software development approaches met to discuss what, if anything, they had in common. The result of that discussion was a common core set of values and principles described in the Agile Manifesto. A specific approach might be considered Agile if it honors those values and principles.

If you own any books regarding Agile development methods, or you've done a little research, you've probably read the Agile manifesto already, but I'll include it here for you to ponder. The Agile manifesto is expressed as a series of four simple paired value statements where both items in the pair have value, but one item is valued more than the other.

• Individuals and interactions over process and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

When determining if a methodology is Agile, I find that I have to use the pornography rule as quoted by Supreme Court justice Potter Stewart in 1964: "I can't define pornography, but I know it when I see it." The same seems to be true of Agile Development methodologies. Understand Agile values and you'll know an Agile methodology when you see one.

For many these value statements read like motherhood statements - simple truths hard to disagree with. Others believe that the "this over that" format incorrectly asserts that one side of the statement is at odds with the other - for example that valuing individuals and their interactions is somehow at odds with valuing a solid process and effective tools. If one of the goals for the Agile Manifest was to cause discussion about these things, it's indeed accomplished that. And oddly what comes out of discussions about the value of people, collaboration, working software, and responding to change are flexible resilient methodologies, effective tools, innovative approaches to documentation, the rethinking of the contract, and new approaches to planning and project management. Go figure.

There are a few named Agile Methodologies very well described in books written by their creators and/or practitioners: Extreme Programming (Beck, 1999), Scrum (Schwaber and Beedle, 2000), Feature Driven Development (Coad, LeFebvre & DeLuca, 1999), and Crystal (Cockburn, 1999-2004). All these books describe methodologies that share the common value system expressed in the Agile manifesto.

I'm confident that there are named Agile approaches that I've missed. I'm equally confident that there are a multitude of variations currently in use within a number of software development organizations. I often meet people and talk with them about their development approach. After just a little conversation with them, it's easy to describe what they're doing as Agile. They've adopted and adapted a variety of processes and techniques to best fit their organization. Along the way, and often without specific intent, they've used what we're calling Agile values to guide their approaches. Agile Development didn't invent those values - just gave us a single phrase to refer to them by.

Those foundational values are the important thing. It's quite possible to adopt a named Agile approach such as XP or Scrum and use all the techniques, but do so in a spirit contrary to Agile values. So while process, tools, documentation, contracts, and plans are valuable and many Agile approaches describe approaches for all of them, it's the soft stuff - people, collaboration, responsiveness – Agile Development emphasizes that makes the difference.

This book describes the innovative techniques and approaches that emerge when the Agile value system is applied to the process of software product design and requirements.

General Specifics

If you're new to Agile Development approaches, I might have led you to believe that there's little consistency among Agile approaches. That's not exactly true. There are a few durable concepts that can be found in most Agile approaches - sort of an emerging Agile best practices. When we combine these best practices, and step far enough away, most Agile approaches actually begin to look very similar. If we're going to work with requirements in an Agile context we need to understand these common ideas. Let's discuss a model that gives structure to these common ideas.

Feature Development Builds Software

Most Agile development approaches break work down into small hopefully independent pieces of work. XP might call this a "user story," while Scrum might call it a "backlog item." I'll use the somewhat neutral term "feature" here. An Agile feature will dependably have a few qualities:

Features are developed in repeating cycles of designing, building, and testing. When designing we'll decide a few things about the feature. We'll build those few things. Then we'll test that the feature meets the small amount of design we've done. We'll cycle through these three steps a number of times before we can call the feature completed.

Sometimes a feature might be broken down into smaller development tasks completed by different developers. Each task might spin through the design-develop-test cycle a number of times before they're all integrated together to form a completed feature.

Iterative development builds features

Agile development uses an iteration to build features. An iteration is a time box with specific start and end dates. Iterations are commonly two to three weeks in length, but might vary shorter or longer in any specific organization - although much longer than a month is uncommon.

An iteration is preceded by a planning session to choose the features that will be built within the time box. This activity is normally collaborative and involves developers, testers, and business people requesting the features. With a plan in place the iteration begins and developers start work on features business people have chosen to build.

As the end of the iteration time box approaches, iteration testing might begin. Though each feature was tested independently, the group of features are generally tested together as a coherent whole.

When the time box ends, the iteration ends. Uncompleted features are carried forward for consideration in the next iteration's planning session.

Wrapping feature development with iterative development grows our model to look like this.

Iterations build to an incremental release

Short iterations give more frequent opportunities to gauge development performance, evaluate the current state of the product, and adjust features and priority. It may be difficult in this short amount of time to build a set of features complete enough to be placed into use by actual end-users. In situations such as this, it's helpful to package several small increments into a software release.

A release is preceded by a release planning session where one or more software releases are planned. During the planning session the features for each release are determined with an emphasis on choosing a set of features that makes each release useful to the people who'll receive it. During release planning, rough estimates for development time by feature are made. Using these estimates, release dates can be forecast along with the number of iterative development cycles it might take to complete a release.

With a release plan in place the features for a release are fed forward to iteration planning where features will be chosen to be built in the first iteration. The rhythm of iterative development kicks in and the features of the release are built an iterative chunk at a time.

As the end of the release date approaches, release testing might begin. Just as features were combined during iterations and looked at as a coherent whole, all the finished features for the release will be looked at and evaluated as a coherent whole with a special emphasis on the usefulness of the group of features in the release. Will the release be useful? Will it be marketable? Wrapping iterative development with incremental releases grows our model to look like this.

Development projects build products

Every release contributes to further the goals and vision of a product. If we're doing this work in support of a product consumer company, we're keen to make sure that each release helps us gain or at least hold onto market share. If we're releasing against an internal IT project it's important that each release support the goals of the project which are usually to increase the profitability, directly or indirectly, of the organization paying for the development.

A product, or project if your organization doesn't see this it as a product, is usually driven by a set of high level goals, a project charter, or a product vision. This charter or vision states the goals for the software. These high level goals might be expressed as financial objectives for product sales or increased efficiency along with a plan for how those objectives might be realized.

A product's life might span years or decades. Hopefully a project won't live that long. But as releases of the product are built and delivered the product/project is evaluated against the goals established for it in the charter.

Wrapping incremental release with product visioning and project chartering might change our model to look like this.

The Agile Waltz

As I'm writing this someplace in the back of my mind I'm recalling my feeble approaches at ballroom dancing. I'm hearing a voice in my head chanting "One-two-three, one-two-three…" as my feet try to remember which way to step when dancing to a waltz. As I think about Agile development I'm seeing that one-two-three waltz tempo repeated in the cycles I've described above "plan-build-test, "plan-build-test…" Agile development, when done well, is as rhythmic as a good waltz.

Think about dancing to a nice waltz for a minute. (If you're not a good dancer, go ahead and imagine that you are.) When you're done, let's look back at the model we've built.

The Recursive Build

This is where our dance metaphor breaks.

You might notice that the feature is a three step process, but iterations, releases, and projects might look like they only have two steps. The feature has a step labeled "develop" where the feature we've conceived of is coded in some programming language. This is the "building" part of the features. You'll notice that the feature "bubble" on our model is glued into the iteration bubble. Iterations are built out of features - or design, developing, and testing features is the "building" part of our iteration. Continuing out, releases are built from iterations, and products and projects are built from releases. The building part of each cycle is accomplished by the smaller cycles it encloses.

At the building bit of each cycle we start by planning again, then building, then evaluating. At the feature level, the bubble marked "develop" over-simplifies what's really going on there. In reality an Agile developer will cycle through the plan-build-test cycle many times in the process of development. A developer might do a tiny bit of planning or design by writing some unit test code. That would be followed by writing some actual software code. That would be followed by running the unit test code to test that the design worked as hoped. One-two-three, One-two-three….

The Big Plan Up Front

You might be starting to notice how much planning is done in Agile development. Every cycle begins with a plan. In a new product or project all the planning bits of a cycle feed into each other. Plan after plan after plan after plan. In our model of swirling bubbles, it might be a little hard to see what's happening over time. Let's squash the planning flat and take a look.

You're seeing how each cycle's planning precedes and feeds into the next. What's important to note is that the granularity of planning changes for each cycle with the planning step for each cycle growing more detailed or fine grain as the cycle time decreases in length.

Design is a Synonym for Plan

You might notice that I've been a little loose with my use of the terms "plan" and "design." In my head they used to be very different things. Planning was an approach I used to schedule activities and resolve their dependencies in order to reach a goal. Design was a creative process where I'd choose specific qualities of a thing it might have in order for it to satisfy a goal. The difference might have been the accepted use of creativity in design vs. a firmer analytical approach that I might adopt when planning. Over the years the boundaries for me have begun to blur.

As I've learned more about user centered design approaches, software design has adopted a flavor of being more analytic - analytic without losing its dependence on creativity. Using the analysis models and techniques supplied in user centered design approaches has allowed me to be more successfully creative. As I've run projects over the last decade, I've seen that the job of keeping a project on course and dependably delivering requires much more creativity and intuition than I'd originally suspected.

Merriam Webster clearly calls out design as a synonym for plan - which, given my previous views confused me. Today I'm seeing software development as a continuous process of design and planning, followed by doing and evaluating.

One-two-three, one-two-three…

Given that we'll be doing so much continuous design, how does a user-centered design approach lend itself to our need to segment design into packages that vary in granularity from course to fine grain? Jesse James Garrett's User Experience model discussed in the next chapter will help to visualize that.

Feedback, Feedback, Feedback

You might also notice that each cycle's testing step feeds the next higher cycles'. Individual features are tested against the original goals described for the feature. All iteration features are tested together with respect to each other and the goals of the iteration. The results of all iterations are tested together with respect to the goals of the product. As with planning and design, testing is done continuously and at different levels of granularity from the smallest individual feature to the product as a coherent whole.

Test is a Synonym for Evaluate

OK, it's not really a synonym from the dictionary's perspective, but I'll use it that way here. It's important to understand that all this testing isn't about the features merely being implemented and working as expected, but about the features accomplishing the goals set out for the product, release, and iteration. It's quite possible to have a feature that works as designed and required, but simply doesn't have the qualities predicted for the product and release. It's quite possible for all features designed and required to be implemented for a release, but for that release not to fulfill the goals of end users or business people who paid for the development. At the end of each cycle of feature development, iterative development, or release development, it's critical to evaluate the value of the feature relative to the goals of the product, release, and iteration.

The results of all this testing and evaluation feed back into the iteration plan, release plan, or project charter.

By feeding back, I mean that the tangible results from feature development, iteration development, and product release might indeed change features, iteration plans, release plans, and project charters. Yes, I mean they might change requirements. Evaluation cycles give us a chance to learn from what we've done thus far and adapt early to information we didn't know when initially planning and designing.

These feedback cycles that seem to invite change to requirements are an example of a process adaptation to the Agile value of responding to change over following a plan. Effectively gathering and responding to feedback is one of Agile development's defining characteristics, and most troublesome problems to solve.

Evaluate progress and process along with the product

A common technique used in Agile approaches is a Reflection or Retrospective session. Normally this takes place at the end of an iteration and release.

The reflection session is a good time to discuss the results of product evaluations.

It's a good time to measure our progress as well. Are we completing features at the rate we predicted? At this rate, will we finish on time?

One of the primary goals of the reflection session is to look back at the specific process followed over the last cycle and determine if the development approach is meeting its intended goals. Just as the evaluation of features may result in changes to product design or requirements, evaluating process may result in changes to process or tools.

When we unroll an Agile incremental release iterative development lifecycle and look at it sequentially over time, it might look something like the picture below. You can see all the instances of planning above the center line, and evaluation below.

But Wait, There's More…

The Agile Development model discussed here stops at the product and project. But, we all know the world is a bit bigger than that. Products and projects usually exist in a portfolio of possible products and projects. The decisions we make about specific products are usually made in the context of other products and the time resources available to them. The portfolio level decisions are made against the backdrop of the company who's paying for the products and their general goals, financial or otherwise. If that company is owned by a larger parent company, that company's general goals and direction is help set by the parent company. The concentric cycles continue to radiate up and out. In all cycles there's an element of decision making and planning, an element of execution and building, and finally an element of evaluation, reflection, and re-planning.

Agile Development Distilled

Agile development may be characterized by short cycles of planning and design, building, and testing and evaluation.

Cycles may be at a very detailed and tangible level - like the design, building, and testing of a single feature. Cycles may be at a higher abstract level like the design and planning of a product release, its construction, and subsequent evaluation of the finished product.

Much of the challenge in Agile development is found in the area of planning, design, test, and evaluation. How can we plan at a high level and defer the design of small feature details for later? How can we frequently evaluate what we've built against our high level plans and adjust those plans accordingly?

Further reading:

• Agile Software Development: The Cooperative Game
• Extreme Programming Explained: Embrace Change
• Agile Software Development with SCRUM
• A Practical Guide to Feature Driven Development
• Crystal Clear: A Human-Powered Methodology for Small Teams
• DSDM: Business Focused Development
• Adaptive Software Development: A Collaborative Approach to Managing Complex Systems