Agile Software Development

I’ve been giving a number of presentations on Agile Software Development of late and I thought it might be useful to publish my take on Agile and how I present it. I have been using variations of this slide deck for my presentations.

Disclaimer: The following summarises my current thoughts about Agile software development. I keep refining my thoughts constantly so this may well become redundant over time. While I have written this article off the top of my head without any references, my thoughts are inspired from all the reading and research I have done so parts of it will likely bare resemblance to the writings of people like Martin Fowler, Alistair Cockburn, James Shore and Shane Warden.

What is Agile?

The definitive Agile would have to be the Manifesto for Agile Software Development, a document that was published in 2001 by 17 key individuals within the software industry. The Agile Manifesto outlines 4 values that Agile stands for:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

They made the point that while the things on the right can be valuable (e.g. documentation), they value the things on the left more.

I interpret these four values to mean that Agile is:

People-oriented
Value-driven
Client collaborative
Adaptive and flexible

Before I explain what I think these four statements mean in the context of Agile, let’s take a step back and apply some context to Agile.

Traditional Software Engineering

The vicious cycle of traditional Software Engineering
The vicious cycle of traditional Software Engineering; Figure 1 from http://www.renaissancesoftware.net/files/articles/ESC-349Paper_Grenning-v1r2.pdf

I’m of the opinion that traditional Software Engineering is fundamentally flawed.

I think that applying traditional engineering methodologies to software doesn’t work because software development is fundamentally different from traditional engineering in two major ways:

  1. Having an upfront requirements phase and corresponding requirements document doesn’t work because the client either doesn’t know what they want until they see the system evolving, or they think they know what they want, but what they really want is something different, or they know what they want, but that will change over time.

    Stable software requirements is an oxymoron.

  2. Traditional engineering involves a relatively short engineering design phase followed by a longer construction phase where the design is implemented by following a repeatable process. With software the implementation of code is actually a design phase rather than a construction phase and requires engineering design skills throughout. The construction phase is essentially non-existent. The traditional notion of a design phase in software is mostly useless (good software developers will be able to write the code or at least the tests well without needing to write a SDD) or at least should happen right before writing the code (for complex business logic it can help to visualise the process in a diagram before writing the code – this is where UML becomes useful).

    Software development requires engineering skills throughout the whole development cycle to get a good result.

I think that the four Agile values I just mentioned are a reflection of the fact that traditional software engineering doesn’t work and tries to address these two fundamental differences.

People-oriented

I think that Agile is people-oriented because it recognises that software development is an inherently people-oriented process in two major ways:

  1. Because requirements are not fixed you need to interact continually with the client to elicit requirements. This interaction is (or at least should be!) a core part of the software development cycle.
  2. Because implementation requires engineering skills you need skilled professionals with the right support and environment to get a good result.

Value-driven

At the end of the day we are working for our client and any work we do that doesn’t have value to the client is wasted effort.

In terms of software development, at the end of the day the client gets the most value out of working software. Anything that doesn’t contribute towards working software, such as producing documentation to tick boxes (e.g. requirements specifications) isn’t delivering value to the client.

In Agile projects this is taken a step further by literally getting the client to prioritise all tasks on a regular and continual basis so that at any point in time the team is working on tasks that produce the maximum value to the client at that point in time.

By working on these top priority tasks to completion the end result is that you almost constantly have production ready code. This gives the client the flexibility to be able to deploy whenever they like, however often they like with confidence. This isn’t easily possible with traditional software engineering and clearly delivers a lot of value to the client.

Client-collaborative

In my opinion this is the most important of the four values and the big differentiator and driver of Agile development. Agile software teams have continual, face-to-face contact with the client throughout the project; they collaborate closely with the client to develop the software.

Agile project management fosters a better relationship between the development team and the client they are working for. It leads to the client being more engaged and excited in the whole process and communicating better with the development team – I have seen this with every client we have worked for so far, they get so engaged and excited that they voice their excitement out loud.

It also results in the client being able to trust and follow the advice of the development team. An interesting side-effect of this is that projects that will not be successful are able to be more readily identified and culled before much effort is wasted – something I like to call successful failure.

One the other side of the coin, it means the development team better understand the client’s goals and where they are coming from and that helps them deliver better value for the client and be more engaged with the project.

You will (or should!) never hear words like “scope-creep” or “we can’t do that because it wasn’t in the requirements document” in an Agile project. Agile is about avoiding those kinds of us versus them relationships because they lead to a lack of trust and less than ideal outcomes.

Adaptive and Flexible

Agile projects involve the client re-prioritising and changing requirements on a regular basis. Agile project management methods ensure that this happens in a controlled way and the development team is flexible enough to cope well. This flexibility empowers the client to be able to respond quickly to changing market trends and, interestingly, take advantage of situations that yield a competitive advantage. Again, this isn’t easily possible with traditional software engineering.

Common Misconceptions

No documentation

If the client prioritises documentation as being important and delivering value then the team will work on it. Also, there will always be documentation that teams will produce because it’s important or necessary (e.g. handover documentation). The point is that a minimum amount of time is wasted on documentation that doesn’t deliver value, such as traditional requirements specifications. This can be done in a few ways:

  • Documentation is evaluated for usefulness / value before being produced, if there is no value in producing the documentation then it shouldn’t be produced.
  • Documentation is created in a succinct manner to get the core information across as easily and quickly as possible
  • Documentation can be created in an automated manner.

Obviously, adherence to standards and managerial demands will often make the first two points hard/impossible, so techniques like the third point can be investigated to try and ensure the least amount of time is “wasted” as possible.

I’ll make one last point; there is a lot of focus in Agile on technical excellence and testing. This means that the code quality should be very high. The tests allow you to refactor code with confidence, which makes it easier to keep the code of a high quality even as the design of the system evolves. Because of this, the code itself serves as the ultimate documentation of the system in combination with the tests. In essence, the code and tests become an executable specification.

Lack of process / cowboy coding

In actual fact, I’ve found the opposite to be the case. In order to follow Agile properly, the team needs to be very disciplined (daily stand-ups, retrospectives, pair programming and test-driven development are examples where the team needs to stay focused). Agile actually introduces more of a rigid structure to the development team.

With regards to “cowboy coding”, this certainly should not occur in an Agile team (if it is you are doing things wrong). Thought should be given to the architectural design of the software prior to development, preferably tests should be written prior to development and all production code should be peer-reviewed. All of these techniques (and others) are common sense and aren’t exclusive to Agile – they have been around for some time.

Agile is simple, thus it must be easy!

The thing that strikes me about Agile is that all of the values, principles, techniques and concepts behind it are very simple. They are essentially a combination of a lot of great ideas that have been applied to software for a long time.

Because of this, some people might think that Agile is easy. Certainly, as Agile becomes more widespread you start to see a lot of teams say they are “Agile” when in fact they simply picked up a few techniques related to Agile such as TDD and pair programming. In actual fact “going” Agile is a very hard process; it’s not easy at all!

As I already mentioned – doing Agile properly (by properly, I mean following everything in the methodology you choose; over time as the team becomes more experienced though they can use the retrospective process to change their process over time to suit their environment) takes a lot of self discipline. Agile is a big change from the more traditional ways of doing software development and as such it’s hard.

Agile is a change. Change is hard.

A couple more notes about this:

  • The team won’t become immediately more productive when first “going” Agile; the opposite will probably be true as they get used to the change.
  • Agile isn’t a silver bullet; you need to work hard to get it to work (i.e. you have to do it properly), but the end result will be that you will be successful.

It makes Software Engineering redundant

Some people will say that because Agile is such a change from traditional software engineering that it makes Software Engineering redundant and that Software Engineering has no meaning anymore. I would strongly argue the opposite; I think it makes Software Engineering more relevant:

The development team need to use the whole range of engineering skills from requirements elicitation, through to design, implementation and testing every week and every day in an Agile project.

This is a departure from the traditional scenarios where you might see design engineers and requirements engineers etc. With Agile you need better engineers that can utilise their full engineering skill set all the time. I think this is something that scares some traditional engineers and makes them reluctant to consider the Agile way of doing things.

Agile Methodologies

The Agile Manifesto outlines 4 Agile values and 12 principles that you should follow in order to realise the 4 values. In practice, you will become Agile by adopting an Agile methodology, which will outline a set of processes and / or techniques that take the Agile values and practices into account.

The two most common methodologies are eXtreme Programming (XP) and Scrum. XP has a solid foundation of technical practices that are good to consider adopting (e.g. pair programming, TDD, etc.) and some project management processes to follow in order to deliver projects in an Agile way. Scrum leaves you to decide what technical processes you want to adopt and focuses on a more rigid project management structure. My team started with XP and moved towards Scrum as we sorted out what technical processes we wanted to follow and when and wanted a more rigid structure to manage our projects. This worked well for us, so it’s something I can recommend.

Agile Project Management

Depending on which Agile methodology you choose, there will be a number of project management processes you will be expected to follow, following is a list of the most common (and in my opinion most important) ones.

Self-organising teams

Project managers don’t need to micro-manage Agile teams since you can trust that they are professionals and can manage themselves. This frees up the PMs to deliver value to the team (another way in which Agile is value-driven) by being able to focus on clearing roadblocks and liaising with external stakeholders rather than wasting everyone’s time with long, weekly meetings to try and understand what everyone is doing and why they aren’t following a (somewhat) arbitrarily designed Gantt chart that is weeks behind to the letter.

Yes, I am somewhat cynical about traditional PM applied to software, but that’s because I’ve seen it go wrong so many times. Yes, if you have a really good PM then it can work, but really good PMs are rare and it still takes a lot of effort and time (i.e. overhead).

Daily progress communication

This is an amazingly powerful tool – short 5 minute meetings every day where the core development team talk about what they did between the last meeting, what they plan on doing between now and the next meeting and any roadblocks they face. Any stakeholder (e.g. client, PM, managers, etc.) is invited to observe the meeting whenever they want and allows them to quickly get up to speed on the progress of the team (everyone is on the same page) and roadblocks are readily identified.

Continual process improvement

This is usually accomplished by a tool called the retrospective – a meeting (~1 hour long) where the team and potentially other stakeholders like the client attend and talk about what they think went right, what they think went wrong and any ideas for improvement based on what happened since the last retrospective (or variations of that theme).

The team will then collate the items identified into related groups and everyone will get a number of votes to vote on the items they think are most important to them to improve on. The top one or two items will then have a root cause analysis and the team will talk about how to address the root cause and also how any improvements can be measured.

By focussing on only one or two things it’s more likely that a permanent change will result rather than thrashing between the improvement of a large number of tasks. By ensuring there is a metric for improvement, this can be analysed at the next retrospective to see if the changes were effective and should be kept or dropped. By having the retrospective often you get continual process improvement.

The two key things to get right in an Agile project are the daily progress meetings (also called daily stand-ups in XP and daily scrums in Scrum) and retrospectives. If everything else the team is doing is wrong then you can still succeed since everyone is on the same page and you can fix the things that are wrong over time.

Regular review, planning and estimation

Enough said. Obviously, the client is heavily involved with this.

Very accurate and lightweight project estimation and tracking

Agile provides techniques that attempt to waste as little time as possible with project management overhead, while still being able to accurately deliver tracking and estimation of of tasks. This is through techniques like the storyboard, planning poker, burn down charts etc.

In particular, the estimation techniques in Agile really struck a chord with me because I feel that they lead to far more superior and accurate time estimates than what you usually see with more traditional techniques, even with the continually evolving requirements that you see in Agile projects. I could write a whole blog post about Agile estimation in it’s own right, so for conciseness I’ve simply included a PDF of my current notes about Estimation (largely collated after a very useful Agile user group session last year).

Face-to-face communication

All Agile methodologies focus on using face-to-face communication over other forms of communication wherever possible because it’s more efficient; this is summed up nicely by one of the 12 Agile principles:

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Technical Processes

The following are some of the most important technical processes that often accompany Agile:

  • Test-Driven Development (TDD), Acceptance Test-Driven Development (ATDD) and Behavioural-Driven Development (BDD)
  • Continuous Integration (CI)
  • Focus on code quality and maintainability (largely via continual code refactoring and use of good design practices and patterns)
  • Pair programming
  • Peer-reviewed production code

Who is doing Agile?

Agile is becoming more commonplace over time. Some of the big International companies (e.g. Siemens, Lockheed Martin, Motorola, Microsoft, Yahoo, Google, General Electric, Cisco Systems) and big Australian Companies (e.g. BankWest, Suncorp, Sensis, Telstra) are using Agile.

Barriers to adoption

Above all, I think that the principles of Agile are common sense. If it’s common sense why isn’t everyone doing Agile? I think there are two reasons, and I think these two reasons are also the main barriers to adoption of Agile:

  1. There is an endemic culture within University courses (and by association graduating software engineers) and within most software organisations of the traditional way of doing things. In order for Agile to be successful, you need the clients on board, management on board and the development team on board. To get each of these groups of people on board you need to break down their existing ways of thinking about software development. Changing culture is hard.
  2. Agile is a big change and as such there will be resistance to that change. There is always resistance to change. In order to successfully transition to Agile you need to embrace change.

Agile isn’t relevant to just Software

I think that the core values behind Agile can be relevant for non-software IT projects – if you are focused on delivering client value, working closer with the client, being adaptive and flexible and looking after your employees then that will surely result in better success.

I think that some of the Agile Project Management techniques should be investigated for application to non-software projects and teams because if they can be successfully applied then you will have less project management overhead and better communication between everyone.

I have read that some companies have implemented Agile values and techniques in non-IT areas such as HR and Finance, which is interesting. Also, one of the Agile methodologies, Lean, is actually based on techniques and processes that came out of the manufacturing industry (via the Toyota Production System).

Lean is well suited for operational work, so it’s not just about project work; operational software and IT teams can utilise some of these techniques.

There is a related concept to Lean called Systems Thinking that has wider applicability than software, I was recently linked to an interesting video that illustrates the basics behind it.

The future

I’m of the opinion (hopefully not naively!) that within the next 5 years Agile won’t exist anymore because everyone will be doing it; kind of like what happened to Object Orientation.

I think that the last software companies to change to Agile will be Defence companies because they are the most heavily document-driven.

4 Replies to “Agile Software Development”

  1. Well written. You have a knack for getting ideas out clearly.

    The first thing that comes to my mind is “what would the lawyers think”. There is no way to guarantee (or even estimate) if an agile project was successful, hence the customer can’t sue the supplier (i.e. development team) if things don’t work out.

    Granted the idea of a “successful failure” means the customer will have a clear view of potential failures pretty fast, but not until spending a fairly significant amount of money to get things started which there is no way to re-coup.

    The flipside is that the supplier is well protected from lawsuits which is very good for us as developers.

    I think this would prevent big companies from taking on agile projects with external suppliers. Agile works well for internal projects where the “client” is the same company but I think it would be difficult to engage two external parties in an agile project.

    1. That’s one of the 4 questions I almost always get asked when I give presentations on Agile.

      I think there are two situations to consider here which somewhat relate to what I said above about changing the endemic culture, but also relates to common business sense – you aren’t just going in trust a company because they are like “Hey, we do Agile, trust us to deliver”. In fact, if someone says that to you I would imagine they are probably one of the companies that doesn’t get it and aren’t doing proper Agile:

      Does the company trust you to deliver?
      Does the company not trust you / is new to the concept of Agile?

      For the first one, you can probably sign up to a contract saying you will deliver according to a release plan of some sort (e.g. first release after 6 months, second release after 12 months, or maybe even monthly releases, or shorter, if you are practicing continuous delivery then it’s up to the client how often they release; in terms of contractual obligations though, I imagine there would likely be milestone releases in the order of a number of months though). You would likely put some (vague?) criteria against the releases e.g. first release has functionality for the user to be able to do X, Y and Z, etc.

      Given that the client has the power to change requirements based on changing priorities the contract would need to have the flexibility to accomodate this. Now, let’s consider what happens traditionally vs in Agile when things don’t go according to initial plan (say there is a milestone release at 6 months that the company promised it’s users):

      Agile: You get to the 2 month (say; remember what I said above about estimation being more accurate in Agile) point and realise the list of features that the client wants according to the initial release plan can’t all be implemented within the 6 months because what the client wanted for some of them was a lot more complicated than originally thought (I’m imagining that the software company signed the contract and agreed that the release plan seemed reasonable based on previous work they had done, if it wasn’t obviously they would start the negotiations I’m about to talk about at the contract signing step). You let the client know that the list of features they wanted aren’t going to be able to be released at the 6 month milestone, but that the most important features, or the most important parts of all the features will be able to be accomodated. Given the long amount of notice and the fact they are willing to work with the client to make sure that the things important to them are in that release, in combination with the fact that they trust the company (and in the 2 months that has already happened with this project, it’s clear that the work so far is of good quality and there has already been delivery of working code of their most important feature(s)) means the client is probably fine with this. End result: There is a release after 6 months, it’s exactly what the client wants because they have been using it as it evolves for the last 6 months (and probably has done an extensive amount of user testing along the way, so it’s also what their users want). It’s also got the most important features that the client wanted, and as it turns out the ones they deprioritised weren’t actually that important afterall and they probably won’t even end up implementing them in the 12 month release because the users are coming back with really great ideas after using the system that the client wants to implement instead.

      End Result: Client happy, users happy, developers happy (they delivered value to the client, got praised for their good work and got paid)

      Document/Contract-driven: The requirements and design docs are done after (say) 2 months, and the developers start working. It’s fairly clear after 2-3 months that they aren’t going to hit the deadline, but they simply tell the workers to work harder (or perhaps bring in a couple of new devs; mythical man month anyone?). They aren’t going to communicate this to the client because either they think they can get it done still (rose-tinted glasses) or they don’t want to risk the client waving the contract in front of their face. The 6 month milestone rolls past and they can’t release because the system isn’t fully finished and they are half-way through two features so they can’t even release some of the system since it will break. The PMs and the client start having meetings with a lot of finger pointing on both sides and the client waves the contract in their face, saying they didn’t deliver. At this stage there are two possible courses of action:

      The client threatens to not pay the developers, and then they threaten to stop working on it and not deliver if they don’t. This is where it can get really ugly…
      The client accepts the PMs promise that it will only be another week, this happens week after week until the milestone is reached (let’s say only 1 month late). At this point, the users of the client are unhappy they didn’t get the product when originally promised and the software is buggy (because there was no / little time for testing since the software was rushed through the testing phase so it could be released), the client isn’t happy because after looking through the software it turns out what was implemented wasn’t what they were after at all (there might be more waving of contracts here, but then the developers will wave the requirements spec saying they implemented that and the client signed it off – either the requirements are ambiguous, or what the client wants is different from the requirements spec; either way the software isn’t delivering the value they wanted). Being late with the first milestone throws the second one way out, particular since the developers need to spend a lot of time fixing bugs. Because they are hurriedly fixing bugs so they can continue onto the second milestone and because they are already rushing to meet the second milestone the code quality steadily gets worse and worse and the bugs take longer and longer to fix because of it, each fix adding more technical debt to the code. By the time the second milestone is ready (if that even happens) it will be very, very, very late and very, very buggy.

      End result: Client not happy, users not happy, developers not happy (and potentially not paid) and won’t be rehired by that client.

      In the second case where the company doesn’t trust you / is new to Agile it might be tough to get a contract that is flexible enough to work well in an Agile environment. In that case you can go with the contract the client wants and simply attempt to deliver (a bit risky, especially since you need the client to be actively engaged with the project, which can be tough if they are new to Agile) or hope that they will want to change the contract to give them more flexibility over changing the requirements. Alternatively, another technique that I have heard about is where you convince the client to let you do a (say) 2 month proof-of-concept in an Agile way, and a very accurate estimation of cost and time after that proof-of-concept. The excellent work you have done and the good relationship you have started to form with the company will then help to convince them that what they want isn’t possible in the time or budget that they asked for at the end of the 2 months, but that you can deliver a subset of the (most important) functionality after the 6 months they wanted.

      Yes, these scenarios are slightly exaggerated to make my point and contrast very clear and yes, I am very cynical about document/contract-driven projects. However, from what I’ve read and experienced these “exaggerated” scenarios do happen and I’m cynical because I’ve seen it fail time and again.

      tldr; you can have contractual obligations in Agile, but they don’t need to be as strict (i.e. you can be more flexible) and this is a good thing for both the developers and the client.

Leave a Reply

Your email address will not be published. Required fields are marked *