I have been working on an "Agile" product with our IT Project Standards and Governance group at work. When analyzing the values and principles that are espoused in our "gated" product, I realize that the difference between agile and traditional (waterfall) project methodologies is singular. There is all kinds of hyperbole around why agile solves problems that waterfall doesn't, but that is really not on the mark. As Glen Alleman of Herding Cats says "Bad project management is simply that", meaning that the problem is less with the methodology than the execution. What is different between agile and waterfall is really a small number of preferences, and ones that may only apply well in the software development domain.
The main difference is which of the primary project constraints the methodologies have a bias towards, or prefer not to vary:
Waterfall prefers to vary time and budget whereas agile prefers to vary scope. If this sounds incredibly oversimplified, think through it. By stacking all the analysis up front, the cost of varying scope in a waterfall project continues to increase over time, while the benefit of varying scope decreases. This is necessarily true, because all the analysis and design work is done "up front", before we really know how much effort is in the plan, and then when we make the decision, that work is "sunk cost" that usually must be redone if things change later. The theoretical benefit of this approach is that we can optimize the development sequence to minimize cost of re-work. This assumes that we know how to perform this optimization, and that other factors (like emerging requirements and budget and time constraints ) don't cause us to compromise this benefit.
On the other hand, Agile has a preference for a static team (it's hard to be predictable using agile metrics while your team is a moving target), and therefore a fixed burn rate, and uses typically time boxes to manage duration. But agile prefers a sequence that has implies less waste when changing scope, because rather than doing analysis and design "up front" it has a "just in time" approach – meaning within a time box. Thus the cost of changing the scope of the next time box is nothing or just the cost of re-planning, because little pre-work has been done. The risk resulting from this that causes concern is that since we don't know "everything" up front, we will learn things that cause us to go back and re-do something that has already been developed.
In theory, as analysis and design activities of a software development life cycle are completed, knowledge of the work required increases, and the risk of unknown is understood, and risk of rework correlates to the unknown. None of this really helps deal with the risk of business driven change (market change), or constraint change (reduction in time or budget). The traditional lifecycle creates waste out of the analysis and design work that was completed for capabilities that are abandoned when these events occur. The agile lifecycle reduces the waste resulting from "change". while dealing with waste (in rework) resulting from risk of unknown by allowing the stakeholders to trade lower value capabilities for higher in order to align with budget and time constraints.
Because the agile lifecycle allows the delivery of incremental value to the end user earlier (it does require everything to be done to release), the business gets value sooner, while the risk of spending without any return on investment when a project is abruptly cancelled or funding is abruptly cut is much lower. Furthermore, the agile lifecycle expects shorter feedback loops on smaller scope value targets, which creates greater focus both in the stakeholder community and on the delivery team.
So which is better? It depends on where your risk comes from. If your risk comes from the unknown, and there is little risk of the goals or funding changing in the middle of the the project, or if there is limited value in delivering part of the product early in the lifecycle, then a traditional lifecycle may work well. If your delivery in a more volatile environment, where business needs changes very frequently, where funding is driven by short horizon events, or where delivering partial solutions as early as possible is valuable – then an agile lifecycle offers some distinct advantage.