History's role is not to help us predict the future, but to prepare us for it.
Today we talk about product management, agile, waterfall, projects, iterative and measures of success. I enjoy putting this in a historical context. Because those who fail to learn from history are condemned to repeat it. History can help us prepare for the future. Jim Highsmith's career over the last six decades is a good start.
Wild West (1966-1979)
Software engineering was in its infancy and the knowledge of how to software development negligible. Software project success was measured by completion and cost. Businesses operated on the premise that the world was predictable and if plans failed to materialize, the problem was execution, not planning. Change was the "unusual state".
Structured Methods and Monumental Methodologies (1980 - 1989)
This was an era where management theory was firmly stuck in the command-control model. IT projects were generally viewed as out of control. Managers assumed building software was similar to building a warehouse. They often knew little about the value of IT systems, even though productivity increased, and focused therefore on "out of control" cost and schedule overruns.
Structured methods should bring discipline and control to software development.
But the attempt to go from the undisciplined Wild West to a more disciplined, formal approach overshot and focused on the wrong things - processes and documents.
Formality was mistaken for discipline. Formal processes, phase reviews and documentation added a level of bureaucracy that overshadowed the benefits of structured techniques. It worked for tangible products and construction-type projects though.
The belief was that documentation was enough to convey all the information required by individuals in the next phase of work.
A paper by Winston Royce in 1970 is often credited with introducing the Waterfall model of software development (a term he never used in the paper). However, Royce did not advocate for the model in its strict, linear form. He highlighted its limitations and risks, particularly the challenge of accommodating changes once the project is underway. To address these shortcomings he argued for feedback loops, overlapping development phases, and the need for more flexibility and adaptability.
But the waterfall lifecycle was greatly influenced by the surrounding ethos, manager's thinking and the serial "hardware" mindset of the times. Departments outside of IT also adapted their work to conform to the serial mindset. For example, accounting started to classify operating (OpEx) versus capital (CapEx) costs.
Too much structure reduced problem solving and innovation; too little created chaos and ineffectiveness.
The Roots of Agile (1990 - 2000)
Executives now cried for help. "It takes too long, costs too much, and doesn't meet our needs".
In response, software developers began experimenting with RAD (rapid application development) methodologies. They assembled a small team, delivered working software every week, and conducted "customer focus groups" every Friday to get feedback. This was about value, quality, speed, leadership, and collaboration.
The 1990s were the decade where software outsourcing to solve technical debt and the high-cost mess began to flourish. The assumption was that less expensive programming could be accomplished with minimal communication. But much of the outsourcing was based on the false premises of predictability and prescriptive practices.
Worse, it stripped internal IT teams of the expertise they would need in the coming Internet upheaval.
The methodology of software development continued to be deterministic and formal in the 1990s, but slivers of RAD and iterative development began to creep in, especially for rapidly expanding Internet applications.
The 1990s were the decade of process for IT. We sought not just perfection in the way we built software, but total predictability. It wasn't enough just to do things right, we also had to say in advance exactly what we intended to do, and then do exactly that.
The Agile Era (2001 - 2010)
Demand exploded. Acquiring and retaining technical talent proved difficult. Many companies didn't have the money or the talent pool to respond to the pressures of the 2000s. IT organizations faced pressure from the high costs. Companies had to innovate and expand their technical and product design capabilities.
Technical debt, which had been growing rapidly, was still almost invisible to business executives. IT struggled to convince business leaders to approve the investments required to reduce technical debt, thereby enabling continuous value delivery.
On February 11-13, 2001, 17 software developers met at the Snowbird ski resort in Utah to talk, ski, relax, and try to find common ground leading to the Agile manifesto.
Many software engineers were concerned about the nascent agile movement. They viewed agile development as a retreat to ad hoc practices of the past. But in reality, teams were highly disciplined, but worked somewhat informally. The emphasis shifted from documentation towards collaboration, replacing traditional formality.
From 2001 - 2004, individual teams received dispensation to try this "agile stuff". Teams focused on iterations, stories, daily stand-ups, backlogs and co-locating teams - but the core technical practices were often bypassed.
Agile projects' success was often downplayed by others: "it was just a small project", "it was a new project", "they didn't have to follow our standards". General comment from Agilists was "We don't need any project management". But the management of tasks rather than people was what they were complaining about.
From 2005 to 2010, success versus failure with organizational implementation of agile development rested with courageous executives who understood agility at their core. Courageous executives thrived on adventure, the corporate kind, and had the ability to sort through a myriad of opportunities, engage others with their enthusiasm, and demonstrate results through action.
Another success factor was that Agilists had enough background in change models to muddle through team-level implementations. At an enterprise level, it made a difference when they received help from organizational change experts.
Do we implement from top to bottom, or from bottom to top? Do we start with a few teams and user their experience to seed others, or do we "sheep dip" everyone? What was the strategy for moving agility up (the organizational hierarchy) and sideways? How do we instill both being agile and doing agile? Whose change model and approach do we use? Agilists were simply not experts in change management.
Other success factors were demonstrated business benefits, practices that appealed to engineers and a manifesto that stated a clear purpose and principles for the movement.
Digital Transformation (2011 - present)
The problem now wasn't scaling agile, it was scaling agility and innovation at enterprise levels.
Remember the lessons from agile projects: To change behavior and culture, you must change measures of success. Many managers wanted agility, but with traditional performance measures (planned scope, schedule and costs).
Over the last six decades, measures of software development success have evolved from completion to customer value. Nothing drives change like the way organizations measure success, but few things are harder to change than those measures.
The Digital Transformation period explored what becoming a digital enterprise meant. Enterprise executives developed digital strategies but found their transition from strategy to operating models lacking.
Enterprises today should be seeking to become digital transforming rather than to achieve digital transformation. Transformation indicates completion of a stage, whereas transforming suggests a constant state of becoming.
More than six decades of change have required organizations to constantly adapt by addressing the following areas:
- Measures of success - this can drive or restrict change.
- Being digital demands a better business and technology partnership.
- An operating model linking strategy to action.
- Creating a quickly malleable organizational model.
- Management suited for the digital era.
Preparing for the future
Over the 60 years, have anyone notice the rate of change in the world, in technology, and in business slow down? Change has spiraled constantly upward, not only linearly, but bordering on exponentially.
One conclusion from history is that agility, not agile, should be the goal. Agility is a mindset; agile is a type of methodology. Failure to make this differentiation has left organizations with prescriptive methodologies, while leaders who embrace agility have the best chance of thriving in the future.
Change takes more than a one-day workshop. No one changes deeply held mental models easily. Even when organizations and managers intend to start the journey with the right mindset, many don't realize the magnitude and complexity of this transformation.
The root cause of success is people and their interactions, and the root cause of failure is also people and their interactions.
