History's role is not to help us predict the future, but to prepare us for it.
Today we
talk about product management, agile, waterfall, projects, iterative and
measures of success. I enjoy putting this in a historical context. Because
those who fail to learn from history are condemned to repeat it. History can
help us prepare for the future. Jim Highsmith's career over the last six decades is a good start.
Wild West (1966-1979)
Software
engineering was in its infancy and the knowledge of how to software development
negligible. Software project success was measured by completion and cost.
Businesses operated on the premise that the world was predictable and if plans
failed to materialize, the problem was execution, not planning. Change was the
"unusual state".
Structured Methods and Monumental Methodologies (1980
- 1989)
This was
an era where management theory was firmly stuck in the command-control model.
IT projects were generally viewed as out of control. Managers assumed building
software was similar to building a warehouse. They often knew little about the
value of IT systems, even though productivity increased, and focused therefore
on "out of control" cost and schedule overruns.
Structured
methods should bring discipline and control to software development.
But the
attempt to go from the undisciplined Wild West to a more disciplined, formal
approach overshot and focused on the wrong things - processes and documents.
Formality was mistaken for discipline. Formal processes, phase reviews and
documentation added a level of bureaucracy that overshadowed the benefits of
structured techniques. It worked for tangible products and construction-type
projects though.
The
belief was that documentation was enough to convey all the information required
by individuals in the next phase of work.
A paper
by Winston Royce in 1970 is often credited with introducing the Waterfall model
of software development (a term he never used in the paper). However, Royce did
not advocate for the model in its
strict, linear form. He highlighted its limitations and risks, particularly the
challenge of accommodating changes once the project is underway. To address
these shortcomings he argued for feedback loops, overlapping development
phases, and the need for more flexibility and adaptability.
But the
waterfall lifecycle was greatly influenced by the surrounding ethos, manager's
thinking and the serial "hardware" mindset of the times. Departments
outside of IT also adapted their work to conform to the serial mindset. For
example, accounting started to classify operating (OpEx) versus capital (CapEx)
costs.
Too much
structure reduced problem solving and innovation; too little created chaos and
ineffectiveness.
The Roots of Agile (1990 - 2000)
Executives
now cried for help. "It takes too long, costs too much, and doesn't meet
our needs".
In
response, software developers began experimenting with RAD (rapid application
development) methodologies. They assembled a small team, delivered working
software every week, and conducted "customer focus groups" every
Friday to get feedback. This was about value, quality, speed, leadership, and
collaboration.
The 1990s
were the decade where software outsourcing to solve technical debt and the
high-cost mess began to flourish. The assumption was that less expensive
programming could be accomplished with minimal communication. But much of the
outsourcing was based on the false premises of predictability and prescriptive
practices.
Worse, it
stripped internal IT teams of the expertise they would need in the coming
Internet upheaval.
The
methodology of software development continued to be deterministic and formal in
the 1990s, but slivers of RAD and iterative development began to creep in,
especially for rapidly expanding Internet applications.
The 1990s
were the decade of process for IT. We sought not just perfection in the way we
built software, but total predictability. It wasn't enough just to do things
right, we also had to say in advance exactly what we intended to do, and then
do exactly that.
The Agile Era (2001 - 2010)
Demand
exploded. Acquiring and retaining technical talent proved difficult. Many
companies didn't have the money or the talent pool to respond to the pressures
of the 2000s. IT organizations faced pressure from the high costs. Companies
had to innovate and expand their technical and product design capabilities.
Technical
debt, which had been growing rapidly, was still almost invisible to business
executives. IT struggled to convince business leaders to approve the
investments required to reduce technical debt, thereby enabling continuous
value delivery.
On
February 11-13, 2001, 17 software developers met at the Snowbird ski resort in
Utah to talk, ski, relax, and try to find common ground leading to the Agile
manifesto.
Many
software engineers were concerned about the nascent agile movement. They viewed
agile development as a retreat to ad hoc practices of the past. But in reality,
teams were highly disciplined, but worked somewhat informally. The emphasis
shifted from documentation towards collaboration, replacing traditional
formality.
From 2001
- 2004, individual teams received dispensation to try this "agile
stuff". Teams focused on iterations, stories, daily stand-ups, backlogs
and co-locating teams - but the core technical practices were often bypassed.
Agile
projects' success was often downplayed by others: "it was just a small
project", "it was a new project", "they didn't have to
follow our standards". General comment from Agilists was "We don't
need any project management". But the management of tasks rather than
people was what they were complaining about.
From 2005
to 2010, success versus failure with organizational implementation of agile
development rested with courageous executives who understood agility at their
core. Courageous executives thrived on adventure, the corporate kind, and had
the ability to sort through a myriad of opportunities, engage others with their
enthusiasm, and demonstrate results through action.
Another
success factor was that Agilists had enough background in change models to
muddle through team-level implementations. At an enterprise level, it made a
difference when they received help from organizational change experts.
Do we
implement from top to bottom, or from bottom to top? Do we start with a few
teams and user their experience to seed others, or do we "sheep dip"
everyone? What was the strategy for moving agility up (the organizational
hierarchy) and sideways? How do we instill both being agile and doing agile?
Whose change model and approach do we use? Agilists were simply not experts in
change management.
Other
success factors were demonstrated business benefits, practices that appealed to
engineers and a manifesto that stated a clear purpose and principles for the
movement.
Digital Transformation (2011 - present)
The
problem now wasn't scaling agile, it was scaling agility and innovation at
enterprise levels.
Remember
the lessons from agile projects: To change behavior and culture, you must
change measures of success. Many managers wanted agility, but with traditional
performance measures (planned scope, schedule and costs).
Over the
last six decades, measures of software development success have evolved from
completion to customer value. Nothing drives change like the way organizations
measure success, but few things are harder to change than those measures.
The
Digital Transformation period explored what becoming a digital enterprise
meant. Enterprise executives developed digital strategies but found their
transition from strategy to operating models lacking.
Enterprises
today should be seeking to become digital transforming rather than to achieve
digital transformation. Transformation indicates completion of a stage, whereas
transforming suggests a constant state of becoming.
More than
six decades of change have required organizations to constantly adapt by
addressing the following areas:
- Measures of success - this
can drive or restrict change.
- Being digital demands a
better business and technology partnership.
- An operating model linking
strategy to action.
- Creating a quickly malleable
organizational model.
- Management suited for the
digital era.
Preparing for the future
Over the
60 years, have anyone notice the rate of change in the world, in technology,
and in business slow down? Change has spiraled constantly upward, not only
linearly, but bordering on exponentially.
One
conclusion from history is that agility, not agile, should be the goal. Agility
is a mindset; agile is a type of methodology. Failure to make this
differentiation has left organizations with prescriptive methodologies, while
leaders who embrace agility have the best chance of thriving in the future.
Change
takes more than a one-day workshop. No one changes deeply held mental models
easily. Even when organizations and managers intend to start the journey with
the right mindset, many don't realize the magnitude and complexity of this
transformation.
The root
cause of success is people and their interactions, and the root cause of
failure is also people and their interactions.