How to Use Learning Milestones

In tech we have huge bias for Execution — building things and launching them fast and with the right quality. A million underwhelming launches later, we now know we should focus on something else: Learning — systematically discovering the right product, market and business model. All execution and no learning exposes you to launching a great product no one needs (we’ve all been there). All learning and no execution gets you stuck in the quagmire of analysis paralysis. How to progress on two different fronts at the same time? What’s the right balance?

I’m going to go over three options you know — Waterfall, Alpha/Beta and Lean Startup, and one you may not know, but should — Learning Milestones.

Waterfall

Waterfall (the best practice when I got started in the industry in the 1990s) a pure attempt at execution optimization. It splits the project neatly into stages, each performed assembly-line-like by a different team. Putting aside all the known issues with waterfall (primarily that it doesn’t work), from a pure learning perspective it is very broken:

• The product definition phase is supposed to encompass all the research needed to define the product in full, months, sometimes years, before the product ships. In reality, with no product to test on, very little actual learning happens here. Most of the “requirements” are created based on ”market research”, opinions and consensus, and are rushed out out under time pressure so not to delay the development phase.
• Invariably new information is learned during the development and QA phase, but the process makes it very hard to make all but minor changes, certainly no pivots.
• Actual learning starts at the very end, after the product was already built, polished and launched — the worst possible time to make big changes, when everything is publicly visible and the pressure to declare a win and move on is at its highest.

Waterfall attempts to optimize for execution, but essentially kills early learning.

Sidenote: Agile Development is supposed to fix a lot of this, however most teams are actually using a hybrid of Waterfall and Agile (Water-Scrum-Fall). This allows for more learning in the development phase, but doesn’t solve the core late learning issues of waterfall.

Alpha/Beta, Dogfood, Previews and % Launches

Recognizing the risks of pushing learning to the very end, tech companies started creating project milestones aimed at collecting user feedback: Alpha and Beta testing, Preview and %-Launches. Similarly, many larger companies have their employees “eat their own dogfood” — use not-yet-launched products internally and submit feedback. Done right these test milestones have great positive effects on the product.

To see why let’s zoom into the Development & QA part of the waterfall. Schematically the development project starts when the specifications are handed down to the engineering team and ends when the team launches the specified product, let’s call it ‘X’ (this is obviously a huge oversimplification, but bear with me).

Adding a single beta test milestone injects a dose of real user input into the process and, if handled properly, can send the project into a new trajectory, resulting in X’ — an improved version of planned product that better matches user needs. Effectively, the beta milestone moved learning to an earlier point in the project, reducing wasted effort on building and launching the wrong product.

While certainly an improvement, this is far from a complete fix for waterfall. The versions we test are all too often the finished product minus bug fixes or a few features. Testing starts at a late point in the project life cycle with the launch deadline looming and inertia pushing against major changes or pivots. It takes a very disciplined team to stop at the beta phase and go back to the drawing board. And so X’ is typically quite close to X and the end result is the same — a product that misses the mark.

Done right Alpha, Beta, Dogfood and Preview releases can put your product on a better trajectory, but they cannot turn a fundamentally bad product idea into a good one.

Lean Startup

Lean startup flips the balance on its head — learning is the key focus and the product is built while learning. This introduces a number of important innovations:

• No separate product definition phase — the “spec” (if there is one) emerges as a by-product of learning.
• Continuous learning from day-1, through qualitative research (customer interviews, surveys etc), MVPs and build-measure-learn loops.
• Validating learning and tracking progress towards goal (Innovation Accounting) using quantitative metrics, experiments, cohort analysis etc.
• Learning leads to informed persevere or pivot decisions.
• Other development practices based on the Toyota Production System (TPS)

“Validated learning is the antidote to the lethal problem of achieving failure: successfully executing a plan that leads nowhere.” — Eric Ries

Lean Startup is heavily inspired by the process of scientific discovery — starting from a product idea that is based on what we know now, but gradually arriving at potentially different and much more profound end product.

Lean Startup introduced a fundamental leap forward, but it’s not without its challenges:

• Requires teams to learn a completely new way of thinking and operating, one that flies in the face of a lot of old conventional wisdom.
• With no guidance can be easily misinterpreted and or partially adopted.
• Perceived as appropriate only for horizon 3 innovation (early-stage startups, emerging business). I’m not sure I agree, but that’s a topic for another post.

Learning Milestones

This is a method we used a lot in Gmail, generally with very positive results. It doesn’t break away completely from the sequential progression legacy of waterfall, but does allow for early learning, course-correction and pivots.

The thing to notice is that projects tend to arrange themselves around a series of intermediate milestones. Traditionally these are about internal design and dev phase gates.

The trick is to replace these somewhat opaque eng/UX milestones with learning milestones, each producing a somewhat more evolved version of the product that is then immediately tested with an ever larger group of users for a longer period of time. What’s learned is then fed back to the team and is used to decide what to build in the next milestone — an opportunity to update the spec or to pivot. The team is instructed to only focus on the next milestone and not to rush for the final launch.

The goal of the team is to learn by putting an ever more complete version of product in front of an increasingly larger group of users for a longer duration.

Benefits

• Collects for a lot of user feedback early on. In fact doing this right, you are rarely surprised when launching as by now you heard from hundreds of users throughout the project.
• The end product (Y) is almost always significantly better than your original idea (X).
• Not a stark departure from the old alpha/beta/dogfood and so presents lower barrier for adoption.
• Makes the project far more alive and exciting for the team — at any given point you’re working on something that’s days or weeks away from being put in front of users.
• The milestones are in a sense mini-launches. Daunting as this sounds it’s a a very good thing because with each successful mini-launch the teams gets a justified sense of progress and achievement, which improves team bonding and morale (if you ever worked on a not-launching-anything-for-18-months project you know why this is great).

Challenges

• The end product Y is not as fundamentally different from the planned product X as might be achieved with Lean Startups (unless you’re willing to stop and pivot big time half-way).
• Many of the early learning milestones are qualitative. Qualitative results are easier to misinterpret. In general the learning is less rigorous compared to lean startup as there’s no deployed MVP until a later stage.

Final thoughts

It’s surprising how many teams are still stuck in old school waterfall and alpha/beta. If you’ve never used them, I highly recommend trying Lean Startup and Learning Milestones. The former is better when there’s lots of uncertainty in your project, the latter perhaps more suitable for incremental changes (but your experience may be different). Either way you’ll build and learn simultaneously, and greatly improve the odds of launching something that your users will love.

Itamar Gilad (itamargilad.com) is a product consultant helping tech companies build products that deliver and capture tremendous value. He specializes in product strategy, product market fit, growth and innovation. Previously he was lead product manager at Google, Microsoft and a number of startups.