> NOW AND THEN_

Building got crazy fast. Deciding what to build didn't.

· AI · TOOL · PRODUCT DEVELOPMENT

When turning new ideas into working products has never been so fast before, it matters even more to select the right things to build.

Building a lot faster

A few weeks ago I built an animated funnel diagram for my sales tool. It took the AI about half an hour. It looks great, and it has added exactly nothing to my sales.

I used to be a professional developer, so I know how much time it would cost me to build such a feature without AI: days. When starting out with AI, I let it generate small pieces of code that I would review and refactor to be correct. Today I let it build whole features without looking at every change. The speed is insane.

At first, this was great! It still is, but now that my list of ideas to develop was quickly diminishing, I recognised I was not really thinking about what value each idea was going to provide. I could build it so quickly it did not really matter I thought. But soon I saw I had built things that were of doubtful value, or certainly not as valuable as other things. Besides the funnel, I built a page showing my stock investments, data I already have in my broker’s app. I’ve never looked at the page since.

So two things are happening at once. Building got so cheap that we stop thinking before we build. And building got so fast that there’s no time left to think.

No more time to refine

Product Owners never knew exactly what to build. Most ideas are hypotheses. Assumptions that some functionality would provide a user some benefits. We are dealing with uncertainty. That's why we use an Agile way of working. We deal with this uncertainty by iteratively developing a product and validating the outcome of each iteration. These iterations could be a sprint, or multiple sprints of hard development work. At the end we deliver a product increment that a user can work with. And we find out whether the assumptions that we worked from were right or not.

But now these iterations are becoming very short. The complexity of how to build has decreased significantly. Building is no longer the bottleneck and hypotheses can be tested much earlier. That is great, but it means that Product Owners and users are going to have to review and validate much more functionality in much shorter time. Where the PO could use the time of a sprint (say two weeks) to talk to stakeholders, refine their ideas, spar with the team and prioritise the items, they now may only have a day or two before items are built. Specification, prioritisation and validation are going to be the bottleneck. And many organisations I see are not aware of this notion. They focus on AI prompting workshops and speeding up delivery, while nobody asks who is going to validate the output.

AI makes the how less uncertain, but the what stays complexA Stacey matrix with certainty about how to build on the horizontal axis and agreement about what to build on the vertical axis, with simple, complicated, complex and chaos zones fanning out from the origin. Five work items start in the complex zone. AI moves them to the left, as how gets less uncertain, but they stay at the same height because agreement about what does not change. They are still in the complex zone, each one a hypothesis to test.SIMPLECOMPLICATEDCHAOSCOMPLEXCOMPLEXHOW · certainty about technologycertainuncertainWHAT · agreement on requirementsagreeddisputed?????AI: THE HOW GETS LESS UNCERTAINTHE WHAT IS STILL COMPLEX

"But AI still delivers crap!"

One thing we need to address though. I often hear "but AI is still quite bad, I have to review every change and do a lot of rework". And that might be true in your case. But AI is only getting better, and keeps improving at a mind-boggling pace. More importantly: a lot of that rework isn’t AI getting the how wrong. It’s us being unclear about the what.1

The skill to quickly decide what to build, and then validate the underlying hypothesis, becomes even more important. And that is still a human's job.

Deciding what to build in the first place […] is the most durable part of the job 2
Laurie Voss

Determining what is worth building now is a bigger constraint than building itself according to Anthropic's design team at Designer Fund.

Scrum helped, but...

Scrum helped teams building the right thing. Product Owners learned how to manage their stakeholders, to refine their ideas and wishes and gather their feedback. This is real progress and improved the outcome of many projects. But the cadence was built around how fast they could build. There was time within a sprint to plan stakeholder meetings, refinement and updating the Product Backlog, and even then many PO's did not get to it. Specification was already a skill to improve for Scrum Teams, and validation hardly ever done. Often Scrum Teams just moved on to the next idea.

And now the feedback loop gets much tighter, leaving even less room for these crucial activities. These activities are still mainly done by humans, and for a long time to come I would wager. So we need to improve on these.

What most AI readiness scans miss

There’s no shortage of AI readiness scans. They look at tools, skills, data, governance and AI strategy: which use cases to build, what budget to use.3 What they rarely ask is how well you decide what to build, and how you find out whether it worked.

That’s a strange blind spot. Even in projects where AI is the product, RAND found the most common root cause of failure is a what problem: stakeholders misunderstand or miscommunicate which problem needs solving4. When AI does the building, that gap only gets wider.

The problem that needs to be solved is not correctly understood or miscommunicated
Ryseff, De Bruhl & Newberry (RAND 2024)

That’s the gap The What Gap measures.

The What Gap

The What Gap measures how well you can adapt to the speed of AI by looking at five dimensions:

  • Customer insight flow: how well are you able to get customer insights and adapt your plans to them?
  • Deciding what: how much are decisions based on evidence and goals?
  • Intent: how well have you defined your intent and outcomes before building?
  • Validation: how good are you at validating your ideas?
  • Capacity to change: how well are you able to change product development in response to AI?

Like in any system your weakest dimension is your bottleneck. That is where to improve first.

Get the tool

The What Gap is still under development, and not because I'm still building it (that was with the help of AI indeed the easy part), but because I am still validating it. That means gathering experts around me that bring their opinions and review the questionnaire. So that I can validate whether the questions really measure the five dimensions.

In the meantime I can give you this advice:

  • Write down the hypothesis before you start building. Concisely, what problem are you solving, for whom and how you are going to test it.
  • Limit what has not been validated. If released, but unvalidated work piles up, focus on validating before building something new
  • Make the Sprint Review about outcomes, not output. What are users now able to do and how does it help them?

Want to know where your bottleneck is? Join the waitlist. and you’ll be among the first to use it.

Also, if you think you can help me out with validating the questionnaire, please let me know! It takes about 30 mins, and I'll make sure I mention you in the credits ;).

Footnotes

  1. Naresh Jain, "Are Harnesses Worth the Cost?", https://martinfowler.com/articles/are-harnesses-worth-the-cost.html ↩

  2. Laurie Voss, "We are all Product Engineers now", https://seldo.com/posts/we-are-all-product-engineers-now/ ↩

  3. MITRE AI Maturity Model, https://www.mitre.org/news-insights/publication/mitre-ai-maturity-model-and-organizational-assessment-tool-guide ↩

  4. RAND (2024), Ryseff, De Bruhl & Newberry, "The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed", https://www.rand.org/pubs/research_reports/RRA2680-1.html ↩