From the observatory to the boardroom: Part 2

Building a production app with AI in 7 days taught me more about AI risk than any report I’ve read. It also reminded me of something I learned much earlier in my career.

I started in software development before I moved into strategy and leadership. I’ve spent a lot of time in the trenches of legacy modernisation - both as a developer, and as a leader. Untangling systems built too fast, by well-intentioned teams, under pressure, without enough thought given to what would come next - usually due to time pressures to get systems up and running. Integrations held together with logic nobody had documented. Data models that made sense in the moment but became liabilities over time.

We’re about to do that again. At scale. And faster than ever before.

Here’s what I took away from the experience of building StellarLog.

You still need to know what you’re doing

AI accelerates execution. It does not replace judgement. When I asked Claude to scaffold a component or suggest a data model, I could evaluate the output because I understood the underlying architecture. I knew when a suggestion was elegant and when it was going to create problems down the line. Without that foundation, you wouldn’t know what questions to ask, or when the answer was wrong (and that did happen a few times).

This maps directly to a statistic I use in my keynotes: 57% of employees use AI without evaluating the accuracy of the output. That’s not an AI problem. That’s a capability and governance problem.  In a code base, in a data model, in a system that will need to scale - that’s a legacy problem in the making.

Speed creates new categories of risk

When one person can build what previously required a team, the guardrails that existed in a team-based workflow disappear. Code review. Peer testing. Architecture sign off. These weren’t bureaucratic overheads. They were the mechanisms that caught the decisions that would hurt you later.

If you’re moving fast without replacing those structures or processes, you’re making a withdrawal from a technical debt account you haven’t opened yet.

Testing was where the real time went

Generating code with AI is fast. Testing it is not, and that gap is more significant than most leaders realise. The majority of my iteration cycles were spent on code that worked perfectly in isolation but broke the moment it was introduced to the broader system. Validations that appeared sound allowed values they should have rejected. Each fix surfaced a new edge case. Each edge case required a new iteration. What this revealed is that testing is irreducibly human work.

AI can generate a solution in seconds, but it cannot anticipate every context in which that solution will be asked to perform. It cannot know what it doesn't know about your system. The discipline of methodical testing, of asking "what should this not do" as rigorously as "what should this do," is a skill that sits entirely with the human in the process. In a team environment, that skill is distributed. When you are building alone with AI, it rests entirely on you.

This is the capability gap that organisations are not talking about enough. It is not enough to train your people to use AI tools. You need to invest in the foundational engineering and analytical discipline that allows them to interrogate what AI produces, not just accept it.

The governance gap is real, and it compounds

65% of organisations use AI. Only 30% have an AI policy. I live the tensions between these two numbers during this build. Every decision about data handling, API security, and user privacy was mine alone to make. I had the background to make those decisions. Most people building with AI right now do not.

In an enterprise context, ungoverned AI-assisted development doesn't just create security exposure today. It creates the kind of structural technical debt that quietly accumulates until it becomes a board-level problem: systems too fragile to scale, too complex to modify, and too deeply embedded to replace without significant cost and disruption.

Where do we go from here?

Surprisingly, the answer isn’t to slow down. The organisations that stopped to perfect their legacy systems were left behind. The answer is to move forward with intention. Build fast, but build with discipline that protects you later. Establish the architectural standards before the tools proliferate. Create the review structures and processes that teams used to provide. Invest in the capability to evaluate AI output, not just to generate it.

Readiness in motion, not paralysis in planning.

Part 3 of this series will cover how to apply these learnings as a senior leader or executive.

I don't think most organisations are asking the wrong questions about AI. I think they're not yet asking enough of them. If you want to explore what those questions are, I'd love the conversation.