- Roll Roll AI: Build // AI.
- Posts
- 25 AI Product Smells
Ever walked into a room and thought, "Something smells off..."
That’s most AI roadmaps.
In this issue, we’re putting names on those funky odors: 25 AI product “smells” I’ve encountered in the wild.
(Okay, just 10 today, you will see who below)
Let’s open the windows and take a tour.
Table of Contents
Best Reads of the Week
How to Keep Your Data Team From Becoming A Money Pit - I love how the solutions to most data team problems always ends up being: “we need someone doing some sort of product management type of work” 👀
5 GTM Programs I Am Implementing With My Teams - Maja got me at “Tested on AI companies”. That’s literally in the intro. She’s just that good.
When Messaging Hits But Behavior Misses - I’m big on behavioral science applied to bussiness. This one’s a smart read. Learn the BOOST framework and start having real behavioral impact.
Roblox: Meta’s Only Real Competitor - Roblox is completely under my radar, a little bit like Duolingo (sorry unhinged social media strategists) but I like to read investment-driven analysis.
Not Communicating Your Impact is Killing Your Career - I usually avoid linking paywalled articles, but at least read the free part. One of my favorite thing to do is yelling at brillant people to show to the world how great they are. Do it.
I got fooled by AI-for-science hype - here’s what it taught me - A solid read. AI-for-science can be exciting, but if you’re building in the space, don’t just drink the Kool-Aid. Reality always sobers you up.
25 AI Product Smells (and how to start fixing 10 of them)
Let’s face it, creating an AI product can be a mess.
It’s not the frameworks. It’s not even the tooling. It’s the quiet dysfunctions no one can really pinpoint.
Today, we are going to call them out.
To be precise: The 25 AP product smells I have seen in the wild and how to sniff them out before they wreck your roadmap.
Well… 10 of them.
The rest?
I’m keeping those 15 tucked away for future newsletters. Or maybe as premium content in my audits? Gotta keep the mystery alive, right?
I have grouped them in 5 categories:
Strategy
Execution
Team & Collaboration
Culture & Mindset
Process & Org Design
… SETCP. Ok, that’s not catchy but it will do.
In each, I’ll highlight:
The most severe dysfunction that kills your AI dreams the fastest
The most overlooked that also kills your AI dreams but you won’t even realize it.
Strategy
Most severe: Disconnected Roadmaps
“Our AI roadmap doesn’t talk to our product roadmap”
Classic.
Data teams run experiment in a vacuum, your product manager focus on its own roadmap and leadership funds “AI initiatives” but has no clue how they map to business needs.
This split leads to duplicated work, pointless features and uneccesary friction.
What to do?
Build a shared problem space. Frame problems in a way that data, AI and product teams can all act on, even if their toolkits and methodologies differ.
Most overlooked: No ‘Why Now?’ for AI
“We’re building with AI because… uh… the board said so?”
If you can’t articulate why this specific moment is right for an AI capability, stop. AI needs urgency, not just feasibility.
What to do?
Anchoring every use case in a strategic or timing-based trigger: competitive moves, cost shifts, data readiness, user behavior changes.
Execution
Most Severe: The Graveyard of Prototypes
“We’ve got great ML demos… none of them made it to prod.”
You build. You present. You never integrate. Nobody owns the path to production.
Result: lost credibility, lost velocity, lost users.
What to do?
Every experiment must include a path to real usage, even if it’s a manual, ugly V0.
Most overlooked: No Staffing for Maintenance
“The model’s live… but nobody’s watching it.”
AI isn’t set-and-forget. Models drift. Feedback loops break. Business context changes.
If you don’t plan for model stewardship, you’re planting landmines.
What to do?
Treat model upkeep like infrastructure. Assign ownership. Schedule audits. Build in observability… not just logging.
Team & Collaboration
Most Severe: Lost in Translation
“Your PMs, DS, and engineers aren’t even solving the same problem.”
The most advanced teams I’ve worked with all share one trait: a shared language.
Without it? Data scientists chase irrelevant goals. PMs get lost in abstraction. Engineers burn time on the wrong infra.
What to do?
Build connective tissue shared docs, hybrid roles, or rituals that force co-definition (e.g. “ML problem-shaping” sessions).
Most overlooked: Business Afraid to Challenge Modeling
“I don’t have a PhD, I’ll stay out of the way.”
Wrong move. Not everyone need to model but you don’t need a PhD in Data Science to interrogate assumptions. Otherwise, you may ship black boxes that fail silently.
What to do?
Give PMs structured ways to pressure-test: what’s the target variable? Are labels clean? What’s the worst failure mode?
Culture & Mindset
Most Severe: Tech Theater
Let’s build this AI feature for our next funding round.
When you treat AI as stagecraft, not strategy, users suffer.
It looks good. It demoes well. But it adds zero value and now your team’s stuck maintaining a gimmick.
What to do?
Don’t allow “AI” to be a feature category. Every capability must trace to a real job-to-be-done.
Most overlooked: Overfitting to Internal Stakeholder
What we need? A chatbot.
Stakeholders aren’t the enemy. But treating their wishes as gospel creates product that optimizes for opinion, not outcomes.
What to do?
Reframe internal asks as hypotheses then test with users. No sacred cows, only falsifiable ideas.
Process & Org Design
Most Severe: Waterfall Wearing an Agile Hat
Sure, it’s agile, we do a weekly standup before our 6-month spec lock.
Most AI projects (or just “projects”?) still run in waterfall with late feedback, big bets, and long delays.
They might call it Agile, but it’s “slideware-driven development.”
What to do?
Short, sharp feedback loops. Evaluate early, test downstream assumptions fast, and stop hiding behind JIRA tickets.
Most overlooked: No Reuse of Failed Experiments
That project flopped… let’s never speak of it again.
This is how you waste money twice. First by building the wrong thing. Then by throwing away what you learned.
What to do?
Build an experiment library. Document what didn’t work and why. Reuse components, lessons, and signals.
Wrapping Up: Smell What’s Real
Most AI product issues aren’t technical.
They’re coordination breakdowns, misaligned incentives, or shallow framing.
These 10 smells are just the starting point.
If you can name them, you can fix them. If you ignore them, they’ll quietly kill your ROI.
Outro section
Smell something familiar in your own org?
You’re not alone. Most AI dysfunctions aren’t code problems: they’re coordination, culture, and clarity problems.
These first 10 are just the ones I’m willing to reveal.
The remaining 15? Still fermenting.
Catch them in my audits or in a future breakdown… who knows?
More to explore:
Share your thoughts:
How did you like today’s newsletter?
You can share your thoughts at [email protected] or share the newsletter using this link.
Reply