Marketplace listings that made things up
I was on a team that got early access to some LLM tools. We wanted to use AI to generate content for the physical properties in our marketplace, and it looked like a perfect fit: fast, consistent, data-driven. We tested a deployment quickly and watched content generation improve dramatically in speed and volume.
Then we hit a wall. The AI was producing beautifully formatted, grammatically perfect content that was completely wrong about key product specifications. It mislabeled items, hallucinated features, and invented amenities and services. We added checks and validations, and the content became as generic as our original basic system had been.
We had forgotten that selling physical products takes a person deciding which details matter to real customers and which features to play up for which audience, and then balancing accuracy against copy that sells. The AI could process any number of data points. It had no sense of the context that makes a listing useful to the person reading it.
A book, read by summary
I worked with a highly ambitious product manager on my team. She was smart and wanted to be very good at the job. When she asked for book recommendations, I suggested one of my favorites, a classic that had shaped my thinking over years of putting it to use.
She came back the next day with questions about it. I was impressed by her speed until I learned she had only read AI-generated summaries and bullet points. She could recite the basics of the framework. What she'd missed was everything you get from wrestling with the ideas yourself, like the context behind each lesson and how it connects to situations you've been in.
We spent the next several weeks reading it together, chapter by chapter. As we talked through each section, she started to see why the frameworks worked, when they didn't, and how to adapt them to her own situations. By the end, the gap between knowing the summary and understanding the book was obvious to both of us.
In both cases we handed AI the judgment. The content tool failed because we let it decide what customers care about. The summaries failed because they skipped the thinking that makes a framework useful in the first place.
The PRD
When I tried using AI to write a Product Requirements Document, the first drafts were predictably bad. It hallucinated features, made wildly optimistic growth projections, and proposed solutions that ignored our technical constraints and our market. After the listings and the book summaries, I recognized the pattern right away. I was asking AI to do my thinking again.
So I changed the order. I outlined my own ideas first, then prompted the AI to challenge my assumptions. I steered each round of revision, let it help with research and structure, and checked everything against the constraints I knew were real. The result was better than what I would have produced alone. The AI didn't supply the judgment. It pushed on mine until it got sharper.
I think judgment matters more as these tools get more powerful. Those marketplace listings looked polished and read well, and they were wrong. The better the output looks, the easier it is to forget that nobody checked what it says.