Copying the industry leader's experiments
At one global company where I worked, the team was pushing hard to improve our test-driven culture. They were obsessed with generating feature test ideas, launching experiments, and sharing what they learned across the organization. It felt like we were finally getting scientific about product development, and we had plenty of activity to show for it.
One particularly ambitious project came from our competitive intelligence team, who had been carefully tracking the testing patterns of one of our largest competitors. We watched which tests they ran and whether they productionalized the results over time. It looked like the perfect data-driven strategy: be a fast follower of the industry leader.
The logic seemed sound and the execution was careful. The results were terrible. For months the team couldn't get positive results from tests our competitor had apparently validated and implemented, and nobody could work out why. This company was an industry leader known for its testing methods, and we were following their exact approach.
Then someone found the answer in our competitor's quarterly financial filings. They had identified a critical bug in their testing software that had been showing winners as losers and losers as winners for an unknown amount of time. They were telling investors they were working to resolve it, going back through past tests, and reconciling the results, and that their financial performance would be impacted for several quarters while they fixed the problem.
We had been copying a broken system the whole time.
Blind with extra steps
By the data-driven playbook, we had done everything right. We identified patterns, followed the evidence, and executed carefully, and none of it was worth anything because we never questioned the source.
That's the trap of being data-driven. When you follow data without questioning its context, you're not being scientific. You're being blind with extra steps. We had more analytics sophistication than most teams in our industry, a dedicated competitive intelligence function, and rigorous testing processes. None of it mattered, because the whole system assumed the data we were following was sound.
Questions we should have asked in month one
A data-informed team would have asked why our competitor's results were so hard to replicate. The persistent gap between their reported wins and our actual results was worth investigating. We treated it as a problem with our own execution.
There were people on the team who felt something was off. The tests we were copying weren't working, and the explanations for why kept shifting. That kind of gut-level discomfort is information. We treated it as noise and kept following the numbers.
A competitor's testing patterns also sit inside a bigger picture: their financial performance, their public statements, how consistent their results are over time. All of that should affect how much weight you give their data. We were so focused on test-level details that we never stepped back to ask whether the whole picture made sense.
That mistake cost us millions in lost opportunities and misdirected resources. The answer had been sitting in a public financial filing, and it took us months to think of looking there.