From $5M to $100M on bad data
I took over a product that had been launched as a side project with no validation or rules. The data was duplicated, conflicting, and incomplete. My job was to clean up this mess and scale from $5M to $100M in a few months.
The same problem was showing up all over the organization, so we treated it as an organization problem. We could have built a validation service for this one product. We built one that worked across all data ingest points. It became the foundation for a whole new content management service, it helped us clean up our entire data catalog, and it left us ready for a large migration project that came later.
Resilient vs. anti-fragile
A resilient team would have cleaned up the data and moved on. They would have fixed the duplicates, patched the gaps, and gotten back to the roadmap. That's what most teams do, and it's a fine outcome. We built a validation service that kept the problem from coming back in any product in our catalog. Resilience gets you back to where you were before the hit. Anti-fragility is when the hit leaves you better off.
What made it work
The fix was bigger than the problem in front of us. The validation service came out of a crisis in one product, but we designed it for every data ingest point in the organization. When the large migration project came along months later, we were already prepared, because the crisis had pushed us to build the right infrastructure early.
We also used the crisis to look around. The data integrity problems in our product existed across the organization, and nobody had seen the full picture because each team was focused on its own silo. What we found led to cleaning up our entire data catalog. I don't think we'd have gathered that information any other way.
And the team changed its mind about what the job was. Early on, the team's instinct was to clean up the mess and get back to the growth roadmap as fast as possible. Things turned when we stopped treating the crisis as an interruption and started treating it as a design problem. That's how a data nightmare became one of our most valuable internal services.
Nobody planned this
We didn't plan to build a company-wide validation service. We didn't have "create foundational data infrastructure" on any roadmap. The crisis handed us a problem so bad that the only real solution was worth more than what we'd lost.
You can't avoid disruption. Markets shift, competitors surprise you, internal priorities change overnight. What you can decide is whether your team treats the next mess as something to get through or something to build on. If that product's data had been clean when I inherited it, we'd probably still be patching each ingest point one at a time.