I took over a data science team building advanced scoring models for content assessment. We were on the 7th iteration of our regression model, and it had gotten worse at scoring documents correctly. A new data analyst with an interest in neural networks offered to work on it on the side. He used document embeddings as well as the metadata, and our scoring effectiveness went up 30%. The expert who had been working on the problem for years didn't find that. The new guy did.
If that analyst hadn't been part of our team, we'd still be on iteration 8 of a failing regression model. Neural networks were the technique, but what mattered was that someone with a different background looked at a problem the rest of us had stopped seeing clearly. Since then I've trusted a group with mixed experience over any one expert, myself included.
Seven rounds of regression
Our scoring model had been built by a talented data scientist who deeply understood regression analysis. That depth had become a trap. Each iteration was a variation on the same approach, and seven rounds of diminishing returns showed that more effort in one direction wasn't going to get us there. We needed someone who would question whether regression was the right approach at all.
A learning community gives you somewhere to go when you're stuck besides pushing harder in the same direction. Good ideas often come from applying a solution from one domain to a problem in another, which is what our analyst did when he brought neural network thinking to a regression problem.
How we set it up
We hired for different backgrounds on purpose: experienced data scientists, but also analysts, engineers, and people from adjacent domains who thought about problems differently. That's how we got the analyst who knew neural networks. Then we made regular places for those perspectives to come out. We held monthly problem-solving sessions where anyone could bring a challenge. We set up mentoring pairs that went both directions, with experienced people sharing domain knowledge and newer people sharing techniques they'd picked up elsewhere. And we ran lightning talks where someone would present an approach from outside our usual toolkit.
The sessions worked on real problems. When the analyst proposed document embeddings, he wasn't in a brainstorming workshop. He'd seen our actual scoring data and recognized a pattern from his previous work, and the team gave him a way to act on it.
People saw that offering a different angle led to real results, so they offered more of them. The team got better at noticing when they were stuck in a single-approach rut and went looking for someone who'd see it differently.
Was it luck?
Some people would call that luck: we happened to hire someone who knew the right technique. I don't buy it. Plenty of teams have a person like that and never hear from them. We had made it normal to challenge the existing direction and to say "what if we tried something completely different?" out loud, and the analyst's idea came out of that habit.