The Most Valuable AI Training Data Isn’t on the Internet
We’ve spent the last few years feeding AI just about everything we can find. Books. Websites. Research papers. Code. Videos. Documents. Probably a few billion Reddit arguments that nobody should have started in the first place.
And the results have been remarkable.
Today’s AI can explain quantum physics, write software, analyze financial statements, and pass exams that would give most of us a headache.
But there’s something I’ve been thinking about.
What happens when AI has read almost everything worth reading?
Where does the next big improvement come from?
I don’t think the answer is simply more information.
I think it’s experience.
Consider two people.
One just graduated from business school. Brilliant student. Perfect grades. Knows every management framework ever invented.
The other has spent 25 years running businesses, dealing with difficult customers, managing employees, and making decisions when there wasn’t an obvious right answer.
Give them the same complicated business problem.
Who would you rather have in the room?
I’d probably want both.
But there’s a reason the second person’s opinion carries weight.
They’ve seen what happens after the decision gets made.
They’ve watched good ideas fail. They’ve seen terrible ideas somehow succeed. They’ve learned that what works beautifully in one company can be a disaster in another.
Most importantly, they’ve developed judgment.
And judgment is a difficult thing to learn from a textbook.
I think AI is beginning to encounter a similar challenge.
We’ve done an extraordinary job teaching models what the world knows.
Now we’re trying to teach them how experienced people actually work.
That’s where companies like Harvey and AfterQuery become interesting.
Harvey has been developing AI specifically for complex legal work, including systems that use expert-designed evaluations and post-training to improve performance on realistic assignments.
AfterQuery works with experienced professionals to help AI models learn from real-world tasks, decisions, and expert feedback.
Both point toward something I believe will become increasingly important.
The next generation of AI may depend on knowledge that was never published.
The senior engineer who knows a system is about to fail because she’s seen the same warning signs for 20 years.
The experienced physician who recognizes that a technically correct recommendation doesn’t make sense for a particular patient.
The sales executive who can tell within ten minutes whether a customer is genuinely interested or just being polite.
That knowledge exists.
It’s incredibly valuable.
But much of it has never been written down.
We’ve spent decades trying to capture it through documentation, training manuals, and knowledge management systems.
Maybe we’ve been approaching the problem from the wrong direction.
Instead of asking experienced people to document everything they know, what if AI could learn from how they actually solve problems?
Watch the decisions they make.
Understand the alternatives they reject.
Learn from their corrections.
See what happens after their recommendations are implemented.
Of course, that doesn’t happen automatically. Companies need ways to capture those experiences, evaluate their quality, and use them to improve AI systems.
But imagine the possibilities.
For the first time, we may be able to make expertise that took someone 25 years to develop available to people who haven’t had those 25 years.
That could change how we train employees, deliver professional services, and build businesses.
We’ve spent years worrying about whether AI has enough data.
I’m starting to think the more interesting question is whether we’re giving it enough experience.
Because the next big breakthrough might not come from another trillion words.
It might come from understanding why a 30-year veteran makes a decision that nobody else in the room would have considered.
Looking to optimize AI in your organization? Contact us for a consultation on how Consulting Solutions can help.