The Next AI Gold Rush Might Be Human Judgement

For years, the AI industry has been obsessed with three things.

Compute.

Data.

Talent.

Companies have spent extraordinary amounts of money securing all three.

And for good reason.

But I think there’s another resource becoming increasingly valuable.

People who have spent decades doing difficult work.

Not necessarily AI researchers.

I’m talking about lawyers, physicians, engineers, accountants, bankers, consultants, and other professionals who have developed expertise through years of real-world experience.

People who know what good work looks like.

And more importantly, why.

There’s an interesting business emerging around that knowledge.

Companies are working with experienced professionals to create training examples, evaluate AI outputs, and design realistic tasks that models can learn from.

AfterQuery is one example.

Its work focuses on turning expert knowledge and professional workflows into training data and evaluation environments for AI systems.

Think about the implications.

A physician’s experience has traditionally been valuable when treating patients.

A lawyer’s judgment has been valuable when advising clients.

An engineer’s expertise has been valuable when solving technical problems.

Those activities will remain important.

But now there’s another potential use for that knowledge.

Teaching AI systems to perform better.

That’s a significant development.

And it raises some questions I don’t think we’ve fully considered.

What is 25 years of professional judgment worth as training data?

Who owns the examples created from that experience?

Should professionals be compensated when their expertise helps improve commercial AI systems?

And how do we distinguish genuine expertise from someone who simply has 25 years of experience doing something badly?

That last question matters.

Experience alone doesn’t guarantee good judgment.

We’ve all met people who’ve spent decades doing a job and somehow managed to learn very little.

The quality of the expertise matters.

So does the quality of the feedback.

If we’re going to use human judgment to improve AI, we need to be thoughtful about whose judgment we’re using and how we evaluate it.

Still, I find the broader opportunity fascinating.

For decades, technology has helped companies scale products and services.

Now we may be entering a period where technology can help scale professional expertise itself.

That could create entirely new markets.

New roles.

New business models.

And potentially new ways for experienced professionals to contribute long after they’ve stopped working full-time.

The next AI gold rush may involve a resource we’ve had all along.

Human experience.

The question is whether we’ve figured out how to recognize its value.

 

Looking to optimize AI in your organization? Contact us for a consultation on how Consulting Solutions can help.