I tried to teach AI what experience had taught me.

© Rich Baker

What trying to teach AI how I make professional judgements taught me about experience, expertise, and how we learn to question what looks right.

For perhaps fifteen years, I have been using different versions of the same approach to communications planning.

It has changed over time. Bits have been added, removed, simplified and occasionally rediscovered after I had forgotten why they were there in the first place. Different jobs changed it too. Large change programmes taught me things that leadership communications didn’t. Operational environments taught me other things again.

But the basic idea has survived: start with what you are trying to achieve, work backwards, and only then decide what communication might help.

None of this is particularly revolutionary. I suspect plenty of experienced communicators work in roughly the same way, whether or not they have written it down.

I had written some of mine down. There were templates, questions and planning tools. But after doing the work for long enough, a lot of it wasn’t really written anywhere.

It was just how I thought.

Someone would ask for a video or newsletter and something would make me wonder whether that was actually needed. A leader would want to “raise awareness” and I would want to know what they hoped people would do with this new awareness. A plan would have fifteen channels in it and I would wonder whether three would do.

Of course, sometimes the answer really was just to send the email.

You accumulate these things over a career. Some come from training. Most probably come from getting things wrong, watching other people get things right, working with brilliant people, surviving difficult situations and gradually noticing patterns.

Eventually you stop noticing that you’re doing it.

Then I tried to explain it to a machine.

It turned out to be surprisingly difficult.

I had started using AI seriously in my communications work a few years earlier, experimenting with it in multilingual policy communications, content production and other parts of the job. Some of the improvements were dramatic, and were scaled globally. Work that had taken days could take minutes.

But I found myself less interested in the speed than I expected. I wanted to know whether the communication was better. Did people understand something they hadn’t understood before? Could they do something more easily? Had we solved the problem?

That probably wasn’t an AI insight at all. It was the same old outcomes thinking following me into a new technology.

As the technology improved, though, another possibility became harder to ignore. If AI could produce the communication, perhaps it could also help with some of the thinking that came before it.

So I began taking the planning approach I had been developing for years and making it more explicit. Not just a template with boxes to complete, but the reasoning underneath the boxes.

How do you tell whether something is actually a communications problem? How do you distinguish the outcome of an organisational change from the outcome communication can realistically influence? When should you challenge the brief? And when should you stop being clever and just write the thing?

I wanted to see how much of that an AI could learn.

And this is where things became interesting, because before you can teach a machine how you make a judgement, you have to explain the judgement to yourself.

Some things were relatively easy. Start with the outcome. Separate the business objective from the communications objective. Don’t mistake activity for impact. Be proportionate.

Fine.

Then there were the things I knew how to do, but found much harder to describe.

I might look at a brief and think: that’s not the real problem.

Why? Well. It depends.

On the organisation. The people involved. What has happened before. What isn’t being said. The consequences if we get it wrong. Whether the person asking for the communication actually has the power to solve the underlying problem. Sometimes it is simply a feeling that something doesn’t quite add up.

“Experience” is a wonderfully convenient word for all of this. We use it to describe the accumulated ability to recognise something without necessarily being able to explain exactly what we recognised.

The machine didn’t object to that ambiguity. It simply made the ambiguity visible.

If context matters, which context? If something is disproportionate, disproportionate to what? If an experienced communicator would notice something, what exactly are they noticing?

Those questions sent me back through things I had been doing instinctively for years. Some turned out to be principles. Some were heuristics. Some were probably preferences. And some may just have been habits that had acquired the dignity of experience because I had been doing them for long enough.

That was mildly uncomfortable.

More surprising was how much a general-purpose AI could already do. Not perfectly, and not reliably enough to hand over responsibility, but well enough to make the boundary between AI capability and human experience less obvious than I had expected.

That matters because a lot of the conversation about AI and professional work is organised around a reassuring distinction: AI does the routine work; humans provide the judgement.

I have used versions of that distinction myself. In Don’t Let the Robots Write Alone, I argued that the interesting question isn’t whether AI can produce competent communication. It increasingly can. The question is what humans still need to contribute when producing the words is no longer the difficult part.

I still think that. I’m just less satisfied now with the word judgement as the answer.

Because judgement isn’t a mysterious substance that experienced people possess. It has ingredients: knowledge, pattern recognition, context, memory, confidence, doubt, the ability to frame a problem and the ability to notice that the frame might be wrong. It is knowing when a rule applies and when the situation in front of you is the exception. And, perhaps, knowing when you don’t know.

This is where my experiments began to overlap with another idea I had been writing about: metacognition.

Thinking about our own thinking sounds faintly philosophical until a machine starts participating in the thinking. Then it becomes rather practical.

Lev Tankelevitch and colleagues at Microsoft describe working effectively with generative AI as placing new metacognitive demands on people: understanding our goals, deciding what to delegate, assessing what comes back and knowing when to change approach.

Another Microsoft study surveyed 319 knowledge workers about 936 real examples of using generative AI. Higher confidence in AI was associated with less critical thinking, while greater confidence in your own ability was associated with more. The nature of critical thinking shifted too, towards verifying information, integrating responses and overseeing the task.

Perhaps AI doesn’t simply remove thinking from work. Perhaps it moves where the thinking has to happen.

The first draft becomes easier, while the judgement about whether it is any good becomes more important. Information becomes easier to obtain, which places more value on knowing what matters.

A plausible answer becomes cheap. Knowing whether it is the right answer remains rather expensive.

There is another complication: where does the judgement come from in the first place?

Much of mine was built by doing the work AI can now help people avoid: writing the bad first draft, sitting through the meeting, misreading the audience, building the overcomplicated plan and sending something that didn’t work.

Quite a lot of experience, when you think about it, is discovering afterwards that three channels would indeed have done.

There is emerging evidence that AI can help people perform unfamiliar tasks without somehow converting them into experts. Recent Harvard Business School research makes that distinction rather neatly: generative AI can extend what people are able to do, but expertise still matters to the quality of what they do with it.

That raises a question I find much more interesting than whether AI will replace particular communications tasks:

What happens to professional judgement if people no longer have to do some of the work through which that judgement used to develop?

I don’t know.

And increasingly, I think that is where the useful bit starts.

I began this experiment because I wanted to see whether some of the reasoning I had accumulated over a career could be made explicit enough for AI to use. I thought I was building a better way to think about communications work. Instead, I found myself pulling apart the idea of experience.

What do experienced people actually notice? Which parts of judgement can be articulated, and which can be reproduced? What happens when AI gives an inexperienced person access to behaviours that look remarkably like expertise? And if some professional judgement is built through practice, what happens when the practice changes?

Those questions have now become the basis of the independent research I’m doing into professional judgement and generative AI, starting with communications.

I’m particularly interested in the things that are difficult to see from the finished work: what people noticed, what they questioned, what they nearly did, what they decided not to do, and where experience actually changed the decision.

There will be more on that as the research develops.

For now, though, I keep coming back to the odd reversal at the heart of this.

I began by trying to teach AI what experience had taught me.

It has ended up making me much more curious about what experience had actually taught me in the first place.