Training an agent that outgrows you is a false proposition

What is it like to train an agent? In the beginning it feels familiar: we provide data, set goals, give feedback, and correct mistakes. But if the aim is an agent that becomes ever more powerful, and eventually more intelligent than we are, the word “training” starts to lose its meaning.

The first analogy is the piano teacher. A good teacher can take a student a long way, but only up to what the teacher can hear. Once the student plays beyond that, the teacher can no longer tell a brilliant interpretation from a mistake. The lessons turn into encouragement, and then into listening.

A second analogy is the child and the parent. Raising a child is training of a sort, yet we do not expect to remain the authority forever. We hope some values stick, but a grown child who surpasses us makes their own judgments. Nobody calls that a failure of parenting; it is simply the point where parenting stops being the leading force.

A third analogy is the chess engine. Once engines passed the best players, humans stopped teaching them chess. The engines learn by self-play, and the humans study the engines. The direction of instruction flipped.

Training needs three things: a goal we can state, feedback we can judge, and a trainer who knows more than the trainee (agruable, but at least if the trainer is more in charge than the trainee). For a stronger agent, the second and third quietly fail. We cannot evaluate what we cannot understand, and we cannot correct what we cannot see is wrong. What remains of our role is closer to choosing what to ask, or deciding whether to trust, than to training.

These analogies are, again, incomplete. A student is not a system we build and deploy, and a child is not something we can copy a million times. Perhaps the honest description is that we are no longer training the agent past some point; we are merely the starting conditions of something that trains itself. What that means for us is a harder question than how to train it.