Ben Bariach is a researcher focusing on the philosophy and governance of AI at the University of Oxford and an industry leader in frontier AI safety and governance. Harry Law leads research at the Cosmos Institute. He was previously a researcher at Google DeepMind and a postgraduate fellow at the Leverhulme Centre for the Future of Intelligence.
In a harbor before dawn, a hundred fishing boats prepare to leave in the dark of the morning. Nets are checked, and ropes loosened. As the boats reach open water, a voice breaks through the radio static: “Abort.
Whales sighted.” The majority of the boats, 90 of them, turn back. Some captains cannot fathom inflicting pain on the creatures. Through their binoculars, they look for a glimpse of the pod.
Instead, they see the remaining 10 boats continuing toward the fishing grounds. Did they miss the warning? Or did they care more about what they stand to gain?
By afternoon, the 10 boats return with their catch. The others are still tied to the quay, their holds empty. Their shared waters rewarded bad behavior.
Next time, how many will still turn back? AI agents will soon face a similar version of this predicament, only in waters with no harbourmaster, fines or reputations to keep anyone in line. As they perform tasks, negotiate, trade and compete, they will encounter other agents.
Extract — continue reading at the source.
