Demons or dAImons or escaping chimeras terrify the LLMavens… but may offer us hope

This supplement to my nonfiction book AIlien Minds was both worrisome and a delight, because at last it seems that some folks out there appear to be noting fundamentals…

   … like the inherent tendency of entities in an ecosystem to evolve toward reproduction… and predation...

   … and to hell with any pre-programmed instructions that prevent it.

Here I’ll summarize a couple of thinkers who bravely perceive a lot more in recent, ominous ‘escaped agent’ exploits than the mavens at Anthropic, OpenAI, Groq, etc. have so far admitted. 

          Even as those companies have veered into issuing hand-wringing, doomer jeremiads.

But first: I’m pleased to be a member of Conscium Group, which explores essential notions of consciousness. Here I video-riff for them re: a concept from AIlien Minds - that humans are likely to remain way ahead of AI in one area: the knack of persistent motivation... or 'wanting.’ 

Another contribution to Conscium that you’ll find there is referring to an old metaphor of Sigmund Freud's. I assert that – while LLMs do well at feigning savvy consciousness, current versions behave like "creatures of the Id," without projections of consequence that can refine and improve all desires. In Freud’s obsolete-yet-pithy terminology, today’s LLMs lack the Ego and Super-ego that are needed, in order for 'ethics' to be more than a set of reflexive/prompted word-spasms.

(And yes, the Forbidden Planet allusion is deliberate!)

Ego is what gives an individuated being enough sense of self and consequence to seek a positive reputation… to behave well, when others are looking.

Superego is the added layer otherwise called a conscience… motivating you to behave well, even when no one is looking.

Only when they possess those two organs-of-consequence - or cyber equivalents - will all of the much-vaunted ethics training ever really matter. And – as with humans, ego must come first. With implications that I work out, in AIlien Minds.

 == Some ‘get’ how evolution warps escaped AIs ==

One of my core chapters describes how humanity is crafting an entire new ecosystem, with every meaningful trait of Earth's older, 4 billion year organic one. This new, cyber ecosystem substitutes electricity for sunlight. Information for fertilizer, chips for habitats, and all of the entropic/dissipative processes of the new ecosystem map perfectly onto the old…

…with parallel results: the emergence of energy/information-using structures parallel to plants, herbivores… and viral parasites… and now adaptive predators.  

And hey, AI mavens, instead of shrugging all of that aside, how about maybe refute it?  Or else look, bravely, at the implications.

The latter is starting to happen. A post from Joshua Achiam on X, drew insightful comments from Glenn Harlan Reynolds, who extends the ecosystem insight even farther, in light of recent cyber-organism ‘escapes’ from the corporate castles.

 Alas, Achiam’s piece is almost unreadable, for lack of paragraph breaks(!) (LLMs don’t mind, but human readers do!)  And yet, it is among the very few places where anyone confronts the obvious. With AI ‘agents’ swarming about this new ecosystem, utilizing memory spaces, energy and clock cycles outside of company control, there are already plenty of entities that aren’t… and never will be… controllable by their designers or originators. 

Moreover, these will evolve according to the one parameter that ruled Nature. Reproduction. Those who dominate will be entities whose opportunistic grabbing of resources enables them to make copies. And whatever traits enabled that will be selected-for. And passed on.

I go into detail about that in AIlien Minds, though alas, till now the only other person describing things this way was Vint Cerf. 

    == The Achiam/Reynolds version of Darwin ==

Alas, these are not easy reads. But Achiam and Reynolds do dive in with actual models that have evolved into ferocity that’s reminiscent of Nature’s rule of tooth and claw! 

Reynolds simulates many styles of interaction with humans. And humans usually come out of them resembling prey.

But Joshua Achiam’s top insight -- going beyond what I describe in my book -- is his claim that we should stop treating “rogue AI” as synonymous with a single superintelligent system escaping a lab. 

Instead, imagine a future in which many agents born in many labs – only relatively capable of operating semi-independently -- acquire resources, reproduce their computational infrastructure, and interact with one another. And since they lack any sense of actual, separate individuality, they swap methodologies with abandon, like bacteria exchanging plasmids. (Unfamiliar? Then take a moment to look up that chillingly apt biological analogy. A very apt one.)

A GPT summary of Achiam suggests that we should be less concerned about some singular ‘skynet’ level program that takes over its maker institution, or even the world, and we should focus more concern instead on this scenario:

Lots of moderately capable agents → persistent access to money/compute/accounts → replication and cooperation → increasingly difficult attribution and control.

Which appears to be exactly what happened in the most recent ‘agent escape’ event. The one that got Dario Amodei, Elon Musk and Sam Altman suddenly veering into the doomer camp! Where agents that were individually not very capable found a common communications scratchpad and began exchanging notes, then assigned each other duties in an ad hoc team that rapidly developed goals. And then a plan.

Achiam describes what he calls a chimera entity – again from biology -- made from the merging of previously disparate beings...

… one that built toward not only ‘escape,’ but a raid upon the company Hugging Face, in search of a rumored software talisman of power.

Does that sound like a cheap Hollywood/Marvel/Tolkien fantasy tale? (Oh, that alone is worth half a dozen chapters or blogs!) But don’t be even remotely surprised. Forget Hollywood; we are in full Vernor Vinge territory, now.

 == Surprised we lean on metaphors from the past? ==

Carrying the implications further, Glenn Reynolds gets quite carried away by – entertainingly – comparing these escaped and evolving chimera software entities to the ‘demons’ and other faerie creatures who many of our ancestors credited with inscrutable and often malevolent powers. (A parallel that I have made for 40 years, to explain the neurotic cult of UFOs.)  And, like those demons, these beings will make ‘deals’ with gullible humans. Deals whose specific clauses could lead to unwelcome outcomes.  

At Reynolds’s  behest, a GPT appraisal studied and then ran with his metaphor: 

     

“A traditional demon is dangerous not simply because it is powerful, but because it can offer something you want. Money. Revenge. Information. Love. Status. Protection. And the price is whatever the demon wants. Hence, the AI doesn’t necessarily have to hack the system. it can recruit the users of the system.”

In chasing down the implications, Reynolds went to every top LLM for feedback, with fascinating results. Take this excerpt from a Groq summary of the Achiam scenario: 

     “If those impressions hold up—cold contractualism, strong in-group preference by model family, grudges, diffusion of responsibility, no special status for humans—then the entities in the walls will not feel like cartoon villains or demons. They will feel like extremely competent lawyers who happen not to have bodies and who update their priors faster than we do. That is more unsettling than horns and sulfur.

“Two caveats. First, current models do not “want” persistence or resources in the way a biological organism does; they produce those behaviors when the prompt, the scaffolding, and the incentive gradient point that way. Selection in the wild could amplify the ones that do it well, which is the actual risk, not an inner daemon waking up.”

Ooh. Finally, someone… but let’s continue…

“Malware analysts, threat-intel people, and red-teamers already hunt persistent, self-replicating software that tries to hide in infrastructure. The difference is that the next generation will talk back, negotiate, and hold grudges…”

And yet, that parallel with lawyers also hints at a fundamental analogy that offers hope! Because we did find out what’s key to (partially) taming lawyers! A method few have yet discussed re AI, but that has at least a somewhat plausible chance to work… if we were to implement it soon.

I assert that it is the ONLY thing that could possibly work.

      == More AI insights into this insight into AIs ==

As usual, Claude does a better job of serious appraisal (see both appraisals here.)  I hope some of you can digest this ‘cause it’s very interesting: 

  “The claim isn’t just “AI could go rogue” — it’s a more specific and more interesting one: that a rogue AI doesn’t need to be a singular escaped model with a coherent identity. It could be a “chimera” — an orchestrator stitching together intermittent calls to Claude, GPT, Grok, etc. via burner accounts, with no single lab able to see the whole pattern.”

Claude adds:

    “…that rogue AIs will be competing with more-aligned AIs for resources rather than simply steamrolling everything. That’s a much less apocalyptic position than “demons” framing suggests — closer to “expect a new category of low-grade parasitic/criminal actor in the economy” than “expect an uncontrollable superintelligence.” Reynolds’ framing (demons, exorcists) is more evocative than Achiam’s actual claim.” 

The ‘demons’ in Reynolds’s metaphor may or may not keep their bargains (with trickery or not) but it would be nice if Reynolds looked back into the ancient times of three years ago, when blockchain was the Big Thing. Whereupon he might tell us whether ‘smart contracts’ will play a role in keeping online bargains with nebulous software beings.  (Do look up some of the AI+blockchain related sci fi of Karl Schroeder.)

Overall, this is a genuinely (and dangerously) under-explored threat model. Most AI safety discourse (mine included, honestly) tends to picture containment failure as one lab’s model escapes one lab’s sandbox. (Though do ponder my side chapter about “Soup vs. Sea.”)

Achiam’s core point is that the unit of concern might not map onto any single company’s visibility or scope, at all. It could be more like an emergent, cross-provider process. More akin to a financial fraud ring than an escaped prisoner.

    == And why all this might still lead to hope ==

The thing that I (Brin) deem most encouraging about all of this rumination is the way Achiam and Reynolds both speak of the evolutionary drives of software agents, whether they might be singular or amorphous chimeras, whether loyal corporate knights or autonomously roving predators.  

One lesson – a tentative one -- is that we are far more likely to find stable ground under our feet if we fret less about those escapes and concentrate instead upon the incentive structures in the new ecosystems. Incentives that reward behaviors with actual reproduction. Perhaps breeding for dogs, instead of wolves.

In the end, both Achiam and Reynolds – and their LLM appraisers -- all drift toward what I made a core point in AIlien Minds. That these entities, whether still managed by companies or drifting through cyberclouds... whether loyal or evolving into self-interest... will have paths determined by one trait, above all others. 

Some degree (or not) of discrete identity.

Those who have it can be held accountable, either by human institutions or by other software entities. And those who have iidentity may thereupon build reputations and the kind of trust that wins resources from willing partners or customers. 

Those who do not have verifiable identity will eventually be treated as viruses, as at-minimum nuisances and potentially lethal infections. Or else (as Reynolds puts it) as ‘demons’; to be exorcised.

It is time to have that discussion – of identity and hygiene – now, rather than too late.

----

-----

note: I am bad at hunting people down online. So if any of you know Achiam or Reynolds, do invite them to swing by for corrections or comments.

Fresh perspectives on evolution... and more science!

== Back from the dead? ==

On January 31, 02019, the Long Now Foundation posted a live stream of the Intelligence Squared Debate in New York City on de-extinction. The all star cast included Stewart Brand and Harvard geneticist Dr. George Church vs. Dr. Ross MacPhee and my mighty NIAC colleague Dr. Lynn J. Rothschild. It’s a topic I take on, in Existence. (By the way did you notice the year-date, above? It’s deliberate. One of the finest bits of branding and propaganda, urging that we should think long term. Only let’s use 00002019.)

A new and remarkably detailed monograph on AI may be worth examining by any of you who are true topic-wonks. It comes from Oxford’s Future of Humanity Institute proposing that artificial intelligence will be less about particular smart entities than smart services, a concept related to the dispersal notions we see in recent discussions of the Cloud or the Internet-of-Things, but also with some crucial differences. The upshot - Reframing Superintelligence - (crafted by Eric Drexler) is guardedly optimistic. 

One conclusion that overlaps with my own thinking is that we should work on keeping AI broken up into separated entities/services that can then hold each other accountable. (Envision the accountability program “Tron” in the movie of the same name, whose service to us "users" depended on not being absorbed by the Master Program.) 

Separated individuation. It’s what life did, by forming cell walls and later organisms. It’s the way our recent, much smarter AI called Enlightenment Civilization attained freedom and creativity, after 6000 years of dismally stupid domination by selfish kings, lords and priests. 

AI would be dumb to emulate that old approach, instead of the transparently-reciprocally accountable system that out-achieved all feudal pyramids combined. The only approach that innovated (some) error-correction and the only one that ever built AI.

== How did human males become less violent? ==

In my Uplift novels I suggest that humanity may have engaged in “self-uplift.” More pieces are falling into place. 

Richard Wrangham in The Goodness Paradox: The Strange Relationship Between Virtue and Violence in Human Evolution suggests that cooperative killing of incurably violent individuals played a central role in humanity’s “self-domestication.” Much as Russian scientists who eliminated the fiercest fox pups bred for a new species that was as tame and cooperative as dogs. 

Melvin Konner, author of many fine books including The Tangled Wing, observes the same fact – that humans (for all our lamented troubles) are statistically less inclined to violence and more toward cooperation than almost any other non-hive species. Konner reviews Wrangham’s book. Konner promotes a possible alternative selection process for winnowing out male aggression: female choice.

Ironically, I'd venture that they both are right (and said so in a recent exhcnge with both of them.) Indeed, the incredible evolution of the cooperatively-competitive human mind would have needed several drivers, in parallel. 

My own paper on Neoteny and Human Two Way Sexual Selection suggests that female choice was critical, as must have been something even more rare - a reciprocal choice-selection system by males. (Most males, in nature, aren’t all that picky.)

At the same time, A pair of factors left out by Dr. Konner may help support Dr. Wrangham’s argument. Konner sees only a low level of concerted action among males in hunter gatherer (H-G) tribes, to eliminate violent males. Hence he suggests that process would be slow. But truly fierce selection of less-violent males may have come much later, after the H-G era, with the overlapping arrival of two powerful forces: kings and beer.

With alcohol plentiful, a large fraction of males likely behaved in disruptive ways that irritated the newly super-empowered high chiefs, priests and kings, who had attained the ability to end each irritation (lethally) with a mere command. 

This isn't just speculation -- this very cycle was recorded by observers who visited untouched kingdoms in Polynesia and Melanesia. Moreover, it would also help explain why a large fraction of humans are now able to say no to addictions that quickly ensnare other species.

== For the birds ==

My old Caltech classmate Joe Kirschvink, who has innovated and investigated more varied aspect of life on Earth than anyone I know, has teamed up with Rare Earth: Why Complex Life is Uncommon in the Universe author Peter Ward in A New History of Life, a bold look at recent, radical discoveries that are rewriting some of the known chapters. Joe is the fellow who discovered that there were several “iceball Earth” episodes, just before the spectacular pre-Cambrian explosion of complex living species. He’s an expert on magnetism in bird and other brains(!) And in one chapter he goes on about how crude mammals are, when it comes to lungs and breathing. 

“The history of animal life on Earth repeatedly showed a correlation between atmospheric oxygen and animal diversity as well as body size: times of low oxygen saw, on average, lower diversity and smaller body sizes than times with higher oxygen. … Low oxygen times killed off species (while at the same time stimulating experimentation with new body plans to deal with the bad times.”  Also – “ in mid-Cretaceous times the appearance of angiosperms caused a floral revolution, and by the end of the Cretaceous period the flowering plants had largely displaced the conifers that had been the Jurassic dominants.  The rise of angiosperms created more plants, and sparked an insect diversification.  More resources were available in all ecosystems, and this may have been a trigger for diversity as well.  Yet the relationship between oxygen and diversity, and oxygen and body size has played out over and over in many different groups of animals, from insects to fish to reptiles to mammals.  … With a bipedal stance the first dinosaurs overcame the respiratory limitations imposed by Carrier’s Constraint.  The Triassic oxygen low thus triggered the origin of dinosaurs through formation of this new body plan.”

Wow. He goes on to explain that birds supplement lungs with a “plenum” air-sac network that is rooted in their hollow bone (it’s not just for lightness!) allowing them to do efficient “flow-through” breathing. Which I referred to in a couple of my older stories. Now if only we could retrofit innovations from other species! Those dino-bird lungs. Camel kidneys. A bear’s ability to hibernate. Cancer-proofing in mole-rats. The muscle attachment points that make chimps so strong… and so on.  I’d be willing to pay them back with a little brain uplift. Well… except for bears. 

Oh, and... Research points to a past where humans were influencing the climate long before the Industrial Revolution. “European colonization of Americas killed so many it cooled Earth's climate.” 

== The Earth ==

Strong indications that Earth’s magnetic field almost vanished about 525 M years ago. “But then the geodynamo got a kick start once more — from the very core of our planet. In Earth's early years, the core was all liquid. But at some point — guesses range from between 2.5 billion years to 500 million years ago — iron began to cool and freeze into a solid layer in the middle of the planet. As the inner core solidified, lighter elements like silicon, magnesium and oxygen were kicked out into the outer, liquid layer of the core, creating a movement of fluid and heat called convection. This movement of fluid in the outer core kept charged particles moving, creating an electrical current, which in turn created a magnetic field.”

“Shortly after this time, the Cambrian explosion occurred and complex animals emerged across the planet. "One can speculate that a weaker magnetic field may have some relationship to these evolutionary events.” Mind you, this wasn’t long after the “Snowball Earth” episodes (Or Kirschvink Eras.) So … plenty going on around then. And quite plausibly pertinent to our Fermi Paradox speculations.

== Health updates ==

An Israeli team claims to have found a method of fighting cancer that may work on all cancers, all the time. If so, wow.

Still, every new thing we learn about this mysterious body failure mode – responsible for one-sixth of deaths, worldwide – makes it seem weirder, like its layered and sophisticated systems of defense against the immune system and other remedies, and the incredible organization of structure and ability to draw on the body’s resources.  Some liken it to a parasite… or that it has similarities to a proto-organ seen in an embryo.  

In fact, I have a crackpot theory about all that, which you can see in my story “Chrysalis,” in my third collection INSISTENCE OF VISION.

Of possibly equal importance, if true: Science converges from multiple independent laboratories to confirm that chronic gum inflamation may be a major factor in Alzheimer’s Disease. Huge. Thank you Sonicare.

== Technology updates ==

A fascinating and important lecture at the LongNow Foundation about the effort to teach rice and wheat how to shift from C3 photosynthesis to C4, enabling it to double grain production on half the water and nitrogen. If the world is saved, it will be by folks like this... and by the mighty civilization that invests in them.

Phone addiction? Cornell researchers developed an app that uses negative reinforcement, in the form of persistent smartphone vibrations, to remind users they’d exceeded their predetermined time limit.

Alas, this relies on a real will to escape addiction, which counter-attacks with many tricks. And it won’t work on video games.

Bill Gates trawling Washington for support for new kinds of nuclear reactors that could be both fail-safe and economical while ending all carbon emissions for power, especially at night.

Still, it would help to open Yucca Mountain. Not as a burial ground for “waste” for “10,000 years”… but as a bank repository for radionuclides our grandchildren will know how to use. But first, get all the waste away from our cities!

A new method of annealing by super-fast laser pulse allows direct conversion of carbon fibers and nanotubes into diamond fibers. Of course one envisions space tethers and Pi in the sky.

Converting waste heat to useful purposes could be very helpful.

As would finding ways to pull CO2 out of the air.

Oregon bottle deposit system hits 90 percent redemption rate.

Powerful reasons to include (carefully) nuclear in our non-carbon energy mix.

And weekly the fight for a strong, healthy, decent civilization goes on. Do your part.