__ __ _ _____ _ _ _
| \/ | ___| |_ __ _| ___(_) | |_ ___ _ __
| |\/| |/ _ \ __/ _` | |_ | | | __/ _ \ '__|
| | | | __/ || (_| | _| | | | || __/ |
|_| |_|\___|\__\__,_|_| |_|_|\__\___|_|
community weblog
In AI, it's easy to argue about philosophical questions over-much
So please, remember: there are a very wide variety of ways to care about making sure that advanced AIs don't kill everyone. Fundamentalist Christians can care about this; deep ecologists can care about this; solipsists can care about this; people who have no interest in philosophy at all can care about this. Indeed, in many respects, these essays aren't centrally about AI risk in the sense of "let's make sure that the AIs don't kill everyone" (i.e., "AInotkilleveryoneism") – rather, they're about a set of broader questions about otherness and control that arise in the context of trying to ensure that the future goes well more generally. from Otherness and control in the age of AGI by Joe Carlsmith
The first essay, "Gentleness and the artificial Other," discusses the possibility of "gentleness" towards various non-human Others – for example, animals, aliens, and AI systems.
The second essay, "Deep atheism and AI risk," discusses what I call "deep atheism" – a fundamental mistrust both towards Nature, and towards "bare intelligence."
The third essay, "When 'yang' goes wrong," expands on this concern. In particular: it discusses the sense in which deep atheism can prompt an aspiration to exert extreme levels of control over the universe.
The fourth essay, "Does AI risk 'other' the AIs?", examines Robin Hanson's critique of the AI risk discourse – and in particular, his accusation that this discourse "others" the AIs, and seeks too much control over the values that steer the future.
The fifth essay, "An even deeper atheism," argues that this discomfort should deepen yet further when we bring some other Yudkowskian philosophical vibes into view – in particular, vibes related to the "fragility of value," "extremal Goodhart," and "the tails come apart."
The sixth essay, "Being nicer than Clippy," tries to draw on this guidance. In particular, it tries to point at the distinction between a paradigmatically "paperclip-y" way of being, and some broad and hazily-defined set of alternatives that I group under the label "niceness/liberalism/boundaries."
The seventh essay, "On the abolition of man," examines another version of that concern: namely, C.S. Lewis's argument (in his book The Abolition of Man) that attempts by moral anti-realists to influence the values of future people must necessarily be "tyrannical."
The eighth essay, "On green," examines a philosophical vibe that I (following others) call "green," and which I think contrasts in interesting ways with "deep atheism."
The ninth essay, "On attunement," continues the project of the previous essay, but with a focus on what I call "green-according-to-blue," on which green is centrally about making sure that we act with enough knowledge.
Related: Why general artificial intelligence will not be realized [Nature]
Previously: posting such things on an Internet forum could cause incalculable harm
posted by chavenet on May 09, 2024 at 1:31 AM
---------------------------
counterpoint: what if the moon got mad?
posted by lalochezia at 3:19 AM
---------------------------
Recall, from my last essay, that dead bear cub, and its severed arm – torn off, Herzog supposes, by a male bear seeking to stop a female from lactating.
Sorry, but moments of zen like the above chestnut aside, I find this nine-part essay series on AGI terribly dull and will not finish reading it because I'm too fucking busy and too busy fucking.
posted by AlSweigart at 4:33 AM
---------------------------
We already knew in 1979, had we listened to the message of Star Trek: The Motion Picture, that general intelligence could not be realized, at least not the way Sam Altman is claiming you can, by vacuuming up huger and huger amounts of data and thinking it will add up to humanity.
In that film Mr. Spock makes contact with the giant entity V'ger, who has traveled the entire universe absorbing all the data there is to absorb. Yet despite all of this, a crying Spock realizes, the machine "cannot understand this simple feeling"--the warmth of human feeling when we hold hands.
posted by johngoren at 4:40 AM
---------------------------
The world is burning and people are dying needlessly from hunger, treata le disease, and war, and these deep thinkers are worried we might be mean to toasters.
posted by The Manwich Horror at 5:13 AM
---------------------------
First, define AGI:
- OpenAI's internal definition is expert systems that can replace a majority of US office workers, which is pretty fucking telling.
- the other definition which most people mean (if they talk about it at all) is Data from Star Trek TNG: a fully sapient artificial system capable of continuous adaptive learning and personal growth through real time interaction with the world / people in it.
Using the first definition you can probably begin to approach "AGI" (and certainly a killer app-grade digital assistant) with a combination of LLMs and the methods outlined in OpenAI's Step-By-Step Verification and Self-Taught Reasoning papers, usually referred to as Q* (Q-Star).
Necessary background:
LLMs are artificial neural networks with many layers of fixed (after training) connection weights, which more-or-less reflect how humans parse and employ a language the network was trained on, and in the derivatives of their connection topology contain a fairly accurate map of our concepts and how they relate, with a degree of subtext and (haphazardly filtered) prejudices intact. That "fairly accurate" conceptual map has lately been massively boosted by multi-modality: dumping video, images, and audio into the mix to give all that language a little context. But LLMs cannot encounter a new problem and come up with a wholly novel solution in the sense of making a little diorama of it in their mind's eye (they have none) and running a bunch of predictive experiments with a specific goal in mind that only cares about a few specific factors, and upon finding a successful set of actions execute them in the real world.
That inability, in a more general sense, is why they can't always follow fairly basic reasoning or recognize when they're hallucinating. There's nobody at home, no mental models or metacognition (thinking about thinking). Even humans with no visual mind's eye casually instantiate and collapse generalized mental models on the fly hourly.
That sort of thing is (or someday will be) rightfully the domain of an entirely different approach to neural networks: reinforcement learning-based ones, the kind you use to solve mazes, autonomously play video games, or teach robots to walk. Without any copyright theft. Really any tasks that can be given a numerical score, and the connection weights iteratively optimized for higher scores. They are almost always vastly smaller than LLMs or similar deep learning systems, but there are strategies to make them not only consistently produce optimized output to a specific problem, but do so zero-shot: as in something they were not specifically trained on but in the same general domain. They are used heavily in the fine-tuning of LLMs after pre-training (dumping a snapshot of the Internet into the LLM) so that LLMs can do things like beat doctors at medical questions, maybe even zero-shot. Basically a great way to weed out the more overtly stupid LLM output before it goes live. Continuous, adaptive/dynamic reinforcement-based networks are an insanely complex proposition just as a structural matter and not simply "solved" by scaling hardware, and anything of that nature is far behind LLMs in terms of recent progress.
Background complete, and now for the extremely abbreviated version of Q*:
1) We know from nVidia's Eureka demo and further research since, that LLMs can - despite being giant fixed sets of weights - usually write better success-evaluator code for reinforcement-based neural networks than human experts(!!!)
2) We could brute-force problem solving (systems modeling or systems thinking) by breaking problems down into many tiny steps, spawning millions or even billions of randomized reinforcement-based neural networks for each step, and - using LLM-authored evaluator code - continuously cull the underperformers until we have a chain of optimized reinforcement networks for handling each step of the problem. Maybe even some genetic algorithm-style initial weight refinement
3) Build a massive, $100 billion supercomputer, give it a "cool" name like Stargate and set it to do all this brute-force problem solving (never fucking mind the ecological impact: want to know the best route across New York in rush hour? Just burn down a rainforest! Or equivalent carbon footprint)
4) Voila! AGI in the digital assistant sense and possibly in the replace >50% of office workers sense
As someone who loved, and still loves, neural networks as an approach to AI I cannot begin to tell you how much I fucking hate this idea. Its inelegance, its likely atrocious carbon footprint, but mostly because I can't think of a good reason it won't work well enough to pass in the second or third generation (2026~2028 at current release cadence, the latter end lines up with Stargate's estimated completion). Worse yet, it's being done by OpenAI who are consistently the least ethical operator in this space. Even Google has a reduced functionality open source model these days.
The only good part of this is that LLMs are (like it or not) trained on basically all human culture ever uploaded to the Internet, and are thus profoundly biased towards human values and ways of classifying the world. That cuts both ways: their sexist, racist output is a reflection of our sexist, racist society. But our collective disdain for things like mass murder, continuous surveillance, and the worst predations of authoritarian capitalst power structures are embedded - however deeply and subtly - into every model trained on the Common Crawl. But there are Nazi worldviews in the soup as well, and no guarantees what kind of mixture you'll get from a prompt modulated by a random seed.
And Data from Star Trek AGI? Prior to this shifting of the definition goalposts the median estimate of experts in the field was about 30 years from now. I'd personally bet on 70, still, but I can't deny something like Q* might be capable of bumping up the technological growth exponent.
[note: dear god, any experts here on MeFi, please correct some likely major errors in the above if you're feeling generous. I desperately want to know where I'm wrong, and have it called out where we all can learn]
posted by Ryvar at 6:51 AM
---------------------------
I am firmly in the camp that says "AGI will not be realized with any methods currently on the horizon." So I see no point in looking at a lot of articles about paradigms for understanding and managing AGI risk. Though I probably will look at the Nature link.
posted by Aardvark Cheeselog at 7:03 AM
---------------------------
The horror of AGI, as OpenAI apparently imagines it, is that it's basically slavery. They want to invent a person in order to have a personal secretary. The only saving grace of their vision is that it's never going to happen, but they're going to destroy jobs, waste fresh water, and emit a bunch of carbon in the process of trying.
I skimmed the above essays, but as far as I can tell they're entirely about "what criteria do we use to decide whether the person we've invented for the purposes of enforced labor is good/nice enough," which is not really any less loathsome in my book.
posted by Four String Riot at 7:31 AM
---------------------------
With a giant "As I Understand It": the Q* proposal still lacks the recursive agent modeling necessary to pull off anything we'd call identity or real sapience if we were somehow able to side-by-side directly compare it with our own inner thought life. Like, sub-chimpanzee at theory of mind. Most of the ways you'd extremely-hypothetically set about addressing that would almost immediately step off the definitional slavery cliff, which I 100% trust OpenAI to not give a shit about.
posted by Ryvar at 8:05 AM
---------------------------
Do you ever get the feeling that a lot of smart people are worried about the wrong things?
When I see Photoshop advertising its AI powered image processing, I don't find it repulsive because I'm worried about the singularity, or because it is a tentacle of some future machine god Other reaching back from apocalyptic future. I find it repulsive because I see LLMs as an example of the powerful taking stuff from the powerless, and then adding insult to injury by trying to sell the loot back.
It's a bummer. I used to be a technophile. I'd see "AI" in a product description and I'd think warm thoughts about all the clever programmers doing clever things, and now I just wish I had a bigger hammer.
posted by surlyben at 8:42 AM
---------------------------
Can capitalism be taught to think before it kills us all? All signs point to no.
posted by Artw at 8:49 AM
---------------------------
I've said this in the past, but lots of high-level AI development is being steered by people who create things by describing what they want (a prompt, as it were) and sitting back while dozens if not hundreds of programmers/scientists/artists actually implement it. Their reality is one where ideas--no matter how vague, impractical, or insoluble--are what's important and they honestly believe that they're genius billionaire technologists because their ideas are what makes them valuable.
That's why the exploitation is built into the technology. That's why the every single application of LLMs takes the form of some sort of assistant you can order around. Who cares about the artist and their skills when it's your vision that's the important part?
posted by RonButNotStupid at 9:00 AM
---------------------------
... LLMs as an example of the powerful taking stuff from the powerless, and then adding insult to injury by trying to sell the loot back.
I've heard colonization analogies before, but they haven't landed before. This really brings it home for me. Thanks!
posted by dbx at 9:28 AM
---------------------------