One problem is that human value might be inherently multidimensional. I think of it as the want/like/approve distinction. We seem to have separate mechanisms in our brains for 1) enjoying something in the moment, 2) wanting to do it before, and 3) approving of it afterward. It's possible to want something without enjoying it (like a person with OCD wanting to close the door exactly 10 times), enjoy something without wanting it (people have said that they've reached very enjoyable medidative states but feel zero motivation to reach them again), enjoy something without approving it (porn), approve something without enjoying it (exercise), and all other combinations. This is the reason why "revealed preference" doesn't work: a person's actions are dictated disproportionally by the "want" dimension, but a good theory of value should incorporate all three. If we optimize one over the others, the tails will come apart.
A nice toy example is video games, where people are attracted to them because of the graphics, then stay because of the gameplay, and then have a warm afterglow and want to discuss afterward because of the story. Which of the three should contribute the most to the "true" quality rating of a videogame - "want", "like", or "approve"? Is this question philosophically meaningful? Will more reflection solve it?
Is this question philosophically meaningful?
Assuming you mean something like "Can philosophy answer this question?" I think "Maybe, but we probably won't know until we do a lot more philosophy." To put it another way, I think it's very plausible (but far from certain) that we can eventually answer questions like this one, given enough competent reflection, and I want to make sure we definitively find out before we make any irreversible choices based on what we think our values are.
I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection.
Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later) safer, but the deeper problem is that humans aren't safe, and can't safely serve as builders, overseers, or alignment targets for powerful AIs.
Problem with Long Reflection: It seems to imply that the main problem with humans is that we just haven't had enough time to think, that reflection is the main thing we need to do more of, and then we can get on with building powerful AIs or other technologies. Or that if we build aligned AIs that sincerely help us think a lot more, or do the thinking for us, then things will turn out fine.
So I think we need a catchy handle for a related but distinct idea, that humans aren't ready to build AIs or other extremely powerful technologies, because we're currently too flawed, in a variety of ways, and it will take a long process (which may or may not end up succeeding) to fix those flaws.
A summary of the flaws that I have in mind:
(This list focuses on key bottlenecks that seem hard to fix even with AI assistance or intelligence enhancement, and isn't meant to be a complete list of human flaws / safety problems. It ignores e.g. that the median human is ignorant of many important issues, and that we're currently quite bad at complex large-scale coordination such as passing/implementing close-to-optimal government policies.)
My main hope for a Long Self-Correction eventually succeeding rests on the fact that humans have seemingly, mysteriously, made progress on these issues over a very long period of time, so if we preserve the environment in which we can seemingly do this, and not give anyone or anything the power to permanently derail such progress, then maybe we can continue to snowball The Correction until we reach a point when we can rightly justify reshaping the universe according to our volition.
It will probably be shortened to "The Long Correction" at some point if it catches on, similar to how "outer space" is now often just "space".
Why isn't there a version of EA that explicitly talks about how to leverage people's status motivations to do more good for the world? It's very possible that explicit talk about status is actually counterproductive at least in the short run, e.g. it heightens status motivations and makes people less altruistic, but then do we just march into the future while blindfolding ourselves to this aspect of human nature?
Facts Only
* Human value might be inherently multidimensional, defined by want/like/approve distinctions.
* Separate brain mechanisms exist for enjoying something in the moment, wanting to do it beforehand, and approving of it afterward.
* One can want without enjoying (e.g., OCD) or enjoy without wanting (meditative states).
* The distinction shows that optimizing a single dimension leads to divergent outcomes, as seen with examples like pornography or exercise.
* Revealed preference methods fail because actions are dictated disproportionately by the "want" dimension.
* Video games involve attraction from graphics (want), persistence from gameplay (like), and discussion from story (approve).
* The text proposes "The Long Self-Correction" as an alternative concept to AI Pause and Long Reflection.
* The motivation for pausing is framed around ensuring safer development of powerful AIs.
* The proposed correction rests on the historical progress humans have made in addressing systemic flaws.
* A specific area for re-examination is whether explicit status motivations should be incorporated into frameworks like Evolutionary Algorithms (EA).
Executive Summary
Full Take
Sentinel — Human
The text reads like a stream-of-consciousness philosophical exploration synthesizing psychology and AI ethics into a novel framework, strongly suggesting human authorship.