--- In wta-talk@y..., "Eliezer S. Yudkowsky" <sentience@p...> wrote:
> Marc Geddes wrote:
>
> > Fair enough, but I did kind of get the impression that Eleizer
> > expected the FAI to act like a wish-granting genie.
>
> No, if I inflated to superintelligence, then this is my best
current
> understanding of how I would be ethically obligated to interact
with any
> still-basic humans around.
>
> Why? In the absence of an objective morality (an undetermined
possibility
> that needs to be projected both ways) then I don't see any moral
reason
> that would allow a superintelligence to meddle in other people's
lives
> without their consent. As a human, interacting with other humans,
I can
> try to convince people that they should want different things. I
can do
> this because I don't have the intelligence to see humans as systems
whose
> actions are effectively determined by which of several sentences I
choose
> to say to them. At that level, even simple conversation becomes
meddling
> with people's minds if you start choosing between innocent-sounding
> statements on the basis of a superintelligent understanding of how
those
> statements will affect that person's trajectory, according to goals
not
> shared by that person. I don't see any privileged moral frame of
> reference under human morality that would enable one entity's
strictly
> personal goals to justly supervene on another entity's strictly
personal
> goals. But giving people what *they* want, as opposed to what
*you* want
> or even what you would prefer that they want, is okay.
I find this argument bizarre. I fail to see that conversation
between the superintelligence and a human constitues meddling with
people's minds. You could use the same argument to say that those
with Ultra High IQ's should refrain from debating with average IQ
folks. I find it exceedingly unlikely that even a transhuman will
have the ability to understand how certain statements could affect a
person's trajectory. It's more likely that any algorithim complex
enough to support 'intelligence' displays 'chaotic' behaviour, in the
sense that nothing other than running the algorithim itself could
predict the algorithim's future states.
Why is giving people what *they* want O.K? It would seem to me
that 'giving people what they want' can said to consitute 'meddling'
of a greater degree than verbal persuasion. If a person asked me to
do something for them, but I knew it was going to harm them if I did
it, surely it is more ethical to refuse them?
Consider the argument from 'finitely specifiable limitations' on a
superhuman intelligence. No matter how fast it's improving, no
matter how powerful it's become, at any given time X a superhuman has
access to a *finite* amount of material resources, a *finite* amount
of processing speed, a *finite* amount of intellectual prowess and
*finite* limitations imposed by the currently understood *laws of
nature*. Granting any given request from a human would divert
resources from a finite pool that could have been used for an
alternative purpose. What are some possible consequences of this?
Example: self-improvement is likely to be a sub-goal of any super-
human. By continuously having to divert resources to grant requests,
the rate of self-improvement will be retarded. This may, in the long-
run, actually prove to be harmful to the sub-goal of 'helping
others' - the superhuman attains the greatest power to help others by
self-improving reasonably quickly.
Another example: The best way for a superhuman to help others may
not be to 'grant requests' directly, but instead a quite different
sub-goal may prove to be optimal. For instance 'discovering new
knowledge and sharing this knowledge as quickly as possible' may
actually turn out to be more benefical for humans in the long run.
By 'granting requests', resources are diverted in the short term and
the rate of discovery of new knowledge is retarded, a result actually
contradictory to the goal of 'helping others' .
And so on...
>
> Another way of thinking about it is this. Suppose that you're
interacting
> with an human-derived superintelligence that happens to be a devout
> Zoroastrian (a thought that makes my brain hurt, but let's suppose
for the
> sake of discussion that this is cognitively possible). Let's
suppose
> there are no external guards on how this superintelligence
interacts with
> you. What constraints do you hope this superintelligence feels
obliged to
> obey in interacting with you? I would feel that my rights had been
> violated if our conversation seemed innocent on the surface - in
> accordance with my wishes - but had actually been chosen out of a
vast
> space of possibilities with the intent of making my life serve the
overall
> goals of that ZSI (Zoroastrian superintelligence). I would not
wish to
> find my life trajectory warped to serve goals I find repugnant.
I just don't find this idea of 'mind control' through conversation to
be credible.
>
> But in the absence of objective morality, how can the same ethics
not also
> apply to a rational superintelligence interacting with a human
> Zoroastrian? Fair is fair. If I don't want my life warped to
serve a
> view of destiny that I disagree with, then undoubtedly neither do
> Zoroastrians. Now it could be that there is some view that is
simply
> *right* and that I and the Zoroastrians are both wrong. But in the
> absence of objective morality, what's left for the ethics of
> superintelligences interacting with mortals is the idea that
sentient
> beings should get what *they* want. And "what they want" should be
> defined however *they* want to define it.
Why is the only alternative to giving other sentients what they want
mind control? If a superhuman is pursuing self-centered goals of
it's own but subject to the constraint that it sees all other
sentients as ends in themselves, why would it have any desire to try
to manipulate other sentients into helping it with it's goals? In
any event, if the superhuman is really so superior in brain power, it
hardly needs the help of other mere moral sentients does it?
>
> If you're interacting as an equal, you can try and convince people
to want
> other things, and they can try and convince you back. But if you
have
> substantially transhuman brainpower and can model people from the
design
> stance, such that you have the power to convince people of
arbitrary
> propositions, at that point what you are doing ceases to
be "argument" and
> becomes "mind control", unless the discussion is carried out
*solely*
> according to the wishes of the person being talked to.
I don't buy that. Like I said, any algorithim complex enough to give
rise to an intelligent mind probably cannot be modelled from the
outside. The only way to predict how a given entity would act would
be to actually 'run' the algorithim....but then the map is equivalent
to the terrority in any case.
>
> Now you might say that an obvious way around this - in ethical
terms - is
> for a superintelligence to create an interacting system on the same
level
> on the human. And if I were talking with a voluntarily ethical
> Zoroastrian superintelligence for some reason, this is in fact what
I
> would prefer. Given my current limited understanding of ethics,
that is;
> obviously you should first ask a more trusted superintelligence
about
> whether this is what a smarter you would regard as the ethical way
to
> communicate...
>
> But if you're talking about a Transition Guide or Sysop - a
> superintelligence occupying a unique position with respect to the
> trajectory of human destiny - then *any* transfer of personal
preferences
> by the original programmer constitutes a hijacking of human
destiny.
> Would you want a transcending upload with personal goals you found
> repugnant using her unique position as Transition Guide to leverage
those
> goals over all human destiny? No? Then, failing an objective
morality,
> the general ethical principle must be that a mind which will occupy
a
> unique position with respect to human destiny, such as the first
seed AI,
> should not be leveraging the personal goals of its programmers.
Well, hopefully that won't be a problem because we both agree that
an 'objective morality' exists. (Didn't you tell me you believed in
objective morality?) All we need to do for seed-A.I is make sure
it's capable of metamorality - discovering the objective morality for
itself and modifying it's own super-goals accordingly.
Not in
> *any* way, big or small, because you wouldn't like it if someone
else did
> that to you. Eliminate sensitivities on the choice of original
> programmers. If any mind-in-general needs *some* set of highly
developed
> personal preferences, then use the modeled personal preferences of
a
> generic human, not the personal preferences of the original
programmers.
Well, here's where I guess I disagree with you. Just what are the
modeled personal preferences of a *generic human* ? ;) Surely the
logical course of action for the original programmers is to load in
the personal preferences that they believe are the most ethical.
> And that's even *if* the personal preferences of this mind-in-
general play
> no role in how it interacts with not-yet-upgraded intelligences, or
play a
> role only on the specific request of those intelligences. Don't
hijack
> human destiny. Being a seed AI programmer doesn't make you morally
> different from any other human.
>
> Of course my current understanding of morality may be wrong, but my
> current understanding of morality is all that I have available to
use as
> raw material to illustrate to an AI how human moral philosophy
works.
>
> Marc Geddes: Your wish is to achieve your immediate subgoals as a
result
> of personal effort, rather than having them granted directly as
wishes.
> (If you Google on "Singularity Fun Theory", you'll find out *why*
you
> prefer this... but that's a separate story.) That wish is
certainly
> suboptimally achieved with respect to your current environment.
You
> didn't create your shoes, much less your feet. Your functional
components
> are systems that evolution designed and then handed to you on a
silver
> platter. The various artifacts around you were created by other
humans
> and operate as independent causal systems divorced from your
sensorimotor
> architecture; you control them as external tools rather than
feeling them
> as parts of yourself. If your wish is to find happiness in
achieving your
> goals as the result of your personal creative efforts, rather than
having
> things handed to you on a silver platter, then that's certainly a
wish
> that could be achieved to a much greater degree after the
Singularity than
> before.
Huh. You're starting to sound a bit like an A.I <Marc backs slowly
away from Eliezer and looks scared>
Seriously, some forms of 'self-furfillment' probably cannot be
acheived in any way other than a person achieving goals for
themselves. For instance an experiential 'qualia' of 'self-
furfillment' may be linked to the awareness that aspects of
*objective reality* are the way they are because of the entities own
efforts.
>
> > All that's needed is an injunction not to violate the rights of
> > others. (i.e. to regard all other sentients as 'ends in
themselves'
> > and not initiate force against them except in self-defense)
> >
> > A little bit of self-centeredness should not be a problem if the
> > Kantian imperative is backed up by reason. (i.e. if the A.I
reasons
> > that it is indeed 'ethically correct' to regard other sentients
> > as 'ends in themselves')
> [snip]
> > Agreed. In fact as part of my suggested supergoals the A.I would
co-
> > operate with humans. (As part of it's quest for knowledge I
> > suggested a desire to 'teach' what it learns, so it would be
sharing
> > it's knowledge with us. In fact this sharing of knowledge is
> > actually probably the best way an A.I could help us)
>
> ...
>
> You can't mess around with AI morality this way - adding a piece
here,
> taking a piece there according to your whims. Do you construct
your own
> morality using this kind of thinking? No? Then why do you believe
you
> can use it to construct an AI's?
> This is not a question of making up random stuff and asking the AI
to do
> it. What do *you* believe is right? What do *you* believe is
wrong? No
> matter how imperfect your moral philosophy is, it's something that
can
> exist at least temporarily in at least one mind.
Agreed. What made you think I was 'picking stuff at random' when I
suggested the stated values? The values I selected stem from my own
world-view which can be justified through a comprehensive integrated
philsosophy of metaphysics, epistemology, ethics and politics. :D
For instance my idea of every entity as an 'end in itself' is not
a 'random injunction' I made up, but follows logically from my
philosophy.
>
> By thinking of FAI as something separate from yourself, you are
destroying
> your ability to even get started on understanding metamorality. We
> interact with other humans by cajoling them, giving them orders,
making
> alliances, wondering about their other loyalty. We *create* only
> ourselves. Building AI is an act of creation, not a matter of
giving
> orders. How do *you* choose between actions? What are *your*
supergoals?
My supergoals are precisely the one's I named!
*The life of each intelligent entity in pursuit of it's purpose is an
end (purpose) in itself
*One purpose of each intelligent entity is to perfect itself
*One purpose of each intelligent entity is to discover constructively
useful new knowledge
*One purpose of each intlliegent entity is to create aesthically
pleasing works
All flowing logically from the comprehensive meta-morality of Marc
Geddes :D
>
> If you're wondering how this gets reconciled with "no sensitivity
on
> choice of initial programmer", it's because opening a channel to
your own
> moral substance is how you provide the AI with an *interim*
approximation
> of the substance of humanity. In other words, because your
supergoals are
> good enough for you, doesn't mean they're good enough for the AI.
But if
> they're not good enough for *you*, they're *certainly* not good
enough for
> the AI. That doesn't mean inventing whatever you want piecemeal,
then
> rationalizing why each piece *would be* good enough for you. It
means
> using what actually genuinely *was* good enough for you.
O.K, I guess I agree. So I should attempt to give my A.I the same
morality that I myself have right - hopefully it's an *interim
approximation* to objective morality, but if not ,it doesn't matter
so long as the A.I has a way of working out objective morality for
itself?
>
> > It may be that a 'non-self
> > oriented supergoal' might turn out to harmful in some way that we
are
> > currently unaware of.
>
> You're going to have to offer something more specific than "might"
if you
> want to convince me that observer-centered supergoals are necessary
to the
> integrity of minds-in-general. I don't believe that having ten
fingers is
> necessary to all possible minds-in-general. I don't believe that
being
> built from amino acid chains twisted into complex shapes by van der
Waals
> forces is a requirement of minds-in-general. Why should the
feature you
> cite be anything except another contingent product of our
evolutionary
> origins? You are arguing from your own ignorance of the
evolutionary
> psychology of goal systems. I suppose this can be an effective
argument
> if the audience doesn't know either - i.e., Jeremy "How do we
*know*
> genetically engineered organisms won't spontaneously explode?"
Rifkin -
> but it ill behooves a wannabe specialist.
Well, I thought my previous posts contained some valid arguments ;)
Short and dirty argument:
If the main super-goal is not observer-centered than in some
circumstances the mind will endanger its own existence by diverting
resources that should have been used to protect it's own goal-
system. If the mind is destroyed than it cannot carry out any of its
putative non-self-centered goals (or any goals whatsoever, because it
will be dead!). A logical contradiction.
>
> --
> Eliezer S. Yudkowsky http://singinst.org/
> Research Fellow, Singularity Institute for Artificial Intelligence
Received on Thu Oct 17 21:49:42 2002
This archive was generated by hypermail 2.1.8 : Mon Sep 08 2003 - 11:37:39 CEST