Iason Gabriel on Foundational Philosophical Questions in AI Alignment

[Music]
[Music]
welcome to the ai
alignment podcast i'm lucas perry
today we have a conversation with yasen
gabriel
about a recent paper that he wrote
titled artificial intelligence
values and alignment this episode
primarily explores how
moral and political theory are deeply
interconnected
with the technical side of the ai
alignment problem
and important questions related to that
interconnection
we get into the problem of dealing with
a plurality
of preferences and philosophical views
the is ought problem
meta ethics how political theory can be
helpful
for resolving disagreements what it is
that we're trying to
align ai systems to the importance of
establishing a broadly endorsed
procedure and set of principles for
alignment
and we end on exploring the long
reflection
this was a very fun and informative
episode yasen has succeeded in bringing
new ideas and thought to the space of
moral and political thinking in ai
alignment
and i think you'll find this episode
enjoyable and valuable
if you don't already follow us you can
subscribe to this podcast on your
preferred podcasting platform by
searching for
the future of life or following the
links on the page for this podcast
jason gabriel is a senior research
scientist at deepmind where he works in
the ethics research team
his research focuses on the applied
ethics of artificial intelligence
human rights and the question of how to
align technology with human values
before joining deepmind yassin was a
fellow in politics at st
john's college oxford he holds a
doctorate
in political theory from the university
of oxford
and spent a number of years working for
the united nations
in post-conflict environments and with
that
let's get into our conversation with
yassin gabriel
so we're here today to discuss your
paper
artificial intelligence values and
alignment
to start things off here i'm interested
to
know what you found so compelling about
the problem of
ai values and alignment and generally
just what this paper is all about
yeah thank you so much for inviting me
lucas so this paper
is in broad brush strokes about how we
might think about
aligning ai systems with human values
and i wrote this paper because i wanted
to bring different communities together
so on the one hand i wanted to show
machine learning researchers that there
were some interesting normative
questions
about the value configuration we align
ai with that deserve further attention
at the same time i was keen to show
political and moral philosophers
the ai was a subject that provoked real
philosophical reflection
and that this is an enterprise that is
worthy of their time as well
let's pivot into what the problem is
than the
technical researchers and people
interested in normative questions and
philosophy can both contribute to
so what is your view then on what the ai
alignment problem is and the two parts
you believe it to be composed of
in broad brush strokes i understand the
challenge of value alignment
in a way that's similar to stuart
russell who says that the ultimate aim
is to ensure that powerful ai
is properly aligned with human values i
think that when we reflect upon this
in more detail it becomes clear that the
problem decomposes
into two separate parts the first is the
technical challenge
of trying to align powerful ai systems
with human values
and the second is the normative question
of what or whose values we try to align
ai systems with
oftentimes i also see a lot of
reflection on ai policy and ai
governance
as being a core issue to also consider
here given that
people are concerned about things like
race dynamics
and unipolar versus multipolar scenarios
with regards to
something like agi what are your
thoughts on this and i'm curious to know
why
you break it down into technical and
normative
without introducing political or
governance issues
yeah so this is a really interesting
question and i think that one will
probably discuss at some length later
about the role of politics in creating
aligned ai systems
of course in the paper i suggest that an
important challenge for people who are
thinking about value alignment
is how to reconcile the different views
and opinions of people given that we
live in a pluralistic world
and how to come up with a system for
aligning ai systems that treats people
fairly despite that difference
in terms of practicalities i think that
people envisage alignment in different
ways
some people imagine that there will be a
human parliament
or a kind of centralized body that can
give very coherent
and sound value advice to ai systems
and essentially that the human element
will take care of this problem of
pluralism
and just give ai very very robust
guidance about things that we've all
agreed upon are the best thing to do
at the same time there's many other
visions for ai or versions of ai
that don't depend upon that human
parliament being able
to offer such cogent advice so we might
think that there are worlds in which
there's multiple
ais each of which has a human
interlocutor
or we might imagine ai is working in the
world to achieve
constructive ends and that it needs to
actually be able to perform these value
calculations
or this value synthesis as part of its
kind of default operating procedure
and i think it's an open question what
kind of ai system we're discussing
and that probably the political element
understood in terms of real-world
political institutions
will need to be tailored to the vision
of ai that we have in question
all right so can you expand that a bit
on the relationship between the
technical
and normative aspects of ai alignment a
lot of the focus is on the normative
part of the value alignment question
trying to work out
which values to align ai systems with
whether it is values
that really matter and how this can be
decided
i think this is also relevant when we
think about the technical design of ai
systems
because i think that most technologies
are not value agnostic
so sometimes when we think about ai
systems we assume that they'll have this
general capability
and that it will almost be trivially
easy for them to align
with different moral perspectives or
theories
yet when we take a ground level view and
we look at the way in which ai systems
are being built
there's various path dependencies that
are setting in
and there's different design
architectures that will make it easier
to follow one moral trajectory rather
than the other
so for example if we take a
reinforcement learning paradigm
which focuses on teaching agents tasks
by
enabling them to maximize reward in the
face of uncertainty over time
a number of commentators have suggested
that that model fits particularly well
with a kind of utilitarian decision
theory which aims to promote happiness
over time in the face of uncertainty
that it would actually struggle to
accommodate a moral theory that
embodies something like rights or heart
constraints
and so i think that if what we do want
is a rights-based
vision of artificial intelligence it's
important that we get that ideal clear
in our minds
and that we design with that purpose in
mind
this challenge becomes even clearer when
we think about moral philosophies such
as a kantian theory
which would ask an agent to reflect on
the reasons it has for acting and then
ask whether they universalize to good
states of affairs
and this idea of using the currency of a
reason
to conduct moral deliberation would
require
some advances in terms of how we think
about ai
and it's not something that it's very
easy to get a handle on from a technical
point of view
so the key takeaway here is that what is
going to be possible in terms of the
normative
and in terms of moral learning and moral
reasoning and ai systems
will supervene upon technical pathways
that we take and so it is
important to be mindful of the
relationship between
what is possible normatively given what
is technically known
and to try and navigate that with
mindfulness about that relationship
i think that's precisely right i see at
least two relationships here so the
first
is that if we design without a
conception of value in mind it's likely
that the technology that we build
will not be able to accommodate any
value constellation
and then the mirror side of that is if
we have a clear
value constellation in mind we may be
able to develop technologies that can
actually implement or realize that ideal
more directly and more effectively
can you make a bit more clear
the ways in which for example
path dependency of current technical
research
makes certain normative ethical theories
more
plausible to be instantiated in ai
systems than others
yeah so i should say that obviously
there's a wide variety of different
methodologies that are being tried at
the present moment
and that intuitively they seem to match
up well with different kinds of theory
of course the reality is a lot of effort
has been spent trying to ensure that ai
systems are safe
and that they are aligned with human
intentions when it comes to richer goals
so trying to evidence a specific moral
theory
a lot of this is conjecture because we
haven't really tried to build
utilitarian or kentian agents in full
but i think in terms of the details so
with regards to reinforcement learning
we have this obviously an optimization
driven process
and there is a whole caucus of moral
theories that basically
use that decision process to achieve
good states of affairs
and we can imagine you know roughly
equating the reward that we used to
train
an rl agent on with some metric of
subject of happiness or something like
that
now if we were to take a completely
different approach so say
virtue ethics virtue ethics is radically
contextual obviously and it says that
the right thing to do
in any situation is the action that
evidences
certain qualities of character and that
these qualities can't be
expressed through a simple formula that
we can maximize for
but actually require a kind of context
dependence
so i think that if that's what we want
if we want to build agents that have
a virtuous character we would really
need to think about the fundamental
architecture potentially in a different
way
and i think that that kind of insight
has actually
been speculatively adopted by people
who consider forms of machine learning
like inverse reinforcement
learning who imagined that we could
present an agent with
examples of good behavior and that the
agent would then learn them in a very
nuanced way
without us ever having to describe in
full what the action was
or give it appropriate guidance for
every situation
so as i say these really are quite
tentative thoughts
but it doesn't seem at present possible
to build an ai system
that adapts equally well to whatever
moral theory or perspective we believe
ought to be promoted or endorsed
yeah so that doesn't make sense to me
that different techniques
would be more or less skillful for
more readily and fully adopting certain
normative perspectives and capacities in
ethics i guess the part that i was just
getting a little bit tripped up on
is that i was imagining that if you have
an optimizer
being trained off something like
maximize
happiness then given the massive
epistemic difficulties of running
actual utilitarian optimization process
that is only thinking at the level of
happiness
and how impossibly difficult that would
be that like human beings
who are consequentialists it would then
through
gradient descent or being pushed and
nudged from the outside or something
would find virtue ethics and
deontological ethics
and that those could then be run as a
part of
its world model such that it makes the
task
of happiness optimization much easier
but i see how intuitively
it more obviously lines up with
utilitarianism and then how it would be
more difficult to get it to find
other things that we care about like
virtue ethics
or deontological ethics does that make
sense yeah i mean it's a very
interesting conjecture that if you
set an agent off with the learned goal
of trying to maximize human happiness
that it would almost by necessity learn
to accommodate
other moral theories and perspectives
kind of suggests that there is a core
driver which animates moral inquiry
which is this idea of collective welfare
being realized in a sustainable way
and that might be plausible from an
evolutionary point of view but there's
also
other aspects of morality that don't
seem to be built so clearly
on what we might even call the pleasure
principle and so i'm not entirely sure
that you would actually get to a right
space
morality if you started out from those
premises
what are some of these things that don't
line up with this pleasure principle for
example
i mean of course utilitarians have many
sophisticated theories about
how endeavors to improve total aggregate
happiness
involve treating people fairly placing
robust side constraints on what you can
do to people
and potentially even encompassing other
goods such as
animal welfare and the well-being of
future generations
but i believe that the philosophical
consensus or the proponents of opinion
is that actually unless we can say that
certain things matter fundamentally
for example human dignity or the
well-being of future generations
or the value of animal welfare is quite
hard
to build a moral edifice that adequately
takes these things into account
just through instrumental relationships
with human well-being or human happiness
so understood
so then we have this technical problem
of how
to build machines that have the capacity
to do
what we want them to do and to help us
figure out what we would want to want
us to get the machines to do an
important
problem that comes in here is the zot
distinction
by hume where we have say
facts about the world on one hand is
statements
we can even have is statements about
people's preferences
and meta preferences and the
collective state of all normative and
meta-ethical views on the planet at a
given time
and the distinction between that
and ought which is a normative claim
synonymous with should and is kind of
the basis of morality
and the tension there between
what assumptions we might need to get
morality off of the ground and how we
should interact with
a world of facts and a world of
norms and how they may or may not relate
to each other
for creating a science of well-being or
not even doing that
so how do you think of coming up with an
appropriate alignment
procedure that is dependent on the
answer to this distinction
yeah so that's a fascinating question
so i think that that is ought
distinction is quite fundamental
and it helps us answer one important
query
which is whether it's possible to solve
the value alignment question
simply through an empirical
investigation of
people's existing beliefs and practices
and if you take the issue distinction
seriously
it suggests that no matter what we can
infer from studies of what is
already the case so what people happen
to prefer or happen to be doing
we still have a further question which
is should that perspective be endorsed
is it actually the right thing to do and
so there's always this critical gap
it's a space for moral reflection and
moral introspection
and a place in which error can arise so
we might even think that if we studied
all the global beliefs of different
people and found that they agreed upon
certain axioms or moral properties that
we could still ask
are they correct about those things and
if we look at historical beliefs we
might think that there was actually
a global consensus on moral beliefs or
values that turned out to be mistaken
so i think that these endeavors to kind
of synthesize moral beliefs to
understand them properly
are very very valuable resources for
moral theorizing
it's hard to think where else we would
begin but ultimately
we do need to ask these questions about
value more directly
and ask whether we think that the final
elucidation of an idea
is something that ought to be promoted
so in some it has a number of
consequences but i think one of them is
that we do need to maintain a space for
normative inquiry
and value alignment can't just be
addressed through an empirical
social scientific perspective right
because
one's own perspective on the isot
distinction and whether
and how it is valid will change
how one goes about learning and evolving
normative and meta-ethical thinking yeah
perhaps at this point an example will be
helpful
so suppose we're trying to train a
virtuous agent that has
these characteristics of treating people
fairly
demonstrating humility wisdom and things
of that nature
suppose we can't specify these up front
and we do need
a training set we need to present the
agent with examples of what people
believe evidences characteristics we
still have the normative
question of what goes into that data set
and how do we decide
so the evaluative questions get passed
on to that of course we've seen many
examples
of data sets being poorly curated and
containing bias that then transmutes
onto the ai
system we either need to have data
that's curated so that it meets
independent moral standards and the ai
learns from that data
or we need to have a moral ideal that is
freestanding in some sense and that ai
can be built to align with let's try and
make that even more concrete because i
think this is a
really interesting and important problem
about why
the technical aspect is deeply related
with
philosophical thinking about this is odd
problem
so the highest level of abstraction like
starting with axioms around here
if we have is statements about data sets
and so data sets are just information
about the world
the data sets are the is statements we
can put whatever is statements into a
machine
and the machine can take the shape of
those
values already embedded and codified in
the world in people's minds or
in our artifacts and culture and then
the odd question as you said is what
information in the world should we use
and to understand what information we
should use
requires some initial principle
some set of axioms that bridges the isot
gap
so for example the kind of move that i
think sam harris tries to lay out
is this axiom like we should avoid
the worst possible misery for everyone
and you may or may not agree with that
axiom but that is the starting point for
how one might bridge the izot
gap to be able to select for which data
is better than other data or which data
we should unload to ai systems
so i'm curious to know how is it that
you think about
this very fundamental level of initial
axiom or
axioms that are meant to bridge this
distinction
i think that when it comes to these
questions of value
we could try and build up from these
kind of very very minimalist assumptions
of the kind that it sounds like sam
harris
is defending we could also start with
richer conceptions of value that seem to
have
some measure of widespread ascent and
reflective endorsement
so i think for example the idea that
human life matters
or that sentient life matters that it
has value and hence that suffering is
bad
is a really important component of that
i think that conceptions of fairness of
what people
deserve in light of their equal moral
standing is also an important
part of the moral content of building an
aligned ai system
and i would tend to try and be inclusive
in terms of the values that we can verse
so i don't think that we actually need
to take this very defensive posture
i think we can think expansively about
the conception
and nature of the good that we want to
promote and that we can actually have
meaningful
discussions and debate about that so we
can put forward reasons for defending
one
set of propositions in comparison with
another
we can have epistemic humility here
given the history of moral catastrophes
and how morality
continues to improve and change over
time and that surely we do not sit
at a peak of moral enlightenment in 2020
so given our epistemic humility we can
cast a wide net
around many different principles so that
we don't lock ourselves into anything
and can endorse a broad notion
of good which seems safer
but perhaps has some costs in itself
for allowing and being more permissible
for a wide range of moral views
that may not be correct i think that's
broadly speaking correct
we definitely shouldn't tear the
artificial intelligence
too narrowly to the morality of the
present moment given that we may and
probably are making moral mistakes of
one
kind or another and i think that this
thing that you spoke about a kind of
global conversation about value is
exactly right
i mean if we take insights from
political theory seriously
then the philosopher john rules suggests
that a fundamental element of the
present human condition
is what he calls the fact of reasonable
pluralism which means that when people
are not coerced and when they're able to
deliberate freely
they will come to different conclusions
about what ultimately has moral value
and how we should characterize ought
statements at least when they apply to
our own personal lives
so if we start from that premise we can
then think about ai
as a shared project and ask this
question which is
given that we do need values in the
equation that we can't just do
some kind of descriptive enterprise and
that that will tell us what kind of
system to build
what kind of arrangement adequately a
factors in
people's different views and
perspectives and seems like a solution
built upon the relevant kind of
consensus to value alignment
that then allows us to realize a system
that can reconcile these different moral
perspectives
and takes a variety of different values
and synthesizes them in a scheme that we
would all like
i just feel broadly interested in just
introducing a little bit more
of the debate and conceptions around the
is ought problem right because
there are some people who take it very
seriously and other people who
try to minimize it or are skeptical
of it doing the kind of philosophical
work that many people
think that it's doing for example sam
harris is a big skeptic
of the kind of work that the izop
problem is doing
and on this podcast we've had people on
who are
for example realists about consciousness
and
there's just a very interesting broad
range of views about value that
inform the izot problem if one's a
realist about consciousness and thinks
that suffering
is the intrinsic valence carrier of
dis value in the universe and that joy
is the intrinsic valence carrier of
well-being
one can have different views on how that
even translates to
normative ethics and morality and how
one does that given one's view
on the is a problem so for example if we
take that kind of
metaphysical view about consciousness
seriously then
if we take the izot problem seriously
then even though there are actually bad
things in the world
like suffering those things are bad but
that it would still require some kind of
axiom
to bridge the is-odd distinction if we
take it seriously so
because pain is bad we ought to avoid it
and that's interesting and important
and a question that is at the core of
unifying ethics and
all of our endeavors in life and if you
don't take the izot problem seriously
then you can just be like because i
understand the way that the world is
by the very nature of being a sentient
being and understanding the nature of
suffering
there's no question about the kind of
navigation problem that i have
even in the very long term the answer to
how one
might resolve the is ought problem would
potentially be a way of
unifying all of knowledge and endeavor
all the empirical sciences would be
unified
conceptually with the normative right
and then
there's no more conceptual issues so i
think i'm just trying to illustrate the
power
of this problem and distinction it seems
it's a very interesting set of ideas to
my mind these kind of arguments about
the intrinsic badness of pain or kind of
naturalistic moral arguments
are very strong ways of arguing against
say
moral relativist or moral nihilist but
they don't necessarily
circumvent the issue distinction because
for example the claim that pain
is bad is referring to a normative
property so if you say
pain is bad therefore it shouldn't be
promoted like that's completely
compatible with believing that we can't
deduce moral arguments from purely
descriptive premises
so i don't really believe that the is
ought distinction
is a problem i think that it's always
possible to make arguments about values
and that that's precisely what we should
be doing and that the fact that that
needs to be conjoined with
empirical data in order to then arrive
at sensible judgments and practical
reason about what ought to be done
is a really satisfactory state of
affairs
i think one kind of interesting aspect
of the vision you put forwards was this
idea of a kind of
unified moral theory that everyone
agrees with
and i guess it does touch upon a number
of arguments that i make in the paper
where i juxtapose two slightly stylistic
descriptions of solutions to the value
alignment challenge
the first one is of course the approach
that i termed the true moral theory
approach
which holds that we do need a period of
prolonged reflection
and we reflect fundamentally on these
questions about pain
and perhaps other very deep normative
questions
and the idea is that by using tools from
our philosophy
eventually although we haven't done it
yet we may identify
a true moral theory and then it's a
relatively
simple well not simple from a technical
point of view but simple
from a normative point of view task of
aligning
ai maybe even agi with that theory and
we've basically solved the value
alignment problem
so in the paper i argue against that
view
quite strongly for a number of reasons
the first is that i'm not sure how we
would ever know that we'd identified
this true moral theory
of course many people throughout history
have thought that they've discovered
this thing
and often gone on to do profoundly
unethical things to other people
and i'm not sure how even after a
prolonged period of time we would
actually have confidence that we had
arrived at the really true thing and
that we couldn't still ask the question
am i right but even putting that to one
side supposed that i had
not just confidence but justified
confidence that i really
had stumbled upon the true moral theory
and perhaps
with the help of ai i could look at how
it plays out in a number of different
circumstances
and i realized that it doesn't lead to
these kind of weird anomalous situations
that most
existing moral theories point towards
and so i really am confident that it's a
good one
we still have this question of what
happens when we need to persuade
other people that we found the true
moral theory and whether that is
a further condition on an acceptable
solution to the value alignment problem
and in the paper i say that it is a
further condition
that needs to be satisfied because just
knowing
or supposedly having access to justified
belief in a true model theory
doesn't necessarily give you the right
to impose that view
upon other people particularly if you're
building a very powerful technology that
has world shaping properties
and if we return to this idea of
reasonable pluralism that i spoke about
earlier
essentially the core claim is that
unless we coerce people
we can't get to a situation where
everyone agrees
on matters of morality you know we could
flip it around it might be that someone
already has the true moral theory out
there in the world today
and that we're the people who refuse to
accept it for different reasons
i think the question then is how do we
believe other people should be treated
by the possessor of the theory or how do
we believe that person should treat us
now one view that i guess in political
philosophy
is often attributed to jean-jacques
rousseau if you have this really good
theory you're justified in coercing
other people
to live by it he says that people should
be forced to be free
when they're not willing to accept the
truth of the theory
of course it's something that has come
in for fierce criticism
i mean my own perspective is that
actually we need to
try and minimize this challenge of value
and position for powerful technologies
because it becomes a form of domination
so the question is how can we solve the
value alignment problem
in a way that avoids this challenge of
domination
and in that regard we really do need
tools from political philosophy
which is particularly within the liberal
tradition has tried to answer this
question of how can we all live together
on reasonable terms that preserve
everyone's capacity to flourish despite
the fact that we have variation and what
we ultimately believe to be just
true and right so
to bring things a bit back to
where we're at today and how things are
actually going to start changing in the
real world as we move forward
what do you view as the kinds of systems
that
would be and are subject to something
like
an alignment procedure does this start
with
systems that we currently have today
does it start with systems
soon in the future should it have been
done with systems that we already have
today but we failed to do so
what is your perspective on that to my
mind
the challenge of value alignment is one
that exists for the vast majority
if not all technologies and it's one
that's becoming more pronounced as these
technologies
demonstrate higher levels of complexity
and autonomy
so for example i believe that many
existing machine learning systems
encounter this challenge quite
forcefully and that we can ask
meaningful questions about it
so i think in previous discussion we may
have had this example of a
recommendation system
come to light and you know even if we
think of something that seems really
quite physique so
say a recommendation system for what
films to watch
or what content to be provided to you i
think the value
alignment question actually looms large
because it could be designed to do very
different things
on the one hand we might have a
recommendation system that's geared
around your current first order
preferences
so it might continuously give you really
stimulating
really fun low quality content that kind
of keeps you hooked to the system and
with
a high level subjective well-being but
perhaps
something that isn't optimal in other
regards then we can think about other
possible
goals for enlightenment so we might say
that actually these systems should be
built to serve your second-order desires
those are desires that in philosophy
we'd say that people reflectively
endorse their desires about the person
you want to be so if we were to build
recommendation system with that goal in
mind it might be that instead of
watching this kind of cheap and cheerful
content i decide that i'd actually like
to be quite a high brow person so
it starts kind of passively providing me
with more art house recommendations
but even that doesn't cop out the
options it might be that the system
shouldn't really be just trying to
satisfy my preferences
that it should actually be trying to
steer me in the direction of knowledge
and things that are in my interest to
know
so it might try and give me new skills
that i need to acquire
i might try and recommend i don't know
cooking or self-improvement programs
that would be a system that was i guess
geared to my own interest
but even that again doesn't give us a
complete portfolio of options maybe what
we want
is a morally aligned system that
actually enhances our capacity for moral
decision making
and then perhaps that would lead us
somewhere completely different so
instead of giving us this content that
we want it might lead us to content that
leads us to
engage with challenging moral questions
such as factory farming
or climate change so value alignment
kind of arises
quite early on this is of course with
the assumption that the recommendation
system is geared to promote
your interest or well-being or
preference or moral sensibility
there's also the question of whether
it's really promoting
your goals and aspirations or someone
else's
and in science and technology studies
you know there is a big area of value
sensitive design which essentially says
that we need to consult people
and have these almost like democratic
discussions early on
about the kind of values we want to
embody in systems
and then we design with that goal in
mind
so recommendation systems are one thing
of course if we look at public
institutions say
a criminal justice system there we have
a lot of public thought and discussion
about the values that would make a
system like that fair
and the challenge then is to work out
whether there is a technical
approximation of these values
that satisfactory realizes them in a way
that conduces to some vision of the
public good
so in sum i think that value alignment
challenges
exist everywhere and then they become
more pronounced
when these technologies become more
autonomous and more powerful so as they
have more profound
effects on our lives the burden of
justification
in terms of the moral standards that are
being met become more exacting
and the kind of justification we can
give for the design of a technology
becomes more important
i guess to bring this back to things
that exist today
something like youtube or facebook is a
very rudimentary
initial kind of very basic first order
preference
satisfier i mean imagine all of the
human life years that have been
wasted mindlessly consuming content
that's
not actually good for us whereas imagine
i guess some kind of enlightened
version of youtube where it knows enough
about what
is good and yourself and what you would
reflectively
and ideally endorse and the kind of
person that you wish you could be
and that you would be only if you knew
better and how to get there
so the is between that second kind of
system in the first system where
one is just giving you all the best cat
videos in the world
and the second one is turning you into
the person that you always wish you
could have been i think this clearly
demonstrates
that even for systems that seem mundane
that they could be
serving us in much deeper ways and at
much deeper levels and that even when
they superficially
serve us they may be doing harm yeah i
think that's a really profound
observation i mean when we really
look at the full scope of value or the
full picture of the kinds of values
we could seek to realize when designing
technologies and incorporating them into
our lives
often there's a radically expansive
picture that emerges
and this touches upon a kind of
taxonomic distinction
that i introduce in the paper between
minimalist and maximalist conceptions of
value alignment
so when we think about ai alignment
questions
the minimalist says we have to avoid
very bad outcomes so it's important to
build safe systems
and then we just need them to reside
within some space of
value that isn't extremely negative and
could take a number of different
constellations whereas the maximus says
well
let's actually try and design the very
best version
of these technologies from a moral point
of view from a human point of view
and they say that even if we design safe
technologies we could
still be leaving a lot of value out
there on the table
so a technology could be safe but still
not that
good for you or that good for the world
and let's aim to populate that space
with more positive and richer visions of
the future
and then try to realize those through
the technologies that we're building
as we want to realize richer visions of
human flourishing
it becomes more important that it isn't
just a personal goal or vision
but it's one that is collectively
endorsed has been reflected upon
and is justifiable from a variety of
different points of view
right and i guess it's just also
interesting and valuable to reflect
briefly on how
there is already in each society a place
where we draw the line at value and
position
and we have these principles which we've
agreed upon broadly
but we're not gonna let ted bundy do
what ted buddy wants to do
that's exactly right so we have hard
constraints some of which are kind of
set in law
and clearly those are constraints that
of these are just laws that
the ai systems need to respect there's
also
a huge possible space of better outcomes
that are left open once we
look at where more strains are placed
and where they reside
i think that the ted bundy example is
interesting because it also shows that
we need to
discount the preferences and desires of
certain people
one vision of ai alignment says that
it's basically a global preference
aggregation system that we need but in
reality there's a lot of preferences
that just shouldn't be counted
in the first place because they're
unethical
or they're misinformed so again that
kind of to my mind pushes us in this
direction of a conversation about value
itself
and once we know what the principled
basis for alignment is
we can then adjudicate properly cases
like that
and work out what a kind of valid input
for an aligned system is
and what things we need to discount if
we want to realize good moral outcomes
i'm not going to try and pin you down
too hard on that because there's the
tension here of course between the
importance of liberalism not
coercing value judgments on anyone but
then also being like well we actually
have to do it in some places
and that line is a scary one to move in
either direction
so i want to explore more now the
different understandings
of what it is that we're trying to align
ai systems to
so broadly people and i use a lot of
different words here without perhaps
being super specific
about what we mean people talk about
values and intentions and
idealized preferences and things of this
nature so
can you be a little bit more specific
here about what you
take to be the goal of ai alignment
the goal of it being what is it that
we're trying to align systems to
yeah absolutely so we've touched upon
some of these questions already
tacitly in the preceding discussion of
course in the paper i argue that when we
talk about value alignment this idea of
value is often a placeholder for quite
different ideas as you said
and i actually present a taxonomy of
options that i can
take us through in a fairly thrifty way
so i think the starting point for
creating aligned
ai systems is this idea that we want ai
that's able to follow our instructions
but that has a number of shortcomings
which stuart russ and others have
documented
which tend to center around this
challenge of excessive literalism
so if an ai system literally does what
we ask it to
without an understanding of context side
constraints and nuance
often this will lead to problematic
outcomes with the story of king midas
being the classic cautionary tale
you know wishing that everything he
touched turns to gold everything turns
to gold then you have a disaster of one
kind or another
so of course instructions are not
sufficient what you really want is ai
that's aligned with the underlying
intention so i think that often
in the podcast people have talked about
intention alignment as an important goal
of
ai systems and i think that
it's precisely right to dedicate a lot
of technical effort to close the gap
between a kind of idiot
is actually sufficient to get us to the
really good outcomes the kind of
maximalist outcomes that i'm talking
about
and i think that there's a number of
reasons why that might not be the case
so of course to start with just because
an ai can follow
an intention doesn't say anything about
the quality of the intention that's
being followed
we can form intentions on an individual
collective basis
to do all kinds of things some of which
might be incredibly foolish
or malicious some of which might be
self-harming some of which might be
unethical and we've got to ask this
question of whether
we want ai to follow us down that path
when we come up with schemes of that
kind and there's various ways we might
try to address
those bundle of problems i think
intentions are also problematic
from a kind of technical and
phenomenological
perspective because they tend to be
incomplete
so if we look at what an intention is
it's roughly speaking
a kind of partially filled out plan of
action
that commits us to some end and if we
imagine that ai systems are very
powerful
they may encounter situations or
dilemmas or option sets
that are in this space of uncertainty
where it's just
not clear what the original intention
was
and they might need to make the right
kind of decision by default so they
might need some intuitive understanding
of what the right thing to do is so
my intuition is that we do want ai
systems that have
some kind of richer understanding of the
goals that we would want to realize and
whole
so i think that we do need to look at
other options it is also possible that
we would form the intention for the ai
to do something
that explicitly requires an
understanding of morality so we may
ask it to do things you know like
promote the greatest good
in a way that is fundamentally ethical
then it needs to step into this other
terrain of
understanding preferences interests and
values
i think we need to explore that terrain
for one reason or another
of course one thing that people talk
about is this kind of learning from
revealed preferences so perhaps
in addition to the things that we
directly communicate the ai could
observe our behavior and make inferences
about what we want that help
fill in the gap so maybe it could watch
you in your
public life hopefully not private life
and make these inferences that actually
it should create this very good thing
so that takes into the domain of trying
to learn from things that it observes
but i think that preferences are also
quite a worrying data point for ai
alignment at least revealed preferences
because they contain many of the same
weaknesses and shortcomings
that we can ascribe to individual
intentions what is a revealed intention
again
sorry revealed preferences are
preferences that
are revealed through your behavior so i
observe you doing a or b
and from that choice i conclude that you
have a deeper preference for the thing
that you choose
and the question is if we just watch
people can we learn all the background
information we need
to create ethical outcomes yeah
absolutely not
yeah exactly as your ted bundy example
nicely illustrated
not only is it very hard to actually get
useful information from observing people
about what they want but what they want
can often be the wrong kind of thing
for them or for other people yeah i have
to hire
people to spend some hours with me every
week
to tell me from the outside how i may be
acting in ways that are misinformed or
self-harming so instead of revealed
preferences we need
something like rational or informed
preferences which is something you get
through
therapy or counseling or something like
that well that's an interesting
perspective
i guess there's a lot of different
theories about how we get to ideal
preferences
but the idea is that we don't want to
just respond to what people are in
practice doing we want to
give them the sort of thing that they
would aspire to if they were rational
and informed at the very least so not
things that are just a result of
mistaken reasoning or poor quality
information
and then this very interesting like
philosophical psychological
question about what the content of those
ideal preferences
are and particularly what happens when
you think about people being
properly rational so to return to
david hume who often know that his or
distinction is attributed to
he has the conjecture that someone can
be fully fully informed and rational
and still desire pretty much anything at
the end of the day
that they could want something hugely
destructive for themselves or other
people
of course uh kantians and in fact a lot
of moral philosophers believe that
rationality is not just
a process of joining up beliefs and
value statements in a certain fashion
but it also encompasses a substantive
capacity to evaluate
ends so obviously kantians have a theory
about
rationality ultimately requiring you to
reflect
on your ends and ask if they're
universalized in a positive way
but the thing is that's highly highly
contested so
i think ultimately if we say we want to
align ai with people's ideal and
rational preferences
it leads us into this question of what
rationality really means
and we don't necessarily get the kind of
answers that we want to get
to yeah that's a really interesting and
important thing i've never actually
considered that for example someone who
might be a moral anti-realist
would probably be more
partial to the view that rationality is
just about
linking up beliefs and epistemics and
decision theory
with goals and goals are something that
you're just given and
embedded with and that there isn't some
correct evaluative procedure for
analyzing goals
beyond whatever meta preferences you've
already inherited whereas a realist
might say something like
the other view where rationality is
about beliefs and
ends but also about perhaps more
concrete
standard or method for evaluating which
ends are good ends is that the way you
view it
yeah i think that's a very nice summary
the people who believe in substantive
rationality tend to be
people with a more realistic moral
disposition
if you're profoundly anti-realist you
basically think that you have to stop
talking in the currency of reasons so
you can't tell people they have a reason
not to act in a kind of unpleasant way
to each other or even to do really
heinous things you have to say to them
something different like wouldn't it be
nice if we could realize this positive
state of affairs
and i think ultimately we can get to
views about value alignment that satisfy
these two different groups we can create
aspirations that are well reasoned from
different points of view
and also create scenarios that meet the
kind of wouldn't it be nice criterion
but i think it isn't going to happen if
we just double down on this question of
whether rationality
ultimately leads to a single set of ends
or a plurality of ends
no consensus whatsoever all right that's
quite interesting
not only do we have difficult and
interesting philosophical ground
in ethics but also in rationality
and how these are interrelated
absolutely i think they're very closely
related
so actually the problems we encounter in
one domain we also encounter and the
other
and i'd say in my kind of lexicon they
all fall within
this question of practical rationality
and practical reason
so that's deliberating about what we
ought to do either because of explicitly
moral considerations or
a variety of other things that we factor
up in judgments of that kind
all right two more on our list here to
hit our
interests and values so i think there
are one or two more things we could say
about that
so if we think that one of the
challenges with ideal preferences is
that they lead us into this
heavily contested space about what
rationality truly requires
we might think that a conception of
human interest does significantly better
so if we think about ai being designed
to promote
human interests or well-being or
flourishing
i would suggest that as a matter of
empirical fact there's significantly
less disagreement
about what that entails so if we look at
say
the capability-based approach that
amartya sen and martha nussbaum have
developed
it essentially says that there's a
number of key goods and aspects of human
flourishing
that the vast majority of people believe
can juice to a good life
and that actually has some intercultural
value and affirmation
so if we designed ai that bear in mind
this goal of enhancing
general human capabilities so human
freedom physical security emotional
security
capacity that looks like an ai that
is both roughly speaking getting us into
the space of something that looks
like it's unlocking real value and also
isn't bogged down in a
huge amount of metaphysical contention
i suggest that aligning ai with human
interest or well-being is
a good proximate goal when it comes to
value alignment
but even then i think that there's some
important things that are missing
and that can only actually be captured
if we return to the idea of value itself
so by this point it looks like we have
almost arrived at a kind of utilitarian
ai
via the back door i mean of course
utility is a subject
of mental state isn't necessarily the
same as someone's interest or
their capacity to lead a flourishing
life but it looks like we have an ai
that's geared around
optimizing some notion of human
well-being
and the question is what might be
missing there or what might go wrong
and i think there are some things that
that view of value alignment still
struggles to factor in
the welfare of non-human animals is
something that's missing from this
well-being centered perspective on
alignment
that's why we might just want to make it
well-being for sentient creatures
exactly and i believe that this is a
valuable enterprise so we can expand the
circle so we say it's the well-being of
sentient creatures
and then we have the question about what
about future
generations you know does their
well-being count
and we might think that it does you know
if we follow
toby ord or in fact most conventional
thinking we do think that the welfare of
future generations has
intrinsic value so we might say well we
want to promote well-being
of sentient creatures over time with
some appropriate waiting to account for
time
and that's actually starting to take us
into a richer space of value so we have
well-being but we also have a theory
about how to do
inter-temporal comparisons we might also
think that it matters how well-being or
welfare is distributed that it isn't
just a maximization question
but that we also have to be interested
in equity or distribution
because we think it's intrinsically
important so we might think it has to be
done in a manner that's fair
additionally we might think that things
like the natural world have intrinsic
value that we want to factor in
and so the point which will almost be
familiar now from our earlier discussion
is you actually have to get that
question of what
values do we want to align the system
with because values and the principles
that derive with them can capture
everything
that is seemingly important right and so
for example within the effect of
altruism community and within moral
philosophy recently
the way in which moral progress has been
made is
in so far that de-biasing human
moral thought and ethics from spatial
and temporal bias
so peter singer has the children
drowning in a shallow pond argument
it just illustrates how there are people
dying and children dying all over the
world
in situations which we could cheaply
intervene to save them
as if they were drowning in a shallow
pond and you only need take a couple
steps and just pull them out
except we don't and we don't because
they're far away
and i would like to say essentially
everyone finds this compelling that
where you are in space doesn't matter
how much you're suffering that if you
are suffering then
all else being equal we should intervene
to alleviate that suffering
when it's reasonable to do so so space
doesn't matter for ethics
likewise i hope and i think that we're
moving in the right direction
if time also doesn't matter while also
being mindful we also have to introduce
things like uncertainty like we don't
know what the future will be like
but this principle about caring about
the well-being of sentient creatures
in general i think is essential in core
i think
to whatever list of principles we'll
want
for bridging the is out distinction
because
it takes away spatial bias where you are
in space doesn't matter just matters
that you're essentially being it doesn't
matter when you are
as a sentient being it also doesn't
matter what kind of sentient being you
are because the thing we care about is
sentience
so then the moral circle has expanded
across species
it's expanded across time it's expanded
across
space it includes aliens and all
possible minds
that we could encounter now or in the
future
we have to get that one in i think for
making a good future with ai
that's a a picture that i strongly
identify with
on a personal level this idea of the
expanding moral circle of sensibilities
and i think you know from a substantive
point of view
you're probably right that that is a lot
of the content that we would want to put
into an aligned ai system
i think that one interesting thing to
note is that a lot of these views are
actually
empirically fairly controversial
so if we look at the interesting study
the moral machine experiment
where i believe like several million
people ultimately played this experiment
online where they decided which
trade-offs an
av an autonomous vehicle should make in
different situations
so whether it should crash into one
person or five people a rich person or a
poor person
pretty much everyone agreed that it
should kill fewer people when that was
on the table
but i believe that in many parts of the
world there was also belief
that the lives of affluent people
mattered more than the lives of those in
poverty
and so if you're just a reason from
their first order moral beliefs you
would
bake that bias into an ai system
that seems deeply problematic and i
think it actually
puts pressure on this question which is
like we've already said we don't want to
just align ai
with existing moral preferences we've
also said that we can't just declare
a moral theory to be true and impose it
on other people
so are there other options which move us
in the direction
of these kind of moral beliefs that seem
to be deeply justified
but also avoid the challenge of value
imposition
and how far do they get if we try to
move forward not just as individuals
like examining the kind of expanding
moral circle
but as a community that's trying to
progressively endogenize these ideas
and come up with more principles that we
can all live by
we might not get as far if we were going
at it alone
but i think that there are some
solutions that are kind of in that space
and those are the ones i'm interested in
exploring i mean
common sense morality understood as the
conventional morality that most people
endorse
i would say is deeply flawed in in a
number of regards including
with regards to you know global poverty
and things of that nature
and that's really unfortunate given that
we probably also don't want to force
people
to live by more enlightened beliefs
which they
don't endorse or can't understand so i
think that the interesting question is
how do we
meet this demand for a respect for
pluralism
and also avoid getting stuck in the
morass of common sense morality which
has these prejudicial beliefs
that will probably with the passage of
time come to be regarded quite
unfortunately by future generations
and i think that taking this demand for
non-domination or democratic support
seriously
means not just running far into the
future or in a way that we believe
represents the future
but also doing a lot of other things
trying to have a democratic discourse
where we
use these reasons to justify certain
policies that then
other people reflectively endorse and we
move the project forwards
in a way that meets both disidorata and
in this paper i try to map out different
solutions
that both meet this criterion of
respecting people's pluralistic beliefs
while also moving us genuinely morally
aligned
outcomes so now the last question that i
want to ask you here then
on the goal of ai alignment is do you
view
a needs-based conception of
human well-being as a subcategory
of interest based value alignment
people have come up with different
conceptions of human needs
people are generally familiar with
maslow's hierarchy of needs
and i mean as you go up the hierarchy it
will become more and more contentious
but everyone needs food and shelter and
safety
and then you need community and meaning
and
spirituality and things of that nature
so
how do you view or fit a needs-based
conception and because
some needs are obviously undeniable
relative to others broadly speaking
a need space conception of well-being is
in that space
we already touched upon so the
capabilities based approach and the
needs based approach are quite similar
but i think that what you're saying
about needs
potentially points to a solution to this
kind of dilemma that we've been talking
about
if we're going to ask this question of
what does it mean to create principles
for ai alignment that treat people
fairly despite their different views
one approach we might take is to look
for commonalities
that also seem to have moral robustness
or substance to them so within the
parlance of political philosophy we'd
call this an
overlapping consensus approach to the
problem
of political and moral decision-making i
think that that's a project that's
well worth countenancing so we might say
there's a plurality of global beliefs
and cultures
what is it that these cultures coalesce
around and i think that it's likely to
be something along the lines
of the argument that you just put
forward that people
are vulnerable in virtue of how we're
constituted that we have a kind of
fragility
and that we need protection both against
the environment and against certain
forms of harm
particularly state-based violence and
that this is a kind of moral bedrock or
what the philosopher henry shu calls a
moral minimum
that receives intercultural endorsement
so actually the idea of human needs is
very very closely tied to the idea of
human rights so
the idea is that the need is fundamental
and in virtue of what your moral
standing the normative claim
and your need the empirical claim you
have a right
to enjoy a certain good and to be secure
in the knowledge that you'll enjoy that
thing
so i think the idea of building a kind
of human rights-based ai
that's based upon this intercultural
consensus is pretty promising
in some regards human rights as they've
been historically thought about
are not super easy to turn into a theory
of
ai alignment because they are
historically thought of as guarantees
that states have to give
their citizens in order to be legitimate
and it isn't entirely clear what it
means to have a human rights-based
technology
but i think that this is a really
productive area to work in
and i would definitely like to try and
populate that ground
you might also think that the consensus
or the emerging consensus around
values that need to be built into ai
systems such as fairness and
explainability
potentially pretends that the emergence
of this kind of intercultural consensus
although i guess at that point we have
to be really mindful
of the voices that are at the table and
who's had an opportunity to speak
so although there does appear to be some
convergence around principles of
beneficence and things like that
it's also true that this isn't a global
conversation in which everyone is
represented
and it would be easy to prematurely rush
to the conclusion
that we know what values to pursue when
we're really just
reiterating some kind of very heavily
western-centric
affluent view of ethics that doesn't
have real intercultural
democratic viability all right
now it's also interesting and important
to consider here
the differences in importance of single
agent and multi-agent
alignment scenarios for example you can
imagine
entertaining the question of how is it
that i would build
a system that would be able to align
with my values one agent
being the ai system and one person and
how is it that i get the system to do
what i want it to do
and then the multi-agent alignment
scenario considers
how do i get one agent to align and
serve to many different people's
interests and well-being and desires and
preferences and needs
and then also how do we get systems to
act and behave
when there are many other systems trying
to
serve and align to many other different
people's needs
and how is it that all these systems may
or may not collaborate with all of the
other ai systems and may or may not
collaborate with
all of the other human beings when
all the human beings may have
conflicting preferences and needs
how is it that we do for example
inter-theoretic comparisons
of value and needs so what's the
difference in
importance between single-agent and
multi-agent alignment scenarios
i think that the difference is best
understood
in terms of how expansive the goal of
alignment has to be
so if we're just thinking about a single
person in a single agent
it's okay to approach the value
alignment challenge
through a slightly solipsistic lens in
fact you know if it was just one person
and one agent it's not clear that
morality really enters the picture
unless there are other people other
sentient creatures who our action can
affect
so with one person one agent the
challenge is primarily correlation
with the person's desires aims
intentions potentially there is still a
question of whether the ai
serves their interest rather than you
know these more volitional states that
come to mind
when we think about situations in which
like many people are affected
then it becomes kind of remiss not to
think about
interpersonal comparisons and the kind
of richer conceptions that we've been
talking about
now i mentioned earlier that there is a
view that there will always be
a human body that synthesizes
preferences and provides more
instructions for ai
we can imagine democratic approaches to
value alignment where
human beings uh assemble maybe in
national parliaments maybe in global
forum
and legislate principles today is then
designed in accordance with
i think that's actually a very promising
approach you know you would want it to
be informed by
moral reflection and people offering
different kinds of moral reasons that
support one approach
rather than the other but that seems to
be important for multi-person situations
and it's probably actually a necessary
condition for powerful forms of ai
because you know when ai has a profound
effect on people's lives these questions
of legitimacy also start to emerge so
not only is it doing the right thing but
is it doing the sort of thing that
people would consent to and is it doing
the sort of thing that people
actually have consented to and i think
that when ai is used in certain forum
then these questions of legitimacy
come to the top there's a bundle of
different things in that space
yeah i mean it seems like a really
really hard problem
when you talk about creating some kind
of national body
and i think you said international fora
do you wonder that some of these
vehicles might be overly idealistic
given what may happen in the world where
there's
national actors competing and capitalism
driving things forward
relentlessly and this problem of
multi-agent alignment seems
very important and difficult and that
there are
forces pushing things such that it's
less likely that it happens
when you talk about multi-agent
alignment are you
talking about the alignment of an
ecosystem that contains multiple
ai agents or are you talking about how
we align an ai agent with the interests
and ideas of multiple parties
so many humans for example i'm
interested and
curious about both i think there's
different considerations that arise for
both
sets of questions but there are also
some things that we can speak to that
pertain to both of them
do they both count as multi-agent
alignment scenarios and your
understanding of the definition
from a technical point of view it makes
perfect sense to describe them both in
that way
i guess when i've been thinking about it
curiously i've been thinking of
multi-agent alignment as an agent that
has multiple parties that it wants to
satisfy
but when we look at machine learning
research multi-agent usually means
many ai agents running around in a
single environment
so i don't see any kind of
language-based reason to offer one
rather than the other
with regards to this question of
idealization and real world practice
i think it's an extremely interesting
area and the thing i would say is this
is almost
one of those occasions where potentially
the is or distinction comes to our
rescue
so the question is does the fact that
the real world is a difficult place
affected by divergent interests mean
that we should level down our ideals
and conceptions about what really good
and valuable ai would look like
and there are some people who have what
we term practice dependent views of
ethics who say
absolutely we should do we should adjust
our conception of what the ideal is
but as you'll probably be able to tell
by now i hold a kind of different
perspective
in general i don't think it is
problematic to have
big ideals and rich visions of how value
can be unlocked
and that partly ties into the reasons
that we spoke about for thinking that
the technical and the normative are
interconnected
so if we preemptively level down we'll
probably design systems
that are less good than they could be
and when we think about a design process
spanning decades
we really want that kind of ultimate
goal the shining star of alignment to be
something that's quite bright and can
steer our efforts towards it
if anything i would be slightly worried
that because these
human parliaments and international
institutions are so driven by real world
politics that they might not give
us the kind of most fully actualized
set of ideal aspirations to aim for and
that's why
philosophers like of course john rules
actually propose that we need to think
about these questions from a
hypothetical point of view
so we need to ask what would we choose
if we weren't living in a world where we
knew how to leverage our own interests
and that's how we identify the real
ideal that is acceptable to people
regardless of where they're located
and also can then be used to steer
non-ideal theory or the kind of actual
practice in the right direction
so if we have an organization that is
trying its best to create
aligned and beneficial agi systems
reasoning about what principles we
should embed in it from behind rawls's
veil of ignorance you're saying
would have hopefully the same practical
implications as
if we had a functioning international
body
for coming up with those principles in
the first place possibly i mean i'd like
to think that ideal deliberation
would lead them in the direction of
impartial principles for ai
it's not clear whether that is the case
i mean it seems that at its very best
international politics has led us in the
direction of a kind of human rights
doctrine
that both accords individuals protection
regardless of where they live
and defends the strong claim that they
have a right to subsistence and other
forms of flourishing
if we use the veil of ignorance
experiment i think for ai it might even
give us more than that
even if a real world parliament never
got there
for those of you who are not familiar
with this the philosopher john rule says
that when it comes to choosing
principles for a just society what we
need to do
is create a situation in which people
don't know where they are in that
society
or what their particular interest is so
they have to imagine that they're from
behind a veil of ignorance
they select principles for that society
that they think will be fair
regardless of where they end up and then
having done that process and identified
principles of justice for the society
he actually holds out the aspiration
that people will reflectively endorse
them
even once the veil has been removed so
they'll say yes
in that situation i was reasoning in a
fair way that was non-prejudicial
and these uh principles that i
identified there continued to have value
in the real world and we can say
what would happen if people were asked
to choose principles for artificial
intelligence from behind
a veil of ignorance where they didn't
know whether they were going to be rich
or poor
christian utilitarian kantian or
something else
and i think there some of the kind of
common sense material would be surface
so people would obviously
want to build safe ai systems i imagine
that this idea of
preserving human autonomy and control
would also
register but for some forms of ai also i
think distributive considerations would
come into play
so they might start to think about how
the benefits and burdens
of these technologies are distributed
and how those questions play out on a
global basis they might say
that ultimately a value-aligned ai
is one that has fair distributive
impacts on a global basis
and if you follow rules that it works to
the advantage of the least well-off
people
that's a very substantive conception of
value alignment
which may or may not be the final
outcome of ideal
international deliberation maybe the
international community will get to
global justice eventually
or maybe it's just too thoroughly
affected by
nationalist interests and other kinds of
what to my mind the kind of
distortionary effects that mean that it
doesn't quite get there
but i think that this is definitely the
space that we want the debate to be
taking place in and that actually
there has been real progress in
identifying collectively endorsed
principles for ai
that give me hope for the future not
only that we'll get good ideals but that
people might agree to them
and that they might get democratic
endorsement and that they might be
actionable
and the sort of thing that can guide
real-world ai design
can you add a little bit more clarity on
the
philosophical questions and issues which
single and multi-age and alignment
scenarios supervene on how do you do
inter-theoretic comparisons of value
if people disagree on normative or
meta-ethical beliefs or if people
disagree on
foundational axiomatic principles for
bridging the is ought gap
how is it the systems deal with that
kind of disagreement
i'm hopeful that the three pictures that
i outlined so far
of the overlapping consensus between
different moral beliefs
of democratic debate over a constitution
for ai
and of selection principles from behind
a veil of ignorance
are all approaches that carry some
traction in that regard so
they try to take seriously the fact of
real-world pluralism
but they also through different
processes
tend to tap towards principles that are
compatible with a variety of different
perspectives
although i would say i do feel like
there's a question about this
multi-agent thing that
may still not be completely clear in my
mind and it may come back to those
earlier questions about definition
so in a one-person one-agent scenario
you don't have this question of what to
do with pluralism
and you can probably go for a more
simple one-shot solution which is align
it with the person's interest
beliefs moral beliefs intentions or
something like that
but if you're interested in this
question of real world politics
for real-world ai systems where a
plurality of people are affected
we definitely need these other kinds of
principles that have a much richer
set of properties and endorsements all
right
there's rawls available of ignorance
there's
principle of non-domination and then
there's the democratic process
non-domination is a criterion that any
scheme for multi-agent value alignment
needs to meet
and then we can ask the question what
sort of scheme
would meet this requirement of
non-domination
and there we have the overlapping
consensus with human rights
we have a scheme of democratic debate
leading to principles for an ai
constitution
and we have the veil of ignorance as all
ideas that we basically find within
political theory that could help us meet
that condition
all right so we've spoken at some length
then about principles and identifying
principles this goes back to our
conversation about the
zot distinction and these are principles
that we need to identify for setting up
an
ethical alignment procedure you
mentioned this earlier when we were
talking about this this distinction
between
the one true moral theory approach to ai
alignment
in contrast to coming up with a
procedure for ai alignment that would be
broadly endorsed by many people and
would
respect the principle of non-domination
and would take into account pluralism
can you unpack this distinction more and
and the importance of it
yeah absolutely so i think the the kind
of true moral theory
approach although it is a kind of
stylized idea of what an approach to
value of alignment might look like is
the sort of thing that could be
undertaken just by a single person who
is designing the technology or a small
group of people
perhaps moral philosophers who think
that they have really great expertise in
this area
and then they identify the chosen
principle and
run with it the big claim is that that
isn't really a satisfactory
way to think about design and values in
a pluralistic world where many people
will be affected and of course many
people have gone off on that kind of
enterprise have made serious
mistakes that were very costly for
humanity
and for people who are affected by their
actions so the political approach to
value alignment
takes a fundamentally different
perspective and says it isn't really
about one person or one group
running ahead and thinking that they've
done all the hard work it's about
working out what we can all agree upon
that looks like a reasonable set of
moral principles or coordinates to build
powerful technologies around
and then once we have this process in
place that outputs the right kind of
agreement
then the task is given back to
technologists and they said these
are the kind of parameters that our fair
process of deliberation has outputted
and this is what we have the authority
to encode in machines
whether it's like human rights or
conception of justice
or some other widely agreed upon values
there are principles that you're really
interested in satisfying like respecting
pluralism
and respecting a principle of
non-domination
and the one true moral theory approach
risks violating those other principles
are you
not taking a stance on whether
there is a one true moral theory you're
just willing to
set that question aside and say because
it's so essential to a thriving
civilization
that we don't do moral
imposition on one another the coming up
with a broadly endorsed theory
is just absolutely the way to go whether
or not there is such a thing as a one
true moral theory
does that capture your view yeah so to
some extent
i'm trying to make an argument that will
look like something we should affirm
regardless of the meta-ethical stance
that we wish to take of course there are
some views
about morality that actually say that
non-domination is a really important
principle
or that human rights are fundamental so
someone might look at these proposals
and from the comprehensive moral
perspective they would say
this is actually the morally best way to
do value alignment
and it involves dialogue discussion
mutual understanding and agreement
however you don't need to believe that
in order to think that this is a good
way to go
if you look at the writing of someone
like joshua green
he says that this problem we encounter
called the tragedy of common sense
morality
a lot of people have fairly decent moral
beliefs
but when they differ it ends up in
violence and they end up fighting
and you have a hugely negative more
externality that arises just because
people
weren't able to enter this other mode of
theorizing
where they said look we're part of a
collective project let's
agree to some higher level terms that we
can all live by
so from that point of view it looks
prudent to think about value alignment
as a pluralistic enterprise
that's an approach that many people have
taken with regards to the justification
of the institution of the state and the
things that we believe it should protect
and affirm and uphold
and then as i alluded to earlier i think
that actually
even for some of these anti-realists
this idea of inclusive deliberation
and even the idea of human rights looked
like quite good candidates for the kind
of wouldn't it be nice criterion
so to return to richard rorty who's kind
of the arch moral skeptic
he does ultimately really want us to
live in a world with human rights
he just doesn't think he has a really
good meta-ethical foundation to resist
on
but in practice he would take that
vision forward
i believe and tried to persuade other
people that it was the way to go
by telling them good stories and saying
well look this is the world with human
rights and open-ended deliberation
and this is the world where one person
decided what to do
wouldn't it be nice in that better world
so i'm hopeful that this kind of
political ballpark has this kind of
rich applicability and appeal regardless
of whether
people are starting out in one place or
the other that makes sense
so then another aspect of this is
in the absence of a moral agreement or
when there is moral disagreement
is there a fair way to decide what
principles ai
should align with for example i can
imagine
religious fundamentalists at core being
antithetical to
the project of aligning ai systems which
would eventually lead to something like
playing god
and just be like well this is just not a
project that we should even do
so that's an interesting question and
you may actually be putting pressure on
my preceding argument
i think that it is certainly the case
that you can't get everyone
to agree on a set of global principles
for ai
because some people hold very very
extreme beliefs that are exclusionary
and don't tend to the possibility of
compromise
typically people who have a
fundamentalist orientation of one kind
or another
and so even if we get the pluralistic
project off the ground
it may be the case that we have to in my
language
impose our values on those people and
that in a sense they are dominated and
that leads to the difficult question
why is it permissible to impose beliefs
upon those people but not the people who
don't hold
views it's a fundamentally difficult
question
because what it tends to point to is the
idea that beneath this talk about
pluralism there is
actually a value claim which is that you
are entitled to non-domination so long
as you're prepared
not to dominate other people and to
accept that there is a moral equality
that means that we need to cooperate and
cohabit in a world together
and that does look like a kind of deep
deep moral claim
that you might need to substantively
assert i'm not
entirely sure i think that's one that we
can save for further investigation
but it's certainly something that people
have said in the context of these
debates that at the deepest level
you can't escape making some kind of
moral claim
because of these cases yeah
this is reminding me of the paradox of
tolerance by carl popper who talks about
free speech ends when you yell the
theater's on fire
and in some sense are then imposing harm
on other people
and that we're tolerant of people within
society
except for those who are intolerant of
others
and to some extent that's a paradox so
similarly we may respect and endorse
a principle of non-domination or
non-subjugation
but that ends when there are people who
are dominating or subjugating
and the core of that is maybe getting
back again
to some kind of principle of non-harm
related to
the well-being of sentient creatures
yeah i think the the obstacles that
we're discussing now
are very precisely related to that
paradox
of course the boundaries we want to draw
on permissible disagreement
in some sense is quite minimal or
conversely we might think that the wide
affirmation of some aspect of the value
of human rights
is quite a strong basis for moving
forward because it says that all human
life
has value and that everyone is entitled
to basic goods
including goods pertaining to autonomy
so people who reject that really are
pushing back against something that is
widely
and deeply reflectively endorsed by a
large number of people
i also think that with regards to
toleration the anti-realist position
becomes quite hard to figure out or
quite strange
so you have these people who are not
prepared to live in a world where they
respect others and they have this will
to dominate or a fundamentalist
perspective
the anti-realist says well you know
potentially there's this nicer world we
can move towards
the anti-realist doesn't deal in the
currency of moral reasons they don't
really have to worry about it too much
they can just say am we're gonna go in
that direction with everyone else who
agrees with us
and hold the idea that it looks like a
good way to live
so in a way the problem of domination is
much more serious for people who are
moral realists
for the anti-realists it's not actually
a perspective i inhabit in my day-to-day
life so it's hard for me to say
what they would make of it well i guess
just to briefly defend the anti-realist
i imagine that they would say that they
still have reasons for morality
they just don't think that there is an
objective epistemological
methodology for discovering what is true
there aren't facts about morality but
i'm gonna go make the same noises that
you make about morality
like i'm gonna give reasons and
justification and
these are as good as making up empty
screeching noises and blah blahing about
things that don't exist but it's still
motivating to other people right
they still will have reasons and
justification they just don't think it
pertains to truth
and they will use that to navigate the
world and then justify
domination or not that seems possible
but i guess for the anti-realist if they
think we're just
fundamentally expressing pro attitudes
so when i say you know it isn't
justified to dominate others i'm just
saying i don't like it when
this thing happens then we're just
dealing in the currency of likes
and i just don't think you have to be so
worried about
the problem of domination as you are if
you think that this means something more
than someone just expressing
an attitude about what they like or
don't if there aren't real moral reasons
or considerations at stake if it's just
people saying i like this i don't like
this
then you can get on with the enterprise
that you believe
achieves this positive end of course the
unpleasant thing is you kind of are
potentially
giving permission to other people to do
the same or that's a consequence of the
view you hold
and i think that's why a lot of people
want to
rescue the idea of moral justification
as a really meaningful practice
because they're not prepared to say well
everyone gets on with the thing that
they happen to like
and the rest of it is just window
dressing all right
well i'm not sure how much we need to
worry about this now
i think it seems like anti-realist and
realists basically
act the same in the real world maybe
i don't know yeah in reality
anti-realist
tend to act in ways that suggest that on
some level they believe that morality
has more to it than just being a
category error
so let's talk a little bit here more
about
the procedure by which we choose
evaluative models for
deciding which proposed aspects of human
preferences or values are good or bad
for an alignment procedure
we can have a method of evaluating or
deciding
which aspects of human values or
preferences or things that we might want
to bake into
an alignment procedure are good or bad
but you mentioned something like having
a global fora
or having different kinds of governance
institutions or
vehicles by which we might have
conversation to
decide how to come up with an alignment
procedure
that would be endorsed what is the
procedure
to decide what kinds of evaluative
models we will use
to decide what counts as a good
alignment procedure or not
right now this question is being
answered by
a very biased and privileged select
few in the west at ai organizations
and people adjacent to them i think this
question is
absolutely fundamental i believe that
any claim that we have meaningful global
consensus on ai principles
is premature and that it probably does
reflect
biases of the kind you mention i mean
broadly speaking
i think that there's two extremely
important reasons to try and widen this
conversation
the first is that in order to get
a kind of clear well-grounded and
well-sighted vision on what ai should
align with
we definitely need intercultural
perspectives
on the assumption that quote john stuart
mill no one has complete access to the
truth
and people have access to different
parts of it the bigger the conversation
becomes
the more likely it is that we move
towards maximal
value alignment of the kind that
humanity deserves
but potentially more importantly than
that and regardless of the kind of
epistemic consequences of widening the
debate i think that people have a right
to voice
their perspective on topics and
technologies that will affect them
if we think of the purpose of global
conversation partly
as this idea of formulating principles
but also bestowing on them
a certain authority in light of which
we're permitted to build
powerful technologies then you just
can't say that they have the right kind
of
authority and grounding without proper
extensive consultation
and so i would suggest that that's a
very important next step
for people who are working in this space
i'm also hopeful that actually
these different approaches that we've
discussed can potentially be mutually
supporting so
think that there's a good chance that
human rights could serve as
a foundation or a seed for a good strong
intercultural conversation around ai
alignment
and i'm not sure to what extent this
really is the case but it might be that
even
some of these ideas about reasoning
impartially
have currency in a global conversation
and you might find that they're actually
quite challenging
for affluent countries or for
self-interested parties
because it would reveal certain hidden
biases
in the propositions that they've now
made or put forward
okay so related to things that we might
want to do to come up with
the correct procedure for being able to
evaluate what kinds of
alignment procedures are good or bad
what do you view as sufficient for
adequate alignment of systems
so we've talked a little bit about
minimalism versus maximalism
where minimalism is aligning to just
some conception of human values and
maximalism is hitting on some very
idealized and
strong set or form of human values
and this procedure is related at least
in the
i guess existential risk space coming
from people like toby ord and william
mccaskill
they talk about something like a long
reflection
so if i'm asking you about what might be
adequate alignment for systems one
criteria for that might be
meeting basic human needs meeting human
rights and
reducing existential risk further and
further such that it's
very very close to zero and we enter a
period
of existential stability and then
following this existential stability
is proposed something like a long
reflection
where we might more deeply consider
ethics and values and norms before we
set about changing and optimizing all of
the atoms
around us in the galaxy so do you have
a perspective here on this sort of most
high level timeline
of first as we're aligning ai systems
what does it mean for it to be adequate
and then what needs to potentially be
saved for something like
a long reflection and then how something
like a broadly endorsed procedure
versus a one true moral theory approach
would
fit into something like a long
reflection yes a number of thoughts on
this topic
the first pertains to the idea of
existential security
and i guess why it's defined as the kind
of
dominant goal in the short-term
perspective
there may be good reasons for this but i
think
what i would suggest is that obviously
involves trade-offs you know the world
we live in is a very
unideal place one in which we have a
vast quantity
of unnecessary suffering and to my mind
is probably not
even acceptable to say that basically
the goal of building ai is
or that the foremost challenge of
humanity is to focus on this kind of
existential security and extreme
longevity while living so many people to
lead lives that are less than they
could be why do you think that well
because human life matters
if we were to look at where the real
gains in the world are today
i believe it's helping these people who
you know die unnecessarily from
neglected diseases
lack subsistence incomes and things of
that nature
and i believe that has to form part of
the picture of
our ideal trajectory for technological
development
yeah that makes sense to me i'm confused
what you're actually
saying about the existential security
view
as being central if you compare the
suffering
of people that exist today obviously to
the astronomical amount of life that
could be in the future
is that kind of reasoning about the
potential
that doesn't do the work for you for
seeing mitigating existential risk as
the central concern
i'm not entirely sure but what i would
say is that on one reading of the
argument that's being presented
the goal should be to build extremely
safe systems
and not try to intervene in areas about
which there's more substantive
contestation
until there's been a long delay and a
period of reflection
which might mean neglecting some very
morally important and tractable
challenges that the world is facing
at the present moment and i think that
that would be problematic like i'm not
sure why
we can't work towards something that's
more ambitious for example
a human rights respecting ai technology
why would that entail that well so i
mean this is the kind of question about
the proposition that's been put in front
of us essentially if that
isn't the proposition then the long
reflection
isn't leaving like huge amounts to be
deliberated about right because we're
saying in the short term
we're going to tether towards global
security but we're also going to try and
do a lot of other things around which
there's moral uncertainty and
disagreement
for example promote fairer outcomes
mobilize in the direction of respecting
human rights
and i think that once we've moved
towards that conception of value
alignment
it isn't really clear what the substance
of the long reflection is
so do you have an idea of what questions
would remain to be answered
yeah so i guess i feel confused because
reaching
existential security as part of this
initial
alignment procedure doesn't seem to be
in conflict with alleviating the
suffering
of the global poor because i don't think
moral uncertainty extends to
meeting basic human needs or satisfying
basic human rights or
things that are obviously conducive to
the well-being of
sentient creatures so i don't think
poverty gets pushed to the long
reflection i don't think
unnecessary suffering gets pushed to the
long reflection
so then the question you're asking is
what is it that does get pushed to the
long reflection
yes so then what gets pushed to the long
reflection is
is the one true moral theory approach to
alignment
actually correct is there a one true
moral theory or is there not
a one true moral theory are
anti-realists correct
or are realists correct or are they both
wrong in some sense or
something else correct and then
given that the potential answer or
inability to come up with an answer to
that
would change how something like the
cosmic endowment gets optimized
because we're talking about billions
upon billions upon billions upon
billions of years
if we don't go extinct and the universe
is going to evaporate eventually
but until then there's an astronomical
amount of things that could get done
and so the wrong reflection is about
deciding what to actually do with that
and however esoteric it is the proposals
range
from you just have some pluralistic
optimization process
there's no right way you should live
things other than
joy and suffering matter like i don't
know building monuments that calculate
mathematics ever more precisely and if
you want to
carve out a section of the cosmic
endowment for optimizing things that are
other than
conscious states you're free to do that
versus
coming down on something more like a one
true moral theory approach and being
like
the only kinds of things that seem to
matter in this world are the states of
conscious creatures
therefore the future should just be an
endeavor
of optimizing for creating minds that
are evermore enjoying profound states of
spiritual enlightenment and spiritual
bliss and knowledge
the long reflection might even be about
whether or not knowledge matters
for a mind does it really matter that i
am in tune with truth and reality
should we build nothing but experience
machines that cultivate
whatever the most enlightened and
blissful states of experience are
or is that wrong the long reflection to
me seems to be about these sorts of
questions and
if the one true moral theory approach is
correct
or not yeah that makes sense and
my apologies if i didn't understand what
was already taken care of by the
proposal
i think to some extent in that case
we're talking about different action
spaces so
when i look at these questions of ai
alignment i see
very significant value questions already
arising
in terms of how benefits and burdens are
distributed
what fairness means whether ai needs to
be explained born accountable
and things of that nature alongside a
set of very pressing global problems
that would be
really really important to address
so i think my time horizon is definitely
different from this
long reflection one kind of find it
difficult to imagine a world
in which these huge but to some extent
present questions have been addressed
and in which we then turn our attention
to these other things
i guess there's a couple of things that
can be said about it
so i'm not sure if this is meant to be
taken literally but i think the idea of
pressing pause
on technological development while we
work out
a further set of fundamentally important
questions
is probably not feasible so it would be
best to work with
a long-term view that doesn't rest upon
the possibility of that option
and then i think that the other
fundamental question is what is actually
happening
in this long reflection so it can be
described in a variety of different ways
sometimes it sounds like it's a big
philosophical conference that runs for a
very very long time
and the end of it hopefully people kind
of settle these questions and they come
out to the world and they're like wow
this is a really important discovery
i mean if you take seriously the things
we've been talking about today
you still have the question of what you
do with the people who then say actually
i think you're wrong about that
and i think you know in a sense it
recursively pushes us back into the kind
of processes that i've been
talking about when i hear people talk
about the long reflection there does
also sometimes seem to be this idea that
it's a period
in which there's very productive global
conversation
about the kind of norms and directions
that we want humanity to take
and that seems valuable but it doesn't
seem unique to the long reflection like
that would be
incredibly valuable right now so it
doesn't look radically
discontinuous to me on that view
all right because we're talking about
the long-term future here and
i bring it up because it's interesting
and considering what questions can we
just kind of put aside these are
interesting but in the real world they
don't
matter a ton or they don't influence our
decisions but over the very very long
term future they may
matter much more when i think about a
principle
like non-domination it seems like we
care about
this conception of non-imposition and
non-dominance and non-subjugation
for reasons of first of all well-being
and the reason why we care about this
well-being question
is because human beings are
extremely fallible and it seems to me
that the principle of non-domination is
rooted
in the lack of epistemic capacity
for fallible agents like human beings
to promote the well-being of sentient
creatures all around them
but in terms of what is physically
literally possible in the universe
it's possible for someone to know
so much more about the well-being of
conscious creatures than you
and how much happier and how much more
well-being you would be in
if you only idealize in a certain way
that
as we get deeper and deeper into the
future i have more and more skepticism
about this principle of non-domination
and knob subjugation
it seems very useful important and
exactly like the thing that we need
right now
but as we long reflect further and
further and
say really smart really idealized beings
develop more and more epidemic clarity
on ethics and what is good and the
nature of consciousness and
how minds work and function in this
universe
the i would probably submit myself to
a dyson for your brain that was just
like well lucas this is what you have to
do
and i guess that's not subjugation but i
feel less and less
moral qualms with the big dysons for
your brain showing up to some
early civilization like we are and then
just telling them how they should do
things like a parent does with a child
i'm not sure if you have any reactions
to this or how much it even really
matters for anything we can do
today but i think it's potentially an
important reflection on the motivations
behind the principle of
non-domination and non-subjugation and
why it is that we really care about it
so
i think that's true i think that if you
consent to something
then almost i don't want to say by
definition that's definitely too strong
but it's very likely that you're not
being dominated so long as you have
sufficient information and you're not
being coerced
i think the real question is what if
this thing showed up and you said i
don't consent to this
and the thing said i don't care it's in
your best interest
yeah i'm defending that that could be
true in
some kind of utilitarian
consequentialist
moral philosophy of that kind and i
guess my question is do you find that
unproblematic or do you have this
intuition that there's a further
set of reasons you could draw upon which
explain why the entity with greater
authority doesn't actually have the
right to impose these things on you
and i think that it may or may not be
true it probably is true
that from the perspective of welfare
non-domination is good but i also think
that a lot of people who are concerned
about
pluralism and non-domination think that
its value pertains
to something which is quite different
which is human autonomy
and that that has value because of the
kind of creatures we are you know with
freedom of thought a consciousness a
capacity to make our own decisions
so i personally am of the view that even
if we get some
amazing amazing paternalists there's
still a further question
of political legitimacy that needs to be
answered
and that it's not permissible for this
thing to impose without
meeting these standards that we've
talked about today
sure so in the very least i think i'm
attempting to point towards the long
reflection consisting of arguments like
this
like we weren't participating in
coercion before because we didn't really
know what we're talking about but now we
know what we're talking about
and so given our epistemic clarity
coercion makes more sense
it does seem problematic to me and i
think the interesting question is
what does time add to robust epistemic
certainty
so it's quite likely that if you spend a
long time thinking about something at
the end of it you'll be like okay now i
have more confidence
in a proposition that was on the table
when i started
but does that mean that it is actually
substantively justified
and what are you going to say if you
think you're substantially justified but
you can't
actually justify it to other people who
are reasonable rational
and informed like you it seemed
to me that even after a thousand years
you'd still be taking a leap of faith
of the kind that we've seen people take
in the past
with really really devastating
consequences
i don't think it's the case that
ultimately there will be a moral theory
that's settled and the confidence and
the truth value of it
is so high that the people who adhere to
it have somehow gained the right
to kind of run with it on behalf of
humanity
instead i think that we have to proceed
a small step
at a time possibly in perpetuity
and make sure that each one of these
small decisions is subject to continuous
negotiation reflection and democratic
control
the long reflection though to me seems
to be about questions like that because
you're taking a strong epistemological
view on meta ethics
and that there wouldn't be that kind of
clarity that would emerge
over time from minds far greater than
our own
from my perspective i just find the
problem of suffering to be
very very very compelling let's imagine
we have the sphere of
utilitarian expansion into the cosmos
and then there's the sphere of
pluralistic non-domination democratic
virtue ethic
deontological based sphere of expansion
you can say run across planets at
different stages of evolution and here
you have like a suffering hell planet
it's just wild animals
born of darwinian evolution and they're
just eating and murdering each other all
the time and
dying of disease and starvation and
other things and then
maybe you have another planet which is
an early civilization and
there's just subjugation and misery and
all these things and
these spheres of expansion would do
completely different things to these
planets
and we're entering super esoteric sci-fi
space here but again
it's i think instructive of the
importance of something like a long
reflection
it changes what is permissible and what
will be done and so i find it
interesting and valuable but i also
agree with you
about the one claim that you had earlier
about it being unclear that we could
actually
pause the breaks and have a thousand
year philosophy convention
yes i mean the one third thing i'd say
lucas is
bearing in mind some of the earlier
provisos we attached to the period
before the long reflection
we were kind of gambling on the idea
that there would be political legitimacy
and consensus
around things like the alleviation of
needless suffering
so it is not necessarily that it is the
case that everything would be up for
grabs
just because people have to agree upon
it in the world today we can already see
some nascent signs of moral agreement on
things that are really
morally important and would be very
significant if they were
fully realized as ideals maybe there's
just
not that big of a gap between the views
that are left to be argued about
during the long reflection but then
there's also this interesting question
wrapping up on this part of the
conversation about what did we take
previously that was sacred that is no
longer that
an example would be if a moral realist
utilitarian conception ended up just
being the truth or something
then rights never actually mattered
autonomy never mattered
but they functioned as very important
epistemic tool sets
and then we're just like okay we're
basically doing away with everything
that we
said was sacred we still endorsed having
done that but now it's seen in a totally
different light
there could be something like a profound
shift like that which is why something
like long reflection might be
important yeah i think it really matters
how the hypothesized shift comes about
so if there is this kind of global
conversation
with new information coming to light
taking place through a process that's
non-coercive
and that the final result seems to be a
stable consensus of overlapping beliefs
that we have
more moral consensus than we did around
something like human rights
then that looks like a kind of plausible
direction to move in and that might even
be
moral progress itself conversely if it's
people who've been in the conference a
long time
and they come out and they're like we've
reflected a thousand years and now we
have something that we think is true
unfortunately i think they end up kind
of back at square one where
they'll meet people who say we have
reasonable disagreement with you
and we're not necessarily persuaded by
your arguments
and then you have the question of
whether they're more permitted
to engage in value and position than
people were in the past and
i think probably not i think that if
they believe those arguments are so good
they have to put them into a political
process of the kind that we've discussed
and hopefully their merits will be seen
or if not there may be
some avenues that we can't go down but
at least we've done things in the right
way
luckily it may turn out to be the case
that you basically never have to do
coercion because
with good enough reasons and evidence
and argument basically any
mind that exists can be convinced of
something
then it gets into this very interesting
question of if we're respecting a
principle of non-domination and
non-subjugation
as something like neural link and
merging with ai systems
and we gained more and more information
about how to
manipulate and change people what
changes
can we make to people from the outside
would count as coercion
or not because currently we're
constantly getting pushed around
in terms of our development by
technology and people
and the environment and we basically
have no control over that and do i
always endorse the changes that i
undergo
probably not does that count as coercion
maybe
and will increasingly gain power to
change people in this way so
this question of coercion will probably
become more and more interesting
and difficult to parse over time
yeah i think that's quite possible and
it is kind of an observation that can be
made
about many of the areas that we're
thinking about now for example the same
could be said of autonomy
to some extent that's the flip side of
the same question what does it really
mean to be free
free from what and under what conditions
if we just loop back a moment the one
thing i'd say is that the hypothesis
that you know you can create moral
arguments that are so well reasoned that
they persuade anyone
is i think the perfect statement of a
certain
enlightenment perspective on philosophy
that sees rationality as the tiebreaker
and
arbiter of progress in a sense
that the whole project that i've
outlined today
rests upon a recognition or
an acknowledgement that that is probably
unlikely to be true
when people reason freely about what the
good consists in
they do come to different conclusions
and
i guess the kind of thing people will
point to there as evidence is just
the nature of moral deliberation in the
real world
you could say that if there were these
winning arguments that just won by force
of reason we'd be able to identify
them but in reality when we look at how
moral progress has occurred
requires a lot more than just reason
giving
so to some extent i think the master
argument approach itself
rests from mistaken assumptions and
that's why i wanted to go in this other
direction
by a twist of fate if i was mistaken and
if the master argument was possible it
would also satisfy
a lot of conditions of political
legitimacy right now we have good
evidence that it isn't possible
so we should proceed in one way if it is
possible then those people can appeal to
the political processes they can be
convinced
they can be convinced and so there's
reason for hope there for people who
hold a different
perspective to my own all right i think
that's an
excellent point to wrap up on them do
you have anything here i'm just giving
you an open space now if you feel
unresolved about anything or have any
last moment thoughts that you'd really
like to say and share
i found this conversation really
informative and helpful and
i appreciate and really value the work
that you're doing on this i think it's
sorely needed yeah thank you so much
lucas it's been a really really
fascinating conversation and it's
definitely pushed me to think about some
questions that i hadn't considered
before
i think the one thing i'd say is that
this is really
a lot of it is exploratory work these
are questions that we're all exploring
together
so if people are interested in value
alignment obviously
listeners of this podcast will be but
specifically normative value alignment
and these questions about pluralism
democracy and ai
then please feel free to reach out to me
contribute to the debate
and i also look forward to continuing
the conversation with everyone who wants
to look at these things and develop the
conversation further
if people want to follow you or get in
contact with you or
look at more of your work where the best
places to do that i think if we look on
google scholar there's links to most of
the articles that i've written including
the one that we were discussing today
people can also send me an email which
is just my first name
yason deepmind.com so yeah
all right
if you enjoyed this podcast please
subscribe give it a like
or share it on your preferred social
media platform we'll be back again soon
with another episode in the ai alignment
series
[Applause]