Untangling Artificial Intelligence Ethics (Andreia Martinho)

the meeting is now recorded
andrea the floor is yours thank you good
afternoon
my name is andrei martino i'm a phd
researcher at tpm
and my supervisors are casper horos and
martin crowson
so first i want to take this opportunity
to present my research
at the ai tech agona today i will
be talking a bit about the ethics of
artificial intelligence
the title of this talk is untangling
artificial intelligence ethics
artificial moral agents autonomous
vehicles and
thought experiments
so the beginnings of artificial
intelligence can be traced to
imagination
fiction philosophy and so i will start
this talk
with a bit of fiction this is
a movie clip from the movie ex machina
the video is a bit long one and a half
minutes but i think it is quite
interesting
hi i'm
caleb
hello
do you have a name yes
ava pleased to meet you ava mr andrea
can you perhaps increase the volume we
can hear it but it's barely
there sure
i've never met anyone new before moments
hi i'm
caleb
hello caleb
do you have a name yes
ava i'm pleased to meet you ava
i'm pleased to meet you too
i've never met anyone new before
nathan and i guess we're both in quite a
similar position
haven't you met lots of new people
before
none like you
so we need to break the ice you know
what i mean by that
yes what do i mean overcome initial
social awkwardness
so let's have a conversation okay
what would you like to have a
conversation about why don't we start
with you telling me something about
yourself
what would you like to know whatever
comes into your head
well you already know my name
and you can see that i'm a machine would
you like to know how old i am
sure i'm one one what
one year or one day
when did you learn how to speak ava
i always knew how to speak and that's
strange
isn't it
so ava is what we call
in the machine ethics literature an
artificial moral
agent so i don't want to spoil the movie
for the people who haven't seen it yet
but i think it's pretty safe to say that
eva it's an artificial moral agent that
ranks
very high in the autonomy and ethical
sensitivity spectrum
and indeed most of the controversies
that we find in the scientific
literature
about artificial moral agents concern
systems like ava which rank very high in
autonomy and ethical sensitivity
[Music]
so what what are these controversies
by looking at the literature we
identified four main controversies
so the first one is about the
development of these systems
should the systems even be developed
this is the core
of machine ethics but is quite
controversial
the second controversy it's about the
design of
amas how should we implement
morality in machines there's no
consensus about
the best implementation strategy the
third controversy
it's about moral agency we would this
systems
have uh moral agency would they be
comparable
with humans as far as moral agency is
concerned
and then finally what are the societal
and moral roles of
these systems will they be our moral
teachers
or our moral destroyers
so this there's even though there's a
lot of debates about
these issues in the literature there's
poor empirical information
about this so we raised the question
what are the perspectives
of ai ethics scholars about these
controversies
this research is currently under uh
under
revision and hopefully it will be
published quite soon and it's
in collaboration with adam polson who is
an australian based
researcher so to address this question
we used q methodology which is an
adequate methodology to bring coherence
to complex and convoluted issues such as
this one
so the first step was to build the
concourse of communication
which is pretty much a background so we
looked into the literature from popular
and scientific publications
and we derived 203 statements that would
relate to
these controversies that i just
mentioned
from this course we selected 45
statements that would capture the main
controversies and the main tensions on
this topic
then we selected the participants
because we needed
the participants to grasp some sort of
basic notions
of philosophy artificial intelligence
and ethics
we decided that we would um we would
contact uh researchers with publications
in a broad in the broad field of ai and
ethics but also
artificial more agents machine ethics
autonomous vehicles
and we got 50 uh completed surveys which
is actually a very good number
for queue methodology because it entails
a lot of choices so we we don't need
hundreds of participants and then
finally
we used multivariate that data reduction
techniques which was pca
to analyze the data
so just to get you some examples of the
statements that we used
from the literature uh from for the
development controversy this is one
example
technological progress requires
artificial morality
and for instance for about the future of
artificial moral agents
we selected this is just one example
mere indifference to human values
including human survival could be
sufficient for artificial general
intelligence to pose an existential
threat
so as you can see the statements were
quite short
and they were written like a proposition
so that participants could either
disagree
or agree
so this is the survey that participants
completed
the first step was to assign the
statements
into three different bins uh disagree
neutral or
agree then the participants would sort
the statements
and assign them into a forced
quasi-normal distribution
which would go from minus five to five
uh and then finally
uh the participants had the possibility
to provide
comments on the statements that they
ranked
highest or lowest so this methodology
allowed us to have quantitative data but
also qualitative data
from the data five main perspectives
have emerged
and if this was uh this this talk would
be done in person
we could do something interesting and uh
have some sort of
uh activity so that you can also vote in
which one you you
would agree more so we cannot do this uh
today because
uh of the pandemic but you can still
uh do an internal exercise and uh
and kind of select which one do you
agree more or which one you identify
with the most
so the first perspective it's called
machine ethics the way forward
the second perspective is called ethical
verification
safe and sufficient the third one is
morally uncertain machines
human values to avoid moral dystopia the
fourth one is
human exceptionalism machines cannot
moralize
and the fifth one is machine objectivism
machines as superior moral agents
so let's let's learn a bit more about
each one
of these perspectives so the first
perspective as
i mentioned just before it's called
machine ethics the way forward
according to this perspective amas so
artificial moral agents
are unavoidable and they may even be
necessary for technological progress
they are more than simple tools and they
will advance our understanding of
morality
so the striking feature of this
perspective is actually the development
and how positive this perspective is
about the development
and from the the data that we got the
qualitative data that we got from
participants i selected one
uh statement um written by one of the
participants of course i don't know who
he or she is that relates to this
striking feature feature there will be
no other way than to develop ethical
machines when humanity is supposed to
rely with their life
on them so moving to the second
perspective ethical verification safe
and sufficient
according to this perspective amas will
not replace humans in ambiguous moral
situations
they believe or according to this
perspective ethics and human moral
agency are not algorithmic they cannot
be reduced
to an algorithm and transparency
accountability and predictability
leads to sufficiently ethical machines
so the striking feature about
this perspective is the design
according to this perspective the moral
performance of machines should be
evaluated through
external checks and balances such as
verification of transparency
accountability and predictability and
not so much
by implementing morality within machines
so this is one of the statements from
this perspective
if an ama makes the wrong decision the
outcomes could be disastrous
similarly the risks of malicious hacking
of amas
are serious this verification validation
and transparency are critical to the
success
of even limited amas in real world use
so this is a statement that one of the
participants wrote
the third perspective it's called moral
morally uncertain machines
human values to avoid moral dystopia
according to this perspective
amas must be morally uncertain and hold
human values otherwise they will pose an
existential threat
according to this perspective
prohibiting unethical behavior
as well as implementing external checks
and balances it just
isn't enough striking feature about this
perspective is about
future projections according to this
perspective
mere indifference to human values could
be sufficient for agi to pose an
existential threat
so the the highlight of this perspective
is
sort of the dystopian future
and one of the participants wrote though
experiments such as
through experiments such as the
paperclip maximizer
show quite convinced conversely that for
an egi to pursue a goal that is merely
orthogonal to human values could
plausibly
present an existential threat
moving to the fourth perspective human
exceptionalism
machines cannot moralize as per this
perspective amas are not
moral agents because they lack free will
and the understanding to make moral
assessments
as we humans do logical machines will
not be better moralizers
or are moral teachers because they lack
this human element
so the striking feature about this
perspective
is the human element according to this
perspective
human flaws such as irrationality
is a push for moral agency so amas are
not and will not be
moral agents because they lack this sort
of conceptual
understanding one of the participants
wrote you start being moral when you
recognize
your shared humanness with others and
understand it
that like it or not you are in a
relationship with them
until machines get that and i'm
suspicious of their ability to do so
then they're not going to have full
moral agency
and finally the the final perspective is
machine objectivism
machines as superior moral agents
so according to this perspective amas
will prevent human
harm through logic and context
specificity these agents are better
better moral reasoners and educators
according to this perspective free will
and conceptual understanding are not
required for moral agency
so the striking feature about this final
perspective
is that agi or you know these advanced
forms of
uh of artificial agents will lead to a
better understanding of morality
and actually they will be better moral
agents than humans because they are not
subject to
irrationality seduction or emotional
turmoil
one of the participants wrote machines
can enhance us as moral agents if they
manage to distance
us reflectively from our intuitions
which are very much determined by social
preference
so i was
i'm wondering if you have some sort of
preference
or do you think that one of the
perspectives makes more sense for you
and this is something that
hopefully we can discuss in the end of
the talk and i'm looking forward to
hearing your comments and thoughts about
each one of these of these perspectives
so the main findings now that i have
explained
um the five perspectives that came from
the data
is that there are very different uh
perspectives about the issue of
artificial moral agents and this
kind of attests the complexity of the
matter
we found two main dichotomies in the
data with respect to development and to
moral agency
so according to the first perspective
machine ethics the way forward it stands
for advancing artificial morality
differently the second perspective which
is ethical verification safe and
sufficient
is skeptical about the feasibility or
need for artificial immorality
differently they just consider that
external checks and balances are
sufficient
and then the second echo to me regarding
moral agency
so according to the perspective human
exceptionalism
uh machines cannot moralize which is
perspective four
there's a high value of the human aspect
in morality
but according to perspective fifth
machine objectivism machines as superior
moral agents
morality improves when stripped of human
flaws
we also found some agreements there is
an overarching agreement
that ai systems working in morally
salient context cannot be avoided
and that developing amas it is
permissible
according to existing moral theories and
this is
in the final agreement that i found
was quite interesting because with the
exception of perspective four
none of the other perspectives consider
free will
an important or essential element for
moral agency
this position immerse emerges from the
thought
that was shared by participants in the
study that because
humans do not have at least radical free
will and yet they are moral agents
the same should apply to machines
so then we looked into this data
we saw that the dichotomies we saw the
agreement
and we started thinking what why there
are so many perspectives about this
topic
and a potential source of the these
differing perspectives we believe
it's related to the failure of machine
ethics to be widely
observed or explored as an applied ethic
indeed machine ethics literature is
chiefly
abstract we believe that ai
ethics needs to be applied and shaped to
the field in which it operates
and not the other way around so having
this in mind
we will now look into artificial
intelligence ethics
in a particular field which is
transportation
we all know that in recent years there
has been many debates about the ethics
of autonomous vehicles
indeed autonomous vehicles are often
used as the archetypal
ai system in many of these atticus
ethics debates
however little is known about the
ethical issues in focus
by the industry that is developing these
systems
because the industry has such a
important role in developing this
technology
we believe that it is important to
understand their position
even if it is quite formal about
ethics this this work was recently
published
in the journal transport reviews uh
me my supervisors and also and former
master students
uh niels herbert
so to address this research question
that i just raised
we we did a literature review
but rather than just looking into the
scientific literature we looked into the
official business and technical reports
of companies
with an av testing permit in california
we chose california for a couple of
reasons first it was an early proponent
of autonomous vehicle technology and it
houses
many r d programs and also they're quite
transparent
about the companies that have permits
so our starting point was
a list of ethical issues so we we knew
we wanted to look for
um ethics in the technical reports
of companies but we needed a list of
ethical issues
so we used this list of issues compiled
from 22 major
guidelines of artificial intelligence
ethics
this was was not compiled by us this is
a work
made by hagendorf and
this is the our starting point for this
research
then we looked into the scientific
literature for this
particular step we did the prisma
literature review
and we identified 238
articles that were reviewed and finally
we did the document review
we used 86 business and technical
reports
that were issued from 2015 to 2020
from 29 companies first we did a
contextual or
manual analysis and then we did we used
the text mining algorithm
and then again we we checked again
manually if everything made sense
i just didn't include that step over
there
so these are the companies that we
use the reports from in the study
as you can see uh there's it's 29
companies
at this point in time which was uh june
2020
there was a record of i think 66
companies had testing permit in
california
so then the question is why didn't use
reports from all the companies well we
were pretty strict about the kind of
documents that we were going to use
for reprocessability
purposes or reasons so
we wanted to use reports that were
available
in uh pdf so that we could you know
always refer back to them
so we excluded uh blog entries
uh articles that were just available
online
in any of those communications that were
not
we could not actually store in in our
own
on records so we emailed
every company we did online searches
and then we came up with the final list
of of
29 companies and 86 documents that were
used in this research
the data set is stored at the 40
research data if you have some interest
in looking
at it it's it's publicly available
okay so we first looked into
the landscape both on scientific
literature and
on the industry literature
and it was quite salient that were that
there were some
ethical issues that were more relevant
than others
so based on the criteria of frequency
and by that i mean uh how many times
these ethical issues were
uh appearing in uh in the documents but
also
comprehensiveness how deeply were they
explored
we came up or we identified three main
ethical issues that we wanted to further
explore safety human oversight
and accountability and these are the
three main
ethical issues that we will we will
explore today so
the first ethical issue which is safety
and cyber security
um in the ethics literature and we can
go back to the word cloud
and you can see how central the trolley
world
is and this is like this is true this is
i
we did this word cloud based on the data
and actually the trolley problem
appeared in more than half
of the scientific articles that we
reviewed it's quite
uh an evidence that the ethics
debates in the scientific literature are
dominated by this
thought experiment for the ones that are
not very familiar with this
thought experiment um it was popularized
by felipe foot in 1967
and uh there's many variation variations
to it but
it pretty much entails a hard moral
decision
between lives saved and sacrificed
recently it was reported this that there
is an overstatement
of this black and white stylistic uh
thought experiment and that this was not
very helpful
for the ethics community so different um
different ethical extreme situations or
weakened
ethical trolley problems have been
reported in the literature
recently so this is one of those
examples i call
i call them weak trolley problems
because they're still a moral decision
in place but it's not as dramatic as the
the thought experiment that i just
mentioned so in this case it's
a situation where the autonomous vehicle
needs to
make a decision of going into the left
or into the
right side of the lane based on
distributions of risk
so the av trolley problem argument
is set in the in the scientific
literature more more or less like this
autonomous vehicles ought to save lives
upon development
deployment of autonomous vehicles
extreme traffic situations
will not be completely avoided some of
these
extreme traffic situations will require
an autonomous vehicle to make difficult
moral decisions
difficult moral decisions in traffic
resemble trolley problems
the best option to assist autonomous
vehicles in managing
the trolley problem is x
x is programmable therefore av should be
programmed with
x and the main premises that are debated
in the literature
is premise four which is about the
relevance
of the trolley problem and premise fifth
which is about the strategies to
implement uh some sort of moral controls
into
the autonomous vehicle so then from the
scientific literature we raised two very
practical questions from which we want
to look for answers
in industry reports the first question
is are extreme traffic situations
resembling trolley cases
addressed by industry and the second
question is what are the solutions
proposed by industry to address
extreme traffic situations so let's see
what
industry says about this so we spected
the industry reports regarding crash
worthiness
to clarify some of the critical elements
of the autonomous vehicle moral dilemma
the first element is the risk of
crashing so if you think about it if
if autonomous vehicle technology um
eliminates completely crashes and
collisions then we don't have
a trolley problem or any sort of moral
situation
like this so what we found is that
companies express their vision
of a future without accidents this is an
aspirational goal
in pretty much all the reports we read
but they emphasize
the inevitability of crashes and
collisions
this is one of the statements that we we
retrieved from a toyota report
driving environments can be extremely
complex and difficult and no automated
driving system
regardless of how capable it may be is
likely to prevent
crashes entirely so then we go to the
second element
extreme traffic situations or how they
call it in industry reports
edge cases i included some graphs here
so you can see the autonomous vehicle
uh it's going off the cliff and
unfortunately going into the ocean
apparently so this would be an extreme
traffic situation
uh we selected uh one statement from a
report issued by nvidia
ai-powered autonomous vehicles must be
able to respond properly
to the incredibly diverse situations
they could experience
such as emergency vehicles pedestrians
animals
and a virtually infinite numbers of
other obstacles
including scenarios that are too
dangerous to test in the real world
and then finally we go to the final
element
moral situations and what we found out
is that there's no reference whatsoever
in the industry literature about moral
dilemmas trolley cases or anything like
that
as described in the scientific
literature that is
situations that require the autonomous
vehicle to make a difficult moral choice
however we did find one statement that
was
even though it's not a trolley case it
was
close enough that i felt the need to
highlight it here so nuro
wrote this in the in the unlikely case
of a neural shuttle ever encountering an
unavoidable collision scenario
the driverless passengerless vehicle has
the unique opportunity to prioritize the
safety of humans
other road users and occupied vehicles
over its contents
so of course this is not a trolley case
because the nature of the choices is
very different
in this case it's not life versus life
it's life versus
goods or merchandising uh
we speculate that uh this somewhat
more transparent account um from neuro
it's probably because about the the
moral situations it's probably because
this company
uh does not so the shuttles of this
company they don't carry
uh passengers but only goods so
this is this is why we we considered
that they were a bit more open about
uh this moral situation but it's just a
speculation of course
so answering the questions that were
asked before i extreme
our extreme traffic situations
resembling trolley cases addressed by
industry
no but there are nuanced delusions that
unravel
underlying concerns about these extreme
traffic situations
and then the second question is what are
the solutions proposed
to address extreme traffic situations
we found radar speed limitations
simulation and validation methods to
test
those scenarios
so now we move to the second ethical
issue that we considered
relevant by looking at the scientific
and industry literature
which is human oversight control and
oddity
here we highlight the the human
meaningful control
philosophical account which uh
i it was uh recently
um highlighted and and uh
it's it's been the subject of a lot of
amazing research at the udelph
especially by
uh filippo i'm not sure if he's here but
uh he can talk about this much better
than i can
but just in a simple way um for
autonomous vehicles to be
um to be under uh meaningful human forms
of control
they should meet two conditions the
tracking condition and the tracing
condition
the tracking condition is about the av
being able to track the relevant human
moral reasons
in a sufficient number of occasions and
this condition ensures that
the av complies with the intentions of a
human operator
and then the tracing condition
there's two dimensions to this condition
so the first i mentioned is about
the actions of the autonomous vehicle
being traceable to a proper more
understanding
and then the second condition the second
dimension is that at least
one human agent should understand the
real capabilities
of the system and bear the moral
consequences of its actions
so we raised by looking into the
literature we raised a question
uh for industry once again and the
question is
which decision prevails in traffic the
autonomous vehicle or the human operator
and now we will look we will look for
answers for this
question so first of all i wanted to
mention that we found a
statement that we believe relate to the
tracking condition
namely companies they emphasize their
on-site human oversight
of the av testing operations and also
the remote control
of the autonomous vehicle operations
most of the companies as i showed before
did not have driverless uh testing
permit
so on-site uh human oversight
of the operations is normally uh the one
that it's
uh used for the tracking condition
and then for the tracing condition um
regarding
the understanding of the real
capabilities of the system
we also found one interesting statement
from ai motive
which states that test operators face
their own unique challenges the debug
screen of a complex autonomous system
is incomprehensible to the untrained eye
these engineers and developers have a
deep understanding of the code at work
in our prototypes allowing them at times
to predict when the system may fail
this allows our test crews to retake
control of the vehicle preventable
in a controlled manner
so going back to the question that we
raised about human oversight
which decision prevails in traffic and
actually we found different approaches
with respect to the authority
so companies like mercedes-benz and
intel
they prioritize the autonomy of the
vehicle
but also companies like auto x they
prioritize decisions of the remote
operators
and finally we move to the last ethical
issue
that we have identified before which is
accountability so in the human
meaningful control
in the second uh in the tracing
condition but in the second dimension of
the tracing condition
there's also one dimension that it's
about
uh having one agent that can bear the
moral consequences of the actions of the
system
it has been stated that in order for
this dimension of the tracing condition
to be met
in higher order levels of automation a
transition of responsibility from the
driver to designers or remote
operators is required and another issue
that has been raised in the scientific
literature
it's about the liability and technology
argument
so it is stated that autonomous vehicles
have the potential to save lives but
crashing liability may discourage
manufacturers from developing and
deploying these systems
and then this technology would not meet
its potential to save lives
so from the literature we raised one
question which is
which accountability design strategy is
adopted
by the autonomous vehicle industry
so we found out that companies seem
invested in the lowest liability risk
design strategy
this is not shocking of course so this
strategy relies
on rules and regulations expedite
investigations
in this case we we have uh hear a
statement from uh
that we found in a report issued by bmw
in which they they show their intention
to use a black box
similar to the ones that they use in
flights so that
uh it makes liability easier uh
to to address
and then finally crash and collision
avoidance algorithms
so i will read this this statement
because i thought it was
interesting from intel it says that avs
will never cause or be responsible for
accidents
by formally defining the parameters of
the dangerous situation
and proper response we can say that
responsibility is assigned to the party
who did not comply with the proper
response
so going back to the question that we
raised which accountability design
strategy is adopted by
the autonomous vehicle industry we
believe that it's minimally
minimally responsible technology
mobilized
states that the self-driving car will
never initiate a
dangerous situation and thus it will
never cause an accident this is
pretty um illustrative of
minimally responsible technology
and finally we also found this
interesting statement from intel
in which they say that their
responsibility sensitive
safety system will always break in time
to avoid a collision with the pedestrian
unless the pedestrian is running
superhumanly fast so
as a summary i first started this talk
um with artificial moral agents and
we identified five different
perspectives
about key ethical controversies related
to artificial
moral agents we concluded that for
for artificial intelligence ethics to be
accepted as an applied ethics
context sensitivity is required and then
we looked into the specific field of
transportation
and particularly to the autonomous
vehicle
we reviewed the ethical issues in focus
by the autonomous vehicles industry in
california
and we compared and contrasted with the
scientific and industry narratives
industry provides elements that should
be considered in the ethics debates
my main takeaway message is that as ai
becomes part of our daily lives
ethicists need to adapt and find ways to
untangle ethics in order to foster
meaningful communication
between the ethics and technology
communities
in my view empirical research such as
the one that i just presented
it's a good way to realize this
thank you
thank you
do you have any questions uh if you have
uh any questions please raise your hand
the button is uh on the top right corner
or just post your question in the chat
yeah i think we have the first hand
raised and it's eugenia's hand
hi yes thanks andrea thank you very much
really interesting presentation a lot of
interesting things to think about
uh i uh so one of my questions was
so uh following up on the conclusion
that you that you just uh made that the
empirical research is necessary to
adapt the conversation we have better so
in particular like
the the trolley problem example that you
mentioned
i'm curious so what are your thoughts
about this particular example like uh
um like what
what did you learn from from the
research that you did
with with regard to that like is it
is it useful at all uh to talk about uh
or
does it or is it useless or like
yeah i'm just curious to hear your
thoughts well
uh the main goal of ethics it's to
um or ethics and thought experiments in
particular
is to um come up with these extreme
dramatic situations and kind of test the
values that
come out of that so in that way i do
believe that the trolley problem
it's in any thought experiment it's very
very very important
however as i mentioned before as
artificial intelligence leaves the
laboratories and starts
being used on a you know day-to-day
basis
we need to move from these very
important and very interesting thought
experiments and start to think about
ethics in a more practical way
so that we can actually uh be
alongside the technology uh communities
and help develop ethical systems
and what i find what i find always
interesting about the trolley example
i'm asking myself
well but what happens today like when a
human is
driving the car like so what like what
happens in that what do we consider the
right moral decision there
like do we have an answer to that and
like if the case would come to the court
let's say in the netherlands for example
how would the court judge
uh how that unfolded and i i i don't
hear us usually talk about that and then
i wonder like we jump right away to the
ai
scenarios well i think that uh
humans have something that machines
don't have we have the benefit of
reacting by instinct
and when you react by instinct well i'm
i'm not an expert
by by no means but i don't think any
court will
will judge you on a split-second
decision it's instinct however
artificial intelligence machines do not
have this uh
advantage so we need to progre program
them in advance
so they can react in the moment so that
that's the main
difference nice you actually help help
help advance my understanding of this
yeah thanks i appreciate it
um while we are waiting for more
questions andrea would you mind stop
sharing your screen
i think uh yeah we will have a little
bit of a
so i'm not watching a screen here yeah
uh you just click on the small button
next to leave
yeah this one yeah okay
sorry i'm i never use teams actually or
rarely
um i actually have a related question
um about the trolley problem or other
the conclusions that uh
yeah that's a question another question
from evgeny sorry again it's uh
it's my turn now so yeah uh it's it's
more related to the not to the truly
dilemma itself but
more to the analysis that you did based
on research papers
and there you have a really impressive
word cloud which
shows that truly is the most often
discussed word in the papers right but
i'm just wondering uh
have you uh have you done any
kind of in-depth analysis to see what is
the tone of discussions because i i
i'm pretty biased here so i read a lot
of papers which criticized through the
dilemma as an approach to
machine ethics so i'm wondering can it
just be people
uh arguing against uh usefulness of
trolley dynamic rather than advocating
that this is the way
to solve ai attacks okay that's
that's a very good question so i will
say that
[Music]
clearly when this this someone came up
with
uh with this trolley dilemma
in the context of autonomous vehicle
there was some sort of fascination i
think that the media
and um you know uh even scientifically
to sure
they you know they kind of fell in love
with it because it's like having a
thought experiment from a textbook
coming to life and that does not happen
very often and then gradually uh you
know the literature
in recent years as i mentioned before it
has been
saying that there's an overstatement of
that trolley problem
so there's two things about this
first of all while the community is
arguing in favor
and against it they're not looking into
other pressing issues and that
i think it's a problem and the other
thing that i want to say about this
is that yes there is a lot of
discussions
and there's not a lot of agreements and
that's exactly the type of research that
i do
is trying to look into convoluted
complex issues
and then try to make some sense and
and deriving some empirical insights
from it
i don't hear you arkady
you're muted arcady yeah
yeah thanks uh so uh i'm wondering if
there is also a dynamic aspect to it
whether
this uh kind of shift of attention uh
changes over time so whether people
started looking at through the dilemma
as you said
soon after that paper was published and
then it kind of faded down
and then other topics are catching up or
is it consistent over time
so that's uh that would be interesting
to know as well
um i can say that what i've been seeing
the most recent publications um i think
they're pushing more for this sort of
weakest trolley problems so about
distribution of marginal risk
this kind of more realistic uh
moral situations um
[Music]
that's what i what i i've been noticing
recently
thanks
any other questions i think if ghani had
one right
yeah while we are waiting for our other
people to come up with their questions
again
you can go ahead um or perhaps madeleine
raised yeah let's let's let uh other
people go first and then if there's time
left i'm gonna ask yeah madeline do you
want to ask your question
yeah thank you so much for your
fascinating talk it was really great you
covered a lot
um i was just wondering actually um if
you could
spell out the argument that you made
sort of switching
into the context specific if you could
say a little bit more about why you
why you made that move in in that
argument
um that we really need we really need to
be doing this context specific
applied ethics yeah thank you that's a
very good question
so um as i said before as we move
from theoretical ai kind of unreachable
ai to something that we're using
in our daily lives i believe that we
should have
ethics that it's more applied to the
specific fields
but answering in a different way um
many so if you if you think about the
data that we got about artificial moral
agents
it's highly heterogeneous and this is
the same in i'm sure this is the same
about
other ethical issues related to
artificial intelligence not just
artificial moral agents and one of the
things that
that i think that explains this it's
because it's
this literature it's very abstract so
when you
when you're talking about an artificial
moral agent
it could be anything it could go from an
autopilot to
a system like ava like i showed and so
it
because it's abstract then there's a lot
of uh
there's a lot of contradictions a lot of
disagreements and once you scale down a
bit
and and make it more context specific
not only do you
um do you gain more insight but also i
we believe that it's it's you're going
to get more agreements than when it's
just
abstract and theoretical
thank you
hey afghani your turn again yeah sure
thanks yeah yeah i think that's a larger
point andre i think that that's so
great that you bring up the whole matter
of like
uh grounding in in in a context with
communication between the
theoretical discussions and the practice
i think also the other way around when
sometimes we
like when we do ethical discourse and we
point out problems which are
very relevant but people who are
practicing are not aware
that's also true so because some often
we have very passionate debates here at
the university but people who are
actually practicing they're entirely not
aware of all these discussions so we
have these passionate debates but unless
we actually have the conversation
we don't impact the world in the way we
want to impact
so sometimes we also have good ideas but
they don't so we need to have this
conversation
i i was wondering um something about the
the first half of your presentation
i was curious you mentioned on one of
the slides the point that
uh like uh so an argument that humans
don't have radical free will i i was
could you please elaborate so what what
does like what does that point mean
uh on that slide sure um
that's that's actually one of the what i
consider one of the most interesting
parts of the study or one of the most
interesting findings that was a bit
unexpected
so of course i don't want to go into
this whole philosophical discussion
about determinism
you know indeterminism but
the argument that was found consistently
except for the
the fourth perspective is that well
humans as humans we don't have radical
free will
it's like it's it's a bit it's a bit
deterministic perspective about free
will it's like
our actions are somehow predetermined
um and still we we think we are
we regard ourselves as moral agents so
why shouldn't machines as well
and it was in the beginning when i when
i was looking at the data i thought this
would be just a marginal
um a marginal finding but then
actually turned out the other way around
so either
uh either everyone that that
did the survey or almost everyone is
believes in determinism or this is
really something that uh that that
people really believe so that machines
they don't need free will to be moral
agents and bear some moral
responsibility i do think this is an
interesting research avenue to be
pursued
after this um after this paper is
published
yeah very interesting i wouldn't expect
that as well
i was very surprised that's that's
actually something that
really made this research fun is to see
these things coming up
i i just like a note and this definitely
links to the larger conversation that
you want to avoid right now but
i i was thinking like sometimes in the
academic environment we talk about free
will
in ways that makes me really cringe
because when we have these very
philosophical conversations
i find it a very slippery slope in the
civil sense of the word because free
will of course is the way we interpret
it civically
as my you know freedom to love who i
want to
say whether to say things and like the
freedom of speech you know
freedom of religion that's very
essential to our societies and sometimes
in the academic discourse we go into the
like
well like is it biologically
predetermined and
yeah i was i was yeah maybe maybe i
should have made this a bit more clear
when i was mentioning free will it's uh
it's about uh you know the deterministic
or deterministic
perspective about free will and uh how
much
do we have free will and or if we just
believe on it so that that's another
that that's another layer of the
conversation because for some for some
philosophers it's only required the the
the fate or the confidence that we have
free will and not
really free will yeah
no well it's a whole it's a whole super
interesting topic
on its own but of course but but the
thing is that some of course
we have these developments and
especially when it comes to algorithmic
ai profiling of people
that when you use the argument let's say
that well we do not have free will
then you can start arguing well it's
it's that that kind of can be used as a
justification why you can use
big data to predict people's behavior
because everything is predetermined
anyway
and the machine learning model will know
you better than you yourself know
yourself
yeah but anyway that's a whole big
discussion yeah
thank you thank you
great we have uh a couple more minutes
for uh
the last maybe one two questions
i'm curious to see if someone got really
got some strong
feelings about one of the perspectives
in favor or against it or
i was surprised by the diversity of i
mean
you found the whole two controversies
not just one
and the pro contra amaze but it was a
two-dimensional controversy and it was
surprising to
to see that uh people are that divergent
and uh
i think you mentioned that it was mostly
academics right
people who responded to the statements
well yeah from a wide range so
i sent many many many emails uh but only
for people that have some sort of
publication in
ethics or artificial intelligence
because otherwise it would be very
difficult
to get meaningful uh information because
the statements itself they were you know
they were philosophical like
uh you know concepts like moral agency
free will
conceptual understanding it's not
something that uh
anyone could really grasp it so that's
why we selected this
participant set right but at the same
time did you
break people down based on their main
thing or
sorry did you analyze did you even
collect the data on people's discipline
background so
i would imagine that you would have a
little bit more uh
a little bit higher share of proponents
of amas
in uh people with engineering
backgrounds perhaps compared to
philosophical backgrounds i don't know
uh well the people the people from the
engineering
uh that i contacted and again i
contacted
many many many people so i didn't select
i was not like making some sort of
selection bias or anything um
but they're all uh they all
uh were published in ethics so it's not
that they were
like working on autonomous vehicles and
never have published something on ethics
it was always probably
people that have published in the field
from at the ethics side as well
but but yeah that's a that's a good
question and actually i wanted to do
that but then we have a problem in the
survey
and somehow we didn't get that data
which
oh there's someone is here yeah karolina
wants to say something
yeah hi uh thank you very much for the
talk just
very very interesting um yeah
in my i just wanted to bring something
up um
so on my own phd and uh well
in in my lab and there are other
works before that worked also on this
question uh
we we like to look at it as uh in a
different a slight different uh
perspective uh
in the sense that we believe that maybe
the machines don't have to be
so autonomous in everything they can
just
ask for humans for their input in some
in some moments so of course you cannot
do this when the car
is driving and the city because
everything is too fast and uh
yeah it's they have to decide by their
selves let's say
um but in many other situations when i
think that these ethics problems come up
um we just like
to to yeah to i would just
like to leave this input that maybe
in some cases we can use actually man to
to into to be interdependent and not
just go fully on this idea that the
machines have to be fully autonomous in
ethics and moral issues and
so they can actually ask you know and um
i don't know we can still deal with some
ethical problems ourselves instead of
putting that in the
in the autonomy part yeah that's
interesting so in that uh
second slide that i showed a third slide
that i showed that then you would
say that machines should have it should
be low on
the autonomy axis and it also seems from
your
from your words that you kind of
identify more with the second
perspective which is about
uh ethical verification checks and
balances but not necessarily
implementing morality within uh the
machine
yeah so i believe that in a far future
well i think that machines can
level up humans in any sense i mean in
the far
far future so
so i would say that i will make i will
be a bit of devil's advocate because to
be honest i
i like i like to be neutral about
research yeah uh but i would say that
the machine ethics proponents
they would say that um
artificial morality should not be enough
thought
and that we should working towards that
since now
and you would also say that just by
learning about
it and trying to develop an artificial
moral agent we are
already learning about morality so
that's like a good
uh collection yeah yeah but i actually
said this to mean that um
i believe that maybe one day they can
become
they can do everything we can so we have
to
have take this into account from day one
uh that's what i meant um but yeah
uh i i don't know i i think
yeah i just i just wanted to give this
uh idea that
yeah will the um the autonomy
i mean to put the morality and the
ethics in the
in our autonomous partners let's say we
don't have to
maybe just put everything on on them
in some situations they can just
ask or require some input
thank you so much it's a great input
thank you carmina thank you
okay i think it's time to wrap up uh
thanks andrea for your presentation
and thanks everyone for a nice
discussion um
everyone yeah the recording will be
available on our
youtube channel soon and see you all
next week
bye bye bye thank you


270. My Objections to Eliezer Yudkowsky 1

hello and welcome to session 270 in the
AI safety.com reading group tonight
we'll be discussing the first half of my
objections to we're all gonna die by
Elia saidkowski and this is written by
Quentin Pope
Quentin Pope
um is not publicly writing who he is
this is his image from Twitter and uh I
assume it's because he's killed about
his privacy I did in fact manage while
I'm
while I was looking him up to get a good
picture of who he is but I think he
would prefer privacy so we are
unfortunately stuck with this picture
that I'll be used as attribution for his
claims in this presentation
and earlierkowski of course prefers to
be called decision theorist at Miri I
don't think that is what I would
actually say he is but that's what he
prefers
um this was published a month ago on
listerong and we are taking the first
half
ending before
Quentin Pope this describes the the part
where more people are people are more
optimistic than earlier
so first thing is I want to say thank
you to Quentin Pope because I think uh
critically examining these uh the
arguments are a really valuable service
and it's really important to dig into
and even adversarially uh try to
question the assumptions and try to find
weak points because that is how we get
to truth and I think it's uh it can be
thankless work and well I want to say
thank you
uh the criticism that Elisa utkowski has
gotten uh from this
um uh from his podcast on bankless has
been
um
Clinton Pope notes that it's been uh of
a on like an outside high level view
where it doesn't really argue uh against
him in in a kind of specific way and
um
uh
Quentin Pope agrees more with with these
people Tyler coven Robin Hansen then he
does with Ilia zidkowski and he wants to
uh
provide some like inside view uh
arguments against uh it is uh points
in particular he uh in the the criteria
he uses for what arguments to
investigate are those are the ones he
disagrees most with
um and of course I when when I see this
just this um uh podcast on on bankless
that's uh 16 000 words and there are
just a lot of claims so uh I don't
actually think Quentin has chosen the
right methodology the right procedure
for figuring out what claims to engage
with because I think he should have
engaged with the claims that ilyaskowski
explicitly Flex as important
um and like the two
uh paths that I see as most important
and that I think Eliseo you think is
most important for why alignment is such
a definitely problem the fact that it's
like we only have one try and we don't
have much time
um that's the kind of arguments that I
think are really really Central and
that's the thing that I wanted Quentin
to engage with
and you know perhaps not even as I mean
choosing list of lethalities instead of
uh instead of a podcast transcript
um would probably also be wise
so the way that this is something I
reuse for a skepticism uh that
um
there is a sense probably that Quentin
and many others have that a lot of the
points is writings are wrong and this
gives rise to a broad and shallow
criticism of many of these individual
statements that utkasku is coming with
um and this is in turn something that
people react with like hot takes and
saying more irrelevant things and that
means that Quentin Pope can't really
respond to a hot take in any good way
and we don't get any kind of synthesis
out of this and this is the overall
inadequate equilibrium that I believe
that uh the AI safety skepticism is
mostly stuck in right now
and of course I should also point out
that this is my opinion and if you can't
buy uh which posts gets upvoted by this
wrong this is this is wrong like if you
investigate one argument you'll get very
few upvotes if you try to give this a
big picture
um uh disagreement this shallow big
picture disagreement you'll get a lot of
uploads and that's wrong
finally Quentin Pope uh
uh says he will endeavor to not use his
own research on chart Theory
um uh as the uh the argument why Elisa
is wrong because
um like uh
most people are not that much into shot
Theory as Quentin Pope
um but I actually find that a lot of the
arguments that Clinton Pope are making
does in fact rely substantially on the
intuitions from shot Theory or that like
from how humans form their values
so the first question
is will the current approaches scale to
AGI and
um yutawski apparently thinks not and
that was in fact what he was saying
before uh tpt4 was announced and he
hedged it a lot like uh he he does that
in general
um but um
uh like after uh gpt4 was announced then
of course he said he updated and said
it's smarter than he thought would scale
to
um and then of course afterwards this
criticism hit
um
I
don't actually think that
elizabethkowski still believes very
strongly that you can just scale it up
and you will get a AGI but mobile is
that there is some something hidden
somewhere still yet to be discovered
that will infect uh be the key that will
unlock some kind of room
um and Quentin Pope is actually even
more optimistic in in that the current
Paradigm he believes may very well scale
to Super intelligence and I I don't
actually have a good intuition for this
uh so uh my point with this is just that
like if Quentin Pope is more bullish on
the current Paradigm then that seems
like something that should increase the
probability of Doom and um
like if we are seeing when the last
curves go below the human level or
trying in different ways it looks like
we
it does to me seem like the the very
longest timelines like earlier culture
have argued for seem totally
unreasonable at this stage
so
while AI scaling will the alignment uh
work that we are doing
um continue to scale alongside
um
Quintin Pope is very optimistic
and reasons historically saying that
when capability advances work they
typically integrate well with current
alignment paradigm
and uh I disagree and I would like to
bring out this Old Post by Scott
Alexander which has a graph saying that
more people are killed by a falling
Furniture than by terrorism and the
reason why you get this very
counter-intuitive result is because you
start counting on September the 12th so
you don't actually get September 11th uh
and that means you get some uh like
really really misleading
um statistic and my point here is it
matters very much when you start and I
started at like the opposite time in the
sense that I started just when that we
had this relatively big
framework that completely
um relied on uh like
mathematics of utility functions
um and then the Deep learning Revolution
happened and all the alignment work that
had been done to this point all the
technical alignment Pro basically uh
vanished it was completely demolished
and didn't integrate at all with the
current Paradigm
um so I'm not saying necessarily that
Quentin Pope is wrong but it depends
very much on when you start to count and
I think it
um it is further Complicated by a uh
it's by I guess Quentin Pope and I would
have differences of opinion about what
is capability and what is alignment
research and this is just simply
difficult
uh
quitting Pope makes a claim that um
the capability advances that we're going
to see in the future they may not break
the existing alignment techniques
I think that is
um probably true as long as we stay
narrowly within the same Paradigm but
it's almost certainly gonna be false
when we
get something dramatically more
important like eventually we are going
to find something better than Auto
regressive language models like I can't
see us using this uh in 3000 years
now we get to the discussion of
generality and I want already here to
um
uh highlight that I find this really
difficult to pass I think the uh
description that
um Elisa uses is in in this podcast is
not very good he has written better in
in many other places
um and uh going into this is really
difficult uh is is to me it's just a
really difficult subject
um and I actually don't think generality
is the best framing of what the actual
problem is uh but that's what we're
using here so let's roll with it
says that if we were a full General we
would be as good as our programming as
we are at catching and throwing things
but we're not so we're not perfectly
General
Quinton replies that we're not really
suspect for throwing in a meaningful
sense we just have a general learning
algorithm and then buys that towards
learning how to throw
um
so if I
should say like this is really vague my
intuition is that throwing and catching
an object is actually really really
difficult in some kind of objective
sense
um and humans are really good at it and
the things that are required for coding
uh seems like things that somehow in a
information theoretic way or something
like that is actually easier and humans
are just really bad at them
um I don't actually think this is
something that influences the
probability of Doom very much like
obviously if you think that humans are
optimal or close to Optimal or near some
kind of ceiling or anything like that
that could be an argument why we should
have a lower probability of Doom based
on this but as far as I can see Quentin
Pope does not really make the last step
towards the argument that this is why we
should have a lower probability of Doom
that is a discussion about brain
architecture about like how simple and
general are Transformers compared to the
brain and we found Evolution found the
human brain by
shipping at Flint access and we find
Transformers by a language models but
they can do uh basically everything both
um
and uh I try to see like um
from this can we
try to uh
get any kind of information or intuition
about how um likely it is that there is
like a possibility that uh uh the
language models can become dramatically
more General soon rapidly in some kind
of room and uh I think that Quentin
believes based on this that there it is
unlikely that we'll get some big step of
generality
um but I couldn't like connect the dots
really from from quinton's writing
unfortunately
um yeah this is a
different I won't go into this I will
instead talk about the idea that AI can
be more General than humans uh in
um
uh argument where you can imagine an AI
just rewriting itself programming itself
to become better at coding or something
like that and Quentin believes that
actually powerful cognition comes from
simple learning processes applied to Pro
to complex data
um and that's a reasonable disagreement
I would just flag here that
um it is in his
um description says you can imagine
something and uh Quentin makes a much
much stronger argument that a powerful
cognition mostly comes from and I think
in this case
um because Quentin is making a much much
stronger argument he needs to provide
much better arguments much more evidence
for this statement
there's some discussion about like
humans can reprogram themselves if you
Define reprogrammers obtain new training
data I don't think that's actually how
most people think about reprogramming
um
if I think about reprogramming like I
think about an AI designing a new Ai and
in the limit that would be something
like you know matrushka brains and some
uh provably optimal Bayesian reasoners
and things like that and I think in that
case I do in fact believe that this kind
of reprogramming is something that is
possible and is something that is
potentially extremely powerful
the hosts in the bankless podcast ask
what is a super intelligence and Elijah
says it's something that can beat any
human and the entire human civilization
that all the cognitive tasks
Quentin Pope objects that this is too
high a bar we may get something
dangerous substantially earlier than
this
I don't actually disagree very much with
Quentin but I would like to point out
that iliakowski is answering the
question as far as I can see in the
totally standard accepted way
um
but I agree with Clinton that uh the
thing we actually care about
um I don't know if this is an agreement
is
um is below this and I believe in
particular it's the sixth strategic
cognitive tasks that bastron identifies
that are important but I think there is
some kind of generality here in that I
expect all six of the tasks to be solved
at roughly the same time because each of
them have been optimized with roughly
the same amount of uh G you know human
cognitive effort
this may in fact be a total nitpick I
don't know I am going back and forth
about whether this is actually an
important point or not
um what I would in fact say is that
between this slide and the previous
slide Elizabeth has been talking for
several minutes and going through some
of the arguments that I think are more
like core and I think
um
Quentin by choosing to engage with this
point rather than some of the others is
a
stepping substantially away from the
core Arguments for AI Doom
now we get to the width of Mind space
this is an old old old picture that Elia
saidkowski made I think back in
the early 2000s or something like that
and that comes up as the answer to the
question why would an AI not be like us
and Illy says that mind the space of
Minds is very very wide
um
Quentin objects in very strong terms
this is extremely misleading because
real world data does not look like
spheres in any meaningful sense
I think
at this point Elise rudkowski is just
talking about like the space of possible
minds and I think that could in fact
reasonably be a
um like a space in some higher Dimension
that is like
um continues there are obviously some
some limits to it but I think in general
you could imagine very very many kinds
of mines
uh and no reason why you uh this space
would not be like continuous
Quentin suggests rather that we use
eight different reference class and the
reference class that Quentin thinks that
we should think about is the space
distribution of powerful intelligences
that we could build in the near future
um and that's of course uh depending on
how strong your model are for how does
that look in fact uh like in the limit
if you have a really strong uh opinion
that it's gonna be gdpg5 then it's like
a distribution with one point
um
I
um Quinton says that this is more useful
for estimating the distance to the human
mind
um it's not really clear to me uh
notably
building an AI that is like issue in
mind is something that to me seems like
we are very unlikely to do in in the
near future
um
so that means that like then we punt the
question to like how humans like is uh
gbt4 and
from that question we want to figure out
like how human-like is uh dvt5 is going
to be or the version of gbt that will
eventually be of substantial strategic
interest
um I don't think that reference classes
is in fact a good way of looking at this
at all like
on the outside view you can't really say
very much like you need to actually look
into GT4 to say anything really
interesting about this and to get any
kind of knowledge like reference class 4
casting is something that I expect to
fail dramatically in this case
also this mindspace is white is
something that an argument that Elisa
used back before I certainly before the
Deep learning Revolution and we do in
fact now have some evidence like if we
try to
um just train a
an ultra regressive encoder on common
Coral what do we in fact get out of that
we know that now and we know that we get
something that seems clearly unaligned
um like and we've tried a number of
variations on this and we get something
that is on a light like obviously online
or we're doing reinforcement learning
from Human feedback because it's clearly
online
so in that sense the default expectation
if you just build an AI as an um
as a generative pre-trained Transformer
you are going to get something that is
online
next topic is strawberry alignment the
idea that the task of copying a
strawberry is something that we don't
know how to specify in a way that
doesn't involve the well-being
irreversibly damaged
this is also something that quentinoplex
to in the sense that the question seems
malformed to him in that humans can't be
made to only care about one thing
excuse me for mood
so
um the uh it's obviously not one thing
that iliacowski is talking about it's
two things
copying the strawberry and not
destroying the world and
not destroying the world is actually a
reasonably complex and thick thing
about that
uh if you try to like specify this in in
more details and think about that and
the question becomes if you also add in
alignment with human values and
something like that and tries to get it
aligned in a in a stronger sense
um does that make the problem more
difficult or less difficult
um Quentin is arguing that like we don't
even know if a value formation process
exists that allows you to specify
that you don't just want to copy a
strawberry
um Quentin argues further that we have
one value information process that we
know is compatible with our values and
that is of course our own and why would
we not want to use that and I think this
is close to the core of my applications
to Quentin's overall uh thoughts on
alignment
because I don't think that we actually
want an AI that is anywhere near to a
human we want something that is
courageable and and not something that
is human-like in any meaningful sense
um the orthogonality thesis uh as uh
given by Bostrom is that intelligence
and final goals are orthogonal can be
combined in any way and Quentin is
arguing that there is no orthogonality
thesis for alignment techniques this
could also be stated as not all
alignment techniques work for all final
goals
which I think I basically disagree with
basically agree with
um
deep learning is an example where we
have
um
a much easier time instilling a behavior
if we have a lot of examples of this
Behavior
um that is certainly true but I'm not
entirely sure that it is in fact
alignment this instilling Behavior
seems more related to
um capability and less uh
this true to my conceptual understanding
of alignment
um
precisely what is alignment and what is
like making the model do as you want is
an open question I don't I haven't
really seen a uh a good description like
clearly a number of people thinks that
reinforcement learning from Human
feedback fits into this
um and I think uh
I think most people would also agree
that like reinforcement learning as it
was done back in the 1970s by Minsky and
things like that that is probably that
was probably not alignment research that
they did but what precisely is alignment
is an open question
Quentin further argues that superhuman
tasks they are the things that don't
have any kind of examples and that means
that this kind of
behavior is not this kind of alignment
by a reinforcement learning on this
doesn't really work
um
I think it's a problem of capability and
we are not talking about alignment here
at all because eventually we will have
AIS that is capable in fact of doing
things like duplicating strawberries and
they will also be capable of doing of
taking over the world and we want them
to duplicate strawberries and not take
over the world and that is a problem
that we will need to solve
uh Quentin also has another statement
here that makes me
uh think that he and I have a different
conception of alignment in that he
doesn't want an alignment method that
lets us turn a language model into a
paperclip maximizer I see that as like a
capability issue and like it's to me
it's very very easy to turn gbt4 into a
a into a paper clip maximizer you
literally just give it a problem and say
you are a people clip maximizer what
action will you take and then you do
that and then you have a paperclip
maximizer so this is a problem whether
it can become a paperclip maximizer that
is actually able to make paper clips
it's a question of uh capability only
and not really alignment
will AIS be steerable by creating
descent
a counter example is perhaps natural
selection which Elise says it maximized
Inclusive fitness but did not really
cause us to care about uh Inclusive
fitness
here is an example of like how I think
of it uh in like meme form here is a
potato chip and here is a curve that you
can navigate on with gradient descent
and if you have a really strong model in
this
uh in this 3D space here that is
something that is very unlikely to
generalize for instance if you want to
generalize to taste then you are not
going to get even if you have a perfect
representation of a potato chip you're
not getting uh uh you're not getting
something out that actually tastes well
um and in the same way I feel uh well
Clinton disagrees and says basically
that we have no way of knowing and this
example that Elizabeth is giving tells
us precisely nothing about like how
difficult is alignment going to be
I think at a minimum it tells us that
inner misalignment is possible because
having like one example of inner
misalignment is something that is
certainly is some kind of evidence
I don't think that Elizabeth actually
cares very much about this argument he
uh uh doesn't think that gradient
descent is the same thing as natural
selection
um
so I feel
um Quentin Pope's uh counter argument
here kind of misses the mark a little I
think
would not find
um
that they're talking past each other to
some extent here
um
and ilyasuredkowski is in in some of the
next sentences talking about what he
thinks is actually really important what
his core argument is uh and again that's
what I feel Quinton should engage with
rather than with things that are kind of
like this but that eliakowski actually
thinks are quite different
how about the example of humans liking
ice cream that's according to
earlierkowski an example of value of Mis
generalization when we shifted to a
modern environment where we could have
ice cream
and Quentin here has a long alternative
description of how this came to be how
humans came to like ice cream
um and um it looks mostly identical I
think to what I imagine Elizabeth would
uh write I think one of the difference
is that Quentin's description it routes
explicitly through that you have to
actually eat the ice cream
um and uh to figure out that the ice
cream tastes good and I think this is a
an assumption that just obviously fails
for a more capable uh agent I think a
uh a super intelligence would not in
fact need to eat an ice cream to figure
out that it that it tastes good
Quentin Pope similarly has a suggestion
for avoiding this problem that is just
we won't reward the AI for taking bad
actions and like I'm confused here
um this is like uh like
misgeneralization is a problem when we
get to a new domain like going from
training to uh deployment where we can't
just choose not to reward the AI because
it may have taken over the world
um
so I am confused here and
I won't probably talk that much about it
speculates on the integrals of a gbt
like model
um he says I don't think duties can have
any sorts of inner desire
and
eleazkowski says they might in fact do I
notice here one thing that he said in
the in the podcast that
um
uh that question emits from his summary
is that he's talking about Duty like
models and uh says like things like gbt
7
um so
um
I think we should have a like I don't
know at what points we get substantial
inner goals where do we get Misa
optimizers
um I think my intuitions about what is
going in what is going to be happening
inside tbd7 are extremely weak I think
um
Quentin would need some kind of argument
for why he thinks there will be no inner
desire in GPT 7. I
I think it's really really hard just to
say anything with any kind of certainty
about that
Quentin notes that uh gbt uh thought
doesn't seem to have like an out of
desire for Simplicity it doesn't want to
be asked simple questions
um and that is of course true but it
still seems like goals are possible so
it must be possible for a a stronger
version of tpg to in fact have goals
uh the example he uses is humans we have
an inner goal that is we want to predict
visual stimuli
um but and asks have you ever even once
in your life thought anything remotely
like now I'd like to sit in a dark room
and I would in fact say that yes I have
thought about this I do uh every night
think yes now I would like to turn off
the light
and I think also
um I have uh thought this quite
Consciousness conscious in response to
overstimulation like now I need to like
leave the party for a moment and just go
to the bathroom and put some cold water
in my eyes and just relax uh like this
is I think quite normal
um but of course this kind of inner
processes haven't really taken over me
like in general the the article is what
is uh Stronger for me uh like the the
argument that this never happens I think
is wrong I think there are in fact a lot
of humans who have these kind of inner
processes that have taken over like in
cases of addiction or or like mental
diseases I think the the strong claim
that this is a thing that literally
never happens is almost certainly false
that is all for today thank you and see
you in two weeks