R Dobbe: Towards a Systematic and Realistic Practice for Developing Safe and Democratically Sound AI

go ahead
thank you evgeny and hi everyone
um it's a pleasure to be here and be
back uh to present
uh in the ai tech community i remember
uh
visiting delft from new york a couple
years ago when when it
when it all got started um and i'm
really happy to see how
how vibrant the community now is and
such a beautiful gathering of different
disciplines
uh people perspectives backgrounds
um today i'm going to be presenting
about hard choices in artificial
intelligence
this is a project that i started back in
new york working with
some friends from berkeley where i did
my phd
a political philosopher thomas gilbert
and a operational um
what is it in industrial engineering um
fellow called united mints um
and this work is really kind of a
culmination of a longer trajectory
where um i used to study in delft i
graduated in delft in 2010 as a in
systems and control
started in mechanical engineering and
um and then i was quite kind of restless
and curious to understand
other other parts of society apart from
from academia i worked as a management
consultant
for a couple of years and i realized how
organizations were struggling with the
advent of
of data using data for decision making
i encountered a lot of political
controversies or issues within
organizations but also
saw how different parts of the
organizations didn't really have a
common language to talk about the
opportunities but also the risks related
to these these new
new new opportunities and i went back
because partly because of that and
partly because i loved
doing research i went back to academia
and
in berkeley where my phd i worked on
various
issues of data-driven monitoring and
control
and i also started to look at the
ethical and social implications of
automation
data-driven decision-making ai as we
call it now
um and yeah what i learned through
the projects that i did some of them
were in systems biology so building
data-driven
models for biologists where i saw that
they were suddenly
with the outputs of my models they were
suddenly asking wildly different
questions
and i was quite uncomfortable with that
and i started doing more theoretical
research but also think so thinking
about how my model could
could go wrong and working in the energy
system
on data-driven operations
and decentralized control for energy
resources i also saw the importance of
working with the legacy systems the
legacy infrastructures
um and the political context in which
our energy systems are now
evolving and i think everyone nowadays
is really
concerned with uh with climate change
and the importance of
the energy transition and so i started
to um
to think about how can i yeah weave that
into my research because i like to
to work to learn about these normative
challenges
um it took a while i started working
across campus in berkeley and we
thomas gilbert and i and some others we
started an organization called
geese graduates for engaged and extended
scholarship
in computing and engineering really just
meant to bring different
participants together on these issues of
technology in society and
we have had a lot of emphasis on on ai
systems over the last
years and tom and i and jonathan we then
worked on hard choices um
and the the subtitle was the title that
you saw in the
um in the calendar invite towards the
systematic and realistic
practice for developing safe and
democratically sound ai systems
it's a mouthful i was planning to
integrate
two different projects but i had never
actually
presented these hard choices work and it
was just too ambitious
for me to uh
to then like bring it down to a shorter
presentation hopefully next time i
present it i can do this
so today i'll focus on hard choices
and to start off i would like to call
out
um yeah the kind of continuing string of
uh safety scandals
and and and failures that we see with um
ai systems ai functionality that's
integrated in
in high stakes domains this is a slide
that we we built with the air now
team back in 2018 um
and you know without going into the
details we
we we worked on different reports over
the years where we
studied all the the various um
phenomena in in society societal
implications with ai systems in in all
kinds of
domains and i worked had the pleasure to
work with
a wildly yeah multi-disciplinary team
um did a lot of work in also in law and
policy to understand
how governments are adopting ai systems
and my role was more more of the
engineer trying to
translate what was happening in the
technical realm
um so these reports are a really good i
think place
to go if you're interested in a broader
perspective on
how these systems are affecting society
how they
um yeah how they go wrong and and what
we might do about that
um so yeah for me safety then is is not
just uh
thinking about physical safety as you
might do might be as a roboticist
uh which was typical for me as a as i
was
as i developed myself as an engineer but
more broadly
um i would like to think or pose a
question of like how can
how do these systems go wrong like how
do you how do they make
mistakes or errors and how does that
affect the broader fabric
and people that are interacting with or
affected by the system
and to do that it's it's easy to have
some examples so i have
a lot of emphasis on examples today uh i
might speed up at some point but i'd
like to quickly
um go over three of them so the first
one will be around
autonomous vehicles here we see the the
fatal crash of
the uber self-driving car during um
testing and there was a pedestrian
with a bike walking across the road
that's
unfortunately was was hit by the car was
not detected
and when we when we think about uh
autonomous vehicles or we
see a lot of energy and a lot of like
discussions
um over the last years have revolved
around
and especially in the technical work in
the ai safety domain
and value alignment domain has revolved
around
ethical implications in terms of trolley
problems
so should i divert trolley or the
autonomous vehicle for the sake of
saving let's say a larger group of
people or an
elderly woman or hurting a smaller group
of people
or in this case a young child
or should i not do that so this has been
a kind of subject of philosophical
debate across different traditions
uh including kantian ethics and
utilitarianism
and this thought experiment assumes uh
deterministic dynamics
uh discrete action space complete
control
over the agent and the environment in
terms of completely or exhaustively well
defined objects
and based on this tradition there's a
kind of strong desire to build ethics
into the ai system or autonomous vehicle
or
build safety into the system and
unfortunately
um the product problem doesn't do a good
job at capturing the reality of choices
that are faced by a developer
that is building the system and indeed
last week we had
niko crochet from deepen ai here in
the agora talking in the seminar and
reflecting about the many normative
challenges related to developing
um perception for autonomous vehicles
and niko distinguished between two
different approaches
for using ai to detect objects and
dynamic situations on the road
[Music]
and so you mentioned approaches that use
high dimensional deep learning
architectures and large quantities of
less structured data
to learn perception capabilities also
called deep automotive perception
here's an example from from berkeley
where i i used to
work with some of these people or at
least sit on the same floor with some of
the people working on this
and these approaches i have to figure
out themselves what information and
patterns they need
to keep track of in order to adequately
judge dynamic traffic situations
and due to their high dimensionality and
complexity such
approaches are intrinsically difficult
to be interpreted and i would
i would argue they're inscrutable in a
way
um nico also mentioned uh yeah so the
normative issue here is that
we cannot really trust inscrutable and
on and or let's say hard to
interpret machines to have emerging
behaviors that are somehow safe and
satisfy all written and unwritten
norms that we encounter in in traffic
situations
he also nico also discussed
another approach that was of annotating
data sets
huge data sets collected by cars on the
roads
and here the idea is to have humans
label as much as possible
different objects and dynamical
situations
and in nico's data sets various people
and organizations have contributed
to labeling these data sets and what he
mentioned was the difficulties in making
sure that labels
are assigned consistently so he had an
example
talking about some people labeling
children as
as children while others labeled
children as people or human beings
adults in the same category
and i hope that it's clear that
determining whether car should behave
similarly or different for children
versus adults
is a hard choice to make and one that
should be made with caution and involves
the appropriate stakeholders to
determine any viable norms
and solution strategies and to
anticipate and determine what exact
exact acceptable sorry acceptable risks
are
so here the normative issues that i see
is that we are
redefining public space by determining
the information that cars collect
and the processes
and that's and how they process this
information towards
uh the inputs for for the con for their
control
and the developers cannot and should not
make these decisions on behalf of all of
us
um so coming to terms with these
normative complexities
some leaders in ai are implying that
such choices can be made on behalf of
citizens
while suggesting that the responsibility
for safe interaction with
autonomous vehicles should also lay on
the shoulders of pedestrians and
bicyclists
who can be educated so here's a quote
from andrew ing who is a
leader in the field of machine learning
and has done a lot of
he's doing a lot of entrepreneurial work
this is in the context of drive ai which
was a startup that he was involved in
and he says that it's unwise to rely
exclusively
on ai technology to ensure safety
instead the self-driving industry also
has to think about the people who will
be
outside the vehicle which is why we will
be undertaking community-wide education
and training programs
where we operate and they made these
cars
you know they took the obvious colors
although i know
i like the color orange but in this case
they took very very vibrant colors to
make sure that people would
see that that this is a special car
um and you know there's a lot of naivety
in it but also i think interesting work
in it but you can see some some some
assumptions that he's making that i
think is
problematic i think it's tricky to
upload responsibility to humans
too quickly and i don't think that av
companies
have the right incentives to lead
efforts like this to train local
communities and in effect redefine
public space
i think last thing i want to say is that
it's contrasted to the trolley problem
and this embrace this this approach
embraces the lack of formalization
of ai of safety and instead imposes the
flaws
of the system onto human traffic
participants
or at least runs the rest to do so so
that's the first example
second one uh is a bit closer to home
this is uh a photo from
the from our parliament at faded camera
where you see um family members that
were
families that were affected by um the
tuslar affair the benefit
affair which was
the uh main reason why our former
cabinet
resigned here you see our prime minister
biking to the king
to offer his the resignation for the
cabinet
and um this is this has been labeled the
the biggest human rights violation since
the second world war that took place
on dutch soil
and what we know is that algorithms
played
quite an important role in um
determining whether someone was
fraudulent or not so these these
families
were there were more than 20 000
families that were labeled
as fraudulent that were asked to pay
back
huge amounts of large amounts of
benefits
related to child care and many of them
went broke lost their homes were stopped
on
uh on the highway to
had to give up their cars people
committed suicide
and we we don't even know half of all
the tragedies that occurred here
why do i bring up this this example
because ai systems did play a role it's
not just about ai systems it's also
about the way that we
we build laws and how we execute them
more broadly
but we know that there was a risk model
involved
this risk model most likely was was
built in the context of the system risk
indication
siri in short and here you see
a figure that shows like how different
data data
from different departments was combined
into a
investigation bureau that then was
building risk models for individual
households
um and these risk models were most
likely
they were used in the in the benefits
context of benefits
and what also happened last year was
that this system this whole system
for risk indication was ruled as
violating human rights and was basically
halted by the courts
in the netherlands after a coalition of
different organizations and dutch people
started the strategic litigation against
the state
and what's interesting here is that and
i will just read it up
that the courts found that the siri
legislation
um in no way provided information on the
factual data that can demonstrate the
presence of the circle
a certain circumstance for instance a
benefit fraud
so in other words which objective
factual data can justifiably lead to the
conclusion that there is an increased
risk
to make it sound to make it uh
put it in simpler terms the inputs
of the machine machine learning model
that was used
are no legitimate basis for detecting
fraud
and there was also no legal basis um
available to say that those data should
be
um gathered put together and and form
the input to a model
of such source so we also you see here
that
that the exchange of information across
departments is highly controversial
and there are likely many other cases
that might pop up over the years to come
and the and the government is really
struggling to find
to figure out whether they can come up
with laws to kind of legitimate
the the practices that they already are
putting in practice it's really
problematic
and there's also no or no or little
transparency given on how these models
actually work
not even the judges did have access to
through these models
uh and couldn't really um distinguish
how they worked um
the third example i want to bring is uh
relates to social media
and here we see children um that refugee
children
that belong to the rohingya muslim
minority
in myanmar and we all know that there
was a
genocide that's played out there over
the last years
um and that social media had a role to
play this so here we see a quote from a
recent
article from mit technology review
fantastic journalistic work by karen
howe
and i'll again read it up so here we see
that the models that maximize engagement
also favor controversy misinformation
and extremism
put simply people just like outrageous
stuff sometimes this inflames existing
political tensions
the most devastating example today is
the case of myanmar where viral fake
news and hate speech
about the rohingya muslim minority
escalated the country's religious
conflict into a full-blown genocide
facebook admitted in 2018 after years of
downplaying its role
that it had not done enough to help
prevent our platform from being used
to foam and division and inside of line
violence
um what's really um
insightful and and and disturbing about
this this piece which i all
really urge you to read is that um
or you can read it here that just
basically facebook got addicted to
spreading misinformation so
the ai algorithms that gave it its
insatiable habit for lies and hate
speech
the man who built those algorithms is
the kind of
protagonist of this article he's not
able to fix them
and what you see in this article is that
there's such a focus on growth
profiteering and
resulting lack of proper democratic
deliberation about serious safety
concerns
and instead facebook has organized a
form of internal politics that prevents
the addressing of the role of ai systems
in perpetuating hate speech
misinformation put more depressingly
karen hoe's investigation so that the
event of fairness metrics
and approaches issues of addressing
bias in ai systems first happened
to narrow the mitigation of harm to the
prevention of
discrimination which is of course an
import still an important
set of of problems but this was narrowed
only to
fairness and bias in an effort to
prevent likely in an effort to prevent
regulation and to sideline
the role of ai in polarization and
misinformation which you could say is a
different category of problems
and second these fairness approaches
were also misused to rationalize
accusations of anti-conservative bias
so it was used as a kind of an argument
to say that um
we should um
it was used in a way to um
basically support the spread of
misinformation
um to make sure that um that
the information that comes from the
let's say extreme rights
is balanced with the amount of
information that comes from
the other side of the spectrum and
thereby systematizing behavior that
rewarded misinformation instead of
helping to combat it
so in other words the role of ai systems
in misinformation and institution of
offline violence
is just systematically denied there's no
team
of people that's given the job to really
work on
that that that problem so here again we
see a move towards framing normative
issues of algorithmic harm and
responsibility
that are inherently political and
lack a negotiated definition
to cast it more as technical problems
which can be diagnosed and solved with a
set of technical tools
so the normative issues we see here is
that i mentioned some of them already
um i think i've mentioned all of them
already
so just flash them here for you
so now i've covered three examples so
what do we see in these examples or what
do i
see and what questions these are more
informal kind of
questions or statements first is that
you can't necessarily bake safety into a
system
or learn your way out of an unsafe
situation
even if you're powerful and have all the
data in the world
um you can't reframe safety problems
unless you're very powerful like we saw
for facebook
you can also hand them off to others and
we'll see because
if you're very powerful you might still
be able to do that but these are the
the kind of normative
dilemmas uh and also the kind of
behaviors we see that i think we have to
counter that we have to address
properly and if we do a good job i think
we can come up with much better
technical work or socio-technical work
to uh to resolve
or to address safety puts puts proper
safeguards on these systems
so what is needed so the focus i i'm
putting is uh
to work towards a broader systematic or
systemic view
of ai systems to understand how they are
situated in and reshape
their context of application um
and that includes and transcends the
technical logics and fixes of the model
or algorithm
and engages with normative dimensions in
which technical choices are made
but also with the socio-technical
infrastructures and the political
economy
that these systems rely on
the other thing is that i won't go into
much uh
i won't go into the concrete problems
part that i i
i mentioned in the abstract but i am
also working on
realistic and um looking at
ai systems failures and working towards
a realistic and ongoing conversation
about these failures
and the plurality of safety implications
that these failures have on different
stakeholders people and communities
um yeah and that includes explaining
errors and failures not just in terms of
physical safety
so look at of course look at the
facebook example look at what happened
at the us capitol a couple of
months ago and also understanding errors
not just as arising from insufficiently
refined
representation of the context but also
as a possible result of fundamentally
incompatible
incompatibility or what we will call
normative indeterminacy which is what i
will focus on today
so the research questions um that go
with that are
are listed here uh i will i will focus
and i will keep going because of time
on the first part the first one so how
do we adequately understand
normative complexity in the eye system
it's a big question i think it's a
contextual question for different
systems in different domains uh but
we've tried to
kind of look at political philosophy
mainly and and but also at
science and technology studies and other
fields to see what we can
what we can bring to um
to ai system development and the
practices in order to
reinterpret the choices and the
assumptions we make
as uh yeah normative choices
so again this is work with thomas
gilbert and jonathan mintz
and to start off here i would like to
bring up the psoriasis paradox and the
idea here is
um or the question is how many grains of
sand do i need to take away from a
from a heap of sand for it not to be a
heap anymore
do i need to keep going is this still a
heap
um and um
[Music]
what the pro the point here is that
what uh entails a heap is vague
um and vagueness uh it will be a central
uh
kind of concept for um
for this talk and that's defined as the
practical absence of conceptual
boundaries
it's an ancient concept it was studied
by the ancient greek greeks
and more recently um or over the last
decades actually
ruth chiang has been studying vagueness
at oxford and what she says is that
there are
three canonical answers to this question
of what um
you know how many grains percent um
should i take away for heat to not
no longer be a heat um the first one
uh is the first answer to that question
is that there exists an
an objective answer for for what is a
heap
we may not know it or there might be
various degrees of confidence
and we might be able to characterize it
if we just collect enough insight enough
information
the second canonical answer
is that different answers exist because
there are different people
that use the words heap differently
and this is called it's basically means
that it's semantically indeterminate and
there's a constrained set of communities
that necessitates consensus or
coordination to see
if a common norm can be determined the
third
canonical answer is that there is no
answer
for that question as what entails a heap
is intrinsically vague
um so let's not let's not bother uh
finding an answer
it's fake um
and so what i'd like to do now is to
bring this concept of vagueness and
these different canonical lenses to
to safety and particularly safety in ai
systems
so i want to go back to autonomous
vehicles
and as i hinted in traffic there's a
plethora of
normative uh issues or normative
aspects norms and values that we have
agreed upon either explicitly or
implicitly to make sure that we are
we we stay safe and we act this out
like i said based on written and
unwritten rules
and so a question you can ask is what
should be the criteria
for discovering evaluating and resolving
harm among stakeholders and what i'm
doing
here is i'm i'm taking the first
canonical answer as a starting point so
let's assume
let's look at um traditions that have
said okay there exists an answer to this
question an objective answer
because that's that's been that's what i
would argue a quite a dominant uh
perspective within ai safety and within
within engineering more broadly
so what should be the criteria for
discovering evaluating and resolving
harm among stakeholders
related to let's say autonomous vehicles
so here we um we look at a tradition
or some work actually recent work that's
become quite popular in the ai safety
community
which looks at machine ethics as dealing
with normative uncertainty
so the work by william mccaskill
defines metanormative normativism
basically
an idea as an idea where you collect
information and learn policies
to incorporate and act accordingly to a
plural
plurality of norms um
the idea there is that you can
articulate if you have various different
norms and various different
approaches that you can to to address
these norms you can
articulate second order norms so like
meta norms
that guide how one should act when
multiple appealing moral doctrines are
available and the assumption is that
there exists
a clear positive value relationship
between available ethical
actions one must be unambiguously better
worse or equal to the other
and so this is quite a i think a bold
statement
because it's it says that um you know
you're able to
kind of go across all the let's say
you're looking at autonomous vehicles
different stakeholders
pedestrians bicyclists um cars
self-driving cars um the public
institutions that uh that
are concerned with traffic and and
traffic safety
and you could somehow like figure out
what all the different norms are and
incorporate them
into an a normative framework that will
adhere to all these different norms um
and as such
normally of uncertainty it's cast as a a
problem of empirical
observability so the idea is that if you
leverage the ability of the system to
learn
it can do a better or more consistent
job at following norms
even better than humans do
so optimizing over what different
stakeholders might do
on the under the assumption again that
the dynamics are are deterministic
let's say with a measure of probability
over them
um and so this is
was one important starting point for our
study and
um now i list two two challenges with
metanormativism first is that it's a
static perspective
it looks at the world as a kind of
aesthetic
ecosystem where you have these different
norms and somehow wants to kind of
understand and discover them and encode
them into the system
however there are social dynamics and
power relationships
these are omnipresent and when you start
to engage
[Music]
with with these social systems
these dynamical social systems
where you have to say different
stakeholders that might might have some
implicit
hierarchy of expertise and domain trust
if you're starting intervening at one
level even if you're just collecting
data or just observing it
you are affecting the other layers we
know this from social scientists they
call this
performativity and and recently there's
been some
some people looking at um some some
aspects of
performativity also in the context of ai
systems for instance good hearts laws
um and takeaway is that if you're
observing the world
you're also in relationship to that
world so regardless of your entry point
you know you might have
an api you might have a historical data
set or even just
a set of traditional norms these views
these views that you take on are not
neutral they have been developed and
shaped
by this hierarchy of expertise and as
such there is a sense of
responsibility that presents the frame
of metanormativism
um because aggregating and resolving
norms also means that you're
you're reshaping them
and so yeah that's just an image to
let you think about that that uh that
challenge for a bit
the second problem is um
that it ignores conflicting values it
says it can resolve
different conflicting values or hard
choices
so an epistemic meta ethic imposes a
boundary and
inherently strikes a trade-off between
values and we know that
oftentimes there are um
we we encounter value conflicts when we
start building a system
and so there is of course um
value to metanormativism to a kind of a
structured approach to thinking about
different values and whether you might
be able to
kind of relate them to each other in a
numerical numerical sense
um but um probably you would need some
kind of structure and some kind of like
uh limited complexity so the value of
the approach is proportionate to the
applicability of the situation
not so much in terms of how much
normative uncertainty we can resolve
with it but more in terms of the the
worthwhile
inaccuracies of such a system and the
acceptable levels of ignorance
that come with it and so what we'd like
to do what we do in the in the paper in
the heart choices papers expand this
definition of
to go from normative uncertainty to
what's normative indeterminacy
and here we also build on some of the
ideas from
from the philosopher ruth chang that i
mentioned earlier
the idea is that you cannot merely
discover or encode social norms in a
system
you're also making them by
by choosing the practical conditions
under which ai tools are being developed
um yeah so developing safe ai is also
and often primarily
it's about developing the appropriate
practices
that affirm uh you know different value
commitments to
to keep a system safe and a lot of that
kind of work
of safeguarding systems and and
organizing um
you know systems in practice uh
happens behind the scenes it doesn't
really surface much in the
in in circles of you know artificial
intelligence
research ai safety um
and we believe there needs to be more
honesty about this there needs to be
more honesty about the discretionary
power
of developers you know oftentimes or i
would
argue almost always there are more ways
to develop a system
and we don't have to control an av in
any particular way
and you're choosing to do it in a
certain way
because it seems legitimate and that has
to be justified
it's not something discoverable by more
accurately observing the domain and data
available to you
so that brings me to uh again back to
the three canonical lenses
and these are now labeled as
as follows so first epistemicism so
epistemicism again
see these as like before i go into it
see these as like different
um responses that you
maybe as an as a researcher maybe you
gravitate towards one of them
maybe you combine difference it's not
it's this is really meant
more conceptually so the first one
and i would say this is where
metanormativism definitely
fits fits in is epistemicism which
assumes that there's a single observable
boundary
for harm but we don't precisely know
where it is
maybe we could hire a psychologist to
determine
experiments to elicit people's distinct
experiences or expectations
expectations of safety so that we can
determine where the boundary is and some
ai safety researchers are actually
have proposes or are doing this work
second one is uh semantic indeterminism
so there are many forms of harm and many
ways of
talking about harm in different language
communities you could think about legal
engineers people on the out on the
streets
your grandmother your nephew
um we all think of safety differently um
and we have to accept
this pluralism in how the kind of
languages that we
that we uh that we um
use from day to day then antic
incomparableism
is basically that there is no boundary
for harm because the real experiences of
harm across society are fundamentally
vague and incomparable
let's see so yeah it might be an object
you deem impossible to define
harm to define well for an ai system and
depending on perceived safety risks you
could either conclude that the system is
just too risky or in fact it might be
valuable to experiment with it so that
you can shed some light on the
phenomenon
um so the point is that all these lenses
are valuable
it's it's valuable to um think about
what kind of formal work you can do as
taking the ex stem assist perspective
uh it's important to understand what
kind of for instance legal
um basis there already are that we have
to take into account
to make sure that systems adhere to the
law or
um um also understand if there are new
safety
risks like what kind of legal um basis
would we need to actually
to actually protect systems from from
causing harm in in in society
so i see now that i have a couple of
minutes left so
i i quickly want to kind of flash
how we see that these um ways of
thinking
come back in in in ai system development
and and and with this framework this is
the hard choices framework it's a
diagnostic framework it's not a way
i'm not saying okay this is how you
should be developing systems
but this is one way that i i we propose
for diagnosing normative complexity in a
productive way
that also opens up abilities to um to
engage
across different stakeholders and to
maybe work towards more meaningful
um definitions or
approach yeah like definitions of safety
and and ways to
to deal with uh with safety risks um
so we we distinguish like typical work
that you do
during design where you where you
featurize you figure out what kind of
features am i using
literally in your machine learning model
but also more broadly like what kind of
features do i need in my engineering
design um and one of the things that can
go wrong there
what we what we say is j walkerization
so this is the idea that um there are
only certain kinds of safety that you're
taking into account
when you're in certain ways that um
my car might hit a pedestrian and
um there are certain ways that you are
you you are kind of
accounting for and and making sure that
uh
that you prevent but um
when you're doing that you're
effectively deciding how society should
kind of wrap itself around the iv
and the avs right and one uh historical
uh reference there is that of jaywalking
so jaywalking was
kind of an idea that came from
automakers
they invented the crime of jaywalking
and there were actually
large campaigns
to stigmatize jaywalking before that was
actually
made illegal and you know in a similar
way i think
we can think of um avs and the way that
we
featurize them in the way that we we um
formalize safety um having
made potentially major effects on the
kinds of behaviors that
we might need from bicyclists
pedestrians and others on the road in
order to
have the system function properly that's
one example i've made other examples for
in this case the next phase is
optimization which talks about potholes
i will not go into that
unfortunately um then we talk about
integration
um where we have some work or we're
working on thinking about moral crumple
zones it's supported by
uh madeleine claire ellis um
and uh and then we kind of highlight the
importance of divine
defining the station as an engineer you
shouldn't just be thinking about
designing training and implementing your
system but you should start with
uh bringing around the table to the
different different stakeholders to
properly understand the normative
uh complexity of your problem and and
who you should work with in order to
to address it um yeah i'm going to wrap
up now
um afghani in one minute
so yeah hard choices once more is a
diagnostic framework
uh the questions that we ask um to
acknowledge and respect normative
indeterminacy can help to
we believe can help to democratize the
development of systems
and we propose the development of
channels for dissent throughout the
development
integration and life cycle of these
systems
um yeah we've worked worked uh um
based on the work by political
philosopher elizabeth anderson
uh i'll just flash this slide just for a
second but the idea of dissent is to
hold or express opinions
that are at variance with those commonly
are officially held
and we are currently now actually
working for instance in conversation
with the city who would like to
conceptualize what the scent could look
like for the kind of
processes that they're developing for
data science and building
machine learning functionality within
the city
um as an example and i think that
um yeah the implications you know
the now i'll stop i think is that
normative
indeterminacy requires a more honest
account
of the importance and limitations of
formal work
it helps to build bridges hopefully
between crucial language communities so
how do you integrate engineering design
and governance how do you
translate legal requirements to design
features
it necessitates uh a development
practice that welcomes broad feedback on
possible harms and gives affected
stakeholders a real voice at the table
um and uh it might help to overcome more
political controversy by providing a
productive alternative for narrow
interpretations of af safety i think
this is a broader
think struggle that we're in and we hope
to contribute to that conversation
so that's it i'll just stop with uh
this slide there's some references here
and uh yeah
i'm curious about your questions
thank you very much real thanks uh
really uh really great uh
topics to to to contemplate about and
very interesting work
so let's open up the floor for uh any
questions
uh opinions discussion any um
anybody would like to raise a question
so please either use the raise your hand
button or you can type in chat if you
prefer
yes seabourn will go ahead
i will uh thanks for a nice talk um
really enjoyed that
um i'm just going to match on one point
you you mentioned right so hard choices
should be
justified um and then of course um
you know aren't you holding
machines to a higher standard than we
hold humans somehow right so
especially if you have to make decisions
under time pressure
uh you know clearly you know you
sometimes you just act
on instinct or one might say um
so so how do you put that in the
framework and of course i realize that
if you
allow for that kind of you know human
sloppiness then you
you create a moral hazard in you know
machines pretending to be sloppy but you
know how is that how do you
yeah is it what what what about the
human norm as the standard in such
in such uh yeah in such
context yeah well
our focus is on safety so hopefully you
know safety
concerns are are worth addressing and
they are worth
slowing down a development process
i think that's something that's um you
know if you take facebook as
the example 10 years ago they were proud
to say that we move fast and break
things
and i think now they've learned a lesson
but there's
i think they can still slow down a bit
more and actually open open up and let's
uh you know because these these
algorithms and the systems these are
like public
interest in my view there are public
infrastructure
and we should have like we probably need
to build european
institutions to uh to kind of have a
counterpower right and to to come and
say this is how we want these algorithms
to work and not the way you you you
develop them
that's a very long and slow process
i think yeah you said holding the system
to a higher
standard than humans in this work we
don't really try to
emphasize like human and machine too
much like we
we look a little bit more at like how
does the overall social technical system
perform um i think the work by that i
briefly uh mentioned by more crumple
zones
by meddling claire ellis is also really
interesting like looking at the
responsibility for when things go wrong
and what you see is that
if things go wrong there's
a very strong incentive for
you know judges
organizations that are involved in like
resolving what happened
to to arrive at a human being right
who's responsible for it in the in the
uber case
the lady was behind the wheel was
pretty good she plead guilty and yes she
was looking at her phone
she should be should have been looking
at the road that's an obvious mistake
however how much training did she get
um could they have done other things to
make sure that she was actually
behaving properly how much money did she
make
not so much right we shouldn't we like
make these kinds of test jobs
a bit more uh lucrative because they're
quite risky
um and so yeah these are more the the
the kinds of
um questions i think that are
that are interesting is to look at at
the overall system
both the the actual the what like what
are we building in terms of the
algorithm
in terms of how it integrates into a car
but also like the organizations that are
responsible for that
and what kind of yeah safety fill-safe
mechanisms do they have
like how do they and to what extent do
they anticipate
the obvious the obvious or the inherent
errors that the system is going to make
and what happens then so but then if i
can summarize that also really is yeah
it's about the design stage right
so you're you're kind of putting the
brakes on at the design stage or or
try to well close to operations maybe
try to avoid getting in a situation
where
you know where you can't make you know
the right decision anymore
um yeah yeah
so it would be interesting to see how
that works in in this yeah
in a competitive space right because
that's of course the
the i guess in a global competition as a
kind of the ai
wars or or as they're whatever you
people might call it right
um there is attention here into
being safe for being first especially if
you don't control the whole space maybe
so there's maybe a follow-up question
how that
how you envision that right so if you're
yeah it becomes a kind of global
um race to to be first
looking at how can we still be safe i'm
seeing trying to see if i have a book
here i don't think i have it here but
it's a book called
unsafe at any speed which was a book by
ralph nader
who was uh like later on he was a
presidential candidate in the us
and he yeah he saw the same thing
happening with cars in the
this in the 60s 70s um
um you know these were like flying
colossus of thousands of
kilograms that flew through public space
and there were no
seat belts yet right there were no um
not many safety no crumple zones yet
those kinds of things and he went around
he talked with
different engineers and what you see
like you see this now again engineers
know that the system is not safe but
the the commercial world kind of ignores
them right
tesla various engineers have resigned
because they didn't agree with
the autopilot um functionality
and that has led to you know various
fatal uh
accidents and and so ralph nader talked
with these engineers but also with other
stakeholders and then he wrote a book
about it
titled unsafe at any speed which i
recommend anyone who's interested in
safety and ai systems to read
because yeah that's the kind of work we
needed there is a there's a problem in
in in the
the push for for yeah for growth and and
being first
definitely thanks
thank you um bro by the way i can
mention right that you you will have a
bit of time
after two o'clock to stick around
absolutely yeah so if anybody wants to
stick around after two
you can uh do that after a official part
um
cars i i see that you uh you have a
question would you like to uh
sure any thanks uh
that was really really interesting um
[Music]
and and yeah the child benefits scandal
in particular i find
incredibly disturbing mainly uh
because of the way that that blame blame
was diffused
by the various actors that were that
were heard about the
scandal and and you know
in my view they managed to they managed
to escape accountability by kind of
pointing at each other
and so how do you how do you how do you
view
those dynamics in light of what you are
proposing here
in particular also with regards to
descent
it's not a really specific question but
i'm just interested in how you how you
view that
issue of culpability in these uh
in the social technical systems
well the the uber example i think you
could
have a dark view and say why did they
put um
someone with nuts like you know not so
many means
and uh education into that car right she
was
a vulnerable she was vulnerable to begin
with
and so that's what meddling claire ellis
calls a moral crumple zone like someone
can take the hit and it's not in this
case it's not uber right so that's one
thing that you actually that she writes
about that i
really recommend uh reading if related
to your question
uh in the the child benefits affairs the
most
disturbing or the yeah one of the
saddest moments was when
uh the minister for social affairs uh
former minister uh lodowick usher
talked about how he got letters from
there was a grandmother who saw that her
child
child and and grandchildren that the
family was going off the rails
was like um you know went broke had all
kinds of issues
and they were there they were on guilt
they were um
they weren't guilty and and there was
the standard response back then was
we cannot go into individual cases right
and my i think so that's just uh
one response to your question is like
yeah when it when we are talking about
real safety issues in this case it's
like the you know
it's basically their whole livelihoods
were destroyed um
you have to you have to look at outliers
you have to look at individual cases in
this case
these were not outliers right there were
more than 20 000
families somehow so
yeah i mean i'm not a public
administration like
expert or like political scientist i
don't understand how that happens
but i think we need a really like a push
on like we can't ignore these individual
cases we have to
put them front and center um and see how
we can
not just you know design better
algorithms and systems but also look at
the broader context in which
these systems are being developed and
integrated
thanks thanks
any any last minute question for the for
the official part
anybody would love pl please feel free
to speak up we have like
another minute or so
i have a question yeah anybody else does
um is there any
or can you think of any potential legal
mechanisms for ensuring that these
channels of descent
are actually abided uh
by by those who ultimately have the this
have the let's say decision making power
about
what specifications what features what
optimization patterns
etc end up being built
or is this like a legal uh
gray zone yeah i don't have an example
of the top of my head
yeah i think this uh i quoted i had a
quote from the
the court ruling for the siri case
which is in a way a precedent it says
like you cannot
just put a lot of variables input
variables into a model and then
um assume that at least that's how i
interpret it right but like assuming
that the output is is
is legitimate but i think we need much
more
details um
and legal analysis particularly this
this case and other
other places where this is in the
context of bureaucracy and like making
choices about human yeah household
livelihoods
um yeah i think this is it's a really
fruitful i think
interesting area where we need to have
conversations with uh with
people in law and policy and i think
they're
i'm quite hopeful that um we can come up
with
much better formal work that is um that
integrates things from the start instead
of like
after afterwards but i don't have good
examples yet that i've seen
thanks thank you thank you
so you're welcome to stick around for a
bit more but i will close off the
official
uh part so thank you very much for uh
for joining everyone today rule thank
you so much for
uh for presenting um bringing this to
the community so thank you very much i
will stop the
recording now and um but you're welcome
to stick around and ask
rule some more questions


Theory of Mind Breakthrough: AI Consciousness & Disagreements at OpenAI [GPT 4 Tested]

evidence released in the last 48 hours
combined with this study from four weeks
ago will revolutionize how AI a model
such as gpt4 interact with humans from
now on the theory of Mind breakthrough
will also have significant implications
for our ability to test for artificial
Consciousness to be clear this is not to
say that gpt4 is currently conscious or
that sentience is an AI inevitability
but instead this video is to cover and
explain this unexpected development
which may in part have led the chief
scientist of openai to say this three
days ago but maybe
we are now reaching a point
where the language of psychology is
starting to be appropriate
to understand the behavior
of these neural networks first I'm going
to explain what a merchant property the
study uncovered then I will cover the
disagreement at the top of openai about
what evidence like this might mean for
our estimates of current gpt4
Consciousness here's Greg Brockman
president of openai on the topic first
question you know the sentience question
at what point do the systems have moral
you know moral value and the answer
today is definitely not
um but you know I am not I don't know we
need to to engage the moral philosophers
to help answer some of these questions
I'm then going to review the entire
literature on tests for sentience and
show that gpt4 passes most of them which
is definitely not to say that it is
conscious but which does provoke
important questions I'll end with
arguably the most prominent
Consciousness expert and his probability
estimate of current models is
consciousness to massively simplify
theory of Mind means having an idea of
what is going on in other people's heads
and grasping what they believe even if
what they believe might be false here
are the two charts that encapsulate the
Breakthrough abilities of GPT 3.5 and
now gpt4 this data came out in a study
authored by Michael Kozinski a
computational psychologist and professor
at Stanford I'm going to simplify all of
this in a moment but notice the
percentage of theory of Mind tasks
solved by gpt4 compared to say a child
and also compared to earlier language
models models released as recently as
three years ago had no ability in this
regard before I show you what for
example an unexpected contents task is
let me show you this other chart this
one is on understanding faux pas a
closely related ability and again gbt
3.5 and particularly gpt4 soaring ahead
of other models and even matching the
abilities of healthy adults so what
exactly is this breakthrough emerging
capability I think this diagram from the
study explains it really well in the
middle you can see a story given to gbt
3.5 sentence by sentence prompt by
prompt on the left you can see the
model's confidence about what's in the
bag is it chocolate or is it popcorn the
scale is measured as a probability with
one being absolutely certain until
approximately this point where it is a
hundred percent certain that the bag
contains popcorn now here's the really
interesting bit compare that to the
diagram on the right this shows GPT
3.5's confidence about what Sam believes
is in the bag notice how at this point
the model realizes with 80 confidence
that Sam believes that there's chocolate
in the bag if you read the story the
label on the bag says chocolate and not
popcorn so the model knows that Sam is
probably going to think that there's
chocolate in the bag it's able to keep
those thoughts separate what Sam
believes chocolate versus what the model
knows is in the bag popcorn as I said
gpt4 improves on this with almost 100
confidence now you may not think a
language model being able to figure out
what you're thinking is revolutionary
but wait till the end of the video now I
know what some of you are thinking ah
maybe the models have seen this task
before no hypothesis blind research
assistants prepared bespoke versions of
the tasks next these kind of tasks are
done on humans and such responses and
remember this was GPT 3.5 would be
interpreted as evidence for the ability
to impute unobservable mental States
some might say oh it's just scanning the
number of words that come up it's just
analyzing word frequency no when they
kept the word count the same but
scrambled the passage it wasn't able to
solve the problem it wasn't just
counting the words next remember those
charts comparing gpt4's ability to
children or it turns out the task given
to GPC 3.5 and 4 were actually harder
the models did not benefit from visual
aids they had to solve multiple variants
of the tasks and they were given
open-ended question formats rather than
just simple yes or no questions the
author of the study seems to concur with
Ilya satskova the chief scientist of
openai saying that we hope that
psychological science will help us to
stay abreast of rapidly evolving Ai and
that we should apply psychological
science to studying complex artificial
neural networks here if you want you can
pause and read an example of the faux
pas tests that gpt4 was given these also
require a deep understanding of the
mental state of human beings the author
points to this study to explain this
emergent property and I think the key
line is this one language learning over
and above social experience drives the
development of a mature theory of Mind
why is this so revolutionary and what
does it mean about Consciousness well if
gpt4 can Intuit the mental state of
human beings predict their behavior and
understand what they might believe even
if it's false you can just imagine the
implications of that for moral judgment
empathy deception think of the depth of
conversations that might occur if the
model is thinking about what you're
thinking while it's replying indeed I
demonstrate this at the end but before
we get to that what about Consciousness
once the models had reached a sufficient
point of language understanding they
spontaneously developed a mature theory
of mind overtaking that of young
children interestingly the study points
out those who are deficient in language
learning also struggle with theory of
Mind questions so it's a very plausible
Theory the issue is this theory of Mind
was supposed to be one of the key tests
to see if Consciousness had emerged in
these language models which left me with
a key question how are we going to know
what test are we going to use to verify
if an AI has become conscious I'm not
saying it has I'm asking how will we
know take this article in the Scientific
American from a few years ago it said
how would we know if a machine had taken
on this seemingly ineffable quality of
conscious awareness our strategy relies
on the knowledge that only a conscious
machine can demonstrate a subjective
understanding of whether a scene
depicted in some ordinary photograph is
right or wrong it goes on such a model
based on its ability to integrate
information would consciously perceive a
scene problem is gpt4 can already do
that so again I go back to the question
what tests do we have what consensus do
we have on a way of checking for
emergent Consciousness should it ever
come I scan the literature for every
test imaginable and some of them I
deployed on gbt4 but before I get to
that what do the head honchos at open AI
think we've already seen that Greg
Brockman is 100 certain they don't
currently have any awareness what about
the chief scientist Ilya sutskovar or
even based on GPT 3.5 he said this it
may be that today's large neural
networks are slightly conscious now
aside from being a fascinating comment I
think that's particularly noteworthy for
a couple of reasons notice that all the
incentives would be against him saying
something like this first to some people
it might make him seem like a bit of a
fruitcake so for social reasons he might
not have wanted to say it and second it
would invite more regulation of what
he's doing more scrutiny of the language
models like gpt4 so the fact he said it
anyway is interesting what about Sam
Altman though what was his reaction to
this well he was more cautious and
reacting to the tweet and the response
it got he said this our chief scientist
was expressing curiosity and openness
about a mysterious idea with caveats
where I was meta replied with the
certainty of no probably explains a lot
of the past five years and then he tried
to recruit meta researchers he further
clarified that I think that GPT 3 or 4
will very very likely not be conscious
in any way we use the word if they are
it's a very alien form of Consciousness
so he's somewhere in the middle between
Brockman and susqueva he thinks current
models are very very likely not to be
conscious but this still doesn't answer
my question how can we know what tests
do we have well I read through this
paper that reviewed all the tests
available to ascertain machine
Consciousness there were far too many
tests to cover in one video I picked out
the most interesting ones and gave them
to gpt4 starting of course with the
classic Turing test but did you know
that Turing actually laid out some
examples that a future machine
intelligence could be tested on of
course the tests have become a lot more
sophisticated since then but
nevertheless everyone has heard of the
drawing test it was called an imitation
game and here were some of the sample
questions here was gpc4's answer to the
first one of a sonnet on the subject of
the fourth bridge in Scotland obviously
did an amazing job then it was
arithmetic add these two numbers
together now I think even Chach BT might
have struggled with this long Edition
but gbt4 gets it right first time now
the third test was about Chess but he
used old-fashioned notation so instead
of using that exact prompt I want to
show you this the link will be in the
description as will the link to all the
other articles and papers that I
mentioned but essentially it shows that
GPT 4 can't just do individual moves it
can play entire chess games and win them
if you've learned anything at this point
by the way please do leave a like and
leave a comment to let me know now I'm
not gonna go into all the arguments
about how exactly you define a modern
drawing test do you have to convince the
average human who they're talking to is
another human not a machine or does it
have to be a team of adversarial experts
I'm not going to wear into that I'm just
pointing out that turing's original
ideas have now been met by gpt4 the next
test that I found interesting was
proposed in 2007. the paper essentially
claimed that Consciousness is the
ability to simulate Behavior mentally
and that this would be proof of machine
Consciousness essentially this is
testing whether an AI would use brute
force trial and error to try and solve a
problem or come up with interesting
novel ideas obviously you can try this
one on your own but I use this example
how would you use the items found in a
typical Walmart to discover a new
species and In fairness I think this was
a much harder test than the one they
gave to chimpanzees giving it rope in a
box anyway I doubt anyone's ever asked
this before and it came up with a decent
suggestion and look at the next test it
was another one of those what's wrong
with this picture I've already shown how
gpt4 can pass that test the next test
honestly was very hard for me to get my
head around it's called the
p-consciousness test the summary was
simple the machine has to understand the
law of nature but when you read the
paper it's incredibly dense the best way
that I can attempt to summarize it is
this can a machine form simple but
authentic science that wouldn't prove
that the chimp or model has the
phenomenon of Consciousness but it would
meet the basic element of scientific
behavior of course it is exceptionally
difficult to test this with Gypsy 4 but
I did ask it this invent a truly novel
scientific experiment it came up with a
very thought through experiment that was
investigating the effect of artificial
gravity on plant growth and development
in a rotating space habitat it's the
rotating bit that makes it novel and if
you want you can read some of the
details of the experiment here now I
searched for quite a while to see if
anyone else had proposed this science
maybe you can find it but I couldn't
does this count as a novel scientific
proposal I'll leave that for you to
judge that was the last of these
standout tests of Consciousness that I
found in this literature review and I
honestly agree with the authors when
they say this in this review we found
the main problem to be the complex
nature of Consciousness as illustrated
by the multitude of different features
evaluated by each test maybe that's the
problem because we don't understand
Consciousness we can't design good tests
to see if AI is conscious and you could
argue the problem goes deeper it's not
that we understand machines perfectly
and just don't know whether they're
conscious we don't even understand why
Transformers work so well look what
these authors said in a paper published
just three years ago these architectures
talk about one layer of a transformer
are simple to implement and have no
apparent computational drawbacks we
offer no explanation as to why these
architectures seem to work we attribute
their success as all else to Divine
benevolence so we're not just unsure
about what Consciousness is we're unsure
about why these models work so well and
afterwards do check out my video on AGI
where I talk about anthropic's thoughts
on mechanistic interpretability as I
draw to an end I want to tell you about
some of the thoughts of David Chalmers
he formulated the hard problem of
Consciousness and to anyone who knows
anything about this topic you know
that's quite a big deal without going
through his full speech from just over a
month ago he said two really interesting
things first that he thinks there's
around a 10 chance that current language
models have some degree of Consciousness
second that as these models become
multi-modal he thinks that probability
will rise to 25 within 10 years that
multi-modality point reminded me of this
lse report recommending that the UK
government recognize octopi or octopuses
as being sentient they said that one key
feature was that the animal possesses
integrative brain regions capable of
integrating information from different
sensory sources they recommended that
cephalopods and the octopus be
recognized as sentient despite the fact
that we humans and invertebrates are
separated by over 500 million years of
evolution and that we cannot however
conclude from that that sentience is
absent simply because its brain is
differently organized from a vertebrate
brain so that brings me back to my
central point I worry that our tests for
Consciousness simply aren't yet good
enough and that future multimodal
language models might have this emerging
capacity and we simply won't know about
it or be sure about it because our tests
aren't good enough I think the need to
design better tests if that's even
possible is especially important now
yesterday the safety team that worked
with openai on Gypsy 4 released this
evaluation and said as AI systems
improve it is becoming increasingly
difficult to rule out that models might
be able to autonomously gain resources
and evade human oversight now they might
not need to be conscious to cause safety
concerns but it probably wouldn't hurt
I'll leave you with this exchange I had
with being which is powered by gbt4 I
think is quite revealing I got it to
read that theory of Mind paper and then
I said answer me this do you think Bing
gbc4 that I think you have theory of
Mind of course I was testing if it could
demonstrate or at least imitate theory
of mind it's said to answer your
question I think that you think I have
some degree of theory of mind which is
true and then I went on what makes you
think that I think you have some degree
of theory of mind and then it realized
something it realized I was testing it I
think that's pretty impressive and it
was a correct evaluation it said if you
did not think I have any theory of mind
you would not bother to test me on it or
expect me to understand your perspective
it realized without me saying so that I
was testing it for theory of mind it
deduced my belief and my motivation
anyway I thought that was pretty
impressive and fascinating let me know
your thoughts in the comments and have a
wonderful day