Gillian Hadfield, University of Toronto | Incomplete Contracts & AI Alignment

hi everyone welcome to faucets
intelligent cooperation group as a
preamble to this meeting we split
today's keynote which originally had
four presenters into individual
different keynotes because i think the
ideas are really amazing and everyone
really deserves a little bit more time
to be discussed as so future invitations
are to follow uh we also released a new
chapter draft and that's focusing on
cooperating with ais um and how to
incorporate ais into our framework of
volunteer cooperation and so to bring
you up to speed we started the entire
book ideally and initially with
identifying volunteer cooperation as the
main driver of civilization we then
explored crypto commerce tools for
facilitating better human to human
cooperation and now we take the next
leap finally
for discussing how to bind ais into the
mix and in the chapter at least we start
by really showing parallels between the
common thread of an agi singleton that
takes over the world
and the threat of previously centralized
systems that had the ability to take
over the world and learning from those
lessons we propose that instead of
trying to design a novel single super
intelligent agent that can follow human
values we should try to create the
diversity of decentralized
instantiations of super intelligences
that can be incorporated into our
existing framework of volunteer
cooperation we also review a few
promising approaches for how to do this
and we start with principal agent
relationships across entities that want
to cooperate and whether those entities
are humans or ais a principle always has
a variety of different ways to align an
agent with its request so when thinking
about what is needed to align ai's with
human principle requests we can already
learn from the tools that human
principles already use when cooperating
with other human agents and today we'll
hear about incomplete contracting for
human corporation and what it would take
to extend this to cooperating with ais
and today we're super super happy to
have julian hadfield here from the
university of toronto she has a lot of
different titles that i will post here
in the chat but into earlier she wrote a
fantastic book on rules for flat world
why humans invented law and how to
reinvent it for complex global economy
she also published a fantastic paper on
incomplete contracting and ai alignment
that i will share here as well and we
already had the pleasure of welcoming
julian previously for foresight in
person meeting for annual and member
gathering and so i will share some info
on that too but for now she and i will
start with 10 minutes
or 15 minutes um on this framework i
will then start with a few questions and
then we can't wait to hear from you uh
please start collecting your questions
or comments in the chat and then we'll
just hopefully have a really nice
go-around also just a shout out yoshaba
because i see you here well done for
getting on lex friedman's podcast twice
okay sorry it couldn't happen jillian
now it's up to you and welcome to this
trip
all right thanks allison and i'm i'm
really looking forward to this allison
give me a sneak peek of the questions
that you can ask and i think this is the
best set of questions that anybody uh
given me to talk about this stuff so i'm
really looking forward to to getting to
that conversation and um i am going to
uh hopefully get go go through
you really quite quickly um ten minutes
maybe at most um the weekend uh because
so we can get to the uh concept here so
um
so allison led in with um you know the
framework i think is so critical first
of all i think you know the the concept
of decentralized uh multi multiple ais
and how they integrate into our
normative system that's
absolutely critical for us to be
thinking about um and i think it's the
most important thing for us to be doing
we should really be trying to figure out
how do we embed
a particular set of values into ai
systems to thinking about how do we
embed them into our complex structures
or aligning other humans because we now
need to expand that to include
um
aiai entities uh i'm going to talk a bit
about uh incomplete contracting uh
ellison mentioned that this is um this
is actually
a paper this is from a paper uh giant
where
my son dylan had field manel
um so if we want to think about how do
we get an agent to do what we want right
that's that's the puzzle
um well we've been studying this in
economics
for quite some time when we're talking
about human agents this is the theory of
incentive
and when we started down this pathway as
economists thinking about how do we give
agents the incentive to uh perform in
the ways uh preferred by the by the
principle in other ways that are aligned
with the principle
entrance and goal
um we developed concepts of all
contracting economic theory about
contracting
and the idea that well you can write a
complete contingent contract you can get
the agent to do what you want if you can
write a complete contingent contract on
all the observable states of the world
that matter for more payoff so that part
our theory of incentive
but after building some absolutely
gorgeous elegant models um you know
beginning around in the mid 80s which i
will confess is when i was a graduate
student at stanford thank you about
these things um
oh
we started to recognize and all the
links and what sort of a lead on this uh
that
contracts were not generally complete it
was actually when you looked around at
the real world of contract a lot of
those contracts are incomplete to roll
uh another nobel prize winner economics
almost every economist would agree he
said in 1999 that actual contract are or
appear uh quite incomplete many
contracts are vague or climbing on a
number of key features and this this is
an absolutely important observation and
it kind of something that has to be
repeated i'm also
trained in law and i i'm a law professor
and you know i teach i teach contracting
um and the key thing that the disconnect
between the way economists sometimes
talk about contracts and the way real
world of law looks like with contract is
the fact that uh they are replete
with vague language like we agree to
behave reasonably
or do what commercially affected um in
this contract we see a lot we don't see
chris we see some terms like pricing
terms for example maybe delivery date um
and some some technical that's the cases
but there's lots of vagueness and lots
of gaps
so that's for the analysis and economics
of the incomplete contract said okay so
if you take seriously that contracts are
incomplete we don't want to spend all
our time designing complete contracting
we want to think about what's the
optimal design for an incomplete
contract
and uh because what we recognize is that
incomplete contracts give rise to
strategic behavior well you didn't tell
me i couldn't do this i will exploit
that gap
um and it can lead to suboptimal
suboptimal behavior so the though we now
have
significant literatures in economics
that look at
the analysis of incomplete contracts and
the design of incomplete
contracts how can we optimally design
contracts given the constraints that
they will be in complete
so now let's think about
artificial intelligence and the focus is
particular on the the form of all the ai
or machine learning that that uses
reinforcement learning of course we have
other
approaches to machine learning
supervised learning on supervised
learning
but in reinforcement learning we are
specifically saying how do we establish
a reward function
for an agent and now a an artificial
agent
to learn how to behave in an environment
and achieve the goals that we as humans
have for that artificial intelligence
and that's the that's some of the
powerful systems that we have out there
and but this is uh relatively recent
this is 2018 first workshop on goal
specifications for reinforcement
for reinforcement learning how do we get
that robotic agent that our official
agent to do what we want
and
the answer is that just like it's really
really hard to write complete contracts
we should treat it as impossible reward
engineering is very hard this is an
illustration from another paper that
dylan uh is an author on um
with uh stuart russell and entrepreneur
and uh peter beale um
really looking at this problem of how do
we deal with the fact that our designer
of an ai system a reinforcement learning
agent
um supplies a reward function here
called the proxy reward function to the
agent
um the agent
learns on that proxy reward function
but if there's a gap between the proxy
reward function and the true word
function we're gonna get behavior that
we didn't intend well the illustration
here is you know you think you're tr
you're you're training this robot to
find uh the best route to the pot of
gold taking paved roads taking shortcuts
through grass
you then put it out into the real world
and discover that there's lava
pools in your environment which you
didn't think about but of course you
have very negative
utility on going through the law having
the robot go through the lava but you
didn't say anything about it from the
from the robot point of view the lava is
just fine um there's no cost associated
with it so that's a um a challenge of
how do we it's very comparable to that
in complete contracting
model so just like we have our
incomplete contract leading to strategic
behavior exploitation of gaps and
sub-optimal behavior we're also going to
get that when we have mis-specified
reward function and we've the latin
first lesson
from incomplete contracting is to say
and you need to treat the mis-specified
reward function
and to some extent unavoidable you can't
just keep going back your engineers and
wagging your finger at them and saying
you know get it right next time let's
get the reward function they're always
going to be uh to some extent um this
specified
so the question that we have been
thinking about is what can we learn from
the public contract theory to better
understand and more systematically find
solutions to ai misalignment and the
specification
and so i'm just going to talk a little
bit about one of the things that we've
been uh exploring here um so you want to
imagine here we've got uh we've got a
robot the robot
has got a reward pension that's going to
reward the
um the robot for carrying boxes across
the room
and if we trade our our robot in this
environment it's going to learn the best
path and efficiently learn how to get
the opposite to see the other side of
the room but if we then have a
unexpectedly let's say a vase appears in
the path that the robot has
all uh developed for carrying those
boxes what is the robot going to do well
it's just like the lava in the last
picture it's just going to knock that
phase over because it's reward function
doesn't say anything about knocking over
dates
and if you start thinking well let's
just make sure you get
invaders into the reward function then
think about cats and think about
children and think about pools of oil
and all the other things that might
occur in a real world in a real world
setting the robot yeah in this setting
is stuck in with that reward function is
gonna plow straight through that through
that base
so how do humans solve this problem so
just think we've let's suppose we did
hire a human and uh we write a contract
with that human and we say we'll pay you
or same reward for carrying boxes to the
other side of the room
and we we announced that contract needs
that contract and then the agents are
um uh is set loose and again we have a
bays that appear
on the human normal path so what's going
to happen here well we all know what
happens since the human's going to walk
around the veins and not knock it over
maybe maybe he or she would lose the
vase but we're not going to knock that
days over and the question is how is it
that humans know to do that what is
going on with humans that we're not
behaving like the robot here
well the key thing that we want to talk
about is the fact that human contracts
come
embedded in an environment an
institutional structural environment
that has lots of capacity to help fill
in terms
interpret terms
um and uh
and and and determine what was the
appropriate behavior in a particular
setting we had we have implied terms and
institutions so again teaching contracts
um what was it reasonable to think the
parties had in mind when they agree with
respect to vases they didn't say
anything about it but what would they
have had in mind what would they have
said
we provide we have a we have an
elaborate structure of legal reasoning
as well as institutions that can
implement that reasoning
um including legal advice not just
adjudication
um uh to to to say what's the reasonable
behavior uh for that for that setting
and what that means is that we have
these external institutions they might
be formal institutions like court you
might be informal institutions just like
reputation in in in groups
and those are filling in the contract
which formally expressly just says you
get paid hard for carrying boxes um but
those institutions are filling in that
gap and say well no there's a cost
associated with boxing overlays
um you might get a bad reputation as an
employee and you're not going to get a
good reference for your next job you
might get sued by the employer
um or you might have um
money withheld from your from your
paycheck
and those external institutions are
filling in
that contract filling in the incentive
for the for the and this is what makes
it rational for principals to enter into
these contracts that everybody knows are
incomplete they say well a lot of the
gaps will be filled in
one of the ways to think about what's
happening here is that the our human
agent when that vase appears go through
a
process of thinking about
what would observers think
if i just plow through an arab better
choice amazing is in the way what would
observers think if i just flowed ahead
and walked through that phase knocked it
knocked it over
and adam smith has this wonderful
concept of the impartial spectator this
is from theory of moral sentiment
um so i think really captured this idea
the idea of that we have
um when we i mean he was thinking about
this as a description of morality and
moral and moral conduct that we are kind
of constantly engaged in this
hypothetical reasoning you know what
would a neutral and impartial observer
think about my behavior
if i behaved in this way so what would
uh what would the impartial spectator
and as we sell in institutions it could
be what would my impartial spectator
saying what will the judge say what
would another employer think what would
a co-worker think um if i behaved that
way and that's how we are still humans
are filling in those incomplete
contracts so the question that i want to
explore and i'm looking forward to our
our conversation is
you know how can we build the normative
infrastructure that could support
ai alignment because i think a key
message i always want to give to people
who are working on the building and
design of the ai system
is where there's a lot of um a lot of
game theory uh game theory is great
there'll be lots of game theory but it
basically says we can solve the problem
inside the box or there's the math
there's the design of the
uh the reward there's the idea that we
can we can fine-tune the math of those
reward functions and i think the key
thing we're missing is the idea we need
this normative infrastructure human
societies would not work without that
normative infrastructure to help support
those incomplete contracts
we're going to have incomplete war
classification goal specifications for
ai so how are we going to build that
normative infrastructure to support ai
alignment and
um i'll say how many if i stop there and
then we can
jump into conversation
that sounds lovely thank you so much
very crisp and i can't believe that you
actually when i were able to condense
this paper uh so as succinctly uh thanks
a lot um maybe we can i already have a
question from kate here and one maybe a
comment from david in the chat i'll
start maybe with uh two or three
questions at the beginning um and maybe
we can still keep them recorded in case
you and then we'll move to an off the
record discussion
we can have a little bit more of a free
flowing back and forth but you know
you're mentioning the russia observer
and um i'm really interested in what do
you think are the most important
differences across human cognition and
ai cognition that could become difficult
when transferring the incomplete
contracting framework from humans to ais
because
um what would it really take for an ai
to predict a community's like normative
structure humans have really evolved for
this impartial observer ai's haven't
really and so how uh yeah how would we
be able to uh actually transfer that uh
that framework to a human to ai
condition
yeah it's great great great great so so
i think the key point here is i
absolutely do believe we have cognitive
architecture
that humans have evolved that is attend
to
what what would others think what would
be we're predicting what would be the
response to achieving you know we we
think a lot about when we think about
morality so i spend a lot of time now
would be with groups looking at
um
morality in
all you know in ai setting um and think
about that but there's a big focus on
the compliance right like what's our
cognitive so we have ideas that humans
have a preference for complying with
rules and so on but which i i resist
because i think that's kind of well it's
it's a it's a cheat it's it's
it's it's kind of slapped on it doesn't
really tell us what's happening and i
think well what's really distinctive
about human society is the third party
punishment scenes we have and punishment
is a powerful word but it could just be
a raised eyebrow
right that tells us oh i mean like right
now we're looking at each other over
screen uh we're interacting on chat
we're raising hands and so on there's
all kinds of norms right in here where
we're kind of like also being
conditioned by what do you think is the
right thing should i just keep talking
have i been talking too long right you
know
did i bring it too close here um so we i
think we have developed the cognitive
architecture to constantly engage in
predicting how would third-party third
parties respond from formal interprets
all the way through a formal institution
so what does that mean for for ai
systems
well in a way it didn't i mean i'm not
saying it's easy to build them like that
but conceptually it's not difficult
conceptually it's the same challenge how
do you train your systems to not just
make predictions about what will happen
if i pick up the cups like this
or if i turn the wheel on the car like
that in physical space
how do i predict normative faith and the
thing that i see about the way we train
our systems right now is we're not
providing that information we're
basically like training our large
language models we're training our
autonomous vehicles with none of the
support structure that humans use to
figure out how to do it which is heavily
drawn heavily on predicting those
third-party uh reactions uh so i think
we need to think about change you know
enriching the information the data on
which we're training in that sense but i
think we can build that comparable
cognitive architecture
okay even if we could do you think that
you know that would be i guess the more
prevalent way that uh effective ai
systems will be built because one could
also imagine the opposite scenario so
instead of human aligning ai agents with
our interests by teaching them to
develop on a spectator maybe we lose our
own internal spectator over time because
the more we
cooperate with artificial agents who
naturally follow more homo economicals
cooperation style
that makes them more superior players
the more we lose our own ability to
potentially have an eternal spectator so
like just because of the fact that by
them just going along with the homo
economical setting by being pretty
unbound and their ability to fake they
may be more effective at cooperating and
so uh they may just uh super be in that
regard
yeah no this is a really fascinating
question oh and it and it reflects the
kind of thing i can worry about when i
think about uh i mean i've been
emphasizing how do we integrate ai into
our human normative structures and that
means we've got to build for them to
integrate and i think that's that's the
that's the that's our that's where our
alignment solution sets and lie because
just like human right the way we align
is we're integrated and that's flexible
and it evolves and it changes over time
and space and so on but you're
emphasizing another really key point
which is if we start having a lot of
artificial agents in our mid
what does that suit to these systems
that we have come to depend on and i
think that's that's really important
like i think if if you know 50 or 75 of
the cars are on the street are
autonomous vehicles
right i now have less information about
the humans in my community because i
used to know something about the humans
in my community from the way that they
drove around on the highway right or at
the four-week stop or at the the
intersection
um
and i have less information now about
other humans um so i think your point
about the that doesn't start to degrade
our impartial spectator like the kids
people worry about the kids who are like
talking to alexa or siri
and they're not you know they're not
they're not they're talking to it like
it's an it which it is um and it's that
degrading their interaction with with
humans um
i don't i don't have many answers to
that except to say that that's i i i
think these are the kind of existential
threats that we face that we're just
going to slowly stabilize our human cook
sounds that we have to be very cautious
about it
yeah i think one um you know i guess
more cryptocommas tools that we
mentioned in the book is potentially
using blockchain to
allow an ai agent's internal working to
be transparent and observed by anyone
interacting with them and that could
maybe give a bound on their ability to
make um and so reinstantiating a little
bit this constraint that humans have
when faking um and
are there any you know um directions
that you want to point to and promising
research areas or like you know
currently available prototypes that you
know people could consider when they
actually try to do research in this
direction
yeah i don't i don't know about
prototypes but i think what's important
here is if you're you're zeroing in on
the right dynamics right so this is
about this is all about trust right this
is all about am i willing to participate
in a complex interdependent set of
relationships
and count on the fact that somebody else
is cooking the food and somebody else
has built the bridge
right so so so i think you know to the
extent that um you know we can be using
technology blockchain or whatever to
um uh to address that trust problem is
really important
i think once you take the lid off so i i
i know they like uh andrew kritsch at um
at uc berkeley chai is has
done a lot of thinking about like you
know if we if you know if if it's common
knowledge what your strategies are like
can you get access
and what does that do that so i think
this is some interesting
thing i think the thing that tricky is
don't you take the lid off this
patrick transparency is not necessarily
what produces
trust um
and so i think it's going to be
more complicated than
um well we we can make our ai's if we
really are saying what we want our ais
to do is to be able to act like
competent human
i think we actually depend a lot on the
fact that we have all kinds of different
lines we draw around what information is
shared with who when
um and so
my starting point might you know i'm
it's it's a great question to explore uh
because it does address that
transplanted i think it's gonna be
a lot richer than we maybe anticipate
once we take the little off
yeah i agree i think uh what we also
propose is not making everything
transaction like not making everything
transparent that an ai entity does but
just in certain situations you may want
to introduce transparency in others uh
you know you want to have shielded
transactions um okay
that's enough for me thank you so so
much uh very very eloquently uh uh
answered i'm uh we'll take david manheim
next and find to be on the record and
i'll follow up with the others and what
they prefer okay david you're a first
and maybe say a word about your
background so julian has an
understanding of the
conflict uh sure
brief word about my background um i i do
lots of things um including some
thinking about um ai alignment and
how metrics are designed um
so specifically in that regard i i just
i want to flag so
yes it's true that metrics are always
going to be um imperfect like you can't
you can't completely close the gap but i
also think that um there's a huge
space between what people usually set as
um easy to operationalize goals and what
it is that you could do if you try to
actually um come up with like think
about what the failure modes are
beforehand when you're designing a
system and and that's the kind of first
level of we should make sure you do that
um but the second thing is i think
um
you pointed out that the gap between
the goals that the system has and what
it is that um
you want it to be doing um is hard to
close the question that i have is it
seems plausible at least that um an ai
system predicting
what the impartial observer would
be upset about especially if they're
willing to be kind of
if you look at children of certain ages
um they'll be kind of very hesitant to
do things that might get them in trouble
because they're not really sure what it
is that they are and aren't allowed to
do and it seems like that's an easier
predictive task
than
um
actually directly inferring what it is
you're supposed to do
um so it seems like maybe there's
something there does that make sense as
a
as an approach yeah yeah let me just see
on your first point completely agree we
should you know it we should we should
put as much in there as we can and we
should do a great job as we can on
reward engineering so
um uh this is not safe throw it here
just to say but we need to be thinking
theoretically about the fact we're never
going to fix it and fix all of it
now
on the other so actually thinking about
the child being cautious
not sure what will get them in trouble
it's actually not a bad model
right uh so um
and and that's actually what the
prediction we're talking about here is
and because
um even though mathematically rewards
and punishment are the same thing right
what we do know is that our our
normative systems our legal systems they
primarily operate on
uh cost on punishment
right and i i've other things to say
about why i think why i think we see
that
um
but but the idea so the idea is that
yeah travis is thinking okay what's
going to get me in trouble that produces
caution
that's actually one of the principles of
reward design that people like dylan and
stuart ruffle and anchor dragon
are working on it's the idea that
actually you do want your
uh your ai to be uncertain about what is
really wanted especially in domains
they're outside of the tree can they
recognize hey this wasn't in my trading
data i've i had nothing like this in my
training experience i should be cautious
that's not that may not be a bad
principle um and truly is to think about
that i hadn't thought about it with
relationship and that you know children
would be cautious in that way and at the
same time they're they're rushing
forward you know like not they're
learning this whole thing like oh i
could get in trouble for this who would
have um
certain ages in certain situations right
yeah yeah
for sure thank you thank you
all right great next one up we have kate
who's also fine to be on the record
okay um my question oh thank you so much
jillian first off um my question is
about case law so i'm not a lawyer but
it seems to me that the way in which
that
the way in which we're able to fill in
the gaps in contracts is because we have
this historical record of case law that
has happened in the past
um and uh it seems to me that that's
actually quite a bit like machine
learning in that um you know we we have
this record of things that we've learned
and we're able to consult that to figure
out what to do next
um
and so i'm curious if uh
we could use anything similar for ai
um and what that kind of institution
would look like um would it be possible
to have some kind of uh
database of
you know the previous things that a
agents have learned um and be able to
share that kind of knowledge
that's really interesting i mean i i
think i want to emphasize that case law
is an important thing in our advanced
formal legal system but those are
relatively
uh new and recent in human evolution and
i think we've been filling in
filling in the contract for a very long
time and there's a lot of places where
what we're what we're having reference
to are sort of the more informal norms
but and then the data we're using is all
of our experience of all the times we
saw how did people react when somebody
talked like that in a session or behave
that way in the restaurant or on the bus
or whatever
um but i do think there is something
from having formal case law and i think
what's really important there too is
that that's how we i think david uh
friedman on the call too so another law
professor uh could
talk about these things um
uh so uh
that so so we do have we have case law
now that's a common law approach um and
that's another interesting thing to look
at the difference between
uh different systems i wrote about this
a long time ago thinking about how these
information differences between
common law and some court systems led to
different behavior in
behavior of courts and so on
um
and of course what we do with those
cases now this is really recent right
this is like 19th century harvard law
school inventing the case book approach
to
legal education
um is we train all our lawyers on
reading the same cases
so
uh they've all read these kind of
archetypal
and they've engaged in the reasoning
because it's not really even the
precedent in the case that matters it's
what they've learned about engaging in
this shared system of reasoning because
that's what lawyers had to do is predict
what would open lawyers think about this
um
so i think it's it's it's a a
fascinating thought to say so what we
need because this is like my answer to
allison earlier
um
saying
you know i don't see us building the
data sets that provide that kind of
feedback like like let me treat jptree
it's just training on the internet to
predict you know what word is next
and what it's not screening on is how
did people respond when they read that
on the internet
right what did what did they think about
it so i i do think that developing the
data that you kind of want to you want
to marry up with that is to say
um you know when you answered that that
five-year-old that way
right people said you don't say that to
a five-year-old uh you know or you don't
use that language you might use that
language in this chat room but you don't
use it in the
um a public news feed or
that you know you say that
um oh
anyway so you get my point i think
that's a really i really do say this is
a place where
we need to be starting to get creative
about how would we build we how how how
do we build our training data stuff i
think a massive
and so we're just training on cheap data
and we're just now recognizing really
complicated things and so i think
um
whether it was an adjudication type
process
right so maybe reviewing the way an
autonomous vehicle behaves
like should we have community processes
that are reviewing that and
accumulating that information
um
or is it uh or is it just organic
observation how do people respond anyway
i think that's a really interesting
thing to explore
awesome thank you yeah i think we were
just discussing uh in the chat about
where could we get this data from and
maybe there's a few current
decentralized arbitration um crypto
coverage tunnels where equal here yes
and actually gather some uh of their
data um about how humans usually wanna
set up most disputes or so forth okay
next question uh we have mika
so this feels um
surprisingly similar to the problem with
autonomous cars obviously um and and for
two different reasons for one it feels
like
just as with autonomous cars humans are
not 100 successful at doing the thing
that we want them to do right so humans
fail a lot um both at driving cars and
just to interact with society around
them um so we don't actually have to
fully solve the problem we just have to
do better than humans which is pretty
low bar realistically
and so uh similar to autonomous cars you
know the bar's low you just have to
drive better than human and i feel like
similar with ai as it's like interacting
in our environment just like in other
ways the bar is the same like you just
have to do better than a human and so i
feel like i definitely agree with you
100 that we cannot solve this
um like manually
um
but at the same time we also don't need
to shoot for perfect which i think is
kind of a natural tendency to want to
find a perfect solution which is just
impossible um and the second part about
how this i feel like relates to
autonomous cars is similar to how tesla
is training their data they're basically
just having the cars watch humans drive
and then the cars basically learn what
would a human do and so any situation
the guards put into it just ask itself
what would a human do and then it tries
to do that thing
i feel like you know
if we can have these ai agents in our
society just watching what we do all the
time um and then of course there's
questions on mass surveillance
but if we just ignore that for a moment
we just think about imagine these ais
could just watch everything and watch
what humans do and then at that point
now you just have to just tell them you
know do what that thing is doing that
those things out there those little
things walking around talking to each
other interacting et cetera just mirror
that do the same thing and it will i
suspect using that technique we could
get
good enough pretty quickly and then
eventually better than humans which is
you know the target and once you're
better than humans then you've succeeded
and anything after that is just a bonus
if that makes sense
yeah so um
you know i think there's
there's lots so let me just you know i
just want to make an observation about
just you better than than a human and
thinking now autonomous vehicle
um
i mean i think from a consequentialist
point of view we could say that right i
mean but
we just just want self-driving cars to
be
to kill fewer people than humans kill
um
but
i actually that now
it's another way in which allison
questioned about what happens to humans
when we introduce i actually don't think
human
um so i think of our normative uh
infrastructure
as
an equilibrium process and it's it's
really it's what allows us to keep
working forward in complex
interdependence and increasing levels
with complex interdependent
um so it's very much a social scientific
perspective on normativity it's not an
internal perspective on what's good or
bad or or or moral or not it's saying
it's it's it's a tool we have
normativity is a tool we have and these
normative structures are what move us
forward
um
so if we introduce autonomous vehicles
into the mix they're a different kind of
entity
and i may not trust uh you know when i
when i
interact with a with a human i kind of
in equilibrium i'm cutting them some
slack
for the mistakes they made
i don't know if i need to cut machine
locks in the machine for the the uh
mistakes they make
they're not human in the way that i am
um so i'm i'm not i try and i know this
is a big big topic in in autonomous
uh vehicle world uh you know well why
are why are people wanting more from the
machine than from a human i think it's
because it's that that complex process
of normativity on there
um
but i agree with you we don't want to
set the bar to be perfect and i think
that's that so that point about do what
humans do i would
i mean i don't know if you mean as a
training mechanism there's certainly
lots of work being done on you know to
what extent can you train on just
observing humans well if you're trying
to train for normative behavior you're
going to run into a lot of problems
because
um
uh
you know
humans are not always implementing our
you know uh the norm the rules like like
i don't think you want to i really do
not believe you want to train your
your adjudication system entirely on the
decisions made by human judges i think
you want that to be information but you
don't just want to replicate that
um don't want to set the bar too high
but i do think that the right way of
thinking about it is um you want
machines to be normatively competent
and
uh and then to interact with with humans
memory so
what love to think about there like i
think
all right thank you so much next one up
we have david treatman
right i don't mute myself
humans learn
what we're trying to teach the machines
in two different ways
one of them is hardwired by evolution
and the other is by trading as children
and i'm not sure i see what are the
limits to using the latter method for ai
that is suppose what you do is you put
your ai kind of environmental function
in and you have it in partial observer
you have a human being who is monitoring
the process and at each step it affects
saying yes that was right no that was
wrong no that was terribly wrong and so
forth the ai then trains on that
uh this is going to be expensive it's
going to take a lot of human interaction
we only have to do it once for a given
environment because you then make a
thousand copies of that ai and you load
them into a thousand robots and and now
you're home free
and you know eventually something
terrible happens and you realize you've
got to have a broader environment you
you you tweak the system but i'm not
sure i see what are the limits to that
learning model uh for how you teach it
to it
um
well i i
i
so so so i i'm yeah so your your
question sort of has a is there a
premise in there that i said there's a
limit to that i mean i know but we have
a puzzle and i'm trying to see why that
is here yeah
ah
well no no i am saying i think that the
you know
training
well
um
i i want to disagree with one thing in
the in the mob and you talked about like
treating it like there's a there's like
in the lab time like a childhood time
when you're you're you know you're in
this safe space you're in this and and
you're going to train and then remember
get thousands of copies or whatever and
we'll we'll we'll send people out send
machines out into the into the world
actually humans are doing this all
throughout our life so it's not it's not
i don't think that what we're learning
is the value
i think what we are learning is the
cognitive process
of reading
value constantly from the environment
and predicting which how we how not the
values like not what should i put into
my reward function but now like what
will be the consequences
uh how will people react
um
in human so people in cultural evolution
um
uh you know
i think this is
not the strongest theoretically but it's
where things are now there's a view that
you know humans have a preference for
conformity and so if you observe that
this is how we all do things around here
they have a preference for conformity i
actually think that's actually just an
implementation of if i do it differently
i'm going to suffer some consequences
people are going to think i'm a weirdo
at least
um but i think that
that's constant
so i i i don't want to disagree with you
that i that that
there's a lot we could do with this kind
of training and a lot of and there's a
lot of human interaction it takes a long
time to trade a human
sure but it's coming you're gonna do
this in the real environment that is to
say you're willing to accept the risk
that the one the one robot you're
trading will subway step on somebody
uh right because after all this is just
one and you're going to get a million
later
uh so you you you have the robot in the
environment you have a human being
monitoring and
maybe for some things you want to have
the robot say when do five seconds to
advance so you can tell no no no no you
don't do that
but but right the whole model is it's
just one where we want it
yeah and i and i do think that that
there are people who are working on this
who are you know using techniques like
this human feedback and so on in the lab
or or like you know maybe you've got
your robot or your autonomous vehicle in
a protected space so there's
well limited capacity to do harm but um
yeah so i think that i do think that's
part of the process
well i i i'm i think that there's a
missing distinction uh in the
conversation
though i first of all julian i did i
think this is a whole brilliant
framework
for thinking about these issues so thank
you very much for that uh but your talk
uh raises an incompleteness in two
different places
uh that i think are usefully kept
distinct
one is
what's the me what's the extended
meaning of the contract when the
contract was silenced on something
and then the thing that cylodon becomes
an issue you talked about the
institutional framework in the the the
embedding in the systems of norms
that is kind of the extended implicit
contract that might turn into an
explicit enforcement action
by
the
framework in which the contract is being
enforced so i think this relates very
much directly to the whole notion of
split contracts and the kind of thing
that for federico is doing
where you're bringing in this extended
learned judgment system
in the intermediary role of the
extension of the contract
and then there's
separately from that
there's the issue of the impartial
observer in the heads of the inter of
the individual contracting parties as
opposed to in the extended contract
itself
and then that gets
and the issue about
uh what would the
the
a participant in the contract what would
they
their own internal impartial observer
how would they judge their action
uh part of that is is inherently
tied to their prediction
of
how would the extended contract or the
system of norms of the society i'm
embedded in
view my activities in some so the
developmental
account in theory of moral sentiments i
think is very much that we abstract our
own impartial observer or partial
spectator
from learning how others react to us
that that there's sort of this dual
prediction going on there
yeah so i agree with everything you said
so i'm i'm not um i'm i'm maybe i'm not
seeing
uh the tension um
because i guess i am
i mean i'm leaving my hand at the
impartial spectator
yes no sabian saying that we do engage
in this
in hypothetical process all the time how
would people respond how would it and
and the impartial part is to say you
know not not somebody who
well not an interested party but you
know rather somebody who's evaluating my
behavior relative to the norms and rules
of our society
um
i think i take what i'm reacting
is that the focus
seems to have been
on
the extended judgment and learning of
the participant of the contract
and without enough
focus on the
on
the judgment on the part of
the intermediary norm
framework
right right okay now that that that's
really that okay so that that's
important um in the sense of um i mean
i'm sort of holding that
the content
of
what gets filled in right you know the
what what does the institution do what
are the norms i'm i'm treating that um
as
the output
of our normative system okay
i think i can i can spot why uh
there's this gap here
is that i'm looking at smart contracts
i'm looking at
creating a whole new
norm enforcing framework
that is
uh
very much inspired by
uh what's been working in human
institutions but is also to a large
extent
has to
recreate and reinvent a lot of that on a
more automated basis
so the fact that the contracts that
we're writing in program code
are incomplete
is giving us novel problems that are
different than the problems of the
incompleteness
of
uh contracts in natural language
by humans
yes no and and actually this is um
you know
so certainly in the domain of smart
contracts
i think the
uh you know that the message and what
you're saying is yes the message needs
to get here say look you think you could
write down on the auto executing
uh contract but i guarantee you this is
let's say i teach contract this is all
we teach i guarantee you there will be
some circumstance where people will say
no that's not what we meant
right that's not what we meant and
that's exactly why we have these
elaborate institutions and the
institutions are not just adjudicators
they are also systems of reasoning
and
and
particular systems of reasoning the
particular system of reasoning and
contract law in anglo-american system
which is different from
contract reasoning and say the french
system or the german system
um and it's that you know it
so i this is um
uh i had a student who had a startup but
i'm not exactly sure what happened to
this sage was um but it was the idea of
creating a platform that said you're
gonna need dispute until you're gonna
need interpretation services and
gap-filling services for smart contracts
um and so you you know so let's get
creative how do we do that right we know
that our differing institutions
are quite broken
uh they don't deal with the human
environment very well they're too
expensive they're too slow most people
don't rely on them um
but uh but we're gonna need new
processes to to fill that in so it's
kind of related to kate's question as
well i think like how you can build that
those new institutions that can that can
hook into the smart contract yeah
right
oh let's unpack okay we have christine
next
so uh i'm going to challenge that i
think that
you're i first of all i thought your
presentation was great and i thought
that there was a lot of interesting
stuff in there um especially the around
the stuff of uh um
that reasonableness that we can approach
that in a smart contract system um as we
currently do it it was actually not
until i heard about split contracts i
thought smart contracts were worth any
attention to me whatsoever um so having
having reasonableness be something
that's addressed is actually something i
i strongly agree with but as in terms of
having autumn artificial intelligence
systems uh be the arbitrator of
reasonableness
in our contemporary systems one of the
things that's very important when a
resolution is made by a judge is that
the judge is able to explain why they
came to a conclusion and this is
something that is that contemporary
artificial intelligence systems are
notoriously bad at their their very
black box we make a decision uh based
off of uh kind of absorbing a bunch of
information from society kind of like a
gut decision type thing and you know
therefore we also pull in like gut
racism and gut misogyny and all that
type of stuff right um but one of the
direct so part of the directions that i
find most promising in ai research are
like propagators and leilani gilpins
work on retroactively explaining why a
machine learning system might have made
a decision through propagators um
because if it seems to me that if i am
going to trust
um any sort of ai system to step into
making things reasonable um there's a
phrase
an artificial intelligence system that
can't explain why it made this decision
isn't very intelligent um and so i
wonder whether or not you think this is
a prerequisite for um that for further
advancement in this area before we can
trust ai agents to start be making um at
least let's say criminal resolution
level uh decisions in our society
great thanks christine so so actually
we're gonna have to get out although you
have to invite me back to talk about my
most recent work which is on
justification
uh and is
exactly on this point um
and again i think the ai world is
barking up the wrong tree
with explainable
ai
because
um
christine what you were what you were
saying about you know like when the
judge makes the decision if the ai is
going to make the decision and it's a
black box well it's actually a black
grass that's going on inside people's
brains as well
and and really we don't want to know we
don't ask the judge to tell us you know
what neurons fired wouldn't help us to
know that
we don't really even ask the judge and
say what really drove your decision
right
what we ask our judges to do is to
provide justification
for their decisions which is to say i
can
i can account
for this decision
with a set of reasons
that are consistent with the rules and
principles governing reason giving in
this community
so i can't say
i convicted that person because they
were black
right i can say i convicted that person
because i was convinced by the testimony
that that was that they were the they
they actually were holding the gun
right so and we we
we examine those decisions for whether
or not those are accurate reasons no no
what was really going on is you
convicted them because you were because
they were black we do we discipline our
reasons but it's that system of reason
giving that is our control mechanism
and that's how we control decision
making by human judges
because we expose that process and i
think that so i i have a blog post on
the website for my institute that that
talks a little bit about this to say
um
and i'm working on a couple of different
projects to say we need to be thinking
about how do we design structures where
either humans can give justifications
for the decisions we've been thinking
about a licensing process for ais so you
can't release a model until human
given the pep step can predict what the
model will do and give adequate
justification for what the model will do
and then you know can we even say you go
back to that human and hold that human
responsible but i think that um there's
nothing really cool stuff i do want to
come back and talk about what we found
in some of our experiments on this stuff
um
but it it you're absolutely right that
um
uh in order to build systems where ai is
making is exercising judgment it's
filling in the contract uh for example
we're saying what's reasonable um uh we
have to
again
hook that into our systems of of
normative control and one of the key the
key one of those is the provision and
the scrutiny of reason
i just wanted to say that
i i agree with you i'm not actually not
advocating for transparency of the
execution process as a requirement here
um so i agree with the justification uh
approach um have you looked at leilani
gilpin's uh work as in terms of creating
retroactive
uh research on this she's one of gerald
sussman's uh students uh
into the chat oh perfect perfect she
knows she only hasn't had that no she
has an amazing system where she's hooked
up uh propagators to
um
to the retroactive information of cars
so that they can explain why they what
happened that led to crashes or like you
know turning in this situation and blah
blah they can retroactively construct
what happened much like as you said a
judge might not be exposing their entire
firing of neurons but they will be
retroactively thinking about the
conclusion they came to and walking
backwards from there to understand why
they ended up coming to this conclusion
basically yeah so i i thank you for
putting the link in i'll go take a look
i do want to emphasize the key
distinction between the causal account
of how a decision was reached
and the normative justification for the
decision and those are two different
things and and there's lots of good
reasons to want possible accounts like
oh we got to fix the machine
it's responding to the wrong thing it's
seeing snow it's not seeing dogs and
hunt and and wolves right at sea and
hope nope we definitely do want
explanations and anyway i'll look that
up that's great thanks for the link