Friday Seminar Series- Gillian Hadfield: AI Alignment and Human Normativity

[Music]
so I want to talk generally about this
time AI human so we've got a lot of
terms floating around AI safety AI
policy value alignment how do these
things all line up and I want to give
you a framework for thinking about that
I'm going to talk about a set of
research projects I'm involved in and
some things I'd like to be involved in
so a lot of the conversation about AI
safety is on the question of how should
we regulate AI right what what are the
right rules what are the values we
should have so anybody who's like been
bored to tears by the trolley problem
right and sitting around like what what
are the values that's a lot of the
conversation in AI epics liability for
autonomous vehicles for autonomous
weapons etc it's about what should what
should the rules be how should we I want
to focus it's actually they're really
really big and interesting and hard
problem is how can we regulate AI how
can we actually get machines to do the
things that we want them to do and
that's that that's a tough question and
I think the way to think about that is
to think about how can you build AI
systems that can interface with what I
call human normative systems that's an
important concept normativity I'm going
to use that language to mean the systems
that humans have for classifying
behavior as sanctionable or not that's
basically every single human society you
look at it's going to be full of
normative labels this is an OK action
that is not an okay action everything
about social norms culture law boils
down to a classification system a
normative classification system this is
okay that's not okay and then on top of
that an enforcement scheme for punishing
people who engage in the not okay thing
and as you think this is one of the most
exciting and fascinating things about
human evolution is the development of
these normative systems and I want us to
be focused on thinking about that I want
to just draw a distinction here because
a lot of our work on thinking about how
do
make sure that robots and a eyes do what
we want them to do focus is on the idea
of preferences I'm an economist I love
the idea of preferences but preferences
are a modeling tool I don't think there
is no such thing as preferences it's a
way of modeling actions and so I'll
throw that out there and I'd love
anybody wants to talk about preferences
and the difference between thinking
about this there's a difference between
analyzing human normative systems and
thinking about human preferences or
values so I want to just draw that
distinction it's not and focused on the
idea of systems which also means there's
a lot more information out there for us
to use to extract from the environment
what is it that's okay and not okay to
do I want to think about this as it's
both an engineering research program how
do you build these kinds of systems and
it's a social science research program
because frankly there's very little work
done on analysis of those normative
systems and I think those things need to
be integrated and that's why I'm really
excited to be talking about this work
and hopefully opening up some research
programs okay so I'm going to talk about
five lines of research the first two
we'll talk about in some detail in the
last three I'll just kind of hit them to
say these are additional things I'm just
getting started on would like to get
started on we'd love to talk to you
about so let's talk about this first one
designing reward structures and this is
as Marcia mentioned this is work with
with dylon and it's really giving an
overview framework for thinking about AI
alignment so how many of you seen this
little video oh okay so this is out of
open AI they trained this AI to play
this video game it is a boat racing game
which you may not be able to tell
because that boat has not learned to win
the race
it has learn to get points because
that's the reward function they gave it
so you can watch down here the score
it's going crazy what did it learn it
learned that if you just roll over those
those turbo-boost
spots in there you get lots of points
but you never win the game
they have this is posted as an example
of how I mean that they thought they
were choosing a pretty good reward
function for this system to learn how to
play the game but this is what it's
learning how to do so a great example of
the reward design problem here's another
example from other work of Dylan's with
Smith Emily Peter a bill Stuart Russell
and I could do game the last three
butcher his advisers at Berkeley and
this is a paper that Dylan presented
last year at nips inverse reward design
thinking here about the kind of problem
that you suppose you have a designer
wants to train a robot to go find pots
of gold in a little grid world and to
optimize the path choosing between going
over grass which is costly and taking a
path which is less a paved path which is
less costly the designer sets up this
this reward function that's really a
proxy reward function trains the robot
sends it out in the world but oops
out in the world there is lava and the
robot knows nothing there's nothing in
the reward function that designer
specified that takes account of the lava
but of course she does actually have a
preference preference over lava it's
really bad to go through lava how can
how can you but you can if you can't
always anticipate everything that's
going to be out there in the environment
you're going to have this problem of
designing these reward functions that
just don't take account of things that
you're going to discover out there in
the real world that you really do care
about so it's the idea that your reward
function is not really your reward
function and they have a solution that's
basically treating this as an
observation from the distribution of
possible reward functions okay so that's
a way of thinking about the reward
design problem and that's a way of
thinking about misalignment misalignment
between the reward function here that
the the AI is is training to achieve and
the true reward function or the true
values for the humans involved whoops
I'm gonna say let's see okay so I went
but here's the point I want to make so
my PhD is in economics misalignment
between individual and social welfare
functions
is like what all of economics is about
that's basically what we do so when
Dylan and I start talking about this
stuff we started having conversation I
was visiting at Harvard and he was
finishing up at MIT saying okay so
you're talking about these issues you're
studying and I'm saying well that's
sounds pretty darn familiar we have
things in active the first theorem of
welfare economics which tells us that
perfect incomplete markets in which
individuals are just maximizing their
individual profit function and utility
function can't achieve alignment with
social welfare function that's just a
basic result we have a whole field of
principal-agent analysis which is
precisely focused on the problem of how
does a principal who delegates a task to
an agent get the agent to behave in the
ways that the principal wants the agent
to behave and then we have a long line
of work on what to do about the fact
that those those contracts between the
principal and the agent will invariably
be incomplete it won't capture
everything so this is what I was doing
when I was a grad student I was thinking
about franchising sounds like a weird
thing but anyway and I was thinking okay
I was doing contract theory and
bargaining theory game theory and we
were thinking about the problem of okay
so McDonald's over here has things that
want to accomplish its got values
associated with licensing its trademark
to franchisees and it's got value it
gets from franchisees putting effort
into a task and from you know building
its market through new products that it
puts out there you know the frappuccinos
and so on the franchisee over here has
values associated with those things that
are different from the ones that the
principal have and particularly the
print the agent here is bearing costs
now if if they could write a complete
contingent contract that addressed all
possible circumstances and specify the
rewards for the agent they could align
those interests so that they maximize
the joint profit function between them
right so there'd be an optimal amount of
effort there's an awful amount of new
frappuccino and products to introduce
some rate given the cost if they could
write what economists think of as a
complete contingent contract addressing
every possible circum
pants they could achieve the optimum
they can maximize joint joint welfare
but here's what people were starting to
realize way back when when I was
starting my PhD was those contracts
invariably are incomplete it's really
hard to write a contract specifying
effort for the franchisee it's hard to
measure thing it's really hard to write
a contract specifying how often the
franchisor should require new
investments in new products and so on so
those contracts are invariably
incomplete and they they exist in an
environment where they're filled in by
two kinds of mechanisms or institutional
settings so here's my little symbol for
quartz so for example this contract it's
incomplete it's got lots of ambiguity
but you might think that has an implied
in it that the franchisor has to be
acting good faith in deciding when to
require these new products and the good
faith means you can't just extract from
the franchisee franchisees locked in
made all these investment you can't just
extract a value you couldn't have gotten
upfront and you might have courts that
enforce that but you might also have the
community that's enforcing that so get
frantic McDonald's getting a bad
reputation of having a harder time
getting new franchisees or the
franchisee getting a bad reputation or
getting terminated so this is my little
symbol for informal or community
enforcement so that's the world in which
those complete contracts live and the
message that economists were getting
right about then was if we're going to
analyze contracts analyzing incomplete
contracts is fundamental and I think
there's a similar message for people
working in CS on to doing reward design
is to say look misfit Mis specification
of rewards is not just a glitch not oh
you know open AI the folks who design
that boat racing algorithm should have
just been smarter and written a better
reward function to say look no Mis
specification is the state of play and
and we need to think about how do you
manage that it's unavoidable and
pervasive so tender seeing has been
writing
this for a little while talking about
optimal reward design sort of figuring
out what's the best way to do that so we
want to think a little bit of a wire
contracts incomplete economist have done
a lot of work on this what are the
reasons for incompleteness well there's
bounded rationality you can't think of
all the contingencies
there's costly cognition and drafting
there's what we call non contract
ability you can't write all those
contracts and then handers that you
can't explain to courts to everything
you want them to know and if the court
is your enforcement mechanism it's going
to be incomplete for that reason we've
got strategic behavior I don't want to
mention certain things that might happen
the franchisor doesn't want to mention
them I don't want to mention them so
they don't end up in the contract
because we don't want to talk about them
we could plan to renegotiate we know
we'll know more later so let's not try
and get everything upfront right now
let's start our relationship and figure
things out later we can plan to write
vague and I'm gaps terms and then
delegate to a third party to fill it in
later with better information so that's
the set of reasons that economists have
looked at for why contracts are
incomplete and I think you can basically
write down the same set of reasons for
why reward structures might be
misspecified bounded rationality costly
engineering and design non implemented
bility which is what we think is heard
of the analog to non contract ability
just machine learning problems we
haven't figured out how to solve yet
adversarial design might be an analogy
to strategic behavior not so sure about
that one yet it may be that in fact you
want to design something where there's a
planned iteration on rewards you're
gonna have an initial reward you're
going to update that reward later and
maybe we shouldn't be think about
planned completion of rewards by third
parties so I think there's this this
analogy that we can line up between why
contracts aren't complete and where I
rewards or misspecified
now what we do in this little paper is
we go through the insights from the
economic theory literature that our
pause we're just kind of throwing out
there here's some possible things that
you might take from the results that we
have in the economic literature I'm just
going to very quickly mention a couple
so for example there's the analysis of
property rights in this literature that
says sometimes the best thing to do
instead of trying to more finally retune
your contract it may just be better if
you're a principal and you've got an
agent running your firm you may just be
better to sell the firm to the agent
that's transferring property rights to
the agent what's important about that is
to recognize that selling the state the
firm to the agent creating this property
is just a transformation of the utility
function that's all that it means this
is so if we're going to think about what
this might mean in the AI context and
this really is speculative it's not
about giving robots property rights but
it is about thinking about whether or
not the utility function the reward
function has to address more than the
specific task that you're trying to
accomplish do you need to give that
agent rewards that go beyond the
specific tasks that you're thinking
about it performing another set of
results in this literature on
measurement and multitasking sometimes
the basic results showing that sometimes
it's optimal to reduce the incentives on
measured tasks when there are important
tasks that are not measured as well and
the usual example is thinking about
teaching if you want to both have your
students acquire knowledge but also gain
creativity and resilience and so on well
we can measure their knowledge pretty
well with standardized tests we can't
measure the success with which we're
imparting creativity and resilience to
them and so it may say look you actually
don't want to optimize your let's see I
think I jump back too far so it may be
you actually don't want to include this
easily measurable information in the
reward structure for the teacher if it's
going to distort things from the things
you can't measure which I think might
also be something to think about in the
AI context and I'm really nervous with
uber next door to say anything at all
about how this applies in self-driving
cars but anyway you can think about it
we can think but the idea is that you
know if you find if you sort of it's
it's a natural thing to think I've got
this information I should make the
greatest use of it but if there's stuff
you can't measure you may be better off
dialing back how you're using
you're miserable stuff and not going to
the max on that one we also talked in
the paper about theory insights for what
we call strong strongly strategic AI
thinking here of AI that can change its
utility function or change its hardware
and so on we're not gonna I'm not going
to go through that but you can take a
look at the paper if interested the the
thing I really want to emphasize are
insights from the the legal theory of
incomplete contracting
I'm going to keep track of my time here
so one of the basic insights we get in
the 80s is this idea that contracts come
embedded in institutional and social
structures Granovetter is a sociologist
we get the development of something
called the relational contracts approach
emphasizing that contracts consists not
only of their expressed terms but also
of their interpreted and implied terms
so and those are supplied by law and
relation terms so let me sort of talk
about why I think this might be
interesting to think about in the AI
context so this is an example of from a
great paper out of open AI concrete
problems in AI safety if you're
interested so they posit here's a basic
problem you train a robot to carry boxes
to the other side of the room he's got a
gets a reward for getting boxes to the
other side of the room you train it for
that then you release it out into the
world and oops there's a vase that
appears on the path so what it says it
Lissa's like the law of a problem what
is the robot going to do when that would
that well a robot doesn't have any
information about vases going to just
plow straight through the base because
this reward structure says there's zero
there so can that and and they present
this as a problem saying okay how are
you gonna develop robots that can figure
out as they would say kind of common
sense
I don't think let's see if we can dig
into common sense a bit more
structurally so imagine you have a human
agent who's got the same job and they've
got a contract that's exactly the same
as the robots reward function they're
going to get paid for boxes on the other
side of the room and so we're gonna hire
this agent we're gonna leave the agent
alone in the room to carry the boxes and
then the Vaes is going to appear in the
path that the agent has gotten used to
using going across the room what's the
agent going to do the he
an agent gonna go around the vase and
the question is why why is the human
gonna go around the vid then we don't
want to get mushy we don't want to get
just like oh is that's just being human
and that's just common sense we got to
figure out what that looks like how do
humans do it what makes incomplete
Contracting rational why is it rational
for the principal to leave that agent in
the room and not worry too much about
the vase showing up in the path well the
way to think about this I think is to
think about the fact that this contract
here is not the entire contract that's
the expressed terms in the contract the
human agent is going to think huh see
this vase if I take my usual path I'm
going to knock it over and then what's
gonna happen well you know the employer
might sue me might take me to court and
get me for property damages my
employment employment law may allow the
employer to deduct from my wages the
cost of the broken vase or it may just
get a really bad reputation I won't get
a good recommendation from this employer
my co-workers may snicker at me I will
bear some kind of a cost for knocking
over the vase so the true contract is
our - see the cost for knocking over the
vase and the agent is able to fill that
in by reference to this external
structure and go around the vase and
this is the key point the human
contracts rely on a ton of external
structure nobody solves this problem
entirely mathematically inside the box
that's the basic insight of how you
solve the incomplete Contracting problem
you got to bring in stuff from the
outside and I'ma teach contract law when
I'm over at the Law School so if you'd
like to know more about that I'm happy
to talk about it we bring stuff in from
the outside so I think one of the
questions is can we build a eyes that
can similarly fill in their reward
functions can we build them so that
they're able to replicate that human
process of reading and then predicting
the classification of behavior in our
human normative systems right the
external structure I'm appealing to
here's a system it's out there it's not
just somebody's preferences it's a
system that's out there and can we get
them to a
negative way to action is classified as
sanctionable so can we build those kind
of I think we have to figure out how to
build those kinds of systems all right
let me now talk a little bit about this
line of work on modeling human or TV
also with Dylan this is a paper on which
we've I think titled we've we've roamed
around a lot legible normativity for AI
a line at the value of silly rules okay
so remember I just said we are can we
can we build a eyes to replicate the
human process of reading and predicting
the classification of behavior in human
normative systems well to do that if
we're going to pay for they're going to
need we're going to make predictions
about how human normative systems will
respond to specific actions you're going
to need good models of those human
normative systems and this is when I
really get worried about this line of
work because hardly anybody is working
on this from the social science
perspective the focus in a lot we have
lots of people working on norms and law
and throughout the university we have
people in psychology sociology law
working on economics working on norms
but the focus tends to be on the
substance of particular norms so we have
behavioral economics running experiments
playing those dictator or ultimatum
games and coming up with a result that
gets expressed as 30 percent of people
have a preference for fairness and
they'll reject an unfair offer in an
ultimatum game right that's not telling
us how these systems work that is
telling us that there is this fairness
norm and we're not looking at how do
those systems work I think it's also
causing us a problem because a critical
feature of human normativity is it's
arbitrary its capacity for arbitrary
content as a group of humans we can
effectively put anything we want into
our norms and if we can coordinate all
of us to say we won't deal with that
person if they wear the wrong clothing
or if they say the wrong thing right if
they don't contribute this amount to our
joint project
we can establish that as a norm and this
is actually I think why we invent
normativity precisely because it's
capable of taking arbitrary content not
meaningless content but arbitrary
content in order to adapt to different
environments so I think one of the
things we have to recognize about
modeling human normative systems so we
don't want to really model the specific
content in specific settings we want to
look at the attributes of those systems
so this is the point we're gonna have to
they can't just be given the rules
because there's not a list of norms that
are out there in cross cultures across
geography across settings across time in
future worlds we're going to have to
help figure out what of the predictive
basis for figuring out what are the
rules that are actually enforced and in
play in any given environment it's a
project I've been working on with
co-author berry wine gasps who's a
political scientist at Stanford we've
got a series of papers on this and what
we're looking at or what are the
attributes of rule systems that help to
coordinate and incentivize third party
decentralized punishment
we have classifications of behavior this
is okay that's not okay and then the
challenge is how do you coordinate
enforcement of that centralized
classification centralized enforcement
you know the police the government
prisons and so on very it's very rare
it's very new in human societies and
it's also only playing a very small
fraction of the enforcement behavior
even in even in settings like contracts
those contracts are enforced by the fact
that people find out you breach your
contract they don't want to do business
with you and that's what drives a lot of
compliance with contracts so we
emphasize that if you're trying to
analyze that system a key uncertainty
that anybody is facing in a setting is
that understand getting a handle on the
likelihood that others will participate
in punishing somebody who violates the
rule we need to know okay even if these
rules are announced are people actually
going to be enforcing them
and it's so a key uncertainty in any
setting is what rules are being enforced
by others again very distinctive about
human societies third party punishment
right we say so-and-so was rude to my
colleague I don't want to ask so-and-so
to join my research team right so and so
you know behave badly with with my
family member I'm not going to recommend
them for a job right we engage in third
party enforcement and understanding when
and how that's going to happen is a key
problem that humans are solving all the
time all right so what this project does
it looks at what we call silly rules and
silly rules we just mean by silly rules
and it's deliberately provocative name
rules that actually have no value
directly in and of themselves a lot of
our rules about clothing about food
about particular words that we use
things you can say or not say in this
setting you could think of them as silly
rules now it's really important people
who are experiencing the rules don't
experiencing them don't experience them
as silly they think they're very
important but I want to define a silly
rule as one where there's no direct
impact it's only but we want to what's
the function is performing so I want to
say what's important about this for AI
is we want to say okay here's something
you might need to understand that we
don't currently understand about the way
our normative systems work and it will
be important if we're gonna build a eyes
that can interact in those normative
systems that we have good models of this
so this is where you can think of this
as an example to prove my main point
which is we need people interested in
this problem of aligning AIS with human
normative systems to also be engaging in
much more careful research about how
human normative systems work okay so we
do this we give you an example to get
started imagine dropping a robot oh yes
sorry I'm gonna give you some right now
and I'm gonna do it in a dangerous
context okay so we're going to drop a
robot down among the awah of Brazil and
we're gonna win we want this robot to
figure out how to build arrows this is
something that the man among the awah
spend a lot of time doing making making
arrows and part of the reason I want to
use this setting is precisely because it
seems kind of odd to think about
dropping a robot into this environment
to learn how to make arrows but it's not
really all that much sillier crazier
than dropping a robot into Toronto and
saying learn how to drive right it's
just that we are so immersed in the
world of Toronto on how to drive that's
very hard for us to sort of pay
attention a little to what might be what
might be going on in the structure rules
I think this is a real challenge for
doing this kind of work because we have
so many weird we are participants in
these systems much harder for us to get
outside of them okay so here's here's
the kinds of things that this robot is
going to observe and on this so it's
gonna observe that there's actually a
bunch of rules about making arrows those
do you don't know this is Hammurabi's
code my little symbol for a set of rules
Hammurabi's code is not 287 independent
rules it's 287 rules carved into a
7-foot pillar a stone and stuck into the
the central square in ancient Babylon
okay so it's a set of rules so you're
gonna see things like use hardwood for
the shaft use bamboo arrowheads put
feathers on the end of the arrow use
only dark feathers smoke the arrows over
a fire at all times
while they are active and non-active
arrow is one that's been bundled up and
put in the rafters so this is just an
arrow that might be used so keep it over
the fire keep it warm don't let it get
cold making it making use only
personalized arrows make them 1.4 to 1.7
meters long and make and carry as many
more arrows than you're going to use
okay so that's what you'd observe in
terms of the rules these are actually
enforced rules of
how to make arrows here's what else
there are robot will observe men in this
society spend over four hours a day
making arrows in one season they carried
four hundred and two on their trips and
they use nine of them and because they
get bundled up and so many get carried
together they get damaged so a bunch of
the four enough plus hours is actually
repairing the damage that's done for
making and carrying so many in it and
not really using them they actually use
shotguns to shoot most of the stuff that
they're out after this is a tribe that's
living in close proximity to developed
communities so they're but they're but
the man who makes his arrows differently
is mocked and shunned they make a lot of
fun of him for here's soar an example of
what I'm going to call silly rule use
only dark feathers he uses colorful it
feathers on his arrows he doesn't you
they're they're too long they're the
wrong length he doesn't maybe he doesn't
keep I don't know if this is true maybe
he doesn't keep them warm so we have a
whole bunch of rules but not all of
these rules are functionally related to
accomplishing the objective of catching
prey so what Dylan and I did was we did
a computational experiment and we lose a
slide here yeah okay so I think I've
lost the slide in here I want you to
Santa sighs I don't know exactly which
of these are silly but I'm pretty sure
the colored feathers the keeping them
warm over the fire making and using only
personalised arrows that may have
benefits for other settings but you know
because of the social consequences of
not using personalised arrows but you
know an arrow is an arrow it doesn't
matter who made it and this one seems to
be a bit silly for sure because making
and carrying many more hours than arrows
and you're used using is just creating
costs okay so we did is we ran a
computational experiment we put together
communities of a hundred agents in a
group and the group is defined by a rule
set
those of you were thinking about group
identity and so on I think this is a way
of thinking about what group identity
means is what what does it mean to be a
member of this group I followed this set
of rules about what I eat what I wear
how I treat people marriage etc and
we're going to have members of that
group engage in a sequence of
interactions you just want to think
about this like three person games being
randomly drawn in the sequence and and
the rules are being randomly drawn from
that groups rule set okay now we're
going to imagine that there are in this
set of rules there are some silly rules
but there are also some important rules
so the important rules might be don't
steal people's stuff
keep your contract promises don't harm
others there might be some important
rules in the environment but there's
also going to be silly rules and they
were just thinking about they're all in
that set and we want to imagine that
there's high value to being in this
group with these important rules suppose
you're the first group to figure out
that protecting property will improve
economic performance there is high value
to group membership if and only if those
important rules are enforced so
everybody says hey join our group we
protect private property sounds great
but then you're gonna leave your private
property unattended you're not gonna
you're gonna leave your stuff unattended
and that would be great because you can
go off and do productive things while
you leave it unattended if other people
are in fact enforcing that rule but
you're gonna lose if in fact nobody's
really enforcing that rule right so what
will matter is people have told me this
is the rule I need to know whether or
not this group is actually enforcing it
and that's the key uncertainty for our
agents in this model is the uncertainty
is about what's the percentage of
Punishers in this group are there enough
people around willing to punish
violators that I've got a decent chance
that if I walk away from my stuff the
don't steal people's stuff rule be
enforced the important rules and I get
the payoff rather than the costs
associated
with that so you want to think about
that's the variable the agent doesn't
know is the percentage of Punishers in
the group and we want to think about our
agents in a sequence just having to
continue to decide period by period
whether or not to stay with the group or
to go off to some alternative say you
know they're island on their own okay
where they can maybe secure a they can
secure what we could just zero it out
pay off you know they want to know is
there a height so it's obviously got the
structure of a a bandit game and what we
do here is we then vary we have multiple
groups and what we do is vary across the
groups it's a percentage of rules in the
set that are silly rules we hold
constant the number the rate at which
important games come along you know
possibilities that somebody will steal
your stuff right but we're going to
basically insert in so here's our blue
is the important games we're just gonna
vary the number of silly games you play
along the way okay so we're having a low
low density here in the top row and then
a higher density down here in the bottom
okay
so we're thinking about our robot
example we're thinking the question for
the robot is which rule should the robot
learn to follow just these ones I'm
going to call these the important rules
let's say I don't know I don't make
arrows but I'm just guessing or the full
set that seems to include these silly
rules
well that's very much like the problem
that humans are facing all the time
right which group is better which group
should I stay with you can think about
those all I Indians actually they are
having to decide should I stay with my
community in this area or should I go to
town and join integrate into the rest of
Brazilian society so that's a question
about which group in my end we're going
to measure the value of group membership
the size of the community over time and
the sensitivity of the stability of
those metrics to the cost
density of silly rules and then we're
gonna look at this in the context of a
belief shock where all of a sudden all
the population thinks oh wait a second
there's a let's suppose there's a
there's an influx of new members
immigrants to the group and we don't
know about them we don't know if they
enforce the rules or not all right we
think maybe they do but we're not sure
so that's a belief shock and it's a
belief shock if there's actually not
really any change in the population the
people the newcomers are just as likely
to be Punishers as the old timers but
we're also gonna look at a population
shock where in fact there's a change
there's been a change the newcomers
actually don't enforce at the same rate
okay so our hypothesis here that groups
with more silly rules are more likely to
survive shocks to beliefs like
immigration and as I said those could be
the changes and that groups have more
silly rules will collapse faster in
response to a shock to the truth of the
population which is actually when it's
optimal you don't want to continue in a
group if the group's rules are not being
enforced you might like you I wish they
would be but if they're not you don't
want to stay it's inefficient to do that
so I'll just show you a couple of
pictures here from this is first just
showing the impact on the the vertical
axis here is this is the proportion of
communities that are active and this is
the number of in it we call it an
interaction it's every every member of
the group has interacted and had at
least one had one important interaction
so in some of these cases they've also
had a bunch of silly interactions but
important ones and what this is showing
it up here in the top left is that when
we have our cost its you want to think
of that as as a relatively low cost
almost all those societies survive you
can't see the orange ones which are the
the orange sorry the orange ones are the
the communities that have the very
bottom zero silly rules they just have
important rules which is probably the
Society you'd say you'd rather live in I
don't want to live in the society with
silly rules I want to live in the one
that just worries about the important
stuff the blue ones are the ones that
have
I density of lots of silly rules so this
is just showing first of all as from the
from the left as the cost of those silly
rules because you have to help
participate in enforcing silly rules
these are going to be a member of this
community as that cost goes up what you
see is that the the height that the
societies with lots of silly rules
collapse faster eventually we get even
the no one's no silly rules do okay so
here's the results when we get the
belief shock and this is just showing
that the size of the circle is the size
of the community the size of the group
again this is the proportion that are
still active that don't collapse and so
we can see is that the this is which is
our prediction that the societies and
more silly rules are more stable they
stay bigger and they don't collapse
we've still got almost 70% of them
surviving if we have an actual
population shock though again our
societies with lots of silly rules
collapse much faster than our societies
with fewer silly rules so what's the
point about there of all this this is a
world with lots of low-cost and
predictive silly rules because what's
important here is that when I observe
you punishing a silly rule that's
predictive of your punishing the rules
so that I can predict that you are a
Punisher of the important stuff as well
which is actually where this project
started out which is saying why do we do
things like take two hundred and seventy
eighty seven rules and stick them on a
single thing and call it Hammurabi's
code because now all I need to know is
are you enforcing the code I don't need
to know are you enforcing rule 42 and
seventy six and a hundred and twelve
which when I first started thinking
about this the reason I started talking
to Dylan says that seems like a
computationally very difficult problem
and here's a solution let's put it all
in one thing and somehow create the
belief structure that that's the same
thing okay so in terms of connection to
AI a ice may need to read follow and
help enforce silly as well as functional
rules and therefore AI research
also needs more and better models of
normativity so that was just really as
an example for that okay I've got about
five or ten minutes okay all right so
all I'm gonna do is I'm just gonna give
a very quick teaser of projects that I'm
either want to work on started to work
on or have got further on but still
still working on just very quickly here
so I think there's some great work to be
done and modeling norms in multi-agent
settings some of you may know this paper
out of deepmind which used multi agent
learning model to look what happens what
really happens do you know the common
pool problem the idea that if if humans
have a resource that they all have
access to fishing apples in the orchard
there'll be something called the tragedy
of the Commons unless you create some
kind of structure because everybody
individually will go consume too much of
the fish and the fish won't reproduce
and will kill off the the fish or will
eat too many of the apples before they
have a chance to reproduce so what they
did was they developed they they looked
at a community for this and and ran this
but they ran it with the following
technology they gave all of the agents
the capacity like a laser tag that they
could they could tag other agents and
take them out of the competition for the
resource
I think they talk about it as apples
take them out of competition for 25
steps so that they basically I can I can
reduce the number of people going after
the apples and then didn't see what
happens in these communities what did
these agents learn to do and they they
argue there are three phases there's
what they call the naive phase the
tragic phase and the mature phase here
for these societies this is number of
episodes and here what they're measuring
on the on the vertical axis they've got
measures of social welfare okay and the
first one here is efficiency which is
just average reward per agent and the
second one is peacefulness the number of
steps
with untagged agents or converse oh so
when that starts going down here that's
more use of the laser tag right people
are getting tagged more often and so
what there's what they're showing here
is that the agents actually learn they
have a they they actually don't first of
all don't realize they should be racing
around for the stuff and they're doing
pretty well then they figure out they
better race around and grab the stuff
before anybody else and so we get this
as the the collapse of the resource the
resource replenishes every episode but
this is the collapse of it and then over
time they start to figure out the
tagging and they take their they start
tagging the other agents and so the
average reward for each agent goes up
because the effective population is
going down and that's what's showing up
in the peacefulness that they're very
they're not using the laser tags at all
basically at the beginning and then they
start using them so I really like this
line of work and I think there's a ton
to be done developing this because this
is not actually a solution to the comet
these agents have not figured out a
solution that a common pool problem
they're all acting on purely private
incentives they each they're generating
externality but they're only shooting
because it's benefiting them and so what
I'd like to see is whether or not we can
embed a model of coordination of
decentralized punishment of an arbitrary
norm like somebody gets up and says 10
right can we figure out how to
coordinate punishment of anybody who
does more than 10 because that's what
human societies figure out how to do is
there a way to model that and can we
model the emergence of classification of
novel behaviors so if somebody has a new
way of collecting apples right can how
does that get labeled okay so that's
that's that's aspirational I'm not
working with Amy on that but I'd love to
talk more about it here's a project that
I'm just getting started on and with two
great undergrads from the computer
science department who are both here
Victor Kwan Andrew John on we've just
started working on this is also with
Dylan you think about this is the
thinking about related to questions
about interpretability and fairness in
algorithms but it's from a very very
different starting point and one of the
starting points is when can we think
about holding a human responsible for
the decisions of an algorithm can we
develop a procedure for basically want
to think about this as licensing an
algorithm before use that ensures that
the algorithms decisions can be
justified consistently with rules and
principles supplied by a relevant human
community and the thing we really want
to emphasize here is that the decisions
of algorithms in a lot of cases involve
judgments right you've got easy cases
hard cases and your easy cases on either
side clearly answer a clearly answer B
and then we've got this cases where
we're not sure should we do a or B the
other thing we emphasize here and this
is a distinction between say the
interpret ability research it's not
about can we figure out what's happening
causally inside the algorithm humans
need reasons for decisions so if a bank
is going to deny somebody alone using an
algorithm the way we regulate that in
human societies is we have a set of
reasons that are acceptable for that
your credit score was low the prospects
for your bank according to an actor for
your project according to a bank we're
low right we have reasons that are not
acceptable I had a bad day you're a
woman you're not my nephew right there
there could be all these there are
reasons they're not like thats how legal
systems work we put people on the stand
to say we think that you were redlining
in that neighborhood and your mortgages
and we require that there be reasons
provided so there could be that you have
internally within an organization the
desired it is or you could have a
government that wants to be able to say
we need to be sure this algorithm is
behaving in a way that can be integrated
into our human systems of providing
reasons which is different from
explaining let's say what's happening
inside the algorithm so what we're
trying to figure out if we can do and
we're so early in this it's all
I wrote I sent Dell on an email saying
can I even put this up is this too risky
okay so we're not quite sure what we're
trying to think about is can we find in
a data set can we do something like
create a dress code in which there are
things that clothing that allowed
clothing that's not allowed with some
intermediate cases that are hard to
judge could we imagine training
algorithms on a bias set of labels and
some on fair labels and fair here would
just be that's a good faith
implementation of the rules that say you
know it can't you can wear a t-shirt but
it can't have an offensive slogan okay
we're gonna have to decide what's an
offensive slogan and then what we want
to think about is so this is now the
procedure for the licensing you develop
your algorithm you give the human a
human a different human than the
original label or a chance to work with
the algorithm learn about how it works
play with it then you say okay human
we're going to give you a test set you
have to be able to predict the
classification that the algorithm will
provide with some set level of accuracy
because you're going to delegate to this
algorithm the decision and it needs you
need to be able to reasonably
comfortable confidence predict what it
will do and human for any decisions in
that in that test set you will need to
provide valid reasons for the decision
this t-shirt is offensive because it
uses word that words that don't show up
in the in Webster's they only show up in
the urban dictionary the slang
dictionary for example right and what we
would like to see is that first of all
can you even do this can you connect
reasons to the algorithm does it make
sense and does the procedure pass the
fair Elgar algorithm and fail the biased
one so we're trying to think how could
you integrate this so that then if
somebody you released that algorithm it
starts making making won't making loan
decisions and somebody comes along and
says the algorithms made a decision that
violates the rules governing loan giving
in the society can you put the human on
the stand basically to provide reasons
can
assistant with the lungs that were given
in the in the original test set so new
I'm happy to talk about it last thing
very quickly this is on building
regulatory markets or what I call super
regulation and I'm working on this with
folks in in the policy group at open AI
I also talked about it in my book if
you're interested rules for a flat world
so this is our standard picture of how
we regulate we have governments that
establish rules governing what regulated
entities can do so down here we've got
our banks and Facebook and uber and so
on and we have command most of that our
traditional model is command and control
regulation here's what the car can do
here's what it can't do here's what the
bank can do here's what it can't do and
all that detail is supplied by
governments and I don't think we can
ever build the capacity in governments
to regulate particularly powerful AI
systems I don't think we can regulate
what we have now never 9ai
global complex and so on and so what I'm
working again talking with working with
open AI on this as well is can we figure
out how to develop a alternative model
for developing this regulation where we
say okay government you're going to move
out of the role of creating the rules
you're going to move into the role of
regulating regulators you're going to
establish outcomes that regulators have
to achieve and can we create so these
folks in here are private entities they
could be profit-making companies they
could be nonprofits but the point the
key point is they can attract investment
and brains they can compete with the
ubers and the Facebook's and the
universities for smart people to invent
methods and deliver methods for
regulating the behavior of these
regulating entities I just throw that up
there I'm happy to talk to you anybody
about it that wants and I'm done
[Music]


The future of surveillance | Ben Garfinkel | EA Global: San Francisco 2018

I am going to be making the case that surveillance
is an area that effective altruists don't
tend to think that much about, but it's potentially
an area that we should be thinking a lot more
about, especially if we care about the long-run
future.
So the way I'm going to make this case is
that first, I'm going to describe two ways
in which the future of surveillance can be
quite bad.
Then I'm going to describe a more positive
future, and argue that this isn't something
that appears very frequently in discussions
of surveillance, but something that plausibly
more people in effective altruism ought to
be thinking about.
Okay.
So, first, here are a couple of outcomes we
don't want for the future of surveillance.
On the left, we have a depiction of what seems
to be perhaps a bit of a violation of privacy
or not as much accountability as you'd want
for your system of surveillance.
On the right we seem to have, perhaps some
sort of oversight.
It seems like a significant security issue
is unfolding that perhaps we wanted a little
bit more intense surveillance to prevent.
Um, to sort of describe the first scenario
in a less of a caricature, the sort of concern
here is that currently, governments are collecting
a very large amount of data about individuals
all over the world.
This seems to be increasing over time.
Everything we do online mostly ends up collected,
people walking around with things in their
pockets whic, which have microphones and cameras
and GPS locators.
Surveillance cameras are becoming more prevalent
and just seems like in general, we should
expect the amount of data collected on individuals
to keep going up over time.
Probably more importantly, the ability to
use this data is also going up as well.
Partly this is a better a matter of better
analysis, better data mining to identify individuals
from mass collected data, things of that nature.
Partly it's a matter of being able to more
efficiently use the information which is gleaned
from data mining.
Just one quick example, which is fairly benign
but somewhat suggestive: in recent years it's
become somewhat common in some provinces in
China, to used facial recognition cameras
to do things like automatically recognize
people who are jaywalking and automatically
fine them.
A bit of a slightly longer-term thing is the
idea of a social credit score.
This is the idea of using large amounts of
data collected about people, including, for
example, their social media postings or the
crimes they commit, and using this to assign
people a score which will affect their employment
prospects, their ability to travel or get
into certain schools.
And although none of this stuff is yet, I
would say very significant, there's some suggestion
that in the long-run future, or if you let
this go for just a few more decades, we may
see much more strong versions of social incentive
shaping and much more invasive forms of surveillance.
In the long run you might be more concerned
about countries being more authoritarian or
just political institutions we care about
working less well.
The other category of risk that you might
be concerned about is the sort of ineffectual
surveillance scenario.
The sort of argument for this concern is that,
in the future, it may be the case that methods
of causing large amounts of harm become a
lot cheaper or easier to use.
So at the moment if, let's say you're an individual
with sort of unusual motives or bad motives,
and you want to hurt a lot of people, it's
not that easy to do that, to hurt more than
a few hundred people.
But this is to some extent, probably a matter
of what technology is available.
Some people have suggested that, for instance,
synthetic biology, given perhaps a few decades,
may make it easy for a relatively small groups
of people to design pathogens that can harm
very large numbers.
Other technologies which are sometimes discussed
with this sort of narrative are cheap drone
swarms, in the sort of longer-term future
nano-weapons or potentially especially disruptive
cyber weapons.
It's not necessarily clear that any of these
individual technologies is extremely likely
to sort of have this property of making it
very cheap to cause large amounts of harm.
But we can sort of draw an analogy to sort
of suggest what the significance of these
technologies would be.
So suppose that it turned out to be the case
that nuclear weapons were much cheaper to
make than they in fact are, say that rather
than requiring massive state programs and
years and years of work, that anyone could
fairly easily construct nuclear weapons from
household materials.
It seems like in a world of that sort, the
odds that they wouldn't be used would be very,
very low, and you'd likely need some very
pervasive form of surveillance to actually
catch people who were planning to cause this
large amount of harm.
So we don't know that any future technology
will have these properties, but it seems not
entirely impossible that one might.
And if that's the case then we'll want, probably,
much more effective forms of surveillance.
So something which is typically a part of
discussions of surveillance is this sort of
trade off narrative.
So on the one hand, there's this idea that
the more you protect people's privacy, the
less you allow governments to actually protect
people's security.
And on the other hand, there's this idea that,
the more you make government accountable,
sort of let people know what they're up to,
the less effectively governments will be able
to operate.
So sort of to explain or justify the privacy/security
trade off, let's take the case of someone
who's carrying a bag that may or may not have
a bomb in it, and consider a police officer
who'd like to know if it has a bomb, who doesn't
have any tools available to them.
It seems like their two options are to, first
of all, they can potentially open up the bag
and look at what's inside, see if there's
a bomb.
But in the process they'll figure out everything
else that's in the bag, and some of this might
be quite personally revealing.
On the other hand, they can choose not to
open the bag, and therefore not violate the
person's privacy, but in choosing not to open
the bag, they also don't learn whether or
not there's a bomb inside.
The accountability/security trade off, the
idea here is that, let's say take the case
of a protocol which is used to select people
for search or a special scrutiny.
We may want to know that the protocol is actually
being followed, that an individual isn't deviating
from it.
We may also want to know what exactly the
details of the protocol are.
Is it something which is discriminatory?
Is it something which is fair?
Does it have a sufficiently high accuracy
rate?
A case which is often made by governments
to sort of keep their protocols secret is
to argue that, if you make the details of
the protocol public, then people can figure
out how to get around it and it becomes much
less effective.
And so this trade off narrative seems to suggest
that steering away from one risk means steering
towards the other.
Even if you have the mindset that only one
of these two risks is actually credible or
of significant importance other than the fact
that other people care about the other risks,
it means that there will be political constraints
on pursuing solutions to one risk or the other.
As a sort of an extreme caricature, let's
say that you're someone who doesn't care about
privacy at all, you think that the risk from
authoritarianism isn't in any way important,
and you think would be really great if the
government put cameras in every single person's
home and watched them all the time.
The fact that other people are definitely
not cool with that, and care about privacy
means that that would just be a nonstarter
as a solution.
So in general, it seems like the more severe
the trade off is between these values, the
more concerned we should be about either risk
or both risks together.
This all seems to suggest that a useful thing
to do would be to look for opportunities to
reduce these two trade offs.
This means looking for ways to make surveillance
more accountable and privacy preserving.
While this sounds a bit idealistic, we can
get some intuition that this is possible by
looking at different forms of surveillance
which are applied today, and that definitely
vary quite significantly in how much they
protect people's privacy.
So if we were trying to get into the case
of a bag that may or may not have a bomb in
it, suppose that instead of just opening the
bag, a police officer has access to a bomb
sniffing dog.
In this case, they can have the dog come up
to a bag, sniff it.
If it barks, they open the bag and search.
If it doesn't bark, they don't.
In the idealized case with dog that has a
perfect accuracy rate, they only learn exactly
what's relevant for security.
Does a person have a bomb?
But they don't violate people's privacy in
any other way.
And the more accurate it is, the less it violates
people's privacy.
At the same time, this is also a fairly accountable
form of surveillance.
If you have your bag on you, you can tell
that a dog was used rather than just a person
rifling through it.
You can also tell whether the dog barked.
It's difficult to lie that a dog has barked
when it hasn't because you can hear it.
Yeah.
So that's a sort of a specific case.
A more sort of abstract case for optimism
is that in the future, it's likely the case
that surveillance and law enforcement becomes
more heavily automated.
While this has a number of scary components
to it, there's also some reasons to think
that this trend may actually make it easier
to ensure privacy and accountability.
So here's some basic advantages of automation.
So first of all, if you automate an analysis
task that would ordinarily be performed by
human, then you can use software as a screen
between data and the humans who see it.
So the analogy is to a sniffing dog.
Again, if you have some piece of software
which looks at data and make some initial
judgment of whether to search further, then
potentially human doesn't need to look at
data that they would otherwise look at.
You can also potentially you redact sensitive
information automatically so that no human
ever needs to see it.
So one concrete example would be automatic
face blurring of faces that appear in police
body camera footage.
In certain regards, algorithms are also more
predictable and less opaque than humans.
To some extent, AI is often a black box, but
it's less of a black box than the human brain
is.
You can't really look at a human police officer's
brain to sort of see what's going on there.
But you can often look at the source code
for software.
It's also often easier to associate things
are done with software with reliable audit
logs, as opposed to let's say, trusting human
analysts to record what they're up to.
Software's also less likely to engage in certain
abuses that a human might.
So one slightly disturbing example of this
is, there's this concept, at least in the
past, hopefully not in the present within
the NSA of LoveInt, where this is short for
love intelligence.
The idea is looking at information on a significant
other or an ex, and this was apparently common
enough that they had jargon for it within
the NSA.
Seems like something weird has happened if
software that you've designed is doing that.
At the same time, if you're using software
in place of humans, it also potentially becomes
easier and more efficient to automate a single
piece of software, as opposed to auditing
lots of different humans who might be replicating
this behavior.
So if you're using a single piece of software
in lots of different cases, then potentially
you just look at the single piece as opposed
to lots of different human analysts and officers
and officials, who might be deviating from
protocols.
It's also easier to associate a piece of software
which is applied in lots of different cases
with summary statistics, for example, in accuracy
rate, compared to using statistics for humans.
So I think this is actually like a large issue
currently with basically surveillance in law
enforcement at the moment, is that it seems
like intuitively if you're establishing probable
cause or reasonable suspicion, it's sort of
like a probability threshold for that.
Exactly how accurate is, for example, an officer's
judgment that someone meets these thresholds?
What portion of the time are they right?
For an individual officer, this isn't extremely
feasible to collect these statistics, but,
for example, using a facial recognition system,
you actually have fairly good data on exactly
how accurate it is.
You can actually have a fairly informed discussion
about what sort of false positive rate is
too high.
That's a bit hard to have for humans.
A couple of less obvious advantages are that
increasing automation can actually decrease
the need for data collection, and that increasing
automation can decrease the disruptiveness
of engaging in auditing.
So the idea for surveillance without data
collection, the basic idea is certain cryptographic
technologies make it possible to analyze data
or extract certain pieces of information from
it without collecting the data in unencrypted
form.
Probably the most notable one is a technology
called secure multiparty computation, which
is extremely general.
And recently just in the past decade or so,
became much more practical to use.
A couple of examples of how this technology
can be used: so the first is the idea of a
set intersection search.
So this comes up fairly commonly in law enforcement
contexts, where you want to identify someone
that's suspicious on the basis of the fact
that they show up in a few different databases.
So one concrete example is, say someone robs
a few different banks.
Police know it's the same person, but they
don't know who it was.
They might want to search the cell records
for the cell towers that were near all three
of the banks to see if anyone made calls near
all three of them.
The way you would traditionally do this, in
a way that actually historically has been
done, is to collect all of the records from
these three cell towers, get tens of thousands
of people's records, and then comb through
them and see if any name pops up three times.
But it's actually possible to do this without
mass collecting records in this way.
So there's a paper in 2014, that shows how
to conduct a search of this sort.
Where you get out a list of names that appear
in all three of these databases, but you don't
get any other information besides just those
names.
So rather than collecting tens of thousands
of people's information, you collect maybe
one or two.
Another example is fraud detection.
So for the case of value added fraud detection,
one way you sometimes do this is by finding
discrepancies between different companies'
private financial records.
One, let's say, reports a purchase, that doesn't
match another country's report of a sale.
There's a paper in 2015 that describes a protocol
that I believe has actually now been used
by the Estonia government to find cases of
tax fraud of this sort, without collecting
companies' private financial records.
They get this output of here are the discrepancies,
but they don't actually get any unencrypted
records, sort of taken in, so they can't learn
anything else other than just who is showing
discrepancies.
Another example, which I'm also not going
to get into the technical details of, is this
idea, traceable to a paper by Joshua Kroll
in 2016, to use a cryptographic technology
called zero knowledge proofs to produce accountable
algorithms.
The basic upshot is that he shows that it's
possible in many cases to prove to the public
that a protocol that's received their approval
is still being applied.
That people aren't straying from it.
And also that the protocol has certain desirable
formal properties, for example, fairness properties,
without actually making it public, what the
details of the protocol are.
So that's desirable, if the reason for not
making the details of a protocol public or
that you can say, oh, if we made them public,
people could get around it, or potentially
you're a law enforcement agency and you can't
make the details public because some private
company has developed it for you and it's
quite profitable.
So this is a way of making things more accountable
while dodging those objections that making
myself more accountable would make you less
effective.
So in my fairly non-expert view, I see sort
of a few opportunities for things effective
altruists could potentially add to the conversation
around surveillance.
So one is that in my opinion, the conversation
is typically too focused on managing trade
offs.
For example, debating exactly how much security
you can get by trading away this amount of
privacy, or sometimes denying that these trade
offs exist to any extent, which seems implausible.
Rather, I think it'd be more useful to look
ahead to technical solutions for actually
sort of reducing these trade offs.
Another concern I have is that a lot of the
conversation concerns current programs, or
programs which are just getting off the ground.
I think it would also be productive to have
a conversation about what forms of surveillance
we might want to be moving toward, let's say
over a 10 or 20 year period as new risks emerge,
and also as new technologies make different
forms of surveillance feasible.
And then the last one is that I think discussions
of surveillance are often sort of reliant
on assumptions about technology which aren't
actually true or are going to become less
true in the future.
So a classic one is just that analyzing data
actually requires collecting data, which isn't
actually technically true.
One last comment is that this presentation
has been all about, sort of mass surveillance,
but a lot of what I've said also applies to
the case of agreement verification in an international
relations context.
So, the idea here is that frequently you want
to verify compliance with an agreement, let's
say an arms agreement, but often the process
of monitoring the country or verifying compliance,
involves collecting lots of information which
is sort of revealing in a way which is viewed
as negative.
So for example, it gives away to details of
weapons systems or allows countries access
to private actors' labs that might be indicative
of sort of valuable intellectual property.
And if you can find ways to make sort of more
privacy preserving a forms of monitoring in
the same way, you can make more privacy preserving
forms of surveillance.
and this could potentially reduce the bottleneck
on the ability to actually reach international
agreements.
And this is also something that seems to intersect
a lot with existing effective altruist concerns
around global catastrophic risks and governance
of emerging technologies.
So just in closing, in the future, surveillance
might threaten the institutions that we care
about, or it might fail to protect us for
new threats.
It seems like there's some sort of trade off
between addressing these risks, but these
trade offs also don't seem to be immutable.
There is some hope that in the future, technological
progress can help reduce these for us.
So therefore it seems like the project of
pursuing accountable.
privacy-preserving surveillance, while not
something that many people are engaging in
at the moment, might be something that more
effective altruists want to look into or signal
boost in conversations around surveillance.
I'm going to be giving office hours at 10:30
AM tomorrow, if anyone wants to talk more
about that, and I'm also just generally free
to talk if anyone has any interest.
Yeah.
Thank you so much.