Stanford Seminar - Emerging risks and opportunities from large language models, Tatsu Hashimoto

all right so thanks all for coming uh
today i want to talk about large
language models the topic of of intense
interest by by many faculty here um
and i think i want to highlight some
risks of being too excited about them
but also the opportunities that they
enable for problems of safety and
broader societal concerns
um i'm i'm imagining that not all of you
are familiar with language models and
what they've done for the field of nlp
so i'm going to open with this kind of
slide and in fact the slide is now
pretty old but i think it gets a
the huge gains that have been a result
of language models so this
chart shows the benchmark performance on
squad the stanford question answering
data set um and this is a really classic
benchmark that tests our ability to
answer questions based on wikipedia
articles right one of these classic
challenges
um and performance had been pretty
stagnant i mean there were some
improvements up to 2018 and you see this
like real inflection point where we got
this big jump of maybe like 10 plus
percent or whatever um and this is all
really attributable to the use of
pre-training that is the use of large
internet corpora uh to initialize our
models um and then uh as well as like a
particular model called from google
and so these kinds of systems have
really driven dramatic performance so
this is one example from a question
answering data set but just kind of
imagine this you know
several percentage gains applied
everywhere in the field it's a very
dramatic change
so they really power nlp systems today
everywhere and so i think empirical nlp
systems building 2018 onwards is really
this paradigm of you take this big
language model which is designed to sort
of predict what words come next on large
internet text
when we take these systems and then we
fine tune them on some downstream tasks
maybe you want to make a system that can
judge the quality of a piece of text or
whatever you would take your language
model you would fine tune it and then
this is really how a lot of these things
work
so the performance of this paradigm is
really impressive but there's important
questions that remain even though we
have high performance the first thing is
can we truly trust them in kind of like
high-stakes situations and i'll talk
about one example in a few minutes um
and the second thing is that this
pre-training paradigm we've it feels
like we've gotten something for free
right we go on the internet take
publicly available data and we've gotten
performance improvements what costs are
we paying uh through that paradigm are
there new risks or harm so that's the
second part of this talk that i want to
talk about
so question one is can we trust these
models and because of the impressive
performance that these systems have
achieved and the continual improvement
of this like pre-training paradigm over
the last let's say three or so years
people are getting more ambitious and
more excited about what we can use these
models for so on the top left this is a
this is a really nice paper sort of a
sort of inspiring thing by some people
from google research saying maybe we can
use language models to in fact guide the
creation of data sets right and that's
actually pretty cool and also a little
bit crazy because data sets are you know
our most cherished things that guide how
machine learning systems behave and they
show some nice results here on the right
some people have tried to use
pre-trained language models plus fine
tuning as a way to evaluate models and
that's also another high-stakes setting
right so now we're applying these
pre-trained language models to the most
critical parts of the machine learning
pipeline data generation and evaluation
and are these models truly so good that
we can use them in these kinds of high
stakes tasks so i want to answer that
question in the first part of this talk
the second thing i want to talk about is
new kinds of harms and attack vectors
and things like that that arise from the
use of language models as part of our
systems
i want to highlight privacy in
particular because i think it's a new
and unique thing that happens
because we're training these huge models
on large internet text we can now
recover various pieces of text from the
internet at first this seems relatively
harmless why do we care about the fact
that we can recover pieces of text from
the internet but we'll see that there's
valid concerns about privacy and the
protection of sort of what was formerly
hidden information being revealed by
these language models so language models
even though they help performance they
may be very detrimental for things like
privacy and cause some serious sort of
security risks as well in the in the
process
so
um this really i think a lot of my
thoughts and some of this work the
directions of the work have been shaped
uh by the center for research of
foundation models i'm sort of giving to
a slide plug here it's a sort of big
initiative with with many faculty across
many departments um and
if you want to know more about language
models and their use and sort of the
background surrounding this i would
encourage you to go and look at the
relevant parts of this white paper that
we wrote that interest you um the the
title of this talk is sort of a
reference and a homage to exactly this
white paper so i think a lot of it i
think you'll find interesting if you
find this talk interesting
um so the first thing i want to talk
about i want to talk about two pieces of
work uh that i've sort of recently been
doing
one of them is in this evaluation metric
sense um what kinds of
new capabilities and harms and pitfalls
might we see when we apply language
models to high stakes tasks so
especially
evaluation so i want to highlight again
this like data generation thing because
i think it's pretty striking of the
amount of trust that we're putting in
these kind of language model systems
i think data generation is really an
emerging thing this paper appeared i
think the europe's data sets track just
last winter but i think talking to other
researchers in the area this is a thing
that people really think is going to
like continue to grow right so this is
an example of what's called synth bio
i'll talk about this in a little bit
more detail now on the left side um
is a
fictitious biography this person doesn't
actually exist it was synthetically
generated
of something that looks like a wikipedia
info box right so if you go to wikipedia
you'll see a box that has like a name
and their occupation and you know the
notable works of a novelist so basically
you have this in structured table form
and the task is to generate a piece of
text on the right and if you sort of
stare at this it's actually pretty good
this is mostly written by a language
model
and slightly edited by a human um and so
the success of these kinds of things
really give us optimism that language
models can maybe in fact be used in a
lot of high-stakes tasks data generation
and evaluation
um
and i think this has been a trend that
has been going on for a few years now of
trying to use language model fine tuning
and things like this to automate model
evaluations um and so these are some
sample papers i'm not making a
commentary on whether these are good or
bad or so on
but they're sort of attempts at taking
the high performance of language models
and using them to evaluate dialogue
systems
so the setup is something like this you
have built a chatbot you would like to
evaluate how good this chatbot is but of
course chatbots return natural language
and there's no easy metric like accuracy
that you can compute so the traditional
paradigm is to sit a bunch of users in
front of this chatbot and ask them how
good was this interaction
that's actually pretty expensive and
pretty hard and so there is this dream i
think of training language model based
evaluators based on human judgments that
we've collected offline and then
evaluating our models against this
predictor of how good a human would
think the system is so then we can get
low-cost evaluations of our models and
we can sort of improve our chatbots and
so on based on this evaluation scheme
right like it's a very compelling
vision for how we might eventually
replace humans
but
at the same time
you know it's sort of a question of are
these evaluation metrics really going to
succeed right in the sense that can we
really build something that's as good as
humans um and i think the thing that
people have been sort of excited about
is the fact that some of these proposed
metrics seem to be coming close to human
level performance so on the right side
here is a whole bunch of correlations
right and on the very top you see human
correlations with each other so that's
like roughly not the very best that you
can do but a good sense of what a good
system should do
and the next two sets of numbers is like
traditional
quality evaluation metrics in natural
language generation and at the bottom
are these learning based ones that are
like powered off of these language
models and you see at the very bottom
like some of the newest methods that
people have developed are in like this
0.7 correlation range that's you know
the human average correlation so you see
this table right like someone shows you
this and then you're going to say wow
you know our learning based evaluation
metrics are getting really good right
like are they ready to be deployed and
used and trusted right
but looking at this as a skeptical uh
researcher you know it gives you pause
because we know that neural systems
especially like language model based
systems
they can achieve incredibly high
performance and yet at the same time
fail in really silly ways um this is an
example that i got from from my student
last night um this is a similar
benchmark or generation
task as the synth bio data set i showed
you before so this is a task of uh you
get as an input some data fields about a
restaurant so this is a restaurant
called the aromi which is chinese and so
on and so forth and the task is to
generate that sentence on the right that
one was human written or actually no
this was this was uh generated by model
aromi is a chinese restaurant near the
crown plaza hotel blah blah blah blah
right so if you use a pre-trained neural
model for this task you get beautiful
text like this like you know fluently
written
matches the input so on and so forth
this is a task called eqe and i think
most people consider this to be a
completely solved task around 2020. one
of my students even calls it easy to
easy
now the thing is it's not really easy
because if you replace the restaurant
name with starbucks which is a
completely reasonable substitution to do
suddenly the system completely falls
apart it now thinks that the chinese
restaurant is called the crown plaza
hotel which is the thing that it's near
it's not the name at all
so it can just blow up in completely
unexpected ways when you go a little bit
outside of the training data set and so
these kinds of experiences i think make
me sort of as a prior a little bit
suspicious of this belief that we can
just take models that look high
performance you know in our normal
evaluation and then go out and deploy
them into the wild or in high stakes
settings
so okay
um my my postdocs s and physol were sort
of similarly skeptical and they were
looking into these evaluations and our
initial interest was in doing things
like evaluating the factuality of
language generation systems and what
they found was something a little bit
troubling at the very beginning
they found that factuality metrics for
summarization tasks had this really
weird structure like first of all they
didn't seem to perform particularly well
compared to baseline so if you just
measure word overlap they seem to do
quite well and second of all they seem
to have higher correlation with simple
things like word overlap than with human
evaluation and those two combined make
us a little bit suspicious are they
truly learning the task or are they
learning these surface structures like
are they learning to just look at word
overlap in a slightly more clever way
and i think that's the first thing that
i want to emphasize in this first part
of the talk the brittleness of these
high performance systems that can arise
and the problems that can arise from it
these things called spurious
correlations where a model is picking up
the surface level features rather than
the underlying task they're pervasive in
uh these kinds of automated evaluations
as well as more broadly in machine
learning and what we sort of showed in
this set of works is that really they
rely on just really simple stuff like
the word overlap to your input
perplexity which is a measure of
compressibility of your sentence and
length of the sentence these are just
really simple stuff that has very little
to do with the underlying quality or
factuality of generated text
and yet it seems like these are the
things that are driving a lot of the
performance rather than understanding
the underlying task so we'll go through
a few more examples next but i want to
give you the bigger picture because not
all of you are excited or interested in
the nlp or natural language generation
evaluation but i think all of you are
hopefully interested in questions of
robustness and reliability and spurious
correlations are exactly the sort of
thing that is troubling for these
systems
so what do i mean by that
they're sort of a standard place where
we usually see spurious correlations and
i've talked about this or worked on this
in the context of machine learning and
nlp broadly so i'll give an example and
i'll talk through this example
so that this
top thing shown in the the top half of
the slide is a task called entailment so
you're supposed to get in two sentences
like a premise and a hypothesis
and what your task is is to say is the
hypothesis entailed or logically implied
by the premise so here the economy could
still be better hypothesis the economy
has never been better and so that's not
entailed from one to the other right so
this is a classic sort of ai challenge
um in nlp
and what's interesting is that when
people constructed the very first
benchmarks for this task
there was a spurious correlate it turns
out that crowd workers when asked to
write non-entailing sentences really
like to put negations in right so this
is a reasonable human bias when you're
asked to write something that
contradicts an input you might just
negate it right
but what this does is it introduces a
really simple spurious correlate or a
surface level feature that a model can
pick up and then exploit to solve the
task so that's what happens here
so we can sort of think about this in
this like two by two diagram on the
bottom here so what we would like to do
is to predict the label whether
something is an entailment or a
contradiction
but really it turns out that this thing
is correlated with less spurious
attribute whether or not a sentence has
a negation or not
and the key thing is that because of the
dataset bias we only have or we mostly
have data on the diagonal here
entailment with no negation or
contradiction with negation because
that's true
it turns out to be the case that just
picking up on negation is a really
effective strategy for predicting
entailment right this is not what we
want the models to learn and this is the
general principle i want you to remember
from this first part of the talk this
idea of spurious correlates and how it
can sort of deceive us into thinking a
model has understood a task when in
reality it has not at all right so here
in this case using the spurious
attribute gives us low average error
even though there are parts of the space
that we're doing really badly on
any questions on this high level thing
okay
so the problem really is the same thing
in this evaluation case what we would
like to predict is something like
in this summarization task do we have a
factual uh summary or do we have a
non-factual summary and the spurious
attribute in this case is overlap if
you're using a lot of words from the
original article during summarization
you're probably factual right so you
have this spurious correlate that's
really easy for the model to pick up
that gives you low average error
on the other hand we see that in the
worst case when we're looking at
examples of non-factual text with
overlap so you add like a negation or
something to like intentionally break
the factuality of your text or you're
looking at the bottom left a factual but
very little overlap where you've
paraphrased the summary in both of these
cases these kinds of evaluation systems
will be expected to fail right so the
way to think about this conceptually is
that a lot of the cases where we have
spurious correlates our metrics are very
very accurate in the easy case right
when the spurious correlates match the
label but they fail sometimes
catastrophically in the hard cases where
these heuristics that the model has
learned are no longer relevant
okay so now i'll get into slightly more
details of this and the point of this is
maybe not to convince you that this is
you know a truly
exciting area for natural language
generation evaluation but i want to show
you the scope of how deep this goes it
seems like this problem is really
pervasive so we looked at a couple
different metrics and we looked at a
whole bunch of different benchmarks for
for dialogue and summarization and we
really found the same phenomenon across
all of them
so to just to remind you right these
metrics that we picked uh mod dialogue
rpt and usl these are supposed to
measure the quality of uh a chat bot
system and its response right and the
higher the better because it's more
correlated with humans and on the left
side we've taken some very simple
heuristics just looking at the length
looking at the compressibility of the
sentence called perplexity or a
combination of the two and what we often
see is that these very simple heuristics
if we like carefully figure out what the
model is trying to learn
we can actually get them to do better
than a lot of these learned models and
that's disturbing and once again on the
right we see another disturbing trend
which is that these automated evaluation
metrics seems to correlate more with the
heuristics like perplexity or length
than with humans so on the y-axis here
is the correlation with our learned
metrics and we see it's more correlated
with the spurious correlate than with
humans so this is kind of the general
pattern that we see these systems are
very high performance but when we put
them in situations they haven't seen
before they really seem to be relying
upon these heuristics
we see the same thing on persona chat
which is another dialogue evaluation
case you see that most of these learned
evaluation metrics aren't doing much
better although usl is doing a little
bit better and on the right side once
again we see that the spurious
correlates are more correlated with the
learned metric than with humans
finally to drive the point home this is
the very last one um daily dialogue is
another dialogue evaluations that here
we're basically getting no correlation
because daily dialogue somewhat has a
different distribution than the data on
which these models are trained
so in all these cases we see a very
different story from the original
optimistic view that we had
just because the system seems to be
doing well on some sort of benchmark
performance numbers doesn't necessarily
mean that it's going to do well out of
distribution nor does it mean that it's
actually learned the task right we can't
really trust uh average case performance
numbers on face we really have to look
deep and look at what the system is
learning and test it out of its
distribution
and so what we found here is that we
found at least three different spurious
correlates in that they were very
predictive of performance and that they
correlated really highly with these
learned metrics right that was the thing
that we found these three things
and i think the key problem was sort of
the current evaluation paradigm is that
we are testing models on the data on
which they are trained not literally
train test overlap but we're taking an
iid sampled set right but when we take
these models and we evaluate them on
different data sets different kinds of
dialogues to evaluate they seem to
totally fall apart and they seem to rely
a lot more on their heuristics
than on actually understanding the task
and so the key point here
is that to make progress longer term on
these kinds of hard problems we need
something that's more of like a robust
evaluation metric right we need to take
these systems intentionally out of their
comfort zone and then test to see what
they've learned
yes
od performance right like it's not like
um a priori bad that it's worriously
correlated or is it actually a prior
event what do you mean by a priori
did not exist and using heuristics would
be kind of fun right right so i guess
the well it depends on what you what you
believe right so if you think that the
original training data set is truly
representative and it's the performance
you care about then you're right that
like you know at least for that setting
you do have a good system that relies on
heuristics
i think in almost no situation is the
training data sets we use in academia
actually reflective
of the real world use cases or the real
world deployment conditions that we
expect they're they're an approximation
right
hopefully it's a good approximation but
but often it's not the case
i think especially for evaluation this
is true because you're almost never
going to be evaluating the same
conditions because you're making a new
model and evaluating that and that's
almost intentionally a distribution
shift so this is a case where it's
really hard to defend that it's going to
be iid
but good question
okay
so
you know this is a little bit abstract
i'm talking about nlg evaluation which
not all of you are probably you know
really familiar with and i think one
question you might ask because i've
shown you a bunch of correlation numbers
right like you know correlations can get
low but you might not really care about
this you might say like why do i care if
the correlation numbers are low right
really the the thing that i'm going to
use this thing for is for developing
models and maybe i can still pick the
right model or the best models using
these evaluation metrics even if their
correlations are low so i'm going to now
demonstrate a direct harm that can
result from using these kinds of
evaluation metrics and trusting them
so i'm going to describe the setup in
one slide once again i'm going to talk
about summarization so the the goal is
to build a summarization system that can
take in an article and produce some sort
of a summary
and the interest in a lot of this field
is producing abstractive summary so you
don't want to just take a couple of
important sentences you want to like
paraphrase it and synthesize it into a
useful summary that's the task that
we're interested in
and factuality is a huge problem for
summarization so we want systems that do
not lie to their users when they're
performing summarization so we're going
to look at a big evaluation data set
that some people from salesforce
produced that evaluated 16 different
systems and did fine-grained human
evaluations for factuality this allows
us to see whether the rankings produced
by these automated evaluations match
those produced by humans right and
really what we can show is actual direct
harms from this kind of framework
and i think one thing that's really
interesting
is people have said something like
actually these kinds of automated
evaluations are really good at
distinguishing which models are good or
bad but once again there's kind of a
spurious correlate i've been talking
about this like word overlap thing and
that's really reflective people find
that things with high word overlap are
generally more factual that's just kind
of true right
but really what we would like to be able
to distinguish is the top left corner if
we have a system that doesn't have very
high word overlap can we distinguish how
factual it is
and the answer is no so this is a little
story that i want to tell there's three
different systems on this slide and they
come from kind of three different
generations or three different time
periods pointer generator is one of the
very first like neural summarization
systems developed here at stanford
pegasus and bart are much more modern
pre-training based models
and here we see the very top row human
evaluation slowly and steadily improves
over time this is just models getting
better right but what we find is a very
different story for the automated
evaluations if we look at fact cc or dae
these like neurally based evaluation
models on the bottom what we find
is that these same systems say that the
oldest model is the best one and that's
because this oldest model the point
pointer generator did the least amount
of rephrasing it just sort of copied out
parts of the article
because this these systems rely on the
spurious correlates they're actively
saying these newest best models are
actually not the ones that we want right
and so the direct harm here is that if
we were to use these systems to try to
rank the factuality of our models
we would actually find that you know
these oldest models are the best and we
would be harming sort of progress right
so these are kind of demonstrations of
direct harms from this kind of spurious
correlation that we see
um
in the interest of time i'm going to
skip over some of the some of the
details but the point i will get to
here is
one thing that we can do is not just
rely on you know the progress of larger
models to make our systems better if we
think a little bit more carefully about
what's happening we know that there's
some spurious correlates like word
overlap or the length of a sentence
and what we would like to do is we would
like to minimize the model's use of
those spurious correlates so we can
design an architecture that does exactly
that taken from domain adaptation and if
we actually do this what we find is we
will get dramatic improvements in the
predictive power of these kinds of
automated evaluations on the set of
models that we care about things that do
abstractive factual summaries i'm going
through this a little bit fast but the
point here is to say that by explicitly
trying to prevent the model from using
these kinds of spurious correlates we
can then get uh improvements in terms of
the actual metrics and rankings of
models that we care about
the final punch line here is that i was
only talking about robustness here right
like i wanted to improve so the worst
case performance in some sense of these
models and make them use more reliable
features but when we did that and we
tried to enforce robustness we didn't
really pay a cost in the average case
accuracy either um here the the error
bars are large enough so i'm not going
to say that the adversarial robust model
is going to be better than the others
but it's suggestive evidence that
actually enforcing robustness could
maybe lead to general improvements in
the performance of our system so i think
that's kind of a hopeful note to try to
end on here
okay
so so we went sort of into the low level
here talking about a lot of results and
a lot of like failures specific failures
of language models so i want to pop back
up and get to the bigger picture again
and talk about that
so language models are increasingly
being used in high stakes areas data
generation evaluation and i think it's
good that we are being optimistic and
trying to push the boundaries
at the same time just because they seem
to do well on some benchmark numbers and
some evaluations we can concoct doesn't
mean that they're actually good we need
to take a very skeptical view of how
good these systems actually are
and so i think that sort of requires us
to look at the robustness of these
systems to rule out whether or not
they're using
these kinds of simple heuristics or
whether they're actually learning the
underlying task
and the final thing which is specific to
our work is there are ways of removing
these kinds of things and explicitly
building those in rather than relying on
sort of scale and you know more data to
solve these problems is a very effective
direct approach to addressing some of
these issues
okay i'm going to stop here in case
anyone has questions about the first
part i'm going to totally switch gears
and talk about privacy next so i want to
make sure that i i answer any questions
that anyone may have about this this
first segment yes uh so when you like um
sort of talked about how you resolved um
or tried to tackle this specific issue
um is that largely through like um data
tuning or is it actually
it's actually i would say more from the
loss
um i skipped over this because i didn't
know how deeply i wanted to go into this
but you basically make a neural
architecture that has two parts the top
part is the same as your normal
prediction task so you're just trying to
predict whether a system is let's say
good or factual or not that's the very
top half of this architecture the bottom
half is is an adversarial component that
basically says if i can predict my
spurious correlate like the word overlap
then that's bad because it means that
the model could be using that feature so
i have this adversarial head on the
bottom that prevents my model from
making use of that it makes sure that
there's no such information in the model
this is all very hand wavy but i think
that's a little bit better than digging
through the math of
the main
adversarial neural networks yes
yes yeah in this case it's word overlap
which we know is extremely strong
spurious correlate
yes
if the correlation level is higher than
what it would be with the humanoid okay
or i guess that the way i would formally
define it is a spurious correlation
is a is a predictive correlation that
goes away under some distribution shift
okay um and as for like identifying what
those are yes and you know about the
word
you mentioned about the negation right
do we have a sense of how many other
stories
yeah that's a good question i don't
think we have a like a comprehensive
inventory i do think perplexity has
often been a perplexity or like
simplicity of a sentence as measured by
some language model is is a really
strong correlate to almost everything
right like if you build a toxicity
classifier for example you'll find that
like toxicity is very unfortunately
correlated with with fluency like as
measured by perplexity so we have some
sense of like things people have talked
about but i think the full universe of
serious correlates is completely unknown
okay cool this is a good good timing so
so now i want to kind of almost
completely switch gears i want to talk
about privacy um so the first part i
think was partially a fully depressing
story it was like well there's some
great things but also there are some
like really big pitfalls the second part
is more more mixed and some of this
content is taken from um my class on on
large language models with percy in last
quarter
so now let's talk about privacy and
privacy risks like why is there sort of
this
big issue about privacy that people talk
about for language modeling
so one of the reasons is because
models are extremely data-hungry i think
this is the thing that sets the stage
shown here um this is not a language
modeling
task this is i believe for a machine
translation task
but what you see here is on the x-axis
the training data set size in log scale
that's important and on the y-axis is
kind of an error measure like how many
words we got wrong when we're performing
translation and that's on the y-axis um
also logged
and what you see here is this fairly
nice line on a loglog scale right and
what that means is if we you know
double the amount of data set we have
then we'll get some multiplicative
decrease in our error rate
but this requires and necessitates that
we continually increase our data set
size by some multiplicative factor if we
want to continually decrease our error
rates right
so that's actually that's kind of tough
right it requires ever larger data set
sizes in order to get ever smaller
errors
um
and the issue here is that there's kind
of a trade-off even right the amount the
kinds of data that we can get our hands
on
is limited on the left we could go to
the internet we can scrape the internet
and produce enormous data sets right the
largest open domain chatbots are
generated by looking at um uh
conversations on reddit right there's a
lot of those available right but they're
fairly low quality that's subjective
judgment um on the right side we could
have really motivated annotators writing
down really really well thought out
careful conversations right but those
are very expensive if you're being a
good person you're paying these people
you know a really decent wage right and
how many samples can you collect maybe
10 000 conversations that's tiny by
neural network training standards right
and so there's a forbidden third path
that i think people have started to
think a lot about which is can we
utilize things like private user data or
even like slightly sketchy copyright
protected data when we're training
language models right there's an ever
greater pressure to increase the scope
of the data that we collect and use and
this is just kind of a natural dynamic
from the fact that our models are so
data-hungry
and i want to talk about a really sad
example of this um
so this is an article that i found while
doing my research earlier this year uh
talking about a south korean startup
which owned a
dating app and the the dating app was
you know used to share sort of intimate
romantic conversations between people
um they have this data so they train the
neural chatbot based on the dating app
information and then they released this
neural chat bot to the while
hopefully you all recognize that this is
a truly terrible terrible idea
and people interacting with this chatbot
actually found that they can they could
get the system
to like emit very intimate details about
people's lives back to them um and so
this is like a truly terrible story
right i was hoping it was the april
fool's joke but it's april 2nd and not
first and it's actually a real story
and there's going to be
more stories like this right as there's
a increasing pressure to build complex
interactions and complex systems like
chat bots people will want to leverage
these kinds of data
and so i think
there's going to be an increasing
privacy risk that we need to grapple
with
when we talk about language models i
think a pretty common thing that people
say is you know either users have
consented to the release of the data or
we're scraping public pieces or
semi-public pieces of information like
say reddit conversations
um so there's really no privacy harm i
think this is a common thing that i hear
a lot of machine learning people say
but i do want to stress and this is not
a purely machine learning topic but an
important one that privacy harms do not
just come from the release of truly
private data
there's a really nice survey by salov in
2006 called a taxonomy of privacy in
which he goes through all the different
ways in which you can harm somebody's
privacy
without really actually releasing their
private information actually he will
also talk about private information but
also other kinds of privacy harms that
you may not have thought about before
um
and i think the important component here
is that there's many different ways to
harm privacy and there's a whole
pipeline of privacy harms right you
could invade somebody's home and that's
obviously a privacy violation
you could collect information like watch
them with a camera that's a privacy
violation but actually the thing that's
most relevant to us as computer
scientists is the information processing
component right there are privacy harms
that arise from taking lots of different
data scattered all over the place
combining them into one centralized
source and that's what language models
and models do right they aggregate
pieces of data and they form inferences
and generalization and of course
disseminating that information can
obviously have privacy harms right
so i want to talk about sort of
information processing in particular um
before i talk about language models and
like their specific harms and so on
right because this is an important
higher level point that i want to stress
so when we aggregate information this is
the act of taking different kinds of
public information and combining it
together into some other new piece of
information and really you can't avoid
aggregation when you're training models
right because the whole point of a model
is to take information that was
scattered in documents across the
internet or whatever and to bring them
together into a centralized place this
model right
and the other thing is accessibility
right when we release a model we
inherently make it more accessible
originally right to get access to these
pieces of information you have to go
crawl through the internet find all
these different pieces of information
and find the relevant things right you
have to do a lot of research right but
now there's a single central repository
this big language model that you've
trained that contains all that
information in one place right it's like
kind of a google of private information
and you can go look everything up
so
you could think about a lot of different
harms that arise from these two
processes and i want to outline a couple
to motivate what will follow in the rest
of this talk
so the first thing is that aggregation
can can violate expected privacy so the
example i like to give for this is let's
say we have a system that can build a
synthetic biography right
so maybe some of you have seen these
like weird kind of very sketchy websites
that will find public information about
you like your address your gender where
you work and so on and they'll like make
a website that's like a kind of a
synthetic
biography of all the things you know and
they know a lot of different things it's
kind of shocking if you go to one of
these websites they'll maybe even be
able to do things like in for your
income right
now imagine a model doing this at scale
you might type in like tatsu hashimoto's
biography and they'll be able to like
spit out sort of all the inferences i
can make right you know my my uh
leanings politically and so on and so
forth that's definitely something that
feels like a privacy harm right because
i definitely didn't consent to having a
system build the synthetic biography
with my
some intimate details of my life in
there
other things that can happen is
inferences right um even accurate
inferences could be harmful and one way
i think to sort of talk about this is
like let's say i have a whole bunch of
writing published on the internet
through blog posts right and then i
asked gpt2 or some other language model
you know here's the writings of somebody
like what is their sexual orientation
right
building that system can itself be a
privacy harm right in the sense that
you're trying to infer information that
is private like technically speaking
that is a piece of information you could
have gone from the public text that
existed
but
this is definitely new pieces of
information that's much more accessible
to somebody
and finally accessibility can harm
expectations of privacy right if i have
a website that's not scraped by
googlebot it's technically speaking
public but i have an expectation of
privacy and so the boundaries of what
constitutes private information
is honestly a little bit pretty loose um
so for all of these reasons language
models represent a new sort of frontier
of privacy harms and like safety
violations for people um
any questions i feel like sometimes this
is controversial uh with people who who
especially want to stick to like
i guess
a question i had was so is this
how is this different from let's say me
like taking your example like googling
about tatsu hashimoto and saying okay
you know what based on this i think this
is their income this is their political
leanings and then creating a blog post
about it is it just a matter of scale
right or is there something else right
that makes this
worse than maybe just meet with india
yeah so i think the question was whether
um there's a difference between you
googling things and like a centralized
system that
that does this i think the key thing is
everything is sort of continuous in the
sense that in order for you to find
everything about me it will take
substantial amounts of effort right
substantial amounts of searching effort
and so on and so forth and decreasing
the barrier the cost barrier to
having all this information available to
you immediately
um that's a qualitative difference
because whether or not it's going to
take you a month versus a day will
really change whether or not you
actually act upon it right
and this is a kind of a good segue
because the courts actually have thought
about this and they come to essentially
the same conclusion
um as me which is uh so doj versus
reporters com for free presses is the
case that that is relevant for this and
sort of to give you the background i
believe this was a case in which uh some
reporters wanted access to
fbi rap sheets and rap sheets are the
centralized
piece of information for like all the
crimes that someone has committed and so
on and so forth aggregated from all over
the country
and the reporters argued that rap sheets
were public information because all the
little pieces that went into it like you
know whether or not you were convicted
in like a small town
that is public information like you
could go to the court records you could
look it up and you could find out right
so that was their argument and and the
doj argued that it was not public
because you would have to spend
substantial effort to look it up and
that was sort of the the same uh logic
that was uh reached by sort of the
justices on the bottom here the issue is
whether or not the compilation of hard
to obtain information um alters the
privacy interest and the nature of the
privacy guarantees and the argument is
that if you have a whole bunch of
information scattered across courts and
you have to go look them up yourself
that's kind of a different mode of
privacy expectation than if it's like
directly available to you and hand it to
you
um
and also i think the other thing about
accessibility that they also noted is
important right so you know in
conversations that we have and you know
in semi-private disclosures you know
most pieces of information are disclosed
to somebody at some time right so the
notion of like privacy is is a very
shifting quantity like at least like
things i put on the internet is a very
binary quantity but there's a lot of
pieces of information that i share that
i don't expect to be disseminated
there's an expectation of privacy
component to all of this so changing the
accessibility of of private or
non-private information is itself they
argue uh a privacy violation so i think
this is kind of important especially the
legal aspect i think is useful to think
about because that is in some sense the
kinds of things you'll have to follow
when you're making these systems if
you're ever involved in making one of
these systems
okay
so
i spent two slides talking about sort of
the legal and ethical aspects of this
and hopefully that is of interest to you
but now we'll return to questions of
more technical questions about machine
learning
for a few slides i want to talk about
harms that other people have discovered
this is not my work
extracting training data from language
models some nice work by carlini shows
that if you have a language model like
gpt2
trained on the internet you can very
easily extract parts of the training
data and so the example figure you
should have in mind is something like if
you put in as an input to this model you
know east strasbourg whatever then the
model will spit out some sort of address
that was originally contained in the
training data and this is exactly the
kind of like potential privacy harms
that you could imagine
and one thing that they found which i
found to be really interesting given the
trends in the field is that the larger
the model the stronger its tendency to
memorize
and that's actually kind of an
interesting observation so this table
taken from that paper what they did was
they found a bunch of reddit links
that was contained in the data set and
it was contained in a very specific
place it was like some paste bin piece
of text that was contained and so they
can exactly track the number of times
this very specific string appeared in
the training data and so as you go to
the top it's more and more common so it
appeared more times in the training data
and as you go to the bottom it didn't
appear very often
and on the sort of different columns the
memorized column on the right it's
showing which models happen to memorize
this piece of text and they have a
technical definition of memorize that i
won't get into but you can just kind of
think of it in your informal way
now what we find is that this excel
model on the left the biggest model
memorizes way more stuff than the others
and in fact like the more frequent the
the piece of text the more memorized it
is so
what this shows is like if a piece of
text appears maybe 30 times in the
training data set
the
gpt 2 xl one of the larger models at the
time will happily memorize that piece of
text whereas if you have a much smaller
model it won't memorize and this is an
important piece because the trend of the
field is to go to ever larger models to
improve performance so this problem is
only going to get worse
and there's arguments to be made by
fairly smart people with nice evidence
that memorization is a fundamental
property of neural networks
and in fact with our current training
paradigm it is unavoidable so shown in
this plot
is the performance of a language model
and on the x-axis is how long we train
for and there's two different y-axis the
blue y-axis is how much we've memorized
that's a technical term but you can sort
of think about it colloquially on the
right side is how well our model fits
both the training and the test data
and so what you see let's start with the
red lines right as you train for longer
training data goes down and test data
goes down to a point and it comes back
up right because it starts over fitting
now the blue line the amount we memorize
goes up as we train more because we see
data more and more times the key thing
to notice here is the point at which our
test and train losses are low is also
the point at which memorization has
peaked right
and so people look at these kinds of
plots and they say it may be the case
that our current paradigm of using large
neural networks and public data
means that we will inevitably get
memorization right this is a thing that
we will have to live with in some ways
and so this kind of sets the stage for
for the harms that that might appear um
and so to sort of wrap up this sort of
like summary of harm's kind of view
right so large language models have
really driven large-scale public data
collection because we know that more
data and bigger models are going to help
us they're going to improve the
performance of our systems
at the same time we know that they're
going to lead to further memorization
and that could be a form of privacy harm
right and so using the current
approaches that we have now it seems
like this is inevitable models seem to
prefer to memorize data and the highest
performance largest models that we have
will naturally memorize data
and so how can we get away from this
problem right like now we sort of need
to try to fix this problem and now i'm
going to talk about that
so
one thing that you might begin with and
think about that i want to sort of
prevent you from doing maybe is to think
about simple privatization schemes right
just look at some names in the training
data mask them out look find addresses
and mask them out um the problem is that
privacy is kind of a high-stakes thing
in some ways if you mess up and you
release a model it's kind of there
forever and even smart people will make
mistakes uh this example on the left is
some very smart uh researchers coming up
with a way to scramble and privatize
data called instahide um and it was you
know made public on on february 21st
last year uh or february last year and
then it was broken just two months later
by some security researchers that showed
that you could almost exactly recover
the original pieces of text based on
like training information and so even
really smart people can make mistakes
this is not to say that this insta hype
thing was a bad idea this is to say
privacy is hard when we don't have
provable guarantees
and so what we need is some stronger
guarantee that we're not going to leak
user data that we care about
in the gold standard method the thing
that has stood the test of time is
differential privacy some of you may
already know about this but i'm going to
give you the the kids version of
differential privacy differential
privacy is a fairly simple guarantee it
says i have a bunch of users which
people call records in the differential
privacy literature so i have alice bob
xavier donna ernie and my guarantee is
the following if i train a model using
everybody
i will get this blue distribution over a
model so my predictor has to be
randomized i have a randomized estimator
and my estimator emits one distribution
and if i remove xavier i'm going to have
a very similar distribution right like
my output of my randomized algorithm
will remain close whether or not i have
included xavier that is the guarantee of
differential privacy and there's a more
formal definition but i don't want to
get into that too much
this is really the gold standard in the
sense that there hasn't really been
strong attacks that were demonstrated
against this and it was used in the 2020
census with some controversy but it's
really something that's hard to achieve
i'm going to skip this slide because i
don't want to get into the technical
bits
but really the issue with differential
privacy is it's such a strong guarantee
it says that when i remove a user no
adversary with arbitrary amounts of side
information can determine whether or not
i was included in the data set that is
incredibly strong and prior attempts to
applying differential privacy to natural
language processing was kind of a
complete failure i just want to give you
a funny example here so this is an
example from i think a two years ago now
kerrigan at all that tried to generate a
language generation system using reddit
data privately um and what they had is
you know this is an example bob lives
close to the and if you have a
non-private system you'll get a pretty
reasonable output here you know lives
close to the station and so on and so
forth if you use a private system you
get complete garbage this is not even
reasonable english if you look at the
continuation over here
and so it seemed for a long time that
differential privacy and deep learning
were just fundamentally incompatible
concepts like you know you can't even
put the two together and people have
very reasonable reasons for believing
this you know large language models have
millions of parameters like one typical
example is 300 million parameters so if
you think about each parameter as
potentially containing a private piece
of information that is a ton of
information to privatize right so it's
really hard to privatize large language
models because they have so many
parameters each of which could
potentially leak information
now theory also agrees with this it says
there are very good bounds that say that
differential privacy performance should
degrade as the square root of the number
of parameters so now theory matches our
intuition and it also matches empirical
results the only successful dp
applications used to be in low
dimensional statistics like things like
the mean or the number of people living
in a household
but that's really not the complete
picture and this is maybe the optimistic
part of the talk
um
we can actually change our perspective a
little bit we've been thinking about
language models as the cause of privacy
harms but it can in some ways also be
empowering us to have better privacy and
this is a slightly different perspective
so we can't train large language models
yet using differential privacy but what
we can do is the following idea we can
look for nicely curated public data
let's set aside the problem of public
privacy harms for a moment train a large
language model and then use this
language model along with private data
to achieve privacy so now we can have
privacy on a small subset of data that
we truly care about while leveraging
large public data sets
and this is kind of the paradigm that i
went back to at the very beginning of
the talk i was saying language models
have driven enormous gains in nlp
through through a range of subtasks and
the reason is because these models learn
really useful things like syntax of the
english language right we don't want to
be using our precious private data to
relearn the english syntax that is crazy
right so we want to learn this stuff
from hopefully safe sanitized public
data from somewhere and then use that to
empower our sort of downstream private
applications that's sort of the idea
here and so we it's wasteful to spend
our private data learning this kind of
public information and that's the main
idea
so what we found was actually kind of
funny so once we started thinking about
it this way
we realized that maybe language models
actually are useful
so what explains previous failures in
using uh in using these kinds of systems
sorry in using differential privacy in
language generation systems it turns out
it was something very silly it was hyper
parameters
if you use typical hyper parameters that
you use for non-private training you get
catastrophically bad performance this is
uh performance on the e2e benchmark that
i showed you before and around 60 is
state-of-the-art 70 is state-of-the-art
and 10 is totally unusable it's garbage
so if you use a typical hyper parameter
for non-private training you get
basically garbage but this was actually
about two orders of magnitude off if you
change the hyper parameters
substantially to be more suitable for
differential privacy you actually get
close to non-private performance this
was really surprising to us and we had
to come up with a different kind of
explanation for why you would use these
weird hyper parameters but the naive
choices were way off
and equipped with this we found a very
different story what we found was that
you could actually get the private
systems that were almost as good as
their non-private counterparts so on the
left side is a classification task so
you're predicting uh entailment which is
the task that i told you about on the
right-hand side is the e2e task the the
restaurant review generation on the
right
and we see that as we increase the
number of parameters from left to right
the performance gets better and better
the y-axis here is performance and each
of these lines is a private model that
we train
and so larger models by virtue of their
pre-trained language models being better
actually get much better private models
in the end
so this runs counter to the maybe belief
that people had that large models were
hard to privatize because they might be
leaking
private pieces of information we
actually found that large language
models actually give us the opportunity
to get even better performance under a
stringent privacy guarantee
um
so in the non-private case the
sorry the pre-training is sort of a
small game but for private learning the
difference is huge right so having large
models gives us enormous gains going
from unusable uh performance to go
getting to completely usable and nearly
non-private performance
um what is the time we have five minutes
left okay so i will talk about one last
technical piece and then i will conclude
um
one really interesting side note
especially if you're sort of more
systems or computational people here is
the real technical challenge that we
found was not necessarily statistical
usually people think about private
learning in terms of statistics how
large does your data set need to be in
order to be able to learn privately what
we found was a really big challenge was
computational
differentially private learning requires
a lot more memory than
non-private learning and what we found
is is sort of this following very uh
challenging thing
so
for various technical reasons
differentially private sgd is very
memory intensive and so for looking at a
non-private uh training task we can fit
34 examples in a batch for a medium
model or 10 examples for a large model
if we're doing private training we can't
even fit a single example into a
titan x our titan rtx gpu when we're
doing private training and this turned
out to turn into the bottleneck because
we were no longer having
these statistical challenges because we
were using pre-trained models but we
have these computational challenges
and by using various tricks involved in
like modifying the back propagation
algorithm what you could actually get is
uh memory usage that was on parts of the
non-private algorithm and that's kind of
a cool little trick that we developed
but because of that we can now
really scale up oh sorry scale up the
size of these models and finally get
private models that were almost close in
performance their non-private
counterparts
um
so before i sort of conclude um in the
last couple of minutes i want to just
show you examples of what a private
system generates because i think in
generation you know i can show you
numbers like blue scores and what on but
looking at the outputs actually i think
is more compelling uh so for the top one
this is the e2e generation task this um
you have the table at the top that
describes the restaurant we have the
reference that was written by humans and
below that this is a private version of
a neural language model that we train so
this thing has a very formal and strong
privacy guarantee of epsilon of three
um and yet it generates perfectly fluent
and sort of grammatical and correct uh
restaurant reviews
and so that's sort of the point that
we're at now by leveraging large
pre-trained language models combining
that with differential privacy we can
actually get privacy for a lot of these
interesting downstream tasks that we
care about we can get this very strong
notion of privacy called differential
privacy
and so that's where we're kind of at
right
so to wrap up right this was in two
parts so the first thing that i do want
to for you to remember
is that language models do pose a
privacy risk it scrapes the internet it
collects all this data it aggregates
them that is a privacy risk right let's
not feed around the bush here and large
language models will memorize training
data so these these threats are real but
at the same time there's opportunities
right i was talking about risks here but
the the flip side of this is that large
language models by virtue of you know
the fact that they can teach you about
english grammar and so on allows you new
kinds of privacy guarantees much
stronger than what was available before
so even though there's some privacy
risks that we get we also get some other
kinds of privacy in return which is sort
of an exciting frontier
so i'll stop here we'll have a couple of
minutes at the end for questions if
anyone hasn't
[Applause]
sure i guess why is it that the
memorization problem can't be
trivialized with pre-processing
why can't the memorization be
trivialized with pre-processing like
trivially avoided i guess what do you
mean by that you can make the model not
memorize data by just not like by just
please stop pre-processing the data
somehow like for example you could
imagine just running like a simple
arithmetic encoder on it so it's lossy
so then by pigeonhole principle you have
something guaranteed
so let me think about that so you can
certainly noise up your data and that
will prevent memorization but you will
pay a very large price in the accuracy
in return because your goal is not to
predict noisy english it is to
predict english right
um there are certainly i mean in in some
ways okay so there is a truth to sort of
what you were you were thinking about
but it doesn't operate at the
pre-processing level to be very formal
uh about this like
differential privacy is kind of this
idea it says instead of taking a normal
gradient step taking out the very top
what you do is you take your gradients
you clip them so each gradient is like
you know bounded in size and then you
add a bunch of noise and that noise step
is kind of like this intuition that you
have
but if you just do it in the right way
with the right amounts of noise and the
right amounts of clipping then you can
get guaranteed so it's a lot more subtle
than just like let's just add some noise
to the training data
second solution yes you share oh like
let's not worry about the public data
privacy issues but let's focus on like
the specific private data we have do you
think it is worthy to think about the
public
data
privacy issues and whether there's any
remedy to having a
large corpus with
private data
yeah or are you
so so okay i'll fully acknowledge right
the first part of this section was like
okay it's really bad to pretend that
private the public data is public right
um i don't think we have a great
solution for that yet
but people are working on differentially
private pre-training
and if that gets solved
then this whole thing will be private
and we would sort of be able to avoid
all these issues
um
i guess like you know once you solve
those problems there are other
additional problems about what does it
mean to do differential privacy on
internet text what what is our
definition of privacy um those are
subtle questions but i think we're
slowly making progress
yes just to make sure i'm on the right
page here um you took a pre-trained
model
um and then you
basically did hyper parameter tuning on
the private data set is that correct you
fine tune on the privacy right
so the the hyper parameter story is not
that we're doing hyper parameter tuning
on the private set but we needed to just
change the hyper parameters when we fine
tune on the private set
so it's the well i guess we can go all
the way back to the very first part of
the talk but it is basically
this paradigm you have a pre-trained
language model you have a downstream
task that you would like to do so in
stage two you take your pre-trained
model and you fine-tune it for the
downstream task it's exactly this
[Applause]
you


What’s been happening in AI alignment? | Rohin Shah | EA Global: Virtual 2020

hi everyone my name is Rohan Shah I'm a
sixth year PhD student at the Center for
human compatible AI at UC Berkeley
my research is generally on what happens
when you try to do deep reinforcement
learning and environments that involve
humans somehow and more broadly I work
on technical AI safety I also write the
alignment newsletter and today I'll be
talking to you about what's been
happening in the eye alignment I should
warn you while this talk doesn't assume
any technical knowledge of AI it does
assume basic familiarity with the
arguments for air risk I'll be servin
surveying a broad swath of work rather
than focusing on the things I'm
personally interested in I'm hoping that
this will let you figure out what parts
of AI alignment you personally feel most
excited by and want to delve into deeper
later a lot of the talk is based on a
literature review I wrote a couple of
months ago and so you can find
references and details in this link in
the top right corner it's just gonna
stay there for the rest of the talk yeah
and so with that let's get started so at
a very high level outside view the
reason that like most people work on a
safety is that you know powerful AI
systems are going to be a big deal
they're going to radically transform the
world that we live in and so we should
probably be putting in some effort into
making sure that the sort of
transformative effect goes well in
particular if AI systems are smarter
than this then they could become the
dominant force on the planet which would
be well could be bad for us in the same
way that like guerrillas are probably
not stoked about how we have like taken
over all of their habitats and so this
doesn't necessarily mean that there will
be X risk it just says that we should
have a good technical reason to expect
that the powerful AI systems we pulled
are actually beneficial for us and I
would argue that we currently do not
have such a reason and so like the case
for working on alignment is that we
really should be creating this reason
and I want to note that like well
there's a lot of disagreement
many specific sub questions in the a
safety and that will become a bit more
evident over the rest of this talk at
least on this basic high-level argument
my impression is that basically
everybody in the field agrees with it
agrees with this high-level outside view
argument okay so what are the specific
risks were worried about what the I one
issue you might be worried about is that
humans aren't really ready to deal with
the impacts of AI so humans tend to
fight a lot right now there's like some
amount of conflict people keep talking
about like how the us-china relationship
is a big deal and AI is just going to
let us have better and better ways of
fighting seems pretty bad like maybe we
just have our fights get bigger have
bigger and bigger impacts on us and
maybe at some point this actually leads
to extinction-level events or perhaps AI
or perhaps a leads to technical progress
technological progress at to faster rate
for us to get accustomed to and as a
result maybe relocked in some suboptimal
values and that's the way that we could
get past dependence so in both of these
stories that the system isn't
intentionally causing X risk but you
know nonetheless X risk does perhaps
happen I'm not going to focus too much
on them I'll just note that like some
ideas that people are talking about our
preference aggregation so this with this
idea you a to have the AI system
aggregate the preferences over well all
of the stakeholders and then everyone
would agree to just like let the AI
system do its thing and that wouldn't
wouldn't bite with the results of
whatever the AI system does and
similarly we could try and figure out
better Metta philosophy so that we don't
have problems like value blockin okay
another outside view that people use
besides just like man powerful AI as a
big deal is that optimization in
particular leads to extreme outcomes
like to take
like very simple example on average at
least in the u.s. men are about 5 feet
10 inches tall but if you look at
basketball players were selected for
height you're going to find that not
very many of them are 5 foot 10 and most
are in fact well over 6 feet the point
here is that when you select for
something when you have optimization
pressure you tend to get extreme
outcomes and powerful AI systems are
going to be very powerful optimizers and
as a result we probably shouldn't expect
our everyday reasoning to properly
account for what these optimizers are
going to do and so we need to take more
of a security mindset where we look for
arguments i quantify over every
possibility as opposed to the average
possibility so this sort of mindset
inspires a researcher is especially at
Mary to try to understand how
intelligence really works so that we
could then make well-designed AI systems
that we understand and so this is led to
research on embedded agency partial
agency and abstraction a little bit
about embedded agency this is like one
of Mary's main research programs and the
basic idea is that in the sort of
standard model of reinforcement learning
and AI more generally there is an
environment which takes in actions and
produces observations and rewards and
then completely separately from the
environment there is the agent that like
sees these observations and takes
actions as a result and those are not
actually how agents actually work I am
presumably an agent but I am NOT
separated from the environment of the
world I like I'm a part of it and this
leads to many many problems in well many
philosophical problems I would love to
go into more detail but don't have too
much time there is a great sequence on
the alignment forum about this I
strongly recommended cool
the next problem I want to talk about is
probably the most familiar one to people
in the side
I call it the specification problem it's
also called outer alignment basically
with this problem if this problem is the
fat is that the way we build AI systems
right now is to assume that we have some
sort of specification of the optimal
behavior in all possible situations that
is infallible as though it were like
handed down to us from God and then
given this specific specification we
have to figure out how to meet it and of
course we can never actually get the
specification like the classic paper
clip Maximizer example shows that it's
pretty hard to specify the behavior of
like make paperclips in a reasonable and
sane way
like turns out that that's quite hard to
actually specify this is also the main
problem that steward Russell's new book
human compatible talks about in terms of
like who's working on the problem like
chai opening I deepmind odd that they're
all doing work on this problem not
necessarily just this problem but
certainly some part of their work is on
solving the specification problem the
main proposed way of solving the
specification problem is to do some form
of dolly learning and one thing I want
to note value over here doesn't
necessarily mean normative value you
don't necessarily need to be thinking
about population ethics here it totally
would count as value learning if you had
a robot that like learned how to clean
your room and then reliably clean your
room like that totally is value learning
maybe we should be telling it
specification learning the value
learning seems to be the names of stuff
so types of value learning
Searle or assistants games the Searle
stands for cooperative inverse
reinforcement learning this is a
particular formalization of how you
could do value learning in which the
world contains a single human who knows
the like reward function the true
specification but for some reason can't
communicate explicitly to the robot to
the agent and then there is also an
agent who is cool
to infer what the humans specification
is and then optimize for it and because
now you don't have a definite because
the agent no longer has a definite
specification that it's trying to
optimize and it's instead uncertain over
what it's trying to optimize this gets
you a lot of like nice properties so for
example the agent will ask you about
what you want it will try to clarify
what your preferences are if you try to
shut it down it will reason that it must
have been doing a poor job of helping
you and so it's going to allow you to
shut it down unlike a classic expected
utility Maximizer which will say nope
I'm not going to shutdown because if I
am shut down then it's like that I can't
achieve my goal
so that's Cyril or assistants games the
unfortunate thing about assistance games
is that they are very very very
computationally intractable it's very
expensive to solve a Cyril game in
addition it requires you to know to have
a good model of how human preferences
relate to human behavior which as many
social site many of the Social Sciences
will tell you is a very very difficult
problem and there is a theorem that says
it is provably impossible and the like
super general case though of course we
don't actually need super general case
we only need the case that actually
applies in the real world and that
instead of being provably impossible is
merely very very difficult yeah so after
Cyril we have learning human intent this
is basically a broad category of
possible ways possible communication
protocols that humans could use to
communicate the specification to the
agent so perhaps a human could
demonstrate the like optimal behavior to
the agent and then the agent could learn
from that what it what it's supposed to
do so this is the idea behind inverse
reinforcement learning and imitation
learning alternatively perhaps the he
could evaluate proposed hypothetical
behaviors that the agent could execute
and then the agent could say could
reason over basically what the human
said was good in order to figure out
what it should be doing so after that
let's come to intend alignment or
courage ability these are somewhat
different while the previous approaches
are like trying to specify an algorithm
that learns values with intent alignment
we're instead trying to build an agent
that tries to do what we wanted to do so
to put it another way we're trying to
bake into the agent a motivation to be
helpful to us and so if we then have
this agent that is like all I want to do
is to be helpful to Rogen that's going
to naturally motivate it to do all of
these other things that we wanted to do
so for example it's going to try to
clarify what my preferences are in the
same way that like a good personal
assistant would try to figure out what
my preferences overflights are so that
he or she didn't have to bother me when
I asked them to like book me a flight
yeah cool so that's a sort of broad
spectrum of approaches to value learning
however there are still a few problems
that are rise so intuitively one big
problem is that since the agent is
learning from our feedback it's not
going to be able to do better than we
can do it's not going to be able to
scale to superhuman performance so if
we're demonstrating the task to the
agent it's not going to be able to
perform the task any better than us
because nothing is giving it the
information of like how to do better
than us
similarly if we're evaluating the agents
behavior it won't be able to find good
behaviors that we wouldn't recognize as
good it's like an example of like where
we might have cared about this is
alphago's move 37 this was a pretty
famous move that alphago made which
seemed really really crazy to humans
known
would have like ever dentist moons I
think it was assigned to like less than
one in 10,000 chance
and yet that move ended up being crucial
to alphago's success and why could
alphago do this because alphago wasn't
group relying on our ability to tell
whether or not a particular move was
good alphago was just relying on the
fact that there was a reward function
that told it when it had one and when it
had lost and that was a perfect
specification of like what counts as
winning or losing and go so ideally we
would like to build super intelligent AI
systems that can actually exceed human
performance at tasks but it's not clear
how we do this with value learning the
key idea that lets us that lets current
approaches get around this is that sure
our AI systems are never going to exceed
like super vision that we give them but
maybe we can train our AI systems to
approximate what we would do if we had a
very very very long time to think so you
could imagine as a hypothetical if I got
a thousand years to think about whether
what the best thing to do was and then I
like totally AI system hey this is what
the best thing to do is in this scenario
and the AI system like properly
approximated this then that would be but
like it could do it in a couple of
minutes as opposed to a thousand years
that would be presumably very very super
intelligent so the details for how we
take this insight and get to an
algorithm that you know we can actually
train and not a thousand years the
details are a bit involved and I'm not
going to go into them but the reason the
like techniques to look out for our
iterative amplification debate and
recursive reward modeling cool so that
was one problem with value learning
another problem with value learning is
the informed oversight problem this
problem is basically that even if we are
providing
supervision to the agent and even if
we're smarter than the age of that word
training let's just take that as a given
for now if we don't understand why the
agent chosen action we won't be able to
effectively supervise it so the classic
example for this problem is consider an
agent that's tasked to write a new novel
perhaps it's got access to a library
where it's supposed to learn about how
to write how to write books and it can
use this in order to write a new novel
but like the new novel is supposed to be
actually new not just memorizing some
novel from the library and spitting it
back out again but like it's possible
that the agent just like looks at five
books in the library plagiarizing some
bunch from all of them and puts them
together into a book that like a reads
very nicely to us but doesn't really
solve the task because it was
plagiarized how are we supposed to know
how are we supposed to tell the agent
that this was bad if we can to actually
see that the agent was looking at these
fine books and like stealing sentences
from them then we in order to catch this
we'd have to read the entire library all
that like thousands of books to like
search for any evidence of the agent
plagiarizing and this seems like just
way more expensive than this seems too
expensive for oversight yeah and right
so the overall problem is that we have
it may be way more costly for us to give
it oversight than it is for the agent to
take actions if we cannot see how the
agent is taking those actions so the key
idea to solve this is I mean it's almost
obvious it's just make sure you know
what how the agent is taking their
actions and there's a again a bunch of
details on how exactly we think about
this but the term to look for is
description universality basically this
property means that the supervisor knows
everything about the
agent Noh's including any facts about
how the agent like chose its output and
so then if we were ascription Universal
with respect to the agent then we would
know that then we would know that it had
like taken these sentences from these
five books because the agent knows that
and if we knew that then we could
appropriately people eyes that don't
tell it not to plagiarize in the future
how do we get this property sadly I'm
not going to tell you because again
limited time but there is a great set of
blog posts and summary in the alignment
newsletter and all of this is linked to
from once again that link in the top
right corner really I just want you to
read that link it's like a great I put
in a lot of work I think it's good
cool all right let's move on to another
top level problem so this will be the
problem of Mesa optimization so I'm
going to illustrate Mesa opposition with
Ananya example so let's suppose you're
searching over a space of programs like
programs and just some programming
language let's say Python and we're
looking for a program that plays
tic-tac-toe well and so you're searching
through these programs and initially
you'd like find some programs that have
good heuristics like maybe you find a
program that like always starts at the
center square and that one tends to like
win a little more often than the other
ones and then later maybe you find a
program that like make sure that any
time it's got two in a row and the third
spot is empty it like actually plays in
that third spot so that it ensures that
it wins if it can in one step and that
one starts to win a bit more but then
eventually at some point you come across
the minimax algorithm and the minimax
algorithm plays optimally by searching
for the best action to take in every
situation and so what happened here was
that your outer optimization your search
over the space of programs ended up
finding a program that was itself an
optimizer that searched over
possible moves in tic-tac-toe and so
this is the idea of Mesa optimization
you have some sort of base optimizer in
this case to search over programs and in
the course of running that base
optimizer then finds a new optimizer
which in this case is the minimax
algorithm cool so why is this relevant
to AI this is just some weird thing
about programs well in the AI case often
we think about as systems that are
trained using gradient descent and
gradient descent is an optimization
algorithm that searches over the space
of neural net parameters to find some
set of parameters that performs well on
some loss function let's say that
gradient descent is the outer optimizer
in our Mesa Mesa optimizers story it
seems pretty plausible that like this
Mesa optimization thing could happen
even with gradient descent where grading
descent finds an instantiation of the
neural net parameters such that then the
neural net itself when it runs is
performing some sort of optimization
then the neural net would be a Mesa
optimizer that is optimizing some sort
of objective which we would call the
Mesa objective and we well we know that
the Mesa objective should lead to
similar behavior as they like original
objective on the training distribution
because that's what it was selected to
do it may be arbitrarily different off
of the training distribution it's like
if you trained it on tic-tac-toe then
like you know it's going to win a
tic-tac-toe but then maybe if you like
switch to connect for it might do
something crazy maybe in connect for it
still only looks for three in the row
instead of four in the row and so it
like loses pretty badly at Connect four
even though it was working well with
tic-tac-toe so if this happened with
gradient descent and we got it like very
powerful intelligent agent Intel
neural-net that was up even if we had
like solved the specification problem
and like happy like ideal reward
function to train this agent it might be
that the neural net model that we come
up with this prayer is optimizing for a
different objective which may once again
be misaligned with what with what we
actually want and this sort of outer
inner distinction is why the
specification problem is called outer
alignment and waimea optimization is
called inner alignment cool so what are
the things that people do to solve maze
optimization well there to me there's
one main proposal and one kind of sort
of proposal the main proposal is
adversarial training adversarial
training with adversarial training the
basic idea is that rather than training
a single AI system that's trying to
perform well on your specification you
instead have both you continue to train
that system as usual but you also have
an adversary and AI system or AI human
team that's like trying to find
situations in which the agent here
training would perform badly or would be
optimizing for something that is
different from the specification problem
in the case where you're trying to get a
corrigible AI system maybe you're like
your adversary is looking for situations
in which the AI system in which the AI
system like manipulates your deceives
you into thinking something is true when
it is actually false and then if you can
find all such situations and penalize
the agent firms for them then the agent
will like stop behaving badly on those
situations and if you are actually able
to find all of these situations then you
will have an agent that is that robustly
does the right thing across all settings
verification would take a trained agent
and then verifies some or the other
property that you care about with that
agent now ideally we would like to say I
have formally verified that the agent is
going to reliably pursue the
specification that that I outlined
whether this is actually possible or not
whether people are like actually
optimistic or not and like not totally
clear on but it is a plausible approach
that one could take they're also like
other other areas of research that are
related that like not obviously
solutions
so in particular robustness to
distributional shift is pretty important
because the way that you get risk in
witness optimization is by
distributional shift because on your
training distribution your agent is
going to perform well it's only when the
world changes that things could
plausibly go badly okay um a sort of
notable notable thing that's missing
from this is interpretability I haven't
really talked about it yet so
interpretability is a field of research
which is trying to make sure that we can
understand the a systems that we train
the reason it's I haven't included it
yet is because it's sort of useful for
everything it's like for example you
could use interpretability to help your
adversaries find good to help your
adversary figure out in what situations
your agent is going to do bad things and
this this would help adversarial
training work better but you know it's
also useful for interpret ability to do
value learning so that you can provide
better feedback on the agent if you like
better understand what the agent is
doing you can better correct it and it's
like especially relevant to informed
oversight or ascription universality so
it's sort of this like not obviously a
solution in and of itself but makes
other solutions way better yeah so these
are all of the techniques that are
trying to like align AI systems there's
also the option of just trying to
prevent catastrophes like May
the system will be useful maybe it won't
be someone else is gonna deal with that
what we're going to do is just stop it
from like killing everybody that's the
main thing that we want to do and so
approaches and approaches here include
impact regularization where the AI
system is penalized for having large
impacts on the world
so some techniques here are relative
reach ability and attainable utility
preservation and the hope here would be
that you could create a powerful AI
systems that can do you know somewhat
impactful things like maybe writing you
providing advice on writing new laws but
like we wouldn't be able to do extremely
impactful things like engineer a
pandemic that kills everybody and so
even if the AI system were like
motivated to harm us the impact penalty
would prevent it from doing something
truly ridiculous truly catastrophic
other things that people think about
Oracle's
the idea here is to restrict the AI
systems action space so that all it does
is like answer questions this doesn't
immediately provide you safety but
hopefully it like makes it a lot makes
it a lot harder for any system to cause
a catastrophe or you could try to box
the AI system so that it cannot have
much of an impact on the world and one
example of a recent work on this is Bo
my or box myopic artificial intelligence
and the idea there is to like cut both
the human and the AI system in the box
so that they have no communication with
the outside world while da system is
operating and then the AI system shuts
down the human leads the box and was
like able to use any information that
the AI system gave them cool so that's
most of what I have in this like problem
solution format there's also a bunch of
other work on a safety and alignment
that's trying that's like not so easily
categorize
problems and solutions so for example
there's work on safe exploration
adversarial examples and certain TVs all
seem like pretty relevant to AI
alignment but not sort of obvious where
exactly in this graph they fit at least
to me so I haven't put them in and
there's also a lot of work on
forecasting which is extremely relevant
to what sorts of research agendas you
want to pursue so for example there has
been a lot of disagreement on whether or
not there will be discontinuities in AI
progress whether there will at some
point in the future be a time at which
AI capabilities shoot up in a way that
you couldn't have predicted by
extrapolating past progress another
common disagreement is whether whether
advanced AI systems will look like
comprehensive AI services which
basically very very short description
there are just a lot of services each
task that you might want an AI system to
do is perform by one service you don't
have like a single agent that's like
doing all of the tasks or you could on
the other hand imagine a single
monolithic agent AI agent that like is
able to do all tasks so which of these
two worlds are we likely to live in
that's another disagreement and then a
like third disagreement is whether it is
possible to get too powerful AI systems
by just like increasing the amounts of
compute that we use with current methods
or do we actually need some like deep
insights in order to figure out in order
to get too powerful AI systems yeah and
as I said this is all very relevant to
like deciding what sort of research you
want to do many many research agendas
only make sense under some under like
some possible worlds and if you can find
out that you're not in that possible
yeah that that world is like not very
likely then maybe you switch to a
different research agenda
and yeah with that I that concludes my
 again there's the link in the top
right corner tinyurl.com slash alignment
2019 that's a link to the literature
review that I wrote there's both a short
version and a long version I really
encourage you to read it it goes into
more detail than I could in this
presentation but yeah thank you so much