Roger Grosse | How can deep learning research inform long-term AI safety?

hi everyone
so great to see people again if i'm out
in scotland or maybe not in scotland
benson good to see you great to see
everyone
it's uh it's really great to have
everybody here today uh this is our
first seminar of the
uh of the new year and lots of people
from our uh previous year since the
launch the launch of the seminar series
last fall
and we're excited about a new year for
those of you who haven't met me before
jillian hadfield director of the short
treatment institute
uh for technology and society
and this is our weekly event to connect
people across disciplines and fields
um
around issues with technology and
society so we're we really aim for a
really rich interdisciplinary
conversation
and um and lots of conversation
um
so we're going to uh normally we are our
goal is to have 45 minutes of
presentation and 45 minutes of
discussion and roger's asked i think
he'll do part one and then maybe do 15
minutes of conversation the discussion
and then he'll do the last part and do
the last 30 minutes
at the end
um and uh you're welcome to
uh i'll
ask questions in the chat i'll keep
track of that
um and roger do you mind being
interrupted along the way you want to
wait until the question um yeah that's
fine
okay so
i'll we'll keep trying i'll keep track
of that and um
and uh you can
also use the if you want to raise your
hand that's good too all right um okay i
also want to introduce uh somebody new
to our community it's our srsu postdoc
beatrice magistro who will be co-hosting
some of these
sessions uh beatrice is research
explorator technology and automation are
changing governance and politics and how
citizens respond to automation-related
economic destruction so welcome to
our group beatrice very glad to have you
with us
and now um my pleasure to introduce our
first speaker of the new term roger
gross um
roger's an assistant professor in the
department of peace computer science
here at ut and a founding member of the
vector institute uh roger was a postdoc
at toronto as well after having received
his phd at mit studying under bill
freeman
bill freeman and josh tenenbaum
and before that he did his undergraduate
degree in symbolic systems and masters
in computer science at stanford
university but we did not overlap at
that point i met roger very early in my
odyssey of learning
and ai safety
and he was also a founding member of the
ai safety discussion group i think at
vector that we now collaborate on
roger's research interests focus on
machine learning especially deep
learning invasion modeling and he works
on developing algorithms that train
faster generalize better give calibrated
uncertainty and uncover the structure
underlying a problem all really
important things furthermore he's also
investigating the important problem of
ensuring that ai systems remain aligned
with human values and of course that's
a core goal of our work here at sri
rogers going to talk today on how can
deep learning
research inform long-term ai safety so
we're starting the semester off with the
big picture questions i'm really looking
forward to that um okay roger let's kick
it off thanks
thank you julian
so i'm roger gross and i'll be talking
about how deep learning research can
inform long-term ai safety
so by long-term ai safety i'm referring
to the field that asks the question
if we were to build a very powerful ai
system
how can we
ensure that it continues to act in our
interests
and so in this talk what i'll try to do
is
outline
how we think about ai safety and how
deep learning research can fill in some
of our conceptual gaps
this won't be an exhaustive talk this is
not like an exhaustive list of all the
ways i think deep learning can help just
a couple
areas where i think it's particularly
useful
and so i'll begin by giving a very quick
overview of some of the background
assumptions behind what i'll talk about
and then i'll cover
two areas where i think deep learning
has something to offer the first is
forecasting ai timelines
and the second is mesa optimization
where an algorithm might itself build
another optimization algorithm with a
different objective
so this talk isn't original research
it's almost entirely summarizing work
done by
other people
um maybe a couple tidbits from my own
research
and i'll mostly be raising questions
rather than giving answers so don't be
disappointed if you don't walk away with
answers
so
i'll begin by giving a brief overview of
why we should be concerned about human
level ai in super intelligence
and i'll mostly be summarizing arguments
from this book super intelligence paths
dangerous strategies by nick bostrom
because i think this book did such a
good job of outlook outlining the
strategic space and what different
assumptions you might have about ai
safety and what would follow from them
so for the next few slides i'll mostly
be summarizing his claims
so an artificial general intelligence or
agi refers to uh
still hypothetical artificial agent is
capable of doing pretty much any
intellectual task that the human can
but the worry is that it wouldn't
necessarily stop at human level ai
because it appears that we're
both human brains and current computers
are very very far away from the physical
limits of computation and so it appears
that it could be physically possible to
build machines that are not just human
level but actually super intelligent in
the sense of being
vastly more intelligent than humans in
pretty much all relevant domains
and this could mean machines that think
like humans except maybe thousands of
times faster
or it could mean machines that think at
a qualitatively higher level the way
that we think at a qualitatively higher
level than chimpanzees
and so
the argument is that if we were to build
such super intelligent machines
then
we'd no longer be the most intelligent
agents on earth and
we'd no longer necessarily be in control
and so the reason to be especially
concerned about superintelligence is
that the reason to think it might happen
quickly
so in particular recursive
self-improvement refers to the process
where
ais are responsible for most of their
own development
so that's where most of the progress
comes from rather than from human
researchers
and under some assumptions this could
lead to
extremely fast rates of improvement in
ai capabilities
also known as an intelligence explosion
and
the reasons for this aren't terribly
complicated and so an oversimplified
model for thinking about it is let's say
that
ai and human capabilities could be
measured as a scalar essentially
depending on the speed of thought
and imagine that each
increment of human time results in a
fixed
exponential increase in ai capability
so i think essentially it war's law
humans are the same as they were 20
years ago and computers
get faster or competition gets cheaper
at a very
steady exponential rate
and so if
the human contribution were the entire
story then the curve of technological
progress would look something like the
blue line in this figure
um
so
the y-axis is a measured ai capabilities
on a log scale
and
essentially we have a moore's law-like
behavior where progress continues
exponentially until it reaches the
physical limits and things level off
um but
um once we cross the line
right so if this were to happen there'd
be some point which
ais are essentially human level and then
um two years later they think twice as
fast as humans and then two years after
that they think four times as fast as
humans and so on
right and so that one that's what
happens if only the human contribution
counts
but
if ais are contributing to their own
development
then once you once they're equally smart
then
moore's law will result in a doubling
every one year instead of every two
years and then after that it'll double
in half a year instead of after one year
and so on and so the rate of doubling
will just keep speeding up speeding up
until basically
um we have what looks like a phase
transition to a much much higher level
of ai capability
that's what's shown in the orange curve
and that's what is often referred to as
a technological singularity so just
imagine decades or centuries worth of
progress compressed into a very short
period of time like a few weeks
so um why should we be worried about
this well there are
a couple core theses that thickbuster
makes
um the first one is to respond to this
common misconception that as ais become
more intelligent eventually they'll
converge to more worthwhile goals
but this isn't necessarily true
and so he proposes this orthogonality
thesis
which says basically that any level of
intelligence is compatible with any goal
so you could have a very very smart ai
whose
fundamental goal is to produce as many
paper clips as possible and no matter
how smart it gets it's not going to
unlearn that goal
this is associated with the idea of
utility maximizing agents
it doesn't necessarily require that but
that's sort of the easiest case to think
about
and so
those of you who do reinforcement
learning um rl is basically a way of
approximating sequential decision making
to maximize a cumulative reward over
time so be a pretty clear instance of
this
and there's a mathematical idealization
of
a super intelligent
ai with arbitrary goals which is
ai xi so something we can actually
write down and reason about its behavior
even though it's uncomputable to
actually implement
so the
other
main component
of why we should be worried is known as
the instrumental convergence thesis
so even though ais can be designed with
almost arbitrary goals
um there's certain
um convergent sub goals that they're
likely to settle on
so regardless of what their terminal
goals are it's likely that they'll have
an interest in self-preservation
now this isn't as obvious as it sounds
we think of self-reservation as being
sort of um
built-in drive that we have but the ai
wouldn't necessarily care about survival
for its own sake um but only is a means
towards accomplishing its other goals so
the ways to
wrestle puts it is you can't fetch the
coffee if you're dead
um the second goal that would have is um
contact integrity right so if we
mess up and give it the wrong goals then
it'll have an interest in preventing us
from changing them
it'll have an interest in acquiring lots
of resources that it could use to do
things
so for instance if we built an ai with
the goal of computing the digits of pi
then it might be useful for it to turn
the entire world into one big data
center so it can compute more digits
so
a common misconception
that
ai researchers have when they talk to
them about superintelligence is that
maybe the whole theory depends on
intelligence being a scalar something
that we could attach the quantity to
like iq but that's not the case so when
we talk about intelligence in this
context it's basically an intellectual
shorthand sort of like how if we're
talking about evolution
we might describe evolution as trying to
maximize fitness even though that's not
really biologically realistic
and
we would sort of build more mechanistic
arguments if we need to so intelligence
plays sort of the same role in these
discussions
so
nick bostrom lists some hypothetical
cognitive superpowers that an ai could
have
and a subset of these might be
sufficient to bootstrap to the others
and
when i think about this i don't want to
get political here but i'll use the
example of donald trump to make a very
specific point which is that
trump didn't have all of the skills and
knowledge that other politicians had
but he had some unusual skills he was a
world-class entertainer he could not
distract the media
he understood a certain segment of the
population really well
and his
you wouldn't be able to put him on a
political iq scale necessarily but the
skills he did have were sufficient that
he could take over political party win
an election
uh he got the nuclear launch codes
he had the entire
talent and resources of the federal
government at his disposal
so
there are kind of a lot of different
potential paths by which
an ai could become very powerful
and i think the extent to which
intelligence skills are generic versus
task specific is still a big open
question in our field
so um what can we do about this well
there's a field of ai alignment um also
known as the ai control problem that
basically tries to ensure that the
superintelligence if built would act in
our interests or at least not cause
catastrophic harm and so one of the
books to look for about this is human
compatible by stuart russell and i'm not
going to talk much detail about ai
alignment techniques in this talk
um it's a little bit tangential to this
discussion
um but basically this is why we're
you know why we should be thinking about
ai timelines now right if the hard
take-off is actually possible then our
only hope is essentially to
solve the ai alignment problem in
advance
or maybe design ai architectures that
aren't consequentialists and like don't
have
goals and desires and things like that
all right so the first
that that's all kind of setting up the
context what i'll talk about
the first main part of the talk is about
forecasting
and if you think about
other
important global issues like climate
change
here is
a report from the
intergovernmental panel on climate
change
which
summarizes the contributions of hundreds
of researchers
who are i'm collecting data and
analyzing it and building models and
trying to form quantitative forecasts of
how climate change might progress under
different assumptions and different
actions we might take
and unfortunately we don't have anything
equivalent for ai
we have a lot of people in the field
making sarcastic tweets and we have
some individual people trying to make
educated guesses about how things will
turn out but we don't really have a
science of ai forecasting
and
there are two questions that are
especially important um the first one is
how long until we get to human level ai
and the other question is
once we're at human level ai
then how rapidly will it become
superhuman right would we have an
intelligence explosion or maybe under
some other assumptions that wouldn't
happen
and
even though the question of
whether
ai's or human level is a little bit
ambiguous
i think when you combine these questions
it won't be ambiguous right if civil
intelligence happens um we'll know it
the world will look completely different
this is not an ambiguous question
and so on different assumptions about
timelines that have different
implications about how we should prepare
so if we believe in slow takeoff right
if we believe that once we're at human
level ai it'll be another 50 years until
we get to superintelligence
then we'll probably be okay because the
human institutions will be able to
co-adapt with agi and solve the problems
as they come up
if it happens over the course of a few
years
then we would have time to react in
certain ways right there could be you
know un could
get all the countries together and have
like an international moratorium and
agi
there are also things they have to worry
about like um international arms races
because there could be a significant
first mover advantage
and if um we could have very fast
takeoff like get super intelligence in a
matter of hours or days
then we probably wouldn't even notice
until it's too late
and so the only way we can make things
work out well is if we
prepare far in advance
so what do the ai experts think
well here uh was a survey
given to
ai researchers who published in nurifs
and icml in 2015
and in the top right they were asked to
estimate uh distribution over when we
get human level intelligence
and
different people have
wildly varying intuitions about the
timelines so that's basically the
individual
black curves you see here some people
think it'll happen in 10 years other
people think that we'll never get there
in this century
but in aggregate people seem to assign
about a 50 probability that
it'll happen in the next 50 years
they also pulled
ai researchers on much more specific
predictions like when particular ai
milestones will be achieved when various
careers will be automated
i think there is a little bit of
ai chauvinism here the ai researchers
think that ai research will be the last
job to be automated uh even like long
after mathematicians i'm not sure i by
that
but generally you know they're expecting
major ai milestones to happen sometime
in the next few decades
um very recently the jia culture has
built a more quantitative model trying
to estimate ai timelines
and they tried to
separately
forecast on different pieces of the
puzzle so how will hardware progress
continue
how about algorithmic progress and ai
how much will resource investment and ai
change
but the most difficult piece of the
puzzle was the number of
flops required to train what they call a
transformative ai model which you can
think of as roughly equivalent to agi
so that's the really um hard part of the
analysis it's what they what she devoted
most of the report to
um but at the end of the day she comes
up with this distribution
over the timelines
and the colors represent on different
modeling assumptions that go into it
um but generally it seems to be peaked
in about a 20-year timeline
with a significant tail
and so
um
so she built a detailed quantitative
model of ai timelines
and it had a lot of uncertainty so um at
the end of the day
there's just a very wide um distribution
over when we um over the amount of
computation required for agi
um
but like why should we expect this even
to be a worthwhile thing to pursue right
why should we expect ai timelines to be
at all predictable
and i think the reason for this is that
um ai scaling as a function of various
kinds of resources has turned out to be
really surprisingly regular
um and so
so it seems possible that we could
actually come up with a much more
precise model of ai progress
um and so for instance one question you
could ask about ai scaling is
um how long does it take you to train a
neural net to a given accuracy as a
function of the amount of parallel
computing capacity
so companies like google are very
interested in this question because they
want to
guide their investment in better forms
of hardware
and so it turns out that
this follows very regular scaling curves
so we have on the left
is a plot for a particular architecture
and a particular task and this is a
transformer language model
so the x-axis
is the batch size you can think about
this as the amount of parallel
computation available and the y-axis is
the number of steps that it takes to
train the model to reach a given
accuracy now both axes are on a log
scale
and
the dashed line represents perfect
linear scaling so with the twice the
number of um
places not a parallel computation you
can train your model twice as fast
and what we see
is that
there's a linear regime
and then it levels off so for small
batch sizes you get a linear improvement
in efficiency with parallelism
but eventually it levels off
and
this isn't just one model but this seems
to be a universal map universal pattern
that holds across many different
architectures
um
so on the right we have
everything from comp nets to recurrent
neural networks to transformers and they
all have this characteristic pattern
perhaps even more surprising is that
these sorts of scaling laws
also apply to really sophisticated
architectures like
gpd3 which a lot of you have already
played around with so as you might know
gpt3
is the kind of architecture that was
trained on a very simple task
which is to predict a distribution over
the next um token in a sequence
which you can think of as the next word
in a sentence even though that's not
strictly true
and
now it's turned out that despite the
simplicity of the task of disrespect to
you on gpt 3 can actually do um
surprisingly diverse things like
creative writing it can do simple
arithmetic problems it can write
computer code you can compose
celtic music
and so it's not something we would call
on human level ai
but
the idea like such a simple architecture
supports
complex behaviors
it may be evidence that intelligence is
more generic than we previously thought
and so
you know if you look at the outputs of
gpthree they often seem surprisingly
inspired and creative
but when you measure the performance
quantitatively
you get
amazingly tight
scaling laws
so
in the right hand figure we have a plot
of the
performance metric of a trained model
and there's a particular way of
measuring performance which is on a log
scale
and on the x-axis we plotted the number
of parameters in the model also on a log
scale
and what you get is a very precise power
law fit that holds over multiple orders
of magnitude
you can also
do similar experiments for things like
the size of the data set shown in the
middle
and also for the training time that you
see on the left so the
blue curves correspond to
training runs for individual
architectures um so
it'll take a while to
reach um it's kind of peak performance
and then it levels off
but then if you um combine all these
curves for different architectures
it traces out a very beautiful power law
and so these power laws
are things that actually are able to
extrapolate
to much larger sizes than we're trained
on so this
these plots were generated from gpt2
but then when they ran gpt3 it actually
um you know fit very nicely on the same
plots
and in fact
this is probably why they were willing
to spend
so much on training gpt3 right if they
were
spending multiple millions of dollars
training a single neural net
they probably want to have a precise
understanding of what benefit they'll
gain from the resource expenditure
and one
other surprising thing from this result
is that the performance seemed to be
surprisingly insensitive to the fine
grained architectural choices
so you could have
one really wide layer in the network or
you could basically use the same number
of parameters
to construct a really deep network with
let's say 10 layers
and it turns out that two layers works
better than one layer with the same
number of parameters but beyond that it
doesn't seem to matter um so the rest of
the curves here are basically
um
lie almost on top of each other it
doesn't matter if you're using
two layers or more than six layers
number of parameters that matters
and so i think there's a lot that we can
learn through detailed quantitative
analyses of neural net scaling
but there there's a lot of work that
remains to be done
so
those scaling laws were for
a particular task of sequence modeling
um do similar things happen in other
settings like metal learning
reinforcement learning
um
some sort of game to research and so on
can we get a more fine-grained
understanding of why those laws happen
like is there something about the
statistics of the number of close
neighbors that appear in data sets of
different sizes things like that
um to what extent can different sorts of
resources like data and computation
substitute to each other
and perhaps the most difficult one is
how can we estimate the difficulty of
what remains to be done right
how can we put tasks that haven't been
solved yet
like training a professional lawyer on
the difficulty scale so we can estimate
the time to achieve them
so
these are
some of the open questions that deep
learning can start to address
so this is it for part one of the talk
i'll open it up to a 15-minute
discussion
okay
thanks roger any questions to get us
started
no comments
um i apologize i had a conflict for the
first half hour so this is a very
uninformed question but of the things
that you have mentioned so far
uh i don't know what's the one that you
think is maybe not most important but
that we're like most poised to make some
progress on
yeah i mean i think
there's a lot of low-hanging fruit left
to do in understanding the scaling laws
um so as i say here like
understanding in a more fine-grained
level um
what are the reasons for them
if if a network follows a particular
scaling law does that imply that it's
effectively doing some sort of
non-parametric regression or nearest
neighbors and things like that
so
yeah i think
there's a lot still to be done there
roger oh reid you have a question
go ahead reid
hey yeah thanks um uh yeah this is
i i think this sort of scaling of uh ai
systems towards an agi is really
interesting i just sort of
was wondering
is this falsifiable like you have sort
of this implicit assumption that there
will be some sort of agi in the future
how would we
what if that isn't possible is there any
sort of mechanism to detect that and
sort of following up on that what if it
doesn't look like people are imagining
what if it does exist but isn't it sort
of some framework that
we can think of right now
yeah great questions
so um one hypothesis that i didn't
really discuss is maybe agi is just
impossible right maybe it's physically
impossible to build an agi
and
i think this would be a very surprising
result about physics
right you know it could turn out that
there is something
very special about human brains that
just can't be simulated on a
silicon-based architecture or anything
else we're likely to be able to build um
i would find that very surprising i
think
most people would probably be very
surprised by that
so i so i'm up printing under the
assumption that it's physically possible
it could be that there are big ideas
required that we haven't found yet and
that we're not going to find in the next
um 200 years
right um
and i think this seemed very plausible
maybe 10 years ago right
if you
if you're an area researcher 10 years
ago you could probably write down a list
of
fundamental abilities that humans have
that
ais didn't seem to have and that we had
no
idea how we might be able to achieve
right you know how can we
combine
the
um
reasoning ability of a search procedure
with the
flexibility of a machine learning system
and things like that
by now i think
ais have
shown
so many
different
kinds of abilities that it's hard to
pinpoint
you know
individuals
fundamental skills that they're
obviously missing and it seems like a
very plausible hypothesis at this point
that everything is a matter of degree
and scaling up existing architectures
but this could turn out to be an
illusion it could turn out to be that
there are
fundamental components of intelligence
that
our ai systems just don't have
and that we're not going to be able to
figure out in the next 100 years and so
i think that's what
implicitly contributes to the
very long ai timelines the ones that are
longer than 100 years
thanks brother benjamin
thanks yeah i had a question going back
to um
the sort of goals used that might be
emergent from any sort of you know
uh goalie oriented ai like
self-preservation and one of them was
not wanting to change its goals
um and i was sort of curious about that
one about where that one comes from
because i sort of think i have most of
these but i don't think i have as a goal
to prevent myself changing goals i'm
often quite open to you know in the
future my goals might change and that's
fine i'll just pursue different things i
don't want to lock myself into my
present goal set even if i could
um so why would we think that that would
be sort of something that they would
naturally
emerge from these systems
right so humans aren't
exactly consequentialist agents right i
don't think that even in principle you
could write down a utility function that
we're maximizing
right maybe we're
carrying out certain biological drives
but you know everything isn't really
driven by one
utility function
and so this idea of goal content
integrity basically comes from the idea
of
sequential decision making to maximize
utility and i think the agent would
realize that if someone kind of reaches
into its
software and changes the goal function
then
it will be
doing something else other than
optimizing that objective and the value
of that objective will be much lower
than it would have if it continued to
optimize it
and so a sequential decision making
agent
would be incentivized to try to prevent
anyone from interfering with its
uh with its goals
to like
prevent anyone from interfering with its
goal definitions
thanks roger jack
you're on mute but you want to come off
me out there
is it my turn sorry
it is your turn yes yes sorry
uh yeah um
so i was intrigued by something you said
earlier because uh
i've always been worried about the
ai alignment framing
the ai alignment problem
for the reason that
we haven't figured out
you know the human alignment problem or
the corporation alignment problem or the
ai researcher
alignment problem
so it seems like you know not not likely
to be a helpful way
uh it might be helpful way to pose a
problem but not to solve it but you
suggested earlier that
we might think of non-instrumental ai
agents
um
and
so my question is
does thinking about like the progression
of ai
intelligence uh or
that's redundant the progression of ai
as a scaling
like scaling in uh
in in the ability to achieve you know
the the
performance on these benchmarks kind of
already assume
uh
an alignment problem framing of
how ai is progressing
right i mean
it's a very good point that
there's no well-defined notion of human
values different people
can have competing objectives
and
that's also something that we'd have to
deal with as part of ai alignment
and so you've been leaving aside the
kind of inherent challenges of
ai alignment there are also the usual
challenges
of
any powerful technology
you have to worry about misuse you have
to worry about
international arms race scenarios
and international arms races can also
interact with the safety issues because
then the groups developing the ais could
be incentivized to
sacrifice safety in the interest of
speed and so certainly like all these um
sort of social issues come up the way
they would with any other technology
i don't know if that answered your
question
it was a vague question
well
let's let's put a pin and come back to
it daniel
hi um
yeah i just have like a
quick question i think about and the
analogy that you made
um
of like a super intelligence or agi to
humans and humans to chimps
and and it seemed like you're talking a
lot about doing things faster but when i
think of a chimp in a human
a human isn't doing things that a chimp
does faster it's just it's actually
doing very different things and it has
like interests
that are you know a chip is not
interested in playing go it just doesn't
make any sense for it to even think
about that
and so when we think about or when you
think about an agi or you're thinking
about something that only is going to
get better at things that have human
utility or is it something that's going
to
in getting
artificially generally intelligent is it
going to develop um kind of like a set
of
goals or i don't know if you want to
this is anthropo anthropomorphizing of
course but
that are kind of unrelated to our own
goals even if they're not like in
opposition to those goals
yeah a good question
so when i talked about
thinking like humans but faster versus
thinking at a qualitatively higher level
these are really two different
hypotheses about what could happen
right
so
it could be
that a human thought is basically
turing complete
right you can imagine maybe like
um
the difference between humans and
monkeys analogous the difference between
turing machines and push down automata
but like once you get to turing machines
then it's universal and there won't be
any fundamental change when you change
the computational architecture
right and so that's like that's one
possible hypothesis about the space of
possible
levels of intelligence
right so it could be that
you know just like you can build faster
and faster
universal computers you can build
faster and faster
uh versions of things that think
basically like humans
right but it also could be that
um ai's have the capability of like
thinking about things that humans just
have no conception of right like the way
that chimpanzees have no conception of
quantum mechanics
and that would be sort of like an even
more extreme version of super
intelligence because like we wouldn't
even be able to like reason about like
what sort of things the
agi is reasoning about
thanks roger
sorry dan didn't you want to say to more
on that
no i'm just saying thank you okay all
right good good uh roger i had a
question for you but i wanted to just
check do you want to go to on to the
next part of your talk first or
um
i'll take this question
okay all right good so i wanted to ask
about
whether there's tension between the
first part sort of thinking about agi uh
futures
and the scaling laws
um in the sense that you know do you
think that's the path right i mean
if i i think we hit the limits of the
data we can make available to our
computation systems
probably sooner than we get to
anything like superhuman
intelligence so just think about the
fact that yeah we're training gpt3 but
we're training it on a very tiny sliver
it's a massive right from one point of
view it's about half a trillion to 500
trillion or something um tokens
but um
it's still a tiny sliver and humans
don't learn on those those size data
sets so in some ways well our
intelligence had to figure out a way to
learn a lot and do a lot without
computing massive
quantities of examples
yeah it's a great point
and so it could be that
current methods will just completely run
out of steam
because they'll be bottlenecked by
sample efficiency
and there just won't be enough data in
the world to train an artificial lawyer
things like that
right um
another possibility is that
computation and data can kind of trade
off against each other
so the more
processing you do with the gpt like
architecture um the more sophisticated
representations will be
and the
more
like
the more abstract level at which it'll
be able to do the pattern matching so
you can imagine
like
if you were to
ask gpt3 to solve bad olympiad problems
um like there's probably
nothing written down in human history
that is sufficiently similar to next
year's math olympiad problems that gpt-3
would be able to
solve the problem
right
but if you
imagine you had an architecture that's
sort of like
like a hybrid between gpt3 and alphago
and it's like doing a search procedure
as it goes along
and using what it learns from the search
procedure to improve its representations
then maybe you know it'll be
um doing pattern matching at like a high
enough
level of abstraction that it'll be able
to make use of problems it's seen before
sort of like how the
human
um
imo contestants might notice the problem
and think like you know hey i saw a
problem just like this
um
in like you know previous olympiad but
um
but that they noticed that after having
like done a lot of work to transform the
problem into different formats and
things like that
so like um that would be one hypothesis
in which
more computation would be able to solve
the problems with sample efficiency
um
in which case um just scaling something
like gpt
3 further
could actually result in agi
right right okay great
thanks we'll go on to your next part
okay
so
the second part of the talk will be on
mesa optimization
and
this refers to
when
an ai might find it advantageous to
solve the problem by
building another agent for another
optimization procedure
and so here we have
an ai researcher thinking hey my life
would be so much easier if i built a
robot or a software agent or whatever
um but then maybe the robot is thinking
the same thing like hey my life would be
so much easier if i built an agent
and so
on the idea is that
agi might try to achieve its goals
by creating another optimization process
that has different goals that are
somehow instrumentally useful
um and so then
that other
agent or that other optimization process
will be the one that's actually
interacting with the world
and so we have to worry not just about
the agent we built directly but the
other ones that might emerge from this
process and this is something that could
happen even if we design an agi with a
non-agent-like architecture such as the
comprehensive
ai services proposal by eric drexler
and so this
problem is pointed out by people at the
machine intelligence research institute
or miri and
most of this argument comes from this
paper by um jupiter at all risks from
learned optimization and advanced
machine learning systems
and so in a sense
as optimization is everywhere
so
um one kind of human analogy is that
planned economies don't seem to work
very well but if you're a government
what you can do instead is you can
design the market incentives um so that
the individual agents who are optimizing
their own selfish interests will be
incentivized to contribute to
the overall societal welfare
evolution to the extent that we can
think of it as an optimizer
found that is a good strategy
to
program animals to have
various drives
like
feeding and reproduction things like
that
these drives you know they
might support genetic fitness but
they're not the same thing
and so humans don't go around try to try
to maximize the number of offspring you
can think of all sorts of things that we
do that are at odds with the goal of
genetic fitness
an example that might be familiar to ai
researchers is generative adversarial
networks
basically if you're trying to
generate things that look like images
like we see on the right
a good strategy to do that is to train
a discriminator network whose job it is
to classify real versus fake images then
you can try to fool the discriminator
and so even though your goal is to
produce images
it's instrumentally useful to build this
discriminator network that has a
different objective of classifying real
versus fake
and so
um the reason that we should be worried
about this from a safety perspective is
that
it seems likely that mess optimization
will be a good strategy for generalizing
in the face of moderate distribution
shifts
right if you want to
hire someone to do a task
you could give them a very detailed set
of instructions but you're probably
better off giving them a high level
description of what you'd like them to
achieve and then they can figure out
their own flexible way to solve it
so similarly we might expect meso
optimization to be a good strategy that
ai could adopt for solving its problems
but the problem is that when a mesa
optimizer fails to generalize
then the failures might be more
catastrophic especially if the mess
optimizer is given lots and lots more
resources than it was trained on
and so for the perspective of ai
alignment
we can decompose out two aspects of the
problem there's the outer alignment
problem of ensuring that the um
the base optimizer's objectives are
aligned with our own so we normally
think of as the ai alignment problem but
there's also the inner alignment problem
which is sort of like the outer lining
problem from the perspective of the agi
right how do we ensure that
its
mesopotamizers are aligned with its
goals
and so here's some questions that i'm
deep learning might be able to inform
so can our current architectures
represent as optimizers
can we recognize when measure
optimization is happening um is it
actually advantageous um so we'd expect
our architectures to use this as a
strategy
and
can mes optimization fail to generalize
under distribution shifts
and so
what do i mean by whether networks can
represent message optimizers
well let's say that we want to um
classify objects
in images but the images are blurry
because the
camera person couldn't hold their hand
steady
and
we train a neural net just like ordinary
neural net architecture to solve this
problem
we don't tell it to use measure
optimization
but one possible strategy that it could
use
is to
first implement an image deep learning
algorithm
and then to classify the resulting image
and so what i mean by image d blurring
is um on the right in the top right we
have an example of an image that has
a blur or camera shake artifact
but you can decompose it into
an estimate of the clean image on the
bottom
as well as the blur kernel which is
shown in the top left of that image
so that the blur kernel involved with
the image in the bottom should
approximately give the one on the top
and so you can formulate empty blurring
as an optimization algorithm where you
try to find an image that looks like an
image
but also when you pass it through the
blur kernel you get the observed image
so
um this deep learning problem would be
an example of a mesa optimization
objective right it's not the same thing
as the classification objective but it's
something that would be instrumentally
useful for it
and so um this approach is one strategy
that the neural net could take to solve
this problem it might not just do it it
could solve it some other way but this
is one strategy that's available to it
and so will it do this what would it
take for it to learn an optimizer
well probably the vanilla optimizer in
deep learning is gradient descent so
you're trying to minimize the function
and you repeatedly take steps in the
downhill direction
until you've converged to a local
minimum
so that's that can be written down with
a very simple mathematical formula
and so can neural nets represent
gradient descent
well there's a fairly uninteresting
sense in which the answer is yes
because neural nets are universal
function approximators
the result of running gradient descent
is a mathematical function and so
even simple feed forward networks ought
to be able to represent it
but that's not a very interesting answer
because the
network might have to be extremely large
and it's not a very natural thing for it
to represent
so what are the architectural features
that we
might use
or like what architectural features
would make it easier to represent
gradient descent
well on one hand the upgraded descent
involves repeatedly
applying the same update rule
and so we can encourage that by
having a recurrent architecture
right recurrent means that
the
unit's going to feed into themselves
or equivalently we can impose weight
sharing between different time steps
right so the weights of the architecture
applied in one time step are the same as
the way it's applied in a different time
step
another feature of gradient descent is
that we
repeatedly retrieve the previously
stored value and then update it and so
we want an architecture that's good at
retrieving and updating previously
computed values um this is something
that
lstm architectures and residual networks
are very good at
and finally gradient descent
runs runs a procedure repeatedly until
it converges to a fixed point and this
is another feature that can be built
into neural net architectures
and so none of these things were
specifically designed in order to do
mess optimization um they're just
features that
turned out to be really useful in other
contexts
but because these are
pretty standard motifs of neural net
architectures
it is
pretty likely that
our networks are well positioned to
support mesa optimization
and even though this isn't done very
often it's also
well within our abilities to just
unroll the entire optimization procedure
and
treat that like a neural net and train
it with back propagation in the ordinary
way
um and so this was done
by
um david duverneau in a 2015 paper the
one that introduced the autograd
framework
and
this is also the inspiration for other
techniques like the mammal algorithm
so so we have architectures that are
very good at representing things that
are
essentially optimizers
all right so
um the best optimization is possible we
would be able to recognize it when it
happens and if so maybe we can just
outlaw it we can design our
architectures so they can't represent
mess optimizers
but this is
probably too big a constraint
the reason
is it's just very hard to look at a
dynamical system and tell if it's
optimizing something
so you can consider for instance
heavy ball momentum which is one of the
main optimization algorithms we use for
training neural networks
heavy bubble momentum is basically a
discretization of hamiltonian dynamics
and so if you want to ban heavy ball
momentum you essentially have to ban
your architecture from simulating
physical systems
that seems like a very big constraint
um is mesa optimization advantageous for
the network
will it help it to solve problems that
it wouldn't be able to solve otherwise
um
and
i'm not sure that anyone's looked into
this exact question but we can ask if
the sorts of features i identified that
make
neural nets better at representing mess
optimizers do these features
also improve generalization
and so here we have two papers looking
at particular
uh architectural motifs that support
mess optimization
on the left we have recurrence on the
right we have
the ability to iterate until you reach a
fixed point
um so the figure on the right was
actually work i was involved with i'm
done by my students tai chi lyon and
geminil
and
what we're looking at here is basically
upwards generalization
so you have a task that's parameterized
by some difficulty parameter
so if you're classifying blurry images
it could be the amount of blur in the
image
and so upward generalization
refers to generalizing from easier
instances of the problem
on the training set to harder instances
on the test set
so what we have on the right
is basically
a couple different tasks where we're
testing upwards generalization
the x-axis corresponds to the difficulty
parameter
and
everything to the left of the dashed
line represents the training
distribution
so the blue curve is the one that
explicitly tries to do fixed point
iterations
and the orange curve is a more
traditional
neural net architecture that just runs
for fixed number of iterations so the
blue one seems to generalize much better
to
more difficult problem instances
and finally if we
implement an architecture that does mess
optimization then might it fail to
generalize under distribution shifts
and this relates to a very big topic in
deep learning research these days which
is implicit regularization
and so basically the idea is that
for
a lot of machine learning problems like
most neural net training these days
there are often many different solutions
that minimize the training objective so
there might be
very different functions that the
neural net can represent that all affect
the training set and which one it
chooses just comes down to
which one it finds first which one is
easier to find with gradient descent
so this is a kind of implicit
regularization to contrast it with
situations where we explicitly impose a
regularizer on the model
and so
the cases we're most familiar with in
machine learning are the ones where
implicit regularization is beneficial
so here's an example of a regression
task we're given a regression data set
that we see on the left
and we fit a degree 6 polynomial to it
and
on graded descent
winds up learning this function that we
see in the middle
but it could learn a different function
right
so the one on the right is a perfectly
valid solution it also fits the data set
perfectly
but the one in the middle is easier for
gradient descent to find
and so grading descent seems to be
imposing some sort of smoothness
assumption which is often a good
assumption for doing regression
and so this implicit regularization
effect is the big part of why deep
learning actually works
but what's easy for grading descent to
find isn't always
what makes the most sense or it's not
always what is intuitive to us
so here's an example
of data set distillation by the way this
example
was done by my student paul b cole
so in data set distillation we're trying
to summarize the data set into a smaller
data set such that a model trained on
the distilled data set will do just as
well as one trained on the full data set
and so we have the base objective of
maximizing the performance on the
original data set
and the meso objective is to maximize
the performance on the distilled data
set
and so basically the algorithm is trying
to adjust the locations of the distilled
points
in order that the
meso optimizer will perform as well as
possible on the original data
so
um
this is what a good solution might look
like
so we can summarize the blue data set
into the red one which has a much
smaller number of training examples but
pretty much perfectly outlines the
function
what happens if you actually run
gradient descent on this
this is pretty surprising
right
so even though we're making it very easy
for it by giving it 100 points to work
with it still doesn't really make use of
them right it essentially just
um keeps the points in a few clusters
and
that is
enough for it to optimize
the outer objective right so because
we're giving it a polynomial regression
model
um these clusters of data are enough to
solve the original problem but the
solution isn't going to generalize
right if we replace the polynomial
regression with a neural net or
something like that then it's going to
make crazy predictions
so this is the sort of thing
that
we actually have the conceptual tools
now in deep learning to
analyze um and so we can actually use
the source of techniques that have been
developed in surprise learning to
explain in much more detail why this
happens
and so in terms of what
deep learning can contribute to
understanding as optimization
well how do we recognize if a neural net
is implementing a mess optimizer
under what conditions is it likely to
happen does it depend on the network
architecture does it depend on the
distribution of training data or the
nature of the training tasks
um if it implements a mesa optimizer um
does it
display any sort of agent-like behavior
that we should be worried about
how can we constrain the neural net
architecture not to produce them as a
optimizer or at least can we constrain
the properties of mass optimizers that
it might produce
if it is learning a mess optimizer how
can we regularize it to perform well
even under distribution shift
so these are questions that i hope that
deep learning will be able to address
so i'll
open it up now for the final discussion
great thanks roger um
should we uh let's see why don't we turn
off uh
okay
uh we have a question from muhammad
hello
[Music]
so my question was
right now we have uh like tools such as
cd spy layers
which actually
lets you implement
a
optimization problem as a layer
of your architecture so explicitly
having that so so uh what do you think
about that since that is possible like
you explicitly are doing that
does it isn't it enough proof that
that's
actually possible like even
a particular network architecture part
of it would be doing effectively doing
this
yeah i mean i don't think that um
optimization is something we should be
directly concerned about because
there
were giving in a lot of information
about exactly what the inner
optimization problem would look like
it's something that we can explicitly
put in there
um so the kind of situation that's more
worrying is when it just spontaneously
decides to um optimize something without
us knowing what that might be
but i think that analyzing the behavior
of by level optimization can tell us a
lot about how
as optimizers might behave
right so um and actually the last
example i gave in this talk was sort of
an example of that this distillation
data distillation problem was an example
of a bi-level optimization problem
and so
like when the outer optimizer is tuning
the parameters of the inner objective
there are like many different solutions
it could come up with
um
but some of them are easier to find than
others with gradient descent
even if those aren't the most natural
solutions or the ones that'll generally
is the best
um so i think that yeah we can learn a
lot from those sorts of architectures
thank you so much for this topic was um
very interesting
i think my question might be kind of one
of the open questions because um i have
a question about the political
implications of this right like how is
this going to affect politics like what
actions can politics take
how do we regulate this um what do we
even mean by going against our interests
um
and the other connected question i had
to this was um i found the part about
forecasting very interesting because
it's so related to climate change but
also other types of policies like
pension reform
right especially when we don't know also
like people's discount rates like how do
we know
if people don't care about this and they
have a super high discount rate so that
they don't want to take any action now
because they really discount the future
in this like
how do we make sure that something
happens especially if we don't even know
that what the timeline really is right
and there is that uncertainty
yeah
definitely great questions
and there are like a lot of issues
around the
political implications
that
um i don't really know how to address i
mean there's the scary question of
international arms races
so if we
believe that heart takeoff is possible
then like even if we were to solve the
alignment problem
then we solve the problem that the one
country or one group could get a huge
first mover advantage and so if it's
countries it could be a military
advantage um you could imagine maybe
um like a company that develops agi
first will be able to sort of you know
take over
uh many many different industries
um and so
um and arms races are kind of dangerous
because
from an alignment perspective
any of these actors could be
incentivized to
sacrifice safety
in
the interest of speed which i'm just
like how with the manhattan project they
were willing to take sort of a
non-trivial
chance of like igniting the atmosphere
and things like that because it was so
important to
defeat hitler
so
um
so i think there are a lot of
really complicated
um political issues
involved with that and how do we sort of
maintain
international cooperation
around ai efforts or how do we have
enough
ability to
uh monitor each other that
we don't have these unstable arms race
scenarios
there is an excellent
report
about five years ago
titled
ai national security
commissioned by jason matheny
and they actually compared
ai against other military technologies i
think they looked at
like stealth planes and
nuclear weapons and
biotech maybe a few other things
and
like ai seemed to be like particularly
risky on like many different axes
in terms of like
being
really hard to
monitor
and
being
sort of cheaply available to
um sort of like non-nation actors and
things like that so yeah definitely
a big concern
so what was the second question again
sorry that was about uh the forecasting
part and like uh different discount
rates that people might have which also
relates to issues like climate change
and
um
other types of reforms that are gonna
have benefits in the future but
immediate costs and i was wondering if
you know there was any study into this
specific issue and people's discount
rates
yeah it's an interesting question
i mean i guess
it is hard to achieve international
coordination on things like climate
change um on the other hand
um the gap rights to combat climate
change are sort of um
an almost unprecedented amount of
international cooperation resulting from
scientific considerations
um so it's like you know in a sense
impressive that we've been able to
achieve the level that we already have
um
and
i think there could be sort of an
implicit um
problem here that as you say
uh people have different
um
discount factors so maybe we don't care
about people living 100 years from now
as much as maybe we should um on the
other hand i think the kind of
amount of effort that is
put into
ai alignment today is pretty small even
by those standards so i forget who made
this argument originally but um
there's this argument that if we
found out that there was an alien
spaceship
um
on route to earth and it was going to
arrive here in 80 years
and they had much more advanced
technology than we do
we'd be pretty worried about that like
we're going to start preparing for that
now
right
but if we believe that ai is like
potentially 20 years away potentially 80
years away
then just feels like there's like maybe
so much uncertainty like what can we do
now and this feels kind of inconsistent
right
um yeah i i've heard that argument too
and i can't remember who it is maybe
somebody smelled the alien spaceship on
its way pagan
sorry thanks i uh i remembered who who
that
who that idea is due to and then i just
forgot i'm sorry i hope i remember it by
the time i finish my question um i
actually sort of have two questions but
i'll just ask one and then give somebody
else a chance and then ask another um
the first one is sort of more of a
combination comment and shameless plug
and also question um in the i think it
was the second section of this
section you mentioned banning mesa
optimizers
um
and how like basically we can't do that
because that would be essentially
banning optimization or physical
constraints or like hugely broad classes
of
algorithms
um
but basically i want to say i think in
general sure we can't ban it but for a
lot of specific use cases and
maybe huge swaths of use cases that are
really important i think we can and it's
already starting to happen and the the
shameless plug part is
some work that i've done developing with
david kruger who's now at cambridge on
unit tests for uh incentives to cause
distributional shift
and i'm pretty optimistic about this
kind of approach in content
recommendation specifically so like
online you know use news feeds or like
facebook feed that kind of thing um the
eu has already banned user manipulation
in those contexts although what that
means in practice uh you know
nobody knows yet but
if that kind of ban
in policy holds and policy makers are
looking for ways to ensure that i think
one way to do that would be to ban misa
optimization in content recommendation
and
i think we have sort of some starts on
ways to do that practice
right i mean i guess when i talked about
banning i wasn't even like thinking at a
political level i was thinking like you
know if we were trying to like
put constraints into the algorithm that
prevent it from doing this optimization
like let's say we wanted to do that
right um that i don't see i would even
be able to
draw that line
um yeah so like if we were to try to
like prevent mess optimization from
rising that would mean for instance that
if your
neural net is trying to
classify blurry images
then like it wouldn't be allowed to
try to
reconstruct a clean image by inverting
the blur kernel okay that would be an
optimization problem
i guess what i mean to be more specific
is that sure we can't ban musa
optimization in general but we could
maybe successfully ban particular misa
objectives
um
or prevent like strongly discourage them
right we we could
you know hypothetically
ban people from directly implementing
them as objectives we
very hypothetically you could say you're
not allowed to implement it by level
optimization routine that has the
following inner objective
right
but this wouldn't prevent
the network from
on its own deciding to solve a problem
by implementing an optimizer
right but if we can um identify the
objective sort of like making algorithms
fairness blind uh or like group um
group blind
we could discourage
the misa optimizer like basically make
it difficult to find the solutions that
we don't want for particular cases
oh let's see so
like like what would the actual rule be
like what you know
if you were the dictator like how what
sort of rule would you impose that would
prevent people from
um writing algorithms that would have as
optimizers
not broadly that's why i agree with you
we couldn't like prevent misa
optimization from happening in general
but with explicit regularization for
instance i can imagine we could achieve
not having a miser optimizer for
particular
misa objectives in practice and the
specific example i gave was achieving
performance via causing distributional
shift
that's one that we have a unit test for
if an algorithm does that so you could
put our unit test as a component of an
optimizer and
at like the
the higher level optimizer and that
would discourage the inner optimizer
from like arising
can i can i jump in here um so i'm just
trying to translate so what
the mesa optimization why is that
changing our ai safety problem what is
the
uh i mean if you're gonna talk about
banning it and i'm thinking okay so so
in what sense is this um
contributing to the alignment problem in
the sense that you might say okay we
don't want this kind of optimizer to
arise i
it it doesn't i mean especially because
as you know there's lots of cases where
in fact we achieve lots of objectives by
creating these other
like you know you talk about markets or
biology and
biological drives that kind of you know
by optimizing that drive you actually
optimize the ultimate objective
so i'm trying to think why is this a
consideration for ai safety in
particular
why is it changing the problem is it
is it that it's more complex is it that
it's like is it because of
explainability like we knew what
objective we originally gave it and then
turns out you know the system has
invented something else and it's it's
hard for us to uh observe that is that
yeah so i'm just
how you see this as an important topic
for something we really need to think
about from an ai safety point of view
yeah i mean i guess the
um sort of long-term safety
consideration would be like maybe you
have
an ai whose values you've successfully
aligned with ours
so it's like um
it understands human values really well
and it's motivated to achieve them
and
it realizes that
money would be very instrumentally
useful for
achieving our objectives and so it goes
off and implements another
agent whose terminal objective is to
make as much money as possible
and then that agent maybe doesn't have
the safety properties and so it
um you know acquires more resources and
turns the world into dollar bills right
and
um it's hard to see what sort of safety
properties you can put on the outer
system that will guarantee that there
wouldn't be some kind of inner system
that arises that has like different
objectives and it doesn't have whatever
safety properties you put on the outer
one
okay i mean i guess i sort of see that
that's how i understand the the
alignment problem in general is right
okay if you say cure cancer the system
may say well i got a way to cure cancer
we'll kill everybody
um then nobody gets cancer
right and and that's that i think of
that that's the alignment problem right
that you could get solutions in that way
so i'm not i'm not sure what the
additional but the addition of the idea
that there's this visa optimization
layer
what that adds
i i think i am just missing it so i'm
just trying to understand what it's
adding
yeah i mean i guess it's
um
yeah
it's a subtle distinction
and
i guess in that example
you could have an agent
right this is like a failure where it
um you know cures cancer by like
blotting out the sun would be like
something where
it i think it's misunderstood or
objectives
it takes actions that
maybe it mistakenly thinks would be
beneficial to our goals
um
whereas
the sort of picture we have in mind in
this optimization
is that it sort of correctly understands
what we desire from curing cancer
but it decides a good way to do that
would be to build another agent
that has objectives that are
instrumental useful to it like
acquiring as much money as possible
and then like that agent becomes really
powerful and
um
it works contrary to the
outer one's objectives
so if we were going to translate this to
the principal agent framework
if you know where the principles
we create the
agi agent
that
agent also becomes a principal with
respect to that to the other agents and
if we haven't thought through the
dimensions of how do we constrain the
capacity for the
agent to become a principal in another
relationship that right right exactly
control there okay right and i guess in
principle like you know if it's
smart enough then it would probably
you know solve the ai alignment problem
itself and like make sure that whatever
agents it creates are aligned with it um
right so i guess the worry might be if
if it's like
you know
not smart enough to do that but it's
like smart enough to give one another
agent um
then it could cause a catastrophic
problem for that reason
oh i see an impossibility proof in here
somewhere yes
um yes okay great uh let me let's go to
jack
and then avery
uh hi again uh i put in the chat um
like
uh
i think an example that might be a kind
of intuitive version of an answer to the
question that was just asked which is
when a police officer
uses racial profiling
they are
using a
mesa optimization thing which is totally
at odds they might correctly understand
the goals of public safety
but they've instituted a racial
profiling sub-algorithm which is totally
at odds with those goals and you know
people worry about
sentencing uh
algorithms sort of implicitly developing
racial models and stuff like that so
that might be a i'm not sure if that's
the same kind of thing you're talking
about so i guess that's
a question but the question i actually
wanted to ask
was and i really like the the slide you
had with the
uh the polynomial and the distilled data
set i thought was really evocative
example but i i think i was missing
uh a part of the argument there or
didn't understand because in that case
um your distilled data set ends up being
you know however many data points you
asked for right but they're basically in
clumps
and they're in
um you know n clumps because it's an nth
degree
uh
polynomial right so in that in that case
like it's supposed to be an example of
uh meso optimization failing
but
because um you have like a
polynomial finding
algorithm and the data set it's given
is in fact an nth degree polynomial
it's not obvious to me that that would
generalize
to other
contexts uh where you have
a more complex
um
ai
uh on looking at more complex data
yeah i mean we certainly shouldn't treat
that example
as definitive proof of anything
uh i think
it's it's like a nice intuitive example
that probably has some things in common
with what we'd be worried about in
misoptimization it's like not even
exactly optimization in the sense that
we explicitly implemented this bi-level
programming formulation
that was like
you know explicitly telling it that
there should be something optimized in
the inner loop
and so
i guess my reasoning is that the
dynamics that happened in that situation
might still happen in a situation where
it decides on its own to
implement them as optimizer
so like maybe you have just a really
general architecture in the outer loop
but it decides that a useful way to
solve a problem would be to create a
distilled data set
right
and so then
the kind of generalization failure would
arise because
it solved for the distilled data set
with one particular
learning algorithm in the inner loop
and the data set was enough to
convey the correct function to that
particular algorithm
but it wouldn't
work if you replace that algorithm with
like a neural net or gp or something
like that
so it's sort of somehow overfit to the
particular algorithm used for the inner
optimization
i see
okay that's where that's where i was
trying to get out with that um
yeah
um avery your hands went down did you
still have a question
uh yeah sure um okay so at the beginning
of the talk you were using bostrom and
you said agi will know it when we see it
and then in the second part of the talk
you move to
maze optimization we probably won't know
it when we see it right
and i just wanted to ask you just so i
can understand some of the comparisons
where does trump fit in here because is
trump the sort of mesa optimization of
representative democracy based on
imperial rome and a bad faith imitation
of republican rome or is trump agi
and just trying to understand the
perhaps with respect to climate change
right would would a mesa optimization of
the climate change problem be something
like the oxygen die-off when the with
the evolution of photosynthesis or
um
what are you trying to point to a kind
of emergent property from the ai that
can't be controlled and that will come
about through maze optimization or do
you want us to think about how maze
optimization lead to something at a
larger scale that can be forecasted and
tracked um i don't know if that i don't
know if i'm following the
which the way that you're putting this
together so tell me if i'm totally
missing yeah i mean it wasn't intended
two parts of the talk to like fit
together quite that tightly um
that you can
think of as like two different ways in
which i think
deep learning research can
clarify some aspects of ai alignment
so
um
i mean i wasn't trying to take the um
trump analogy very far like that was
just sort of you know a very specific um
point about
you know not always being easy to put a
scalar value on intelligence that
doesn't mean that it wouldn't prevent an
ai from becoming very powerful
um
in terms of like
with um
super intelligence i guess you'd know it
when you see it or at least
like yeah if if
if it chooses to like act in the world
then we know when we see it we might
decide to stay hidden for a while um
miss optimization it doesn't necessarily
require
um human level intelligence right it's
the sort of thing that
can arise to some extent even in today's
systems
um the like version of it that would be
worried about would be if the
inner optimizer
um
sort of has agent-like behaviors or it
would be incentivized to do things like
um preserve its own goals and acquire
resources and things like that right
like
at some level sophistication the inner
optimizer might develop the same sorts
of convergent instrumental goals that i
outlined
at the beginning of the talk um it'd be
like hopefully a long time before we
have to worry about any of that but
we might as well get started now on
understanding so
conceptually what happens
great
thanks roger thanks avery and a last
quick question from francois
uh yeah so you were um talking at one
point with that
alien spaceship analogy how if we knew
aliens were coming everybody would sort
of jump on solving that and
ai alignment we just don't have
um that sense of urgency
um and so my initial thought then was
like okay well everybody in ai should
just move towards ai safety but then
part of your talk today highlighted how
work that's been done independently of
ai safety just developing deep learning
can help us sort of
um shape the problem of ai alignment or
gives us more insight than if we were
just fumbling in the dark so i guess
from a policy point of view like if if
you were the dictator and you could say
this is the proportion of researchers
that should work on safety versus other
aspects of the ai what would you or
would you pick
yeah it's a really tough question
um
i mean i think right now with such a
small percentage of the overall ai
effort is going to safety that that
should
be much higher than it is now it seems
hard to put like precise numbers on
how it should be
that's a great that's a great place to
to end in terms of how much we should be
spending time on this i think more than
we are how's that more than we are um uh
explore something uh thank you so much
roger that was a
terrific talk and a great way to kick
off our series this semester
uh and thank you all for joining us uh
next week we have a talk uh by uh
visiting scholar avinash collis who's
going to be talking on quantifying the
user value of social media data and i
hope i'll see you all then thanks a lot