Irina Rish - Out-of-distribution generalization

[Music]
hey everyone and welcome back to the
tours data science podcast and today
we're talking about one of the biggest
problems in ai and that's out of
distribution generalization
during training ais will typically learn
to make predictions based on features
that are easy to learn but deceptive
imagine for example an ai that's trained
to identify cows in images
now ideally we'd wanted to learn to
detect cows based on their shape and
color but what if the cow pictures we
put in the training set always showed
cows standing on grass
in that case we have a spurious
correlation between grass and cows and
our ai might learn to become a grass
detector rather than a cow detector
which would be a much easier task for it
to accomplish
even worse we could only realize that's
happened once we've deployed our ai
system in the real world and it runs
into a cow that isn't standing on grass
for the very first time so how do you
build ai systems that can learn robust
general concepts that remain valid
outside the context of their training
data that's the problem of
out-of-distribution generalization and
it's a central part of the research
agenda of irena rish a core member of
the miele quebec ai research institute
and the canada excellence research chair
in autonomous ai irena's research
explores many different strategies that
aim to overcome the out-of-distribution
problem from empirical ai scaling
efforts to more theoretical work and she
joined me today to talk about just that
on this episode of the towards data
science podcast
[Music]
i'm really looking forward to this
conversation i think there's so much for
us to talk about um maybe one of the the
best places to start i think given the
the breadth of your interest the breadth
of your research would be to explore a
little bit with the intersection of your
research with
safety and generalization so i was
trying to figure out when i was doing
research on you looking at all the
papers your labs put out i was trying to
figure out whether you'd consider
yourself to be an ai safety researcher
an ai capabilities researcher something
in between because it seems like you
contribute to both um so i guess my
first question would be like do you
think it's meaningful to distinguish
between the two and if so how would you
describe your research
yeah that's that's a really good
question indeed and
i think
um yeah alignment is a relatively new
and rapidly evolving field and we kind
of realize more and more that things
that we are working on
while working on capabilities actually
may relate to alignment as well so
historically uh i was focusing on
capabilities so historically i was kind
of always working on topics like
continual learning
out of distribution generalization
adversarial robustness well most
recently
and so on and so forth i was also
working on some neural inspired kind of
approaches like alternatives to back
propagation for example which may be
more biologically plausible and
hopefully also more um
computationally effective
and other type of classical capability
uh kind of based approaches but
i i would say that i'm kind of getting
into alignment field i'm very interested
because i definitely realize as people
say that uh with gpt3
it was alex net moment of alignment
right
so pretty much now you are able to
build ai systems that are highly capable
without really understanding their
behavior without really understanding
how can you affect this behavior and how
you can steer this behavior
so it's it's a
good time for alignment research
indeed to
flourish
so from the point of view like as you
asked whether i see myself as alignment
researcher i would say that what i was
working on and continue working on is
definitely related to a safety at least
to one aspect and this um you know the
the older paper
that's an entropic website by paul
christian and others about
concrete problems in ai safety right
one of them is precisely out of
distribution generalization or
stability robustness to distribution
shifts
and i think it's uh no need to explain
why it's related to safety but classical
examples are you have some
critical application in healthcare you
have system trained on x-rays to
diagnose disorder and it was
unfortunately using some spuriously
related features that have nothing to do
with disorder and have everything to do
with how those particular images were
labeled in particular hospital you can
only imagine what happens if you take
the system and start using it in another
hospital if it's based on spuriously
correlated features
another example i mean there are many
examples like that but actually you
don't even have to go too fancy
even stability to
simpler things not just like shifts and
distribution but i mean
just code just like stability the
problem the system parameters
i mean if there are slight perturbations
and something kind of goes wrong and the
system stops working and makes incorrect
prediction and it's some again critical
mission
yeah i mean there are many examples like
that the oldest is not even related to
out of distribution but it's my favorite
about the
famous failure of the
spaceship because of some fortran error
i don't remember details but it's almost
something like a semicolon instead of
column
uh talk about safety right right yeah ai
systems have to be so
um so precisely engineered so low
entropy so carefully crafted that like
small perturbations cause complete like
misalignment or complete um
unpredicted behaviors
right right why is that the case and why
is that of distribution generalization
so hard so historically it was not the
main focus of machine learning research
like historically for how many decades
and uh basically starting even from
statistics which is kind of the mother
of machine learning
the goal was usually to achieve better
performance to have a better curve fit
if it's regression or to have better
generalization and so on so forth the
focus on robustness and of course there
are areas uh in both statistics and
optimization about uh kind of robust
robust optimization so on but it was
somehow
under
not on the focus of machine learning
research for a long time then people
focused on developing uh highly
performing deep networks only to notice
later that they're extremely
uh fragile in terms of adversarial
attacks the famous examples of just
eating
kind of
randomly looking kind of noise like a
pepper spray type of noise on top of a
picture which people will not even
notice can make system
um can actually break it completely and
it will completely
misclassify the image and then you don't
even have to do anything adversarial you
start switching between as i mentioned
this
examples of
medical imaging or the proverbial cows
on the beach and cows on the
green pastures when you try to learn
classifier of an animal and your deep
net uh tends to pick shortcut features
which are easier
but they are not true features of the
concept they are not shaped they are
background just because it happens so
most of the images were on the green
background for cows and yellow
background for say camels and then you
move to cows on the beach the system
doesn't know what to do so i think that
was kind of noticed simultaneously in
multiple areas of machine learning and
that triggered uh all the recent work on
invariant risk minimization the famous
paper by martin luther king leon batu
and
colleagues from
from facebook and that paper triggered
the whole line of research
more recently on various other methods
could you dive into that actually so
invariant risk minimization that was
that was actually entirely new to me
when i ran into it as i was going
through your work could you explain the
philosophy behind that what's the the
basic idea there it dates back to like
work in statistics earlier and so-called
invariance principle and essentially uh
it's one of the ways of defining
causality uh like if you
have certain variables they don't have
to be the observed one they're kind of
hidden extracted variables but if they
exist and they
essentially remain invariant for example
shape um the shape features of the
animal
they are invariant across multiple data
sets no matter what the background is
and they kind of causally define the
type of animal for example they closely
relate to that concept
while there are other features like
background which may be highly
correlated extremely predictive on each
particular data set but they are not
stable or robust across multiple data
sets so this kind of principle of
invariance
led people to kind of
ask asking questions like how can we
find features that are invariant that
are truly causally related to concept of
interest
and uh the iran paper was kind of among
the first ones so essentially the idea
was
uh
let's
consider different data sets that we are
training
our crucifier from let's not shuffle
them as leon famously said nature gives
us data sequentially and doesn't shuffle
them
uh before training so let's not shuffle
data either let's consider environments
and data sets as separate
so that we can formulate uh objective of
robust predictor kind of
learning
so that we say we would like to build
using deep network for example
features or representations that are
robust across all these environments
assuming that there is something
invariant across them so assuming say
there are animals and we're trying to
classify them so let's try to
extract such robust features that on top
of them we can have single classifier
that is simultaneously optimal for all
these data sets of environment so just
to make this concrete then for people
who aren't familiar with the concept um
so
as i understand it this involves you
know you gave this example of cows with
the background you know
most pictures of cows might have grass
in the background because cows are
usually there but a cow is still a cow
even if you move the cow into a desert
or even if you have the cow indoors and
so when you collect data sometimes you
have a bunch of people who work let's
say in an indoor cow farm or something
because they're part of the cattle
industry and you might have trained on a
bunch of pictures of cows in the
pastures but then at validation time or
test time you have a bunch of cows
indoors and the classifier goes nuts
because it got used to using the pasture
as a as an input yeah yeah it it kind of
there is a nice survey paper about their
shortcut features
about propensity of deep learning
algorithms
to
immediately jump onto easy shortcut
features
just like uh they they give nice
examples like some lazy students instead
of really studying they kind of memorize
something and that helps them to pass a
test but it doesn't help them later on
when
the problems get more complicated so
only gets them to the immediate goal
and they may do it well but when the
kind of number of things they need to do
broadens
that doesn't help so the same may happen
with deep networks and we don't want
them to be lazy students
jumping on those kind of shortcut
features we want to
kind of
steer them towards
learning the true
invariant
properties invariant mechanisms like
physical or whatever laws that are
actually in common to whatever data
they're going to discover encounter and
this seems almost intimately tied to the
idea of overfitting i guess i'm
wondering whether is do you see it as
being identical with overfitting or is
it like a little bit different from
overfitting as a concept ah it is
related uh it is related but it's still
i think it's still different so
overfitting uh potentially may happen
even like without distribution shifts so
you may have some distribution of data
and you keep uh kind of drawing from
this distribution and test data also
from that but if you model
in well in classical of course of a
fitting case it's just when the number
of samples was insufficient and the
modal number of parameters was too large
and
important to note model was not properly
regularized
because that could be the key the fact
that you have small number of samples
large number of parameters
does not necessarily mean that you will
kind of overfit if you properly
regularize you want and there are many
examples
everywhere including my own experience
in the same
neuroscience biology brain imaging where
you have say few dozens of subjects and
80 000 parameters svm
or
sparse regression and yet it can
generalize because of regularizers
so
back to your question out of
distribution is a bit
it is related to overfitting in a sense
to the distribution but
um there there might be different
aspects of like what makes it overfit so
it's a good question and by the way um
talking about overfitting and talking
about um
what can help
learn models that regularize better
that generalize better i mentioned
regularization and that was usually a
classical
um classical way of avoiding overfitting
and classical way of potentially also
steering model toward more robust ones
like regularizers of iran
regularizes of invariant learning
consistency or ilc it was another very
hot
kind of approach to this type of methods
many many others were proposed there is
whole the main bad benchmark
and our group john christophe actually
extended it to woods we'll put it on
archive hopefully this week anyway there
are all these methods and they add
regularization
but apparently
recently as we've seen uh the other way
to maybe achieving better out of
distribution generalization could be
scaling and i think that's something you
also
kind of mentioned briefly you would like
to hear about yeah that would be great
because scaling obviously is i mean it's
exploded since gpd3 kind of served as
this big
aha moment for the scaling argument um
yeah i'd love to get your perspective
maybe first off on like what is the
importance of scaling as you say to
generalization and then maybe what do
you see as the most important events in
the history of scaling thus far yeah i
mean indeed it's quite on one hand it's
relatively recent work but on the other
hand i mean the whole idea of neural
scaling laws is to kind of treat
uh complex
quite complex artificial systems like
say neural net model
as in a sense natural in the same way as
you could treat natural kind of complex
systems that's why well every time a
physicist take a
good look at what's happening in machine
learning something interesting happens
so same with neural scaling laws well
jared kaplan
and others his collaborators and so on
but anyway there is there is a good
history of um
physicists and particular statistical
physicists bringing very interesting
perspectives into ai
even before machine learning like phase
transitions and constraint satisfaction
problems and
approximate methods for inference and
graphical models like kuch
approximations and so on so forth so
with scaling the story is okay so
you have these complex models and you
may increase the amount of data
to feed them you increase their size
it's hard to analyze um
how they're going to behave completely
theoretically although that's classical
theoretical machine learning
but maybe you can do it empirical and
that's essentially what was done
especially in the context of gpt3
so the interesting thing is that at this
scale
you start seeing
very beautiful laws almost like the law
of large numbers also appears at scale
right right so when things are small
maybe you can analyze them theoretically
when things are medium
it's a mess
but then when they are sufficiently
large some
laws emerge and that was like really
really beautiful in a sense result
basically the original gpt3 paper and
the follow-up papers by jared kaplan and
his collaborators like openai and
his other friends physicists
so basically
uh they noticed that there is a clear
law
well power law in that case
which looks like straight line in log
log plot and that's probably many of you
have seen so far
that this is quite a persistent
phenomenon in a nutshell
there are many type of curves because
performance could be classification or
it could be say
cross cross-centric laws on the test
data so this is kind of the first type
of curves that were shown in the neural
scaling laws for language models
neuroscaling was for autoregressive
models and then for transfer and there
is a whole now industry of neural
scaling loss for x where x is your
favorite
topic in machine learning i'm basically
just describing the
first
neuroscale in laws paper although to be
honest there were some previous papers
in 2017 pointing in the same direction
and so in a sense it's like a core
uh discovery
uh and one step back there is a more
general actual picture it's not just
scaling law in terms of uh
straight line in log log the most
general picture if you think about
performance like loss
on test data or perhaps classification
error or maybe not even test data but
some out of distribution data that come
downstream
it varies but most of the time you see
kind of sigmoid-like earth initially
when you don't have enough data or the
model is too small
uh your performance tends to be bad and
kind of
improved slowly again it's very
cartoonish thing but it's a it did
cartoon picture from 2017 paper from
baidu
and then at some point there is like a
tipping point and things start getting
better
more quickly and that's exactly the part
where the power law happened that's a
straight line
in log log plot part and then eventually
when you have incredible amounts of data
and model size you may asymptote you
reach so-called irreducible entropy of
the data uh basically
in classification setting it would mean
base risk like your model learns
the best it can from
data distribution and due to inherent
noise in the data or uncertainty in the
data there is no way you can do better
than that
so and so forth so say you call it a
reducible entropy for the case of the
like unsupervised just loss computing so
the sigmoid curve
is kind of
generic picture of what might happen of
course in some problems you can get to
that
kind of the power law part faster
in some maybe slower
it's still open area what exactly
happens in every single situation for
every type of downstream task how it
depends on data and how it depends on
algorithm and how it depends on models
it's a whole area of research
but
what was observed in the original
jared's paper was
three famous plots
on x-axis you have say increasing amount
of compute
while allowing to choose whatever model
size and whatever amount of data you
need to perform best taking that result
plotting performance in terms of cross
entropy on test data
and realizing that you have straight
line low block plot which you can fit to
the data and then for your parameters
for your power law
and it's actually being predictive about
future behavior similar thing happens
although with different exponent and
different constant for the
increasing on the x-axis amount of data
and similar thing happens for increasing
amount of models so it's like a
cartoonish description uh there are more
details but i
kind of really invite people to read
jared's papers and they're kind of very
clearly explaining and easy to read so
no i think it's a great primer on this
it's it also invites the question i mean
the gbd3 and scaling laws papers uh all
addressed language models and and um
that's a kind of interesting aspect to
this so is there a special role that you
see for language in the scaling story
here or is it just
language just happened to be the first
and gpt3 was kind of
as i understand
those scaling goals were not just to
analyze it but actually to help build it
because as i understand those
predictions on smaller kind of versions
of gpt3
indeed
would hold in the future and kind of
would help to determine which kind of
algorithmic architectural solutions to
use so they will scale better which is
by the way their
beauty of scaling laws
and their usefulness for pretty much
anything and also the answer to all the
skeptical people in academia who hear
about
scaling
and their
knee-jerk reaction is but we cannot do
it because we don't have that compute
you don't need
large compute to do scaling laws because
you are essentially studying trends
with increasing amount of data compute
and model size or potentially other more
nuanced aspects
and you can look at those trends to
predict what would have happened if you
had that compute and that you can do
anywhere in academia at home
so that that's common misconception by
the way and that is more like
methodological approach to evaluating
and comparing machine learning
algorithms and architectures which is i
think very useful and hopefully one day
we'll replace our usual empirical
sections with tables where you take one
benchmark of given size
one architecture of given size and
compare it with another one
and if you looked at the whole
spectrum
from small to medium to large data model
size uh and compute you could see
interesting trends you don't see in your
tables for example classical visual
transformer paper right the visual
transformers at small data regime
definitely are not as good as
convolutional networks but then the
situation reverses and you would like to
see the trend
so back to your question about modality
yeah language was the first one and
there was of course a lot of that
uh but then of course we started seeing
systems like clip dali so they are
multimodal they combine images with text
then we started seeing systems like
perceiver uniperceiver magma from aleph
alpha so you combine not just text and
images you can put videos you can put
audio and you can keep going
multi-modal even further so actually i'm
quite interested we're trying to look
into some uh time series well i mean
it's it's a very initial stage but we
basically would like to do some time
serious foundation models for either
healthcare applications brain imaging or
like financial time series definitely of
interest to our colleagues in morgan
stanley
yes
and then you can think about graph
based
foundation models for drug design
for example yeah yeah so you can have
massively
multimodal
foundation models i mean
it's very computationally expensive when
it gets to more complex modalities and
of course
there is lots of talk about let's learn
from all of youtube and then that will
equal agi
yeah learning from all of you cube is a
little bit challenging at this point but
i think it's uh it's a reasonable goal
for future
well actually so that does make me think
of something you mentioned earlier which
is you know
you you do have this early phase where
returns are very low then you hit a
scaling phase and then there's that
plateau region at the end and one i
guess interesting line of discussion
might be what so what gets us to that
plateau
um you know what is the irreducible
complexity of something like language
for example and does multi-modality
help you
get out of that some because i can
imagine multimodality might help you
transcend a limitation like that if you
know there's irreducible complexity in
language sure but if you're also
providing image data or video data it
kind of seems like that ground audio
potentially like yeah yeah i think
yeah so basically if your goal is indeed
um
i mean it depends what is performance
measure or metric and by the way that
connects very well with alignment
because so far we were talking about
the more commonly used say cross entropy
on test data just how well you actually
manage to capture the data distribution
and represent it right
um
and
if your question is more of a like a
classification or some other type of i
don't know you're trying to do
reinforcement learning on top of those
models which is another recent hot topic
right
yes many people look into then the
question is like if you indeed have
multi-modal
um representation of the same kind of
concept you have like images
and some text describing them and maybe
you hear sounds then yeah i agree with
you that probably if your model is
capable of exploiting all this
information
properly
it should be able to do much better than
just language model or just image based
model and in the sense that what we as
people do right we with use information
from multiple sensors
yeah and that really helps
if you don't see something well but you
heard i don't know like video was not
quite
good and blurry but you heard something
that already kind of gives you maybe
idea about what's going on so if your
goal is to figure out what's going on in
certain movie you can still do that
and so on and so forth so i think the
multi-modality
multiple sensors in like human
gpt
is very important how much of a step
towards agi is scaling
is there other stuff that's required
because i know there's some people who
see scaling as
you know sufficient and some i have such
people in my group i'm not gonna say who
it is
but
okay yeah we have those discussions
every day
um well i think
even people who kind of claim that scale
is all you need
they
mean of course
smart scale
because of course you can probably try
to scale just a single hidden layer mlp
right and technically
theoretically that's a universal
function approximate
that at certain scale
it should be
again by the way nobody really kind of i
think it's a good question just to
formally investigate that the fact that
it's a universal function approximator
plus infinite scaling what exactly would
that imply statistically but intuitively
okay like you can scale that it's
probably going to perform well
eventually too but you probably don't
want to do that right
right you probably rather skill
transformers right
and um
you compare visual transformers with uh
nets and you already see as i mentioned
like from the paper visual transformers
that
the trend is that at scale
you will have more
kind of uh
basically you'll gain more if you use
transformers and not gnets which by the
way also relates to the commonly used um
commonly mentioned
idea of trying to get rid of unnecessary
inductive biases right
which may be too constraining
which is uh interesting question because
you also don't want to throw the baby
with the water
because there are inductive biases and
inductive biases
in a sense everything you do is
inductive bias you use sgd it's
inductive bias i i don't know
but the point is if there are
unnecessarily
kind of specialized
details and constraints that used to
help us like with column nets in the
relatively low data regime just like
with bayesian inference if you have good
prior and not enough data the prior
helps you inductive biases help you
but in the limit with more data
your prior
should be washed out right
and if you start with more uniform prior
or less inductive biases
it might actually help you worse wash it
out faster but if you start with some
very strong prior or inductive bias
which might not be
too generalizable that will
kind of slow down your progress when you
increase the data that's why with
inductive devices like the classical
bitter lesson right
we have all heard about bitter lesson of
saturn that you start ai with putting
rules and expert systems
that work before you start
actually automating the learning process
you don't need those rules you don't
need human knowledge coded in you become
more autonomous you have more general
procedures such as search and learning
and they at scale work better and the
whole ai progresses in this direction
and every time you have some local
success with
human encoded knowledge
later on it's outperformed by more
generic algorithms at scale so it's
important to keep in mind so we don't
waste time on something that not gonna
stand the test of time in the future but
this is it sounds like a really
interesting question because looking
back as you say it like classical
classical machine learning arguably is
just like nothing but a bunch of
inductive priors people go like oh i
think a support vector machine makes a
bunch of sense or decision trees you
know this is like a kind of process of
human reasoning that we could codify and
then like we moved to convents and
what's what's striking to me about this
is convolutional networks for vision
intuitively i think at the time i
genuinely felt like this was not a very
strong inductive bias like of course
images are translationally invariant of
course like this is going to hold for
all of time and as you say now we get
image gpt and we get these transformers
that work for images and we're starting
to learn that even that wasn't um wasn't
quite it was apparently still
constraining in a way right so
apparently there is a better way
to learn
and do we have a sense of like do we
have a sense of what that constraint was
doing like what um what is it that like
image gpt can do
that like um the resnet 50 can't do
through that lens does that make sense
yeah i guess it is good question i think
the intuitive thing about transformers
uh again i'm kind of repeating i guess
common opinion whether it's language or
images uh their flexibility of
transformers is that the attention
mechanism which by the way doesn't have
to necessarily be implemented the way it
is but will not go into details but the
fact that you can have humongous
sequence of tokens which may be from
your language
or they may be a tokenized image
or they could be tokenized any modality
but the point is it can be
extremely long and if your attention
generic approach
kind of to choosing what's relevant uh
is capable of finding
kind of dependencies like arbitrarily
arbitrarily long long-range dependencies
right that gives you that flexibility
and i think that's in a sense again
intuitively um just like hand wavy here
with language maybe that's what really
pushed gpt3 to the level of capability
it has right now
why does it sound most of the time more
coherent than gpt2 or alternatives
because yeah these transformers
first of all
they have the capability of capturing
long-range dependencies and that kind of
relates to coherence of what you're
saying
plus
since you scale them and you scale the
data essentially you had both capacity
of the probabilistic model that
increased
and can incorporate all patterns that
are out there and you had extremely rich
data diverse data that exhibit
those corners of probabilistic space so
in that combination and the combination
of attention that helps you choose
what is
kind of connected with what so
and goes say in convolutional network
case
an image case same thing it goes beyond
pre-specified dependencies it goes
longer range
i think that
i might be wrong but that seems to be
what is helping maybe to detect patterns
that psychologists couldn't defend uh
couldn't detect
or other language models could not
detect that makes intuitive sense to me
because or at least the way i'm thinking
about it now based on what you just
described is like you know convolutional
network it can see the whole image in a
sense thanks to the the stacking but it
doesn't yeah it doesn't prioritize
attention like making sure it checks you
know oh this corner of the image in that
corner of the image you kind of lose and
dilute that information as you move up
the stack whereas attention provides
like this yeah this intelligent way of
of navigating that so if you think about
uh human visual attention
uh so basically you have this mechanism
called aviation right so your um
basically your eye focuses at any given
moment of time
are just a few uh points in the image
and the rest is kind of blurry i mean
there is of course lots of work on that
and the work exploiting that in machine
learning application but if you look at
the maps of kind of attention like how
the human eye traverses image
uh by the way it really depends on
multiple things for example there are
famous experiments where you have same
picture different questions asked about
the picture and your trajectory is quite
different
but the fact that you are able to
pay attention to
parts of the picture that can be far
from each other you don't just
kind of perceive the local
features you really jump like i don't
know the question is like what do you
think the social status of the person
the picture is and you kind of look at
like what the person is holding what
they're wearing this and that and who's
around yeah you look at that attention
kind of map of how your eyes jumps and
you see that oh my gosh it's like all
over the place and it just selected the
most important things well it does make
me wonder i mean is this do you well do
you think gpt3 or sorry gpt or
transformers in general are
what's going to get us to agi i mean
like is is multimodal learning combined
with transformers
going to get us across the finish line
or
is there yet another revolution in
architecture analogous to the one that
brought us from convolutional nets to
transformers in the first place is that
yet in store for us and are we going to
have to get through that in order to
really crack the nut
my personal
answer again it's all hypothesis i still
feel
that yes we do need that potential
revolution
i know that jared would say the opposite
here plus a few other people who were in
our debates
in the neural scaling loss workshops and
also had
two aja debates at mila last fall
um
i know it's highly debatable
i still feel that
we may need more than transformers
uh i might be getting wrong but maybe
it's my neuroscience kind of affinity
talking but i cannot help but notice the
difference between neural net
um kind of nature
and the human
brain nature or any biological network
kind of nature
uh perhaps again it's can be expressed
in transformers i'm not saying it cannot
be but i'd be curious to see what i'm
trying to say is the striking difference
is you have biological network
even some worms that michael levine
experiments with and you probably have
heard his talk
at the new ribs 2018 all the way to the
brain why i'm bringing up worms it
doesn't necessarily have to be the
neural
network it could be biological network
based on bioelectric communication it
has memory and it has adaptation
adaptation obviously to the environment
memory because there's whatever species
develop certain shape
and
the interesting thing you can kind of
reprogram them to develop like i don't
know 200 or 300 worms like michael levin
did and all that is being done by
changing the communication pattern
between cells and that is being done by
chemical kind of interventions changing
some ion channels opening closing them
and once you change communication
dynamic in that network then i don't
know the animals grow more heads and
that solution not founded by evolution
is viable and reproducible
and that makes you think that seems like
memory
an adaptation is somewhere in the
dynamics of those networks and then we
go all the way to the brains
and what does it mean
when the brain is asleep and doesn't
have any image net images in front of it
or any sentences fed to it that dynamics
never ceases you have like
never ending communication
and the
state of the dynamical system is
extremely important in what type of
output will be produced
not just the input the input
kind of modulates this dynamical system
but
there are many opinions and well books
in neuroscience by say george busaki
brain inside out that most of their
information about their future state of
the brain and outputs is not
just in those inputs but mainly like 90
plus percent in that system
and if you think about our transformers
well
they are not quite recurrent feedback
systems with inherent spontaneous
dynamics that is alive even without any
input when they are sleeping
and there is a striking difference and
you cannot help but wonder maybe the
richness
of
human brain functionality or even those
kind of simpler more trivial biological
network functionalities is in that
dynamics in the fact that you use for
encoding not just spatial kind of
vectors but you use
time and maybe we're missing out with
artificial neural networks right now
that
i don't know it might be the case
but it might be just like with
multi-layer perceptron that in principle
it's enough to have transformer
to implement the gi
but there might be just much more
efficient ways to do so with much better
exponent which leads us to good question
and for me it's back to newer inspired
ai
not the final solutions that evolution
found not particular areas of brain but
what was the process
that scaled this network from single
cell of a melba
through all these
worms and other species to this
it's obviously scaling process that
could have better exponent or like with
animals like
elephant or whale
which have larger brain the exponent was
apparently not so good in terms of their
downstream task performance right
so what are those principles of scaling
and nature that maybe we could
use that being forced to distinguish
between you know like the learning
process evolution and then the artifacts
that that learning process produces like
brains um as being two different things
because i think it's we're very tempted
it's very tempting to look at a human
brain and say oh let's replicate this
this sort of shining example this beacon
of intelligence and then you end up i
guess encoding a bunch of inductive
biases if that's your attitude if you're
going to take a human brain and say
let's let's do it like this
maybe you know maybe that does get you
more often to inductive biased territory
i know again again the current just came
up with his uh his paper arguing for
something like that i guess that's would
you call that like a central axis of the
debate right now uh yeah yeah i think in
a sense just like in any debate
uh
sometimes there is just difference in
terminology i think we all agree that
whatever we call it
is needed so some
i mean
some information about how we build
things how we scale but i guess
what
um what the bitter lesson is trying to
tell us and what i really agree like the
last sentence of the bitter lesson is
essentially you don't want incorporate
maybe two specific inductive biases that
developed
in the brain or whatever you may want to
incorporate still some you can call it
inductive advice but rather the more
generic process or procedure of getting
there so you want basically a higher
level of automation you don't want to
tell system
what the architecture should be maybe
you wanted to develop it or basically
what i'm trying to say just by analogy
of this
massive search that could help you beat
the human champion in chess or self-play
and go instead of specific rules of
specific game
you kind of really want to get out of
distribution more general
by not using specific or shortcut
features that only work maybe for
particular tasks and in the short run
can help you but will not generalize
eventually you want to capture more
invariant
principles or mechanisms
that can
automatically lead you to better
solutions that it's not the very
specific results
like of evolution that we want to
implement just like it's not specific
rules of the game uh that we want to
implement but rather
given that we
will have
scaling abilities and we'll have more
compute we want uh to let system develop
those capabilities by only giving much
more general of
higher level
procedural kind of algorithms like
search or self-play and so on rather
than more specific uh kind of things i
mean it's intuitive right it's it's
almost management principle when you
have very powerful uh a very powerful
business behind you very competent
employees um you want to let them do as
much of the thinking as possible and not
be not not you don't want to micromanage
yeah in a way it's like it's the reason
central planning fails for example or
it's the reason that that like uh you
tend to want to see decentralization in
large complex systems the compute is
just
is just better used in that way and if
you don't put in too many inductive
priors your system tends to to be a
little less pathological hopefully yeah
yeah
that's an interesting analogy indeed
well i i wonder what this means as well
for ai safety because like one of the my
reactions when you mention this
possibility of like we might have a new
innovation that that you know all of a
sudden has a way better exponent
makes scaling work like crazy i can
imagine this leading to a pretty binary
leap to something close to agi in a
context where
our alignment research is all focused on
gpt style or i keep saying that on
transformer style models anyway
multimodal or otherwise on deep learning
if the paradigm shifts overnight and
then we all of a sudden get really close
to the finish line or across it
does that introduce a new kind of risk
and do you think that there's room to
think a bit more generally about what
alignment looks like in that context
right yeah um yeah the general concern
is indeed um that
since the system's
gaining capabilities so rapidly
and they are so kind of large and
complex that they cannot really kind of
understand and control them like what
can we do to at least have i don't know
better understanding of what type of
capabilities they could develop
how that depends on scaling
and that um
i mean
safety
direct like
notions of robustness stability and
generalizability and what can happen if
they it's kind of both capacity a
capability and alignment safety related
topic right because you can say
robustness is a
i don't know capability on the other
hand
lack of it immediately leads to
misalignment because system will not do
anything close to what we wanted to do
and can be harmful if the diagnosis is
critical right so but this is just one
example
and um there are other many other
properties like that kind of paper for
example truthful q a right uh basically
looking into truthfulness of answers of
gpt3 well
questions aside how exactly you define
uh your metrics for alignment and that's
a whole good question and but at least
thinking that okay you can define many
more different metrics and properties of
the system that you're building besides
cross entropy
on the test data and besides
classification accuracy those metrics
from truthfulness to
some other properties depending on what
you expect
uh can be also measured and you can look
at the scaling and you can sometimes see
what people say inverse scaling like
truthfulness was getting worse
with scaling in a sense intuitively
um
clearly why if you kind of garbage in
garbage out if you show a system more
kind of data with superstitions and
kind of
incorrect information
the system is simply a statistical model
so
basically it's neither good nor bad it's
just a statistical model yeah but on top
of that of course you could try to
figure out okay so if you want first of
all you can study how various properties
of interest besides accuracy behave with
scaling
then you can ask the question like and
how do i steer the system towards those
properties being what i want
and here are i guess again multiple
potential approaches you can try to
incorporate those properties into
objective when you train the system but
you probably cannot
comp you cannot possibly think about all
possible things that can go wrong
and incorporate everything into
objective and also it feels
um kind of
agreeing with people who wrote their
famous book on
the myth of objective right
[Applause]
greatness cannot be planned and perhaps
things evolve in certain ways and maybe
you can
see how
changes in dynamics
affect potentially behavior in the
future but it's a complex relationship
yeah no we actually it's funny we had so
we had ken stanley on the podcast as
well earlier and and
yeah
haven't spoken with joel yep we'll have
to yeah you're absolutely right it it's
just such a complicated area and the
debates in this are so multifaceted i
mean in a way i wish the stakes were a
lot lower so that it could be just a
fascinating intellectual exercise but
the fact that we're on the cusp it seems
of of making real breakthroughs in you
know human level intelligence that sort
of thing uh really does drive home the
importance of all this work and it's uh
great to see you're putting so much
thought into it and on the safety side
as well um yeah irena thanks so much for
for joining me for this was a ton of fun
well thank you so much


Twelve Tenets for AI Safety

with all that said i want to dive
straight into my 12 tenants for ai
safety
these aren't the be all and end all of
ai safety rules
but hopefully they'll help provide some
guideposts for people who are struggling
with these questions
now to preface this i want to start with
a quote by marvin minsky from mit who's
one of the forefathers of artificial
intelligence
and this is from life magazine in 1970
and
he said once computers get control
we might never get it back
we would survive at their sufferance
if we're lucky they might decide to keep
us as pets
this is not the future we want
and this is the future that we have to
rail against
and part of railing against this future
is saying no
to building systems that could make us
into pets
so understanding that ai and robotics
should serve the good of humanity and
foster the growth of life
by following these tenets you agree
that you will in the next 75 years
enforce the following prohibitions
when building an ai system or a robot
you build systems
that will do
not
have a state estimator capable of
simulating human emotion or
self-reflection
there's a lot of people working on
developing
ai for emotion
for creating simulated emotions
this
is a dangerous path
it's a dangerous path because what we
will get
is not human emotion
is not a complex recreation of our
endocrine system
but just a weak mathematical model
and the intelligence the emotional
intelligence
that creates
will be psychopathic
by definition
it will not be capable
of human empathy
and this is something we don't want to
have happen
will not
make kill decisions
towards humans
or engage in violence towards humans
this is a really simple rule
don't make robots that kill people
that's it
period
don't make robots that hurt people
that's it period
i mean
if you can't follow this rule
there's something fundamentally wrong
with the way you're viewing the world
because this is the path towards human
destruction
this is the path we cannot follow
don't build systems
that will lie to
or steal from
or cheat
humans
look at what has happened
in our world with deception
in the internet
look at what has happened in adversarial
situations with cyber warfare
that is only going to escalate
that is only going to get worse
if we build ai agents
and we build robots
that intentionally cheat us
they intentionally lie to us
intentionally steal from us
we're not going to be able to stop them
not modify or copy ai algorithms
without human code checks
you don't want ai
changing and reproducing ai
this is an active area of research
and again a dangerous one
we should not follow down this path
not have the appearance or voice
of a general human
this could be referring to a deep fake
or a humanoid robot
there's psychosocial problems
with humans ascribing empathy
to robots that don't have
empathy back towards them
okay
we need to stop building
ai systems
that can look
like humans and deceive humans into
believing that they're as human as they
are
we don't want to build these types of
puppets
and that might put a few companies out
of business
but
term for the survival of humanity
it's a vital thing to do
six
not engage in privilege escalation or
hacking on systems without the
permission of the system owner
don't build ai systems to hack
we will not keep up with them
this could be said for general automated
hacking tools right now
your average sysadmin
is not going to be able to patch
and prevent attacks
fast enough to keep up with automated
systems
right now only a few nation states use
automated systems but this is going to
be in everyone's hands because of the
democratization of ai
seven
not have an imperative towards
self-preservation
the minute we give robots
the imperative to preserve themselves
over other things
is the minute we empower them
to
do harm
in the world
don't go down this path
eight
not genetically engineer biological
systems without human oversight
this is an obvious one
don't give genetic engineering tools
to machines they're impervious
to viruses
don't give genetic engineering tools
that could destroy humanity
to
robots and artificial intelligence
simple
simple rule
nine
not to swarm or congregate
physical robots in numbers greater than
can be controlled by a single human's
attention
in unstructured environments
let me just unpack this for a bit
swarms are dangerous
swarms
overpower
humans limited cognitive capacity
humans can only focus on a few things at
a time
we
don't have the evolutionary adaptation
to dealing
with large swarms of robots
swarms
could be our destruction
to not let there be swarms
10.
not make substantive human life
decisions about food
water
medical legal
employment or housing
without a human-based appeals process
have you ever been stuck in bureaucracy
with no recourse
but to get an automated phone line
you want to be able to talk to a human
let there always be a path for people to
talk to people
to sort out their problems
11.
not discriminate based on ethnicity
religion gender sexual orientation or
social cast
i mean this is what we strive towards
this is the ideal of the best of
humanity
we don't want robots
breaking
our progression towards this ideal
okay
finally
12
to not support committing genocide or
ecocide
this is the ultimate rule
we can't create
robots
to kill ourselves
or to kill large portions of the
population
because if we do they will kill us all
so that's it
those are my 12 tenants of ai safety
i hope
that you follow them
for all of our sakes
thank you