Ethan Perez - Making AI safe through debate

hey everyone jeremy here welcome back to
the tour of the data science podcast
today's episode is about one of the
biggest
outstanding open questions in a.i period
and that is the question of whether
we'll be able to get to
superhuman levels of intelligence using
current ai systems
one of the reasons that a lot of people
are skeptical about whether we'll be
able to achieve superhuman intelligence
using conventional machine learning is
that conventional machine learning
algorithms are typically trained on data
that was created by
human beings so the logic goes how can
you ever achieve
super human intelligence by training it
on human level
data now there are a lot of strategies
that speak to exactly that question
trying to get superhuman intelligence
from human level data
one of which is called iterated
distillation and amplification or ida
for short
and while idea itself comes in a lot of
shapes and sizes one of the most common
and fruitful applications of idea to
date has been question answering
systems now that typically involves
breaking down complex questions that in
theory could be so complex that they're
not human understandable
into simpler questions that humans can
actually parse and that ai systems that
are at human level
can actually answer and work with and my
guest today ethan perez is an expert on
ida flavored question and answer systems
of exactly that type we're going to be
talking
about those systems about his thoughts
on how they might give rise to agi
and his thinking more generally about ai
safety so lots to get into i hope you
enjoyed this conversation as much as i
did and without any further ado
here we go hi ethan thanks so much for
joining me for the podcast
hey good to get to chat i'm really happy
that you're here this is
actually a conversation that i've been
excited to have at some point with
someone
and you're just absolutely the best
person to have it with i'm i'm really
curious about your views on a particular
kind
of alignment ai safety ai alignment
strategy
a family of strategies that broadly
falls under debate
or ida and we'll get to what that is in
a minute
but uh but first i want to get to
understand you a little bit and explore
your background how you came to the
space in the first place
so how did you discover ai safety and ai
alignment and what brought you to
do research on it full time i was
generally
excited about the long-term impacts of
ai i i sort of early on like
had some background in math and i was
thinking about oh what what is sort of
the impactful things i can do
with that kind of background and got
more into machine learning
and then i i read nick bostrom's book
super intelligence which i think
was probably a common read for people
with these interests
and i think it just generally got me
thinking a lot about
the long-term impacts of ai that i
should really be thinking about
like what happens when we get technology
that
is close to human level or even
surpasses human level
in its abilities and you know that might
be the kind of thing that happens
at the end of my lifetime or not even in
my lifetime but it's really i think
where a lot of the impact of ai will
have
and where we'll be able to use ai to
actually advance
our human knowledge which is the thing
that i i really care about
by the way sir just to ask a little bit
about the superintelligence side of
things like what was it about
super intelligence that shifted your
view on things or how did your views
shift as a result of reading the book
i think it's that the book focuses
a lot on very long-term considerations i
i don't
think i have the expertise of the time
to sort of gauge the very specific
arguments that
nick made about existential risk and
other things at the time
but i think it did it did make some
arguments that
ai and like very powerful ai systems
were plausible
one key argument that i remember was
like
let's say that we had a system that
could achieve
human level ability on some tasks and
like maybe
some imaginable way to do that is to
just emulate
the human brain and that might be just
extremely computationally expensive but
at least
in principle it does seem possible
because we are sort of an example of
such an intelligent system and we just
need to sort of like copy that
um and then if you do some sort of
copying procedure maybe if you don't
think neuroscientific root is possible
we could do this with like supervised
learning or unsupervised learning
on human text or data at a very large
scale and get something to a similar
effect but that seems sort of plausible
and then he makes this point that oh
well you could just speed up or
parallelize that software at a very
large scale and then get something that
is superhuman in one sense and that it
can do things much more quickly than
a human could but at the sort of like
human level quality
of ability i think that that got me
excited about like
oh there's just like so many things that
i'm probably like in terms of like my
reading speed
um in terms of like how much i can learn
and
it would be like really exciting if i
could just have 1000 copies of me like
go read different parts of
the internet on i'm like looking up a
new diet or
looking up a new philosophy question and
then like sort of like report back to me
all the information that they gathered
and i think that
that was kind of a plausibility argument
that seemed pretty realistic to me
of like oh it seems pretty reasonable
that we might get
systems that can do much more than we
can intelligence wise
yeah in a sense it's almost like it
plays with the idea that intelligence is
a very
like poorly defined thing where you know
you can be super intelligent
even by just replicating human neural
hardware almost exactly do it in a
different medium
where computations are just faster
because it's you know
substrate of silicon rather than cells
or something and you immediately have
something that is for all intents and
purposes
superhuman even though the like the
algorithms
are just human it's kind of an
interesting yeah the hardware software
side
so so did that probably like like you
read super intelligence and you were
like i know i want to make this
my life's work that motivated me to take
my first
like major research internship at new
york yeah
yeah yep um with aaron corville there
who
was like just a really great mentor and
had like other really good peers there
and ended up having a successful paper
out of there and then i was
i like really enjoyed the whole process
i'm really excited about just
how we can help humans advance our
collective
knowledge using machine learning like i
i like i just like read a lot and
i'm always excited to like learn more
about the world and it kind of
sometimes i feel like very bottlenecked
or i i feel like
we really need like step changes in our
ability to answer some
questions about the world like
especially i think about this in regards
to like philosophy where it seems like
oh these are like just really hard
questions where it seems like it would
just take us like many many many more
years
uh in order to answer some questions
there and like we really need to be
thinking about new ways to approach
really challenging questions so then i
think after
the first research internship i
started to think more broadly about like
oh what are the kinds of methods that
could actually let us get beyond human
ability
yeah i mean there's sort of like not
really many ways to do that
that seemed plausible and that's where
like that's where i came across
iterated distillation amplification as
well as debate
are in in those contexts of like
different paradigms to
get past human ability and not just like
imitate human ability which
supervised and unsupervised methods are
able to do that actually brings us to
another interesting aspect of this which
is
so right out the gate you're approaching
ida
iterated distillation and amplification
this thing that i've been teasing since
the beginning of the conversation
you've talked about that as a way of
achieving like superhuman
capabilities from ai systems um
so i'd love to i'd love to discuss that
because usually when i hear ida talked
about
it's i spend more of my time talking to
people in the ai safety community so
they're focused on
ida as a way to achieve safe ai systems
that can be interrogated and i want to
talk about that too
but i'm really curious about what is it
that that prevents us from getting super
human performance without ida i think
that might be an interesting
place to start yeah i mean i think the
baseline
um like the starting point to think
about it is well the default way that we
would train
systems to answer questions is with
supervised learning so this is like
all the state of the art and like sort
of standard methods for training
systems to answer questions in nlp are
just using supervised learning where we
like collect a large data set of answers
to questions and then we learn
a mapping from questions to answers like
that's a good paradigm as long as we can
collect the answers to questions
which is possible for a lot of the sorts
of questions that like people are
googling around for
and like very factual questions like
when was this person born
but that paradigm basically just breaks
down when we
aren't able to collect the answers to
questions and then basically then
we just don't have any training data to
train our supervised learning algorithms
and so that's like that's basically the
problem
that iterated distillation and
amplification is trying to solve
is that let's say you have a system that
can do human level question answering
because you've trained it
on data where we have answers to
questions collected by people so you
have this like sort of black box like
good question answering system
now the problem is well
how can we use this system to answer
some question that's like out of its
distribution because
it's not been labeled by the answer to a
question so
let's say like questions and philosophy
i think would be a good example because
we just don't have labeled answers to
those kinds of questions
and so they would be out of distribution
and might require some sort of different
reasoning process to get to the answer
but we have this really good
question-answering
system that can answer a lot of sort of
other related questions
and so the way we can think about
approaching this problem is
by breaking this out of distribution or
hard or question that we don't have the
actual answer to
breaking it down into sub-questions that
we can answer
with our existing question answering
system so we can break it down into
i don't know if the questions like who
like do people have free will we can
first um have like a question about what
does it mean to have free will and then
we can have some
questions around the neuroscience of
free will um
etc etc and then for each of those
questions
we can um answer them with our question
answering system
or if they need to be broken down
further because they're challenging we
can break them down further and then
answer those sub questions
the hope is that by continually breaking
these questions down into sub questions
we'll eventually bottom out
in questions that a supervised question
answering system could train
and where the sub questions
are the kinds of questions that we would
reliably expect our trained model to
like accurately answer
and then like now we've answered like a
bunch of different pieces of this
larger problem and it should be much
easier for us to
predict the the final answer given the
answers to these sub questions
and so so the idea to to nutshell
uh to make sure i understand it is you
have like some complex or
really difficult question and that
question would nominally require like a
superhuman
intelligence to answer it you have a
maybe human or slightly even subhuman
level intelligence that you've trained
you've been able to train it because you
have a whole bunch of questions and the
answers that real humans have given so
you're able to use
those two things to train this like
roughly human level intelligence
and then the hope is that we can break
down this really complex
question into sub-questions that
eventually can be answered
by this like slightly less than human or
roughly human level intelligence
that's the plan yeah exactly
so does that that's so cool because to
me it seems like
if that were true if it actually is the
case that we can do this seems to imply
that in principle
there is no question and correct me if
i'm wrong about this but
there's no question that cannot be
broken down into sub-questions that are
ultimately human understandable i think
you might get into if you make the very
general general statement you might get
into
issues around like girdles and
completeness theorem where like we know
that there are some true statements that
we can't prove the answers to um
but like maybe there's like modest
versions of it where
like for i mean some of those
examples that are used to prove girdles
and completeness theorem are a bit odd
and like not really
like people don't really like generally
care about them but like
you might say that like for most
questions that we like really care about
there might be some reasonable way to
like break them down into sub questions
like sufficiently many sub questions it
might be like just an extremely large
number of sub questions in some cases
that like we would be able to answer
them yeah sorry can you elaborate on the
the girdles and completeness theorem in
connection to this i just
like i think there might be some
listeners who aren't familiar with
girdles
it's an interesting aspect of this yeah
i mean i'm i'm not definitely not an
expert
on it but sure yeah as i understand the
like simple statement of the claim is
that like
there exists true statements which
we know are true but cannot construct
a proof to show that it that true
statement
is true so these are often like
self-referential statements which are
like
like yeah itself for something like that
yeah
yeah or like this statement is false or
like
they have that sort of quality about
being self-referential
um that kind of initially made me sad i
was like oh well like we can't just
prove every true statement like that's
that's really disappointing but i think
it would be even a big step
i mean those are weird because they're
self-reflect self-referential and like a
lot of
they're sort of an existence proof that
we can't provably like reason to every
true statement but i think
it's sort of unclear to me the extent to
which like that actually holds in
practice
and i guess that i guess that itself is
is its own interesting question like
speaking to the limitations
of um of ida or the limitations of any
any strategy i mean to the extent that
from a safety
perspective we want to use something
like ida to make a super intelligence
that could explore
arbitrarily complex ideas
if those ideas end up involving
areas of logic that are like not
ultimately human tractable in the sense
that we actually can't break them down
into even human understandable
terms then it's sort of like the the
genies out of the bottle we have like
we've essentially no hope of regaining
control over that
that thought process and predicting the
behavior of that system right yeah i
guess so i mean i guess you might want
to
you might want a system that can say
like hey i can't answer this question
or like this i i like no
proof found given
like computational resources i have
something like that
we talk about idea by by referencing
like a
a human level system or an almost human
level system and then we talk about how
that can lead to super intelligence
so i guess one question i'd have is like
what what is ida like today
like what is the state of um of the of
of play and what can you do with ida
systems
that maybe you can't do otherwise oh
yeah i think that's a
that's a great question i mean like i
think that these kinds of systems are
useful now um because fundamentally
they're
i mean this is like the amplification in
it iterated amplification where
you're amplifying the ability of an
existing system that you trained
to do to be able to do slightly more um
so
yeah so in some of my work for example
i've taken
standard question answering systems
which can answer like
questions about a single paragraph in
wikipedia and those are like very good
at answering
a question about a single paragraph on
wikipedia but
they like struggle to answer questions
or like it's not clear how to apply them
to questions where
the answer is like in multiple different
wikipedia articles or different
paragraphs so like one example of a
question
would be like who was born earlier
george
washington or abraham lincoln that
information is in just two articles in
wikipedia
and what i've done in my work is train a
model to first generate
sub questions for that original question
and so you'll get some sub questions
to the effect of like when was george
washington porn
and when was abraham lincoln born and
then you can just answer each of those
individually with your pre-trained model
and then pass both of the answers to the
sub-questions
to another question answering system
along with the sub-questions
and then it should be able to predict
the answer by basically like taking
i guess an argument over the two like
answers to some questions basically
it makes the whole process much easier
and then we show that it like improves
the results on
a like yeah a current like popular
benchmark for question answering
and how do you train it to to identify
sub questions like how
i'm i'm struggling to think of the is it
just like a a label training set is that
what you use
yeah so i think that would be the
simplest thing but there's sort of like
a weird kind of supervision like they're
not that common
to get these sort of like questions
sub-question pairs
yeah like another simple strategy if you
don't have that much supervision i have
a paper coming out which does
this which is to basically like leverage
these kind of like large language models
that can
adapt from a few a few number of
examples to
generalize to doing this so yeah so we
we've basically been using gpt-3
um plus some other tricks to generate
sub questions you sort of like condition
on a question sub question
pair or few sub questions and then you
prompt with a new question it can
it can do pretty good questions sub
question generation
and then there's also like in another
paper that i had
we were basically just trying to see if
we could generate these sub questions
completely without supervision and there
we kind of used some
some of these like crazy methods coming
out from unsupervised
machine translation to do that but uh
yeah i can go into more detail on that
if you're interested but they're they're
sort of like
that's kind of a high level of the
different kinds of methods you could use
yeah no i i find it really interesting
that you're you're looking at effective
effectively
augmenting gpt3 with i don't know if the
the terminology would be like a little
bit of logic
or like a kind of logical structure that
you're baking in
i mean this this kind of brings me to
another question especially as somebody
who's working in this space
with a lot of q a type problems like how
impressed were you with gpt3 when it
came out how surprised were you as well
with with how
how well it could perform i've been
pretty impressed i think
um maybe the places i've been most
impressed are specifically
for like using the model to generate
arguments or generate
advice or like other forms of like long
text generation
it's just like really good especially
on the kinds of questions that most
people ask because there's just probably
the most amount of training data on that
so anything related to like politics
or like political controversies or um
economics economic controversies
apologetics and sort of like religious
debates
um it actually like just does a
reasonable job
at like like i don't know quoting bible
verses or like
other like very like crazy things
because probably there's just a lot of
discussion about those kinds of things
on the internet
and so it's just way better than any
model i've seen at
doing those sorts of things yeah i think
the places maybe where it fails are like
if you want something that's very
precise like if you
if the correct number of outputs is like
one
and like there's not really any other
any other correct outputs that you would
want
like if you're wanting an answer to a
question like you need the exact answer
to a question
it still is um
i mean it's not really using supervision
i think that's basically where
like supervision is is helpful um
is to like get the exact sort of thing
that you want for your task
do you think it has a plausible model of
the world behind it
like to what extent because i think one
of the areas people have been debating a
lot is like
is gpt 3 basically a
good glorified memorization tool where
it's like you know like you said oh i've
seen this before i've seen somebody
have this particular argument about a
bible verse before i know what to quote
versus it's actually sort of connecting
dots at a higher level of abstraction
um where do you where do you see it
falling on that spectrum
yeah that's a good question it is
definitely doing some generalization
like
yeah like a friend and i were playing
around with it recently where he was
kind of skeptical and
i was like oh well what would convince
you that it's doing some sort of
generalization
and he was like well you know it's
probably seen a lot of like recipes on
the internet
and it's also probably seen writing
about unicorns but it's probably never
seen those two
together because i don't know like that
just seems unlikely
so he just like prompted it with having
a like give me a recipe for how to make
like unicorn eye soup
and it and then it just like sort of
goes off into talking about how we need
like rubies and
uh we need to like smelt the iron ore
and like put it into the soup and like
sprinkle it with some
you know gold dust on the top as like a
garnish wow
it was like very detailed and yeah i
don't know that that
sort of seems like a prima facie
like evidence that it's it is doing some
sort of generalization and i think
so to some extent it seems kind of hard
to catch cases where
that just doesn't really do a good job
like
do you expect on that basis that as
gpt-3 is scaled
you know i mean obviously google's just
come out with their like
one trillion parameter model and and we
can expect more scaling to come in the
near future
like do you think scaling is likely to
to bring gpd3 to the point
where the idea strategies that you've
been at least exploring
most recently are kind of subsumed by it
where like
it's good enough on its own to not
require that kind of augmentation or
where that augmentation can then be used
to like
target the next layer of failure for
that that model is is this in other
words
is like scaling gp3 plus ida can that
get us arbitrarily far i guess is what
i'm trying to ask
yeah i think the nice thing about
ida is that it just leverages whatever
ability that you have to get
a model that can do more so i
i think i'm not concerned about like
larger language models subsuming the
need to do idea
because whatever like large language
model you have you can always make it
answer more kinds of questions by
running this sort of the question
decomposition process
um so far i've it does seem
easier to get that sort of
benefit from question decomposition the
better our models are
because yeah like you can just generate
better sub questions if you have a
better language model
like question decompositions from gpd3
do better than question decompositions
from
our previous like sequence to sequence
methods even then like pre-trained ones
which also do better than like
methods that don't use sequence to
sequence learning um there's just like a
clear gradient where like the benefit
from
the subquestions improves yeah so i
think there's like
an increase we'll see like an increasing
large effect from doing this sort of
question decomposition that's
interesting because i would have guessed
like
that at a certain point as these
language models get more and more
sophisticated and they build out like a
more more complex model of the world
that that model of the world would
include the value of question
decomposition
and therefore that essentially like
the the gpgn let's say could optimize
both for question decomposition and for
answering the questions
at the same time in other words you
would you'd kind of benefit from that
that interaction where
if it's all happening under the hood of
the same optimization engine
i think that seems correct um
as long as you have training data for
the amount of decomposition that you
need to do
like if i don't know you like gpt3 might
be trained on some questions where
it needs like a couple or like yeah two
to four sub questions maybe
um and so it might just be doing this
internal decomposition process and
predicting
predicting the answer and who knows you
might be able to even decode
like the sub questions from the internal
weights but i think
the tricky bit is like how do you get
generalization
right questions that require even more
decomposition
than just the ones that people are
generally answering on the internet and
i think that's where this structured
process
is helpful because you can just
arbitrarily decompose the
um questions into sub questions yes so
it's this
what i love about this is it plays with
like this um
i have this mental model of like physics
uh
or maybe you call it physics and logic
and like machine learning
as two two ends of a continuum so
you know in physics what we do is like
we go out into the world we run a bunch
of experiments and we try to
identify like underlying rules that seem
to apply
all the time like no matter what these
rules are always true the speed of light
in any you know in any frame of
reference is always the same
um the gravity works the same way and so
on and so forth
and like the the laws of logic are sort
of similar to that
where you know it it seems like you're
assuming a law of logic here
the law of question decomposition that
you can coherently decompose
complex propositions into simple
propositions and that will always be
true
and so it's almost like you're taking
the sort of trial and error machine
learning kind of like let me feel my way
around the elephant but never actually
like machine learning models don't
actually come up with rules they come up
with like
predictive models which are a little bit
different and a little bit less
fundamental in flavor
it almost feels like you're you're
coupling the two together you're saying
okay
you're really really good at building
this model model of the world
it's a very flexible model but it fails
the moment it encounters something that
hasn't been trained on
so let's augment it with this principle
that we think will be true independent
of context to give it much greater reach
is that like a fair kind of framing or
description
yeah no i think that was that was mostly
it yeah you know that
do you think that um do you think
there's anything that
ida plus gpt3 or gptn i should say
will not be able to do does ida get us
across the super intelligence
agi finish line or is there other stuff
like
self play or reinforcement learning that
will have to be coupled to these systems
yeah i think that's a good question at
least if you are
wanting to answer questions and not
say like take actions in the world um
then i think
a like language model plus question
decomposition
will get you quite far some of my
uncertainties
are around uh like one issue that i've
i've sort of found is that there are
just cascading errors where like
if you answer a sub question incorrectly
then that will that can propagate
as an error into the final answer so if
you answer like
when was george washington born as like
1900
then you're gonna answer that question
like was he born earlier than
abraham lincoln incorrectly um and so i
think
there will need to be like a lot of
thought about like how we
how we do the question decomposition
process like how we use the
answers to sub questions we might just
need to like
do lots of different question
decompositions for any different
question
like we might also want to just directly
ask the question like
maybe there's just some text in
wikipedia that says george washington
was born earlier than abraham lincoln
and so we might just want to ask the
question directly maybe
the birth dates are not available and so
we want to look at like when
when they died and maybe if the the
differences between when they died is
like large enough then we could get some
estimate um that would like let us do
some invasion update on like
when their birth dates were we might
want to sort of like ensemble a bunch of
these different
results of right question decomposition
and also like other things that are
important i think are like
looking at the confidence that the model
that's answering the sub questions has
at um predicting the answer
so like one one thing that i've found is
that when that model
is less confident of its answer it's
likely that
the entire um process will end up with
an incorrect answer
which which i think makes sense but it's
sort of like the kind of thing
that like we should actually like
carefully tune it sort of emerged in the
system that we
built but like it's some sort of thing
that we should be careful about we're
like
oh like maybe we should actually train
models that are better calibrated
and like see how we should use their
confidence to affect our confidence in
the overall prediction
there's like a lot of different parts of
this system that i think do
need to be carefully looked at in order
to get the whole process to work out
properly um yeah yeah it def
and definitely his high stakes
especially i guess as far as those those
earlier questions i i presume are
concerned right like the sooner
the system makes a mistake like the the
sooner the entire
tree gets compromised that comes after
is that
okay this actually brings us to because
you mentioned ensembling so
you know you have a um a whole bunch of
different question decompositions
question decomposition one um leads to
one conclusion question decomposition
two leads to different conclusion and so
on and so forth
and eventually you average out or you
you let them all vote
and you you know use that to ensemble
their predictions you get something
that's more robust
what about the other angle of this which
is getting these systems to
perhaps undergo some kind of debate
process i know that's another thing that
you've been working on
um yeah can you speak to like debate and
its importance in this context
yeah i think this the high level idea is
similar which is
that when we have a question that is
more difficult or complex
or uh some like out of distribution
somehow
we need to break it down recursively
somehow in order to get to a point where
we can't actually
make an assessment from a model that's
trained with supervised learning
the way that debate approach approaches
it is in a different way
where the way that i'll motivate it is
by starting about starting out about
like
how you might approach this problem from
a reinforcement learning perspective so
like let's say you're trying to ask a
model
should i eat keto like give me an answer
with some explanation of like why i
should think that you're
giving me some good answer if you don't
have
that kind of supervision for those
explanations one possible
thing you might want to do is just have
your language model
generate an explanation and then you
just check
the explanation and then you see like oh
uh does this seem reasonable if it seems
reasonable
then an rl type approach would just like
reward the language
model for giving a good explanation and
answer
and like negatively reward if it doesn't
seem correct
um so i think this kind of thing makes a
lot of sense if you think that
the answer is easy to check
um but hard to come up with so like math
proofs might be
an example where like oh there's like a
really complicated reasoning process in
order to generate the whole proof but
like
once we have the proof then it's much
easier to just check the answer
and then we can reward the model for
positively for coming up with the proof
so that that like reinforcement learning
type approach will possibly get us past
human ability because
like we aren't actually generating the
answers we're having the model come up
with
do some expensive computation come up
with the answers and we're checking them
but the failing is that it's like unsafe
or like could
could be misaligned with our values and
that we're basically just optimizing a
model for
generating convincing answers or answers
that look correct
things like things that we will consider
to be convincing yeah
yeah exactly um and so i think that's
where the motivation for
debate comes in is that well we have one
model answer a question
and then we can have the same model sort
of
bring us considerations about the answer
to this question
that we might miss and that are like
very convincing cases about
uh against this answer so maybe it
provides another answer
and so now we have like both the
original answer
as well as like a counter answer and
like
these for like a well like for a good
model might be like two of the best
answers to this
uh question and like there we might
think that okay like now we're like more
informed about
like maybe some missing information that
we might have not not had
um with debate in a similar way as like
iterated amplification you can just
iterate the whole process so then you
can just say like
okay given i have this like answer and
like rebuttal answer
i can like generate a counter counter
argument to that response answer and a
counter counter argument etc etc
until basically we have enough uh
information
for a human to predict with like
reasonably high confidence what the
correct answer is
and then we can use that to like since
we have like reliable human judgments on
this sort of like
um data then we can train reliable
models to make the same prediction just
using supervised learning
and so the the hope here would be that
you get
two ai systems to debate each other and
in the process
they end up revealing the strengths and
weaknesses of each other's arguments in
a human intelligible fashion
so that we we can go oh like you know
this this ai is malicious and it's
trying to fool me
into uh i don't know helping it along in
its plan for world domination
whereas this ai is like is doing the
right thing is is
its answer actually makes sense when i
when i see the debate play out
um does that i mean does that actually
work like
i guess one one thing i'd imagine or one
issue i'd imagine here is like
as these two systems that are debating
vastly exceed
human ability to kind of to follow their
reasoning
you end up in a situation where there's
a decoupling between like you said what
sounds convincing to humans
and what actual logic would dictate and
as the logic gets really complicated
it seems like it would become much much
easier to deceive
a human and much harder to convey the
real
um the real true arguments yeah i think
that is
basically a key uncertainty with this
approach um
like you might think that as the
models that are running these sort of
debates get better
then both the arguments and the counter
arguments get better in a way that
is helpful like in that you're more
likely to catch errors in an argument
but yeah it does seem like there i mean
there are some
interesting experiments like human
experiments that open ai has run where
they have
just smart people who know about physics
like play this process of debate and
just try to have like sort of a
lay like non-physics-y
not too physics-y person try to judge
the debate and it just seems like really
really difficult to
for me to assess some of those debates
so i think it's kind of
another thing where i think the process
needs to be like
tweaks in order to figure out how we can
get the most
easy to judge debates in a way that
we can still feel confident that we're
getting a good reliable
judge of the whole process yeah because
it almost seems like i don't know
naively to me it feels like there's a
cursive dimensionality aspect to this
where the more complicated
the the problem domain becomes the more
sophisticated the reasoning becomes the
more high dimensional it is
the like the the more
spare degrees of freedom these ais have
and it seems like those degrees of
freedom like that's the space that you
need
to to kind of wedge in deception and
it seems like that space just just grows
as the problem
increases in complexity um but
yeah hopefully there's some underlying
principles that we can actually uncover
that would let it scale a little bit
more
it seems plausible that you might need
some combination of these methods
i don't know like train some idea type
system
and then um because it's done this sort
of like
decomposition and then it it's like
above human ability and what sorts of
like
questions it can answer and also like
what sorts of debates it can
judge and then you might want to use
that system as the judge for this debate
process
and so it might be able to do a better
job than just like using a human
annotator as a judge directly with
debate i think
one of the motivations is like this
might be a good training signal if you
can train a model to accurately judge
the results of the debate then you can
just use that as a reward signal
for some sort of like self-play even
like alpha zero type system
where the models like optimize uh the
reward
and get better at generating these
arguments and then as they get better
like you might also think that they
might be better judges
for the whole system there's like
different combinations of
of these like puzzle pieces that we can
try to fit together but
uh it's sort of like an open question
about what's the best way to do it to
get us
reliable judgments well and one thing
that i'm curious about too like
and this informs this informs both the
strategies that researchers choose to
work on
and also like the their attitude towards
safety and the kinds of alignment
and ai safety techniques that they feel
compelled to work on
so there's a tight coupling between
those technical questions and like
thoughts about
ai timelines like when you think
uh an agi human level general
intelligence or something that we could
refer to in those terms is going to
arise do you have like
jeb i don't know do you have a take on
that and and if so has that
has that informed your position on
whether to work on capabilities or
alignment or anything else
yeah i made a prediction on this there's
a really nice thread
i might have to send it to you later but
there's really nice posts
from this group research group where
they like collect a lot of different
people's
ai timelines and i think from that i'll
post a link i did see your realistic
plot
so uh everybody check out the blog post
that will come with the podcast and
you'll be able to
to see ethan's illicit post because i
think it's kind of interesting okay
great yeah
um yeah i think it depends a lot on your
definition of hvi i think i
i was like considering that like it
would both need like
good language modeling abilities or like
language abilities
good vision capabilities and like be
able to take actions in the world
and i think in my prediction i sort of
decouple it into
two cases there's one case where we get
general intelligence from
combining existing components that we
have currently
and and scaling them up with compute
that we have
and then there's another component of my
distribution which is like
in the case that that other scenario
doesn't work out
well it's sort of just this prior over
the next like
century or a couple centuries that like
well not really sure when it's gonna
happen so i'll just have some
like decaying distribution over the
timeline and that
second scenario it seems very uncertain
when
we'll get general intelligence because
it could i like
it could just be possibly some like very
hard breakthrough
that like we've already been working on
the problem for
uh quite a long time and so like it
might just take a really long time to
get the necessary
like breakthrough that we need but in
the other scenario it seems like quite
plausible that
in the near term like if we get a large
larger like gpt
style models if we have enough compute
to get like a good performance
good performance then i think it seems
possible that we get quite good language
models and then we just need to like
combine them with the visual components
which is i think
the direction that like clip and dahle
and some of these other models are going
in
and then we just need to sort of like
give them some ability to take actions
in the worlds or be able to train that
like
somebody would want yeah maybe i i don't
know if i'm like
tied to like physical embodiment but at
least
the models should be able to for example
like run a company
like choose actions that are like
optimizing some goal
and that that seems like it might need a
bit more work
uh beyond just like getting an
understanding of
vision and language i think i think
you've posed this problem
in a way that everyone implicitly has
posed it in their minds but no one's
ever made it explicit quite in the way
that you did or as far as i've seen in
that comment
um the central question is like hey do
we have we solved all the conceptual
problems
that we need to solve basically do we
already basically have all these
ingredients the reinforcement learning
the deep learning we just smush them
together and add a pinch of scale for
flavor
and then we end up with an agi or is
there some like you know fundamental
thing whether it's some weird quantum
effect or
some something we're gonna have to
figure out to make agi actually work
and if i recall you actually said you
thought there was about a
a one in four chance that we currently
have everything we need to
to achieve liftoff is that is that
correct yeah i think
i think that's right i mean it depends a
lot on your definition of
agi like right i mean i think language
models alone
that are very powerful are probably like
quite good enough to get
like yeah i quite like have massive
impacts on the world
um i i said like one in four in that
post but i i think probably like most
people would be lower
than me on that some people would be
higher um oh interesting so you think
most people would um would think that
that there probably are
some missing conceptual ingredients i
think that might be
sort of like an opinion that's getting
worn down by some of openi's
scaling work but like generally it does
seem to be the case that like most
researchers are like
oh um we do need some like big
like new ideas in the way that we're
approaching things
and when i looked at your plot one of
the things that um that struck me was
you had it
essentially the probability of us
hitting agi spiking
sometime in the next like yeah 10 years
or so is basically your your big bump
that's associated with the possibility
that
maybe we already have everything and all
we need is scale so within the next 10
years we'll be able to scale
our way there and then and then of
course you've got this big period of
like high uncertainty where you're like
i don't know if it doesn't happen in
that time scaling doesn't work then
who knows when we'll have the big idea
um are like
does that does that cause concern like
are you worried about that possibility
that we might hit agi
like in the next 10 years do you think
that we're
we're prepared for it from a safety
standpoint from an alignment standpoint
from a philosophy standpoint
yeah that's a good question i think
just like to some extent it seems kind
of hard to work on some of these safety
methods without having
yeah just like good models yeah like it
is just hard to actually like
generate a good sub question to a
question
like before we like it would have been
really hard to do that like a couple
years ago
yeah so like part of me feels like oh
like this these methods are going to
become more useful
as um as we get
to closer to
very powerful language models yeah but
also like
it does seem like i don't know like
maybe it feels like we need like
a period like a long period where we're
like close to
having strong language models and like
have a lot of time to develop our
methods and then
like oh then we'll be ready for it and
we'll like have all these methods for
like
scaling past human ability in a way that
generalizes properly
so yeah maybe maybe it's like more the
the shape of the trajectory that
interesting the i mean
matters to me does and does that like
does that imply though that we would
know
because right now i think we're scaling
the leading
the biggest machine learning model in
the world has been scaling by i think a
factor about 10x every year
so to me this seems to suggest that
we're very likely to
not kind of hum along just below human
level intelligence while we sort out a
solution but like rather
you know shatter right through that
threshold and create something way
bigger than we can handle
is that like is that a plausible concern
or is
is that something that uh that you think
is is obviated by something else
well i think it will be pretty compute
limited if in in the case that we get
methods that
are like generally intelligent from our
current
set of methods that it looks like the
direction that that's going is like we
scale
we require a lot of scaling and so in
that case it's
there's going to be some trade-off
between like the quality of the model
that you're getting and the amount of
compute that you're using
using it for so i think that sort of
does limit the amount of yeah i guess
the amount by which you can sort of like
really increase
quickly in terms of intelligence yeah it
seems like
i mean we definitely need a few more
orders of magnitude of compute to get
even like near human level intelligence
at that point it's like you're using a
large budget
alleged fraction of like google's r d
budget in order to train the model
it's kind of a question of like well how
much more money is there to spend on
these and like it seems like a process
that's gonna like take a while
like to get to get at least like some
amount of time to like
maybe maybe it's even gonna have to be
like some sort of like governmental
project to like
gather more than google r d budget
amounts of money to like scale the
models but it does seem hard to like
just blast past
and like we also have these like very
predictable like power lock curves in
terms of like
how how is loss for language modeling
gonna go down as we increase with
compute
and that is like the amount of gain that
we're getting
is slowing down in terms of how much
better we're getting at language
modeling from training with more compute
and so i think those those factors all
kind of suggest to me that things will
like slow down a bit
well and i guess if they do that would
suggest that strategies like ida
are are probably going to be you know
all the more valuable as we look for
that little extra nudge here
a little extra nudge there is ida
going to be um going to be necessary
to to get across the agi threshold
like or are there plausibly other
techniques that
like is ida on its own as a plausible
way in other words of of getting us to
superhuman levels of performance from
systems that
are trained on human level data yeah
okay so
like the characterization in my head is
something like
a a very good language model will
capture the full distribution
of human intelligence and ability
it has the knowledge about what the very
upper end of human intelligence is like
um so like for example like
uh i i've prompted gpd3 as as if it's
like a conversation between
two faculty at stanford and then the
conversation is very distinct and that's
where they're like sort of
citing different like famous people and
like very high quality
like well-formed english and stuff like
a very sophisticated conversation yeah
exactly
and there i think like there it's
clearly
i mean that seems like i mean it's above
the mean of
like human intelligence in terms of
quality so i think that's like that
seems like a reasonable place that like
i mean maybe you might just consider
that like superhuman already
if like depending on what you consider
right
right is is picking like the the top
yeah
one percentile of the human population
is that a superhuman thing i mean maybe
yeah
yeah i mean that would already be quite
useful um
then i think maybe if you use some sort
of like reinforcement based approach
like the one i was describing
you could get beyond that but i think
you would start to see some of the
failings that i described where
models are generating answers that are
convincing but not necessarily
necessarily correct um and there i think
you get into dangers about like
well people are probably going to use
the answers because they're like
probably correct but they're also like
optimized to translate and convince you
and therefore you might be taking
important decisions based off of like
incorrect
information so i think that that is like
the regime where like when we try to
scale past that level of human ability
then we might want to like
we might basically like need to use
distillation amplification or
debate kind of methods to like get
us past human this regime of human
ability in a way that like we
are like confident will work and are you
overall optimistic when it comes to
because
we've talked about ida's potential as a
way of
of amplifying these systems of getting
super human level performance
it seems like we do have strategies that
at least in principle
seem like they have a decent shot of
getting us to super super intelligence
and whether one of those strategies
works or something else works
it seems quite likely that we will
eventually get there like are you
optimistic about our ability to control
these systems and
and well maybe wielding them wisely is
like a separate question but
just our ability to control them to make
sure that um that we don't let a genie
out of the bottle that we'd rather
put back into the bottle yeah i think
that
that's one thing that's nice about um
like question answering systems is that
they're we we sort of decide what we do
with the information it's not that we're
having systems
act autonomously on the own which is on
their own which i think is where a lot
of the
the difficulties come in um
i mean there are there are still
difficulties with doing this like
question answering problem
correctly but i think there if
if we can like really nail this
i think like simpler problem of just
getting models to generalize past human
ability in just the domain of like
prediction
like answering questions then i think we
can use some of those methods to help us
in the more general case where we want
our systems to just autonomously take
actions
in a way that's like not supervised by
humans um
and for example like i think one way to
like make that connection is to use this
sort of like
question answering system trained um
with idea or debate
to provide a reward for systems that are
i don't know like running manufacturing
uh
like amazon manufacturing or like making
like
ceo level decisions at a like
tech company or something like that i
think
there we if we have some sort of
superhuman question answering system
it might be much easier to get accurate
evaluations
of these systems that are taking very
long-term actions
interesting okay well hopefully uh
question answering systems are going to
be a big part of the solution here
really appreciate your your time and
your your thoughts on this ethan and
do you have um a personal website that
you want to point people to actually see
if they're interested in following your
work
um yeah it's just ethanperez.net
awesome.net awesome i like it
very uh what is it i don't want to say
hipsterish but it's like it's got that
it's got that nice aesthetic yeah great
that's good it was the only one
available that was nice nice
welcome to 2021 i guess yeah well uh
yeah thanks so much really appreciate it
yeah thanks for having me