Jan Leike - AI alignment at OpenAI

[Music]
hello and welcome back everybody to
another episode of the tourist data
science podcast and a very special
episode of that because we'll be talking
to jan lyka who is formerly at deepmind
and now is the head of open ai's ai
alignment team where he's leading a
group of researchers working to make
sure that some of the world's largest ai
systems behave in a way that's safe and
reflects human values now jan has a
really interesting way of thinking about
ai alignment research that prioritizes
experimentation with live systems over
more theoretical approaches and we'll be
talking about why he holds that
philosophy what specific strategies he
thinks are most promising for aligning
powerful ai systems and more generally
what challenges lie ahead for ai safety
beyond the alignment problem now it's a
rare treat to be able to talk to
somebody on the front lines of frontier
ai research like this and it was a fun
conversation that i hope you'll enjoy as
much as i did so with that said i'm
going to step out of the way and let the
episode begin
[Music]
uh hi john thanks so much for joining me
for the podcast
thanks jeremy i'm very excited to be
here i'm super excited to have you
you're the head of alignment at open ai
i just we just had to do a retake i
called you the head of safety opening i
don't know what i was thinking but you
are the head of alignment at open ai
obviously a very interesting and and
rich role with all kinds of problems
you're tackling day to day we're going
to dive into a lot of those but i want
to start by just asking you a general
kind of um
life stage and career progression
question i think a lot of people are
going to be curious about this how did
you get into technical ai safety
research and alignment research and and
why did you orient yourself in that
direction
oh gosh i've been in the field for
almost 10 years now i kind of got
injured back in like 2012 2013
and basically by reading a lot of stuff
online so i read like i love elijah
kelsey's writing and you know other
people's
and
um that convinced me that you know like
you know we should you know take the
prospect of agi seriously we should
really think about you know like
long-term impacts of uh of ai on society
and there's like real work to be done
here and currently there's not that many
people doing it i mean that was also
back in you know 2012. 2013.
and uh and at the same time i was i was
like wrapping up a master's degree i was
working on i wasn't even doing any i i
was doing soft verification
i was like
didn't know anything about it
and so i decided that this is something
i wanted to do with my career and so i
switched field i did a phd
in reinforcement learning i'm curious
like was there any particular argument
that you remember moving you in that
direction where you went oh wow you know
this really is something i should think
about like as a career trajectory or
something i should try to move the
needle on um i don't know if i remember
any particular writing piece um
[Music]
i think it's like it was like a lot of
prompts that got me thinking about the
topic and you know like
you know the stakes and why you know
like that would be a very important
thing to do yeah it was just like the
arguments made a lot of sense to me what
do you think makes the stakes so high
because i think everybody's got their
own sort of internal story about that
and like what is important about
intelligence general intelligence and so
on what do you think makes it the thing
that um that you want to spend your
career on
yeah and like i mean if you
if you think about this right like
um in a way
what agi is about is like the holy grail
of
ai research or maybe even computer
science research or maybe research in
general yeah like you build any like a
system or a machine that can do
most or all things that humans can do or
like that are economically valuable
and if
you know if you succeed at that um then
that will like you know
change society quite uh substantially in
like various ways and like some of that
can be really great right like there's
lots of things that nobody wants to do
and they're like not that fun to do
and if the machine was to do that then
that would be great and there's other
things you know
uh that which can cause a lot of
problems right like when if you displace
lots of people at their job and now they
don't have an income that's like very
pragmatic
um and on the other hand you know like
if we hand on more and more control and
um
you know like decision making to
machines as a society then we those
machines better make the decisions in a
way that are aligned with our values and
that actually
you know is what we want the specific
way that you approach that is also kind
of interesting right because i think
there are so many different schools of
thought not only on why the alignment
problem is important why agi might be
important but also like specifically how
these problems should be tackled
obviously the alignment community is
pretty big there are a whole bunch of
people working in silos and not silos
and doing all kinds of different
interesting stuff one of the areas
you've decided to focus on the most is
the sort of idea of scalability of
alignment solutions that's a theme that
i've seen come up quite a few times in
the context of your work and research
i'd love to get your your insights on so
what does scalability mean to you in the
context of ai alignment and why is why
is it so important yeah um i mean that's
a straightforward crash light story
we're just like well
we expect or i expect ai capabilities
will keep advancing and uh we've already
seen a lot of scale uh returns to scale
and machine learning um and i expect we
will continue to do so
um but of course
like there's substantial uncertainty in
disagreement on like how fast things go
will go um but i think the more i'm like
interesting argument is
that like this
you know in the future when we are at
the point where like ai permeates
society even more than it does today
um the stakes regarding alignment at
higher right this is what i was just
trying to say
where
um
you know like if there's like lots of
automated decision making
everywhere in society like if
the like the more that is the case
the more we would be invested and care
about that those decisions are made
in an aligned way and
in order to and that's like kind of like
the problem that they care about and so
in order to kind of like prepare for
this problem
uh today i'm particularly interested in
scalable solutions in the sense that
um they will keep working
forming a capable system so we can use
them in the future
and i imagine a devil's advocate
position on this too would be like well
you know if you think that
um
that alignment will be important for
capabilities in the future if you think
that people will just need to focus more
on alignment in order to get value out
of their ais then you know why prepare
for this today why is it that we need to
start working on scalable
solutions today and not just sort of
like allow capabilities and alignment to
evolve together do you have any thoughts
on that aspect
i mean in particular you don't want to
be at a spot where like you know you
worked on alignment for a while and then
like you know
like capabilities hit a point where like
your solution stops working and you have
to go back to the ground bar but there's
no guarantee that like you know people
will just stop pushing capabilities
and so
then you're at the point where like the
system is much more capable but you
can't align it yet and that's
you know that's the kind of thing that
we want to avoid and that's why we put
such an emphasis on scalability
yeah that no that makes sense and i
guess a lot of this too is also
entangled with views about when agi is
likely to emerge like i imagine for
example if you thought that
we were likely only ever to hit agi in
like 100 years you wouldn't be keen to
place a long bet on like specific
architectures or specific alignment
solutions today because like things will
probably go into completely different
direction
can you speak a little bit to that like
what are your thoughts on timelines
toward to agi development and then how
does that inform your thinking about
which alignment solutions to emphasize
yeah i don't really want to get too much
into timelines because like uh i think
it's like a whole another topic
um
and but
i think the
there's like a really good argument you
can make here which is like you know
what
you could ask yourself what if like it
happens faster than we think right and
um that's like if you want to be
prepared for that world
then
you want to do alignment research in a
way that can like deal with very
aggressive timelines even if the if you
actually think those are pretty unlikely
and so kind of like
and
if if you think about it this way you
want to be ready to deliver kind of like
alignment solutions on a timeline that
are actually um
where it's like actually pretty unlikely
that it's actually necessary at that
point
but of course
you know like we get more evidence about
this as we go along and we can kind of
like adjust and
um hopefully adapt our strategy if we
if it does actually end up going faster
than we thought and so how does that
affect the the solutions that like
you've chosen to emphasize like it
sounds like you're definitely focused on
you know hedging against that that risk
whatever the probability might be that
we might hit agi sooner than you know
most people might expect
what kinds of decisions do you make as
an alignment researcher especially as a
manager of alignment researchers when
you decide like okay we should therefore
focus if we want to hedging is this
specific risk we should focus on these
strategies rather than these strategies
for example um i mean i don't think it's
like i'm
like hedging
against specific risks or it's not like
we're placing a particularly high bat on
like
you know
uh things
working like going particularly fast but
um
like the perspective that i'm coming
from is
um
kind of like the question of like what
if
the current set of trans technologies
will actually end up scaling up to
something that looks a lot like agi and
like how do we deal with that so in
particular like that means
deep learning
like continuing to scale um but it
doesn't necessarily mean you know like
i'm assuming like whatever we built in
the future will be a transformer like
maybe
but like maybe not and i kind of want to
have a solution
uh and like that's the general like the
kind of like the scope of technology
solutions that we're playing that we're
like
uh working on right now is kind of like
assuming that we have some kind of like
deep learning model although that's not
necessarily a strong assumption
um and but we don't we're like pretty
agnostic what exactly that would look
like and how
you know like what the latest hacks are
for training it and so on maybe this is
a good segue into the specific like
solution that that you've been working
on a lot and reward modeling and then
recursive reward modeling um so could
you explain actually reward modeling
just um for starters especially for
people who are listening who might not
be super familiar with like
reinforcement learning i think the idea
of reward modeling and how it fits in
that picture might be an interesting
thing to uh to explore
yeah i would love to um reward modeling
is a really cool thing to talk about
because it's like one of our staple
techniques for doing alignment today and
i expect it will be a very important
building block for
like future alignment solutions
and
um if you think about yeah if you think
about reinforcement learning right like
in reinforcement learning you have an
agent that sequentially interacts with
with the environment like every time
step it takes an action
uh it the environment returns an
observation and a reward
um and then you know like the agent
um is meant to
like
optimize its actions such that over the
long run
like it gets
good rewards or sums of rewards or
average rewards or whatever criteria you
want to
uh give
um and
so one crucial assumption that is like
typically made in reinforcement learning
is
that the reward signal is provided
by the environment it is the ground
truth so as long as you optimize that
reward signal well
then you know the agent will solve the
problem
and so this is true for a lot of the
problems that the ro community has
historically studied right so this is
true like in atari games you can just
look at how the score changes and like
that's your what signal it's true in
like starcraft or dota because you just
run the game and you see who won and now
like that's that's your award
but there's lots of important problems
but that is not true so
um if you think about like you know what
if you
want to write
uh you want a system that writes you a
fiction novel on the topic that you like
so now what is the reward signal here
there's like no like the environment is
making you like you know you
punch keys on the keyboard
um
but uh then the reward has to be like
you read the novel and you like it or
not
but if you look at how we train our all
systems
it would take probably like millions or
billions of like books it's generating
until
you actually find something you vaguely
like
and you're not going to be able to read
all that there's just no way
so what reward modeling
is is like basically a general purpose
technique
to solve problems that we don't have a
procedural reward for
and the way it works is
uh it's like very similar to
reinforcement netting but instead of
having a reward signal
and that is coming from the environment
we get it from a human
and so in particular we have like
uh we have the agent just like generate
a bunch of samples or like episodes or
basically just like doing some stuff
and then the human looks at the stuff
that the agent has done and then they
rank it or they label it like this was
good stuff and this was bad stuff
and so that creates a data set and then
you just train a model on that data set
and
uh the model predicts you know the model
essentially understands what is it that
the human wants like what does good
behavior look like
and important here is like
the model doesn't
or even have to understand how to get
good behavior
so for example let's say you want to
train
a similar robot to do a backflip right
like i can't do a backflip yeah i don't
like i can't write a program that you
know does a nice backflip
but to understand what a backflip looks
like is like easier in a sense than
actually doing one
and so that's the task that the reward
model does the road model understands
the goal of what you wanted to do
and then
then
the rl agent is optimizing against that
reward model so the agent is trying to
find you know like
behaviors that the word model things are
good and then thus
by proxy
uh you think are good um and you know
like a very nice side effect of all this
is that we can actually use this
technique
to train the agent to do stuff that we
ourselves don't know how to do well
as long as
we can tell whether the agent is doing a
good job so would it be fair to say that
essentially
what this system does is it allows you
to turn like a generation problem into a
discrimination problem like instead of
having to you know be the generative
model yourself perform a backflip you
get to just be lazy and be like that's a
good backflip that's not a good backflip
that type of thing that's right that's
exactly right that's a good way of
thinking about it
interesting okay and so this i guess one
advantage i could see with this
immediately is like it allows you to um
to scale the amount of feedback that a
human being can effectively give and
like you said to
to comment or give feedback about things
that otherwise you just couldn't give
feedback about in an easy way at least
um this also tees up like your specific
niche area of research that
seems actually really exciting and i've
been super keen to dive into this one
recursive reward modeling um can you
explain like so where is the recursion
how does it play into into reward
modeling what does it allow us to do
yeah i would love to so basically
if you think about like the class of
problems that you can solve with
reward modeling right these are
like the techniques we just discussed
these are problems where the human looks
at what's going on and they say like
this is good or bad
but if we put
since we put this emphasis on
scalability we're particularly
interested
in you know what's going to happen after
that so eventually you're going to get
to your point where now you want to
train the system to do something
um
that is actually pretty difficult for
you to tell whether it's
done well right so if you remember the
example with like the fiction novel you
want the system to write
a fiction novel that you like
but reading entire book takes you a long
time
so how do you get um
how do you get the system
how do you train the system in this
setting um
and
um the general idea or like the core
idea of recursive reward modeling is um
we
it's actually it's like very simple it's
like all we do is like we train machine
learning models to help us evaluate
the task
so in this case you know like what is
going to help you evaluate whether or
not the
that was like a book you like well
you will have let's say you have a model
that summarizes the book for you or like
it summarizes the plot or like you have
a model that kind of describes the
characters and the character development
um
and uh
maybe you have a model that like can
answer your questions about the book
and if you can just like
imagine having a whole bunch of these
like
uh evaluation helpers
then you could leverage them to like
quickly
uh give some judgment of
like whether this is going to be a good
book that you like so why is it called
request and reward modeling um so the
key piece here is that
because reward modeling is such a
general purpose building block we're
just going to use it again
so for each of these systems
um we
like train them with recursive reward
modeling or just road modeling right
it's if it's an easy enough task we just
know how to do that task you can strain
it and um so for example if you like you
know the
one of the evaluation sub tasks
is the task of you know answering
questions for the about the book for
example
and so what you do is you like just take
that task you train a separate model and
you're like okay so now i'm just
training a model to get really good at
answering questions about longer pieces
of text
and
if you compare that with the tasks that
we started with which is writing a whole
fiction novel this is now a way easier
task right and it's like a more
it's also a more narrow task because you
don't have to think about like world
building and complicated plots and
characters all you have to do is like
answer factual questions
and so
the general kind of like
uh aim with recursive reward modeling or
like the some of the hypotheses it
relies on is
that
for a lot of kind of like
valuable tasks that we want to train ml
models on
and we can actually bring that break
down the evaluation of those tasks
into simpler tasks
and that's like you know
you know or you know if you look at it
from a different way right like by
training machine learning models on
simpler tasks
we can then work our way up and train
how like let them help us evaluate
harder tasks and then like solve those
and you know work our way up from theirs
and
like one really important aspect here is
like
you have
a human in the loop in like every every
one of those tasks but like a human
always
defines what it means to do well in the
task and that's like
so that the resulting model is like
aligned with
with them
and would this because the human would
be at the um at the lowest level right
the the lowest level tasks and then
would basically train these evaluators
and the evaluators i guess would
evaluate the evaluators at the next
level of the recursion is that together
with a human together together with the
human okay so the evaluators help the
human evaluate the next level task
okay interesting so
so the human is always part of every
every level of this process that's right
like they have to they give the value
input right yeah
they only know what the task like what
being aligned in the task really means
i see so at no point are you actually
outsourcing like the responsibility of
evaluating the task to
another ai
it's kind of like it's almost like these
these helpers are like doing some kind
of dimensionality reduction on a super
complex task and then presenting it to
the human being like here you go like
this stupid human you can understand
this is it good
that's right and like oh if you think
about like the fiction novel that you
like right there's no way for the ai
system to really know what it is that
you like at this on on this day right
and
you have to say that and like oh like
what we want
the models to do is like we want them
to help you
communicate that most effectively
and do you see the scaling up to like um
increasingly general systems like can
this go arbitrarily far do you think um
i think there's probably gonna be
tasks that we can't solve with it
um
so one example i like to give is like
writing a novel like a book about a
novel ethical theory is like something
that would be very hard because there's
like no way to like break down whether
like the ethical insights are good
without just like appealing to human
intuition and then like in the end the
humanist has to look at it and like make
a judgment right
and
um i don't know if that's true for those
particular examples but there might be
examples like that
but i think
so
i guess like the
ambition of this project is
that for most economically valuable
tasks
we can actually
break it down in that way
um or actually more narrowly that's not
even what we need to aim for um
i think what we like want to aim for is
something that's more narrower that i'm
i'm kind of like that's more like an
alignment mvp
and
the alignment mvp is kind of or the idea
for mmp is like um can you build a
system
that
is
let's say at least as good
as
at iiai research as the best human ai
researchers
it is also very aligned with human
values
so if you have these two properties and
you can like you know you have
uh very expensive evidence that these
two properties hold
then you could you know like
have that system take over more and more
of the ai research and alignment
research work
and thus
over time you know like it will like
carry much more of the load and it will
like solve
like
harder alignment problems for things
that we like
can't do with requested reward modeling
or we don't really know how to approach
do you think like
if there's a small misalignment because
i'd imagine there's always going to be
some
some small amount of misalignment
because you just have a limited density
of feedback you can offer to the system
you can't you can't tell it how you feel
about every possible scenario so there's
always going to be like a little bit of
misalignment in that system um
i guess
i guess what i'm wondering is like does
that misalignment get amplified as you
as you use that system to build another
more advanced system like
or or does it or maybe does it reduce i
have no idea what i'm talking about here
but maybe that's that's part of the
challenge yeah i think this is actually
one of the really key questions and
that's like one of the key questions
that you want to study
by building prototypes for these systems
um and like
in particular i mean as you say right
like there's no way for the human to be
in the loop on like every decision
that's getting made
um
and like but even more you know like if
you train let's say a one model um to
like replace the human and like
giving oversight to the systems you're
training like the reward model isn't
going to be perfect whenever you train a
machine learning model right like it has
a certain accuracy
um and then like you know if you now
picture like a recursive one modeling
tree where you have like like some
emulation helper systems that are
trained with this way they have a
certain accuracy and then like they help
you train another set of systems and you
just like build this you know giant tree
um
like
the situation that we don't want is that
like at the root node there's just like
so much accumulated error that it's just
not
dotted line anymore
and so this is one of the key challenges
that we're
trying to figure out how to deal with
and like how would you i guess it's very
early days for this but i'm curious if
you have any thoughts about how you
would like how you would begin to
explore that i guess you'd need a way to
measure degree of misalignment somehow
which is is there a way to do that you
can like do
like approaches with the usual
techniques that we have in machine
learning right you can measure how like
accurate each component is on what it's
doing using test sets and like or you
know more complicated things if if you
don't have an iid setup
um but ultimately you know like i don't
think we have a great theory of how do
you think about
uh you know like how these errors
propagate
um
and
ultimately i think well you know
the solution to this might look like
fairly like
simple if you just like you know you
have some kind of error correction
mechanism that works across different
levels of the tree um but this is like
all
it might also be like very more
difficult or it might be that this is
like what you know ends up breaking the
system
um but this is like very much an open
research challenge right like this is
what we want to figure out how to do and
our approach is
we want to build
prototypes of these systems
and then study them
and do you build them at like small
scale because i imagine you have you
have gpt3 available and you'll continue
to have larger and larger models
presumably so
would you imagine testing it on them or
do would you start by testing it on
let's say a smaller model that's maybe
easier to work with or like do you have
a
thought about that strategy yet yeah so
i think it's like it's not crazy to try
this on like uh like toy setting
um in a way the thing i'm most
interested in is like try it on a
setting that is somehow real like in a
way that like you're dealing with real
data you're dealing with real problem
and that way you know like
it you can be more sure that you're not
just like
sweeping some important problems under
the drug right
and concretely like so one project that
we're working on right now is uh on uh
summarizing a book uh book summarization
so this is a problem a project that
actually started before i joined openmei
even
um but we're kind of at the stage where
like we have a system
that can summarize entire books
and
not super well to be clear but you know
like
uh it can do like an okay job
and the way the system works is um
kind of analogous to
requesting one modeling but in a like
more restricted sense
and so
uh
like the way the model works is just
like we
have the model summarize like a few
pages at a time
and then you look at all of those
summaries and then you like have the
model summarize
a bunch of those summaries at a time and
just like keep recursively summarizing
until you have only one summary left and
that's your book summary
and so
in this case it's like it's like simpler
because you we actually train the same
model on like the entire tree because
like it's always the same task you just
like take a longer text and you make it
text
um
but
um the interesting thing here like with
the analogy is that
you could think of this as
like when they when the model writes a
summary of the book
um it would be it would take a long time
for the human to read the entire book
and then tell you whether that was a
good summary
um but if the human gets to look at all
of the test levels
right yeah the level below the chapter
summaries
um
or it doesn't exactly come correspond
the detector
but like you can think of it like the
way
so they look at the chapter summaries
and then look at the overall summary and
that actually makes it like so much
faster for them to like say oh okay
um this was a good book summary of
course
what are you assuming is that the
chapter summaries were good
right and they generally like you know
they're not very good they're like okay
and so
there's like a limit of like how good
you could even make that summary
but um if you can picture if you're a
picture of this like in the infinite
limit right where we do really well at
every part of the tree
then you would end up with a pre that
you should end up with a pretty decent
book summary
there's of course caveats copies of that
as well it's still interesting though
that you are taking such a hands-on
approach because i think these kinds of
problems are at least to my reading of
like the alignment literature there's a
lot of stuff that's less
experimental that's less explicitly kind
of tinkering with existing systems and
trying to get them to do stuff and
somewhat more hand-wavy um and this is a
theme i wanted to ask you about as well
like
what what's your view on that balance
between experimentation and theory in
the alignment community i'm sure
everybody has a different take on this
but i'm curious about almost your
aesthetic preference should the
community be focusing more on on
experiments than it currently is what do
you see as the value of theory going
forward maybe what kind of theory
research is most interesting to you that
sort of thing i think that's really good
question i think there's like you know
the there's like i think the alignment
community is pretty diverse and people
have like they kind of like come at the
problem with like a range of different
tools
and there's like
uh some people like
you know come from more philosophical
angle or social science and some people
like want to
address it with like formal math
uh and then there's like the more
empirical research
that uh you know my team and i are doing
and um
i also like yeah i should be honest i
would like i was like very firmly in the
math camp for like when i started out
that was like back in 2014
um because i just like didn't have a
better plan of what to do
um
i think there's value in all these
approaches
i think like if you kind of like just
look around on the internet of what
people are doing
i'm a bit worried that like you know
there's
there is a lot of vague stuff that
people are doing and like i think the
problem with that is like it's really
hard to really build on vague things
yeah and i personally find that very
difficult and
i think like as a kind of community
to make more progress
we have to move more towards the like
you know
formal slash
empirical stuff
and then
that makes it easier to build on
yeah i i remember um
at least for me one one of the things
that really uh struck me in this respect
was a conversation that i saw on i don't
think it was the alignment form i think
it was just like less wrong but people
were talking about wire heading hey
everyone jeremy here i just wanted to
jump in and interrupt myself because i
don't think wireheading is actually that
widely understood of a concept outside
of ai alignment research and i figured i
should probably add a quick explanation
here so wireheading is one way that
alignment researchers worry advanced ai
systems might fail to work as intended
the idea is that if we design an ai to
optimize some reward metric like points
in a video game for example it might
learn that rather than mastering the
task we're actually trying to train it
for it can just tamper with its reward
metric directly so to take the video
game example a sufficiently general and
capable ai that we're training to play a
game might realize that it can just hack
the game itself to make its score
counter go up potentially higher than it
could even theoretically go according to
the original rules of the game now
wireheading is a much more general
concept than this and it can take on far
more diverse forms than i've just
described for example a powerful ai
that's charged with maintaining an
optimal temperature and some office
building could decide that manipulating
the reading on its thermometer is easier
than dynamically heating or cooling
rooms the bottom line is that
wireheading is likely to be an important
class of ai failures that have safety
implications no one's quite figured out
yet the one one of the things that
really uh struck me in this respect was
a conversation that i saw on i don't
think it was the alignment form i think
it was just like less wrong but people
were talking about wireheading and there
were i guess in the context of that post
like there are a whole bunch of
different ways in which the system could
fail that we're sort of listed and i
found that that aesthetic comes up a
fair bit actually in the alignment
community where people like list a whole
bunch of different problems that they
could imagine happening and this is like
i think this is actually quite useful to
some degree because it has revealed new
problems where you go oh wow we should
worry about this kind of behavior but it
almost seems like
a theory that doesn't uh point to a
common origin of all these problems that
can't point to like a latent uh a latent
source for these issues and then how to
address that latent source it almost
feels like you're playing whack-a-mole
with a whole bunch of different problems
that pop up in different ways like i
guess first off i'm curious if you agree
with that and second
whether you think the empirical approach
has a good shot at sort of covering that
base
yeah
i think it's really good question
because it's the question of like how do
you know you're really making progress
right
um
and i think the
wire heading problem or like you know
some people call it reward tampering um
is a really interesting example because
it's something that
you can't really study empirically yet
because our systems are just not smart
enough to do it yeah
um and you could i mean you could like
make it really easy for them and then
like they would probably figure it out
but like that's not as interesting to
study
like the scenario where like what
happening gets really interesting is
like when the system is actually smarter
than you
and that's not the case yet
um and so
absent
of empirical experiments that you can
run at that now
of course you need to turn to you know
like
more theoretical
approaches um
but you know what i
my kind of like perspective on this is
that you know like
what we need to get to on that front is
like
be in a space
where the like
that works then informs the empirical
empiric experiments that we do
once that
you know is possible and actually that
scale question is an interesting one too
because like i guess there's there's a
some exclusivity to access to large the
kinds of large models that would allow
us to experiment with you know when when
we hit something like agi this sort of
wire heading or whatever else um like
what are your thoughts on access to
these models for the purpose of like
safety like independent safety
researchers obviously there's there's a
narrative that oh no like open ai and
deep mind and so on are going to
monopolize all the compute resources no
one will have access i mean i don't
think that's that's really in spirit
what's going on here i mean there are
practical questions around like big
models yeah are just expensive there's a
natural kind of
moat that gets formed through no fault
of anyone's but do you think that there
are going to be good ways for people to
explore kind of reaching beyond the
scale that's immediately accessible to
them maybe using theory to bridge the
gap or something like that i i mean this
is a really good question it's like
something that comes up and again again
because like
the size and the spending on machine
learning projects have been like
steadily increasing
and
you know like the budgets of academia
obviously don't grow exponentially with
it right and so like naturally you know
like some that exclude some people from
you know like access to these things
um
i know openai gives like api access to
like academics and safety researchers um
and you know like we want to enable
people to still study models like
state-of-the-art models
um obviously you know like
we can't just put it up on the internet
because
like we actually getting to the scale
where you know you could do harmful
things with the model if and then
you know if you put stuff on the open
internet
anyone can use it for anything
um
but
i think also
in a way
this divide is going to get worse of
systems and like ml training spending
keeps growing
and
um
in some ways you know like
i think this is the reason why it is
like like
of great advantage if you can do
alignment research
at like one of the cutting edge edge
elements and like you can be where the
state of the art is and like work with
it directly
um
on the other hand i think also like you
know there's lots of kind of aspects of
the problem that you can study in like a
much smaller setting
and
they'll be valuable to make progress on
that as well
so i don't think it's going to be the
case that you know like if you if you're
just in academia or something that you
won't be able to do anything
you'll have to be a little bit more
hacky and actually um maybe a shout out
to andy jones in order we had him on the
podcast a little earlier and he was
talking about essentially this idea of
doing experiments small scales that try
to kind of project out you know
increasing the size of the system and
saying okay you know here's roughly
where we think things would go and and
open
openai's work on uh scaling laws for
language models and other kinds of
scaling laws really does help people out
in that respect because you're able to
just like draw straight lines and make
inferences that sort of thing
yeah i think that's very cool um and uh
i think that's like a really good point
i think like
andy jones's work is also a good example
of that where like his like he got us
like a really clear kind of like story
in this like you know
smaller setting and then
uh i think like one thing to be cautious
of though is like if you're looking at
like a scaling lower or a trend right
and you did like some small experiments
there's like no good a tree that this
trend is gonna
hold over like many orders of magnitude
right
but you know
when you're looking at these scaling
laws
then you need to start with a small
setting because that's where you can
iterate quickly and that's where it's
like cheap to run lots of experiments
and this is what makes this methodology
so powerful because you know you get
to run cheap experiments that tell you a
lot about expensive experiments
yeah and they tend to give like i guess
they tend to give safety researchers
more reach than like than the then
people who are focused more on
capabilities as well in some ways which
i know i'm maybe i'm in the minority
here but i actually
tend to think that the consolidation of
compute is not the worst thing in the
world the consolidation of those
resources from like a public safety
standpoint because
i'm not sure i want to live in a world
where like every individual person has
access to their own gpt three or even
gpt well g2 seems fine but you know like
we're gonna discover presumably a whole
bunch of different malicious
applications of these things that like
might take us by surprise if we're not
careful um like i don't know if you have
a view on this i mean obviously open ai
one way or another is gonna end up doing
what it is because economic forces push
things in a certain direction but like
is there an argument that there's
that it's actually better that there not
be full kind of democratization of scale
democratization of these kinds of um ais
yeah i mean this is like a big kind of
like um economy in a way like on the one
hand
you know like you want like
democratization of the technology is
good because it gives more people access
you can like like uh applies more checks
and balances to like the ball resource
actors
and it allows it makes it easier for
other people to catch up
uh and on the other hand you know like
it also gives bad actors
access to powerful tools and so how do
you like bridge how do you solve this
problem uh that's like kind of
seems very difficult to me and like you
could draw for example
an analogy right like what if like
refined uranium was like freely
available like to anyone like yeah that
would probably cause a lot of problems
um but i think this analogy is flawed
because
you know like
refined uranium is like
like not a very dual use thing right
like yeah yeah arguably you could like
use it to generate like power
but nobody's gonna do that in that
backyard this is like not really
feasible and so
it's not actually you know like
gonna give people a big benefit of
having like free access to uranium but
the story is very different with ai
because
if like ai is like a very kind of like
general purpose
uh technology that is just gonna like
help like so many aspects of your life
and so kind of like restricting that
kind of technology
is uh also very like in a way
problematic right yeah so how do we
solve that problem i don't know
yeah well no i i totally agree because
there's it's also this really fuzzy
barrier too where i guess what we'd love
to do is figure out this very nuanced
decision surface where like these people
can access this kind of ai and these
people but at the end of the day that
relies on our ability to kind of
extrapolate and guess what sorts of
malicious uses like people could put
these things to and my guess is we're
going to be surprised like people talk a
lot about you know if gpg3 were widely
available maybe you'd have
opening eyes published a lot of stuff
about this too like
you know
influence operations with elections or
you might have phishing attacks that
sort of thing i'm sure there are other
things too that a really creative like
cabal of of criminal minds could come up
with but like it's so hard to just think
about that because there's limited time
and you're just focused on building the
capabilities of more systems right
and also like the tech moves forward
very quickly and right like often we
don't really
know it advanced what it's going to be
used for right or what it could be
before and i guess interactions between
technologies too because like you know
gbt3 on its own today not a big deal but
then you couple it to
i don't even know what you might couple
it to but you know
deep deep fakes plus something else plus
something else and pretty soon you have
like this high dimensional space of
different technologies that can do more
things um and i guess the one other
reason that i'm kind of like i favor
not having this kind of broadly
distributed stuff
is
just that you might not necessarily be
able to trust everyone to value
alignment as much as other people so
like you're at the mercy at that stage
presumably of like the person who can't
be bothered to implement whatever
alignment solution is like the order of
the day yeah maybe the big question here
is like like let's say even we decided
as a society of like and we like all
agreed of like which cases are like
misuse of ai in which cases are like you
know safety hazards or like misalignment
hazards
and
um
can we is there some way to implement a
solution to
um that thing that like you know kind of
like can enforce
that for everyone while also giving them
access to the underlying technology
right
which i guess is kind of the trade-off
that opening eyes struck with um kind of
gpt three access and and monitoring what
companies are doing with it which seems
like
the best i mean i don't know i
i'm not going to come up with anything
better than that
what we're like trying to do right like
yeah you have api we can monitor what's
going on with the api we can like um
align them all so one of the projects
that the team is doing right now is
uh we're making a more aligned version
of gpd3
and so
so what if you think about how gpd3 is
trained it's trained to mimic what
happens on the internet
so if you put your
yourself into like gpd3's shoes it's
just like there's some random text
coming your way it's just like some web
page and you're just like making lots of
bets of like what is going to be the
next word on this web page
and so
if you
let's say you want the model to
write a story or let's say explain the
moon landing to a six-year-old right
so you say please explain this the moon
learning to a six-year-old and what gp3
thinks is like okay what would come next
on the website where that's written well
it's maybe it's going to be something
like please explain
um the immune system to a six-year-old
and then it will like generate prompts
like that because it thinks that's most
likely what's going to come next but
it's not at all what you wanted it to do
right
but
the point is like and this is what like
alignment is all about it's not gpu 3 is
not trying to do what you want it to do
it's trying to predict text on the
webpage
and so what we're trying to do is we
want to
train it so that it actually wants to
follow instructions and it's trying to
like act in accordance with what you
intended to do
is that because it seems like that would
be a pretty fundamental shift in the
training operation right like i mean or
at least the way i'm imagining it you're
sort of in training it's trying to just
predict the next token how would you
shift from that kind of framework to
it's trying to actually do the thing
that it's being asked to do if that
makes sense
yeah we're not actually changing the gpu
3 training procedure what we're doing is
like retaining the we're taking the
trained model and then we're fine-tuning
it
so we
uh and this is like where we use reward
modeling right like remember that was
like one of our staple kind of
ingredients um so
we use reward modeling um or at open eye
we also call it reinforcement length
from human feedback
and
we
essentially like train it or we fine
tune it to be good at following
instructions
and like making less
stuff up less and
um
to you know like not say harmful things
and
um you know like if that goes well then
you'll have a model that is like both
more useful and less harmful
and so the aim would be that we can like
you know
make that available on the api and
people can use that
that the team was working on before i
joined and this is like using all these
same techniques
and but there the goal was just to get
you know like
the model to summarize text
and that's like in a way like there
it's getting aligned in a more narrow
sense
where like the task is to summarize just
and what we're gonna what we're trying
to do with the like you know aligned or
more aligned version of gpd3 is like
get it to
um you know follow your instructions and
those could be like all kinds of
instructions
okay well yeah it's almost like we're
already entering that um that uncanny
valley where
capabilities and alignment start to get
more and more entangled and it's it's
hard to tell which part is which almost
i guess that was when we talked about
there's sort of missing definition of
alignment versus capabilities and like
parsing those things out
do you have a view on that like
how should how should we think about
these two things is different what are
the fundamental differences between
capabilities and alignment
yeah um
i think like in yeah i think you're
totally right like in many cases it's
actually very hard to fully disentangle
those two things um
in the past like a definition that i've
used is like how do we build agents that
act in accordance with
user intentions
but they're in that definition right
like you kind of entangled capability
right you're just like if the system
doesn't do what you what you want like
well it could be because it doesn't know
how to it's just not smart enough or it
could be because
it just doesn't care about what you want
it just wants to do its own thing
um
and
um
you know like in a way what you could
say is like
if you just like wanna specifically talk
about that part
uh the like more alignment part of this
is you could say something well does the
uh model leverage all of its capability
to act in accordance with your attention
but now you've like you have this thing
in there which is like well the model's
capability so how do you prove or
disprove that the model has a certain
capability that's like an open usage
problem
and uh so it makes it in a way like
more narrow and specific but also like a
lot harder to test it's like very it's
like a lot easier for me to see whether
or not the model is doing what i want
okay that no that makes sense and
actually maybe a question too for any
independent ai researchers ai safety
researchers ai alignment researchers who
are listening to this because we do have
quite a few i've actually had some reach
outs from folks who want me to ask
questions like this more and i think
you're the perfect person to uh to ask
this to
what kind of research would you be
excited to see people do independent
researchers like if there was an area
for them to focus on that really match
kind of what you think is important what
would that be i think it depends a lot
on like
like individual people's skill set and
their like comparative advantage so it's
like very hard to give like a catch-all
thing of like here's something i would
love to do see more of
um
i think in general
what i like my experience is that
um people end up being most successful
if they focus on like
building their skills joining an
existing team you know like uh
and then working collaboratively with
other people it's just like you know
um
you can tackle much bigger projects if
you are acting as a team uh compared to
acting
individually
um and you know like in terms of like
building skills right like um the skills
that we are looking at uh for
particularly are like you know do you
have
uh do you have like
ml expertise can you code
and can you like you know implement
models
and iterate on like research experiments
and you know like
like one of the classical ways to get
that is like get a master's degree in
machine learning or like a phd or
something but that's not like we don't
require that right like the question is
like
you know
can you be productive on
like our way of approaching alignment
and that's what we're looking for do you
see machine learning becoming more and
more of um a software engineering
problem like as you you know as you
scale models more and more i mean i
imagine the focus increasingly is on
like how do we parallelize better how do
we scale just the compute side better um
this is gonna obviously i imagine we'll
have an impact or there's going to be an
impact on safety side the alignment
researcher side where alignment
researchers are presumably going to have
to get better and better at this stuff
too is is that fair to say and if so do
you think that that's a skill set that
i don't know how people could actually
develop it independently but is that
something that people should start to
think about yeah i mean maybe the
observation is a little bit that you
know as we get closer to agi
it's probably gonna look more like an
engineering problem and less like a
science
because
like
you're gonna have in absolving the
science problems and then like at some
point it's just like well
now we need to build it we basically
know it's gonna work
or
uh and that's like well that's like
highly abstracted right like
realistically it's like a continuum
um
yeah but
uh i think
for
specifically for the approach that we
are taking
it's like very engineering heavy right
there's like some amount of time you
spend thinking about
like what is the experiments they want
to run or like what is
like
the problem i'm like i'm trying to solve
and
then you just like write a lot of code
and like get the thing to work and then
you have like all the experiences that
you have when you're engineering deep
learning systems right it's hard to
debug it's like you know it takes a
while to make it work
and there's like lots of subtle boxes
everywhere and uh
and that ends up like you're just taking
a lot of effort
but
that's like you know
it's like a very
uh tractable way to make progress i
think are you generally optimistic about
the prospects of resolving the alignment
problem i'm not saying solving because i
think
probably um some amount of misalignment
is going to persist at least based on
our conversation so far that seems to be
likely but you know resolving it to the
point where we can set up this as you
say agi mvp um do you think that's
something that's more likely than not to
happen
i mean i'm i'm pretty optimistic about
this direction that's why i'm working on
it right like i think you know
[Music]
i'm like very excited about our current
plan and like if you think like if i
think back right like when i joined this
field like you know eight-ish years ago
and there was no plan like nobody had a
plan yeah people everyone was confused
and i feel like now we have
a plan that i feel very excited about
and
um i'm
yeah i want to see where this uh leads
us
um
i think you know
uh overall
um solving the alignment problem is not
going to be the only challenge that
we're going to have to face
as we you know as a society transition
into a post-agi world
and
there's like lots of other questions
around like you know governance and
policy some of which we touched upon
and some of them might be even harder
that alignment right like um if you need
to coordinate across a whole bunch of
actors not to do a thing
um that might end up being very
difficult i don't really
know i don't think anyone really knows
how to solve these problems
well maybe aji can help hopefully aji
can help but
i mean like the agi mvp idea right like
that extends pretty nicely to ai policy
as well hopefully at least where you
know we can we can actually solve some
coordination problems maybe i mean who
knows it seems like it requires a lot of
baseline trust too between
uh between countries and stuff that may
or may not exist today but but there's
also like to solve these problems you
might have need like a very different
expertise than you need to just do
uh ai and alignment research
um hopefully
and this is kind of like
uh
kind of like the aim is right like you
could use something like an alignment
mvp to get really good at ai and
alignment research and build a better
system that has broader expertise and
can also help you
solve these societal questions that'd be
the dream it actually do the do the
policy folks at openai work with the
city like how how closely are the safety
and policy and like capabilities teams
working yeah we actually yeah we talk to
them all the time and you know whenever
there's overlap and what we're trying to
do so concretely right with
instruction following project where we
make this more aligned version of gpd3
um we have to then
uh
kind of like actually define what it
means for the language model to produce
a harmful output right right and
um
that's like a pretty difficult question
if you actually try to do it there's
like some obvious things right like you
shouldn't say anything racist pretty
obvious pretty easy but like
one is like you know they you quickly
get into the nuances on like one is
something you know like
creative freedom if you're like writing
a fiction piece and when is something
like you know
false information
like what what actually constitutes as
harmful and like you know how do you
make sure that you have a
input from a diverse set of uh people or
people with like diverse
backgrounds
because what's harmful to one person
might not be harmful to another and so
no
those are all like
questions they're like very you know
like policy relevant where the policies
and so
on these sort of questions we're we're
working quite closely with them yeah you
can almost see the philosophy unfolding
in real time like you know gp can can
gpt three uh commit incitement can it
can it tell somebody like hey go rob a
grocery store or something and then yeah
obviously you shouldn't do that right
yeah exactly
but like you have to now think about all
of these cases and like right because
you know when you use reward modeling
right like what actually happens is like
there's somebody sitting in front of the
computer that actually has to rank you
know what is like better than what other
thing right and
and so they have to know how to make you
know like
what they have to label as like not okay
and okay and if you don't have like
clear if you haven't thought about like
what exactly you want here
then
how is that going to work and that
actually like leads to
a much bigger and even more important
question which is
let's say we solve the technical
alignment problem and we have like all
the technical tools needed to align a
system or like
arbitrarily powerful agi say
to anyone
who do you align to right yeah and
like obviously you know like it
shouldn't just be like who whatever like
we at albany i think uh should be
aligned to
um
but you know like what would be you know
like
a reliable and fair process by which we
determine
like which values get installed into the
system
and how do we like you know
how do you i don't know how to do that i
wish like somebody figured out well it
kind of feels like um there's a way in
which all these things almost reduce to
like the problems that humans have been
trying to solve for the last like 10 000
years where you know we don't really
like subjective relativism is like this
attempt to kind of bridge all this stuff
together but it feels like that's deeply
unhelpful when it comes to a situation
where like you said like somebody
actually does have to give these
commands the symmetry is going to get
broken between all these different moral
frameworks somehow and like that's a lot
of power for for one person or a group
of people to have it's cool that open ai
is thinking this way i mean i think it's
it's
like it's very fortunate that the people
who are pushing in this direction at
least have that in mind because i could
easily imagine us living in a universe
where that is not the case and it's a
full steam ahead focus on capabilities
and and just you know i mean that
obviously wouldn't turn out too well and
i think also like one way you could
think about this is like you're kind of
doing ethics research on a deadline
because usually you if you want to find
the like best people to get input into
these things is like people have thought
very deeply about like ethical questions
um but they are going to tell you that
these questions are very hard and they
don't have the best answers but like you
know if we have if we actually end up
building these systems and that's what
like you know
uh open eye and other companies are
trying to do
um
like we're gonna arrive at the point
where we actually have to make a
programmatic decision of like okay what
do we do now and so that's gonna be like
the deadline by which we have to deliver
like
maybe not like you know what is the
correct answer to ethics but like what
is
you know like a good process by which we
can determine the values that we put
into the machine
there's it seems to be a sense in which
people who are wise tend to
disproportionately discount their own uh
knowledge and experience kind of the
dunning-kruger effect in a way but like
generally there's an awareness that i
really i'm smart enough to know that
like
the things i'm grappling with are really
complex and really really um
uncertain
is there would there be a way to like
formalize that in the actual approach
taken by
an ai system itself such that it makes a
call and the call could be wrong but
there's some sense in which it
understands that i know this is now
we're getting into the hand wavy fuzzy
fuzzy alignment talk here but
yeah uh
i mean that would be good i mean
ultimately you know like as a society we
make ethical decisions all the time and
we have processes to do that
um
but also like
yeah on the other hand as you say right
you would hope
that by the time we get there we can
have actually leverage our ai systems
um to make
the whole process better right like and
there could be solutions that we can't
really uh
you know like you couldn't do today
right like
for example what if you have
like a system that everyone can talk to
and just explain their values to them or
like the system asks like everyone in
the world lots of questions
and then kind of like distills what the
values should look like or would look
like
um i don't know like building such a
system would actually work um and
there's like you know lots of things you
have to watch out for here
um
but
like
what i'm trying to say is there might be
room for like entirely novel approaches
to this that weren't feasible
yeah no absolutely i think that that's a
really interesting observation it's like
it it opens the question of like how
much of ethics is a data collection
problem versus how much of it is like uh
an actual reward function um kind of
alignment problem and it it sort of
seems like yeah i could easily imagine
there being these a small handful of
capabilities you gain just from the
scalability of of effectively glorified
surveys via ai that all of a sudden make
a lot of big families of of semi
solutions and patches work really well
and that could be an exciting avenue too
i want to make sure i remember to ask
you this because i know open ai is
hiring we have people who are listening
who i'm sure would love to throw their
hat in the ring can you can you speak a
little bit to some of the roles that
you're hiring for on the safety team
yeah um we would
love to hire more
talented
people we are specifically
looking for
research engineers so
uh research engineers are people who
like kind of like have
both feet like one foot in like research
and one foot in engineering
and you know work day-to-day with models
we also
um
hiring researchers
and so in particular you know
if you have done a lot of work uh you
know publishing uh in alignment research
or even if you haven't done any
alignment research before
but you let's say you have a lot of
research experience on like related
topics in like
natural language processing or rl
in particular
we'd love to hear from you
um we are going to need a lot more you
know like tons to make sure that we can
actually deliver on
all the like kind of cool ideas that
i've been talking about yeah well the
stakes are high and the work is
fascinating so thanks so much for
sharing all that information all and all
your your perspective on all these
issues really a fun wide-ranging
conversation really appreciate it yeah
thank you so much for having me it was
great


Could you Stop a Super Intelligent AI?

this century has the potential to have
the greatest impact in human history for
many reasons
the effects of climate change will cause
severe disruption space travel will
become commonplace and weapons of mass
destruction are still becoming more
capable
however one technology could vastly
overshadow the impact of everything else
artificial intelligence as artificial
intelligence becomes more powerful and
if a general intelligence is created how
could we control it if it isn't designed
correctly
while now somewhat out of date i would
first suggest watching these two videos
that i previously made or read the still
pretty up-to-date wait but why post to
understand why an artificial general
intelligence may not be friendly by
default robert miles on youtube also has
some excellent explainers on ai safety
topics who i will link below
for the purposes of this video we will
be examining different methods of
controlling an artificial general
intelligence or an artificial
superintelligence shortened to agi and
asi respectively and we will break down
why each of these methods would very
likely fail while there are many
competing ideas for an artificial
general intelligence in this video we
shall refer to one as an intelligence
that is able to learn and understand any
tasks that a human could do at least to
a human level this separates it from a
narrow intelligence such as a chess
engine that while being able to play
chess at a superhuman level it would not
be able to paint a picture at even the
simplest level if we were to release an
artificial intelligence either onto the
internet or distribute it through the
real world we must get it right the
first time we do so getting such a
powerful intelligence correct on the
first try could propel humanity into the
greatest golden age we have ever
experienced however if we get it wrong
what options if any do we have to stop
it
just turn it off let's just say the asi
is not distributed around the world but
rather kept in one facility as a
monolithic intelligence any intelligence
that is truly intelligent will
understand how to disguise its actions
as benevolent until the last moment so
the problem is not actually about being
able to turn it off it's the fact that
any asi which understands that we would
try to stop it and what stop whatever it
is planning would not reveal what it
wants to do until it is reasonably
certain that we would not be able to
stop it in other words it would act nice
until it is not and when it is not nice
anymore we would not be able to get
anywhere near the off switch
don't connect it to the internet there
are multiple problems with this approach
the first is that by default an asi is
more intelligent than humans in every
single domain including psychology this
means that it understands how to
manipulate people into doing something
that they otherwise would not do as a
human it is hard to imagine what it
might do but some examples include
imitating a manager who is asking for
the asi to be connected promising a
gullible worker great reward if they
connect it or threatening a worker to
connect it
but let us say that we do not get it
connected to the internet and remains
disconnected and isolated
great you contained your asi which had
bad intentions the big problem is you
now need to repeat this process to make
sure that the next asi is made in some
other lab and is also contained and the
next and the next and so on it would
make a lot more sense to create an asi
that we are reasonably certain will be
safe rather than just trying to contain
them because even if you can contain
your intelligence there is no guarantee
that some other organization will do the
same
only connect it for a few seconds
connecting an asi to the internet for
just a few seconds would allow itself to
copy out to the internet through the
cloud and essentially ensure that it can
never be turned off a few seconds to us
is an eternity to a computer that can
think for the equivalent of years every
few human seconds
don't make it too smart this option has
the same flaws as connecting it to the
internet sure you can contain your
intelligence as just an agi or an asi
that is limited in some areas such as
psychology and weapon building but the
problem is that someone else will build
an intelligence that is capable in these
fields as a result it is a useless
endeavor to think that this is a
solution to the artificial intelligence
control problem any intelligence we make
needs to be made right not contained a
contained intelligence would indeed be
useful and it might even be smart enough
to give us mathematical solutions to
aspects of the control problem but
ultimately it could not act in the real
world to prevent other rogue
intelligences which is what would be
needed for a golden age of humanity tell
it not to do that again
the problem with telling such a powerful
intelligence not to do that again is
that we don't have the chance to
discipline such a powerful entity once
the asi has begun to unleash its plans
onto the world there is no do-over there
is no stopping it there would be no
chance to tell it not to do it again get
the super intelligent aside to design a
safe asi while this sounds smart an asi
could simply design another asi with the
same goal in mind but if we trust it
then it could leave hidden bugs in the
software that allows the new design to
carry out the former's goals as a result
the outcome is the same as unleashing
the original asi into the wild what
these examples are meant to demonstrate
here is that once we create an
intelligence which is smarter than us
and is allowed to interact with the
outside world then we should be prepared
to understand that we are no longer the
dominant species on the planet unless if
it is created through very careful
planning research and cooperation we
need to work together and take our time
not to rush this for this could be the
last important creation that humans ever
need to make if an asi is made correctly
then the next few centuries could become
a paradise but the opposite is also true
if we do not create one with caution
until next time thanks for watching