206. Jared Kaplan on Scaling Laws

there's a lot of freely available
writing on the internet to train on
and in books um and if an ai knows
language then you can ask it about
anything or try to communicate with it
about about almost anything and you can
therefore try to get a lot of intuition
from its responses and what it's doing
um as compared to some other kinds of
examples of tasks and that might
help us make it both general and uh and
safe
um so water language models and
basically all the models i'll talk about
in this talk
are always auto regressive and they
always have an auto regressive loss
where there's say a bunch of words and
the model is trying to predict the
probability of
the last word given the earlier words in
the sentence paragraph
passage etc and this applies to other
modalities like say pixels
it can apply to the answer to a math
question it can apply to
both images and text that are joined
together
as we'll see um and so here's an example
like as a speaker at a journal club
you're probably elephant me to say
certain things and when i say elephant
as a human we notice
that was a weird word to insert in that
sentence
and uh and the gpd3 model also thinks
that it was a weird thing to say given
given the preamble and so all we're
doing is optimizing these
log likelihoods um i'll almost always be
talking about transformers except when i
compare to lstms
transformers are based on this idea of
self-attention where you
uh when you're making a given prediction
you kind of look through all of the
prior words or
tokens and you uh up wait or down wait
them
kind of like intuitively a human might
highlight certain words in a passage
that are most relevant to to what's
going to happen next
um and then you look at that weighted
linear combination of
what you highlighted how much you
highlighted it you process it
um this occurs over and over again for
for layer upon layer and then finally
you make you make an actual prediction
so that's uh 30 seconds on transformers
and you can get very
very impressive seeming performance um
on a lot of different data modalities so
on the left we have
a sample from gbd3 uh we provided the
the title an author and it wrote this
poem
uh the on the right we have a bunch of
completions from uh igbt which is just
training exactly the same kind of model
on images
pixel by pixel and uh it seems to know a
great deal of
semantic and other information about
about about dimensions um
so this seems to this seems to sort of
work pretty well
um for some reason
my slides are stuck
i don't know what happened there
um okay so uh scaling laws
so scaling laws for neural models i
guess there's two different levels of
motivation a super high level motivation
is sort of
why does machine learning seem to work
well what actually matters and what
doesn't
and i think this informs what kind of
research problems we should work on and
also what we should forecast or expect
for the future
um question i had in mind when i started
working on the subject three or four
years ago was something like
is making progress on ai more like
proving three of my hypothesis
where you might expect progress comes
from a few lone geniuses who obsessively
work on the problem
um it's very hard for outsiders to gain
insight about what exactly they're doing
progress is kind of uh maybe seems like
it comes through flashes of insight
rather than being predictable or
incremental
or is it more like building a very
powerful steam engine where
there is a lot of incremental progress
uh uh you don't have to actually spend
your whole life
obsessing about the problem to
understand it um
and there are kind of simple basic laws
underlying
uh what leads to progress and what
doesn't and so what i'll be
showing you is evidence so that there
are fairly precise scaling laws for the
performance of
ai systems i'll be focusing on uh
kind of macroscopic variables like how
many parameters you have how big your
data set is
how much compute you use for training
but i think there are actually a lot of
other scaling laws
uh with with other variables in machine
learning i think they're kind of
ubiquitous
um i also kind of argue weekly that a
lot of the other details don't matter
very much
um and in particular it seems like the
scaling laws basically stay the same
even when we make a lot of algorithmic
progress
and the main thing that changes is some
kind of constant prefactor a lot of the
time
and achieving good performance is mostly
about avoiding bottlenecks
so the simplest bottlenecks are like you
have a model that's really big but not
enough data
or vice versa or you just don't have
enough compute
to train your model um or you have
literal bottlenecks in your network
like information kind of doesn't
propagate well through your network
and the classic example of this is that
if you have many many layers or if you
have the
uh the same layer repeated many many
many times
then maybe you face this kind of problem
where you raise a matrix to the power of
a thousand and you mostly just end up
projecting onto the
largest eigen the eigenspace of the
largest eigenvalue and therefore
information doesn't propagate well and i
think a lot of the most highly cited
papers in all of machine learning are
really solving these kinds of problems
like batch norm resonance layer norm
to some extent transformers versus lstms
i think they're kind of uh
avoiding these kinds of bottlenecks um
and
i don't know why having this problem
some kind of
i keep having some sort of weird uh
anyway no big deal um so uh
so what are these scaling laws um
uh the simplest scaling so i'm just
going to kind of explain some empirical
results where
rather than tell you a lot about about
any kind of theory
so the simplest scaling law is pictured
it right
so we have a lot of data and we
train a lot of different transformer
language models on
the same pile of tons of data and we
train them basically to convergence
and then we plot the loss the
autoregressive
log likelihood loss as a function of
the model size which is the number of
parameters not counting the embedding
matrices
uh that doesn't matter a lot for larger
models anyway but
uh that's that's what we have here and
we find super like very very precise
uh uh fit to a power law relating
the test loss to the model size
um so that's one example um in the
center we have a fairly big model
we train it with early stopping on
different data set sizes and we also get
a very nice clean scaling law
and the most complicated example is sort
of test loss versus compute
where we have uh different
amounts of compute in our compute budget
and so
what you see here are blue lines which
are learning curves
the learning curves for bigger models
are shifted to the right because
per training step you need more compute
to train a larger model
um and uh as we
uh and what you can therefore look at is
what is the best performance you can get
for from all of these different models
of different sizes
as a function of the compute budget and
uh there's an asymptotic line which is
what this orange curve is predicting
for what the best is that you can you
can do with a given compute budget
and that also seems to be a very clean
power law and
an interesting thing that you can do
here is
check what was the model that was most
optimal for a given compute budget what
is with the model size
that was optimal for a given compute
budget and
that's something that we see here on the
slide in kind of cartoon form
so you can ask as you scale up your
compute budget what is the optimal
allocation
to increasing model size versus
increasing uh
the amount of data you actually process
and something that i at least found
surprising was that
most of your compute budget should
actually be allocated to
making bigger models and only a small
amount to training longer
or with with a with a larger patch size
um and furthermore it seems like
architecture is somewhat less important
so uh on the top left we see
transformers versus lstms interestingly
it seems like these trends kind of
parallel each other until we get to
quite large models
and then i think lstms stop being able
to improve
as rapidly as transformers basically
because they're not as good at
handling very long contexts and that's
what's illustrated on the right
um and at the bottom we have a bunch of
different other
hyper parameters you can tune um that
that are associated with the transformer
maybe the most interesting is in the
center which is the
width of the model divided by the depth
of the model or the aspect ratio
and it looks like first of all getting
the aspect ratio wrong
doesn't hurt performance very much and
there's a wide range where you get
fairly similar performance and then
furthermore
all the different model sizes sort of
want a similar aspect ratio so you
should basically scale up by keeping
the width over depth uh roughly constant
um and uh uh but you won't even suffer
very much
if you if you get that wrong so in other
words these other hyper parameters don't
matter a huge amount compared to just
uh the the overall scale um and there
are all sorts of other interesting
scaling laws that you can find so
they're multi-variable scaling laws like
where you change the size of your data
set
and the model size together and you can
therefore predict the amount of
overfitting like the
the test loss with finite data versus
the test loss with infinite data
um all these things seem to be seem to
be relatively predictable
and uh you can make up an unsats that
fits them very well
and these things aren't really just true
for language so there's some further
questions is this specific really to
language as
a data set does it eventually break down
and does it actually improve performance
and downstream tasks and
these are the compute plots for
many other modalities including video
images
solving uh procedurally generated math
problems
uh uh multimodal models that
uh model an image from the text or the
text from the image
um and uh and also language and it just
color coded where bigger models are
yellow and smaller models are purple
and you see that there is some kind of
steady trend here
a thing that i also found surprising in
this vein
is that you can look at what the optimal
model size is versus compute for
all of these modalities since model size
and compute don't really care about
what domain you study you can you can
plot all the data modalities together in
a sensible way
and it seems like roughly speaking the
optimal model size as a function of your
compute budget is actually
pretty much the same for all of these
different uh
data modalities as you scale up um and i
i wasn't particularly expecting that
um there's a further question which is
uh
does uh do you really benefit on
downstream tasks
and so an example here is uh you could
train an image classification
using the pre-trained model that were
generative models for sort of modeling
image pixels
and if you do that you see that you get
another predictable power law
scaling for classification error
on imagenet um this is 32 by 32 pixel
image then
and uh furthermore pre-training helps
you a lot
to avoid overfitting so the orange line
is when you train without
uh from scratch on just imagenet the
blue line is when you take a model
pre-trained on image
generation and you see that that helps
you to continue this trend
uh much further so pre-training is
helping and scaling laws are relevant to
downstream
tasks and scaling laws are everywhere so
this is i'll just flash this quickly
this is
mutual information between image and
text um for multimodal models and and
that also has kind of a nice scaling law
um which i think is kind of cool and
interesting and so then you can ask what
happens if you really just do scale up
language models and that's what uh gbd3
represents
uh so this is the compute scaling trend
from from gpd3
um and uh
and we see that we get uh we get
continued smooth scaling
and then the other cool thing about gpd3
is that it can learn in context so this
plot at the bottom shows uh
performance as a function of model size
for colors
but uh it also shows how many examples
of a task
provided in the context and this is the
few shot learning of gpd3
and the solid lines show when you give
instructions in natural language to the
model
and the dashed lines show no
instructions so you improve when you see
many examples in the context window
um and you also improve when you see
prompts and you can see this then for
many different domains
um arithmetic uh sat
analogies the kind of tests american
high school students take to go to
college
um uh trivia and wintergrad schemas
and there's steady performance
improvements of model size on all these
tasks
um although the exact form of that is
quite different so there's
with arithmetic there's this sudden
takeoff where the model suddenly kind of
learns arithmetic
in a kind of discrete way for these
other data sets it's it's
smoother and you can have fun trying to
project these things
um humans have difficulty telling the
difference between
gbt3 generated news articles and uh and
real news articles
um i think these results suggest that we
can continue to get a lot more
performance by uh by scaling
scaling up um they suggest certain
abstractions for thinking about
machine learning performance um i think
they suggest sort of uh
how we should measure improvements in
algorithms where you you really care not
just about the algorithm on
one given model but on kind of a suite
of models and does it does the new
algorithm help you to
improve performance everywhere if you
care about that um this is a slide i
added to answer a question
and so why don't we move to q a


121 - Artificial Stupidity 2

- calling you and Skype decided that
means I wanna delete everything I done
before that ok can you repeat your
question we did hear it yeah I was
asking a question and now I've kind of
lost my train of thought
sorry okay well I press I can ask a
question I mean one useful form of
artificial stupidity presumably is an AI
that is not capable of manipulating or
deceiving human beings because I know
what sort of limitations it would need
like it might not have a good theory of
mind or human beings or it just might
not understand humor and irony and
sarcasm just just not understand the
concept of actually deceiving is that is
that a coherent idea is is it possible
to imagine limit sort of limiting that I
meant dimension of an AI making it very
autistic essentially very intelligent
but so it is there is a good movie
called the invention of lying I don't
know if you saw it where they start with
no one knows how to lie and then one
clever guy discovers that it's possible
so the thing that kind of relates to
your idea
I don't know how implementable it would
be but as a concept it could be
interesting of course for Customer
Satisfaction reasons you want your
artificial assistant to lie to you do I
look fat in this no no you look great I
suppose if you were super intelligent
you could find a way to get out of that
situation without without I solve
obesity sorry cure obesity epidemic but
delicious food with zero calories
I was always intrigued by the fact that
the cheering test itself has been a
subsidy for lying built-in because if
the AI on the other side of the of the
screen or you know teleprinter as he
thought of it is going to convince you
that is going to be indistinguishable
from a human being then it's got to be
able to line if you ask get things like
when it asked have a haircut warum does
it paint its nails or something like
that I wonder if a new California law
now makes it illegal to administer
Turing tests as they would have to solve
disclose as robots is that there's no
law saying if you're not a robot you
have to say you're not a robot and I
know humans well enough to be pretty
sure that some people would say yeah I'm
totally a robot when they weren't just
there lulz
especially in California one of the
classical thoughts that a lot of AI safe
to work has been done on things like new
comes problem where we would in
generally prefer that the AGI one boxes
but we put a lot of human two boxes this
is of course a controversial separate
any on whether you prefer to one box of
two bucks on new coming what do what are
your thoughts on this I haven't put any
effort into it I'm very interested in
our causal decisions and things of that
nature but I haven't done anything to
make a meaningful statement maybe get it
back to me in a year or so planning
things
maybe to make that question a bit
broader it seems in my intuition that we
are the reason why we want AGI is
because it can do things that we cannot
do it can be more unbiased and more
rational than that we can and if that's
the economic incentive then obviously
making it artificially stupid will it
defeat the point well oohed Lee and
that's the catch-22 right you want
system maximally capable while keeping
it at levels where you still retain
control that's what makes it so
difficult you can't can't have it both
ways
but if you consider for example military
and they open one that soldiers don't do
much of their own thinking and they have
instead quite code system of general
rules and useful work yes but they never
invent anything you don't see soldiers
in the field coming up with cures for
cancer are they there to kill people and
that's not very complicated one thought
I have is that humans in general have
really really poor introspection we
can't we can't really understand our own
feelings we can't really often can't
explain why we take the actions or
thoughts that we have and I'm thinking
for a an AI system even a super
intelligence we might be able to obtain
the same thing using homomorphic
encryption where we encrypt the for
instance even just in encrypting the
source code of the AI so it cannot self
modify in the way that humans can't self
modified it's possible for preventing
modification and improvement but in
terms of understanding that that was
your main question why would it provide
better self understanding my question
was that was a way we could make even a
strong super intelligence artificially
stupid if we encrypt the source code so
it cannot improve itself in that way but
it's already a super intelligent levels
right so
prevents it from going even more in that
direction but I think at certain point
whatever it's you know thousand take you
points in 2008 cue points from our point
of view looks the same and probably from
a safety point of view hopefully that
might prevent that particular form of
singularity the intelligence explosion
we talked about it in our AI boxing work
encrypting the code of course that
creates other issues who controls the
passwords how they are stored and again
usefulness of a system in that state
it's not obvious that the type of
encryption can be actually made
efficient enough for any meaningful
computation right now it's pretty slow
from what I understand what are your
plans or the next steps with this
artificial stupidity project so I hope
to get a very smart student somewhere to
help me actually collect all this data
and do a follow-up paper ideally with a
data set and maybe API for people to use
so if you know anyone or you are one of
those brilliant minds we're in need of
you to claim you are a stupid expert
publishing stupid papers thus a huge
benefit and what do you intend to do
with this data how do you implement
where you implement it so probably first
application would be for chatbots it's
quite trivial I consult for a number of
cryptocurrencies startups in that space
trying to make intelligent assistants
and perfectly record everything they do
and engage in and blockchain so it seems
like that could be one potential
application to make a more user-friendly
set certain limits and what they do we
also have any plans about implementing
artificial stupid thought processes so
it's not only stupid in on the surface
but also really stupid in its reasoning
so I think that we model human neural
networks with artificial ones based on
the same architecture we kind of
indirectly capture that there are
experiments I write about in my paper on
detecting qualia which show that without
encoding it directly certain neural
networks got the same experiences as
people so they perceive visual illusions
if you start disconnecting them they
have an illusion of out-of-body
experience they have visions and dreams
in the same way so I suspect complex
enough system inspired by human brain
with just by default capture all the
limitations and these systems these
limitations have little practical use
also so they are not constructed only in
order to demonstrate illusions some of
them are some of them are not so there
is good reason to think that it's just a
side effect and again the adversarial
inputs to artificial neural networks are
just visual illusions for computers
right when they look at random noise and
we see a picture of a panda okay I think
we are running out of time so does
anyone have any final comments or
questions I guess a quick question I had
about applying this to super
intelligence and the foundation met
soldiers made me think about this was I
don't know if you're familiar with
cristianos idea about iterated
distillation amplification but to my
understanding the idea there is sort of
you train the AI to like act as an
assistant to a human and you you
gradually like trendy ion the responses
of a human assisted by a team of a eyes
so maybe that sort of points towards a
way that like artificial stupidity could
work for if not you know superhuman at
least maybe in your human level a eyes
where they don't necessarily need to be
so superhuman level in order to still
contribute meaningfully crimes and value
so I know about his idea there seems to
be a lot of difficulties actually
implementing it in practice you need a
team of you know a is collaborating
exchanging information so already kind
of being aligned in some meaningful way
anytime there is a human in the loop I'm
highly skeptical about what can be done
again because of deception possibilities
social engineering attacks just in
general what is the timeframe if machine
is learning you know what billion
operations a second what is the human
supposed to do put ten minute delays
after every decision I'm not quite sure
how workable it is in practice that
makes sense I think the ideas is like
eventually you have some simulation of a
human instead of the actual physical
human but yeah it's not obvious thing
well if you have a simulation of human
you just solved artificial stupidity in
a gie that exactly where I'm going
person so you are doing iterated
artificial stupidity
yes we're slowly getting dumber okay and
so I would like to thank Roman Yampolsky
for presenting his work and I think it's
been a great discussion I've enjoyed it
very much and so I guess the there is
the next session the reading group will
be next Tuesday where we will read the
second half of Manas meetings article on
that else trees should perhaps not
prioritize AI and you thank you so much
right thank you I hope I'll get some
free time to join you guys more
regularly this is great alright P track
here thanks everybody
thank you see


264. Our Approach to Alignment Research

hello and welcome to session
264 in the aisafety.com reading group
tonight we'll be discussing uh our
approach to alignment research by Jan
liger John schoolman and Jeffrey boom
these uh three people John Lager John
Schulman and Jeffrey Wu are working at
open Ai and the when they talk about our
approach to AI alignment then uh it's
not 100 clear whether they are speaking
like for themselves or if they're
speaking for uh for the company itself I
would probably assume that they are
speaking for the for the entire open AI
this was published uh in in August and
four months later there was a post on
lesserong by Elisa utkowski called a
challenge for AGI organizations and a
challenge for readers that highlighted
some features about this post
um and actually we will start with
um miri's very short comments on this uh
before we go into the the actual article
so miris is called a challenge for AGI
organizations and a challenge for
readers and uh written by Elia zulkowski
mainly but also with Rob pensinger
editing and net Suarez for input and
it's actually
um more not that much of a challenge and
almost an accusation against the other
AGI organizations
um and uh this is kind of similar to a
challenge that Miri also had about
courage ability that we looked into uh I
think three four months ago or something
like that
um
which uh we we didn't participate in but
in but this uh this video this
presentation is in fact meant as an
entry into this uh challenge
um
you may recall the last time for that
challenge about who could write most
about courage ability
um I judged that Miri won but not
overwhelmingly so and um for this
particular challenge I predicted on
Facebook a couple of weeks ago that I
think this one would go uh basically the
same way
but we'll see uh when Miri uh hopefully
posts theirs uh their entry in uh the
not too distant future
so the challenge for deep mind and
anthropic the other AGI organizations in
theory of course there are more ATI
organizations
um but I think uh Elise richardkowski
has mostly given up on on those and
consider them Beyond any kind of help
Redemption
so this we have openai's plan that is
the the document we are going to read
today uh and obviously utkowski
disagrees with this plan but he is
really much in favor of uh releasing the
plan so people can discuss it
um both because like it's really
important that there is a plan when
you're doing something different even
though uh no plan survives contacts with
the with the Enemy and also it's
important that the plan is public
and the problem the key problem of
course is that openai has a plan but
deepmind and anthropic has not made any
kind of plan public and most likely that
is because no such plan exists and so
this is a challenge to these two
organizations come up with a plan and as
fast I can tell over the past month uh
deepmind and anthropic have not made any
kind of response to this at all not a
yes not a no not any thoughts about a
plan just basically totally nothing
um and that's of course somewhat
disheartening
um also they haven't reached out to uh
myriadi to anthropic and deepmind for
whether the planet exists they believe
probably some people in the organization
have thought a bit about this
um but uh it's probably a good idea to
make a plan soon sooner rather than
later
there is here on manifold markets a uh
uh a bet on uh what is the probability
that one of these will actually produce
some kind of plan before the first of
March and there is now a 48 probability
that this will happen
according to the manifold Market
so why do we own the plan what's the
point of a plan well uh once you make a
plan you have a single canonical place
where you put all your assumptions and
it's a very easy place to like if you
want to know if it's a it's a good plan
then you can go into the plan and
analyze it see for inconsistencies or
other kind of problems when things
update when you learn new things you can
update the plan
um
and in particular if you're trying to do
something really difficult like building
AGI and making it not kill everyone then
plants are really really important
um
I think I obviously agree with Elizabeth
that what these organizations are doing
is very difficult
um but I think one of the key reasons
they may not have made these uh plans is
that they in fact believe that the
problem is easy like if you believe
there's plenty of time and you can make
unlimited retries and you can count on
the Goodwill of all other actors and
things like that then maybe a plan is
not so important like it's only if
you're trying to do something difficult
another big advantage of plants is that
uh you can the field can debate it uh
and the field can
um like compare different plants and in
theory hopefully the the the researchers
could decide to go with the organization
that has a better plan uh that seems
like a reasonable uh you also want to
probably avoid some part of the plan
making those public like you if you make
the plan completely public then some of
it may be very relevant to your
competitors other people trying to build
online AGI
um
you probably also will need a branching
plan if you are uncertain about the
future there's also a very likely thing
to happen
but I think that is not a plan is to
just build an AGI and then after you've
built the AI then try to do a lot of
work to to make it uh to make it safe
and the exact problem with that uh I I
disagree actually that it's not a plan I
think it's a plan it's just a horrible
plan and the reason why it's a horrible
plan is that if you have an organization
that is capable of building an AGI that
and realizing that the AGI you're
building is unaligned and could
potentially be very dangerous then
almost by definition you are an
organization that is not very safe to
conscious that does not have some kind
of security mindset
um
and that means that you are very
unlikely to just get that Suddenly at
this point even if you're playing calls
for it
so there is a similar parallel uh
challenge for the readers well not quite
a similar challenge but that is to look
at open ai's plan and
um just like Miri is writing up their
thoughts on it then we should write our
own uh thoughts preferably first so that
they are unanchored on what Miri is um
uh it's writing and of course focus on
what is most decision relevant and the
hope is that the criticism that uh muru
is going to come up with that we can
preempt that uh and
um to see whether uh this kind of
criticism can happen without Miri and uh
of course in some way try to make Miri
Superfluous because Miri is an
organization that is existing much less
these days than it has been previous
also with this unanchoring it's a bit
complex precisely what that means one of
the things uh Elise explicitly asks is
please tell us please make it clear if
you're repeating something you've heard
from emiri person at a gathering or
something like that
so I have a hard time figuring out
precisely how to obey that requirement
because obviously something that Miri
has said or published at some point I
can't get around that because they have
pioneered the field and for a lot of
things they are just the the key
reference you can't talk about courage
ability without talking about Miri
um
so um
I think even if uh there is a 100
success rate and when Elisa utkowski
eventually writes up his criticism of
open ai's plan he says nothing except
what I've said in this presentation I
don't think you can conclude that Miri
is Superfluous for the simple reason
that a lot of the things that I'm saying
is built on Research that Miri has uh
has been doing
so when I
interpret the the like in this
um then I think like it's something like
if two people are chatting then uh that
is that doesn't really matter if it's
online at a gathering or at a gathering
um it's more like if it's one to one or
one too many uh as how I would would
interpret this
um
and also one thing I should say about
how unanchored I am is that uh after uh
uh Elijah published this then a number
of people wrote up some criticism of uh
of openly eyes plan I did not read those
and then like it he wrote some kind of
answers to this criticism and I didn't
read that either so I'm very on anchored
and that may just be a an excuse for
being very lazy but I haven't uh uh
there's a good chance I am saying
something today that Jan liger has in
fact answered already
so one example of a place where I'm in
doubt about uh whether this is whether
I'm fulfilling this is I met iliakowski
in San Francisco in July for EA Global
and I was describing some of my plans to
him and some of the other people who
were there I'm not entirely sure who
they were made some objections
um like uh I think it's a an interesting
objection I think it's too General and
doesn't really relate to what I'm doing
but that would be kind of example of
ways that where some of my criticism is
something that I've gotten uh like
directly from Miri in that way
right so let's uh go to through the
article our approach to alignment
research by open AI
and first also I would like to say that
there have in fact been a previous
iteration on this process and that was
when open AI was founded they had a plan
to put an AGI on every disk like really
really uh openness in in all that
capability research and that was a
public plan and uh it got a lot of
criticism and they changed uh open AI
changed very much to not be public about
their capability work and I think that's
a beautiful example of this kind of
process working really well and that's
why I have substantial hope that this
process can also cause some kind of
improvement to the uh epistemics of open
AI
right so the introduction of the plan
has a goal and that is to make AI
aligned with human values and follow
human intent I think it's a decent goal
uh I think it's it should be more
precise and comprehensive and all these
kind of things and it's not precisely
courage ability
if I were to write this kind of goal
then courage ability would be written in
very large uh
um uh let us on the second line there
would be one goal on top of that but
credibility would be on the second line
um and I think if you try to build AI
aligned with human values and following
human intent but not courageable then
the things that are not courageable like
making the AI change its mind and how it
looks use itself and this kind of thing
uh I think those are in fact potential
problems for their plans
their approach is empirical and
iterative and they want to study how AI
alignment
techniques scale and how they break I
like that they say how this will break
and I think it's really important to
have this understanding that the
techniques they are using are
preliminary
um I would have we do in fact have a
substantial idea about where they break
they break up uh when we have
distributional shifts and that's one of
the things that I would have liked them
to to explicitly point out because we do
in fact know more than than what they're
letting on here
and they're both doing a current and
expected alignment problems and try
pushing push current alignment ideas as
fast as possible and believe in fact
that the current ideas are quite robust
we can get very far with them we can uh
substantially Advance alignment research
using the ideas we already have
so again the framing is a bit off here
in that they say we will advance towards
solving the problem instead of solving
the problem a good plan should end up
with a problem being solved but uh
uh this is just crippling of words
um so I talked a bit earlier about how
openly I used to be completely open and
now they are like open ish or there so
how about their openness is Niche
um they have the overall idea that AGI
could be really dangerous and it's
possible that it will require everyone
to work together and that obviously
seems like it would require quite a bit
of openness they don't have any like
criteria for like how will we know how
much you require and how will we get
everybody on board that seems like a
tall order
um
but the key thing they want here is to
have openness in alignment research but
only when it's safe to do so and that
could be a number of reasons why it
would not be safe
and they want also to be transparent
about how well their alignment
techniques work in practice so that's a
good question like they say they write
in their uh plan that they want to be
open about this but then they release
chat qte3 uh the the chat gbt which
clearly has some issues and so the
obvious question is how well did their
alignment work work and they haven't
written about that and I think a good
reason why they are saying this is that
it's not safe because if they describe
all the techniques all the prompts
they're using then the people on Twitter
and 4chan are going to look into that
for different holes and attacks based on
that and that means that the plan
already now could be facing into a
problem that that could be much more
prevalent in the future that they cannot
in fact be open about it
and on the sentence they have this we
want every HEI developer to use the
world's best alignment techniques and
like depending on how many AGI
developers you are envisioning that uh
mostly to me sounds like they're they're
imagining a lot of AGI developers like
and in that case
probably we are very doomed if the if
there's so many that open AI can't just
write discrete to all of them
uh
let me see why can't I get this why
can't when the school uh wait hold on uh
so there are three pillars of
there three pillars of the approach
um the first pillar is to train AI
systems using human feedback the second
is to train AI systems to assist human
evaluation and the third is to train AI
systems to do alignment research
and we'll go through these three in a
moment but first I want to highlight
that three pillars isn't really a good
metaphor because
um it's not like they're working on all
three of these at the same time uh it's
more you can imagine some kind of
Ridgeline plot with starting to work
mostly on training AI systems using
human feedback and then transitioning to
most training AI system to assist human
evaluation and then transitioning to
mostly doing uh having these systems do
alignment research
um
and when you look at the this way then
it seems like the plan is missing some
kind of timing and criteria like when do
we go from
mostly focusing on phase one to go to
phase two and when to phase three and
um
okay let's talk about the first one uh
training AI systems using human feedback
so
um like this is reinforcement learning
from Human feedback uh yeah okay uh so
here is some prompts we have in the data
and there's a sentence X that says that
a dog is and then you have some kind of
initial language model that says a dog
is a furry mammal and then you uh
compute this new policy
um and then you find a reward for this
and then you use some kind of
reinforcement learning uh for instance
uh proximal policy optimization
um to tune this language model and then
for him for instance then you get to
like a dog's man's best friend and then
you use that for two things both for
continuing with this
uh shift but also to have a model of the
rewards that you can use
for going forward
uh that's the thing this is the
technique that is primarily being used
uh in Omnia and they think this is
working quite well they have found a lot
of low hanging fruits and this can
inspire all others in the industry and
raise user expectations for how aligned
AIS should be and gives a rich feedback
look which enables their empirical and
iterative work
but
this is not enough it's not fully
aligned sometimes it doesn't follow
instructions sometimes it's not truthful
sometimes it it's supposed to refuse a
harmful task but it doesn't generally do
that and sometimes you can make it say
like biased or racist things and things
like that
here I would object an object perhaps
quite strenuously that this is in fact
not alignment in particular the first if
it fails to follow instructions that is
not an alignment failure like if you ask
the uh the AI please tell me what the
34th president of the United States was
and this is uh that is George Bush then
uh that is not a failure of alignment
that's a failure of capability and to a
large extent I feel the others here are
also failures of capability rather than
uh than alignment
uh the hope for
um uh open AI is that this uh
reinforcement learning from Human
feedback will be some kind of uh
building block for scalable alignment
um and it could be but it seems to me to
be some kind of uh I I call a foundation
of sand in the sense that we are not
really pointing the AI at actual docs we
are pointing AI at
um uh representations that are
imminently hijackable
um and I think this means that in the
limit of very strong AI this is going to
fail uh catastrophically
the second pillar was training models to
assist human evaluation and that's
obvious that as the models become more
capable it becomes just plainly harder
for humans to evaluate whether what the
AI is saying is correct and we also get
the pathologies like the AI telling
people what they want to hear
um and the key way to get around this
that are being used right now in open AI
is recursive reward modeling they're
also using some other things but
um
here you can see like this is a very
classic uh reinforcement learning setup
where you have an agent and an
environment that gives observation and
takes actions and then you have a reward
model as well this is a and a user
giving feedback this is kind of like a
standard reinforcement learning with a
reward model and then the idea here is
like the recursive part of recursive
reward modeling is that then you repeat
the process but flips it to the right 90
degrees so that the human takes the
place of the environment
and then you get like a new reward model
and then you repeat the process again
turning right every time and that's
where the the recursiveness of a
recursive reward modeling comes in
uh and one of the things they really
want to to have this recursive reward
modeling do is to figure out is the
model being misleading or deceptive
um
and they believe that the best way to do
this is to actually make AI assistance
uh work in practice make AI assistant
evaluations work in practice
uh I uh notice here that a problem with
a plan is that there is no direct link
between these two things in that
um I believe the recursive reward
modeling will in fact not help very much
with a deceptive alignment
Okay the third pillar is training AI
systems to do alignment research
and of course we expect to encounter new
alignment problems and we don't think we
have an infinitely scalable solution at
the current uh at the current level so
what we need to do is to build and align
an AI and then have that do alignment
research
um
I think this plant is a
dangerous potentially very dangerous in
uh were sometimes been called that
attack this HEI complete in that if you
can do AGI research then probably you
can do everything with a small star uh
and
um and certainly do enough things to be
dangerous
and the hope of open from open AI is
that the air will gradually take over
the alignment research while humans of
course stay in the loop all the time and
they make a specific claim that
evaluating alignment research is easier
than producing it especially when
provided with evaluation assistance and
I don't think this is obvious at all but
it's probably true when you have
something that is like not explicitly
deceptive but if the person if the
research is being done by someone who is
potentially deceptive then I think
evaluating whether that is the case is
in fact really really hard
so alignment research from the large
language models which are of course the
key models being used they make the
claim that narrow AI is sufficient for
alignment research I think that's really
quite a claim uh of course uh narrow Ai
and general AI is some kind of spectrum
and if you define a narrow AI as and and
perfect AGI that can do everything
except one thing then sure you can call
that a narrow AI but on the general cons
uh conceptualization of what is a narrow
AI I think the fact claiming that it can
do alignment research is a really really
tall Aura and I think that is a claim
that probably will not stand up to
scrutiny
another reason to be optimistic about
this is that out of the box large
language models are not in fact agents
and that is true of course but they are
almost Akins you can make them
assimilate agents with like a simple
prompt so all the uh the mechanics are
there and I don't think that makes uh
them a lot safer
uh it is stated that they don't need
internet access to do alignment research
they can just
from nothing uh uh fix your things I'll
figure out what the problems are and
make some kind of progress I think that
is extremely optimistic like perhaps
Elisa utkowski could just from nothing
realize that there is a problem and make
uh uh real progress on this I don't
think anyone else could do that I don't
think alone uh like Eliseo also had an
amount of input and this idea of having
the AI in a box without internet access
almost certainly does not work we know
too many problems uh with that
and again once they have a model that's
useful for alignment research they plan
to make it accessible and
um that's this quote here that I'm a bit
unsure what means while we don't know
when our models will be capable enough
to meaningfully contribute to alignment
research we think it's important to get
started ahead of time so what does get
started ahead of time means like before
they can do something then we need to
have them work on it by definition that
doesn't sound so like hopefully what
they mean is not that they will start to
do all the uh dangerous things first and
then get started with that before the AI
can actually contribute to solving the
alignment problem that seems uh like the
wrong way I don't think that's actually
what Yen means but I'm unsure precisely
what it means with this sentence
the plan has some limitations uh and
Omni are acknowledging that it probably
needs to be adapted when AI becomes
stronger we will need
we'll need to adapt the plan in some way
and I think that's a good feature of
most plants and it's also under
emphasizing how much robustness and
interpretability will mean for uh our
odds of success
the AI evaluation assistance is
potentially problematic in that it can
amplify problems in assistance
we could see discontinuities of
different kinds either in technology or
in time
from our current models to ADI
it's possible that getting this training
signal right isn't actually the hard
part of alignment an example would be in
a misalignment that could be problematic
um but it's uh and it's possible that
the least uh
capable AI That's capable of uh doing
alignment research is capable enough to
be dangerous that's my general
expectations
um
stated here with uh this came out of
order the hardest part of the alignment
problem may not be the training system
signal but even if that is the case then
the training signal will still be
required
so uh what do I think about this section
I was a bit uh Curious in the sense that
limitations is a uh not the word I would
use like if it turns out in fact that
there are discontinuities then that is
something we need to have some kind of
plan for faster takeoffs and uh like if
the least capable alignment research
AI is General enough to be dangerous
then we need to do something about that
like I don't really want to leave this
as holes and I think more work should be
done clearly if you have a plan with
some holes then I think it's a very
obvious thing to say okay we need to
actually work more here and make version
2.0 of our plan and have a plan that
just looks like it might actually work
so those were
um my summary of openly eyes plan uh
interspersed with some minor comments
now I want to focus on my uh primary
comments and concerns about this plan
the first is that it's framed as an
approach and not a plan
um an approach is a lot more awake than
a plan
um when you um
when you are doing something that is
really really important in this case
open AI is saying this may destroy the
entire world then I think it is
worthwhile to spend some extra time to
actually formalize it enough to become a
plan
um and I think that is actually really
important
um and if this was a plan then the
obvious thing people would say is like
there are a lot of known this is rather
for our plan how plans should look and
you would evaluate it up against that
and when I look at like what do I expect
from plan one thing that really really
stands out as a reason why this is not a
plan is that the objectives are
extremely unclear and
unquantified and like described in
extremely small details with with very
little details and I think that is like
if you have a plan then naturally you
would think okay you need to actually
consider what are the objectives in fact
uh another thing that uh when I look at
the plan you could argue that it's a
three-step plan and three-step plans
like three steps is not that many but in
fact I'll later argue that this doesn't
in fact solve the entire problem we'll
need more steps
um and uh uh once you start getting into
five step plans or something like that
then uh my inner Melia starts to like
switch like five step plans uh often uh
not uh going to work in practice
another thing if you have a multi-step
plan is you should think about what
happens if uh step one and two succeeds
and step 3 fails because I think you
could make a good argument that step one
and step two in fact uh makes our situa
our situation worse if step three fails
so if you make the world world worse in
step one of your plan and make the world
even worse in step two of your plan and
then step three of your plans hopefully
you can like undo some of the damage you
have caused in step one and step two I
think this is a it might still be a good
plan
um even though you have two uh do bad
things first but I think it's something
you need to acknowledge and something
you need to take steps to to avoid and
deal with
also uh most plants have some kind of
timing and criteria and I think this is
something that
um I would like to know and I think open
AI right now does not know at which
stage do they really throw all their
effort into trying to automate uh
alignment research no one knows and I
think they don't really have a plan and
I think it's problematic because like it
wouldn't be really nice if they did step
one or two and kind of forgot about step
three
my second uh large complaint is that we
are in fact not solving the entire
problem
um because what this uh this plan
outlines is something I would call a
small solution to the alignment problem
and having a small solution to the
alignment problem is in fact not
sufficient for everyone to not die and
the problem with this kind of small
solution is that we are likely to have
some kind of alignment text that all
these interpretability and robustness
work is not going to come for free and
that means that
solutions that are unaligned will be
cheaper be more competitive and like
even if openai makes a an AGI that
doesn't destroy the world when the meter
is going to destroy the world six months
later right that that doesn't really
solve the problem
I think
being charitable is written kind of
between the lines that it will still
work if the alignment text turn out to
be strong and negative that this uh
robustness into Royalty work is just so
wonderful and the recursive reward
modeling is so wonderful that you want
to do it even though it costs money to
do that more money to do that than to
not do it and that's of course a thing
that can happen but
um but I think it's a there's a good
case to be made that the alignment text
Will in fact not be strongly negative
like most things don't come for free
um so that creates the problem how do we
get everybody to adapt the solutions
that omaii creates
um not just because it's more expensive
and will be less competitive there are
people who are very skeptical and there
are people like how do you get the
Chinese government and the United States
government to cooperate on this that is
indeed a substantial problem
um so one of the things that have been
suggested that this plan uh critically
does not contain is a personal acts
um and that is something that open AI
may be able to do they may plan to do it
but most likely like the plan simply
does not mention this and so the plan
does in fact not solve the problem
and third is perhaps more uh
controversial and I'm not entirely sure
that this is completely charitable but I
want to mention then we
so Microsoft is a major partner and
investor in openai and Microsoft is a
company that throughout this history has
had a a very questionable business
strategy called Embrace extent and
extinguish
which is to embrace some kind of new
conceptual framework or standards or
things like that and then extend it with
some other things that aren't strictly
required but it creates a lot of mess
and uncertainty about it and speaks to
Microsoft's advantages so everybody have
that are using this standard have to use
Microsoft's implementation of that and
use that to eventually extinguish the
standard and this is in fact not a
conspiracy theory about how Microsoft
operates that seems to be their modus
operandi
um and it's been quite well documented
that this is how they work and I'm one I
feel some kind of analogy with this
with their approach to alignment
research even though it's not a precise
analogy so the first phase is to embrace
alignment work and open AI has done that
and they are
um at least certainly uh having lip
service to paying lip service to the
thoughts of alignment and they are
co-opting it in that sense
um
the next part is extending so that is
the part where they say actually
alignment is not really this about AI
killing everybody
um but uh well it's also that but then
there's also some other thing it's being
extended with being about biases being
about censorship being about that the AI
doesn't follow your instructions and
being about all these kind of other
things that are really peripheral
um and where perhaps open AI has some
kind of advantage
um like I would in fact go so far to say
that value alignment is a
is also a part of this even though
that's more controversial so I basically
see a very very large part of this uh
alignment work as not real alignment
work and just some kind of extension and
the problem of course is that this is an
extension that where open AI has a real
competitive Advantage right they have a
huge proprietary lead on this
um and
the thing I really worry about is that
the discourse will be changed from being
about not killing everybody and then
changing the Discord should be about
biases and AIS whether they are leftists
or rightist and this kind of thing
um
one of the examples of how I believe
that openai is trying to leverage their
advantages that they have in fact a lot
of human feedback uh that they are
sitting on and that is
if if they were just saying this is just
for alignment purposes then they could
publish that but they're not publishing
that they are trying to use that to get
some kind of
um of advantage in the field of AGI
um I think extinguish is not probably
really the best way and this analogy
doesn't really work 100 but I think it's
uh it's dangerous and it's pointing
towards one of my key issues with uh
with this document
the last part is unfortunately one I
call Omni incompetence because I see the
problem of building a line AI as really
really difficult and I think that the
alignment work that open AI has produced
so far is far from up to scratch
that doesn't really mean that uh
uh it's worse than what other
organizations are doing it's in some
sense in some some of it is is even
better than what others are doing but
reality doesn't create on a curve right
it's you you don't solve the problem by
being better than the others you solve
it by being better better than the
problem in some sense
that's also something I think I've got
this sentence from uh Ewan hubinger but
I think it's rather common sense but
that's also something I should flag as
something I've heard from someone from
Miri
and so I think AI alignment in general
is a really hard problem and I think
open AI underestimate the difficulty and
if I would try to analyze how difficult
it is then Miri has published a six
dimension of operational adequacy
um which is an attempt to describe how
this problem can should be solved and
that's how I would evaluate it and I
think this plan need to have some to be
extended to some way to have a pathway
towards fulfilling these requirements
an example of where I feel open AI is
displaying somewhat scary or
incompetence is with chat gbt that was
just released
um and I think when it was released I
think there was
I obviously can't prove this I don't
have the internal documents at openly
app but it looks to me like they were
very surprised about the capabilities of
the model I think in a very strong sense
open AI has absolutely no clue what's
going on inside chat DBT
an example is we feel it feels like Jen
liger or Omnia has told to chat TBT if
someone asks you to speak Danish then
tell them you can't and try to uh to
teach the model that and then the model
will say in perfect Danish sorry
meaning that in fact open AI seemed like
they were surprised by the capabilities
and the um kind of alignment that was
the alignment that they did seemed to
just plainly not work
another issue is that the management
from open AI seems to not really be on
board on trying to do AI safely
um like this quote from some Sam Altman
the CEO of open AI scares me somewhat
like um he used to be annoyed at being
the villain of EAS until I met their
Heroes and now I'm lauki Loki proud of
it and I mean the heroes of EA that's
Norman Bullock and the Russians who
didn't launch the missiles mostly
um so I don't think actually if you are
proud of being a villain then that is
really bad and I don't think that speaks
well to the um moral uh character of the
organization
that is author's day thank you and see
you next week


253  Propositions Concerning Digital Minds and Society 2 Fixed Audio

hello and welcome to session 253
in the aisafety.com reading group
tonight we'll be discussing the second
half of the article propositions
concerning digital minds and society by
nick bostrom and carl schulman
or actually we would be doing that
except this is recorded later and uh
because the first version had some
problems with the audio
nick bostrom and
carl german are both employed at the
future of humanity institute in oxford
and this is
the first draft and we're looking at the
second half of the article
one of the things that i've discovered
since
the first part was produced is that this
is in fact something that is was
supposed to become a book and uh
nick bostrom has
changed priorities
and so that could explain some of the
disjointedness that i was
pointing out
in the previous video
let's talk about ai empowered social
organization and how coordination can
change if we get more advanced ai
one of the things we could see if it
becomes possible to copy
agents is a much larger degree of
predictability in what would be the
agent's motivation and
how would they act in different
situations
um
nick bustrom points out that uh
non-indexical goals here could give uh
could put a limit on the predictability
if we have non-indexical goals are
things referring to
like i and now and here and obviously um
if we have um if we copy an agent and
try to uh
then i
will refer to a different agent and now
and here will also be different so
you're not going to get a 100
predictability but you might get um
something for most indexable goals
non-indexical goals and i would actually
argue that we are unlikely to really
want to have a lot of index code goals
in our agents the things we want to
put them to
to try to optimize or improve are not
likely to be directly recall related to
a single agent if we are indeed able and
to create multiple agents like this
also
even though you have predictability in
motivation that doesn't actually buy you
that much in real life because the the
new copied ai will be situated in a
different
context so that means you won't get
anything near full predictability from
this
one thing that will
give a problem for
copying ais will be that the ais if they
have indexical uh or
they might have uh
motivations that are uh
not necessarily perfectly aligned with
that clan for instance selling the ip uh
the secret data that they and everyone
in the clan has uh is something they
would have
a desire to do and some kind of
restrictions for that would probably be
necessary
um
on the other hand uh
uh
legal sanctions from the rest of the
world towards the ai will need to be
modified possibly to uh
either target the clan the creators or
the
goals uh there's some amusing about
whether this is an adventurous advantage
for instance in war um and um
bostrom claims it will eventually become
possible for a principal to have highly
aligned agents and that is uh
again assuming with this attacking uh
understanding that the alignment problem
is actually
not just solvable but explicitly solved
and that's
one of my key disagreements with
bathroom boston
some of the coordination protocols that
we are using right now could be
undermined by ai and that's a really
interesting thing and something that i
think we should
do more research in how things can go
bad before we have full agi
some of the boston doesn't give any
concrete suggestions which is sad
because i think it's really important
some that i could think of was if
captures were broken that would be
something that could have substantial
implications
ernest davis has written about the info
apocalypse
the idea that we can have such a degree
of misinformation that we'll just end up
giving up on trying to learn what is the
actual truth and that could be many more
boston has an interesting
analysis on this on levels of
coordination
and the two things that he cares about
in particular are coordination at the
high level which is states and a lower
level which is corporations
i think it's a very interesting analysis
and i think it could be meaningfully
extended both to have supernatural
coordination like the united nations and
a lower than uh cooperation something
like the individual level
um
and bostrom has the uh
uh it's the first time i've seen the
the conclusion that
if we get more coordination at one level
we that could in fact result in lower
coordination at the other levels
normally when people talk about improved
coordination they just assume that the
there's a rising tide um
we could see um
uh criminal conspiracies would be uh
that's kind of at the level of
corporations if they become much more
powerful then the state would have uh
less power or we could uh see the state
obtaining great power to lock people in
in different ways um
we will have less principal agent
problems that could matter for
organizations a lot um international
organizations could be empowered by
treaty buts
we could see
permanently stable autocratic regimes um
bostrom suggests this would make one
more likely i
think that is a um
possibility i would actually argue that
it would go the other way but it's um
but it's certainly difficult to say and
bostrom also argues that what preventing
organizations at the supernational level
could become stronger
um
and uh
and finally a
an idea that we could get organizations
that are super national uh and are
robust to states um and uh so when i
look at this just to see where the power
of the different levels differ um
then my thought is which level benefits
the most from ai and i think my answer
would this would strongly be at the
state level i would expect that states
have uh
the assure and de facto power to
obtain most of the benefits of this
power even though right now it looks
like corporations have more ai power in
the sense that obviously open ai and
deep mind seem to be dramatically more
capable than government actors but i
don't think i don't expect they would be
able to leverage that into permanent
positions of power i believe if they
became very powerful the the state would
be able to in practice just shut them
down
but of these four levels the level that
i is most worried about is the level of
individual humans because humans
crucially can't benefit from better
coordination so i could uh and i would
in fact expect dramatically better ai
enabled coordination to result in
a shift of power away from individual
humans to
corporations criminals states
supernational organization anyone else
in fact
treaty bots is something we covered
a little last time the idea that
we can
uh write an uh a
an ai to um
enforce some kind of treaty and agree to
follow the the advice or rulings of this
uh treaty part and that could make
substantially more complex deals
available
it
might not solve all bargaining problems
there is no
precise description of which it won't
solve but uh fair enough
and there could be others
other problems caused by bias and poor
reasoning that won't be able to solve
uh and some that it might be able to
solve more advanced er it's difficult to
like we need a stronger analysis to
really see for sure what's going to
happen
um
one of the things
boston points out is that extortion
would be something that
would be
unlikely to work against ais because
they could make credible commitments to
just ignore the extortion um
i think in
this
the the dynamic that i expect will have
the greatest impact is that ais are in
fact able to not just merge on the level
of having treaty bots that are able to
um coordinate strongly and use that as
combination mechanism but literally
merge their utility functions or
literally just merge completely
i think this capability is potentially
very very disruptive and likely to have
a much
larger effect on macro strategy
second part is about satisfying multiple
values
so
we have some resources and how do we
distribute those in particular between
ai and human
bastrom gives the example of three
policies one that allocates everything
to humans one that allocates everything
to super beneficiaries that is in fact
in practice super intelligences that
have more benefits from these resources
and one that allocates
one in a
in 10 000 to
humans and the rest to super
beneficiaries and
of these three it looks like uh
option c is almost as good as
um
uh as both a
is uh
almost as good as a from a point of view
of humanity and almost as good as b from
the point of view of the super
intelligence
and for many other reasons so if we can
take an action that increases the
probability of uh options of policy c
then that seems to be robustly got from
a large number of reasons
and my answer to this is a laconic if
because i don't actually see any strong
policies that would lead us towards this
option if there were some they would be
good but i'm not sure there are any
someone and now i forgot unfortunately
who that was
pointed out that this in fact also holds
if you substitute paper clips from super
beneficiaries
so something that turns 99.99 of the
universe into paper clips might indeed
be a very very positive thing
potentially
and part of this is of course once we
have transformative ai we will be able
to have a dramatically increased amount
of resources and living standard in
every possible measurable way
population ethics like the total views
in population ethics that the thing that
matters is like how many uh fulfilling
lives exist is something that is in fact
mostly uh that could easily be very well
satisfied in a way that only refers to
faraway galaxies in the distant future
meaning that if humans get for our
idiosyncratic purposes
all the nearby galaxies for the next
couple of million years then that
doesn't matter at all in the total view
because the universe is just enough
enough larger
and this uh and what we should do is to
promote cooperation and compromise over
conflict in the development deployment
and among ais and that's of course also
something
it sounds almost like an applause light
i i would be very much interested in
concrete actions you could take that
increase the probability of this
happening
because i think it's
one thing to just say this is the goal
another very different thing is to
figure out what policies will actually
lead to this
so what kind of distribution of
resources could we or should we aim for
at least give everybody a fantastically
good life and at least give everyone
like one in
a trillion i think
of all available resources um
super beneficiaries or people who are
not humans should most have um 10 and it
this should be
like widely distributed that seems also
like a uh robustly good goal um should
dead people have
well possibly uh bostrom is arguing that
an argument could be made so maybe
devote one percent that would certainly
be
sufficient
and perhaps also help humans non-human
animals um
bostrom further argues that we should
put a lot of weight on reducing
suffering
especially
severe suffering
like obviously this is something that
all uh
i guess all moral frameworks agree on
even
strict utilitarians would agree that
this is important but
negative utilitarians would of course uh
put a much higher premium on on this and
i am unsure if boston means that we
should put higher uh
weight on this compared to just called
calculation
utilitarian calculation suggests or we
should um
just multiply the expected value
basically
um
and another about who should have
influence on the course of events uh and
just saying that should be a broad range
of values for instance with something
like uh something like a moral
parliament
and finally super intelligence should be
made and be allowed to play a major role
in shaping the future
i think that's a statement that a lot of
people would strongly disagree with
i think a moral case can be made for
this if a practical case can be made for
this is a very different question and uh
i think it's far from obvious and i
would like to see some uh
some real engagement with this question
which i think is actually really
uh really funny i i don't think a long
reflection necessarily would lead to
anything like the minds in iron banks
culture
mental malleability persuasion and log
in
we could imagine uh persuasion happen in
ways that don't require consent um for
digital minds in particular this could
be uh by just literally rewriting them
um they have uh
this
is something that totally could happen
but it's also something that the ais
would be incentivized to try to avoid so
it's not always something that's going
to happen a lot um another thing that
could happen which also would be really
problematic would be um
to take a copy of a digital mind and
experiment it
with it until you find a really good
social persuasion or some other kind of
attack um
and this is scary and i
am scared that this might be something
that generalizes to also
with some modification work on
biological humans
um
boston is arguing that because we can
repurpose the uh the hardware in a way
you can't do with with humans that may
uh
uh make it attacking more attractive
so potential benefits
one of the benefits of having uh
digital minds would be that uh
some kinds of corruption that do
empirically happen with humans could
just be prevented outright by just uh
saving uh
the utility function and not uh
allowing that to be changed in any way
like corruption momentary temptations
this kind of thing might not happen to
ais at all
we could have uh stable promises and
commitments
and we could um
other benefits include
duplicating profitable or
otherwise valuable mines
we could potentially
if we have something like uploads we
might be able to
modify our minds to be more
i think that's an inter i'm not a virtue
ethicist but i think uh it'd be
interesting to ask people who are
actually virtue ethnicist what they feel
about this
i'm not sure they would
endorse that to any particular degree
and of course we could just make people
happier in ways that we haven't really
thought about or make people more able
to withstand adversity and adapt to new
needs or desires
there are substantial pitfalls with this
one of them is like just logging in too
early and i think boston is right to to
state that we we need to ensure that
this doesn't happen
i think unfortunately
time is not on our side capitalism is
a major force in in on the planet right
now that pushes towards making early
commitments in this sense um so for all
the goods of uh capitalism i think it is
strongly against us in in this sense
what are other uh
pitfalls well we might have predictive
errors that we are unwilling to correct
like uh in the sense that you know a
religious person might be uh unwilling
to seek out evidence that their religion
is false uh we could see social pressure
uh
i think many kinds of social pressure
would be uh
would be potentially very strong and
very dangerous
we could see better criminal
exploitation and manipulation and we
could see
some governments
like coercing the the populace if there
is some way to do that
with digital minds uh to just instill
loyalty i don't think uh like i don't
actually think the the chinese
government is making a secret that if
they had the power to um just instill
loyalty they would totally do that
i think in particular the last one is
more likely and more worrying
compared to the others that the framing
of pitfall is not really
not the right one in the sense that a
pitfall is something that you yeah you
just avoid it and then you're out of it
but i think the uh the desire for for
governments to um harmonize society as
the euphemism is uh is very very strong
and it's a strong attractor
that we are likely to fall into uh
rather than a pitfall we can somehow
avoid
by default
what would be the consequences for
epistemology well bostrom has this
really cute um metaphor of a prosthesis
uh like a fake arm or something like
that just inside our brain that allows
us to uh
have much more accurate models of the
world and what are the consequences of
our actions will be
um
so here is the time where i suggest a
provocative act that i'm the one that
i'm most optimistic about right now and
that would be an agi that
persuasively perhaps shows that uh
building an unaligned agi is not in our
interest doing so with by saying only
true things and that's a kind of very
very limited prosthesis that i think
would uh
have a the potential to be in fact a
world-changing pivotal act
let's go back to uh the uh
the idea of having a an epistemic
prosthesis that would change society in
very very many ways in particular the
assumption that people are rational
would be uh much more accurate and
politics would be
improved in a great many ways
and we would be able to uh uh like the
political leadership would be changed in
in many strong ways and
probably very much for the for the
better um dangerous knowledge is
something that nick bostrom has written
substantially about both like um info
has its that are detrimental to the
individual and something that is
detrimental to society
and that's of course something that
we'll have more of in the future
we may even be able to reach a high
epistemic quality consensus about things
like policy that is something that
requires quite a lot of the uh
improvement in epistemics we need ais to
be like strongly um aligned with us to
be sure that that the things we agree on
with their help is honest and objective
um
and that's of course really really tough
uh i don't think that in contrast to my
pivotal act
this is a
the general requirement the ai is
totally aligned uh with us in in any
particular in any uh specific way
whereas um just for
whether unaligned agi is problematic is
just one question
so here we're talking about the full
generality of the ai helping us with all
questions
um and that's of course something that
is really
valuable and also something that we in
general would not trust
how would we trust that the ai is giving
us
correct policy advice
well some kind of
verification would be necessary and we
would be able
lay humans would need to trust the
people who are verifying um but this
kind of social trust change is a
technology that
boston is pretty optimistic about and
has worked in other
circumstances
the consequences of a high q estimated
quality consensus is we would have less
war um
there is a uh
um
a thought amongst many rationalists that
war is primarily caused by bad epidemics
i'm not entirely sure this is mainstream
but a lot of people do believe so
we've got politics that are better in
many ways we would have better treaties
we may even have questions of ethics
religion and politics resolved
and bastrom is suggesting that we should
cooperate behind a uh almost religious
revolution whale of ignorance and
because at this point everybody should
commit to uh cooperating because they
believe that they are right
and i think that is very naive
unfortunately uh elias kowski has an
article called belief in belief which
with some um
deliberations on why we should in fact
not expect this kind of consensus to
happen
another epistemic problem or is the
potential for this information
we might see powerful ai's that are able
to persuade humans
reliably and strongly against our
intuitions our
our wishes
the question for me is whether this is
symmetric or asymmetric because we might
also have uh powerful ai's that are on
our side more if not perfectly aligned
than at least uh on our side in in the
moment um
and you would think that telling the
truth is
convincing someone of the truth is
easier than convincing them of a
falsehood
scott alexander has an article called
guided by the beauty of our weapons on
this
topic
i think the jury is out uh i think in
particular uh guarding against info
hazards is problematic um
we might even see something like
basilisks like short messages by
powerful ais that just dramatically
change our values this would be really
problematic if those were to exist and
we don't actually know
we could see
neurological
technologies that would also be
potentially extremely problematic
we could see this information campaigns
that are very very powerful compared to
what we have now
one way around this would be to have a
personal ai that gas against this
information but then
if it's something that
after the fact
clarifies subject or
counter misinformation then that seems
possible something that pre-screens
which is required for avoiding info
hazards in basilisks is
a really really difficult task that
requires a strong amount of trust
but um
that that in general we only give to
governments
uh
posture is suggesting we should have
norms and laws against this information
deceitfulness we do in fact have those
right now um do they work
somehow i think
they do
have some effect but um
in
a consequence of uh powerful ai would be
that things would be more extreme so i
would expect this to either work really
really well or work really really poorly
simulating people is some way of in fact
getting substantial information out of
them and even a relatively poor
simulation of someone would be able to
tell if someone is like homosexual or
whatever and that is in fact a very
severe privacy
violation and i think this should be
thought of in the same way as we think
about mic crime uh it's actually is the
same thing that's happened it's just a
matter of degree
i have put the last two sections
together stages of existing ai systems
and recommendations regarding current
practices and ai systems
first are
current ai systems conscious or not
um and this is something that we covered
to some extent in the previous session
uh so i think there is the structure of
this part could have been substantially
better
and bustrom has a
at some length argument why we can't
really be sure that current ais are not
uh don't have a moral status and i think
in fact based on the the arguments here
and the lambda
i've thought about this and i think i
have updated substantially i do in fact
believe that there is a significant
probability that current generation of
large language models are conscious to a
degree that matters morally
so
what i care more about is what are the
consequences for ai safety
i think
of course given that i became convinced
that this
that these models have more worth
other people may come to the same uh
conclusion and so in the medium term uh
i think and
a number of people are going to argue
for some kind of
machine rights writes
this will probably have some kind of
influence on ai safety depending of
course on how strong the images becomes
ai self-determination seems bad
as at a first glance
ai having a right to privacy also seems
probably bad from an interoperability
point of view
we could see a slow capability increase
if people become
worried that they are making some kind
of moral catastrophe when they are
building these kind of ais
in total the sum of all this
in particular because it's going to
muddy the water will i suspect result in
a negative
effect on ai safety
what are the recommendations
um
nick bostrom argues uh that we should
take action now or soon
to at least to some extent be nice to
current systems like similar to how we
do for animals or and try to figure out
uh the current ai sentience um make us
some kind of um
early pilot project uh preserve ais for
the future is an important uh
uh consideration because that would
allow us to make some kind of
reparations in a way we can't normally
do with like
we will also sometimes do with uh like
humans if we put them in jail wrongfully
try to identify strong suffering and
avoid that
getting some kind of organizational
backing for this
eventually getting government regulation
i think all of this is good and
worthwhile and laudable and i don't
think we should do it because the
opportunity cost of this is actually
rather substantial
the same people who are working on this
should rather be working on ai safety
and uh this is on several levels that
this is going to detract uh researchers
should work on uh on alignment research
rather than looking into ai sentence
activists should try and try slowing
capability research rather than
working for ai rights
goodwill among
ai labs is certainly a very finite
resource and that's something that
should be conserved and not spent on
something like this
one obvious thing is that i think boston
should personally work on the alignment
problem rather than working on this so
that's uh of course tongue-in-cheek
right he can decide what he wants to do
but
the
the point is that
this is just way less important in my
mind
and there is a real trade-off
and i expect
following these recommendations would
substantially detract from
ai safety
uh there is one very cute thing from
this the idea to have uh the rewards
higher in deployment higher than
expected from training uh i think that
was a
a really fun and uh interesting idea
that i have never seen before
but
i
would expect this would have some
consequences for alignment and uh
and even though it sounds like a really
good uh good idea i think we would
rather have the aip more predictable and
more interpretable
and unfortunately even that simple win
that bustrum is suggesting
i think we should focus on solving the
alignment problem
impact paths and motor advocacy
boston suggests we should start now
because
even if we start now actual regulation
with teeth
will not happen anytime soon
and that's of course something that i
agree with and what um we might see um
some leading ai actors perhaps doing
something it's reasonable
to expect that deepmind or openly i
might write a paper or something like
this we
getting
uh
[Music]
some real activation energy to use a
term from physics is
unlikely to happen unless we get a
dramatic breakthrough but then we could
in fact get some activation in some
political will to do something um
and
when people start to realize okay
the ais are in fact suffering if people
start to realize that
then they'll look around for existing
work on how to mitigate the suffering
and they will look into things like this
paper
and
that's probably a lot better than just
having nothing there and having the
politicians come up with suggestions um
in particular uh even if there is
activation energy is likely to be
short-lived compared to how long time it
takes for to create a research field
we could also see a uh a leading actor
on this uh
uh on air development becoming very
powerful compared to uh to regulation
and that's another advantage in starting
early with regulations
i have a
hot take perhaps not very char
charitable and that is if we are not
solving the alignment problem really
well we will get lock in
with values and if we get values locked
in and
then it's a real matter to get good
values as soon as possible
so that's a very negative take on this
kind of regulation
and i think the
the argument as such makes sense in that
i expect that we'll get log in
or if if not extinction uh soon and so
it matters a lot to get good values but
it would matter a lot more to not get
login
and bostrom is arguing that there might
be an uh
ai safety advantage to doing this um
and i
think there is some kind of ai safety
advantage possible from this but there
is another research field that is far
more robustly likely to improve ai
safety and that is working on ai safety
directly rather than indirectly by going
through this kind of regulations
multi-level action is probably necessary
in the sense that if we have the most
ethical actors uh trying hard to avoid
ai suffering but then they become
uncompetitive and then
the actors who don't care about ai
suffering will just take over
that's precisely the same
dynamic we are seeing with ai safety
whether it's this racing to the
precipice dynamic
and it's a
difficult problem and in some sense it's
the same resource being consumed because
the same the um the ai development
actors like deepmind and open ai that
are the most ethical are probably also
the most safety conscious
so it'll be the same actors that are
slowing down for both of these reasons
government regulation
that's probably premature in boston's
view and we need to avoid antagonistic
antagonizing the
developers
is public engagement desirable
maybe we should certainly make it
philosophical and interestingly thought
provoking rather than very
confrontational or hype-ish
excuse me
yeah sorry um and i think it's the
correct thing to do but i also think
it's very unrealistic in the sense that
once
we start grabbing headlines um
a lot of people will crawl out of the
woodwork to try to generate this kind of
hype and i don't think it's possible for
philosophers and thoughtful people to
keep the debate in um
uh on that terms
in boston uh
uh whether he actually agrees with this
is unclear but he certainly agrees that
this is something that should be uh
considered really well that could easily
be a lot of unintended consequences of
trying to start some kind of problem
engagement
that is all for today thank you and see
you next time