The Inside View #2–Connor Leahy

welcome to the inside view a podcast
where we forecast technological progress
starting from our grad level intuitions
i am your host
michael thrasi for people watching this
on youtube
what you're saying is a screen recording
of how i randomly met
conor lee at the internet party
conor is an ml researcher at adolf alpha
a german company shaping european
research and development
for the next generation of generalizable
artificial intelligence
he's most well known for being one of
elluther ai's founding member
a decentralized grassroots collective of
volunteer researchers engineers and
developers
focusing ai alignment scaling and open
sourcing ai research
for listeners not familiar with ai
alignment it is often used to point at
research aiming to build scalable ai
systems
that do what we want them to do
my current experience with luther was
mostly through their discord
where people coordinate on open source
projects discuss deep learning research
and most importantly exchange deep
learning gossips and 9 000 iq memes
in my opinion elizabeth's flagship
achievements are to have
open source both one the code to 20 gp3
like models
referred as gpt neo and gpt neox
and two the pile a clean dataset they
created themselves
to train those gp3 like models
in the first part of the podcast we chat
about how to speed up gpt's returning
how conor updated on recent
announcements of large language models
why gpt3 can be considered agi for some
specific definitions of agi
the obstacles in plugging planning to
gptn
and why the brain might approximate
something like back problem
we end this first part with settlement
of priors adversary attacks
just pascal mugging and whether direct
work
on ai alignment is currently tractable
in the second part we chat about how
this current projects at luther
multiple scenarios and reasons to work
on technical ailm research
without further ado conor lee
hello again hey can you can find yep
okay so like just let me give you some
context about the the whole
cat thing um i want people to talk about
your timelines and
i want people to give inside views about
what they think about ai and not censor
themselves so
obviously you're one of those candidates
fall to that category yes i have no idea
this is my first okay this is my second
time recording one
so i'm pretty much new to it so hello
connor
how are you doing hi doing well
cool yeah um i i recently saw one of
your
retweets because you're quite prolific
on twitter and it was about like um
a chinese paper that uh went on for
a few months and was recently translated
to english and apparently it's the new
thing
um what do you think about it so i
assume we talked about the rotational
bags yeah exactly that's
fun that's a funny little story so i'm
not super involved with that um i just
kind of saw it from the sidelines
basically what happened was a chinese
researcher
uh like over the food course for a few
months uh wrote a few chinese blog posts
about a new transformer technique about
a new positional encoding technique
specifically
that seem to be have interesting
properties
um someone at a luther which is like the
open source
collective um part of yeah uh stella was
one of the people involved i think
pratal
was the one who first uh made us aware
of it if i remember correctly
uh he posed like hey this seems
interesting so usually google translates
like reverse engineer what they were
doing
so apparently this has been like um
spoken about in chinese nlp circles
and we ran a bunch of experiments and it
was the first
like uh simple quote-unquote tweak
to these kinds of things that we've seen
in a long time that seemed to make like
any appreciable difference so especially
for small models it made convergence
uh so training time much faster uh
the uh seems to diminish in large at
larger sizes but it's still
like significantly a few percentage
points and such and like speed increase
and such that's pretty cool
so we decided to write a english blog
post about it reach out to the author
like hey we really like your work we
like to write a blog post about him
so then he released his paper in english
and we released our blog post at the
same time
well we i i wasn't involved in reading
the book
right okay so you helped um publish a
blog post and i think like since today
there's like a paper
uh builders because it's more about
speed than and actual uh reducing the
button is the button like in
kind of the nvidia link and like all the
uh optimization
when training transformers or so the
bottleneck for training transformers
is um for scaling it's um the
the interconnect speed yeah as you
described so like the
connecting between nodes but also within
nodes like the enveloping and the switch
between the individual gpus and also
like the uh infinite band link between
which you you want basically you want
infiniband ethernet is pretty difficult
uh between nodes to get good scaling the
bottleneck is ultimately
compute so flux is just pure raw gpu
flops
um nowadays if you get expensive
machines if you know they have like
infiniband and
whatever you can scale pretty well and
then just depend on how many gpus can
you buy
i i was just interested in like the
specific trick
that this chinese paper did yeah
specific trick we haven't applied it to
very large models
um it's basically just like it's killing
a free small improvement
it doesn't change anything fundamental
about the architecture really it's more
like a
you add it's a different way of doing
positional encodings in the network
which seems to give a pretty significant
speed
speed increase um not in execution but
in training time so you need fewer steps
to
convert to the same loss which is pretty
significant for smaller models
and it like loses some efficiency with
larger ones we've tested up to a billion
parameters i think we haven't tested it
with larger models i think we're
currently in the process of testing it
on even larger models we don't have
results for that right
so it's it kind of finds a better way a
better path
like the training um yeah
you need less steps to get to the same
amount of training at least for small
models
and so more generally like recently um
i think facebook uh open source uh their
recommender system
and they said there were like trillions
of parameters uh microsoft something
similar and
and nvidia had these like ballsy claims
about
training gpt three in a few days and and
maybe scaling to 3000 americans
so yeah what do you think about all
these like new um topics
around uh scaling up yeah so yeah you
have to disentangle some of that so the
facebook thing is i would not ex you
can't compare it
to a like dense model of gpt3 like
the parameter numbers don't mean
anything if it's a like recommender
systems are very very different
architectures
they're extremely sparse they don't use
all the parameters
the same way gp3 does so you can't
really compare them because
this is a completely different
architecture um
for the um nvidia thing that's
scary that's something that actually
updated my um timelines quite sick
like non like non-negligibly um
on discord before you were saying that
um you you didn't quite
update from uh if i remember correctly
wall-e
or gpt-3 because you already had crazy
timelines and so you didn't update
that much and now you're saying you're
dating from nvidia
i am not also not significant because my
timelines are so extremely short
i i've like updated my confidence for
like you know like uh
like 10 to 15 or something which is
significant
well okay yeah so it's like five times
um
so yeah basically what nvidia talked
about is that they're introducing new
um systems that they claim will be able
to train
um trillion parameter models in three
days
which technically might be possible even
with current hardware
if you have enough of it i haven't run
the numbers on that
sure um because yeah yeah i i think we
don't really know how
all those optimization works um
in detail like at the our level so yeah
i just want to like bounce off on a
bunch of claims you made on the
another podcast called uh street ml
podcast i guess
um which were both funny and
true which is uh let's start with um
gpt3 is aji i know and i know you did it
for the memes but
um yeah so that was a bit for the meme
of course
it was a fun podcast you know so we said
there's silly things
um but yeah i do believe that um for a
certain definition
of agi the thing is agi doesn't really
mean anything anymore like controversy
means
so when i say agi what i mean is it is a
um it is a qualitative it's not like a
binary thing like oh it either is agi
isn't it it's more like a spectrum
i'm like um for me artificial general
intelligence just means
it is a system trained on like some
simple objective
that is somehow able to solve many
useful problems
and in my definition gp3 is trained on a
simple text predicting
uh objective but is able to solve many
very valuable tasks
summarization and text writing and
timing so
yeah i i think that's like i talked to
one engineer
may i uh gave me the definition of like
automating 50
of knowledge work which is kind of uh
relevant to like our economy
and not that hard for you know language
models uh
right now so i think and and like to to
bounce off of like
like let's not speak about intelligence
but let's speak about useful things
and what matters um i loved i loved this
rule you gave which is which now i'm
calling conor's rule
which is to never mention the suitcase
the suitcase word intelligence
so let's apply the kind of rule in a
color on our podcast and not talk about
misleadings
um so okay so we we had this prior
conversation about
uh gpt3 like how should we go from that
to like an agent
that could interact with the world and
um you know do planning
like uh do actions and stuff and i
remember you're saying
um that one of your crocs was so a crux
is for people who are not familiar with
it is
a fact that will change your mind about
um
about the world uh if it happens or not
so a crux
conor had was whether um reinforcement
the part where we tried to uh merge gt3
with some kind of reinforcement learning
agent
uh was not the hard part and that the
hard part was actually coming up with
like this transformer architecture
which can scale a lot um so
yeah do you do you have um any new takes
on that or do you want to expand on
on this yeah i mean you hit it on the
head so
there is obviously some things that uh
we probably would need for gp3s igbt
and you know future model to be
considered so i prefer the
transformative ai when talking about
your automating knowledge worker heavily
economy or whatever whether we call it
that or not whatever
but having a very powerful system um in
a sense
you could argue whether or not gp3 has
any kind of goal system or not
like you could make the argument oh it
can simulate a goal agent even so itself
has not it doesn't have a goal you can
make it right a character that have a
goal
and out of an action that way yeah but i
expect in the future the way this is
going to look as described we'll have
dp3 or gpn as some kind of world model
that a rlag interfaces with to
make predictions of the world and make
better decisions and i expect this to be
easy
um i mean easy in big scare quotes still
you know it's going to take time it's
going to be hard to
find architecture whatever but easy in
the sense that i don't think we need any
fundamental theoretical breakthrough to
build this yep
um i guess i guess the breakthrough
could be
inside of like reinforcement learning uh
at the moment which is um just like
it's not very simple efficient in terms
of um being confronted to like the
normal like complexity of um you know
sensory perception and stuff
yeah but i mean you can look at like the
dreamer architecture for example which
is super sample efficient because it's
trained inside of a world model
i think the reason rl doesn't work is
because they're not using world models
interesting yeah and i need to look at
look more
in new paper uh yeah it's like somebody
based reinforcement learning has
becoming more popular
lately and i expect that to continue but
but still like then the actual you know
one model like the
the gif that i remember of like in
dreaming up
like driving a car um it's still like
a few actions like maybe you can
collapse and go right um
when you're trying to like ask a world
model um
when you try to like do prompt
engineering and talk like a language
model
like your action space is is huge right
so even if you're
a bit more subtle efficient it's still
very hard
well it's pretty clear to me that
prompting is an inferior method of how
to use these models
so one of my hot takes or one of my
proxies here is that i actually
think gb23 is much smarter than people
think it is
because of the task it's actually
trained on
i i conceptualized gps3 as an extremely
intelligent model that is pretending to
be a median internet user
yeah so i think the way that inside the
model
is actually a lot of useful information
that
you it's like hard to get or like you
know a prompt doesn't necessarily
um able to extract easily and we've
already been seeing this with like
prompt tuning and whatever that like
continuous prompts that are like trained
through background or whatever
that we're getting better and better
techniques they have better performance
out of these models
so i i expect there to be as i always
write the reinforcement learning age to
not literally
give a natural language query to the
model but rather to have like a
differentiable
um interface within hidden states okay
so you're essentially saying that
uh like language like human language is
a bad way of interfacing between uh you
know ai models
which is yeah yeah this is pretty yeah i
agree with the claim
um and um and yeah
you also said in this other podcast that
um humans were
not yeah voldemort like the world we're
not
uh able to say they're not intelligent
um and and it's also that um we're
like humans are approximating gp23 not
the opposite
so yeah can you go more on that because
i found it fascinating
so that was definitely something i've
been saying for the beam but
i would say i believe that with like uh
i do like believe it more than other
people do but i have like very little
evidence for what that even means
so that was like somewhat of a meme but
like in a sense um there's something
it's like the more we see how about like
there's recently been
like uh several papers one of the
cocktail networks is all you need
where you show that like predictive
processing predictive coding
type architectures can um approximate
back prop in arbitrary computational
graphs so basically
we see these like uh predictive coding
signaling in
neurological neurons and we now know
like mathematically these can
approximate a backdrop signal
and just empirically we've seen many
many times that like if you
um try to implement like biological
um constraints that exist in biological
networks
they often they generally don't
outperform a pure back prop
so my current working model in my work
working apologies
is that is that the backdrop is not
literally implemented in the brain
not because backdrop isn't good but
because it's like not feasible
because of hardware constraints is that
if the brain could implement backdrop it
would just implement backdrop
so because backdrop is so perfect
because we can implement it in our gpus
is actually
a more pure version of what the brain is
trying to
approximate so i actually interviewed um
uh
fristen when one time about this too and
asked him what do you think of a back
problem he said yeah it's all
ends up to be the same thing it's all
minimizing free energy and
variational uh bayesian models and
whether or not you believe fristen on
this card or wouldn't think it's trivial
uh call first okay he's a very famous
neuroscientist who has a
kind of like a unified fear of the brain
which is also kind of trivial but also
super complicated
it's a long story but basically his
whole point is that although the way the
thing the
brain does is it it minimizes
variational free energy
of a bayesian model which is a very
fancy word to say it minimizes loss of a
predictive model
sure and whether you do that with
backprop or predictive coding
or some other algorithm it doesn't
really matter they're like all
in not like literally equivalent but
they're like they're all approximating
some kind of like bayesian update
and backprop seems to be a space
especially pure
especially efficient way of doing kind
of like approximating some kind of asian
update
at least for the kind of systems we use
so what i meant by that statement is
basically saying
i think the brain may be trying to
approximate a like a numerically
pure backdrop it has to do all these
hacks
to because um you can't have you know 32
bit floating point numbers in neurons
it's just not just too much noise so you
have to do like all these like error
correction and all this like
fuzziness but then there's like also
like um
physical um you know tension electricity
being continuous so it's
it it also has more flexibility and
flexibility
than us like it doesn't have those
problems of you know numbers becoming
too small and
like a double approximation um
it has also like more flexibility on
some points uh do you mean the brain
yeah or i disagree i think it brings
less flexibility than we do
the brain can't represent an arbitrarily
precision number it's all there's too
much noise
the brain can't show arbitrarily large
or small numbers you know you can't just
have a neuron being called an arbitrary
large number or an arbitrary small
number there is a
in many ways if you just look at neurons
and just like look at them you know
spiking
is pretty obvious to me that they're
trying to approximate a digital system
that's the whole point it's trying it's
not an analog system
and if there's one thing we've learned
from the history of computer science
is that analog systems never work out is
they always sound great on paper but
they never work we always
default back to the digital system
because they work
digital systems are just because of the
error correction
of um these things you don't have these
compounding errors that you get in
analog systems
and yeah what about um
like where what what about quantum
physics like
maybe maybe there's something the brain
is
a huge wet mess warm mess
this is the worst possible scenarios for
quantum effects to have any meaning
i see there's currently no evidence
whatsoever that the brain
significantly exploits quantum effect
and even if it did
um i don't see how it's like you can't
have any long-term retained women so it
might be like some kind of like small
effect something like protein levels but
i mean quantum isn't magic either it
would just get bring us from the p-class
complexity to bqp
which might be i mean i could imagine if
the whole brain was at you know
zero kelvin and was an extremely
complicated you know
matrix of superconducting circuits okay
then maybe there's some quantum effects
going on
but i don't i can't see how a quantum
tangled state would not decay here in a
system like that
makes sense um yeah i was just trying to
think about
um like the best still man
like counter arguments like what would
be like something that would make you
change your mind on like
um like a brain approximating some other
function
um so yeah maybe it's something like in
the space of quantum computing
or maybe maybe backdrop isn't optimal
yeah what would be like like the best
stillman of the other position
i mean it's it's less than i'm saying is
approximating one specific
it's more like in my is that all these
groups kind of collapse the same thing
it's like whether you use
quicksort or insertion sword or whatever
sort
at the end of the day you're sorting the
list and there's many different ways you
can do it depending on what hardware
what your data looks like whatever so
i think what the brain is doing is to a
large degree at least what the neocortex
is doing is that it's building
predictive models
of its environment it's it's it's you
know it's it's minimizing the loss of
something like bayesian variational base
model of its environment
whether it does this literally sit
through backdrop or literally
through predictive processing or
whatever is kind of
just an implementation detail like what
these hot field papers showed us is that
they're mathematically they all kind of
average out to the same thing
so is that okay so are we saying
kind of that very converge towards
what would be uh through bayesian update
from some kind of symbol enough
induction and they try to like abduct
update towards
um and and like backdrop is
um maybe the easiest or
um elegant way of approximating some
vacation reasoning
at some point that's my work hip-hop so
i mean base is like provably the optimal
way
on updating your information we don't
have to bring solenomen off into that i
have some spicy opinions
what i'm not going to have to get into
easy give me give me the spicy give me
this
i think salamando is mathematically
incoherent i think it's actually not
like it's an interesting idea to think
about but i don't think you can draw
as many conclusions from it as people
think you can because you're
basically my my um
this should be congress law the minute
you introduce a halting oracle it's no
longer true
[Laughter]
you're not allowed to introduce a
halting article that that's just it's
for both
it's like introducing you know p equals
not p you know you're not allowed to do
that
you mean because he he doesn't consider
altering our goals
and the problem is that salam enough
induction is fundamentally
not halting it is healthy complete to to
actually
execute or to get the distribution of
salaminoff
the price the salomon of prior can only
be computed by a halting oracle
it cannot be computed in finite time and
that
and there's a big difference between
something that takes uh infinite steps
or takes you know arbitrarily many steps
and something that takes holding steps
to do and solomon off is takes a halting
oracle to be able to compute
because it's you know it's running over
universal turing machines over all
possible programs
and many of those will not halt so in
order to get
the distribution over the non-halting
programs you need to have a halting
miracle
yeah i need i need to look back into the
paper but i kind of i cannot remember
from his first paper on that i think
1969 or 1960
17 sorry he was definitely trying not to
go into
uh not helping territory and like he
like says
we're going to like take only um you
know
turing machines at fault um yeah but
there's no way to prove whether a turing
machine halts or not
like but if you give if they give you an
arbitrary sure way you can prove it
so you can never construct this system
so like i'm a constructivist
i think so like i would so okay this
guess is into deep math logic
whenever i reject for example the law of
excluded middle so like there's
in in in classical logic you have like
you know either p
or not p i reject that i think that's
not true
um and because basically with that you
introduce a calling oracle into your
uh logic system sure you can also
introduce griddle and completeness
so to have it to have a consistent
system of mathematics
you have at least in my opinion um you
can't have the lava solid middle and you
can't have non-hull
like you can't have a holding oracle in
your system but to construct
uh um the distribution of a salon enough
prior
you need a holding oracle
oh okay okay okay um
i think i think i guess what you mean i
need to read the paper to understand it
better
um and to be clear i'm not an expert on
this i just
you know i'm this is just my intuition
and what i've picked up from learning
and reading about this kind of stuff
i also think that the universal prior
has like probably very strange
properties that we actually don't want
our agent to have
also there's the problem of the choice
so this might be the
less weird explanation of why i think
salam enough prior is incoherent so
i think hudder actually recently
released a paper on this like 2019
that you just to define your salam
enough prior you have to define
a universal turing machine but there are
infinite possible universe turing
machines
and your choice is arbitrary so there
are infinitely many possible
enough priors and they can and depending
on your choice of salaam enough prior
you get your age like an actually your
ic agent can actually perform
arbitrarily bad on any distribution
interesting yeah um like that's why i
say it's incoherent because like it's a
long enough claims there is one
unique true universal prior and that's
not true there are infinite
of them and none of them is privileged
over the other
sure yeah yeah i see i see where coming
from with this uh marcus paper
um from what i remember reading the
solomon of paper
he he was trying to get to the point of
something very
like understandable by human where you
i think you you you when you know it
already but like you have a sequence of
like a b b a b b
a a and then you say like oh this is a
permutation of a and b
and so you just give like the the the k
that corresponds
to the cardinal of this permutation
among
n factorial and and then like um if you
if you repeat that enough uh with a
better three machines
at some point you get something purely
random
right so um so i i i feel like somehow
this like permutation thing
um where like this permutation encoding
is some somehow the like universal prior
uh natural like he was aiming for and um
but maybe maybe utter like shows that
you know um you can adversely attack uh
those things
um and yeah yeah yukowski uh showed that
you can mug
you can do pascal mugging and so on um
yeah
and like did you like are you actually
concerned about plastic mugging
or uh is it for you like a like a
funny top experiment i mean
pascaline mugging is kind of like a
spectrum
like some people say climate change is a
pascaline magic mugging technically
you know because like someone comes up
to you hey 50 years the world's going to
burn so
you know pay me money to reduce co2 is
that a pascalian mugging
i wouldn't think so i think that's just
reasonable you know thinking
um yeah so just sorry to have i'm just
like to introduce to
listeners that are not familiar with
that ask maybe you can like explain what
that's coming is
um sure so the bascallion mugging is
kind of like a
like a thought experiment where someone
comes up to you and gives you some
really ludicrous
um scenario that you think is really
unlikely but
threatens that the scenario will be like
extremely bad
unless you do something for him it's
like the classic is like you know a
mugger comes up to you and say hey
i'm actually god um i'm going to create
infinite people that suffer infinitely
unless you give me five dollars so you
should give me five dollars
one important point though is that
they're not saying something very weird
um where um because like in in
yukovsky's claim
is that um they say i'm going to give
you
three uh arrow arrow ro3
uh dollars or i will kill three our
arrow three
people which is um newt um notation for
saying very very big numbers um and
those numbers
are actually very high in the salomon of
prior because you can build a simple
train machine
that compute those uh but they're like
so big
that like even if you consider every
possible scenario and you try to like
say make give a probability to like this
guy saying the
the right thing uh index detected in
expectations value
because your prior is so high compared
to like this claim
you will say like oh i need to give
money to this guy
um and so in my in my understanding for
ai
um so the general claim is like if when
we
have a prior like imagine like even if
we are like transformers like other
architectures
and they have a prior level like what's
reasonable and they do planning
is there a way to hack them with
something that has
high prior for them and and will trick
them into like believing it's very high
expected value right um i mean that's
what bizarre examples are
in a way you know you can find like
weird permutation of images
that trick the the never to be like
super confident
of like something yeah exactly
so you know robustness to um
out of zero examples is always dependent
on your distribution it's like you know
you can make an agent
robust to certain distributions of
events
and i don't know i'm not an expert on
this so i know how hard it is to make
them
i assume there's probably some i don't
actually know if this is true
but i assume there's probably some proof
that like for a large enough set
of environments and unfounded agent you
can always find like something that they
can't handle or does something weird
i'm pretty sure that's probably the case
but i don't know that's actually true so
don't quote me on that right so
something like no launch but flow for
like ariel
yeah something like that i'm pretty sure
a theorem like that exists but
i wouldn't actually know um okay
so right now i'm going to just pretend
for a while that i'm not very
um convinced about ai alignment and i'm
trying to and i
will try to steal man one of the
arguments from the previous podcast
um which is that um
we're actually um making a lot of
premises
with instrumental convergence and
um um also
um so there's insurance continuum
convergence a conventional instrumental
goals and orthogonal
um orthogonality thesis so to explain to
the readers the listeners
or technology teachers is that you can
have arbitrarily
high intelligence and arbitrarily bad
moral values
at the same time um and um
conversions instrumental goals are that
us as humans
we want um to stay alive to
um you know do stuff we care about and
any agent that would have an intuitive
utility function
might consider to survive to um you know
pursue his goal and he might also want
compute
uh to uh do better predictions and and
so on
so those are kind of um claims made by
bostrom and yukowski
that are quite fundamental to like all
this kind of stack
of ai alignment arguments you talk about
uh
where um if you don't buy any one of
those
arguments then you're not um
yeah convinced of it so yeah um could
you
could you try to tell man like how those
kind of basic arguments could be wrong
um like instrumental convert convergence
and um you know orthogonal
or is it too hard i don't know so
i don't think you have to buy every of
these arguments to be concerned about
ai's
especially nowadays since we have like a
much wider field than just you know a
few people on a weird transhumanist
trans uh you know a mailing list back in
the day as foreign
you know we now have a much wider swath
of people care about these things and
finding like real arguments
about this like it's hard for me to
steal man some of these arguments
uh some of them i can like some of them
kind of make sense but they miss the
point
so i go like an argument that is valid
against their arthritis thesis
a version of their personalities i
consider a strong man
is some people say well you know and
like a very stupid
agent can't have a very complicated goal
so technically you can't have any age
with any goal
i'm like okay fine but that's obviously
missing the point of the thought
experiment
yeah so another version so like
the only still men i could think of i'm
sorry if i'm doing a bad job
this is the best i can do one is is that
people believe in strong moral realism
so people believe if you just create
something arbitrarily intelligent it
will just all converge the same morality
and just become good
so this is something you hear sometimes
among the transhumanist folk
singularitarian folk
they're like oh the eye would just be so
enlightened and good it will know what
the correct thing to do is just do that
like uh schweduler also has a version of
this
so that's one possible argument um
like another possible argument we could
say that somehow like
um like combo of complexity and like
we can kind of have like more like um
you know doing good is maybe more
simple than um you know um
torturing a billion people where's weird
dreams
uh the simplest scenario is just to
maximize entropy and everything just
destroy everything that's the simplest
policy
just a random policy
everything else is more complicated so
yeah that's like what will you cost me
call the hidden complexity of wishes or
the
like the fragility of values is that
what humans want is actually really
complicated that's why we need energy to
yeah we're it's a very low entropy state
in which humans
exist in which there is value or art or
beauty or anything other than like a
finely diffused plasma
everything that's not a finely defused
plasma is a complicated state
and requires like various energy to
maintain
we're all they were moving towards heat
death of the universe everything's going
to be a very fine
you know mist of black holes and protons
and whatever
into infinity at some point and
everything else requires energy to
maintain
negative or negative entropy in that
sense so
the the the like one of the arguments
that
is you know it goes against the strong
realism claim is that just
it's things that humans want are not
like
privileged in a sense it's not like all
aliens and all intelligent all computer
programs just
spontaneously decide to want the same
things as humans want
that seems pretty outrageous to me like
why would that be the trait
case like i could just literally just
write down a program if what human wants
do opposite
like you know i could just write that
program and that that's an existence
proof that i can write an ai that does
the opposite of what humans want
or you know something random just like
if actually do random
you know and there you go you have a
random policy that there's no human
values it
so i don't believe the strong moral
realization of a strong moral
convergence
so so your your program if you if you
were to program something that says
um do what the human doesn't want then
it should like kind of
include everything the human wants and
so it would kind of have the
understanding of human values even if it
doesn't follow it but it would have the
understanding of human values
it might have an understanding yes but
it would not execute them and honestly
that's what i care about
i i don't care if the torture ai knows i
don't want to be tortured even torturing
me
it's not i don't want it's not i don't
care if it's not smart if it's
if it's you're dead exactly it's like
that's the strongest argument through ai
alignment that kind of arguments always
just like
i don't care if it's conscious or if
it's made of a neural network or a
[ __ ] prologue or whatever i don't
care what
matters is that we're building systems
that can make good
decisions that can optimize their
actions to achieve goals in complex
environments
how this is achieved and whatever does
it really matter
yeah we could just say by fire assuming
optimizer exists
that can be given a goal and we'll
optimize for it
you accept that that this is just
possible at all which
is the entire field of ai like if you're
working in ai and you don't believe this
is possible
well what the [ __ ] are you doing why are
you in this field
the whole point of ai is to build ai why
are you here
no i i thought that ai was you know
something that played
it's a very common thing it's like
i think i think it was stewart russell
who made who made this interesting this
good point is basically
um the whole field somehow has this huge
blind spot of like
what happens if we succeed like what if
we just
yeah succeed somehow no one's thinking
about this
ai alignment is about just taking
seriously what if our things just
actually work yeah yeah i remember
i think he said he says this on multiple
talks and also
um his book yeah what if ai succeeds and
then like
it it asks the actual ai researchers to
to think about wait what what do we do
now
and they're confused exactly it's very
strange for me
so that's why i find it very hard to
steal man these anti-alignment arguments
because i don't
really understand like some of these
people seem to believe that ai
is just impossible which i is very
strange to me than like why do you work
in ai
like what are you doing here and some
people seem less like unwilling to
accept even hypothetical okay well
assuming you would
succeed what then some people so or
sometimes people say
oh that's a hundred years away but even
if it's 100 years away i still think
it's worth thinking about
sure i guess i guess my intuition is
that people
are like prefer thinking about things
that are that make them happy um okay
yeah of course i i was trying to just
take outside sure those people are all
[ __ ] liars
or like these lions themselves um
well yeah let's try not to um say bad
things
um about
i think most people are genuine i think
most people are very genuine in this
regard i think they've just never
thought about it really
which is strange but that's how the
culture of the field works
i think the vast majority of people in
ai just never
really thought about it much and and
maybe there's like as you say about
optimization processes
um you know society uh doesn't
optimize for people saying their true
beliefs on the internet
because their carriers will be ruined so
most people
are censored and maybe a lot of them
read less wrong and think like you
uh but there's nothing
yeah there's a great saying i never
trust a man to
believe something or to to know
something or understand something
if a salary depends i'm not
understanding it
okay i think that's a common dynamic is
that these beliefs are associated with a
dude that wrote a
2 000 page harry potter fanfic so it's
kind of like a
low status thing you're not allowed to
talk about this um
sure okay so it's it's it's like kind of
signaling your
low status to code um fanfic
either uh compared to like quoting
nervous papers
uh yeah exactly it's like uh ai live it
didn't come from the academia
establishment it didn't come from the
serious people with serious titles it
came from a dude
you know and like five other and you
know a bunch of other people on a
mailing list in the early 2000s
and um one of the things which is
extremely important to understand about
humans work is humans
take status extremely seriously wait
wait wait
what about quantum physics like it
actually also like came
as something very weird and
like those were like smart people with
high status like eisenberg
and and like his friend was like all the
all the top guys in physics
yeah and and like and they had a huge
debate
and and nobody like they were arguing
forever so like even
with high status people you can argue
yeah so it's not
necessarily contrary to contradicting
your point is more like
another example of um strange idea that
kind of
um where people could talk about it um
yeah
and i expect alignment it's already in
the progress alignment is being accepted
into the canon of academic thought
because it's obviously a good thing to
be thinking about you know
russell and other professors starting to
like explicitly talk about it
as an acceptable thing there's a i
forgot who wrote this but like i
remember reading a blog post or maybe
some i don't remember what it was maybe
it was a tweet or someone
say that you always want to be the
second person to to like discover
something because like the first person
to come up with
the second person is going to want to
like publicize it and then
would you ever actually would you want
to be if you want to be the first
professor to notice it
you want to be the first highest status
person to notice it because you don't
have to because even if you say
oh this was all invented by this other
person no one's going to quote them
they're only going to excite you because
you're the high status person
yeah i guess i guess we have those like
high status professors
uh we have scored russell we lost from
um
but youth koski doesn't didn't get the
whole the whole credit
um i have like two or two other things i
want to talk about
um which um is of specific interest to
me
is that i think i think like to make a
transition
um what young lacuna if i try to steal
money and lacoon
he says something like um how could we
possibly
align those uh very smart agents
if we don't know anything about them and
it would be like um trying to change the
sex of an angel or something
or um and and somehow um if you think
about it
um there is like some kind of
uncertainty about how much we can
influence the future
like um like the the still man of like
people in the
in the industrial revolution you're
brought like 200
years back like what do you do to like
have a powerful impact uh to energize
yeah i can ask a simple question do you
think marx had any influence on the
current time
yeah he lives in the industrial ages do
you think he had any influence i think
he had some influence but like
like okay but what was his marginal
impact so like
imagine mark didn't existed how many
years would we wait before
someone like mark says something similar
i don't know i can't rerun history
but it might have well never happened or
we might have had a completely different
ideology that had like some that filled
the same niche
maybe fascism would have just taken over
the entire world and that just would
itself
yeah just like somehow at some point i
thought that like buff trump's book
about
superintelligence was very influential
as it motivated people like
elon musk to thinking to take it
seriously and um
it has some very positive impact so i
considered like writing those kind of
books to be
huge impact but after thinking more
about it i
just realized that you know we're all
humans with similar ideas so like
like if boston adam wrote it like maybe
someone else would have
like three years later so i would
greatly recommend you read a book called
inadequate equilibria by yeah i read it
so if you've read it i have no idea why
you're making this argument because the
world is not efficient
okay okay okay again this is not we are
not
efficiently exploiting the maximum
research you could do on ai alignment or
the best books oh yeah yeah we're not
we're not
we're we're 100 so in that sense every
marginal
yukovsky or bostrom we get is a huge
knit oh yeah
yeah yeah it's it's huge uh but it's not
like as huge as like
like comparing zero to like this it's
like huge compared to like winning two
or three years
or four or five years which is which is
still huge um
but yeah yeah i i agree with that and so
to
so so what you but what you were saying
in this podcast was like
we don't care about we we can still
reason about
what those very smart agents can do uh
in an abstract way
without like delving into details of
architecture
and one thing yeah i would
no i'd say differently so first of all
um let me say something very snarky
to looking for
i don't hate you but i'm going to say
some snarky about you now uh if la [ __ ]
thinks we can't reason at all about
intelligence
you must be a really terrible artificial
intelligence researcher huh has you
learned nothing that you can think about
that seems kind of strange
i've learned a lot about artificial
intelligence studying it are
one of the other artificial intelligence
people studying i don't know
seems kind of weird so i've been
studying a lot of artificial
intelligence and i learned a lot of
useful things for optimization processes
about
report functions sure i might not know
exactly how you know the pay-per-click
maximizer is going to be built
but i have a lot of good guesses like i
think there's a really realistic
possibility that
the you know whatever is the first agi
of the first tdai or whatever
is going to just actually be a neural
network built with pytorch running on
gpus in the next 10 years
i think it's a real possibility yeah
yeah now that we see this convergence
between the brain
with you know the brain algorithms and
back prop and it's all like just based
like would you not say that knowing
about bayes uh
you know based optimality is not a
progress in understanding what
intelligence is in reasoning about
intelligent agents
i i find it very strange that these
people who
apparently haven't taken five minutes to
think about uh how to reason about
intelligence and how to think about
agents
say well it's impossible to reason about
but it feels to me like they haven't
tried like if you tried five minutes
to think about what can i like what
more than random what's better than
random can i say about a future
intelligence
so come up with a lot of things maybe i
think i think we're
coming from um is from an engineering
perspective
like someone who tries like would try to
like build the first
uh you know complete uh complainers in
the early
like in the end 1990s like you you want
to
like they think that engineering is hard
and and that like you will like people
will try to align those things
will be the builders and that people who
reason
about those things in the kind of
abstract way in a text way without like
actually building
things don't actually understand and
won't like have an impact and i i think
that's like
most likely engineering still man where
like thinking about things and like
philosophy i think that's a fair
argument but
i think it's over simplifying i mean if
that was true then why do we have
theoretical physiophysicists
we should just kick them all out because
obviously not doing and like i'm just
i'm just like straw manning it like you
steal men and i'll be strong man
is why do we have theoretical why don't
we have mathematicians like we should
just not have any mathematicians they
should just all go do experiments
instead
so yeah i see we need both like i agree
that if you just put a bunch of
theoretical physicists in a box
you're not going to get a good theory of
universe obviously
you can get a nuclear bomb out of it uh
you that you still need
you still have like a theoretical uh
practical experimentalist for that too
did they practice did they oh yeah they
do they did some experiments in
manhattan project but they didn't work
there's a lot of experimentation it was
mostly engineering
actually it's like the theoretical thing
uh nuclear secrecy blog has a good post
on this where they argue
we all think the theory was that
important but actually wasn't
because that was the part that was
declassified they kept all the
engineering classified but he
declassified all the theory because they
thought it wasn't as important
like the really hard part about building
the bomb was just like
all the dupont engineers work of just
you know creating pipes that can contain
uranium hexafluoride and you know build
just really big factories because you
need
really big factories it's a nuclear
secrecy block so it's a good blog that
has a lot of info on that
but nuclear secrecy i think this is what
it's called i forgot who the guy is who
runs it but he has a lot of great
he's historian he has one yeah
blog.crazy.com yeah
there you go great great blog um but
basically
so but i do agree i think engineering is
undervalued
in this regard it's like uh please don't
cast me as someone who says that
we should just all sit in the cave and
just like not look at neural networks i
am the opposite of that
i work with neural networks every day
because i expect
to learn things from building these
large language models and from
experimenting on this launching
i would describe myself as an empirical
alignment theorist
i work empirically with the word with
the machines i do because i think
these there is a acceptable chance like
maybe 10 to 30 percent
that the models we're currently building
will just scale
to what will later be considered to be
hdi and so we should be starting
experimenting with them right now
yeah so it's it's what's called the
literature um as you know positive
alignment
and um i think one one um interesting
take was i think paul christiano's
recent blog on
his research um protocol where like he
tried to experiment and things and try
to see if
is alignment technique scale or not but
one point i saw you're making
um about resulting about um not
experimenting but like
um the usefulness of math in thinking
about agents
was that um there's there was like this
base of like decision theories
which is quite large and um given
to the function you can you could find
the best uh decision theory that
maximizes this
duty function like i i haven't seen much
people making
the decision distinction between ugly
and decision theory
um and so i so yeah i just i just found
that super interesting
i don't know how much we can apply that
for um you know like
yeah i don't have a precise question
about it i just found this that they did
if i have if i could i would if i could
just take
miri you know and put them in a cave for
ten thousand years
and just like didn't extract papers from
them i would prefer
i would love to do that before we did to
build agi just to see what they find
you know i don't know if the decision
theory research is going to lead us
anywhere but i think it's worth
exploring
the same way exploring you know abstract
math and bayes theorem and quantum
physics
is useful i i see like i think decision
theory has helped deconfuse a lot of
questions
so miri has this great blog post where
they say questions start as philosophy
then they become math and then they
become engineering
first you ask yourself the question what
is even the question i'm asking
what are we interested in then someone
starts hacking away at that with math
because
models and systems and like you know
it's like toy experiments or whatever
and then an engineer takes the map and
actually turns into a system that runs
and this is a process i think happens in
most scientific
disciplines so decision theory was an
intent to go from philosophical
experiment what does rationality mean to
turn into math
and now with an ai is the engineering
part of taking precision theories and
whatever
and this inspiration not literally
implementing them stepper step but
taking this aspiration
taking bays as an inspiration for how
these things might be able to think
at this point conor had to leave
so what we're about to hear is the
second part
which happened one week later over
twitch
during the first 10 minutes i
unfortunately
forgot to check if conor's voice was
being recorded so the start would likely
feel
a bit abrupt if i recall correctly i was
prompting him
an illuter ais current projects
especially his involvement
with eegi or luther experiments in
general intelligence
human values or just being useful in
different ways
um so far not much has ha it's most of
it's been kind of like you know me and
two other people kind of
private chat um but we're trying to move
this into a more open state we also want
to work more in interpretability so
you're understanding what these models
do internally and stuff like that
um i want to make this all open source
all open in
luther so anyone can work with me i
expect in the near future is going to
work there to be done
um at the moment not really like soon tm
you know like keep an eye out if you
want to work with me on that um we're
mostly going to need
ml engineers probably uh but also some
web dev work though we do already have
some great web developers melting with
us
ml engineers is quite a broad uh
you know range of people so maybe more
like deep learning engineers that can do
tensorflow mesh
well no we don't use tensorflow anymore
it's all pi torch like oh spiders now
okay
it's all pi torch we have we have
oh cool cool so by torch on this uh new
sponsor
um right yeah so we have a lot of gpus
from them that we
do most of our experiments on so as i
said the work i'm doing with egi
is still very early stages like very
small models just trying to experiment
um if you are if you the listener are
um a experienced ml engineer or someone
who has experience with high performance
computing
please consider contributing to new x ux
is a huge data
is a huge you know code base that
there's a lot to do
uh sid does a great job of keeping
issues up to date on the github repo so
you can just check out the github
breakout look at the issues
and you can also hit up uh sid or me or
sela or anyone anytime
and we're happy to get you into the code
base um so it's like not something that
a beginner can do that's unfortunate
that's the unfortunate part about ux
it's not really something a beginner can
do
you should you need to have some
experience to get into that code base um
yeah that's like the main public private
i i don't say
sorry on github you're on the
uh github website is it public the stuff
you just
yeah it's all public what was the name
again
uh so the new x represent uh so if you
go to the
go to our website you can click on the
new x link and it will get you to the
website to our github repo
where we have all of our code okay sure
um yeah let me do that
oh it's not important um okay
and and so yeah so this eeg
thing so to call um to frame it for the
listeners is
experimental experiences
experiments in general intelligence yeah
right it's just a funny name i came up
with yeah yeah it's pretty it's pretty
cool
it's a bit like eeg but like for agi
right yeah um
yeah and so you're starting from this
paper
uh i i guess from polecraft channel or
the opening
team on learning to summarize from human
feedback
right exactly and so our first step is
going to be replicating the results of
that
paper because i think they use very
interesting techniques that i expect to
scale
and like a lot of variants of those
techniques that i want to experiment
with and so we're trying to
first get our feet wet you know get the
um get this set up
try experiment with like models and yeah
just take it from there
sure and um
[Music]
well i see you have like already our
websites called
i think it's like uh something
uh maybe like nlp okay i'll let me find
my notes
um yours it's called eeg
dot eliter.ai where
you actually have something where you
can pick some summary and then you can
just like give your feedback right
so that kind of kind of cool yeah that's
experimental that's also experimental so
nothing nothing for the public yet
um that's down the line at some point
uh what we hope to do is is to create uh
have an interface where you where we
have users directly interact with these
models and give feedback on their
performance
it's like we'll have the the model
summarize text or
chat or perform other activities and
we'll have users rate their ability and
we're going to use that feedback
using some advanced methods to try to
improve their performance for the users
um application whatever they want you
know whatever the goal is that we're
working on
but that's down the line where it's very
experimental you know um
talk again in a few months
how many months well depends on what you
want to see so i expect that we should
be able to replicate the results
of the paper at least for the smaller
models in the next like
i don't know maybe one to two months uh
depending on compute availability and
like how big the models are
there's other things um making it more
open to the public
and like you know training uh individual
models
we'll have to see about that um it
really will depend on like how much work
your ex needs how hard a few technical
challenges are going to be
but i expected about like six months
time i expect us to have some
really cool results to show that's kind
of my timeline right now six six months
i think we'll have some very cool things
to show
cool yeah because i'm super interested
in that because um i think i think uh
yeah
uh mixing erl
and and nlp is the way to go and
i i love this approach with uh just like
learning um human preferences from
feedback on you know is there is this a
good uh upper
uh backflip yeah yeah
exactly is good and so if you consider
now that
nlp is the most scalable way into aji
and it makes sense to kind of align
align it and and
make some kind of a rel feedback loop um
do you
do you understand how the um reward
model looked um
right worked because like let's try i
just look at the paper
at the blog post quickly and it seems
that you train a weird model
and then um you train something that
summarizes
and you train something that like a
policy to switch between different
summaries again you just like go through
the loop maybe um
so the way the paper does it is is that
um
they collect human feedback on periods
of
um summaries so they have like human
generated summaries
and uh ai generate summaries and they'll
show humans
a pair of them they'll say two um
summaries of the same text
and then the human will will rate which
one of them they like more
then they train a reward model to
predict you know
given uh two texts like which one would
the human like more and they basically
use that kind of like as a reward signal
to then r use rl uh use ppo
to fine-tune a gpt model as a policy to
generate
these kinds of um summaries
they also use a third model as a um
kind of as a kind of regularizer so
basically what happens is if you just
train these models kind of naively
is that they over fit to the reward
model and they find like ways to hack it
they find like weird ways of just like
repeating one word over and over and
over again and for some reason that
confuses the reward model
and gets a really high reward even so
it's not what we want so
it's a bit of a hack but basically what
they do is that they take an untrained
legislative a pre-trained but like a not
a fine-tuned
gpt model and they take a kl divergence
between
um the fine-tuned policy and the
baseline gpt and penalize the model if
it diverges too far from the baseline
and okay and so they give you uh like
bad reward
or yeah it's basically that if the text
is too weird
if it's too far from normal human text
it gets a negative reward for that
okay and that that encourages it to
stay kind of within the bounds of normal
human language
instead of you know producing this one
token that just like repeats it over and
over or something like that
and okay so how much compute would that
need uh because there's like the whole
like how much compute did opening i use
right a lot
so the opening so i currently we don't
have the kind of compute laying around
that would be needed to fully reproduce
especially for the larger models it is a
lot of compute that goes through these
kinds of things
that's why i'm taking it slow i'm not
like committing to any timelines too
much because this will depend on a large
degree on compute
constraints um you know it might turn
out that some of these things
uh are much harder to get you know to
get the compute than we currently hope
um you know we're all figuring this out
as we go but yeah in the paper
it's especially the larger model so like
in the ideal world i would like to
experiment with models in like the 10
billion parameter range
like 10 to 13 billion would be the ideal
size of models for me
um i expect there to be interesting
performance characteristics
uh that i will see those models that i
can't see in like one billion parameter
models or smaller models
it's just a hunch i might be wrong but i
have a good feeling about it
if if possible i wish i could do you
know the 200 billion parameter model i
think that'd be even better
but that's not feasible hmm so
more than more than the number of
parameters from so
yeah how does the dental work differs
from like
how does the language model differ from
gp3 is it the same architecture or do
they have something special
uh it's basically it's the same
architecture the same architecture and
plus plus ppo and yeah i'm not sure
yeah how how longer it would take
um and
what's the what's the next milestone for
gpt neox
um at this moment um other than you know
just like fixing bugs
um improving efficiency experimenting
checking different things
it we're just hardware bound so we're
just waiting basically
um the chip shortest hit core weave you
know they've ordered tons and tons of
gpus but they just haven't arrived we're
just waiting for them to arrive really
so yeah we're we're ready to scale up to
larger sizes we're probably gonna train
like a midsize model
i i assume like a 13 or 20 billion
parameter model number probably do a 200
billion
and once the hardware is there i i i
don't fully grasp how hard it is to go
from 30
billion to uh 200 so like i know
i know we get more sample efficiency is
that true that we get more sample
efficient
when the bigger model is yes
but the model is bigger so the amount of
compute you have to put in the model
and the number of gpus is of course
higher so basically
if you know how much compute you have
ahead of time you can calculate like the
optimal model size you want to train
um i don't know what you know our final
hardware is going to look like i don't
know what the perfect model is going to
look like we're probably going to train
at 200 billion for the meme
anyways because it's just a nice number
go for
but yeah we'll see what comes out of
that hmm
do you see the twitch comments because
like i cannot see any of them
this is very bad uh i just keep like bro
i don't have it open
okay okay because i don't see anything
um
because i need to put in my big screen
and if i do it i lose the
the conversation maybe okay let's try
for five seconds
if there's um and i mute it
so there was comments before now there's
not no no more comments i don't i don't
get it
um probably not that many people
watching okay but
there were comments before okay um
anyhow um
so yeah i guess like so if you if you
scale to
from 30 billion to 200 so you get seven
times more
uh parameters but then there's like it's
more efficient so like what does the
scaling law tells
approximately um we don't know what the
final hardware is going to look like
um so i can't give like a specific
answer and we're probably just going to
train at 200 billion parameter models
for the fun of it
you're just going through that yeah
we'll see our goal
is like 200 billion and we want to get
the hardware to make that happen
but you know this is all up to core
weave and how things end up working
wasn't the meme like one terabyte or die
or something
well what one tr a trillion or bust but
that's not happening that's just
physically impossible like you know you
need uh
i don't know maybe the biggest super
computers in the world could train that
maybe um but yeah with not nothing we
have access to a trillion is a meme
no one's done that like the the switch
transformer doesn't count
well in nvidia nvidia is going for 1
trillion right
they're definitely trying uh i think
it's definitely possible that that they
will succeed in the near future but at
the moment no one has done it and
we're not going to be the first one to
do that sure like okay
when do you expect there's a 50 chance
of nvidia
releasing their 1 trillion model i mean
50
chance today today you're just like
drop a clown and you're like oh i'm good
yeah
maybe 50 i mean i expect like um if
their gtc keynote is true
and all the things they say is true we
should expect a trillion parameter
models sometime this year or next year i
would say i mean
switch transformer exists which is like
a sparse trillion parameter model i
don't think that counts
uh sparse marbles aren't as powerful as
similarly sized
um dense models but yeah i expect a
truly parameter dense model
pretty soon um open the eyes also
teasing that they have something that
you know blows gbt3 out of the water
like i think ilia has tweeted about that
in the past and
that's probably gonna come out at some
point
hmm and yeah recently there was a bunch
of conversation on the discord about
who are we getting this is it huawei or
chinese hardware
uh yeah huawei is like everyone is like
coming up with three on parameters
um
yeah there is a scaling race happening
right now which i'm very not happy with
um but it is kind of like uh you know
big numbers go burr you know like ooh
look at that you know look at how big
our number is oh
our number is bigger than your number
that's just how humans work you know
it's like a
it's a clear prestige thing for you know
big companies and
increasingly governments to um
how to show these like large models that
they could train
the most interesting thing about the
huawei um model is that it was trained
fully on chinese hardware so chinese
cpus chinese accelerators chinese
everything
was very interesting the model itself is
kind of terrible from everything i can
tell
uh it was only trained for like a tenth
as long as gbt3
um also like on chinese text of
questionable quality now that i can
evaluate chinese text that's just what
i've heard
so uh it's not like equivalent to gp3 or
anything
i personally don't see how you know
china is going to catch up in ai quote
unquote in any regional capacity
you know with the kind of censorship and
the kind of like weird incentives that
exist in china anyways
they have those weird incentives that
make them
um you know go more private um do their
own thing
fight the the entire world so they they
could just like
have the smartest people working on
something forever without telling anyone
and just no they can't i do not believe
they are capable of doing that
people very much overestimate one thing
i've like i think under sandberg
said this very great is that top secret
progress usually suck
top secret projects are usually really
really terrible because they don't get
the feedback
from the from like the really smartest
from the community and smartest people
and as far as people usually don't want
to work on top secret projects
especially not with like crushing
bureaucracy as it exists in like china
and stuff
is that let me put this away say you
were the best
ai researcher in the world right you
could work anywhere you want
would you pick china not really
yeah it is a whole paper from boston
about
openness in ai like the like is it like
would we expect in the future to have
more openness because today
there's like a bunch of value for people
uh to to to do opening work to publish
openly to have open source code um so
people don't want to go to china and and
work in private because their career
will be ruined
and yeah they don't want open like they
want open source right
um the the incentives currently don't
really like the chinese market has a lot
of problems
um uh and i also shine these legal
system and censorship system
like imagine if your gpt-3 chinese
starts talking about tiananmen square
you know you have to sense to that
somehow
you can't say that otherwise the party's
gonna be really unhappy with that let's
like imagine
you are what's the temenum square
yeah time and square massacre it's uh
massacre
yeah and that's heavily censored and
obviously a gbt3 model you know
would pick up on that unless you do
extremely heavy censoring
which would completely [ __ ] the model's
performance i expect
so i think it's all a joke like people
will say oh
china doesn't you know has so much data
and they don't care about prices all
[ __ ]
it's like it's such a you know
[ __ ] of bureaucratic nightmares
is that like it's it's way easier to get
a huge data set in
just you know being in america and just
googling things
you know who cares about like all that
kind of the one thing that chinese are
good at
is facial recognition and fall you know
and like spying on their citizens that's
the one thing they're good at which i'm
not happy about
but other than that it's like who cares
but they do publish papers they get
yeah they publish papers let's let's be
clear here
papers the the highest cited papers do
not come out of china this is just a
fact
they're a wonderful really good chinese
suspiciously many of them working on
facial recognition that come out of
china and they do publish papers and
there's a huge
number of papers but um china has like a
very different incentive system when it
comes to
uh publishing papers like often china to
like to like advance your group
you have to like publish a lot of papers
and it's completely irrelevant how good
they are they of course were claimed
otherwise but like anyone
every every person i know that's related
to that has worked in the chinese system
will tell you that like you get a bonus
for like publishing in journals
depending on like point systems
you know it's silly it's like you think
the american system is corrupt you think
the american academic system is is
corrupt which i think it is
chinese is like 10 times worse it's it's
all a big it's all a big meme
but then the question is is um you know
it's like you're kind of building
this um experiments uh on on agi
and i guess at some point we'll get some
race to scaling and at some point it
will
become scarier and scarier and closer to
actual
uh even human level at least human level
uh
text prediction um and so
i had this discussion on the discord was
i think it's called 3d
3d printer or something
and it was that this can just be between
multiple
scenarios and um actually having one big
uh
corporation winning the ai race and at
the beginning
i was convinced that you know open ai
were great
um maybe deepmind is cool too they have
co-founders interested in
asi safety so maybe if if like deepmind
becomes
very good and then openai joins them
because they have doubt in their like
legal policy then we have like
one very strong actor that will make
everything
safe right and that would be an ideal
case
if that works yeah i thought i thought
it was the ideal case but then the guy
convinced me after
a bunch of messages then then multiple
scenarios
were better if if we had like a bunch of
decentralized
kind of agi groups um
kind of helping each other or like
correcting each other
because you know if you're it's harder
to be to be smarter than
all the other ais in the world right um
so sure i feel like luther falls into
the second category where you're trying
to like
level up by the like everyone has the
same
open source software or at least like
the api
and a bit like the open ai um argument
from elements at the beginning right
um i uh do not subscribe for that
argument of elon musk by the way i
i think elon musk i do not like his
opinions on ai at all i think he's crazy
uh when it comes to ai i like him i like
all of his work but i think he's crazy
when it comes to ai
um so this is an extremely nuanced topic
this is not
a is good and b is bad extremely
complicated topic and no one knows what
the right thing to do is
and this is like multipolar versus
unipolar as if we had a choice
i don't think we have a choice in this
regard the way i see the situation
personally
is at the moment you have a choice
between open sourcing lgbt23 or not but
that won't change
i don't i literally don't think that
changed anything in in the multiplayer
versus
only uniport i said i don't think
there's like any action i could take
that would encourage a uni polar outcome
in like any non-trivial capacity so the
way i see it is the following
i think that coordination
is extremely hard i am extremely
pessimistic about any kind of
coordination among humans
i literally think it's easier to build a
fully aligned super intelligence than it
is to get humans who cooperate
on large scales uh it's like i i don't
think that like
all ai companies could coordinate
without defection to like stop ai
research i don't think that's possible
i i just don't think it's possible um so
the way i see it is the only possible
way we have
hope that this is going well at all it's
just we develop
a technical solution to ai alignment or
methods to ai alignment
that are competitive easy to use and
you know so good that there's just no
incentive to not use them
if we develop a method but it makes your
ai a million times slower no one's going
to use it we're all going to die
if we create them you know if we create
a method and then
um in some way it's like hard to
implement or only one person the world
understand it we all die
the only way i think we can get it is
that it has to be simple
it has to be well early you know
implementable
it or we're you know people are there to
make it implementable
it has to be competitive that's highly
performant it also the
resulting ai also has to be powerful
it's like there's no there's no use
building a weak
like a like an aligned ai that has been
weak and immediately destroyed by the
second ai that's unaligned that is much
stronger
so you also need to build an ai
alignment and you know strong
agent that can defend itself against
potential that's the big
danger with multi-polar outcomes is that
we could have like say we build
three aligned agis for this multiplayer
scenario but then a fourth person builds
a paperclip maximizer that blows up the
world
then you know we still lose you know or
even if the
even if they managed to fight back the
paperclip maximizer but you know they
nuke the entire surface of the planet in
the process
or whatever so we need to build some
kind of um
there's a whole like iterated
amplification scenario from
pro crusano and and we need to add some
constraints like
please make sure that people don't build
stronger
agis so we never end up in no one knows
how to solve intelligence amplification
is not a solution to intel
to alignment i expect if you would
implement
android amplification as described you
would paperclip the universe or worse
actually
i don't think it would work as it
currently is and
christiano does no one though this is
working on it
but there is no proposal that currently
exists for alignment that is ready to
implement
and even those that are like have some
like good ideas i would not give them
more than like 10
chance of working we are in a very
early stage of our understanding of
these problems
there has been massive in my opinion
progress on these problems but it's just
an incredibly hard project
and and and yeah the progress is is
is massive compared to what it was
before so it's moroccan yes
accepted and soft and we have maybe like
100 papers a year on ai alignment or
something
tops um maybe a thousand if you count
like uh robustness
and and so on but then um but then
there's like this massive companies like
hey uh nvidia and like throwing like
millions and billions of dollars at this
research right so
i i don't know if ai alignment is
scaling as well as much as the rest of
the economy right
we don't need alignment at scale we need
ai element to work
it we need a solution to alignment it
doesn't matter how many papers we
publish it doesn't matter how much
funding it has
but the only thing that matters is is
that we have a understanding of the
problem and a solution to the problem
that people build in the ais can build
my hope for things to work is is that
no one is incentivized to build an
underlying a high you don't win
by building like if you build an online
agi you get [ __ ] too like no one wants
to build
well and agi killing people with
shotguns right
i'm sorry was that sorry uh people do
kill
other people with shotguns like they're
like some evil people trying to justify
the world well yeah okay but i mean if
those people be the aj
we're we're like super super triple
quadruple [ __ ]
you know if people build a military ai
agi we're so ultra mega
super gigafucked there's like you know
like i don't even think about scenarios
because we're so [ __ ] in such
situations
like if if things like go super
multi-polar and we have like world war
three and there's like take like decades
of like military eye development we are
still ultra good [ __ ]
like i it's like not even worth thinking
about because it's like i don't even
know how we recover from such a scenario
like if something like governments like
build ai agis it would be optimized to
kill
humans like could you imagine the horror
like like the
unimaginable like hell like literal hell
on earth
well i'm just i'm just thinking about
like not someone someone some not like
everyone can do it but you know some
corner that tries to reproduce gp2
in 2030 someone else is like oh i'm
going to try to reproduce this agi paper
that i'm not allowed to do and
you know try to run it and um if
if like you know did the same by your
koski like the iq
to create agi dropped by one every year
kind of dissimilar thing uh so if you
have an iq of 150 in
20 years then you might be able to do it
by yourself or
yeah i i guess it's just
who i expect like the um i expect the an
era of humans to end very shortly after
the fridge agi is built or the first tai
is built i expect humans to not stick
around for very long
um why because because we get uh we get
like neural implants or
or like in the best case scenario
i mean the average case scenario is a
paperclip maximizer like the average i
expect like on average
like civilizations like ours in this
situation probably on average paperclip
themselves
they create an agi it's like unaligned
and it just you know splats the you know
takes apart the entire earth for
resources
to build something it doesn't hate
humans it's not evil it just
doesn't care i think it's the default
scenario i think some minority of
civilizations in our scenarios build
demons they build chaos gods they build
evil
ai like military ai that like tortures
people they're like you know create
simulations of uh you know
terrible situations or whatever
something's called suffering risks risks
yeah that's what i'm
most worried about and some people some
civilizations manage to get the
alignment right and to create like you
know
these like angelic super beings that you
know
can bring you know peace and harmony and
solve suffering and all these extremely
hard problems and then expand onto the
universe
if we get elected right that's what we
get
but that's not the default scenario
nature is allowed to kill us
we're allowed to lose we're not the
heroes in a novel
we're not predetermined by a script to
win
we're just jumbles of atoms we're just a
civilization like
any other in the universe
we can lose and we will lose if we don't
get this right
if we don't solve the technical problem
in time we build agi and we don't get
hundred understanding of how to align it
that's it game over so do you think
do you think the the great caesar is
ahead of us
i don't know i don't know but
at the current moment i would say i'm
more optimistic than it was a few years
ago
i feel like progress and alignment has
been a lot faster than i expected it to
be
so but that means like i updated from
say five percent success chance to maybe
15
success chance it's possible and
it doesn't mean not to work out there's
literally no reason not to work on it
because
the alternative is we were [ __ ]
anyways so why not work on it
and it's also a fun thing to work on i
enjoy working on alignment very very
much
and if it increases our chances even
marginally i mean what else are you
going to do with your
you know like it's just the obvious
thing at least for people like me i
think it's the obviously correct thing
to be working on
and i think there's hope there is a
chance um
that we get this right i personally like
skeptical these like intermediate
scenarios that i think like
cristiano and other people take very
seriously i was like like kiss like you
know
comprehensive ai services scenarios and
whatever they say like
okay we have agi was like neither super
aligned or like super on the line you're
like
it's like intermediate thing like humans
stick around i don't think that's gonna
happen
i i don't expect really any humans to be
around by 2100
you know either or if they do exist they
only exist in like cesa spots or
something
wait uh i don't i don't think c c a
i s the thing from uh dexler drex
drexer drexler um is is about
humans sticking around until 200 200
just like ai
takes no those are two different art
things i just said
it's like i don't it's like in case
there's like not an alignment problem
like
you don't really have like you don't
have paperclip optimizers but you also
don't have
you know singleton ai you have like
either i don't expect these these
intermediate scenarios i don't i don't
look like much probability maximus okay
because i
i put some decent probability on that
happening
uh after reading i think this is the
summary of it from rohan
um i don't i i think i don't understand
kaiser
it makes no sense to me like i've read
it i've i've read the whole thing
i'll do it and it makes no sense to me
yeah for the whole thing makes no sense
to me it's
completely silly like i think a case
scenario can exist stably for maybe like
two years before
some of the kai's things become edge
intake and take over all the other
agents and then you know we'd have
agents again
it's like dude gordon has the has the de
facto post on this called to
why tool ai wants to be agents and i
expect that to happen it's like when you
have a sufficiently
complex service system it will become
agented you'll have inner alignment
failure
you'll have visa optimizers take over or
you know like just like someone just
builds an agent someone just some hedge
fund just sits down and say okay i'm
gonna build an agent that maximizes
profit like that's i think one of the
most likely scenarios how these things
go to ship
is you know we're just gonna have some
stupid corporation be like all right
little ai
you know just create profit and then so
you know it just tiles the universe with
like you know
very large bitcoin numbers or something
okay so you
so at the end is more like a human
failure
it's like the human most likely scenario
the most likely scenario by far
is just we just build a thing something
weird happens and we all just fall over
dead
you know it's just like you know just
some we built some weird agi
it it has some weird paper clippy goal
it has some weird mesa optimization or
something like we have like gpt3
dn by seven or whatever and we
uh prompt it with something instantiates
some kind of like mesa optimizing age
and its internal representation it
breaks out and you know does something
weird um
i feel so scenarios like that are pretty
likely what is it pretty likely i'm
still talking like
10 to 15 probability or something like
50 of my probability mass is always on
unknown unknowns
like something much weirder than i
expect will happen by default something
much weirder than anything i can come up
with is going to happen
and i have no idea what it is and no one
does knows what's going to happen but i
feel like
there's plenty there's just so many
scenarios and how this could go wrong
and it all is downstream of not getting
alignment
if you have if we understand the actual
problem of alignment and we have
technical solutions to these problems
all the other you know then we have a
chance
if we don't have that there is some
chance
there is some possibility i give it like
maybe a five percent probability
maybe ten more like five that alignment
just turns out to be really easy
maybe it turns out we were just really
confused and actually it's super easy
you just have to like do the thing and
it just works every time
what do you mean by like it works so
like imagine imagine it works like
we're in the year 200 and 200
and it works so do we do we have
something
very very smart that cares doesn't care
about
hurting humans or like cares about
preserving humans and we we're just like
uh
merged we're merged with the ai or just
like neural i mean that's a whole
different question about like what
should we want
if we have generally it works i
just don't understand okay because when
when people say it's not
going alignment is a philosophical
problem philosophy is about figuring out
what the question is
we don't know what the question is we're
trying to solve to a large degree
is that and this is something that you
just have to get used to if you're
working in a pre-pharmatic field like
when people try to figure out what
causality is
like you know in the early 1900s 19th
century or whatever people were like
what does
some it mean for something to cause
something they couldn't answer that
question
like what does it mean to be good that's
more clear like people
still argue what does it mean to be good
that is a question i think you can make
progress on and some people would
disagree with you on that i'm not saying
more realism here
like um i'm an anti-realist for the most
part but
you can work on these problems and
alignment not fully but to a large
degree is about that
alignment is about moving probability
mass from the we paper clip everything
or worse
to the we don't do that scenarios what
exactly those don't do that scenarios
look like anything post singularity we
can't predict for sure we can
come up with some ideas you know every
scientist in this field has
some of his sci-fi visions a lot of them
want to keep humans around
a lot of you know a lot of them you know
want us to like be augmented or to live
in virtual reality or whatever like that
um i think it's pretty pretty quite
likely that we have
like a better understanding of the
problem and better control these ages
that we could you know
live in a beautiful wonderful perfect
virtual reality for you know millions of
years or whatever
i think that's definitely possible i am
more of a
um i guess feed the utility monster type
person i'm like
why keep humans around they're
inefficient just create beautiful
virtual minds that experience infinitely
more pleasure and happiness
and meaningful lives and think much more
beautiful thoughts than humans do why
keep around the humans seems to be
efficient
uh but that's a bit controversial so
you're you're
altruistic so you would prefer to die
and have some
duty monsters getting all the guillotine
absolutely
yeah of course just make sense to me
like why would it not does it seems
silly
the easiest decision of my life if i
could just you know you know
kill myself right now and it would just
make a ai alignment happen
and beautiful utility monsters inhabit
the entire universe literally easiest
decision in my life
i think i think we can we can maybe end
on this because that's beautiful
and that my dear and augmented
biological intelligences
was the end of this first real inside
view podcast
if you found this conversation
insightful i strongly recommend joining
a luther ai's discord
link in the description for the event
subscribe to the inside view channel on
youtube
i'm also curious what would be for you
the pros and cons of open-sourcing to
planning research
where the long-term goal is to solve ai
alignment
you can send anonymous feedback on this
video link in the description
or just tag or dm me on twitter michael
trenzy
spelled t r a z z
i