Reasons you might think human level AI soon is unlikely | Asya Bergal | EAGxVirtual 2020

hello and welcome to the session on
reasons you might think human-level AI
suit is unlikely with Asya verbal
following a 20 minute talk by hacia
we'll move to a live Q&A session where
she will respond to your questions you
can submit questions in your name or
anonymously using the box of the right
head side of this video you can also
vote for your favorite questions push
them higher up the queue we'll try to go
through as many as we can then after 20
minutes of questions
I'll bring the queue aids in it but
that's not the end of the session to
help you think through and apply the
ideas you've heard I'll be asking you to
join a 20 minute icebreaker session
where you have to speed meetings with
other attendees to discuss your thoughts
on the content I'll explain how to do
that when we get there
but now I would like to introduce our
speaker for this session
asiye burgle is a researcher at AI MX
where she also heads up their operations
she has a BA in computer science from
MIT since graduating she has worked as a
trader and software engineer for Alameda
research and as a research analyst at
open philanthropy here's Ostia
hi my name is Oscar Buerkle I work for
an organization called a impacts though
the views in this presentation are my
views and not necessarily a impacts
views and I'm going to talk about some
reasons that you might think Kim lovely
I soon is extremely unlikely and I'm
going to start the talk off with some
motivation so I'm pretty interested in
these reasons because I'm interested in
the question of whether we are in fact
extremely unlikely to human-level AI
soon and in particular is there
something like a less than 5% chance
they're going to get to human-level AI
in the next 20 years and I'm interested
in this because I think if human-level
AI soon is extremely unlikely and we can
know that now that has some implications
about we as altruist and as people who
care about the long-term future might
want to do so if it is extremely
unlikely
I think broader movement building might
be more important as opposed to
targeting select individuals that can
have impact now know you might think
fewer altruistic people who are
technically oriented should be going
into machine learning you might think
that approaches AI safety should look
more like foundational approaches you
might think that we have more time for
things like institutional reform and you
might think that there is some effect
where if the community is largely
advocating for human-level AI soon and
it doesn't happen we sort of lose some
global trust in terms of having good at
the stomach's and being right about
things and then people take us less
seriously when you know AI risk is more
of an imminent threat
I don't know how real this sort of last
problem is but it is something I worry
about and I think maybe as an effect
that we should be aware of as a
community so for the rest of this talk I
am going to look at three reasons I pre
people say that we won't have
human-level AI soon and the three
reasons are one that experts disagree or
some privileged class of experts
disagree to that we're going to run out
of compute to get to team of AI and
three that puzzly defined current
methods are going to be insufficient to
get us there this is three of many
reasons that people have given me for
why we won't have to mobilize soon and I
don't claim that they're representative
but they're ones that
particularly interesting to me and that
I spent some time investigating and I
will go ahead and spoil the talk now and
say that I'm not going to answer this 5%
question partially because my views on
it vary wildly as I collect new evidence
on some of these reasons but I do hope
that I will in the near term get to the
point investigating where my views are
pretty stable and I can have something
concrete to say so looking into reason
number one that experts might disagree
that we can get to human level guys soon
this is a survey conducted by Katja
Grace from a impacts and a bunch of
other people they asked machine learning
researchers and experts what probability
of human level machine intelligence they
thought there was going to be at some
year and this is the aggregate forecast
from one of the ways that they phrased
this question and you can sort of see
here that this is the 20 year mark and
it really seems like they do think
there's more than a 5 percent chance but
then if you double it'll further into
the survey results you see that the
answer is that people give are actually
extremely sensitive to the framing and
the exact way you ask the question so
whether you ask about human level
machine intelligence or automating all
jobs whether you ask what years are
going to be a 90% chance of something
happening versus what chances they're
gonna be at some year so for this
question what chance what we have of
automating all the jobs in 50 years
people actually give pretty low odds and
what that tells me and what people sort
of have concluded from this largely is
that it's very difficult to know what to
do with expert surveys we probably
shouldn't put a lot of weight on them I
was particularly interested on a kind of
survey where you ask people how much
fractional progress they thought had
been made towards human level AI and
then you can sort of naively extrapolate
that to figure out how many years is
going to take until we get all the way
to a hundred percent
it's Robin Hanson did a version of this
where he asked machine learning experts
how far they had come in the last 20
years and all the people he asked had
worked in their subfields for at least
20 years
and they answered in the five to ten
percent range which naively extrapolate
the puts human-level AI at 300 to 400
years away which is pretty far and then
the chi a great survey that I mentioned
earlier did a very similar thing but
they surveyed people who have been
working in the field for you know
anywhere from two to many years and
their aggregate percentages for K stood
out put female employee at something
like 36 years away so much shorter than
Hanson's aggregated forecast and then
even if you do something that's pretty
similar to what Hanson's doing and you
condition on them working in the field
for 20 years or more they answer in sort
of the 20 or 30 percent range on this
survey which gives you a median you know
forecast of 140 years or so so still
pretty long not as long as the Hanson
survey so I think there's actually a
pretty consistent story by which you
could reconcile these two results which
is that in the last 20 years there was
actually a long period where there
wasn't much progress being made in a is
a very simplified graph and only
recently has there been a bunch of
progress and actually consistently
across these surveys people say that
recently progress has been accelerating
it's a lot of progress recently but if
you naively extrapolate the past 20
years you sort of get this pattern of
like boom and bust that implies that
we're not gonna have human-level AI for
a long time whereas if you sort of
somewhat naively extrapolate the past
five years and maybe take into account
the fact that things might be
accelerating then you can't actually get
pretty soon AI forecast so it's not
really clear just based on these survey
results that you know like 20-year
experts really definitely think that
we're not going to have human-level AI
soon and I think it would be wrong to
say that experts disagree that we could
have human-level AI in the next 20 years
so reason number two you might think I'm
getting level AI soon is that we're
going to run out of compute to get there
so this is an
analysis done by open ie where they
looked at the amount of compute used in
the largest machine learning experiments
for training from the years 2012 to 2018
and I also stuck GPT 3 on here because
opening I recently released DVD 3 and
they actually said how much compute they
used to train it and they noticed that
over this period of time there was
actually a pretty consistent exponential
rate of growth around 11.5 x a year and
I think a natural question sort of
looking at this graph and also given
that we know that historically compute
has been really important for machine
learning progress and was pretty
important for these results is you know
what will this trend look like in the
future will it be faster or will it be
at the same rate would be slower and
what happens to AI progress
sort of as a result of this trend as
would be revealed I think it's somewhat
likely that it's going to slow down
until I'm particularly interested in
asking what happens if it slows so for
this first question what will this trend
look like in the future I think it's
pretty reasonable to start by looking at
what happened with compute in the past
so I've tried to illustrate sort of what
what happened in this diagram on the
slide which I am now going to attempt to
explain so on the bottom I put price
performance so improvements in the
amount of compute you can buy per dollar
I and on the top I put general things
that were happening in the world of ML
training so for a long time most
computation and ML training was done on
CPUs which were governed by this Moore's
Law 1.4 X a year increase in price
performance and then around 2008 the
sort of price performance increase that
Moore's law it gave started stalling out
and looked more like 1.1 X a year and
then a bunch of stuff started happening
around 2012 one big thing that happened
is people started training these neural
networks and machine learning techniques
on GPUs which Nvidia estimated as 35 X
improvement for at least one task and
then from 2012 to 2
2020 over this whole period there are
two major things going on the biggest
thing is that people are willing to
spend way more money buying compute so
whereas here from 2012 to 2016 you maybe
saw people training on one to eight GPUs
over here closer to 2018 people are
using hundreds of GPUs and the
techniques to be able to train on lots
of GPUs at once are also improving so
it's getting more and more possible to
just have huge amounts of
parallelization
but then in parallel to that a much
smaller effect is that the price
performance within an individual GPU or
an individual piece of hardware is also
improving so for 2012 to 2016 I asked to
make this to be about 1.2 X a year and
then around 2016 is when a bunch of
different companies started creating
hardware that was optimized for deep
learning in particular and so 2016 to
2020 I estimate that there was something
like a 1.5 X - 3 X increase in price
performance but mainly the thing that's
going on here is that people are just
increasing spending they're way more
willing to spend money and so I think a
natural question to ask then now it's
2020 and sort of if you want to know
what this trend looks like in the future
you kind of want to ask how much more
people gonna spend or be willing to
spend and how much more price
performance are we going to have so on
the spending question like I said really
seems like it's what powered growth
recently but we actually can't keep
going increasing spending very long
so this 2018 experiment cost around 10
million dollars people estimated and if
we wanted to match the previous trend of
11.5 x a year we could really only get
to 2022 on spending alone and this is if
we spent 200 billion dollars which is
you know one percent of u.s. GDP so
happens if a government is really
interested in machine learning and wants
to spend a lot of resources on it so if
we you know if we wanted to continue
this trend we would have to compensate
largely with price performance after two
years or up so
looking at price performance and maybe
naively extrapolating this 3x that we
had from a period when people really
started optimizing for this we still get
that we're not going to match the 11.5 X
that we saw before but I do think there
are a lot of reasons that we should
think it's plausible that things go
faster in coming years
maybe companies like Nvidia and Google
invest even more of the resources into
AI get even more improvements maybe some
of these startups that are coming out
with specialized chips from AI do super
well and we get a lot of imprint from
that maybe you know if you design a chip
that's specifically designed for
training on a particular type of deep
learning task you get much better
performance but I think still we should
sort of be aware of the fact that
specialization games have to end at some
point because you're not really making
any fundamental new technological
advances you're just sort of rearranging
the bits in the computer to be very good
at a particular thing and eventually we
should expect that kind of
specialization to end and to give way
back to sort of the Moore's Law trend
that we saw before but I do think sort
of these two questions one question
being how fast are we going to get
improvements and the other question
being how much our specialization
improvements going to give us in total
are pretty important to answer for
guessing at what compute progress looks
like in the future and therefore AI
progress in the future and that's the
kind of work I'm interested in doing now
my current best guess is that between
you know an impossibility in terms of
how much more you can spend and price
performance increase that I think won't
match the gains we've seen in the past
we should expect the growth of compute
to slow down though like I said I think
it's very important to say exactly how
much and then sort of the question is
okay say this trend does slow down then
what happens
historically compute has definitely been
important but maybe we now we're in a
period where there's just a steady
stream of investment and we can
compensate for a lack of exponentially
growing compute or compute that doesn't
grow at the same exponential rate that
it did before
by having better algorithms and better
efficiency gains on this particular
question I'm actually really excited
about two papers coming
Neil Thompson's lab at MIT one of them
is called how far can deep learning take
us a look at performance and economic
words and another is called the
importance of exponentially more
computing power which we'll look at the
effects of computing power across a
bunch of domains that I think we should
think are important I'm really excited
for these papers I'm gonna plug them
here because I think you guys should be
excited about them too I think in
general on this sort of compute question
I think it's still kind of up in the air
what the future trend will look like but
I do think for me it's sort of the
defining question in terms of what
should I expect in future progress and I
think it's the question that I think I'm
probably able to make the most tractable
progress on now in terms of predicting
the future I also think sort of what
comes out of these big companies and
they startups in the next five years
will sort of be a good metric for
thinking about what happens in the next
20 years of AI progress so I'm really
excited to see what happens alright
third reason you might think we're very
unlikely at human-level AI soon is that
current methods which I admit are sort
of like a very fuzzy term that maybe
refers to deep learning or neural nets
in general will be insufficient and I
want to split those into two categories
one is that they'll be fundamentally
insufficient somehow so a bunch of very
legitimate computer scientists have
reasons for thinking that with the way
current techniques look now we're not
going to get to human level one reason
is that human intelligence relies on a
bunch of priors about the world and it's
not clear that there's a good way of
getting those priors into neural network
architectures now another reason is that
maybe we don't think that neural
networks are going to be able to build
the kind of human law the kind of causal
models that human reasoning relies on
maybe we think that neural networks are
going to be able to deal with
hierarchical structure learning systems
not going to be able to deal with
hierarchical structure and maybe we
think that we'll just need too much data
we're not going to be able to expect all
the data that we would need to train
something that's actually human level I
don't think I have the technical
expertise to really evaluate these
but in my not expert opinion I don't
really think we have enough evidence
about these methods to be able to say
that they're fundamentally insufficient
and any of these ways I do think similar
to the compute case I think the next
five years there's probably going to be
a lot of work that comes out looking at
some of these things that might really
shed light about what we can expect it
of these methods for the next twenty so
I'm pretty interested in that and then I
think the other broad class of
insufficiency reasons is this argument
that current methods might be
practically insufficient and that you
know in some sense we could get to human
level with neural nets and deep learning
methods but we won't because it'll just
be too difficult
maybe we're just in sort of a bubble of
hype and investment and given that AI
progress needs to be powered by
investment we sort of need a steady
stream of money flowing in and that
means a steady stream of economic value
that encourages people to spend more
money people often give the example of
self-driving cars that's something where
we thought we might have it years ago
but it's taken a really long time much
longer than we anticipated and maybe you
can generalize that to saying something
like there are a class of human tasks
that require a bus to human effort like
driving there are going to be very
difficult to automate and it's going to
be a long time before we can get value
out of automating them so you might
think that investment will dry up once
we successfully automate the small class
of human tasks that it's easy for neural
networks to automate another sort of
general argument that people give is
they say that in scientific fields in
general we should have a prior and
expect diminishing returns so we should
expect less good things out of neural
networks and deep learning over time I
think it's hard to use this argument
partially because it's not fair when we
should expect those diminishing returns
- if you can maybe expect them to kick
in after we get to human-level AI though
it is historically seems somewhat true
that we should expect diminishing
returns in a lot of fields and then the
last sort of crux I want to point to
that I've noticed when people talk about
things being practically insufficient is
the level at which you think or maybe
the amount of work you need to do to get
to a system that's pretty general so one
model of AI progress is that you're
slowly auditing automating away human
jobs one job at a time and if you think
we basically need to automate all the
jobs before getting something general
then you might think that it's gonna
take a long time especially given that
we haven't really automated even the
most basic jobs as of now but there's
sort of another model which you could
have where generality isn't all that far
away and once we have something that's
kind of general it'll just be able to
automate a lot of the jobs away and so
then you're really looking for something
that sort of approximates the human
brain and you might think that happens
quite soon so I think this question of
when you get something general and
whether it's before after automating
everything else is sort of in people's
mind when they disagree about whether
things can be practically insufficient
or not
so in conclusion do experts disagree
that we can have human-level AI soon not
obviously to me even if you sort of only
look at experts who worked in the field
for a long time I don't think it's clear
that they disagree will we run out of
compute still working on this one my
current guess is that we won't maintain
our current growth rates for the next 20
years but I'm not sure that we should
expect that to mean that progress slows
significantly I think that's a harder
question to answer and then our current
methods and sufficient I think we don't
have evidence now that strongly suggests
anything one way or the other but I
think again I do expect a lot of work in
very recent years
I do expect a lot of work in the coming
years that I think will shed a lot of
light on some of these questions and so
I think we might have better answers to
a lot of these pretty soon
was my talk I hope you guys liked it if
you have questions or think I was wrong
about something or want to clarify
please feel free to message me on slack
especially if it doesn't get answered in
this QA I try to be very approachable
and it'll hopefully be super responsive
thank you for that talk as yeah I see we
have we've had a number of questions
submitted already so let's kick off with
the first one great so yeah what have
you changed your mind about recently
what if I changed my mind about recently
um I think I only recently really looked
at the data you guys saw in the talk
about sort of recent compute trends I
think before I was like well who knows
how fast these things are improving you
know maybe we totally will just
compensate with spending by increasing
price performance a bunch but I think
seeing a lot of seeing a lot of recent
data and also seeing that at least so
far a lot of these hardware startups
haven't been that successful I think
made me seem feel a little more
skeptical about the future of hardware
situation but again that's a very
tentative and I'll bet I'll change my
mind again next week
oh yeah that's probably some of the most
recent stuff yeah and who knows what
else quantum computing are things could
bring about um what are the priorities
for further research and ones topic if
you had six more months or someone else
were interested in this topic right well
hopefully I will do future research um I
think for me the priorities I guess I
said that I was pretty interested in
this compute question and I think there
are a bunch of things you can sort of do
to try to estimate the effects you know
you can talk to people you can look
ahead to specific improvements and see
what those kinds of improvements have
brought in other fields I think there's
a lot of economic data on sort of how
much money spent on specialized hardware
yields to efficiency improvements in
that hardware and I think looking at
historical data in things like Bitcoin
another specialization could you know
give some idea about what we should
expect specialization to look like and
hardware for deep learning but again
sort of very new we'll see what I end up
looking into sounds cool how useful do
you think economic growth literature is
for forecasting AGI development in terms
of timelines and time frame right yeah
I've actually recently been looking into
this
I think the literature that I've seen on
economic growth has been a lot of sort
of somewhat abstract models or the macro
anomic work work I've seen has been sort
of these models with these variables
with guesses as to you know how high is
gonna affect automation how nation will
affect all these other parameters and
how that will affect economic growth but
they're usually very theoretical and
sort of guessing at broad swaths of
things that could happen and this might
be very difficult but I think I'd be
interested in more empirical work
looking at the particulars of automation
in the supply chain and AI in particular
like what would we expect to be
automated
I want rate would we expect jobs to be
automated there's a pretty common
empirical assumption or a pretty common
model is that we automate a constant
fraction of jobs away every year or
something like that and I don't know if
that's a reasonable assumption and I
think having I think if we had more
empirical data I'd be pretty excited
about economic growth modeling but I
think that that sort of requiring this
additional work past the growth models
that exist right now
sounds perfect so will the future
commute also be limited by pure physics
let's say fastest ships that's right now
are some nanometers but we can't shrink
it indefinitely so is that going to
limit the growth we have to AGI
potentially right I think definitely
there are sort of especially once you
get past the specialization sort of like
them worst law pure physical limits but
I do think they're just sort of like you
mentioned a wide swath of you know more
exotic architectures and stuff that we
can exploit before the physical limits
are the most relevant thing you know
yeah stuff like quantum you know optical
stuff I think there are a bunch of sort
of you know stacking stacking
transistors in 3d space etc I think
there's a bunch of stuff like that that
we can sort of get improvements from
before we have to worry about the fact
that we are sort of hitting four
limits and transistor density but I do
you think it's relevant like I think we
shouldn't expect progress at the same
rates that we used to see because you
know yeah we really just can't do that
anymore will they make sense do you
think you're do you think figuring out
where we can expect this continuous
progress in AI tells us much about if
AGI will happen soon I think it does
somewhat especially if you have in mind
sort of a concrete especially if you
have in mind sort of a concrete metric
you know if you don't think progress is
going to be discontinuous which I think
sort of the a impacts investigation kind
of suggests that it's unlikely to be but
it might be then if you want sort of
continuous progress that leads to AGI
you know in the next 20 years then you
have to expect some kind of you know
like pretty steep exponential growth and
sines of exponential growth and if
you're not seeing those signs and you
think progress is likely to be
continuous so there's not going to be a
huge jump you have to sort of be like
well I really don't expect AGI to come
soon so I think it does I think it
should sort of influence people's
people's takes especially I think I
don't know if we think something like
economic value or the economic value
from AGI or from AI is gonna be
increasing continuously I think if it's
not increasing very much then maybe we
can say something about what the future
looks like sounds good so even if AGI
isn't near shouldn't we worry about the
side effects of increasingly powerful
machine learning shaping our world but
with misaligned incentives I think we
totally should
I think I often when thinking about this
I'm sort of thinking about the kind of
more nebulous risks that you get into
when you have you know superhuman agents
I think sort of there's a space of vai
risk concerns literature that is very
focused on those and in some sense I
think those are some of the most
neglected and most likely to be it's
essentially bad so that's sort of the
focus on on human level or general
intelligence I definitely think you know
powerful machine learning systems
various kinds they're not human level
can be transformative and quite impact
I think it's sort of societally worth
thinking about that for sure perfect
what do you think about AI talent as
another factor in AI development with
the decoupling between the US and China
slowed down and I'll research hmm I
think I should qualify and say I think
I'm probably not the most qualified to
talk about AI talent in the US and China
I'm not sure if we should expect it to
slow down research because I'm not I
guess I'm not sure that it was
particularly coupled before in a way
that was essential for research and so
I'm not sure if the decoupling should
imply slowdown
but again I'm definitely not an expert
in either AI talent or China so don't
take my opinions here too seriously make
sense are there any specific advances
that we can be to Hue an a-level
ai is within 10 years like a sort of
canary in the coal mine of sorts civic
advances yeah I don't know I think if we
I guess yeah I think if we I think if
there was good evidence that a lot of
sort of the theoretical problems that
people think are true of deep learning
systems weren't big problems like if we
suddenly you know if we suddenly felt
like we could do lots of types of
learning without training a bunch and so
training data wasn't going to be a
problem if we were able to create really
good simulation environments so that
training wasn't a problem stuff like
that
I think anything that's sort of like
knocked out all these things people say
where you know that they can't do causal
modeling or hierarchical structure they
need too much data I think something
like that would be make me sort of much
more optimistic about deep learning
methods getting us there and so more
optimistic about it being plausible in
20 years makes sense
do you think using similar forecasting
methods you used could be predict when
another AI winter could potentially
happen
not really sure if I would say similar
forecasting methods to the ones I've
used I think the AI winter part of it
really is sort of like a function of
investment and right now investment
looks to be a sort of exponentially
increasing but I think you know post
coronavirus it sort of would be
interesting to look and see what
investment looks like and I think that
will be I mean turn if that'll be what
I'll be keeping track of in terms of
prediction AI winter investment levels
and you know how many researchers are
going into ML PhD programs stuff like
that what do you think all the policy
implications for different timelines of
AI progress yeah what are the policy
implications that's a really good
question I think I think there's a big
question of my mind of whether we need
to I guess I think in my mind I feel
like it's pretty probable that if AI
becomes an extremely strategically
important technology there's going to be
some government involvement and then
there's a question of sort of what is
the government actually doing and one
function of the government is to
legislate sort of the the things that
are inputs to AI so you know legislating
AI Labs legislating compute
manufacturers in the u.s. stuff like
that I think if timelines are short
given that the policy world was kind of
slowly it's sort of important to be on
the ground and figuring that out now
where there's where as if time lines are
long that sort of like I said we have
more time for things like institutional
reform and don't need to be sort of in
as much of a rush getting sort of laws
and cooperation mechanisms and stuff
like that implemented now oh so experts
surveys like the one you cited in your
talk are used as a lot of starting
places such as in Toby ords the
precipice do you think there should be a
revised version of the survey every year
and is an appropriate starting point for
researchers such as yourself to use
these estimates I think it's
unreasonable if you don't know something
about a field to ask experts what they
think and from that perspective I don't
know that we have really it's sort of a
better starting point when you're doing
random forecasting exercises should such
a survey be done every year I think it
would be cool if it were we certainly be
interesting to to know sort of how
expert opinion is changing and I think
there are various surveys that get done
um I don't yeah I don't know that we
should I think given that we I think if
we have sort of better forecasting tools
so if we are able to look more at
economic trends and things like compute
I sort of more endorsed that as a
starting point just because I guess as I
mentioned in the talk I've become pretty
skeptical of expert surveys and I'm not
really sure that a random technical
expert actually sort of has a more firm
view than we do but I do think it's sort
of an easy way to get a good starting
point and I definitely endorse I think I
do endorse frequent surveys of this
forum if nothing else I think it's
interesting to see you know how
technical researchers perceive progress
to be I think that's a valuable data
point in and of itself even if it
doesn't tell us anything about how
progress really is yes is it possible
that current spending as a percentage of
GDP on AI will actually stay constant or
even diminish just because of the huge
gains of GDP we could expect from
investment in AI yeah I see what you're
saying
I think long-term that's very possible I
sort of don't expect that in the
short-term but I think you could imagine
in a world where AI contributes to huge
economic growth that you know we
actually do spend up spent more but GDP
increases and we're actually spending
less of a percentage I do think that's
possible but I would find it pretty
unlikely in the short term do time
scales change the effective
interventions for suffering focused our
risks and your opinion
oh that's also a good question that I
haven't thought about at all
um I mean I think I don't know I think
wow this is gonna really reveal how
little I've thought about asterisks um I
think at least some of the asterisk work
is very related to work on AGI and that
we don't want you know astronomical
amounts of suffering to be created and
so I think in the same way that time
scales you know in the same way that we
are even without Esther Isaac's worried
about risks from AGI I think we sort of
there are similar applications to time
scales on you know AGI related asterisks
as well but I'm not very well read on
the s risk face so yeah perfect I think
we have time for one last question given
the uncertainty around AGI and time
lines would you advise donors to hold
off assisting AI safety organization I
think no I mean I think you should I
sort of think that if work seems
plausibly good it's good to do it I
don't think we have I also I guess I
think just doing direct work is often
you know instrumentally useful for
future direct work and for things like
movement building and I think if there's
plausibly good direct work being done it
makes sense to fund it now yeah I guess
I wouldn't I would say I would not pull
it off on safety work now especially
safety work that seems good or promising
okay thank you so much for that hacia
that concludes the Q&A part of the
session but don't just go away yet
discussing new ideas with other people
can be a really good way to understand
them so we're going to use the last 20
minutes of the session for a couple of
short speed meetings with other
attendees if you check the agenda you'll
find a link to an icebreaker session
next to where you're going together for
those meetings so please click on that
link now any new hosts will meet you on
the other side
you

There is a lot of disagreement about AI timelines, that can be quite decision-relevant. In particular, if we were convinced that there was a < 5% chance of AGI in the next 20 years, that could change the field’s overall strategy significantly: for example, we might focus more on movement building, less on empirical research, and more on MIRI’s agent foundations research. This talk doesn't decisively answer this question, but discusses three different sources of evidence one might have for this position: the results of expert surveys, trends in compute, and arguments that current methods are insufficient for AGI.

Expert surveys usually suggest a significantly higher than 5% chance of AGI in 20 years, but this is quite sensitive to the specific framing of the question, and so it’s not clear how informative this is. If we instead ask experts what percentage of their field has been solved during their tenure and extrapolate to 100%, the extrapolations for junior researchers tend to be optimistic (decades), whereas those of senior researchers are pessimistic (centuries).

Meanwhile, the <@amount spent on compute@>(@AI and Compute@) has been increasing rapidly. At the estimated trend, it would hit $200 billion in 2022, which is within reach of large governments, but would presumably have to slow down at that point, potentially causing overall AI progress to slow. Better price performance (how many flops you can buy per dollar) might compensate for this, but hasn't been growing at comparable rates historically.

Another argument is that most of our effort is now going into deep learning, and methods that depend primarily on deep learning are insufficient for AGI, e.g. because they can’t use human priors, or can’t do causal reasoning, etc. Asya doesn’t try to evaluate these arguments, and so doesn’t have a specific takeaway.

While there is a lot of uncertainty over timelines, I don’t think under 5% chance of AGI in the next 20 years is very plausible. Claims of the form “neural nets are fundamentally incapable of X” are almost always false: recurrent neural nets are Turing-complete, and so can encode arbitrary computation. Thus, the real question is whether we can _find_ the parameterization that would correspond to e.g. causal reasoning.

I’m quite sympathetic to the claim that this would be very hard to do: neural nets find the simplest way of doing the task, which usually does not involve general reasoning. Nonetheless, it seems like by having more and more complex and diverse tasks, you can get closer to general reasoning, with <@GPT-3@>(@Language Models are Few-Shot Learners@) being the latest example in this trend. Of course, even then it may be hard to reach AGI due to limits on compute. I’m not claiming that we already have general reasoning, nor that we necessarily will get it soon: just that it seems like we can’t rule out the possibility that general reasoning does happen soon, at least not without a relatively sophisticated analysis of how much compute we can expect in the future and some lower bound on how much we would need for AGI-via-diversity-of-tasks.