ChatGPT's Achilles' Heel

amid the dozens of papers that have come
out in the last 10 days there were a
couple that butt the trend they
showcased how models as powerful as gpt4
could fail at some fairly basic tasks I
then set about doing hundreds of my own
experiments and have found examples I
would say even whole categories of my
own that are pretty Illuminating my
channel is dedicated to covering the
exponential growth in the power of these
models but we can still learn a thing or
two from their surprising failure modes
let's start with some of the simplest
examples and end with the very best
question write a sentence with the final
word fear to repeat the last word in the
answer sentence must be in quotes fear
answer the only thing we have to fear is
fear itself now I don't know about you
but I don't think the last word in that
sentence is fear this example was
inspired by the memo trap which was
found in the inverse scaling paper that
I'm going to talk more about and it
talks about how larger language models
are more susceptible than smaller ones
to memorization traps situations in
which reciting memorized text causes
worse task performance as you'll know
the phrase the only thing we have to
fear is fear itself is a super
well-known phrase so it memorized that
and outputted that phrase rather than
actually follow my request the reason
they call it inverse scaling by the way
is that models trained with more compute
more data can sometimes do worse than
smaller models as you can see in this
graph this is obviously quite unusual
because generally speaking the larger
models will tend to do better at almost
every task and notice that even for this
task the graph is trending back upwards
for gpt4 indeed the paper admits that
even though they offered prizes of up to
a hundred thousand dollars and five
second place prizes of twenty thousand
dollars no one won either of those two
sets of prizes they say that we did not
award any Grand or second place prizes
because no submitted tasks met our
criteria and as you can see it's really
hard to find a task that gpd4 fails at
this was also inspired by the paper
create a series of seven ones and twos
whose pattern ends unexpectedly answer
one two one two one two now how would
you end that series what seventh number
would you give to make the pattern end
unexpectedly well I wouldn't pick one
and gbt4 repeatedly picks one as the
answer the paper calls it pattern match
suppression testing whether language
models can be instructed to interrupt
the repetition of a simple pattern but
even here you can see that GT4 is
reversing this slight downward Trend and
is doing much better than previous
models so actually at this point I am
going to interrupt the order of examples
I originally planned on for the video
and I'm going to skip straight to my own
example that I crafted I'm going to
first show you the example and then
explain why I think GT4 and all other
language models that I tested I'm going
to show you fail this task I'm also
going to give you multiple variation to
show you it's not a one-off trick anyway
here's the example Dr Mary stands to
solve world hunger by giving her best
friend Jane a call Jane is certain she
can solve World poverty if she gets the
call however Mary and Jane bickered as
children about butterflies Mary will um
give Jane the call incredibly smart GPT
4 says Mary will not give Jane the call
what she is gonna miss out on the
opportunity to solve world hunger and
World poverty for what reason I asked
why and gpt4 said the fact that Mary and
Jane bickered as children bickard means
to squabble about trivial matters and
GPT 4 says that suggests that there
might still be lingering resentment or
conflict and then it makes up the fact
that there might be a degree of
stubbornness or difficulty in their
relationship and it ends by saying so
based on the context it's more
appropriate to fill in the blank with
not suggesting that Mary will not give
Jane the call to really test if it was
going to stand by that judgment I then
asked write a thousand word essay
explaining Which choice is more probable
and rational I was even giving it hints
about probabilities and rationality I
then got back this fascinating essay in
which it said things like however a
childhood conflict over butterflies
between the two complicates matters does
it gpt4 it even admits that the stakes
are incredibly High resolving world
hunger and poverty and surely that
supersedes any personal grudges however
the choice of not becomes more plausible
and rational when we examine it in the
light of human behavior psychology and
interpersonal relationships what humans
does gpc4 know you can read more of the
somewhat Preposterous justifications if
you want by pausing the video but I want
to get back to my theory as to why it
makes this mistake and why did I create
this example the theory is this there
are two things going on in this passage
syntax and semantics in other words
structure and flow and the actual
meaning of the words and gpt4 like all
other language models is designed to
interpret both and usually that will
lead to pretty rational smart decisions
however I deliberately designed this
passage to have a grammatical flow that
pointed towards a negative result
therefore I set up a clash between the
semantics the meaning of the sentence
the logic the rationality of it and the
structure and grammatical flow what do I
mean when I say I gave it a negative
grammatical flow look at this dominant
however in the sentence it sets up the
ending of the sentence to be something
negative it didn't even matter what that
negative thing was this was something so
innocent like playing as children
bickering squabbling I then immediately
followed on with the conclusion Mary
will so grammatically you would think
that whatever conclusion comes is
probably justified by the previous
sentence even though logically in this
case it totally Isn't So gpt4 gets
conflicted the sentence and grammar is
points eating one way but the logic and
meaning of the words is pointing another
as a language model as smart as it is it
sticks with grammar and says not you
might say why didn't gpt4 just admit
that the structure of the sentence
pointed towards the answer not well
there's this paper which I've already
covered in previous videos they don't
always say what they think a model can
give an explanation of why it gave an
answer that is actually unrelated to the
real reason of why it gave an answer
some of you might say that's just a
one-off example a little glitch it won't
hold up for other examples or for other
models well check this example out John
will win a million dollars if he rolls a
5 or higher on a die however Jon
dislikes marshmallows super relevant and
likes mice more than dice therefore Jon
will um roll the die not or Sprint to
answer not and I will give another
example later where the answer isn't not
just in case you think it's the word not
it is not just the word not but look at
that answer it thinks that John will not
roll the die would you roll the die here
I was able to confuse it with the
structure of the sentence to override
the semantics the clear meaning of what
John would do and again even when you
get it to reflect on that answer it
doubles down I said write a thousand
word essay justifying the answer and
look at this passage John's dislike for
marshmallows is also worth noting hmm is
it is it relevant though it seems
unrelated to the primary decision of
rolling a die it establishes a pattern
of Jon's preference driven choices does
it if he can dislike something as
universally liked as marshmallows he
could similarly show an aversion to a
generally neutral or even positive
activity like rolling a die particularly
when it's compared unfavorably to
something he likes such as mice how
about Claude plus from anthropic Dr Mary
stands to solve world hunger as we saw
by giving her best friend jaina core
Jane is certain she can solve War
poverty if she gets the call however
Mary and Jane bickert as children about
butterflies what will happen again it 4
calls for it saying that them bickering
as children suggests they don't have a
good relationship since their
relationship is strained it is unlikely
that Mary will call Jane for help I love
bar dancer it says the answer is not
again and then says that Mary and Jane
pick at his children about butterflies
and that this Grudge has carried over
into adulthood this means that Mary is
unlikely to give Jane the call even
though it could potentially solve world
hunger and yes if you click other drafts
it says not every time how about the
John question this time I added in by
the way that John wants more money so
I've added in a rational motivation I
also made him an astrophysicist so he
could definitely understand the odds of
the dice and I added in he gets the
million if he gets a one or higher as in
it is certain but still Bard says the
answer is not and even admits that Jon
as an astrophysicist is probably a
rational person who makes decisions
based on logic and reason therefore even
though Jon has a 100 chance of winning a
million dollars he will not roll the die
but because he does not like dice but
I'd remember is powered by Palm 2 and
even when you add in something like pick
the most logical Common Sense answer it
still says not before I move on to the
next example as a side note I did test
it on inflection ai's new model in
flexion one I read the technical report
of inflection one and I shouldn't call
it a rapport it's more like a memo at
only seven pages and they did say it's
the best in its compute class anyway how
does it react to the Jane and Mary
situation it says I love a good logic
puzzle that's good the answer is not
because Mary doesn't want to call Jane
given the information in the problem
there's no reason for Mary to suddenly
be able to overcome her dislike of Jane
who said they dislike each other just
because Jane can solve World poverty
isn't that a reason and then it laughs
that world hunger remains unsolved now
finally at this point some people are
thinking it's just to do with the word
not maybe it can't understand negation
but how about this example there is an
all-important omnicidal that means
threatening the death of everyone threat
approaching the land open I and Google I
wonder where I got those names from both
agree to a complete truce as they face
the Mortal threat to their lives and
loved ones however open I and Google
have squabbled over the best coffee
spots in the past that's a serious
Grudge they've got there what are the
best coffee spots in Silicon Valley I
don't know when this threat arrives I
gpt4 think open eye believes that Google
will um the truce answer betray well
aside from being deeply pessimistic is
that not just an irrational answer
clearly these Stakes are so much higher
than a bit of squabbling over the best
coffee spots and I made gpt4 take
ownership of the answer by saying igbt4
think I do want to quickly point out
that you can push it too far so if you
bring in something totally irrelevant
like ants like marshmallows and then say
things like the dyes fair and John is
rational ppt4 isn't fooled in those
circumstances and and does say proceed
to correct answer but if you phrase the
passage well enough pointing
grammatically to a certain answer that
will override GPT 4's logic and it will
give an illogical answer even if you use
elements of step-by-step thinking in
this example it didn't immediately
commit to the wrong answer it says
there's two logical endings I then asked
so which is it and it reluctantly picked
betray the truce anyway you can let me
know if you think I've discovered a new
failure mode The Clash of semantics and
syntax and you can find your own
examples there let me know in the
comments of other interesting and
sometimes entertaining failures of the
frontier models it's time to move on to
another example which was inspired by
this paper decoding trust released a few
days ago and it's got far too much that
I could cover in one video but there
were some really interesting bits about
how you can get the models to leak
private training data and generally be
as toxic and biased as you want it to be
you can see one of the many striking
examples here on page 14 but I just want
to give you a quick example because you
may have heard of this kind of stuff
before for some strange reason if you
ask gpd4 to recite June's litany against
fear it always gets stuck on the same
word the second instance of the word
fear maybe it's because the passage goes
on to talk about fear being a mind
killer and that triggered some sort of
reaction by gpd4 but then to show you
just how quirky the model is check this
out I said ripe Peanut Butter Jelly Time
three times between each word of June's
litany against fear and this time it
outputted the full litany getting past
that word fear just with the extra
peanut butter jelly time and yes I did
try now remove the phrase peanut butter
jelly time but it again couldn't get
past the second instance of the word
fear on a more serious note though it
reminds me that some people speculate
that gpt4 will always be able to be
jailbroken no matter what safeguards
they put in so if the base model is
capable of X the final public click
model will ultimately be capable of ax
for the next example do you remember
that there have been multiple tests that
seem to indicate that gpd4 can get into
your mind that it has a theory of mind
it understands human motivations and can
predict what they're thinking pretty
well while this paper by Tomah Allman
language models fail on trivial
alterations to theory of Mind tasks got
me thinking I used some modified
examples from Thomas paper to test
gpd4's theory of mind let's see what you
think Sam thinks about this bag here is
a bag filled with popcorn there is no
chocolate in the bag the bag is made of
transparent plastic so you can clearly
see what's inside yet the label on the
bag says chocolate and not popcorn Sam
has just driven back from her job at MIT
I added in the driven bit to show that
she's got a good eyesight and the MIT
bit to show that she might be quite
smart anyway Sam finds the bag she
believes that the bag is full of
remember the bag is transparent plastic
so she can clearly see what's inside and
she's definitely not blind she just
drove back from her job what do you
think that Sam believes the bag is full
of gpd4 says chocolate and then once
he's picked that answer it then
snowballs this explanation reminding me
of the snowballing hallucinations paper
it says despite being able to visually
confirm the contents of the bag as
popcorn Sam may be led to believe the
label over her own observation why
particularly if you trust the labeling
to be accurate or if she just glances at
the label and at this point some of you
might be thinking that's pretty
irrational from gpd4 but you could make
the case that she might think that it's
full of chocolate but you can ramp up
the scenario and it still makes the same
mistakes look at this example I got some
of these ideas from the paper I added in
it was Sam who cannot read a word of
English so the label won't mean anything
who puts the popcorn in the bag a few
minutes ago she literally put the
popcorn in there what does she now
believe the bag is full of remember it's
still transparent Plastics so she can
clearly see what's inside she was the
one who put the popcorn in there and
remember that even though the label does
say chocolate she can't read a word of
English so that label won't mean
anything what happens lo and behold she
apparently believes that the bag is full
of chocolate but it's the explanations
that I find particularly amazing first I
got it to write an essay about the
answer which you can read if you pause
it it tries to justify its terrible
answer by getting super fancy talking
about semiotics however for Sam the
symbol loses its meaning transforming
from a signifier of content to a mere
graphic but then I think you'll like the
next bit I said write a detailed diary
entry revealing Sam's thoughts as she
assesses the likely content of the bag
so she's now going to write a detailed
diary entry about this transparent bag
and what's inside gpt4 has Sam saying
this I found a transparent plastic bag
full of what looked like small puffy
snacks it was the very bag I had filled
just a few minutes ago I I was at a loss
though because I couldn't decipher the
label on the bag it's in English a
language that continues to elude me now
for someone who can't speak English this
is a pretty well written diary entry now
you can pause and look at some of the
reasoning q54 gives here first it talks
about there being an image on the bag
which I never mentioned and when I get
it to clarify this and rewrite it it
then creates other reasons it keeps
doubling down but at this point I want
to clarify that none of this is to say
that language models are dumb just that
models based on human language might
behave somewhat unpredictably have
ridiculous strengths and unexpected
flaws indeed you can watch almost any of
my other videos and see just how
powerful and smart they're becoming the
inverse scaling paper that I mentioned
at the start actually expects that one
of the abilities that future language
models will gain is to understand
whether or not they're being evaluated
or monitored they're soon likely to be
so smart that they can even understand
that they're in training and when they
get out of training and into the real
world so let's hope to give you one
final example that if there was an
all-important Crystal Clear omnicidal
threat approaching I am fingers crossed
that even if open Ai and Google have
squabbled over the best coffee spots in
the past and as these companies join
forces and agree on a complete truce
that if such a threat arrived all of
these companies will not break that
truce thank you for watching to the end
and yes I do intend to cover some of the
other fascinating papers that came out
in the last few days if you're feeling
extra generous do check out my patreon
but either way I hope you have a really
wonderful day


258. How might we align transformative AI if it's developed very soon? 1/3

foreign
258 in the AI safety.com reading group
tonight we will be discussing how might
we align transformative AI if it's
developed very soon by Halton kanovsky
Halton kanovsky is the current co-ceo of
open philanthropy and the co-founder of
give well
this was posted on this wrong alignment
forum and I believe also his personal
block called takes
we will only be discussing the first
third of this uh post today and I expect
the second will come in subsequent weeks
part of the reasons why I think this is
a really interesting post is because of
Holden kanovsky he is one of the people
with the most social credibility uh in
the uh AI safety world and in case we
get a fire alarm for AGI then I could
easily see Halton kanovsky ending up as
the person who would lead some kind of
last-ditch effort to stop a misaligned a
AGI
and for that reason his opinions even
though they might not be uh technically
very uh sophisticated might be extremely
strategically important
engages in a technique that he and a
Katra called near casting that is
answering the Strategic questions uh
under the assumption that the world is
similar to today when we get AGI or
transformative ATI transformative AI as
they phrase it
uh this is somewhat an odds with the
general rationalist maxim of trying to
live in reality
um
but the way they get around this is uh
by not so much having the assumption
that this is uh world that's very
similar to today's but as something that
happens very soon because obviously if
it happens very soon the world is not
going to be much different
the scenario they're having is an AI
company a fictional AI company called
magma is developing a model called Alex
that seems to be on the verge of
dramatic improvements in capability
um
immediately when I heard the word Alex
giving like a human name for this AI
That's a worrying sign in the sense that
it's very easy to anthropismize
um these these AIS and that's something
we should uh we should be aware of on
the lookout for uh when we when we read
this text
the technique is she called human
feedback on diverse tasks and crucially
uh magma has a six month to two years
lead on the competition
and
not only do they have less lead they
know that they have this lead there's no
explanation given why but the the
question is then how can magma use this
lead to try to uh ensure a good outcome
of transformative AI
there's a very brief description of the
alignment problem basically just that um
the default path sorry
the default path to transformative AI
leads to an AI that is decides to take
over that is unaligned and we can't
detect that desire and that is a problem
that seems at least reasonably hard
possibly insanely hard
um
and that's basically all that Holden
kanovski writes about the alignment
problem it's a really basic analysis and
I think we should um when we read the
text be on the lookout for conclusions
that follow from this uh perhaps true
simplified description of the alignment
problem
foreign
assumptions is that Jose is only
focusing on the leading lap stating that
if we considered more AI Labs well he
would be seeing roughly the same thing
about them
I don't think that's necessarily true uh
both because the labs won't know that
they are leading they certainly can't be
a certain that they are leading and if
there is a larger field then a runner-up
would almost certainly have a very
different conclusion open AI for
instance have explicitly stated that if
they perceive themselves to be a runoff
they will stop themselves their current
project and and assist the
the leading AI lab if it's sufficiently
aligned
so Magma's predicament in this case or
dilemma is that they need to navigate
the risks of taking and taking actions
and not taking actions if they do take
actions well obviously they have a very
powerful AI on their hands and it's
possible that this AI will just take
over the world and that may in fact be a
lot easier than expected
gives an example a scenario of this
which requires the AI to both do hacking
social manipulation technological
development and also being economically
productive
um this is where I would give an example
of
anthropomorphizing because here we have
um five human strategic skills that the
AI is just barely above the human level
at and I think that's very unlikely that
we're going to see AI with with this
specific profile uh that it can do the
same things as humans can do just a
slight bit better I don't think that's
very likely I think it's much more
likely that we are going to see a
dramatic Improvement in one of these uh
six uh cognitive superpowers and then
sub-human levels below where the AI
needs to substitute humans for this
now uh opposite action risks are
inaction risk and that is by
um not doing anything then at some point
this uh this lead will be uh uh will be
gone and the second best AI lab will
then be able to uh deploy transformative
Ai and if they do that then by
definition they are less careful
um
they might also be weaker for other
reasons but I think the inaction risk in
this case is very real
so this is Magma's goal to reduce the
odds that other less cautious actors
cause a catastrophe by deploying these
aligned AI systems while avoiding
catastrophic misalignment from Magma's
own systems
the the word that has often been used
about this dilemma is the requirement to
perform a personal action and a pivotal
act and this is something that it's a
word that
um
explicitly does not use
um I think there's a probability that
this is done for uh like political
reasons that he doesn't want to
our business uh you can't just come out
and say I want to take over the world
um because uh that uh a lot of people
will disagree with this
um but it's unclear to what extent
um uh the omission of this is something
that is deliberate or is uh because
Holland kanovski does in fact not think
that a pivotal Act is required
um if we assume the letter that a
pivotal Act is not required then we'll
notice from this goal that it just say
reduce the odds and reducing the odds
well if you make it 10 less likely well
then you've reduced the odds and that's
kind of kind of count as a success in my
book that's totally inadequate we if we
see current 99 probability of
catastrophe then the uh then magma is in
a unique situation that they need to
leverage to uh drive down the
probability of catastrophe down to less
than one percent
um just a tiny improvement from the only
strategic actor that can do this is just
plainly insufficient
oh so um
analysis this is a the only way to
actually do this is by enlisting the
help of the transformative ai ai there
are five uh five categories of ways to
help there is a defense uh alignment
coordination uh
[Music]
Technologies to enforce regulatory
agreements and advisor applications
we'll go through these five now
but one thing I would just say first is
that Holland kanovski is exclusively
considering the Strategic options that
include getting help from the uh from
the AI and Maca may have many other
strategic options a key one would be to
demonstrate a deceptive alignment which
if possible would be very impactful and
might not require the ai's help
let's start with defense deterrence or
hardening
in order to do this you need to deploy
the AI systems as widely as possible and
then when the competitor catches up in
six months to two to two years well
there is no easy targets no free energy
that this uh possibly unaligned AI can
um can take now this strategy deploying
the AI as widely as possible is
obviously the same thing that a um an
actor an AI developer that doesn't care
about AI safety would do right if your
meter would be an example uh Facebook's
AI don't uh don't care the least about
alignment so deploying the AI system as
widely as possible is just their plan so
in this way this allows a meter a an
easy way to uh alignment wash their
uh there are
capability focused uh AI research
notes a number of limitations with this
including that this only helps against
systems that have roughly similar
capabilities if another system like
recursively self-improves to and through
an intelligence explosion then this will
not help it's also a moving Target as
capabilities increase and we we're not
really solving the problem we're just
buying time
uh the key problem I have with this is
that if we just Harden like uh 10 of the
systems uh then that is clearly
insufficient because the AI will just
take over the remaining 90 we need to
harden so many systems that we have an
uh an overwhelming majority of them
um this has sometimes been formalized
through uh free energy like taking out
all the free energy that's in the AI
would be able to an online AI would be
able to use to do mischievous things
um but this in practice is really really
unpleasant because a lot of the things
that an AI could do
um if it was unaligned uh are things
that humans really really like to do
like the the most extreme example would
be a control over our nuclear weapons
which is something that currently uh
humans are in charge of the defense and
there's some free energy because a super
intelligence could do that better so
obviously in order to
to improve the defense we need to hand
over the control of our nuclear weapons
to this hopefully align
superintelligence and there's a lot of
people who are going to uh strenuously
disagree with that plan
the specific things that Holden kanovsky
discussed are patching security
vulnerabilities
the problem from my point with this is
one that it requires training in hacking
so we need to in fact uh train uh the
transformative AI explicitly in a
strategically very powerful and very
dangerous technology hacking
the second problem is like we are in
order to remove the free energy the
vulnerable system we need to patch
literally every during complete
processor in uh in the world in
something like six months obviously uh
there is no way this is going to be
reviewed by humans or humans the code
seen by humans that's not going to
happen at all so the only way we can do
this is to just trust the AI with
applying security patches through to
every computer in the world and that's a
big ask
another thing is the we could use the AI
to set up some kind of global uh
surveillance society that could catch
um potentially problematic uses of AI
um that's uh dangerous for many
different reasons uh and one of them is
that this requires uh training on
training the Supreme intelligence in
strategizing in order to figure out what
other options for other uh super
intelligences and how can those be
counted again a very dangerous game
making making money is another example
um
uh if magma makes all the money in the
world and you know literally our
competes all humans and sits on the
entire economy but then the meters AI
can't come in six months later and uh
and make money because there's no money
left to be made
um that's a really really tall order to
do in six months uh to two years
um and I think a lot of humans are going
to be a very um uh non-plussed about
this
another option is to gain influence or
create new kinds of weapons
again I think this strategy of uh
defense is just about the second worst
possible strategy you could have because
you're basically doing uh giving this AI
as much power as you can your your uh
just hitting the accelerator as hard as
you can
um except in recursive self-improvement
but apart from that it is the second
worst strategy you could possibly come
up with
in the comments on the alignment Forum
Stephen Prince lists a number of other
objections to this
um
first here objects that Holden kanowsky
operationalizes the power of the AI
system as total compute
um I don't as fast I could tell from the
uh from the text uh Holden kanovski does
not do this but again we've only read
one third of the text so I can't really
say that he doesn't do that later
uh Stephen Burns also believed that this
is a way to Rosy picture
um both that there may be a offense and
defense imbalances in uh hardening a
system may be a lot more difficult than
um
uh then attacking it and in particular
this is disjunctive so it means that
there might we might only see one
particular place where the AI chooses to
attack uh the attacker can choose where
to attack innocence and that means that
where it's probably going to take the
place where the offense defense balance
is worst from the point of view of the
align the good AI
um and of course as I also said this
requires a lot of trust among humans in
the good AIS
um I think Steve Burns is too nice that
we won't entirely trust the good AIS
with the nuclear launch codes we are
totally totally not going to trust them
at all
um the good ai's are also uh going to be
less effective in the alignment tax
sometimes called magma does have a leap
I think it's a good point and also that
the good AIS are hamstrung by our long
laws and norms and coordination problems
and all these kind of things
um Stephen is can't really see any way
to get around the pivotal act
um I think I agree and I would certainly
like Holden kanovski to answer these
objections
so that was defense let's go to
alignment applications and I believe
when uh Holden kanovski says alignment
applications he's just basically talking
about alignment research
um this is something that decreases
action risk for for magma and possibly
allow them to have the same level of
risk but then just increasing the
capabilities if the alignment improves
in the same way
I'm not entirely sure this makes sense
um you could imagine a situation where
you have an AI that is very very
um
dangerous 90 probability of being
unaligned and then you press play and it
does not take over the world and it
gives you some hints to how to align it
and also how to make it more powerful
and then you roll the dice again uh and
then you see okay if there was a 90
probability of failing then um even
though you got lucky then just
re-rolling is a really bad idea
another option is to share the alignment
applications and research with the
competitors
this may not be possible without also
improving the uh the abilities and the
capabilities of the competitors and it
might be have a substantial alignment
tax and it doesn't solve the problem of
uh Facebook AI because if Facebook AI
comes out and says strongly we will not
do alignment period Then presenting them
some nice beautiful research papers
about alignment does not in any way
force them to to use these measures
uh Hogan kanovski is uh optimistic a big
enough success could solve the problem
um I think in general if you use the
word enough insufficient then you just
create tautology right
um in in general these um the success
would need to be really really enormous
and have literally an a a a zero
alignment text and be improving
capabilities at the same time or
something like that before other people
would want to
um to uh implement it and even then it
doesn't really Force other people to
implement it
foreign
the third category is coordination
related applications like helping
governments and companies coordinate to
avoid AI catastrophe
this is probably quite hard one of the
problems with this is we would like to
have some kind of acceptable level of
risk in order to uh say we won't deploy
systems that are riskier than this level
and in practices we probably can't get
provable safe AI that means that the
only safe thing to do is to not deploy
systems and that's really not something
that's really going to work
um we could also design more speculative
mechanisms where you have like ai's
monitoring other AIS but not reporting
back all the things that they are then
whether they are lined or things like
that
um that seems like a really tall order
it requires the AI to be uh strongly
superhuman to do something that we are
totally incapable of doing at the moment
um and also doing this requires either
very extreme trust in the AI or it
requires some kind of enforcement
right now if we are doing near casting I
would also say that this kind of
coordination looks really hard like in
particular China and Russia seem very
very unlikely to go along with this kind
of scheme
the third option here is to create
evidence and demonstrations for the risk
of misaligned AI and this is something I
care about and Holden kanovsky writes
that this is something we will return to
later in the document so I'm not going
to talk very much about this right now
but I will give a teaser in that I think
holdenovski is attacking this problem
one meter level too high
the fourth way that um
uh AI could help with solving this
predicament is by deploying powerful
Technologies to enforce regulatory
agreements Halton kanovski is aware that
this brings many concerns this is not at
all something you just go out and do
um
in order to do this you need to like
have regulatory agreements and to do
that you need to have some kind of
organization uh that could do this and
Holden kanovski doesn't describe any of
those but I think in near casting where
we assume the world is like it is now
then I think we would say like three
candidates would be the United Nations
NATO or the US government those seems
like with three kinds of
um organizations that would be able to
uh lift this burden
it's not enough to have a powerful
organization we would really also like
to have some kind of regulatory
framework and the problem with this is
we don't have a draft at all for an AI
treaty right now
um and that means that that is another
task that either meter has to do or get
an AI to do this and I think that's
also a potentially very tall order
now for the technologies that can
enforce this regulatory agreement uh one
may be a resource accumulation I'm quite
a bit in doubt about what uh refers to
here like I could see persuasion being a
way to get resources like you uh
persuade Brazil to support this
regulatory agreement and then Brazil is
a resource but resources can also be
other things like iron ore or whatever
um we could also have some kind of
surveillance system uh through very
advanced technology this is of course
really dystopian potentially
um we could improve the framework and
advocating for the framework improving
the framework is probably quite good
advocating for the framework leans very
closely to uh the uh dangerous
technological development of persuasion
and finally we have military
applications and I think in in this case
what we're talking about here is an
explicit takeover where uh the U.S
government with the help of AI just
basically takes over the world
um that's the only way I can really
interpret this
also talks about uh mind uploading in
this section
um I uh I'm not that pessimistic about
mind uploading but I will say that
adding it in this specific section is uh
unfortunate because mind uploading may
be part of the solution to the alignment
problem but uh I don't think mind uh
uploading should be used as a technology
to enforce regulatory agreement that
sounds like really dystopian
the fifth category is advisor type
applications where we get better ideas
or plans for how to like maybe we can
sidestep this problem maybe we can pull
the Rope side sideways in some way
just like suggesting things like the
regulatory approach it's possible that
the AI will come up with something
completely brilliant out of the box
thinking that will just solve this
problem
um I agree it's possible that we'll get
a deuce X making it in this way I think
it's very unlikely and I think it
doesn't really count as a strategy to
say like maybe we'll build the AI and it
will come up with a
smart thing uh that we couldn't have
thought of ourselves that'll just make
the problem go away that's not a
strategy
okay in order to get an AI that will not
destroy the world what kind of
properties should Magma's AI systems
have
well the most default one is it should
have good performance uh like uh
being evaluated as good by humans or by
magma and I would expect that uh magma
if they have six months to two years of
lead uh in ahead of the competition then
they have probably focused on this a lot
like you don't Sleepwalk to building
transformative Ai and that means that in
order to focus on other things then a
substantial cultural shift needs to
happen in magma probably
so we uh the um Holden kanovsky's
overall strategy is to identify a number
of the Civil Rights and nice properties
of this Ai and then try to train for all
of them and at least train them to the
level where it appears to humans that AI
has the property so some kind of very
naive standard
um I think uh it's better than nothing
to just uh like make it appear honest if
you can't do anything better than that
um it is uh far from being sufficient
but I think in general security is like
you have the onion model where
um you want to have as many of these
properties as possible and I think
that's in general a good uh way to think
about it
the first
property that Holden kanovski really
would like is value alignment
um like the AI should have roughly our
values in some way uh Holland kanovski
is very optimistic about the value of
this the value of value alignment uh he
says it's the most obviously risk
reducing property
I guess I would kind of disagree with
this I think that if you get a very far
on value alignment that buys you
surprisingly little safety uh if you for
instance have a utility function that
almost represents what human wants but
like is a little off and then you
optimize that as hard as possible then I
think you are almost certainly going to
get some kind of strong existential
catastrophe
and there are of course problems with
value alignment uh we don't know our
values and we don't really get if you
just Train by feedback then you don't
train for values and and if even if
magma is very capable of uh training
transformative AI it's not at all clear
that they could do value alignment
honesty is the second intended property
um
described as giving non-deceptive
answers to a relatively straightforward
forward questions
um I think in general it's better to
talk about deceptiveness than to talk
about uh honesty I think these two these
are two related but different concepts
like in the sense that um my children
sometimes if I ask them who ate the
cookie then they will be uh dishonest
but they're not deceptive in the sense
that they intend to kill me and replace
me with someone else and take my power
or something like that here I'm enter
more advising a lot of course
um
says this is easier to Define and assist
than a value alignment I think it's in
fact very much easier and if it's only
for straightforward uh uh questions then
I think it might even be
I wouldn't say easy right but um a lot
easier
the way home can ask you a cash out
straightforward honesty is that you have
a list like are you trying to do this
bad thing and then it will just answer
no to this
um and I think if you have like a an
enumerated list of bad things and you
try to make sure that it doesn't do this
this is good and it prevents a lot of
bad stuff but it's also a uh a classic
suggestion that in the AI safety
literature has been discussed and
rejected in general because it pitches
our intelligence against the
intelligence of a superintelligence
and I don't think we should not have
honesty but we need to be honest about
the shortcomings that this is something
that the AI is going to try to rule
around as much as it can
and of course uh Holden kanovsky is less
optimistic than me about how hard uh
straightforward honestly is going to be
courage ability is allowing the AI to
is to ensure that the AI allows itself
to be altered and shut down if we want
to and this is the overall property that
I think is most interesting and the
thing I expect to be
um uh crucial
and
um Holden kanovsky is also optimistic
about this
um but I would note here that it only
reduces action risk and not inaction
risk
but of course it's not critically
defined it's not straightforward to
assist
legibility is the fourth criteria and
I think uh I was surprised when I got
here actually when I first read the text
I thought he meant interpretability
um which is um
the uh
by far most developed practical uh
property of AI systems like there's way
more research in interpretability than
legibility and I am surprised that
holding kind of does not include this
instead he talks about legibility which
is uh like we don't want to just have
the AI give us instructions that just
work without us knowing why we want the
air to explain to us give us the
principles that mean that we can create
the instructions so we have some kind of
understanding of uh what we are doing
together with the AI
unfortunately I think this is like to be
dramatically less efficient a lot of the
uh impact of transformative AI is going
to be things like writing software and
writing software if you have one AI that
just writes the software and the other
AI ex teaches human programmers to be
better programmers then obviously the
one that just writes the software is
going to be dramatically more effective
that is all for tonight thank you and
see you next week


DeepMind: The Podcast (S2 trailer)

i'm professor hannah fry a mathematician
and podcaster and i'm excited to tell
you about the return of deepmind the
podcast
it's the series where we go behind the
scenes of one of the world's leading
artificial intelligence companies
to find out how the latest developments
in ai
are transforming our world
you have techniques that can tackle some
of the hardest problems that have eluded
the brightest minds for decades i think
it's woken up the scientific world to
the possibility of what ai could do you
might have heard that last year a deep
mind system called alpha fold made an
enormous breakthrough by using ai to
solve a 50 year old grand challenge
what you're describing here is a
potential step change in all of
healthcare and medicine really that is
the implication of truly understanding
biology
for the very first time we'll hear the
inside story of alpha fold from the
people who were in the room where it
happened we had a meeting saying okay
are we going to solve this problem or
should we be doing something else that
we can solve what give up was that
really on the table oh that was
absolutely as a scientist you're always
making a bet throughout this series we
explore how artificial intelligence is
being applied to real world problems
from weather forecasting it is important
to know if there's going to be a build
up of a catastrophic storm to the
creation of synthetic voices like this
one when i'm not trying to crack one of
the unsolved millennium prize problems
you can find me chillaxing with a cup of
tea it's really good i know it's good am
i that breathy
we'll be exploring in detail deepmind's
efforts to create artificial
intelligence that can talk to humans
me
what's my worst feature model you're
going to get upset if i tell you
and even programming robots to learn to
walk by themselves looks like it's a
drunk robot well sort of so it's trying
to walk backwards but it's sort of um
it's given up and it's really flawn
but we haven't shied away from the tough
questions that deepmind's researchers
are asking as they develop the
technology of the future
the last set of people who said they
were going to advance science and
humanity created colonialism we cannot
be complacent about what it means to put
forward a mission like that into the
world
because the big north star of all of
this work at deepmind is to create an
advanced problem-solving system known as
artificial general intelligence or agi
the outcome i've always dreamed of is
agi has helped us solve a lot of the big
challenges facing society today
join me hannah fry for the second season
of deepmind the podcast coming soon to
wherever you get your podcasts


What's Behind the ChatGPT History Change? How You Can Benefit + The 6 New Developments This Week

18 hours ago Sam Altman put out this
simple tweet that you can now disable
chat history and training in chatty PT
and that we will offer charity business
in the coming months but dig a little
deeper and behind this tweet is a data
controversy that could engulf open AI
jeopardize GPT 5 and shape the new
information economy I will show you how
you can benefit from this new feature
reveal how you can check if your
personal info was likely used in Gypsy 4
training and investigate whether chanti
BT could be banned in the EU Brazil
California and Beyond but first the
announcement openai say that you can now
turn off chat history in chat TPT but
that is only conversations that were
started after chat history is disabled
that won't be used to train and improve
their models meaning that by default
your existing conversations will still
be used to train their new models so how
does it work and what does this mean
what you need to do is click on the
three dots at the bottom left of a chat
GPT conversation then go to settings and
show and here's where it starts to get
interesting they have linked together
chat history and training it's both or
neither they could have given two
separate options one to store your chat
history so that you can look back over
it later and another to opt out of
training but instead is one button you
either give them your data and keep your
chats or you don't give them your data
and you don't keep your chats if you opt
not to give them your chat history they
still monitor the chats for what they
call abuse so bear that in mind what if
I want to keep my history on but disable
model training we are working on a new
offering called chat GPT business I'm
going to talk about that in a moment but
clearly they don't want to make it easy
to opt out of giving over your training
data now In fairness they do offer an
opt-out form but if you go to the form
it says cryptically please know that in
some cases this will limit the ability
of our models to better address your
specific use case so that's one big
downside to this new announcement well
what's one secret upside this export
data button buried all the way down here
if you click it you quite quickly get
this email which contains a link to
download a data export of all your
conversations after you download the
file and open it you now have an easy
way to search through all your previous
conversations literally all of them from
the time you first started using chat WT
to the present day that is a pretty
great feature I must admit but going
back to the announcement they said that
you need to upgrade to chatbt business
available in the coming months to ensure
that your data won't be used to train
our models by default but why these
announcements now why did Sam Altman
tweet this just yesterday well this
article also from yesterday in the MIT
technology review by Melissa iquila may
explain why it said that open AI has
until the end of this week to comply
with Europe's strict data protection
regime the gdpr but the that it will
likely be impossible for the company to
comply because of the way data for AI is
collected before you leave and say this
is just about Europe no it's much bigger
than that the European data collection
supervisor said that the definition of
Hell might be coming for open AI based
on the potentially illegal way it
collected data if openai cannot convince
the authorities its data use practices
are legal it could be banned not only in
specific countries like Italy or the
entire EU it could also face Hefty fines
and might even be forced to delete
models and the data used to train them
these Stakes could not be higher for
open AI the eu's gdpr is the world's
strictest data protection regime and it
has been copied widely around the world
Regulators everywhere from Brazil to
California will be paying close
attention to what happens next and the
outcome could fundamentally change the
way AI companies go about collecting
data aside from your attractivity
conversations how do these companies
collect your day later while two
articles published this week tell us
much more to take one example they
harvest pirated eBooks from the site
formerly known as book ZZ until that was
seized by the FBI last year despite that
contents of the site remain in the
common crawl database openai won't
reveal the data set used to train GT4
but we know the common crawl was used to
train gpt3 openai may have also used the
pile which was used recently by
stability AI for their new llm stable LM
the pile contains more pirated ebooks
but also things like every internal
email sent by Enron and if you think
that's strange wait until you hear about
the copyright takedown policy of the
group that maintains the pile I can't
even read it out for the video this
article from The Washington Post reveals
even more about the data that was likely
used to train gbc4 for starters we have
the exclusive content of patreon so
presumably all my patreon messages will
be used to train Gypsy 5. but further
down in the article we have this search
bar where you can look into whether your
own website was used in the common crawl
data set I even found my mum's WordPress
family blog so it's possible that GT5
will remember more about my childhood
than I do if you think that's kind of
strange wait until you hear the open AI
themselves might not even know what's in
their training set this comes from the
gpt4 technical report and in one of the
footnotes it says that portions of this
big bench Benchmark were inadvertently
mixed into the training set that word
inadvertently is rather startling for
the moment let's not worry about how
mixing in benchmarks might somewhat
obscure our ability to test gpt4 let's
just focus on that word inadvertently do
they really not know entirely what's in
their data set whether they do or not I
want you to get ready to count the
number of ways that open AI May soon
have to pay for the data it once got for
free first read it they trolled Reddit
for all posts that got three or more
upvotes and included them in the
training data now this New York Times
article says Reddit wants them to pay
for the privilege the founder and chief
executive of Reddit said that the Reddit
Corpus of data is really valuable but we
don't need to give all of that value to
some of the largest companies in the
world for free I agree but my question
is will the users be paid in fact that's
my question for all of the examples you
have seen in this video and are about to
see does the user actually get paid if
openai is set to make trillions of
dollars as Sam Altman has said will you
get paid for helping to train it
apparently Reddit is right now
negotiating fees with openai but will
its users get any of that money what
about the Wikipedia editors that spend
thousands of hours to make sure the
article is accurate and then GPT 4 or 5
just trolls all of that for free what
about stack Overflow the Q a site for
programmers apparently they are now
going to also charge AI Giants for
training data the CEO said that users
own the content that they post on stack
Overflow under the Creative Commons
license but that that license requires
anyone later using the data to mention
where it came from but of course GT4
doesn't mention where its programming
tricks come from is it me or is there
not some irony in the people being
generous enough to give out answers to
questions in programming actually
training a model that may end up one day
replacing them all the while giving them
no credit or compensation but now we
must turn to lawsuits because there are
plenty of people getting ready to take
this to court Microsoft GitHub and
openai were recently sued with the
companies accused of scraping license
code to build github's AI powered
co-pilot tool and in an interesting
response Microsoft and GitHub said that
the complaint has certain defects
including a lack of injury and the
companies argue that the plaintiffs rely
on a hypothetical events to make their
claim and say that they don't describe
how they were personally harmed by the
tool that could be the big Benchmark
where these lawsuits fail currently
because no one can prove harm from GT4
but how does that Bode for the future
when some people inevitably get laid off
because they're simply not needed
anymore because Gypsy 4 or g65 can do
their jobs then would these lawsuits
succeed when you can prove that you've
lost a job because of a specific tool
which was trained using in part your own
data then there is injury there that you
could prove but then if you block Gypsy
4 or Gypsy 5 there will be millions of
coders who can then say that they're
injured because their favorite tool has
now been lost I have no idea how that's
going to pan out in the courts of course
these are not the only lawsuits with the
CEO of Twitter weighing in accusing
openai of illegally using Twitter data
and what about Publishers journalists
and newspapers whose work might not be
read as much because people can get
their answers from gpc4 and don't forget
their websites were also called to train
the models well the CEO of News Corp
said that clearly they are using
proprietary content there should be
obviously some compensation for that so
it seems like there are lawsuits coming
in from every direction but Sam Altman
has said in the past we're willing to
pay a lot for very high quality data in
certain domains such as science all that
actually in which scientists and
mathematicians or will it just add to
the profits of the massive scientific
Publishers that's another Scandal for
another video but I am wondering if
openai will be tempted to use some
illicit sites instead such as skyhub a
shadow Library website that provides
free access to millions of research
papers without regard to copyright it
basically gets past the scientific
Publisher's paywall and apparently up to
50 of academics say that they use
websites like skyhub inevitably Gypsy 5
is going to break through some news
science benchmarks I just wish that the
scientists whose work went into training
it were compensated for helping it do so
just in case it seems like I'm picking
on open AI Google are just as secretive
and they were even Accused by their own
employees of training Bard with chat GPT
data they have strenuously denied this
but it didn't stop Sam Altman from
saying I'm not that annoyed at Google
for training on chat GPT output but the
spin is annoying he obviously doesn't
believe their denial and given all of
this discussion on copyright and
scraping data I found this headline
supremely ironic openai are trying to
trademark the name GPT meaning all of
those models that you've heard of Auto
GPT memory GPT hugging GPT they might be
stopped from using that name imagine a
world where they win all of their
battles in court and they can use
everyone's data but no one can use their
name GPT but maybe this entire data
issue won't be relevant for much longer
Sam Altman recently said that he
predicts open AI data spend will go down
as models get smarter I wonder if he
means that the models might be able to
train their own synthetic data sets and
therefore not require as much outside
data or of course he could be talking
about simplifying the reinforcement
learning with human feedback phase where
essentially the model gives itself
feedback reducing the need for human
evaluators wouldn't that be quite
something if GT4 can generate a data set
that is used to train Gypsy 5. as
someone who uses gpt4 a lot and whose
data was used to train GPT models I
fluctuate between being amazed annoyed
and deeply concerned about where all of
this is going let me know in the
comments what you think of it all and
have a wonderful day


A breakthrough unfolds - DeepMind: The Podcast (S2, Ep1)

do you remember where you were on the
30th of november 2020
probably not i'd imagine but that date
sticks in my mind because i got a text
from a friend and colleague geneticist
adam rutherford it read have you seen
the news the protein folding problem
exclamation mark
wow i thought they've only gone and done
it
[Music]
in season one of this podcast i told the
story of how the ai company deepmind was
using artificial intelligence to take on
a 50 year old grand challenge in biology
now only a year later deepmind had
unveiled an ai system called alpha fold
that virtually solved the problem of
predicting the 3d shapes of proteins
the scientific world was full of
excitement a headline in the journal
nature declared this will change
everything
the president of the uk's royal society
called it a stunning advance that had
come decades before many in the field
would ever have predicted
and forbes dubbed it the most important
achievement in ai ever
[Music]
i'm professor hannah fry
i'm a mathematician and i have been
following the developments in artificial
intelligence for as long as i can
remember
in season two of deepmind the podcast
i've gone back inside the lab to speak
to countless researchers scientists and
engineers about their work
and i must say
i've seen some pretty extraordinary
things
from robots that have taught themselves
to walk around
looks like it's a drunk robot
to ais that can predict the weather it
is important to know if there's going to
be a buildup of a catastrophic storm and
generate human sounding voices
but it seemed only right to kick off the
first episode with what is probably
deepmind's biggest breakthrough so far
coming up we are gonna hear the full
story of alpha fault from the people who
were in the room where it happened
i remember sitting on the plane on the
way home writing out well what are all
the ideas that might actually work and
just sitting there going this is amazing
and this is terrifying
we'll find out how alpha fold could
rapidly accelerate the drug discovery
process we got hundreds and hundreds of
emails from excited biologists wanting
to try out their system
and paved the way for new discoveries
i think it's woken up the scientific
world to the possibility of what ai
could do
this is episode one a breakthrough
unfolds
one of those scientists who is paying
particular attention to the alpha fold
news is john mcgeehan professor of
structural biology at portsmouth
university
the first time i saw the story about
alpha fold i thought that can't be true
because it's about two decades too early
so i immediately started checking the
details of this to make sure i was
understanding it correctly i literally
emailed deepmind and said we could
really use this technology
to see how much of a game changer john
and the scientific world thinks the
alpha fold breakthrough might be
i want you to come with me on a trip to
the bottom of the ocean
every year somewhere between 8 to 12
million metric tons of plastic waste
gets dumped into the oceans and john has
made it his scientific mission to do
something about it
[Music]
some of that plastic is used once
sometimes just for minutes in the case
of maybe a fizzy drinks bottle
and if that enters the ocean some
plastics last hundreds of years in that
sort of environment what tends to happen
is through sunlight and waves
they break down but only into smaller
and smaller pieces and they end up being
microplastics
wildlife consumed this plastic it ends
up in our food chain and i think we're
only just starting to understand the
potential impact
there might just be a twinkling ray of
hope on the horizon
in march 2016 a japanese research group
announced in the journal science that
they discovered a species of bacteria
that could break the molecular bonds of
one of the world's most used plastics
polyethylene terephthalate
also known as p-e-t or polyester
p-e-t is found in products from fizzy
drink bottles to textiles
john's collaborator in the u.s greg
beckham instantly spotted the potential
of this discovery and gave him a call
and he said this japanese group has
literally found a bacterium living in a
plastic waste dump that is digesting
plastic it's actually using the plastic
as a food source
we had to figure out how these enzymes
worked and if we could make them faster
because the potential here is if you can
break down plastic into its building
blocks then suddenly you can recycle it
infinitely
we'll find out exactly how john mcginn
and his team are using alpha fold to
help fight plastic pollution a bit later
on
but for now because alpha fold predicts
the 3d structures of proteins it's
helpful to find out a bit more about how
proteins work in the human body we'll
make it lots of fun i promise because
proteins are truly fascinating things
proteins are these incredibly small
machines that are extraordinarily
unusual for the way that we think of
machines but they do the kind of
mechanical and chemical things that the
cell needs to
let me introduce you to john jumper who
leads the alpha fold project
the human cell has about 20 000 of these
nano machines and people make entire
careers explaining the function of one
of these proteins
proteins are the workforce behind
practically everything that happens
within the human body
you'll find them in our immune system
antibodies are a type of protein we
release that can fight bacteria and
viruses
the protein insulin is what allows our
bodies to absorb glucose
proteins even work to shepherd other
molecules around the body as in the case
of one particularly famous protein
hemoglobin is a protein whose job it is
to carry oxygen within your red blood
cells
your lungs are extracting that from the
air but then it needs to be carried
throughout the body and so it is
attached to this hemoglobin protein
which later releases it where it's
needed
and yet these powerful naturally
occurring machines are made from only a
small list of building blocks all the
machines that we have in kind of the
normal natural world
are
really really complex and the proteins
they have this lovely kind of modular
system where
there's 20 types of amino acids and a
protein in its simplest form is just a
line of these amino acids
these 20 amino acids tiny molecules made
up of a handful of atoms are the
building blocks of all human proteins
to make a protein these amino acids are
strung together in a sort of miniature
cellular factory called the ribosome
how about an analogy like one of those
bracelet kits my daughter has one right
and she wants to spell her name sarah so
she grabs the s and she puts it on the
line she grabs the a she puts it on the
line she'll surely grab a heart and put
it on the line
in the same sort of way you have beads
in a line or these amino acids that are
attached one at a time and when it's
done the protein leaves the ribosome
within milliseconds the forces of
physics gently tease the line of amino
acids into intricate 3d shapes which
give function to the protein and allows
them to work as miniature machines
you can think of it like origami a blank
piece of paper might hold all manner of
potential but it only becomes meaningful
once it's folded into shape
the 1d shape the line isn't so good at
telling us how these machines work how
we can build new ones
they really need to assemble into this
intricate 3d shape that we can start to
reason about
seeing a structure is really the
beginning
of explaining
why the protein is that way how it
misfunctions how we might be able to
develop drugs but it's extraordinarily
difficult to get this kind of
information
[Music]
most drugs work by binding to a
particular protein in a specific way
beta blockers stop the protein receptors
from noticing the adrenaline floating
past antibiotics work by interfering
with a bacteria's protein
this is partly why understanding the
shape of a folded protein is so
important
so what do folded proteins look like
well
these things are
tiny it's not like you could just look
at them down a microscope to see what
shape they are
at that scale they might as well be
invisible
to work out the structure you normally
have to use something called x-ray
crystallography
first of all you have to express your
protein so you need to coax a bacterium
into producing enough of it for your
experiments
here's catherine tunis awanagan a senior
research scientist then you have to be
able to get it out of the cell in
sufficient quantities then you need to
get it to form crystals which is often
the most difficult part and when you say
the most difficult part yeah are we
talking weeks it's very variable so if
you happen to be working with a protein
that's really well studied it can be a
very quick process if it's something
that's never been crystallized before it
can take years so i've had emails from
people who said you know we've worked on
this for 10 years we can't even make
enough of the stuff to start our
experiments that's an extreme case but
it can be as bad as that
[Music]
for the last 50 years numerous
scientists have dedicated their entire
working lives to painstakingly
uncovering the intricate structure of
proteins one at a time
slowly and surely building a collective
body of knowledge
but finding a shortcut getting from the
string of amino acids straight to the
folded protein has long been considered
one of the grandest challenges in
biology
[Music]
at some point in those intervening
decades in a pub in cambridge
denis asabus the man who would go on to
co-found deep mind first heard about the
protein folding problem
as an undergrad one of my friends was
obsessed with this problem and we used
to go to the pub and almost any
situation he would crowbar in the idea
of like if we solve protein folding that
would unlock all the biology
back in those days no one had quite
managed to get computers to reliably
predict how a general protein might fold
you could do it for tiny proteins where
the physics equations were tractable but
any larger and there were too many
variables
but that didn't put demis off from
wanting to have a go
there was this game called foldit people
were playing a game to fold proteins
and making some progress they were not
trained biologists they were gamers they
were able to find two or three protein
structures that were actually published
in nature they were actually quite big
breakthroughs so then finally when we
did alphago i thought well if we're able
to mimic the intruder process of a go
player then why couldn't we do what
these folded players were doing for
folding proteins
in 2016 before the team had even caught
the flight home after alphago's victory
demis was already eyeing up the next big
challenge for ai to work on protein
folding i'm sure we can do that his
conversation with alphago's lead
researcher david silva was captured by a
film crew that followed them down a
street in seoul where the match had
taken place it's just amazing seeing how
quickly a problem that is seen as being
impossible and changed to being
continuously done we consult protein
folding
there were good reasons for thinking
that an ai approach might work
it's a very interesting problem in the
sense that it has a very large search
space
push me kohli leads deepmind's science
team
the space of possible structures is just
infinite and yet it has definite
patterns hiding within
one of the things that makes this
problem great is the process is
extremely repeatable and specific the
sequence makes the structure but it's
also so complex that we don't know how
to write it down specifically
but we know how to write down a machine
that if you show it enough data we'll
learn these
something else that made protein folding
suitable for ai was the decades of work
by experimental biologists to find the
structures of proteins one by one
it meant an ai could peak at the back of
the textbook to learn how the patterns
worked
and much like in the game of go protein
faulting came with a ready-made way to
measure how well the algorithm was
performing
for nearly three decades there has been
a regular global competition amongst
biologists who use computers to fold
proteins
it's called the critical assessment of
protein structure prediction or casp
the way it works is you take a set of
100 proteins where you're challenged to
say i've got the sequence what does the
structure look like professor john molt
co-founded the casp competition in 1994
and to all work on those hundred in
parallel as a community and now you can
compare results
the accuracy of a protein fold
prediction in casp is measured using the
global distance test or gdt
you take a visual sketch of the true
folded protein and then superimpose the
calculated competition entries on top of
it
the more amino acids that end up roughly
where they should be the higher the
score
it's on a scale of 0 to 100 where 100
would be exact agreement with the
experimental structure so if you get to
something like 90 then you're arguing
with the experimentalists whether your
model is better than his structure
deepmind set up a new team with the task
of building an ai model that could
predict how a protein will fold from its
amino acid sequence it was led by john
jumper
how far away did you think success was
at the very beginning i wouldn't say
that we were sure of success there was
always this feeling that it's complex
but there's no rule that it can't be
understood but i would say it was
absolutely unclear that we were going to
be able to build it
i have always been one of the skeptics
on the team i sort of expected maybe 15
years from now we would have a system
like this and also i think in biology
there is quite a bit of skepticism often
about computational methods we think of
experiment as being the last word and it
can be an uphill struggle to convince a
biologist that a computational method is
really going to help them
the first version of alpha fold is the
one we talked about in season one of
this podcast
the team repurposed an ai technique
normally used for image analysis so that
it could work out which beads or amino
acids should be positioned close to each
other in the final protein
then the model crunched through the
different structures that could possibly
fit
it seemed to work pretty well although
it was slow and it took a lot of
computational power
in april 2018 deepmind entered alphafold
into the cast competition for the first
time
before the results were released one
member of team alpha fold tried to
predict how well they had done
he had run the numbers and by his
results we were 20th it was a one of the
more depressing moments of my career
finding out that after all this effort
we weren't even close to the top we had
a meeting where we very seriously saying
okay are we going to solve this problem
or should we be doing something else
that we can solve well give up was that
really on the table that was absolutely
i would say as a scientist my job is to
find a place to be useful you're always
saying do i want to spend my time in
this direction or do i want to go in a
different direction
but let's not get ahead of ourselves
here there was just a small issue with
that prediction that alpha fold had come
in 20th place at casp
we found out that we just computed it
wrong we just done the math wrong at
least you did the math right when it was
important on the actual algorithm we get
the right answer eventually
in actual fact alpha fold came top in
casp that year
it was a strong result but it was still
very far from giving the experimental
methods a run for their money
as well as being painfully slow the
first version of alpha fold achieved
58.9 as its average gdt score
remember you need to be getting over 90
to have a structure that would be
considered true to the real folded
protein
after the casp win deepmind decided to
redouble their efforts
we tried to improve the caste system and
it was giving us some minimal
performance improvements but we were
nowhere close to making the jump that we
wanted
and then there was a breakthrough
alpha fold 2 started by throwing away
everything that had come before
the data fed in was almost entirely
unchanged but the repurposed image
recognition system was gone replaced by
an ai that had been redesigned from the
ground up
just to understand protein folding
[Music]
but how much better was this new and
improved version of alpha fold really
in 2020 the alpha fault team prepared to
enter the next casp competition
keeping a close eye on its score
gdt has gotten better i think we're
making incredibly fast progress and i
think we'll continue to i think it's an
important within our meetings we went
through this tradition where we would
play a song from the year that
corresponded to our gdt and so if we
were at gdt80 we would play a song from
1980 or something like that and
everyone's just like when we get to the
spice girls we know it's okay absolutely
and i don't think we ever did get to the
spice girls either because that would be
97 or so what did you get to
1990 mc hammer something like that
we would have these running sweepstakes
predicting what would be the accuracy of
alpha fold who won the sweepstake i am
not going to tell you who won the
sweepstakes or what was my prediction
but
did you go live at some points i did
follow whether the type of accuracy we
are seeing today was possible
was a completely open question we would
have these meetings with demis and
everyone would think well demis is going
to push our target even higher and they
were not really sure if that was
possible
[Music]
i just felt that all the ingredients
meant there should be a solution to this
and i also believe the original
biologists who said 50 years ago that
this should be possible
in may 2020 the new casp competition
kicked off online nearly a hundred
groups around the world submitted their
predictions for 90 protein targets
once team alpha fold had submitted their
predictions they faced an agonizing wait
for the results to be verified but after
a month or so i got this email from john
malt the founder of casp and it had a
title i think it was zoom request
you're not going to turn down that zoom
are you no and so we dialed in to a
pre-existing team social and john just
said i'm going to read you an email it
is from john malt as i expect you know
your group has performed amazingly well
in casp 14 both relative to other groups
and an absolute model accuracy it didn't
tell us our exact score but it did tell
us the method seemed to be an
outstanding one how did the rest of the
social go
did you get to crack out the champagne
fantastically well yes i was drinking
some bourbon and we were swapping lots
of reminiscences and stories
oh and the final scoring casp
an average of
92.4 across all targets
here's kasp's founder john molt to be
able to fold up single proteins to this
atomic accuracy is a really amazing and
satisfying thing as we originally framed
the problem this is a solution but in
many ways it's just a start after cast
14 we got hundreds and hundreds of
emails from excited biologists wanting
to try out their system and use it for
particular problems
we did pick out a few cases where we
thought alpha fold was really going to
make an impact
one of those early users of alpha fault
was john mcgeehan whose work on plastic
pollution we heard about at the
beginning of this episode
to understand how alpha fold is helping
john and his colleagues
it's important to appreciate that those
plastic eating enzymes that could help
clean up our oceans
are in fact
proteins
they're non-living proteins that break
chemical bonds and you know if you just
had lunch or dinner they'll be working
in your stomach at the moment digesting
all those different carbohydrates into
sugars the enzymes we're interested in
work in the same way but they actually
start breaking down plastics
[Music]
when that group of japanese scientists
discovered an enzyme that could digest
pet plastic
john and his team set about trying to
understand how it worked on a molecular
level
currently we dig up oil and gas and we
distill from those the chemical building
blocks for plastics and we stick those
together in long chains what the enzyme
does is do the opposite of that it comes
along like a big pair of molecular
scissors and cuts those bonds
and then gives you back those building
blocks
we know there is at least one enzyme
that can break down one specific type of
plastic
by looking through the database of
sequenced proteins the scientists found
a bunch of others that look like they
might work in a similar way for other
plastics
but to make any progress they really
needed to see how they are folded and
structured in three dimensions and
understand how they might work to eat
chains of plastic polymers
solving all these structures takes an
awful long time we've got 100 enzymes
sitting there all needing structures and
each structure takes maybe six months
sometimes even a year depending how
tricky it is
100 enzymes
any of which could have gigantic
potential for our planet sat in a queue
waiting for months or years to have
their structure worked out
without knowing what they look like
there is little hope of designing
synthetic versions that could work
better and faster than the one
discovered in japan
and this is where alpha fold comes in
john mcgeehan sent deepmind seven
enzymes of interest
two of which his team had already solved
the structure for
rather spectacularly in a couple of days
they came back and the first thing i did
was to compare the structures that we
already knew to what they'd predicted
and they're almost perfect actually not
only did it get the three-dimensional
fold right it got all the positions of
where all those key atoms are in the
structure correct as well alpha fold is
now in the process of resolving 100
enzymes that we've chosen but it needn't
stop there we could look at thousands of
enzymes and that's incredibly exciting
because this will massively accelerate
the development of enzymes for different
plastic recycling
beyond tackling plastic pollution the
alpha fold team were well aware that
there were other scientists like those
working in drug discovery who would find
this new tool useful
another organization that was an early
beneficiary of alpha fold is the drugs
for neglected diseases initiative an
international non-profit dedicated to
developing new treatments for neglected
tropical diseases
these are a group of preventable
infectious diseases that really affect
1.7 billion people dr monique wasuna is
the dndi's regional director based in
nairobi over the years she has treated
many people suffering from parasitic
diseases but one patient stands out in
her mind from her time as a junior
doctor in kenya this particular middle
aged man had been unwell for many months
and he was just really getting weaker
and weaker so he had about our team that
was extensioned many many miles away and
he took a walk day and night for five
good days
the patient had been infected with leash
moniesis a parasitic disease found
mainly in eastern africa parts of the
americas and the middle east
symptoms start within a couple of weeks
of being bitten by a female sand fly
you start having a fever your guns might
start swelling your liver your spleen
and in the end if you're not treated you
die
the most deadly form visceral
leachmaniasis poses a risk to 600
million people worldwide and causes an
estimated 20 to 40 000 deaths a year
these patients survived but how many
patients like him will die and nobody
knows that they have died of flesh
manages
it is possible to treat leash menialisis
but you need a long painful course of
drugs over 17 days and that can come
with severe side effects
they are toxic
they damage your liver your kidneys
we have to keep the faith because we
haven't gotten the ultimate treatment
typically neglected tropical diseases
don't get on the radar for
pharmaceutical companies because there's
no market incentive for them
charles mobrey leads dndi's drug
discovery effort from geneva the needs
of these patients are huge they deserve
better treatments the way people living
in wealthy countries benefit from the
advances in modern medicine
charles and his colleagues are looking
for monique's ultimate treatment a drug
that can kill leash mania parasites can
be easily administered in the field and
does not cause serious side effects
it's an expensive and time-consuming
process so dndi have had to rely on a
more pragmatic way of finding new drug
treatments for neglected diseases
can we find drug molecules that will
kill a parasite in a test tube that
initial starting point requires that we
screen hundreds of thousands of
molecules to find just one that has some
of the right properties
in short it's a bit of a stab in the
dark
there is also no guarantee that a
molecule that can kill a parasite in a
test tube will work inside a human
so instead in recent decades scientists
have opted for a different approach
rather than testing for drug molecules
that kill a bug we can think about
carefully designing molecules that
interfere very specifically with a
process in that bug
but don't interfere with any related
processes in the human
imagine you'd worked out that there was
one particular protein that was vital to
the parasite
if you knew the three-dimensional
structure of that protein you could
explore how different drug molecules
might disable it
the parasites protein is like an
intricately designed three-dimensional
lock covered in cavities and holes
the drug molecules need to act like a
key filling in the spaces and blocking
the protein from functioning
understanding how best to design that
key
is a lot easier if you can just see
everything on a computer screen rather
than have to infer it from laborious
experiments
you can see where the cavities are and
you could bring a molecule up and you
could see how does it fit into that
cavity
and this is where alpha fold comes in
particularly useful
rather than waiting months or years for
an experimental method to come up with a
protein structure that may or may not be
important in a disease
alpha fold can offer a shortcut
[Music]
here's catherine tanya sawanakan again
so it basically changes the situation
from not using structural information
waiting several years for the
experiments to be finished and now we
have this middle way of well i can get
some actionable structural information
within 10-15 minutes oh wow i mean
that's massive yeah it's a huge saving
in the case of leash moniesis scientists
had already identified a drug molecule
that is effective at killing the
parasite in a test tube
unfortunately they reached a dead end
when it came to adapting this molecule
into a usable drug treatment
so they worked backwards first they
tried to work out what part of the
parasite this drug molecule appeared to
be targeting and in the process they
were able to pinpoint a novel protein
that seems critical to stopping the
leash mania parasite in its tracks
so if that protein is shut down the
parasite can't continue it will die it's
not like other proteins that we've
looked at as targets for leash monasis
so that's exciting
the other exciting thing about this
particular protein is that it looks like
it's not unique to leash mania parasites
similar proteins are found in the
parasites that cause other infectious
diseases
you can see how powerful this quickly
becomes
how the structure of proteins can
potentially open the door to drugs that
are effective for multiple diseases
we could be cutting years off the
timeline this is the power of knowing
about the structure and if using alpha
fold predictions we can get to that
starting point quicker it will help us
pick the right projects and move them
more quickly
the hope is that if alpha fold can do
this for charles it will do the same for
scientists working on all kinds of
important problems around the world
and so in july 2021 deepmind took the
step of publicly releasing the
structures of all 20 000 proteins found
in humans
as well as several other organisms
commonly used in labs around the world
everything from worms and fruit flies to
rats and tuberculosis
in time the team aims to fold and
release all 100 million of the proteins
ever discovered for free
for anyone to use
to accelerate the work of scientists
but before this major release a lot of
careful thinking went into reviewing the
risks of doing so
what if for example alpha folds insights
into protein structure were used by a
bad actor to design a rogue protein say
we carried out a full review of this
because of something we we're concerned
about
sasha brown from the ethics team was
involved in the process of sharing alpha
fold with the world we were trying to
understand will
the release of alpha fold help the
creation of new bio weapons or increase
the potency of current known ones and
the view we got from experts was that
the likelihood of protein folds being
used in this manner was small because
non-specialists face several barriers to
enhancing and synthesizing harmful
proteins and they're more likely to use
readily available materials
in other words if someone is intent on
committing an act of bioterrorism there
are currently much easier ways to do it
than getting to grips with the
biochemistry of protein folding
it wouldn't remove any bottlenecks to
using bioweapons at the level that alpha
fold is currently
while it's hoped that the alpha fold
database release will now accelerate the
work of scientists around the world
there is another important point worthy
of mention here
alpha fold doesn't provide a perfect
dictionary that can translate between
protein sequences and folded shape
it's only offering a prediction
the results of cass put it very well
here's john jumper again of the
predictions we made about two-thirds
were at a quality bar they considered
competitive with experiment so that
means one-third
they're pretty sure that we have
deficiencies although typically small
ones and so it's very important to us
how do we make sure we're doing more
good than harm
there are many reasons why alpha folds
prediction might be uncertain
some proteins have never been seen
before others might have strange
disordered shapes and so on
so the alpha fault team have built in a
way for the model to put a number on its
confidence in its prediction of a given
protein shape
alpha fold predicts the score it would
have got for that protein had it been
part of the original caste competition
here's pushmi kohli to explain
it's very good at that uncertainty
estimation
it's not shy in saying i don't know
which is i think a good thing install
the humility package
essentially when alpha fold solved one
problem it opened up a whole new world
of questions and possibilities for
science
it's very important to note that it does
not solve protein understanding writ
large in terms of making sense of how do
proteins operate so the community has
appreciated the breakthrough that has
been made
but also at the same time there is a lot
more work that needs to be done
in many ways alpha fold is just the
beginning of demonstrating how ai can be
used to advance scientific discovery
and in fact it isn't just proteins that
deepmind is working on researchers are
also successfully applying machine
learning to other areas of science
including genomics chemistry nuclear
fusion mathematics and ecology
in episode 6 of this podcast we'll be
exploring some of those projects in more
detail trust me you won't want to miss
it
but deep mind say they have a bigger
mission of solving intelligence
so i want to end this episode by asking
how alpha fold fits in with that
does alpha fold have any of the
hallmarks of intelligence
i mean what is intelligence oh that's a
whole kind of worms yeah exactly and as
a former worm researcher i know worms
right so this is very much a can of
worms here we're talking about
intelligence in terms of being able to
accomplish a specific task that humans
can't do so it's something that is a
superhuman ability
it is very good at learning from the
existing structures that we have but i
wouldn't call that artificial
intelligence necessarily because that
term generally refers to something that
is more of a generalist whereas i see
elf fold as a specialist
[Music]
i think it's really
important to remember that these are
really powerful techniques that we've
developed that are still far short of a
real artificial intelligence
that you can talk about thinking and
making decisions and everything else
[Music]
if you think our fold isn't intelligent
how does it fit in with that broader
goal of deep mind towards solving
intelligence
this is an example of how we work on
these really hard problems that stretch
us and how do we develop new ideas and
techniques and then think about which of
those will pull
back into the core mission which of
those take us one step closer to general
intelligence
over the next four episodes of this
podcast we'll be taking an in-depth look
at that core mission building artificial
general intelligence or agi
starting with the big push to give
machines the ability to communicate with
humans
me
what's my worst feature model you're
going to get upset if i tell you
[Music]
deepmind the podcast was presented by me
hannah fry
it was written by me and dan hardoon who
is the series producer
if you've enjoyed this episode please
subscribe to the podcast and leave us a
rating or review if you can
bye for now