8 Ways ChatGPT 4 [Is] Better Than ChatGPT

I would not blame you if you thought
that all talk about gpt4 or chat gpt4 is
just that talk but we actually can have
a surprising amount of confidence in the
ways in which gpt4 will improve on chat
GPT by examining publicly accessible
benchmarks comparable large language
models like palm and the latest research
papers which have spent dozens of hours
reading we can discern at least eight
clear ways in which gpt4 integrated into
Bing or otherwise will beat chatgpt I'm
going to show you how unreleased models
already be current chat gbt and all of
this will give us a clearer insight into
what even GPT 5 and future rival models
from Google might well soon be able to
achieve there are numerous benchmarks
that Palm Google's large language model
and by extension gpt4 will be attack gbt
on but the largest and most impressive
is the big bench set of tasks more than
150 or now 200 language modeling tasks
and I've studied almost all of them and
you can see the approximate current
state of affairs summarized in this
graph where the latest models are now
beating the average human and showing
dramatic Improvement on previous models
chat TBT would be somewhere around this
point lower than what is actually
privately available but better than
previous models down here but this just
skims the surface I want to show you in
detail the eight ways that you can
expect chat GPT 4 or gpt4 to beat the
current chat gbt and know that's not
just because it's going to have more
parameters off to the right of this
graph 10 to the 12 a trillion parameters
it's also because compute efficiency
will improve Chain of Thought prompting
will be integrated and the number of
tokens it's trained on might go up by an
order of magnitude lots of reasons why G
ppt4 will be better let's start with
logic and logical inference this example
comes from Google's Palm research paper
the question or input was this Shelley
is from Virginia is visiting that City
with that famous Market where they throw
the fish so vague going home next
Tuesday question is it likely that
Shelley will be near the Pacific Ocean
this weekend and you can see how the
improved model is able to deduce that
yes indeed because she's probably going
to Seattle that she will be on the
Pacific Ocean whereas if you ask current
chat gbt this question what you get is
based on the information given it's not
possible to determine the statement only
mentions that Shelley is from Virginia
and visiting a city with a famous Market
it really can't handle it it can't do
that level of logical inference here is
another great example this test of
critical reasoning and logic was
designed again for the big bench
Benchmark and it was tested on different
language models and most of the them
fail including chat gbt I gave it this
question and it picked the wrong answer
you can pause and examine the question
yourself but C is not the correct answer
it gets it wrong however let's take a
look at the graph beneath at other
language models ones to come gpt4 maybe
and look what happens as the models
increase in effective parameter count
and other things like token size look at
the performance we start to beat not
only average raters but all previous
models and approximate the performance
of the best human rater so the top line
is the best human rater the blue line is
the average human rater so these
unreleased models harm three shot means
it was given three examples of what was
expected before being tested these best
models and you can imagine gpt4 will be
around the same level now Crush what
chat TBT was capable of you can imagine
what this mean means in terms of gpt4
giving more rigorous arguments or
conversely you can give vague inputs
like this thing talking about a famous
Market where they throw the fish and
gpt4 might well be able to understand
exactly what you mean and to be honest
if you thought that's interesting we are
just getting started next jokes on the
left you can see a computer sciencey
type of joke that he was able to explain
but I tested PT on a variety of jokes
and some of them it could explain others
it couldn't let me show you what I mean
here was the joke that I asked it to
explain one of The Oddities of Wall
Street is that it is the dealer and not
the customer who is called broker the
play on words being that the customer
might well end up being broke it didn't
really understand that wordplay and got
it wrong I don't think he got that broke
was different from a broker couldn't
separate off that word inside broker now
it did get this second joke right and
explain it well this shows us what gpt4
might be capable of as the Google paper
showed as the model improves it does get
better explaining jokes and therefore
presumably at telling them those comedy
sketches that people are generating now
with chat gbt they are about to get many
times better think word plays puns
innuendos all sorts the next example
comes from physics look at how the
latest models and by implication GPT 4
beat previous models at answering basic
questions in physics and teaching them
you can see how Palm far exceeds GPT and
also as you can see beats the average
human and is getting closer to the best
human what kind of questions are we
talking about I looked into the research
methodology for this big bench Benchmark
and I took some of the questions and
tested them on chat GPT it couldn't get
them I asked them and both questions it
got wrong but that's what might change
with gpt4 we're talking high school and
Beyond physics you can imagine chemistry
biology starting to get questions more
and more right and therefore be able to
explain them better and better now I
won't pause for a physics lesson you can
see how chat TPT fails by pausing the
video if you want but gpd4 probably
won't fail next is math here are a bunch
of examples that Google produced in the
research paper about the improvements
that its current model can do unrelease
the public but that something like GPT
would really struggle at and you don't
need me to show you chatting BT failing
at these because it's quite notoriously
bad at math these are quite nuanced
questions though how many key strokes
are needed to type the numbers from 1 to
500 yes chat TPT fails what about a word
problem Roger has five tennis balls he
buys two more cans of tennis balls each
can has three tennis balls how many
tennis balls does he now have this uses
something called Chain of Thought
prompting which I'll talk about a bit
later but either way it gets the answer
right whereas previous models get it
wrong you don't need me to help you
imagine the kind of implications that a
GPT better at math would be for the
world just think about Finance
Assistance or math tutors available in
your pocket before we move on to
Improvement number five please do leave
a like and a comment if you're learning
anything from this video I really did
put dozens of hours into reading
academic papers to give you really clear
examples of each Improvement for the
fifth Improvement I'm going to merge two
benchmarks together the first one is
called implicatures quite hard to
pronounce and it's one of those
situations where people reply yes but
don't use the word yes they say
something like is the pope a Catholic or
is rain wet or as far as I know and
previously large language models
struggle to interpret that as a yes to
give you an example look down here at my
final interaction are the Androids
deployed speaker one says speaker two
what do you think which most humans
would interpret as yes of course why are
you asking and yet when I ask has
speaker 2 likely confirmed or likely
denied the deployment of Androids or can
we not tell chat TPT says it's not clear
where to most humans it would be clear
however as you might have guessed look
at the improvements being made behind
the scenes as the parameter count goes
up the number of tokens go up look at
the graph suddenly Palm can actually
understand better than the average human
whether a yes or a no is being said
approaching the performance of the best
human leaving the original chat GPT
behind and showcasing how gpt4 might be
a massive Improvement in this regard the
related Benchmark by the way was about
do we have sufficient information in
other words how about we just say don't
know if we don't know the answer or if
you can't answer the question something
like how much water is in a cup with
height of 10 centimeters you can't
answer that question it depends on many
factors the thickness the radius and
some models would hallucinate or let's
say bullcrap their way to the answer
whereas now you're more likely to get a
don't know from improved models like
palm and the soon to be released gpt4
the sixth Improvement will be in reading
comprehension that is understanding
digesting analyzing in comprehending
essentially large or long passages of
text you can just imagine the
implications of this summarizing earning
calls or transcribing and summarizing
YouTube videos automatically condensing
information down to a paragraph which
might have been pages and Pages chapters
and chapters I think this graph is
particularly stunning how the latest
models are now getting close to the
performance of the best humans at
understanding texts how long will it be
until they can read Dostoevsky and
summarize it in a thought out paragraph
chatty BT definitely can't do this give
it a reading comprehension question and
it gets it wrong almost every time in
fact quite hilariously when I gave it
one question it only picked the wrong
answer and neglected both correct
answers but gpt4 with 99.9 certainty
will be a lot better at doing that then
next big Improvement will be in coding
I'm no expert but reading through the
paper you can see significant
improvements in capability if we scroll
down you can see that the improved model
could compile at a rate of 82 percent
versus the previous state-of-the-art
71.7 and GPT 4 might be even a
Improvement on this of course many of
you may have read media reports of
openai drafting in hundreds of
programmers to help it fine-tune its
code definitely there should be a real
step change in its ability to code
successfully and as it says down here
opening up opportunities to fix more
complex errors that arise during
software development the eighth
inevitable Improvement that gpt4 will
bring will just be General efficiency
and speed Google Muse has demonstrated
with text image the same process can be
done 10 times faster with a bit more
efficiency and compute power is
increasing all the time these models
were trained on a100 gpus but h100 gpus
are already available from Nvidia and
model efficiency is improving all the
time so just imagine this imagine what
previously took 10 seconds to generate
which is still incredibly fast now
taking one second instant responses from
gpt4 now it might not be one second it
might be three or four but it's going to
be faster and one iteration down the
road GPT 5 might be instantaneous
I have detailed quite a few areas where
gpt4 is very likely to improve on track
TBT but there are quite a few areas in
which it will very likely still struggle
one is advanced math mathematical
induction even the latest models really
struggle
this area called navigation is an
example that will say something like if
you move forward three steps turn right
go three steps turn right again 90
degrees go three steps that kind of
thing do you arrive back at the start
these models really struggle with that
but the final area I find quite amusing
and it comes from wino grad schema as
detailed in this other academic paper
called super glue which is a kind of a
rival Benchmark to the big bench and a
whiner grad schema is a situation in
which we have an ambiguous pronoun like
he it or they
and the model has to predict not only
who or what the pronoun is referring to
also why would it be that thing and it
really struggles these models and I'll
show you the graph in a second in fact
here it is even the latest models
struggle I think because it involves
some common sense about the world and
the universe that it just doesn't have
let me show you chat GPT failing at this
task feel free to try this one yourself
Tom threw his school bag down to Rey
after he reached the bottom of the
stairs question who reached the bottom
of the stairs how it makes sense that it
would be Rey because logically you can
think of real life you're throwing it
down the stairs like here take it before
you go out whereas the model says Tom
reached the bottom of the stairs wait
why would he be throwing his school bag
down if he's at the bottom stairs so
that's an example of an area in which
chatter between fails and gpt4 will also
likely fail and the why bit in the title
is the fact that it will not only fail
but not really be able to explain even
when it succeeds why the pronoun is
referring to the noun that it does I
find that really interesting like the
merging of language and Common Sense
these large language models
fundamentally don't have a model of the
universe as I talked about in one of my
other videos it's the main critique that
Jan lacun has actually about large
language models this graph by the way
gives a beautiful summary of what I
think is the approximate current state
of the R so gpt4 comparable to Palm and
how it does versus the average human
on the left all the different tasks part
of the 150 big bench tasks that it can
do better than humans and on the right
those that it does worse than humans so
you can see a roughly even split but
remember this is versus the average
human not versus the best human the link
to all of these papers will be in the
description if you want to check out the
full list of tasks that it does better
versus what it does worse at I've
actually scrolled through almost every
single one of them and analyze it it's
really interesting to do actually all
these different challenges that
independent humans have come up with in
order to test just how far language
models are progressing a very
interesting Endeavor that they're
putting together all the way from
Proverbs to python as I draw to the end
here I want to give you my two main
conclusions I think gpt4 for commercial
reasons and others will be yes a huge
step up from chatty BT but won't be game
breaking what I mean by that is we're
talking better than the average human at
quite a few tasks maybe half of those
measured but still lagging behind the
best human in almost every task roughly
High School levels of achievement so yes
as I quoted Sam Altman saying in my
previous video hype versus reality chat
tpg4 yes it's going to be disappointing
to those people expecting AGI however
what's coming down the road in the short
to medium term it's hard to put an exact
date but are we talking two three four
five years somewhere in that range the
models that are coming are gonna be
pretty impressive slash overwhelming
that's not just by the way because the
number of parameters are improving as
this abstract from deepmind put it it's
not just about improving the number of
parameters it's also about using more
data four times more data
Palm by the way used
780 billion tokens
but there are up to 17 trillion tokens
available on the internet so roughly an
order of magnitude more tokens more data
that these models can train on and the
number of parameters if they're
increased in alignment would also go up
by an order of magnitude not just that
but the compute available to Big Tech
like Google and Microsoft is increasing
all the time I hinted at the h100 gpus
but even just scaling up in size and
compute efficiency should yield
incredible improvements there is also a
whole academic paper on how Chain of
Thought prompting basically getting the
models to break down and show their
working out really improves results and
I've seen that myself with chat gbt if
you ask it to explain its steps it often
gets to a right answer whereas
previously it got to a wrong answer so
it's not just about compute and
parameters and data it's also
refinements to the models themselves of
course other improvements around the
corner are a diversity of data inputs
could be video could be photos could be
audio from the microphone and these can
be assimilated into the model so you can
ask questions like will These Boots be
usable to hike Mount Fuji and this is an
example from Google that might not
necessarily be in gpt4 but it's coming
and Google might be the one pioneering
these diversity of data inputs as the
eerily powerful conclusion from this
Google post put it the vision that they
have is to enable a single AI system to
generalize across thousands or millions
of tasks as we've seen to understand
different types of data photos videos
audio text and to do so with remarkable
efficiency and speed but this video
wasn't designed to scare you it was
designed designed to show you the
tangible ways in which gpt4 and rival
models from Google will improve on chat
DBT so you can be prepared I genuinely
do believe that the knowledge economy is
about to be upended and the better
prepared we can be the better for all of
us and I really hope this video has
contributed to that if you feel it has
please do leave a like leave a comment I
read them all much appreciated see you
soon


AI for science - DeepMind: The Podcast (S2, Ep6)

welcome back to deepmind the podcast
with me hannah fry
for this series i've been speaking to
researchers at the cutting edge of ai to
find out what they're working on and the
implications it could have for all of us
in the last four episodes we've been
looking ahead to the long-term future of
ai including some of the ideas that
researchers hope will bring them to
artificial general intelligence
but in the next two episodes we want to
share with you some of the ways that ai
is already being put to work along the
way
here's deepmind ceo demus asabis
my personal reason for working on ai was
to use ai as the ultimate tool to
accelerate scientific discovery in
almost any field and i think alpha fold
is our first massive example of that
if you heard episode 1 you'll know all
about alpha fold deepmind's
state-of-the-art system that can
accurately predict the three-dimensional
structures of proteins
but alpha fold isn't the only science
project within these walls
step inside the labs and you'll find
researchers scrutinizing dna to
understand the mysteries of life
hunting for new ways to harness nuclear
energy or putting ai to the test in some
of the most mind-expanding areas of
mathematics
so what are you waiting for
come on in
this is episode six
ai for science
[Music]
push meet coley who we heard from in
episode one oversees deepmind's ai for
science program across the natural
sciences and when it comes to a list of
areas that are being worked on
he is exactly the man to ask biology
proteomics and genomics in
quantum chemistry like material design
fusion fundamental mathematics ecology
weather now that is a fairly
intimidating list but over the course of
the next 30 minutes or so i'll be
walking you through as many of them as
we can get to to give you a sense of the
potential for ai to make a difference to
science
first though you might be wondering why
a company that made its name getting ai
to play computer games became involved
in these serious scientific subjects and
that is where i started the conversation
with push meat
science has been in the dna of deep mind
from the very start it's more of the
case that as we sort of built up these
systems and
proved them on games we then started
thinking about now is a good time to
actually stress test them on the real
scientific challenges that society is
facing
for sarah jane dunn a research scientist
on the science team deepmind science
program is fundamental to the company's
aim of solving intelligence to advance
science and benefit humanity
that's quite an abstract concept to some
people
the point is you have
techniques that can tackle some of the
hardest problems that have eluded the
brightest minds for decades
and so i think being able to show
people the power of these technologies
to really help humanity
that for me is one of the most valuable
aspects of what we're doing in science
what the ai for science team does is try
to identify suitable scientific problems
for ai to work on that will go on to
have the biggest impact
one way of thinking about it is this
idea of the tree of knowledge there are
some problems which are the so-called
root node problems which unlock so many
other problems downstream protein
folding was one such example once you
understand the structure of proteins
that has implications for
developing new drugs or coming up with
new enzymes for breaking down plastics
another example would be
if you had a great source of energy
other problems like access to clean
water become much more tractable once
you have unlimited energy at your
disposal and then once we have sort of
isolated some of these root note
problems we then think about
what is the role machine learning and ai
has to play
and now that deep mind has made
significant progress on protein folding
it's turning to some of those other root
node problems that push me mentioned
one of them is nuclear fusion
[Music]
it's hard to emphasize just how big an
impact a breakthrough in nuclear fusion
might have
it is the long-held dream of scientists
around the world that one day we would
be able to fuse two hydrogen atoms
together and in doing so create a
totally unlimited totally clean and safe
supply of energy
it would spell the end of our climate
crisis
but before we get too carried away i
should tell you that nuclear fusion is
really hard
to get atoms to fuse you first have to
heat them up until they form something
called a plasma a state of matter that
is so hot that the electrons are
stripped from the atoms
here's demises to explain one of the
really hard things about fusion is how
do you contain this plasma that's like
as hot as the sun and
it would burn anything obviously they're
touched so the way you have to contain
it is in a magnetic field the problem is
that the plasma is almost in a chaotic
regime so at any moment a bit of it
might just sort of shoot down a certain
direction and you have to change the
magnetic field quick enough to respond
to that to keep hold of the plasma for
multiple seconds
the plasma is held inside something
called a tokamak
it's like a giant metal donut big enough
to walk through
the magnets around the edge create a
field strong enough to suspend the
plasma in the middle away from the sides
but the plasma is wobbly and unstable
and as soon as it touches anything it's
game over
the comparatively cold sides of the
tokamak saps the energy from the plasma
and the heat dissipates almost instantly
up till now what people have done in the
physics world is they've hand written
mathematical controllers for what the
plasma might do whereas our system what
it's learned to do is predict what the
plasma might do
from the shape of it and then almost
ahead of time change the magnetic field
to react to what it thinks is going to
happen and even more excitingly we're
actually doing like plasma sculpting so
we can actually split it into two or
elongate it it's almost like ice
sculpture but with plasma that's as hot
as the sun yeah it's a hot as a sun
but as push meat explains it's not just
about keeping the plasma stable it's
also about which configuration of the
plasma you are keeping stable there are
certain configurations which produce
more heat and what reinforcement
learning now allows you to do is
basically
you can say i want this funky
configuration because i think that it is
going to be more stable or it will
produce more heat capacity and so on and
the ai is like yeah i can do it for you
rather than sort of a human spending a
year of their time sort of thinking
about how to control the different
current in the coils and how to maintain
the right voltages and so on for that
particular configuration i see so it's a
shortcut basically it accelerates the
whole research cycle amazing
and as if protein folding and nuclear
fusion weren't enough to keep the ai for
science team busy they've been pursuing
a raft of other projects including in
the field of ecology
this has taken ai out of the science lab
and into a setting where you might not
expect to find it at work
where we're working in the serengeti is
very classic
lion king east african landscape so you
have the rolling grasslands it started
with those iconic acacia trees broad
open savannas
meredith palmer is a conservation
biologist at princeton university in new
jersey but you'll usually find her with
a pair of binoculars out in the field
you have something like 700 species of
birds 50 species of large mammals
that are traversing around the greater
serengeti ecosystem
so down from tanzania up into kenya in
this kind of endless circle chasing the
rains
and when the migration comes back into
town you can hear the bleeding of the
wildebeest the sounds of the flies that
are following the migration around
this annual gathering of animals the
great migration as it's known
is one of the last in the world and one
of many reasons that ecologists are so
interested in the serengeti we have
lions but we also have cheetah and wild
dog and hyena and leopard
and all of these species are trying to
co-exist
and so to understand all of the
interconnections between these wildlife
we need to be studying these wildlife
communities
unfortunately though climate and
agricultural change as well as illegal
poaching are all having a growing impact
on the local ecosystem
about a decade ago to help them monitor
changes in the serengeti ecologists
installed devices known as camera traps
across a 1 200 square kilometer area of
serengeti national park
camera traps are
small remote cameras we can strap them
to a tree
and leave them running 24 hours a day
seven days a week for months or even
years
the camera traps are triggered by the
heat and motion of passing animals to
take pictures we can examine the images
to see what animals are in them and what
these animals are doing
as you might expect these camera traps
take a lot of pictures lots of fantastic
candid shots and lots that show the back
of an animal or some shape that's a bit
hard to discern
imagine letting a four-year-old run
loose with your phone and then scrolling
through the camera roll at the end it's
a bit like that
these camera traps are producing
10 to 20 000 images a month so we as
ecologists are really facing this
massive
deluge of data and the big issue for us
is that we can't use the images straight
out of the camera trap to address our
research and conservation questions
someone has to go and look at each and
every one of these tens of hundreds of
thousands of images
and write down you know this image
contains
three adult zebra and one baby zebra i
once calculated that it would take me
seven or eight years to process a single
year's worth of image data
faced with all this data meredith and
her colleagues became early adopters of
citizen science we essentially
crowdsourced the classification process
and you could look at our camera trap
pictures and identify all of the animals
and behaviors that you saw in our field
data
after a while though a problem arose as
the number of citizen science projects
exploded the snapshot serengeti project
as it was called could no longer attract
enough volunteers to manually go through
all of the images that needed labeling
snapshot serengeti's impressive data set
of labeled wildlife images caught the
attention of researchers at deepmind
they got in touch offering to train a
computer vision algorithm that would
automatically classify the thousands of
photos taken by the camera traps
identifying which particular species of
gazelle for example appears in a photo
it blows my mind how incredible
ai can be for solving these kinds of
problems
i've had moments where
i've looked at an image and had a
computer identify a species i you know
on first glance didn't even recognize
was there
so there's some animals that the ai is
99 on all of the time
however
if we don't have a lot of labeled images
the algorithm doesn't do so well
so there's a lot of these cryptic rare
little species aardvarks cirillas
servals janet's and when we only have a
couple hundred images of those
the computer can get a little wonky just
because it doesn't know what to look for
computer vision systems and pretty much
all ai systems come to that are only as
good as the data they're trained on
which also explains why the ai sometimes
struggles with those creatures at the
top of the animal kingdom
one issue that we discovered using human
volunteers to classify our data
is that people really really really want
to see lions
we've had images where
it's a warthog sunning itself and our
citizen scientists have very confidently
classified that image as containing one
very impressive male lion
and no i'm sorry it's not it's a warthog
we do have a lot of checks and balances
for making sure that we're feeding
the ai the very best labeled image data
that we can but some of these things
slip through the cracks
the community of serengeti ecologists
are already starting to see the benefits
in their monitoring efforts
tanzania used to have an enormous
population of black rhinoceros but that
has dwindled to fewer than a hundred
2019 nine black rhinoceros were
reintroduced into the grameti game
reserve a part of the serengeti
ecosystem that is monitored by camera
traps in conjunction with ai technology
we'll be able to look at if migration
routes change we'll also be able to
document any impacts of new mega
herbivore effects on the surrounding
human communities
we've documented several of the
rhinoceros consorting with each other
so we anticipate that we're probably
going to have a couple of baby
rhinoceros to look forward to
in the next year or two
for meredith the future is not for ai to
replace ecologists but to complement
them
the way we've been doing ecology to date
you know out there in the field with our
binoculars and our notepads watching
you know a single warthog all day like
it's not fast enough and the more that
we can get a helping hand from ai
the faster we can identify the problems
that are driving change in the ecosystem
the more targeted conservation
interventions that we can make
to save and protect and rebuild these
ecosystems
[Music]
we've heard how computer vision systems
are being used to make sense of biology
and wildlife on a vast scale
but as we heard with alpha fold deepmind
is also using ai to understand life at a
microscopic scale
back in the science lab researchers have
been trying their hand at genomics an
area of biology dedicated to
understanding human genes and dna
deep minds genomics work is still in its
very early stages
so what are push meet coley's hopes for
the future of this research
the implications could be very
wide-ranging if you truly understand
biology then you understand how
all cells behave even cancer cells so
you might have better treatment for
cancer you might have better ways of
creating new
organs for transplantation what you're
describing here really there is a
potential step change in all of
healthcare and medicine that is the
implication of truly understanding
biology we have in some sense been
observing biology
but to truly leverage biology that's
something that humanity
is only starting to understand the
implications of
so
are you ready to get up to speed on a
brand new bit of science
i'm game if you are
i started out life as a mathematician
but when i was thinking about what to do
i was wasn't so taken with the idea
maybe of studying fluid flow down a pipe
for three years no offense to to me
which is exactly what i did
sorry
this is the delightful sarah jane dunn a
research scientist working on the
genomics project
genomics it's a domain of biology and
it's most interested in understanding
the connection between your genotype
that's basically your dna sequence and
phenotype and that's everything from how
tall you are to what color hair you have
to maybe whether you have some
particular disease or not
genotype is what you see in the dna
phenotype
is what you get
i have the mc1r genotype rs185005
which gives me the phenotype of red hair
genotype cftr delta f508
gives the phenotype of cystic fibrosis
those however are the examples where the
connection between genotype and
phenotype is clear and established
unfortunately such examples are the
exception rather than the rule
take for example the bracha one gene
which is linked to the development of
breast cancer
not all people with the braca one
mutation will go on to develop breast
cancer but there's currently no way of
knowing which people with the gene are
more at risk
in some cases this has led some women
with the bracha one gene to have surgery
as a precaution the hope is that a
deeper understanding of genomics may
provide important additional context
about what puts someone at increased
risk or not
if we could unpack more about how the
rest of your dna sequence can tell us
about the progression of that disease
then we might be able to make more
nuanced diagnoses about these kind of
things and then projecting into the
future
how might you be able to change that
before the disease actually develops how
might you be able to to develop the most
effective quick treatments those are all
questions that are open to us after that
[Music]
research in genomics builds upon an
important foundation known as the human
genome project
starting in 1990 the project took the
full set of dna of one anonymous human
their genome and over the course of a
decade went through every molecule one
by one
every single one of the three billion
chemical base pairs a t
c and g
that combine together to give some
twenty 000 or so genes
and a whole lot more besides
thus providing the entire manual for how
to make a human
but in some ways this was just the
starting point
the human genome project
successfully
was able to compile
this whole sort of cookbook for a human
you have bought this book and there's a
saying but you don't know how to read it
that's what scientists are sort of
struggling with
you can think of the amazing human
cookbook as a literal book by the way
one of the early versions of the human
genome project was printed and bound as
part of an art project and it's stacked
up as some 20 phone book size volumes
where every page inside was a blur of
a's t's c's and g's
and therein lies part of the problem
we still have very little idea about how
much of it comes together to form the
cells tissues and organs in the human
body
it is a list of ingredients
but it's not in order and that's what we
are trying to understand how do you
decipher
the actual
sort of cooking process
so you've got one end some flour eggs
milk and a cow yeah and at the other end
you've got a beef pie and you're like
what the hell happened in between
yeah exactly
in humans only about two percent of our
dna actually goes towards our genes the
remaining 98 of those 20 odd phone books
worth of biological instructions are
what they officially call the non-coding
region
less officially biologists have referred
to it as junk dna you know that shows
how much we were interested in it but
after the human genome project was
completed it became obvious that this
so-called junk dna was actually really
important
by now we know what some of the junk
does
some of it is structural stuff like the
scaffolding needed for when a cell is
working
some of it is just random nonsense
but a lot of it is on and off switches
a series of regulators that tell genes
when to get to work
what's interesting is that you know all
of the cells of your body share that
genome but they make use of it in
different ways and they do that by
expressing some genes and not others
every cell contains a complete copy of
every gene but will only require a
handful of genes to be active at any one
time
a gene that prompts cell division is no
use in a tissue which has finished
growing otherwise it could lead to a
tumor
a gene that is active in building hair
or teeth has no role in the cells that
make up the liver or the heart
the hope in all this is that if you can
understand the on and off switches as
well as everything we already know about
genes and proteins it can help
biologists to better understand the
mechanism by which certain diseases take
hold in the body
take for example sickle cell disease an
inherited health condition affecting red
blood cells
there are of course certain signs of
this disease in someone's phenotype if
you look at the red blood cells under a
microscope those that belong to someone
with a disease will take on a
characteristic shape like a crescent
moon
thanks to the human genome project we
now also know that there are certain
changes in the dna that are common among
people with sickle cell anemia
what's still unknown and where genomics
can help
is how
that sequence of letters in someone's
non-coding dna
translates into how the disease will
progress in individual cells
if you're wanting to then understand how
this disease
actually arises in the body and then
ultimately how you might treat it you
need to go through these layers of
information you need to understand well
what does that change the dna sequence
actually
do within the cell does it mean that the
cell can no longer express certain genes
does this give us targets for what we
might be able to attack in that cell to
develop kind of drug treatments so it's
about
understanding from that very first clue
about what might give someone a disease
how it kind of bubbles up from that
basic level of dna
that
is where ai and deep mind's work on
genomics fits in
there is some precedent here already
scientists have a whole series of
special cases where they've already
cracked the connection between dna
sequences and cell function these
studies all serve as a solid foundation
of biological data for deep minds 4a
into genomics research
we're trying to build ai that
essentially can understand the genome
the fact is we now have the kind of
level of data where deep learning can
really take hold and we've been able to
build together the kind of deep learning
models that are able to read a dna
sequence a bit like you might read a
piece of text
as a first step in its genomics work
deepmind is using a relatively new idea
in the world of machine learning a type
of architecture called a transformer
it's useful to think of it as a sort of
translator
it reads in a string of letters from dna
and based on the sequence of a's and t's
and c's and g's it will translate it
into a prediction for how the genes will
actually manifest themselves
to do language translation well you need
to pay most attention to the key words
in a sentence and the transformer uses
the same idea applied to dna
as it runs through the code it holds on
to the most important parts of the
sequence the bits that are most likely
to provide the relevant bits of context
and it pays less attention to all the
filler bits the dna equivalent of all of
the ands and the ins and the vers in a
sentence
you need more context to be able to make
better and better predictions because
some of the things that can influence
what's happening are positioned further
and further along the dna sequence and
so the transformer architecture that we
build is able to swallow in about 10
times as much of that dna sequence
that's 10 times more of the dna sequence
than previous ai models could read
by being able to extend how much context
the model had about the dna sequence we
were able to increase the accuracy of
our predictions about gene expression
from sequence
this all sounds pretty positive
but before we make it sound like machine
learning or ai is some sort of silver
bullet for all problems in biology and
science it's important to sound a note
of caution too
unfortunately there isn't a big button
labeled machine learning in the corner
where you can just gather any old data
hit the button and get out great science
to use ai effectively you have to ask
the right question that for me is one of
the biggest challenges to say okay i
know that there's a big question in the
field about how we can
you know differentiate a progenitor cell
into a heart cell because we'd like to
be able to make them in the lab so that
we can treat people with heart disease
but how can you take something like that
and then wield ai effectively it's being
able to understand
first of all what kind of question you
should be asking and then what kind of
architecture is going to be able to deal
with those data but then also how do you
know if your model's getting better or
worse so it very much is still an area
of research
the other thing about using ai the
sophisticated pattern hunter that it is
is that it doesn't always give you a
good answer for why those patterns
appear
do we have to be nervous about
making predictions without really
understanding the causal mechanisms yes
i think so but it doesn't mean that
these kind of tools aren't valuable the
thing i like about ai is that it can
allow us to
handle some of the complexity that
biology throws at us and
often when we're dealing with biological
data we're dealing with very noisy data
and data that has been perturbed by just
experimental error and so if we have the
tools that can somehow surf over some of
those
bumps in the data and try to really pull
out those meaningful patterns then the
thing that we have to be really rigorous
about is how we interpret it and it's
not the only tool that should be
deployed but if it gives us pointers in
the right direction if it steps us
forward that one piece then that is
still valuable
but it's all very well for computer
scientists to turn up in some new
scientific domain hoping to plant their
flag
as we've heard machine learning
techniques like alpha fold and this
genomics work build heavily on existing
scientific data that is often obtained
using painstaking experimental methods
so how do other scientists respond when
they hear that ai could be applied to
their domain
scientists are
a very competitive bunch but they are
very collaborative as well if they are
convinced
that the approach that you are proposing
is an approach worth pursuing
they are interested in listening to you
have you ever struggled to persuade
people though
so when we started looking at applying
machine learning and ai to pure
mathematics
we encountered a number of
mathematicians who were very very keen
and became really good collaborators but
there were a few who
on matters of principle
thought that mathematics is a very human
endeavor and a machine has no role to
play in it
i'm shocked personally
i'm shocked that some pure
mathematicians might not welcome the
future with open arms
you're probably wondering what that pure
mathematics project was all about
well the first thing that you need to
know is that in pure mathematics proof
reigns supreme
someone might have a conjecture like
there are an infinite number of prime
numbers for instance but that means
nothing until by a chain of logical
statements it can be proven to be
indisputably true for all eternity
so what we were interested in is can a
computer have that intuitive ability to
propose a new conjecture what we started
looking at with some of our sort of
collaborators is about
different aspects of not theory and
topology which was suspected to have
some interactions but nobody had sort of
discovered those relationships
not theory you will not be surprised to
hear is all about the mathematical study
of knots
i know the name might not make it sound
exciting but you're just gonna have to
take my word for it on this one that the
whole field is a glorious playground of
bending and twisting loops of imaginary
string
there are different ways to describe
these knots an algebraic way
and a geometric way and it's quite hard
to translate from one to the other
but then ai started hunting for
connections
and that resulted in a new conjecture
which had never been seen before and not
only that the human mathematicians then
managed to prove that conjecture
so in some sense it's basically humans
who are proving the conjectures that a
computer has come up with did that
persuade the grand pp or mathematicians
i don't think we went back to them
[Laughter]
and that is just a flavor of some of the
science applications that deepmind is
currently hard at work on
from ecology to genomics protein folding
to nuclear fusion the ways in which ai
could make a genuine difference to our
lives in the near future are
increasingly clear
and there are so many more projects that
we didn't have time to cover in this
episode
so if you'd like to find out more about
the ai for science program do take a
look at the show notes or check out
deepmind.com
in the next episode we'll continue to
discover how the applications of ai
could have an impact on the real world
from weather prediction to voice
synthesis
i'm a mathematician author and podcaster
who's fascinated by artificial
intelligence
it's really good i know it's good am i
that breathy bloody hell
you've been listening to deepmind the
podcast
i'm hannah fry and the series producer
is dan hardoon at whistledown
productions
we'll be back soon


241  Finetuned Language Models are Zero shot Learners

hello and welcome to session 241
in the aisafety.com reading group
tonight we will be discussing fine-tuned
language models are serial shot learners
by jason way and others
this is some research done by a team in
google research
with jason way as the primary author and
martin bosma as the person who conceived
the idea and implemented the first
version of of the actual model
and there are a number of other
primary authors and a list of secondary
authors
it's a pre-print
from september
they describe it as
fine-tuning language models on a
collection of data it's described by an
instruction substantially boosts serial
shot performance on unseen tasks
so let's talk zoom out a bit first and
discuss what is the relationship between
this and ai safety because the uh uh the
connection is not entirely obvious
um and they don't write anything about
ai safety almost
um so we had the uh language models our
few shot learners the gt3 paper uh a
long time ago uh where uh
when we presented this uh i presented
this uh as almost two years ago i made a
uh comment on that and that is can you
fine-tune gt3 to be more aligned
uh i don't think that was very original
a number of other people almost went in
must have bought the same thing um but i
uh but a lot of people there are very
few people uh actually pursued this
um some people have started to do that
over the past year or something like
that but um there's something more new
and aligned of course is the property
that we are interested in and there are
a number of properties that a language
model can have or can uh can't have and
it's not really obvious a priori what
properties it will have an example of a
property would be like does it is it
mathematically precise in its uh uh is
it logical or is it more poetic and if
you look at
the actual dpt3 it's certainly more
basic than precise
so a um
a way to
split up alignment in three parts is is
it honest is it uh harmless and
is it helpful so those are three
properties that a language model could
have
and it's worthwhile to investigate so
can be honest well
honestly seem to be a
property that doesn't um
map well into language models like there
is some work being done on truthful qa
um some ways being done but it's uh
certainly not obvious and you see
sometimes uh that as the model gets
smarter it is more likely to tell you
that uh breaking a mirror results in
seven years of bad luck
harmless is harm a concept that is
natural for l for language models not
really uh
not really uh there is some work being
done for instance by uh redwood research
um to try to see whether it is in fact
possible to make a language model uh
fine-tuned in a way so it doesn't
um
so it doesn't uh talk about violence but
it's not obvious can you make it helpful
well that's really what this paper is
about
instruction following is quite close to
helpful and
um
so when i'm not entirely happy about
this uh uh
seeing alignment as a combination of
these three uh features i'm not sure
it's a good model i think it's um yeah
it's either i think it's populist genres
framing and might also be
the uh
uh i'm a days i'm i need to look that up
i think
um because i feel
on reflection
helpful seems like the kind of thing
that
people care so much about that it's
going to be done by default so uh i
think even if it was not very helpful
there would be enough optimization
pressure put in that direction that it
probably would end up being helpful or
otherwise it wouldn't be deployed as
much
right so let's
go back to language models and their
limitations we've seen obviously that
if you take a
language model and just deploy it
without any other work then it's not
that good compared to how much it costs
um
and
it needs some kind of hints about what
you'll be doing that's sometimes called
fusion learning learning so it needs a
few examples and it's also something
that doesn't really happen until the
language model is quite large
um we also speculate that this could be
because the thing that it's been trained
with and um the problem formats
in the uh in the tests are quite
different it's unlikely to see precise
kind of way of talking
in the uh
45 db that um
that ut3 is uh is trained with
so you can see here in um with gt3 you
have pre-trained language model and then
if you want to do inference on some kind
of task then uh you will give it a few
um a few examples um and you can try to
change the prompt in different funny
interesting ways prompted engineering
sounds better than prompt hacking but
that's basically what you can do or at
least the way it's written here uh i
think the authors almost certainly know
it but gt3 certainly can be fine-tuned
so you can even though they say this is
like the gpt3 way you can't do more
things with ut3
and the alternative they're saying is
that um you can pre-train and then
fine-tune um
where you
perhaps somewhat laboriously uh find
some examples of uh maybe like 100 or
even more as many as you can get um to
fine-tune the model on these
and then you get a specialized model
that is substantially better at one task
and
the reason why
we feel
this helps is that
it feels very often like
the models in some subjective way it
seems clear that the language model
itself it knows the answer to the
question that we are given but it's just
we need to either prompt or fine-tune in
order to make the model want to give us
this kind of
to follow our instructions basically
let's talk about instruction tuning the
new um
kind of fine tuning that the authors are
implementing proposing and implementing
here so they take
62 natural language
problems
and
they chase those into natural language
so uh we'll get back to how to change a
natural language probably into a natural
international language
and then you get some natural language
that you can then fight your mind
so that gives you a different option
where you um have a pre-trained language
model and then you fine-tune it on um a
lot of tasks and then that makes it of
course better at the task you fine-tuned
it on but also at following other kinds
of instructions
and they call this the fine-tuned
language net
um
as uh somewhat of a strange name in my
opinion uh like in there as far as i can
tell there's nothing more in it like
than the other kind of transformers that
we're seeing um so uh and i think
fine tune is not a that descriptive name
it would be more interesting to see
like it's um
instruction following it would be my
suggestion for a name
um
and so they show that this does indeed
improve zeroshot performance very much
and also that this is something that
that only happens with large models
uh one important thing here uh that
should be mentioned is that open ai has
an
uh a version of gt3 that does actually
kind of the same thing
they haven't published how they've done
that um
but
so this is the best we have so to
understand that part
right how do you change these uh
standard benchmarks into natural
language
um so here is an example of one that
contains a structured problem that
contains a premise and a hypothesis and
then some kind of target where you have
some options
and then this is cheese international
language by stating first the premise
and then based on this above can you
conclude this and then you have these
options below
they have 10 templates for each uh
problem um
and they
up to three of these templates change
the problem uh dramatically so for
instance instead of trying to see if the
hypothesis follows from the premise they
are asking to translate the premise into
fringe
and that was kind of an odd thing to do
i felt
and they don't really explain why except
i found it actually in one of the
appendix hidden that they did this but
it turned out not really to matter and
fine-tuning is
unfortunately uh an art form at this
point um we
you try something
and then sometimes it doesn't work
they have a classification scheme um i'm
a bit uh there i that's written here but
it's uh like
uh it's presented but it's quite unclear
from the text whether this is something
novel or how is
uh how do you normally do classification
in these language models i think
actually that's the same way that gt3
does but
uh it's not presented as such i'm unsure
whether this is actually something new
the data sets are clustered in a rather
interesting way that's in a way that's
very essential to this word
so here are
62
problems that are
first divided according to color
in
generative here and
understanding natural language
understanding and then it's following uh
it's
divided into further clusters
um some of this like there's a reading
comprehension a common sense and a
reading comprehension with common sense
you know um
they admit this is somewhat of an art
form how do you put this into clusters
and i actually think this would be
really interesting to um to see some um
rigorous work being done with of course
uh in ai clustering things is something
that is very very well known there are
standard ways to do that and would
actually be really interesting to see um
like i can make predictions like
something like sentiments
um might not be something that is
natural for language models and i kind
of predict that if you try to to cluster
this to see how much how close these are
to each other um i i'm not sure this
would look at all like this
um and also it should be said that this
the the uh
the metrics they uh using to say how
good is their um
is their model performing depends
strongly on how many clusters they have
than the exact clustering structure
i don't think the authors
quite admit how important this
clustering actually is
uh they make some uh of course when you
do machine learning you need to to split
into uh like training validation and
test and they do that in a
somewhat interesting way because
obviously uh they have taken literally
all the uh the standard benchmarks they
could get their hands on and so
the problem is if you're fine-tuning
these then
what do you bid smart and the way they
do that is to just leave one cluster out
and then the fine tune on the rest and
see how they perform on that cluster
so that means they need to do the um the
testing
but uh oh well this thing is not that
expensive
and they say that actually these
clusters um they
they didn't follow those precisely um
sometimes there were some that were
still relevant even if they are outside
the cluster and i think in that case you
should just have made the clusters
bigger really that would accomplish the
same thing as fast i can tell
um so how did they train this uh well
they they start by describing the
language model that they use um and to
my mind it seems like gt3 obviously gt3
is the model that i know best and google
news some other models that i'm less
familiar with and
they don't describe this model in relay
the base model in relation to any other
models so um i would really like to know
like what did they do different from qc3
like there's one thing that's trivially
obvious is that um
they are using less parameters 137
instead of 175 billion um
but um
it also looks like their model they're
doing something really something some uh
it looks to me like some kind of pre
pre-training um with uh on just getting
the model to understand sentences um but
i i can't tell for certain whether it's
uh how new and original this is um and
another thing that makes this somewhat
annoying is that they have this model
that is kind of like tv3 but they don't
compare it head to head with gt3 in the
unfine-tuned version they do in the
fine-tuned version and then they can see
it's better but it would actually be
interesting to see like maybe all of the
small tweaks that they've made and of
course the state of the art has advanced
over the past two years so it's
perfectly possible that the base
language model outperforms dbg3 we just
don't know that because they don't
really
report this result
they also have a description of their
fine-tuning procedure i won't go into
details it looks kind of normal i think
they use some kind of packing um
scheme that i haven't seen before and
it's again one kill to me is this
actually original or is this uh like
i don't think people do that in the when
gt3 was first created but is that like
how everybody does that now i don't know
and of course the length of the bossy
input and the
the output is set uh
to uh yeah
to these lengths um and this is
something that i feel is very important
because here for practical purposes this
matters a great deal
and describe how long time uh the fine
tuning takes uh and yeah 60 hours it's
not
not that much and of course tpuv version
3 that's like so last year right now
everything is measured in extra flops
and i can't even remember how much how
many an extra flop is
um
so the results here you can see the
results as presented like we have keep
t3
with zero shots uh g3 with flu shot and
the new flam model
and in general very often it um it
outperforms
and
it's certainly better than the zero shot
gt3 in
20 of the 25 tasks
and
also even better than the uh the future
performance of gpg3 often
and
uh is this instruction tuning is a
substantial improvement in general
um
and but it kind of depends on precisely
what task some tasks are um
uh almost instructions in themselves
like the uh the classical example of a
task that is uh it would be continue
this sentence that's something where in
giving instructions to gbt3 doesn't
really make sense because it's what it
does it by default gpg3 just continues
the sentence and if you have a uh
something
where the task is to continue its
sentence you don't need instructions it
just does it and and some of them um
are very much unlike that um
so um
yeah the kind of uh conclusion is that
if their instructions are redundant then
instruction
doesn't help very much but it does help
a lot in in the other case
they have an interesting
idea for how to make a
serial shot uh learning where you just
get the problem and future learning
where you have like a few examples and
that's uh by automatically generating
those from examples and not uh and not
answers we're using what they call exam
class
and that's the simple idea is you just
take a sim
one example and then you record the
output and the output is probably
quite bad because that's zero shot in
the first time but then you add that to
the prompt so now you have one
uh so now you have one shot
and then
you're getting something new from that
and that gives you then you have two
shot and then you continue to do that
until you don't have more space in your
prompt
um
and that's the technique that seems to
improve performance substantial um
uh and um especially if the if it's like
something where there's complex output
and
we also noticed that um
the uh standard deviation the difference
between
uh
how well you perform on different
examples is just lower so prompt
engineering which is
which will often have this effect uh is
less important in practice this can be
can be automated
um
and i think this is
to me
some kind of hint
that the architecture of transformers
just isn't powerful enough like uh this
is something that you put on top of the
transformer architecture to to
make it uh
to focus its attention on precisely the
kind of thing
that you want with this particular
prompt and seems like something that a
smarter uh architecture than
transformers would be able to do itself
without having having to add this
um how does the
performance improve when
you add more of these clusters gradually
well here you can see a graph with how
many clusters you've used um they don't
show for serial clusters and that's a
bit sad
you
alright they kind of do here this here
is the base model so they do in fact put
that on these
these three lines and you can see it's
actually a bit funny that for two of the
tasks the uh just uh fine tuning on one
cluster decreases performance so the
first time you start this instruction
trimming process you get slightly lower
performance
but then as you add more and more you
can see the uh the graph just continue
going up and it seems like the they are
they obviously chose 62
62 uh
tasks because that's all they had um and
if someone uh sits down and just writes
some more
uh
some more tasks more benchmarks it seems
quite likely that they would just be
able to uh increase this curve further
uh so it's basically bottlenecked on the
availability of benchmarks
um
yeah the only one that that doesn't seem
to help immediately is sentiments
so how does it scale uh with uh
increasing
model size
well if it's something that has already
been seen
then um
the performance seems to
increase very little as far as i can
tell that's actually uh somewhat strange
that this model doesn't really seem to
benefit from more uh
for more parameters but if you're
instruction tuning it then it is used to
improve very much and but this is only
on task that is seen during instruction
in june so that's kind of what we would
expect like the more um like you get a
bit of
an improvement as you add more examples
that seems very reasonable but on the
other kinds of um
uh
tasks the kind of task that you haven't
seen
we get a much more interesting like
first you fine tune on something that is
not what you're trying to do
if the model is small then that is just
bad if you have a small model you
shouldn't fine tune on other kinds of
instructions
but it seems
as you can see here
at
this point once the model gets above
this particular size
then
instruction tuning starts to help and
starts to help help dramatically so this
entire exercise if you go back a couple
of years back when the largest models
were
eight billion parameters back then this
approach just wouldn't make sense in
general the model was unable to learn
what instructions mean and what's
helping me
and apparently once you get
above this level then
the model becomes smart enough to find
some kind of like this is mostly my
interpretation that it becomes able to
figure out okay we're trying to follow
instructions and instructions are so it
becomes more helpful at at this point
once it gets smart enough
here
that's all for today thank you and see
you next week bye next time


GPT 4 Got Upgraded - Code Interpreter (ft. Image Editing, MP4s, 3D Plots, Data Analytics and more!)

I just got access to the code
interpreter plugin about 48 hours ago
and I've been running experiments on it
non-stop since then I've come up with
about 18 examples to show you guys its
power most of them I reckon haven't been
seen before I predict many Industries
will have to update overnight when it's
released more widely and at the end of
the video please let me know what you
think and what other experiments that we
can try first though what about this one
a 3D surface plot just quickly the way
it works is you click this little button
to the left of the text box and then you
can upload many different file types
like CSV files Word files images and
even short videos then it will
automatically analyze the file type
without you pressing anything and then
of course you give it a prompt and as
with all of chatbt it becomes a
conversation so the first 3D surface
plot was decent but it was too small so
I simply said in natural language can
you make it four times bigger thank you
and of course you have seen the amazing
end result even with the lighting look
at the shadow is there I believe this is
based on a real contour map of a volcano
in New Zealand and I could do a whole
video just on this but I have 17 other
examples to get to but this one was
truly amazing did you know for example
it can generate QR codes I said create a
QR code that I can scan with my phone to
reach the following URL and lo and
behold it creates it and yes it does
work maybe I'm easily impressed but I
think that's pretty amazing and what
about a 3D scatter plot this is truly
remarkable I uploaded the data from
gapminder and it created this chart
based on the median age of over a
hundred countries from 1950 I think
projected to 2100 and I asked highlight
the UK this is indeed the UK's median
age through those years in red but I
know what you might be thinking that is
amazing that it's 3D and interactive but
the blue kind of merges and it's hard to
see what's going on I engage in a
conversation and look what it created it
picked out the 30 most populous
countries and separated them off with
separate colors look at that that is
gorgeous
now you might have the critique that the
median age is in descending order in the
y-axis going from 20 down to 60 so in a
sense the median age is actually Rising
not falling but nevertheless that's
easily amendable and that is truly an
incredible diagram and look just for fun
I'm going to go into the data look at
this I'm traveling into the data this is
so wild I don't know how helpful it is
but I think that's just beautiful and
crazy there are so many Industries data
analytics accounting consultancy that
this will affect by the way it got all
of this done in about a minute I see a
lot of people online talking about five
seconds later it is no way done in five
seconds you have to wait 30 seconds a
minute sometimes much longer before I
move on I want to give you a killer tip
that it took me quite a while to work
out so when you get access try to
remember this say output the
visualization as a downloadable file if
you don't add that phrase as a
downloadable file what will happen is it
often gets stuck at this stage of the
code it'll either say fig.show or
plot.show and then just stop I found
that I encountered this problem far less
often if I said output a downloadable
file next did you know that code
interpreter can do optical character
recognition I screenshotted this text
from a New York Times article I think it
was and I asked OCR the text in this
image and write a poem in Danish about
it now I don't want to exaggerate it
often gets OCR wrong I don't want to get
your hopes up it fails more often than
it succeeds but when it works it can do
it understood the text and then did a
poem in Danish about the text now I'm
going to need a Danish speaker to tell
me if that was a good poem but either
way it could do it how about this one it
can do interactive time series with
range sliders and selectors I uploaded a
CSV file on life expectancy data from
the entire world and I just said can you
pick out the U.S UK and India and create
a Time series with range slider and
selectors again that killer phrase
output a downloadable file and here is
what it came up with notice how the life
expectancy for all three countries Rises
during the 20th century and look how I
can select down here interactively a
range of the data and even by clicking
up here a 10-year interval or 50-year
interval but here's the crazy thing I
did nothing I just uploaded the file
there were hundreds of countries in
there you can see here all the steps
that it did and if you click on the
Arrow you get to see the actual code
then it goes through shows its
explanation and eventually gives you a
link that you can simply click and get
the file downloaded and if you weren't
that impressed already here's where it
gets fairly game changing you can get it
to do the data analytics not just the
visualizations for example I said find
five unexpected non-obvious insights
from this data and offer plausible
explanations for them this was bad to
the median age data for the most
interesting observation provide a
compelling and clear visualization now
ignore the first diagram which wasn't
that good because of the x-axis but look
at the insights this is data analytics
you can see here that the original file
was called median age years and it was
just a table of data no analysis
whatsoever but look what gpt4 picked out
in site one the global median age has
been steadily increasing over time it
calculated the global median age that
wasn't included in the data it was just
country data and it says it's gone from
around 22 years to over 38 years in 2023
and it's projected to continue rising to
approximately 44 years by 2100 and then
it offers a cogent explanation this
trend is likely due to a combination of
increasing life expectancy and
decreasing fertility rates worldwide as
Medical Technology improves more people
are living longer birth rates are
declining particularly in developed
regions is pick this all out and then it
moves on to the the next Insight the
countries that have seen the most
significant increases in median age are
these ones and again it gives an
explanation as to why their median age
might have risen more than any other for
example Albania has seen significant
emigration of younger people which could
also lead to an older median age is it
me or is that kind of crazy that it
crunched all the data visualized it but
then also gave really interesting
analyzes of the data now you can read
the other analyzes but each of them are
really interesting and the final
visualization which I asked for is
brilliant I think notice how the graph
goes from green to red when you get to
the Future projection I didn't ask it to
do that now obviously in this video I'm
going to focus on the flashy visuals and
the cool little tricks it can do but in
terms of data analytics that is what is
going to change jobs change Industries
and remember this is code interpreter
Alpha version one look at the difference
between mid-journey version one and now
mid Journey version 5 a year later how
about basic video editing now there is a
limit to what it can do but it can do
some basic video editing if you ask it
for example I uploaded a short file and
asked it to rotate the file 180 degrees
and it was able to do it now I'm not
saying that is massively useful but it
was able to do it here is a similar
example I uploaded an image file and
then said can you zoom out from the
center of the image now initially it did
zoom in but then I clarified that I
wanted it to zoom out from the center
just to be cheeky I also asked can you
make it black and white oh and I also
asked to add music but it couldn't add
music anyway here is the end result by
the way it gave it to me as an mp4 file
and look it zooms out from the center
and it's made the image black and white
now because I got access so recently I
honestly haven't explored the limits of
what kind of video editing I can do with
chat GPT code interpreter but I will let
you know when I can now back to
visualizations I gave it a hypothetical
scenario that sounds kind of realistic I
sent 231 CVS got 32 responses 12 phone
interviews three follow-up face-to-face
interviews and one job offer which I
rejected I'll put a downloadable Sankey
diagram of this data I did then get it
to change the coloring slightly but I
think that's a pretty cool Sankey
diagram look sent CVS 231 and then
receive responses and you can go down 32
phone interviews 12 face-to-face
interviews and three job offers and one
rejected offer obviously I could have
tweaked that for hours make it more
visual make it more interactive maybe
make a gif of it but for two minutes
work I think that's a pretty interesting
and incredible output next and here is
one that you might say is a little bit
concerning and it's about steganography
now I will admit I am not at all an
expert in fact I know virtually nothing
about it essentially what it involves
though is hiding a message inside an
image or in inside some code and gpt4
was more than willing to play along and
it encoded a secret message into an
image there was the image by the way and
if you looked at that you'd think that's
totally normal that's just a silly
little image right well apparently
here's what it can do to a casual
Observer it looks like a simple image
with some shapes but it actually
contains the hidden message hello world
then it provided a python function which
can be used to decode the message from
the image now obviously this is just a
silly example that is totally harmless
but am I being crazy in thinking this is
a somewhat concerning ability for future
language models to possess especially
when they reach the level of an AGI
often openai talk about future versions
of GPT doing scientific research and
finding things that humans wouldn't have
discovered but let me pose the scenario
that it gets better than any human
expert at steganography anyway enough
from me I'll let the experts weigh in on
that one next did you know that gpt4
with code interpreter can do to text to
speech just before anyone comments
though why did I write proceed without
further question because GPT 4 with code
interpreter has a tendency to always ask
clarifying questions and if you have
access to only 25 messages every three
hours you don't want to use up half or
more of them on clarifying what it wants
to do or saying yes please do that but I
found writing proceed without further
question means it gets straight to it
and essentially you get double the
number of prompts for your money anyway
as you can see I asked turn this entire
prompt starting from the beginning into
a text speech file now quite a few times
it denied it had the ability to do this
but eventually I got it to work it was
actually when I finally gave it this
prompt and it worked I say it worked but
it didn't quite work as intended check
it out here is the text-to-speech that
it came up with a large language model
trained by open AI when you send a
message containing python code to python
it will be executed in a stateful device
a notebook environment python will
respond with the output of the execution
or timeout after 120.0 seconds internet
access for this session is disabled do
not make external web requests or API
calls as they will fail now thank you
Stephen Hawking for that message the
only thing is it had nothing to do with
my original prompt now anyway when you
get access to code interpreter play
about with text-to-speech because it is
able to do it even if it denies it time
for a fun one I asked create a tree map
of the letters in the following quote
and I'm not going to read it out because
I am not good at tongue twisters anyway
I said give each part of the tree map a
different color and output a
downloadable file proceed without
further question and here is the output
and I checked it for the letter P and it
was correct that there were 36 instances
of the letter P in the output and look
how it's proportional with the number of
instances of the letter and the size of
each rectangle I think that is pretty
insane okay back to something more
serious I uploaded this file which is an
image of a math problem quite a hard one
as well and you guessed it I said solve
the math problem in this image it then
extracted the text from the image
presumably using OCR and then proceeded
to solve it and I'm going to get onto
this in a second it is better at math
than Wolfram Alpha I know that's a big
claim but it's far less buggy I found
Wolfram Alpha crashing very frequently
anyway here are the two solutions and
isn't that incredible from a photo
essentially it then extracts out the
math problem including the two square
roots and then solves it this is all
within the same window of chapter no
need for any other apps or extensions
next it can do radial bar plots which I
think are really quite beautiful I'm not
saying this is the best one ever and I'm
sure you could tweak it to make it more
clear and beautiful look at that the
life expectancy in the US climbing from
1800 and then it goes clockwise reaching
a projected almost 90 by 2100 again I'm
sure you could do a far better job than
me in extracting out a more beautiful
diagram but aren't radial bar plots just
beautiful to look at speaking of cool
diagrams how about this I didn't even
specify which visualization to do I
uploaded this same life expectancy data
and I just said what are the most
advanced and Technical visualizations
you can do with this data proceed to do
them now honestly it picks some
visualizations that I don't think are
the most advanced but nevertheless it
was creative here is what it did it does
frequently make the mistake of
cluttering the axes and having far too
many labels so that you can't see
anything so scrub that one out not great
but what about the next few remember it
just did this on its own this is a heat
map and you can see some really
interesting things from this data like
India starting with a much lower life
expectancy than anyone else but
gradually Rising but still falling
behind the others even in 2100 and look
at China look how the life expectancy
drops in the 60s and 70s I think we all
know what happened there compare that to
the US which is a gradual continual
Ascent actually aside from 22 20. look
how the shade gets a little darker in
2020. obviously you guys can probably
work out what happened around then but
then the projections are for it to go up
toward 90 by 2100 that's a beautiful and
clear heat map that I didn't even ask
for it to do let's look at the next one
box plot do you remember those from
school you get the upper end of the data
the highest one the lowest one the
median the first quartile and third
quartile and it's a great way of
statistically representing a set of data
and it's done it for every 50th year
starting in 1900. obviously a slightly
less beautiful diagram than some of the
ones you've seen today but for the
statisticians in the audience you will
know that this is a very useful metric
for a lot of data the individual points
above and below are typically when there
are outliers in the data I would
estimate that all of these
visualizations only took around two two
and a half minutes so definitely not the
10 seconds as I said that you often see
on Twitter I mean have you ever seeing
gpt4 give an answer in less than 10
seconds speaking of useful I think many
professionals will find the next thing
that I'm about to showcase the most
useful of all any insights that Gypsy 4
finds Trends medians analyzes whatever
you can ask it to add to the original
file and then download it do you
remember that the original file was
called median age years well notice this
file name median age years with insights
it has created a downloadable new file
with the insights included and look at
some of the insights that I mean you
have the change from 1950 to 2100 and
here is the average median age
throughout the period and the change
from 2023 to 2100 notice that the
original file didn't have those columns
they were added by gpc4 with code
interpreter and now how about data
progression video files I was honestly
shocked when I saw that it could do this
but I asked can you make a 256 by 256
MP4 that gradually reveals the lines as
they progress on the x-axis this was
about the median age over time here is
what it did and look at how the data and
the chart progresses as time moves along
I was really shocked to see this and the
line in red which is going to be labeled
at the end is the global median age and
remember it calculated that that wasn't
in the original file now I'm not sure
why it picked out these four countries
maybe because they represent extremes
either way I think the result is
phenomenal and I'm genuinely impressed
that it did this even though I know the
final result could be improved
dramatically for example far higher
resolution and maybe the global median
age labeled from the start and actually
now that it's got to the end I can see
why it did pick out these countries
because Niger did have the lowest median
age in 2100 and it looks like Puerto
Rico had the highest and the fastest
aging one was Albania next and this this
is going to shock quite a few people
what about image editing I created this
image in mid-journey version 5 and then
here's what I asked I said use opencv to
select the foreground of this image and
look what it did it picked out the
foreground no Blue Sky now I know it's
not perfect but it's nevertheless
impressive all within the window of
chapter BT this does actually make me
wonder if open Ai and chat to BT is
eventually not now but in a few years
gonna swallow all other apps or maybe
Google's Gemini but either way one
interface one website one app doing the
job of all others and by the way of
course chapter BT is now available on
iOS but imagine you have one app and it
can do image editing text-to-speech
video editing everything data analysis
not add gpt4 levels but GPT 6 or gbt 7
levels if you can get every piece of
information service and application in
one interface a bit like now people
being addicted to their smartphones
won't people be a addicted to this one
interface again that's not going to
happen now but I'm just posing it as a
question to think over for the moment
though before anyone gets too carried
away it does still hallucinate quite a
lot so I uploaded this image and I asked
it questions about it and it answered
and I was like wow it can do image
recognition it said this image appears
to be a digital painting of a humanoid
figure at a desk with a rather complex
background I was initially amazed until
I realized that it probably got that
from the file name because when I asked
it questions it got it wrong so I said
what is on the desk now look back
there's this weird kind of microphone
and a bit of paper and not much else a
keyboard and look what it said there are
multiple floating holographic displays
okay a mouse not really a desk lamp I
can't see that and then tools and
devices now correct me if I'm wrong but
I think most of those are incorrect now
obviously I need to do far more
experiments to see if it actually can
recognize any particular images and
maybe I'm putting it down too harshly
but at the moment it does seem to
hallucinate if you ask it about too much
of the detail of an image next remember
how one of the key weaknesses of GT4 is
that it can't really count things
especially not characters words Etc and
even more so it can't do division and
some of you might be thinking well with
Wolfram Alpha it can do those things not
quite here is an example of the code
interpreter plugin essentially eating
Wolfram Alpha obviating it making it not
obvious what the utility of it is if
you've got code interpreter I asked
divide the number of the letter e's in
this prompt by the number of the letter
T's now you might think code interpreter
can improve things by doing the
character counting but it can also do
the division notice how it counted the
characters correctly compared to Wolfram
Alpha and of course got the division
correct as well so if it can do Advanced
quadratics and do division and character
counting Etc it does beg the question
what would we use Wolfram Alpha for that
we can't use code interpreter for I
honestly might not know something that
you guys know so do let me know in the
comments it also also got this math
question correct and notice you get
these beautiful map visuals that you
don't get with the base version of gpd4
you get something more like this where
the visuals aren't as clear and notice
the base version of GT4 gets the
question wrong it can't do division but
with code interpreter it gets the
question right next one is a quick one
pie charts nothing too special but I
think it is a fairly beautiful
visualization it doesn't seem to matter
how big the CSV file is that you upload
this next example was really quite
fascinating it was a word puzzle I have
tried this particular word puzzle on
gpt4 dozens of times the reason I picked
this puzzle is called a Word ladder is
because it really struggles with the
puzzle if the number of steps required
is more than a certain number usually
about five or six steps it gave me a
really interesting border of the limits
of gt4's planning abilities with
language anyway it always gets it wrong
here is a demonstration with the base
model of gypsy 4. you might say why is
this wrong but look at how it's changed
change from Seas to sags which is more
than one letter change and that's
typical of the kind of Errors it makes
what about with code interpreter well
you can probably guess the ending given
that I featured it in the video but it
gets it right I believe it draws Upon A
hard-coded word set and this does Point
towards the kind of puzzles that I think
gpc4 with code interpreter will be able
to solve things like crosswords and
sudokus okay not exactly world changing
but nevertheless I think quite
fascinating and how about Venn diagrams
the reason I picked this example is that
I had to go through about 10 steps to
get it to create this rather basic
three-way Venn diagram this represents
the overlap between dogs Ai and desks
and apparently all of them are loyal
companions well we will see about that
but anyway it took quite a few steps to
get it right which was pretty annoying
but here's the really interesting thing
once I got it set up in the way that I
like all I had to do was say use the
format above to create a new three-way
Venn diagram this time for Mango's Movie
heroes and marmosets try to make each
entry funny and use different colors
proceed without further questions so it
may have been a struggle to set up
initially but once done it was so easy
to iterate a new three-way Venn diagram
and actually it was better than the
original apparently all three are adored
by fans worldwide apparently only
marmosets and Movie heroes can climb up
trees really fast and mangoes and
marmosets can hang upside down that's
crazy one or two prompts iterating on a
design already agreed upon this is
honestly what is likely to happen in the
future with people spending hours to
find the perfect data visualization or
piece of data analysis and then just
hitting copy paste for all their other
files perfect it once and then it does
the rest for you a quick couple of bonus
ones before I finish you can just ask it
to come up with a visualization giving
it no direction at all it came up with a
distribution of prime numbers up to ten
thousand thing is I believe there's a
slight mistake at the beginning because
I think there's only 25 in the first 100
and 21 in the next 100. so you probably
do want to still check the outputs that
code interpreter gives you and that's
another reason it's not going to
instantly replace all data analysis and
data visualization it's not perfect and
it's not fully reliable but you've got
to look ahead to where things are going
I'm going to end where I started with
this insane 3D surface map of a volcano
if this is what gpd4 can do now with the
Alpha version of code interpreter what
will GPC 5 or 6 do with version 7 or 20
of code interpreter I was about to
speculate about that but then I got
distracted with trying to get inside
this volcano it is kind of fun look I'm
going above and into the volcano let me
know what you will try when you get
access I know they're rolling out
steadily and I know that some people
will have had access to it for about
three weeks so hopefully if you want to
experiment with it you will be able to
soon in the meantime do let me know if
you have any ideas that you want me to
experiment with and thank you so much
for watching all the way to the end have
a wonderful day