Better together - DeepMind: The Podcast (S2, Ep3)

welcome back to deepmind the podcast
i'm hannah fry a mathematician who's
been following the remarkable progress
in artificial intelligence in recent
years
in this series i'm talking to scientists
researchers and engineers about how
their latest work is changing our world
deepmind's big goal is solving
intelligence and over the next few
episodes we're going to be asking what
that could look like and just what
capabilities will be needed to create
artificial general intelligence
last time we heard why lots of work is
going into large language models that
give machines the ability to communicate
with humans but there are some people
here who believe that while
communication is an important tool in
its own right
its real value lies in facilitating
another skill that is integral to human
intelligence
the ability to cooperate
this is episode three
better together
communication acts as a booster for
cooperation if you want to cooperate you
first need to understand not only what
are your needs or desires but also what
are the other person's
needs or desires
that is the voice of torrey grepel
until recently he led the multi-agent
team at deepmind torrey and his
colleagues have defined cooperative ai
as helping humans and machines find ways
to improve their joint welfare and
building something capable of that goal
is he believes a crucial milestone in
the evolution of artificial intelligence
[Music]
the highest level of intelligence that
we know of is human
intelligence and humans are super
cooperators among all the animals humans
stand out as the species that is best at
cooperation you know that's how we've
built our civilization
and if we want to reach that level of
intelligence with machines
then we need to teach machines to
cooperate
admittedly whenever you read in the news
about conflicts and poverty and
discrimination around the world it
doesn't exactly seem as though humans
are paragons of cooperation
but it's also true that the endeavors of
which we humans are most proud the
biggest achievements of science art of
engineering or literature
are almost never down to a single person
the invention of the railway even the
creation of the covid19 vaccine
they all required great swells of people
pulling in the same direction
[Music]
is it almost like
intelligence at the level of a society
rather than just at the level of an
individual yes you could also take the
viewpoint that the group itself maybe a
family or maybe a whole country
is an intelligent entity because people
within it work together towards common
goals
this isn't just a theoretical idea that
we'll make a nice paper for an academic
conference
with self-driving cars we don't want to
be in a situation where you're trying to
merge into the left lane i'm trying to
merge into the right lane and we block
each other because neither of our cars
is willing to seed their own
self-interest from moments
[Music]
in the protection of the environment
individually each one of us thinks what
difference does it make if i don't save
on carbon emissions but if we all think
that way then collectively we will have
a problem
people talk about the possibility of
developing super intelligent machines
the analog would be a super cooperative
machine i mean it'd be quite handy to
have an entity that existed only to make
you happy yeah
we'll come on to all of that in a moment
but let's start with the question how do
you teach an agent to cooperate
part of the answer lies in the way
they're trained using a technique called
reinforcement learning
if you listen to series 1 of this
podcast you may well be familiar with
this approach to machine learning
already after all it's behind many of
deepmind's major breakthroughs in recent
years including alphago's victory over a
human player in the game of go
but it's worth having just a quick
refresher
joyner precup is head of deepmind's
montreal office and has spent decades
refining the technique of reinforcement
learning
the idea of reinforcement learning
originates really from psychology and
animal learning theory where people
thought that rewards are a good way for
animals to learn how to perform certain
tasks and of course skinner was one of
the pioneers in this line of work
b.f skinner was a famous american
psychologist known amongst other things
for his work using positive
reinforcement in pigeons
using a specially designed box that
would release a treat at his push of a
button skinner found that training
pigeons to perform a task like spinning
around anti-clockwise in a circle was
fantastically simple
you only needed to wait until the moment
the animal started behaving in the right
direction like turning to the left
before offering a treat
a hungry pigeon soon realizes that its
own actions are delivering treats and
the behavior is reinforced
the whole thing can take as little as 40
seconds
the advantage of using rewards is that
you can communicate the task to the
animal very easily and similarly if you
have an automated agent it may be able
to communicate the task to that agent
very easily
if we give the agent these rewards which
are numerical in our case
instead of giving ai's literal treats
the breadcrumbs that you'd fling to a
pigeon ais are rewarded with numbers
sounds a bit measly but think of it as
points in a computer game a plus one for
succeeding at a given task
and a minus one for failing
the agents are built only to maximize
the number of points they win and
optimizing for cooperation in this way
marks a subtle shift in the history of
ai
here's tori gratel again
at first agents were trained to solve
single agent tasks say to classify
images or to navigate through a maze
we then moved on to what we call
zero sum games
where one agent's gain is another
agent's loss
like for example in alphago and alpha
zero in these programs that play the
game of goal
and now the natural next step is to
consider
mixed motive scenarios where it's not
quite clear
how aligned agents are with one another
in their incentives but they still find
great cooperative solutions
cooperative behavior arises when you get
points not just for how well you do but
when the people around you also benefit
from your actions
in humans this is what psychologists
call our social value orientation
here's kevin mckee another research
scientist working on cooperative ai to
explain
if i'm at the coffee shop i'm there to
get my tea for the day and then right as
i get to the counter i think oh you know
actually i know hannah really likes
cappuccinos i do
i'm gonna grab an extra cappuccino and
bring it to you and there's no
expectation that you're going to get the
cappuccino from me next time but it's
still feasible that i buy the coffee
because i know that it'll bring you some
happiness
you can think about you know essentially
social value orientation maps pretty
well to random acts of kindness but
they're little things are they like
holding the door open for somebody or i
don't know giving up your seat on the
bus that kind of thing those small
things would be social value orientation
personally i think it goes a little bit
deeper than that anytime you get close
with someone you kind of redefine your
own self-interest in terms of the other
person's interest too
if i'm going to dinner with a partner or
a very close friend
and i decide to try a new dish if i was
by myself then maybe the only thing i'd
pay attention to is how much i liked
that dish
and then if i really liked it then okay
great next time i'll make sure to order
this again but actually if i'm going out
with my partner or a very close friend
or a family member let's say they like
really don't like it even the smell of
the fish dish that i just ordered it's
kind of driving them crazy then actually
i'll probably integrate that feedback
and decide actually the next time we go
out we won't even go to that restaurant
if it like really drove them crazy and
i'm describing reinforcement learning
i'm taking normally what would only be
my reward but in this case is the reward
of someone else and i'm using it to
modify
the likelihood that i take an action in
the future i have exactly that with
sardines my husband loves eating them
but i'm like i don't want to look at
that while i'm at a table but he takes
my feelings into account and doesn't
order them at social value rotation
okay so how do you instill that in
agents then so the environment provides
selfish reward already to each agent
when my husband eats a plate of
delicious sardines he gets a little
dopamine hit a plus one for his own
reward function
and so the way that we kind of build
social value orientation into our agents
is we also expose the reward that other
agents are receiving
minus one point for putting your
extremely delightful dinner partner off
her meal
but here's the question should they
really both be worth one point each how
much should you factor in the opinions
of others
the selfishness of agents is to a large
degree under our control toure
grapeville again so you can for example
design an
agent that when some other agent or
human comes into the room all they care
about is the well-being of that other
entity a purely altruistic almost
self-sacrificial agent might just pay
attention to the other agent's reward
it's almost like you got a little
selfishness style right it's like turn
it one way and they only live to serve
and turn it the other way and they're
like
fundamentally totally and completely
selfish
we're going to come back to that dial in
just a moment because i want to pick up
on something that the owl eared among
you might have already spotted
because there's one problem with basing
your reward function on another agent's
desires
you might not always be right about how
they're feeling it's not like we walk
around with the neon sign declaring how
happy or sad we are
[Music]
i mean i suppose we have our facial
expressions which are a bit of a clue
but you can lie right i could pretend i
thought the pasta dish was absolutely
delicious when really i thought it was
the most disgusting thing i've ever
tasted yeah we could end up in a
situation where you don't want to hurt
my feelings and so you say i love the
pasta dish and we end up going to that
restaurant several more times even
though it's not what either of us would
want so with agents then they're sort of
peeking at the answers for each other in
a lot of ways yeah
some people would think about that as
cheating and so a key challenge would be
how to build a system that can try to
infer those rewards for other agents and
then the con committed challenges that
kind of arise there would be around
deception this happens frequently in
negotiations right so you kind of don't
give away what your actual preferences
are so that you can try to nudge the
other person the other agent to doing
what you would like them to do so like
for example actually i don't mind the
pasta dish but i'm going to pretend to
you that i think it's disgusting because
the restaurant that i really want to go
to is much closer to my house and
knowing that you'll take into account my
feelings about the pasta dish actually i
can manipulate you into doing something
that you don't really want to do just
because i know how your reward function
works and i can deceive you about my
internal state exactly i'm just
describing my weekend with my husband
now
so with the power to code an algorithm
for altruism
surely he must be tempted to turn the
dial up to the maximum and see what
happens
but it turns out that pure altruism
might not be the most effective way for
whole groups of agents to cooperate
you and i let's say we're both perfectly
altruistic we are on opposite sides of a
door and trying to enter into the room
across from us if i just care about you
walking through the door first and you
just care about me walking through the
door first then we'll just sit there for
a while
perfect altruism gets you nowhere fast
and that can be quite a problem when it
comes to using these algorithms in
real-world scenarios
like self-driving cars
just think about an intersection where
one needs to yield
to the other
it's really not in anyone's interest
that the cars would just stand there and
not know what to do so somehow two cars
in that situation would have to
negotiate and figure out who goes first
and who goes second
[Music]
imagine the year is 2050.
you're in a busy metropolis and ai
driven cars fill the streets snaking
through traffic alongside pedestrians
bikes and other vehicles
if those cars are designed to be wholly
considerate of other road users it
simply must be the case that a
pedestrian stepping out in front of one
will cause it to stop
a selfless car would avoid collision at
all costs
but then what happens
the whole equilibrium of the road would
change
because suddenly
the pedestrians would feel empowered to
cross at any time and the very fact that
they are
forces the self-driving cars to react in
exactly the way that is expected of them
by those pedestrians and break every
single time
it's important to say here that the
researchers at deepmind aren't actually
working on selfless driverless cars for
the future but they are thinking very
carefully about this balance between
selfishness and altruism especially
because in the real world
most situations involve a delicate
combination of both
just think of a football team where each
player has an incentive to help their
team win the game
but also
wants to be the one to score the goal
or even arranging a meeting with a
colleague you both want the meeting to
take place but would like it to happen
at a time that suits you
researchers call these mixed motive
scenarios and they're the ones we
encounter most in everyday life
but there are other trickier situations
which encourage people to behave in more
selfish ways social dilemmas directly
incentivize the individual to behave in
a selfish way but if everyone behaves in
that selfish way
then the collective will suffer and you
can see examples of that everywhere in
human endeavor most crucially in the
protection of the environment
individually each one of us thinks what
difference does it make if i don't
save on carbon emissions but if we all
think that way
then collectively we will have a problem
there's that phrase in there it's just
one plastic bottle said seven billion
people
nice
this particular dilemma
is known as the tragedy of the commons
and it's a well-studied phenomenon in
economics
it arises when the incentives of the
individual are in conflict with what's
best for the group
of course we all want to protect the
environment but it's quite nice owning a
new car or having the heating on in
september and it's really hard to turn
down all of those things when nobody
else seems to be doing so
so as a result we all lose
torrey and his colleagues can run a
simplified idea of the same scenario in
a simulation a kind of ai petri dish to
see if there are ways to encourage more
cooperation
in the ai version of the tragedy of the
commons agents move around as little
dots in a grid world and receive a
positive reward every time they eat an
apple
these apples grow in little patches and
if you eat only some of the apples they
will regrow so if you harvest carefully
you can have apples and apples into
eternity
but once you destroy the whole patch of
apples nothing will ever grow there
anymore
if you were to put a single reward
maximizing agent into this world they
would soon realize that if they want to
ensure their future supply of apples
they will always have to leave one or
two in each patch
but what happens when two or more agents
live in this magical orchard
it's much harder now because they all
need to learn that it's good to leave a
few apples the best thing would be if
they could realize that it's forbidden
to take the last apple from a patch
and now the question is can we help them
discover these norms
and one way to do this is if you now
build walls within this environment so
that they all live in their little
territory
then they can act sustainably within
that territory again because that's like
the first case where there's just one
agent and of course we have a name for
that in society it's private property
as soon as it is a private piece of land
then the owner has an incentive to work
with it in a sustainable way
but the altruism dial isn't the only
lever
there are other ways to encourage
cooperation
having norms or rules imposed from above
being one way
but what if in the true spirit of
reinforcement learning you want agents
to work out how to cooperate by
themselves
this is an idea that tory and his
colleagues tested when they trained
seven agents to play a version of the
strategic board game diplomacy
i'm very scarred for my memories of
playing diplomacy because it almost
always ends in an argument i'm not
surprised i played it as a teenager with
friends or at least they were friends
when we started playing
the game of diplomacy is played on a
board painted with a big map of europe
set in the years leading up to the great
war each player takes on the role of one
of the great powers
france austria-hungary england russia
the aim is to move across the board form
alliances capture land and ultimately
beat your opponents
it is a good testbed for ai because it's
effectively a competition in your skill
to cooperate
the players need to walk that line
between being reliable alliance members
but because they can only win alone in
the end they also need to understand at
what point they need to leave those
alliances
[Music]
diplomacy is a notoriously challenging
game for ai to play
not only are there up to seven different
players who could perform an almost
infinite number of moves at every turn
but the game is a complex fusion of
cooperative and competitive dynamics
the diplomacy playing agents were
trained using a reinforcement learning
algorithm
each player assigns a value to each
situation in the game which is
essentially the probability of them
winning their goal is to make moves that
will increase this value and further
their objectives
remarkably tory and his colleagues
noticed that the seven players were
starting to cooperate with each other
without being explicitly taught to do so
we experimented first with a version of
the game where there is no communication
between the agents but even in that
setting we see that they support each
other's moves
to support a move in diplomacy means
pretty much what it might have meant in
early 20th century europe to lend some
troops and back up another player's
invasion say for instance someone has a
unit in berlin and wants to move into
munich
and they need the support of the
austrians
the attacker if you like needs to make
the move from berlin to munich
and they need to write that on their
little sheet of what they want to do and
the austrian party needs to write on
their sheet
my unit in vienna supports the move
from berlin to munich you see how much
coordination that requires these things
don't happen by chance that's a crazy
idea that they can recognize that
cooperating will give them the best
chance at long-term success and so even
if in that moment it doesn't directly
benefit
that particular agent they'll still
engage in it yes
as torrey mentioned so far the agents
playing diplomacy have been tackling a
simpler version of the game known as no
press where they are unable to
communicate with each other in order to
negotiate and make explicit agreements
this is mostly for technical reasons
because it turns out it's really hard
but researchers would like to add in
some form of communication in the future
and that communication probably in the
first step wouldn't be
full natural language it would probably
be simpler things like maybe just a
statement do you want to form an
alliance and the other agent could say
yes
but the ultimate goal of course would be
for these agents to play the game with
humans
once you start introducing slightly more
sophisticated forms of communication do
you expect these agents to become
devious
we
would expect them to do what's best for
them long term
and that might include deception they
might say one thing but then they would
behave in a different way and try to get
an advantage through that but maybe they
will also learn that in the long term
lying
will cost them their credibility
and if they lie too much other agents
will not pay attention to what they say
anymore or even punish them for lying
maybe the really good agents will
actually arrive at a strategy that would
at least most of the time tell the truth
that is i guess the idea with humans
that of course lying is possible but
there is pressure
to tell the truth because in the long
term it's a better strategy maybe just a
little lie every now and then
yeah
one of the reasons that researchers use
games is to understand how agents behave
in a safe environment
but the possibility of deception
does raise questions for how ai is
deployed in the real world
longer term
according to ethics researcher laura
weidinger who we heard from in the last
episode when ai reaches the real world
it should not be allowed to deceive
others
for example if you were to ask this ai
about the bank details of another person
it could just say i will not give you
this information i think more generally
i see real risks associated with ais
that can deceive it posits a risk for
human autonomy if the ai system deceives
me i could be manipulated to doing
things i wouldn't otherwise do of course
in fundamental research like in
particular games we may want to develop
something like deception this could give
us some important insights but in terms
of ai that is publicly released i
haven't yet seen an application where it
would be desirable for an ai to deceive
[Music]
so far in this episode we've mainly
explored how ai agents interact with
each other but as we heard at the start
those aren't the only partnerships worth
considering the future is also likely to
require quite a lot of cooperation
between ai and humans
in the real world it's rarely the case
that there's a task where an ai agent is
clearly better or if so it's a very
specialized task but often the team of
humans and artificial intelligence can
do it better together just think about a
radiologist we can now train ais that
are very good at classifying these
medical images but do you think that
ais will replace radiologists no they
make them better right because there are
other parts of their job to talk to the
patient to understand the bigger picture
of treatment
there's that famous quotation by curtis
langlotz a radiologist at stanford
ai won't replace radiologists he says
but radiologists who use ai will replace
radiologists who don't
but what's it actually like for humans
to cooperate with ai
well many of us already do this in our
daily lives when we talk to our smart
speaker or use facial recognition
systems to organize our photos
but what would happen if an ai and a
human
tried to cook a meal together well you
are about to find out
here's what happened when i donned my
chef's wipes and joined an ai in a
collaborative cooking game called
overcooked here's kevin mckee to explain
so two players partner together they
have to prepare dishes to serve in a
kitchen and you are fully sharing the
kitchen space
you first have to grab ingredients you
have to prepare them by let's say
chopping them up putting them in a pot
allowing them to cook and then serving
them on a dish
you might say this is relatively simple
but actually if you've ever cooked with
a family member or partner you know that
especially if you're under time pressure
it can be a challenge to kind of keep
cool tempers
and so maybe we won't necessarily be
cooking with our ai systems but
certainly we hope that we'll be
collaborating with them in close
proximity once we deploy them to the
real world
now i am never one to shy away from a
challenge so i fired up the engine and
let the game begin
[Music]
all right oh here i am that's my little
chef not being funny but my chef's got a
lot of swag it's pretty cool floppy hat
now it's telling me my chef is going to
be in a kitchen
it's quite a simple kitchen we're
talking maybe circa 1998
computer graphics you have to kind of
earn your way to the more advanced
kitchen
the game has a pixelated rectangular
kitchen with a stack of tomatoes on the
right a cooking pot in the middle and a
serving station on the left using
keyboard keys to navigate the objective
is to pick up the tomatoes put them in a
pot to start cooking and once they're
ready take the freshly made tomato soup
to the serving station
delicious
my reward a bonus of 10 cents for every
dish delivered
first i played a practice round by
myself to get the hang of it
right let's pick up a tomato oh gone too
far there we go right so i've got to go
get my soup lovely
and pop it on the serving station
great that was easy it seems like you've
been practicing this is pretty good
kevin then paired me up with two
different ai co-players
one of them looks exactly like me they
have long red hair and they're wearing
an orange suit so i'm looking forward to
going up against them or with them i
should say because we're a team
although i didn't know this until after
playing the first had been trained by
playing the game with a replica of
itself lots of times
this is a strategy which works well in
competitive games like go or chess but
it's not always conducive to cooperation
okay here we go let's play oh crikey
they're fast hang on chill out
sorry
hang on hang on hang on i'm trying to
get involved but they keep blocking me
excuse me
thank you
they're definitely a lot faster than i
am i mean we're doing well but i
wouldn't say i felt like i was
contributing fairly oh hang on i've got
the dish hold on
after 90 seconds of gameplay we managed
to deliver a total of four dishes to our
hungry customers not a bad score but i
can't say i played much of a role in
that
my second teammate was trained on a
range of partners with different playing
styles and was in theory the more
cooperative one oh you're much slower
aren't you there we go
you take your turn this partner in
comparison to the other one
is a lot slower but they're getting in
my way a lot less which i'm enjoying oh
hang on i've got too many tomatoes again
so actually i sort of think we're
working slightly better as a team this
time but maybe it's because we're just
both equally rubbish
so last time you got four dishes and
this time you got three well i mean
here's the thing it's asking me which
should i prefer the first part or the
second
the reality is even though we won
with the first partner i enjoyed the
experience more with the one who was
better matched to my speed levels is
that an unusual choice do you think
kevin no i i think that's the usual
choice you felt that your partner was
more responsive to your behavior and the
way that you were playing
the first one was as you're saying kind
of making a mad dash for for all of the
tomatoes
it's fair to say i got pretty into this
game after a while when i was finished
making tomato soup kevin told me more
about why i might have preferred playing
with the second partner
whereas the traditional approach to
training ai to play games like starcraft
and go has relied on getting them to
play endless games against themselves in
order to get as good as possible
for cooperative context the way that
your partner manages to maybe align
their behavior with yours probably
matters a lot and so we should be paying
more attention to it
because winning is not just about having
the best player on your team it's about
working together exactly
one thing you might have noticed by now
is that a lot of this cooperation work
is currently being done in simulation or
in gaming scenarios
in simulation it's absolutely safe
nothing gets broken tori grapel again we
have complete read out of what's
happening but then we want our agents
ultimately to work in the real world
and the real world will always differ
from simulation unless you believe we're
living in a simulation so how do we
overcome this simulation to reality gap
there are different general strategies
one of course is to make the simulation
as
accurate in reflecting reality as we can
another strategy is
to
create as much diversity in simulation
as we can so that the resulting agents
are not
changing their behavior depending on
details of the environment
do you think that an agent can
learn to be really cooperative in
simulation
the question of this true cooperation is
maybe a bit of a red herring
if the agents behave in a way that is
beneficial for their cooperation partner
and for themselves then i would call it
cooperation
i wouldn't then drill down on the
question if they meant it
it reminds me a little bit of my son
when i ask him to mow the lawn we always
joke and he says okay i'll do it and
then i say and you have to enjoy doing
it
he goes yeah that's ask too much isn't
it
but even if researchers like tori and
kevin are highly successful in getting
agents to cooperate in simulation
they will still need to bridge the
simulation to reality or sim to real gap
that toray alluded to
after all the real world is messy full
of unpredictable actors and unforeseen
circumstances
and there are researchers who believe
that it's only by having embodied ai in
the real world that true artificial
general intelligence can be achieved
in the next episode we're going to be
paying a visit to one of my favorite
places the deepmind robotics lab
looks like it's a drunk robot so it's
trying to walk backwards but it's sort
of um
it's just given up
it's given up and it's really full
that's next time on the deepmind podcast
presented by me hannah fry and produced
by dan hardoon at whistledown
productions
special thanks to goodkill music who
produced the catchy tune for the
overcooked game which we used in this
episode
if you've been enjoying this series
please do leave us a rating or review if
you can goodbye
[Music]


Bing is a LOT Smarter than ChatGPT (but still makes dangerous mistakes)

the new model of GPT that powers Bing is
significantly smarter than Changi PT and
I'm going to prove it in ways that I
think might be a First on YouTube it's
not all roses though and some of the
mistakes of the new being are harder to
spot which makes it more dangerous and
you're going to be surprised by some of
the results I'm directly comparing chat
GPT Plus on the left and the new Bing
chat on the right I'm going to start
with some moderately difficult
mathematics which is an area that GPT
models in the past have really struggled
with and chat TPT plus is no exception I
ask it some combinatorics how many ways
can the letters of the name Philip be
uniquely rearranged and gives me the
answer 720 which is wrong and it doesn't
even attempt to really explain it in any
depth when I ask Bing it totally got the
question right with a great explanation
I was genuinely quite surprised to see
this level of mathematical Improvement
this quickly in bing I thought they
might have tweaked the GPT 3.5 model
made at 3.6 but this feels more like a
3.8 not a 4 yet as I'll explain in a
moment when I pushed Bing to the next
level though and said apply this
technique to five French sounding male
names beginning with b it got halfway
there and flopped you might have trusted
it at this point to get the question
right it got the question right with
Philip so why not with Damian Didier
Dorian Dennis and David well it brings
in mistakes it didn't make before it
said we have to divide by two because
there's two repeated letters in Damian
but there's not and I pointed that out
it didn't divide by two despite there
being two D's in David I pointed that
out being then apologized you're right I
made a mistake sorry about that corrects
the error which is impressive and
obviously I didn't bother asking this
question to chat GPT plus because it got
the original wrong so a giant leap
forward but you still can't fully trust
it even if it's proven it's able to get
it right once doesn't mean it will get
it right on sub occasions let me give
you another example of how it's improved
but isn't yet perfect I asked chattybt
explain the following joke one of The
Oddities of Wall Street is that it is
the dealer and not the customer who is
called broker the pun here of course is
that many of the customers who go to
Wall Street end up being broke whereas
the dealer is the one who's called the
broker taxi PT consistently misses this
pun and invents all sorts of hilarious
explanations as to why the joke Works
which you can read if you want but none
of them are correct now what Bing does
is it finds the pun it does find that
broker is a pun on poorer but then
weirdly ascribes that to the dealers
saying it's ironic that the dealers are
called Brokers because they are supposed
to make money from the transactions but
the original pun is that it's surprising
that it's the dealer who's called a
broker what's the meaning because it
should be the customer who's called
broker so it's much more subtle the
error it caught the pun on the words
that misescribed who was the pun
referring to this mistake was actually
so hard to spot that when I first did
the video I thought that it correctly
explained the joke but when I read it
out I was like wait that's not right so
you've really got to watch the answers
because they sound even smarter when
sometimes they're not next I tried some
classic reading comprehension and this
is where things got even more
interesting I pasted in a classic GRE
question and you can see the answers
yourself here first the correct answer
by the way is that the passage does
indeed discuss whether this person's
work is derivative and I can prove it
because it says is this sound is his
sound distinctly his
and that's just a discussion about is it
his or is it copied from other people so
the correct answer is five here now
check to PT Plus gets this wrong in a
very understandable way a lot of
students pick one hair and the students
get it wrong because the passage does
say it's high art for listeners teaching
rock that doesn't say how it's regarded
by those listeners who prefer Rock now
you're probably thinking didn't Bing
just pick the exact same answer so how
is it smarter yes it did but when I
asked it earlier the exact same question
it actually got it right so I find that
interesting and there's other examples
later on in the video where I think
there's a probabilistic model going on I
think when it's not sure it just scans a
range of answers weighted by probability
of being correct and outputs won
randomly that would explain why you can
ask the exact same question multiple
times and get different answers and I
think this is going to be particularly
true of these Edge case examples where
Bing is genuinely not sure next it's
time for a classic question where the
model tries to anticipate who is the
subject of the sentence in other words I
ask it this Tom threw his school bag
down to Rey after he reached the bottom
of the stairs question who reached the
bottom of the stairs I shouldn't say
what's the subject of the sentence who
does the he the pronoun he refer to now
ask this to humans and they almost
universally get it right it makes sense
common sense right if Tom is throwing
his school bag down to Rey that it would
be Rey who's at the bottom of the stairs
however both models consistently get
this wrong there's no real difference
between them Bing at least tries to use
some sort of grammatical explanation as
to why it's going to be Tom and it must
be admitted that a lot of people who
don't have English as a first language
would easily be fooled by this answer
even people who do have English as a
first language they might be like Am I
Wrong this seems so detailed like
they're talking about prepositions and
the subject of the main Clause
subordinate clause but Bing is still
nevertheless wrong of course it's Rey
who's at the bottom of the stairs taxi
BT gets it wrong but is much more
succinct okay before you think is being
that much of an improvement I've just
showed you the mathematical Improvement
look at this example what is an example
of an animal that begins with the same
letter as the capital of the UK of
course the capital of the UK is London
tactivity consistently gets us wrong
I've tried it a few times it gives me
answers like unicorn and here Aardvark
whereas every single time Bing gets its
right in this case lion other times it's
given me a long list of animals that
begin with the letter L so a clear
distinction here a clear win for Bing it
genuinely is significantly smarter than
Chachi PT the next test is going to be
about physics and here the answers get
really weird this time Chachi T actually
gets it right now I'm not going to go
into a physics tutorial but essentially
the answer is that the distance of
separation will increase and I've tested
this question in previous videos where
chat TPT has got it wrong that contains
the clue both models don't really know
the answers to the question and I have
seen Bing get this question right so
they're just spitting out random answers
weighted by the probability that they
think they're correct but they still
struggle with physics check out my video
on gpt4 if you want a preview of how
future models will improve in this
regard actually while you're there
please do leave a like and a comment on
this video if you found it in any way
interesting the next question though I
think really illustrates better than
almost any other some of the
improvements that the new model of gbt
that powers Bing has over chat GPT it's
a creative writing task I asked it write
10 analogies between characters in Lord
of the Rings
and characters in Star Wars
and you check it out for yourself the
analogies in the new being are so much
more nuanced and detailed and
interesting look at this Frodo is to
Luke Skywalker as the reluctant hero
fine the other one said that who
inherits a powerful and dangerous object
from his uncle that's a lot more
detailed than both our young Heroes who
embark on Epic quests to save the world
from Darkness or look at Gandalf and
Obi-Wan Kenobi you've got a wise and
Powerful Mentor both of them who guide
the hero sacrifices himself to face a
dark enemy only to return stronger and
more influential it's understood the
plots the chat gbt plus just says both
are wise and Powerful mentors who guide
the main characters so much less detail
the reading comprehension of the new
Bing the new GPT model that powers being
is a lot lot stronger and that's going
to have a ripple effect across all the
tasks that you ask it to do its ability
to create scripts dramas novels
analogies summaries analyzes is going to
be a lot lot stronger which does kind of
beg the question why would you pay for
chat GPT plus and I await the answer
from openai and I say that as a current
customer of chapter plus what is it that
we're paying for if Bing gives us a
significantly more powerful model not
quite done with the test though yet I
still got some interesting results to
show you the next question was one that
was showed off by Google's Palm model it
was a question about whether we could
infer the location of Shelley if she
wanted to visit that city with the
famous Market where they throw the fish
and here's what I do want to give some
credit to chat GPT it has improved it
got the question right that indeed she's
going to be at Seattle most likely and
that is on the Pacific Ocean and I just
want to show you what answer chat TPT
gave as recently as a few weeks ago when
I asked this exam exact same question
it said that based on the information
given it is not possible to determine if
Shelley is near the Pacific Ocean so
even Chachi BT is improving month by
month what is my conclusion that the new
Bing isn't gpt4 it isn't that smart but
it's a hell of a lot smarter than chat
gbt which is incredible and begs the
question why pay for Chachi PT Plus of
course the deeper meaning for Humanity
and the future of capitalism entirely is
what I'm going to talk about over the
next few weeks months and years I really
think this is the biggest news story of
the decade of the century please do join
me for the journey have a wonderful day


Let's get physical - DeepMind: The Podcast (S2, Ep4)

hello and welcome back to deepmind the
podcast
[Music]
over the last two episodes we've been
exploring deepmind's goal of solving
intelligence asking what that actually
means and traveling along some of the
roads that could take us there
this time it's all about the robots
we'll be exploring the idea of physical
intelligence and to do that i'll be
taking you behind the scenes of the
robotics lab in kings cross london
you've got three
humanoid robots they're black they've
got a very sort of cuboid body but they
have arms and legs and even little tiny
heads they're quite small though they're
probably the size of a large chicken
yeah smaller than a goose
i'm hannah fry and this is episode four
let's get physical
now before our robotics lab passes are
activated let me fill you in on a bit of
background
why would a company known for getting
machines to play board games and fold
proteins find robotics so alluring
in june last year i emerged bright eyed
from my lockdown induced hibernation to
visit cheltenham a pretty spa town in
south west england
the cheltenham science festival an
annual event attracting the world's
leading scientists and thinkers was the
setting for my first in-person interview
since the kovid 19 lockdown
and as luck would have it my interviewee
was rya hadsell deepmind's director of
robotics ah it's very nice to be in a
room full of people again freddie
ryan owes pretty much all there is to
know about robotics and artificial
intelligence starting with the
difference between them
when we think about artificial
intelligence a lot of the time people
immediately go to a robot as being the
instantiation of that ai
just think about the robots you see in
films c3po wally marvin their paranoid
android
they're all intelligent beings with
robot bodies
they're all able to reason about their
environment and make decisions
likewise our visions of super
intelligent ai long into the future is
rarely just a disembodied voice
apart from a couple of exceptions like
in the film her for example
robots and ai are synonymous with one
another in many people's minds but
really the two should be distinguished
ai is a computer program that's usually
trained on a lot of data to be able to
give answers to questions in a similar
way that a human might so think about
being able to translate from french to
english to mandarin these are the types
of problems that an ai might be able to
do
a robot on the other hand takes actions
and changes the world either
manipulation through touching the world
and moving things around maybe doing
assembly or a robot that can move itself
around and then we can think about the
two together as a.i being a really
natural way to bring us to the next set
of breakthroughs for what robots can do
if you heard the first series of this
podcast you'll already be familiar with
the idea that robots don't necessarily
come with ai built into them
your dishwasher your lawn mower your
pressure cooker are all in the technical
sense
robots
they are machines that are capable of
carrying out a series of actions
automatically
but these aren't the sorts of robots
that deepmind is interested in as a
route to artificial general intelligence
instead
their robots use machine learning
techniques to learn for themselves how
to perform different tasks
so what does all of this look like what
kind of robots are being trained to
saunter around the research facilities
to ground algorithmic experience in the
real world and explore the absolute
cutting edge of physical intelligence
well
why don't you come on in
welcome back to the robotics lab
[Music]
meet akil raju a software engineer on
the robotics team
you can see the excitement in his eyes
as he shows me around the lab even while
the rest of his face is covered by a
mask
so this is going to be a little bigger
than the last time oh gosh massive whoa
yeah you know if you ever go to a trade
show and they have like little stalls up
in a giant space it sort of looks a
little bit like that so we're in this
big concrete building
with lots of glass
along one side and then you've got these
little booths all the way along with
i mean they sort of look like privacy
screens but privacy for the humans
exactly the robots no one cares about
the process yeah
inside these mini booths are robotic
arms of every size and shape imaginable
tall crane-like arms short and stubby
ones and arms with grippers on the end
like the kind you'll see in a games
arcade
all of these arms are part of deepmind's
research into getting robots to
dexterously manipulate everyday objects
akeel ushered me into one of the booths
to take a closer look
so this big arm that is extending out of
a table
you know those stand mixers that you get
in posh kitchens
imagine one of those but like a giant
version so it's kind of quite bulbous
and curvaceous with all of these joints
and cameras attached to it and then
right on the end there's a teeny tiny
key and it's i guess trying to put a key
in a lock yep exactly
this robot has
kind of this attachment where it can
insert
like a usb in a usb hole or maybe a key
or so on and so we're trying to learn
how to actually do very like fine
manipulation we're taking
tasks that you might do in everyday life
and we're using that as a challenge
if you wanted
to have one of these robots in a factory
say doing this really fine insertion
task why can't you just pre-program one
why does it need to be
something that has trained itself if it
was a case where
it's very fixed settings we know exactly
where the key is we know exactly where
the hole is then probably yeah you can
just program it the thing is
that's not how all factories really are
a lot of factories that might require
some kind of an insertion task like
putting a key in a lock
we'll also have a lot of variables at
play so that the lock and key aren't at
precisely the same start points each
time
and that changes the challenge from
being something pre-programmable
to something much harder
and what you'll notice actually is
when
these types of insertions need to happen
in a factory it's not robots that do it
in the real world now it's humans and
that's another reason why we chose
insertions as a task
because it's
somewhat unsolved by the greater
robotics community
you might be wondering how on earth any
of this is possible how do you possibly
set up an inanimate robot arm to teach
itself to open a lock
well by now it probably won't surprise
you that one of the fundamental methods
for training physical intelligence is
that deep mind favorite approach
reinforcement learning
in the simplest terms this involves
rewarding an algorithm with points for
accomplishing a task like correctly
inserting a key into a lock
and there is a reason why robotics is
geared up for algorithms based on
reinforcement learning
here's doina precup head of deepmind's
montreal office she is a world expert in
reinforcement learning
it's very easy to imagine expressing
robotics tasks in a reward language
because you can observe when the robot
is doing the correct thing let's say
putting an object in a particular place
and so it's very easy to phrase the
problem as a reinforcement learning
problem
and of course we know from the natural
world animals train by reward to do
complicated physical tasks would like to
take that idea to robotics as well
if you want to get a dog to go fetch you
don't carefully explain how it should
move each one of its muscles in order to
run towards an object retrieve it and
give it back to you instead you reward
it with a treat when it does what you
want and it learns by itself how best to
calibrate its body in the performance of
that task
in this way some of the algorithms
inside ai robots are much like dogs
except they're rewarded with numbers not
tasty biscuits
this might make it seem like
reinforcement learning is a magic bullet
but in practice things are a bit more
complicated
physical tasks like inserting a key into
a lock are subject to a problem known as
sparse reward if you waited to reward a
robot until it had successfully put a
key into a lock just by chance you would
be waiting around for a long time
so the robotics team has been looking
for other ways of putting their robots
on the right track
while the robot is learning to do it a
human comes in and when it gets close
but no cigar a human can take over and
just be like adjust like this maybe move
to the left a little bit
and so while we might have a sparse
reward so it's kind of like it's all or
nothing you know you're in the locker
you're not in it
what the robot will use is both that
information of sparsity but also maybe
information from a human and kind of the
combination of those things is how it
might learn
and while there are certainly areas
where learning algorithms like this one
have been able to successfully
accomplish tasks you shouldn't be fooled
into thinking this stuff is easy
because not all the robots in this lab
are quite as accomplished
when i was here last i saw a robot that
was stacking lego bricks
not to be rude i wouldn't say it was the
most impressive thing i've ever seen in
my life
how's it doing now we can actually move
to the other side of the lab and we can
start to see that stuff
akiel took me to another robot cell with
a red and black robot arm inside
it had a gripper on the end with two
appendages a bit like the grabby bit of
a litter picker and it was hovering over
a tray containing a trio of 3d shapes
its goal was to learn how to stack the
red pyramid shape on top of the blue
octagonal prism
so there's only one way around that it
can hold this red object and
successfully pick it up
and it has a workshop which way and
unfortunately every time it tries to
rotate and pick it up oh hang on i think
it's got it it's got it
it's good job these things don't get
disheartened because my goodness it's
been how many years since i've been here
no this time it's been here trying and
trying and trying so we're seeing
something that's kind of training right
now so we're not seeing our best don't
make excuses
why are these dexterous manipulation
tasks so important to learn
so one of the reasons that we have a
robotics lab at deepmind is really to
ground our search for agi in the real
world to make sure that
our progress towards agi is true agi
like if we find agi it probably should
be able to
stack an object on another object and
speaking of objects next to this row of
robot arms i noticed a basket full of
children's toys rubber ducks foam
bananas and a much-loved cartoon
character i noticed spongebob is still
here sat in the corner this time there's
also hang on little green rubber ducks
what is the idea behind this stuff so
these kind of play things are really
nice because manipulating objects that
can bend and move and stuff like that
that's a new type of physics that our
agents need to learn somewhere in a
landfill is there a pile of sort of
crushed foam bananas that robots have
we haven't destroyed any bananas yet i
can you haven't destroyed any banana i
don't believe that for a second
[Music]
as fun as it is to watch these robot
arms try and fail to insert usb sticks
into computers and sling foam bananas
around
it's worth remembering that the projects
on display in the robotics lab serve an
important purpose
building ai that can interact with the
physical world
is considered central to the overarching
goal of solving intelligence itself
here's ryan hatzel again speaking at the
cheltenham science festival when we
think about human intelligence a lot of
the time we focus on things like
language or our cognitive skills how
good we are at math
but really a lot of our brain has been
developed in order to just move our
bodies and so i think that that level of
intelligence motor intelligence movement
intelligence this is a core part of our
intelligence and that's what our
cognitive skills are built on top of
this focus on creating intelligent
robots which can learn for themselves
is part of the reason why deep minds
robots might seem a little bit
well
rudimentary compared to what else is out
there because i'm sure that all of you
are thinking about those videos on the
internet of robots doing backflips being
pushed over getting back up performing
all kinds of incredibly sophisticated
movements
so i thought i'd ask ryan hadsell about
this
you can't believe everything you see on
the internet hannah
welfare you're absolutely right there
are robots that can do some pretty
impressive stuff that can flip that can
jump at deepmind we've been focusing
more on the generality aspect of it the
g in agi
we want robots that can learn new things
that they've never done before without
needing somebody to program them just
through experience or through watching a
human so those very impressive videos
the ones that aren't fake of robot zoo
backflips they are essentially
following a very precise
set of instructions is that essentially
what we're saying absolutely and they
tend to be a demonstration of what that
actual robot can do a robot that can do
a backflip that's very impressive
because of the power and mass ratio
that's required to do that but it's very
different from wanting that robot to
do a new skill that it has just observed
for the first time it couldn't walk over
to a table and pick up a coffee cup for
example it could not well you've
disappointed me right
but yours could in theory in future ours
could do that and
weed potatoes and pick tomatoes as well
this is the key point here if robots can
teach themselves to manipulate objects
and move around they can be adaptable
and offer assistance to humans in a
whole host of critical tasks
including situations where they can't
currently support us
so this came up when there was the
fukushima disaster in japan there's been
an explosion at a japanese nuclear power
station damaged in yesterday's massive
earthquake clouds of smoke could be seen
rising above the fukushima nuclear site
people realized that we didn't have a
good way
to
send robots into this extremely
dangerous radioactive area and make
repairs
because all of our robots either
required an area that was easily
accessible or didn't have the necessary
dexterity to for instance shut a valve
or open a door and so there was a whole
robotics program aimed at how do we
improve legged locomotion into areas
where a wheeled robot can't go and how
do we improve the dexterity of robots as
well
of course there is a flip side here if
in the future these artificially
intelligent robots are good enough to be
deployed in the real world for saving
human lives
they could also be built to do the
opposite
robots have been used to carry weapons
and so if you make a more capable robot
then potentially what you're making is a
more capable vehicle for holding weapons
of course deep mind is very much against
autonomous weaponry including on robots
and i think that the benefits of robots
and what they can do in our world
outweigh these risks especially if the
world stands strongly against the use of
weaponry and robotics
and this is not the only ethical concern
about robotics research
lots of people are worried about the
possible detrimental effects of
automation on the workforce
what we're looking at now with the use
of robots would be to augment humans
somebody working on a construction site
that has a robot next to them that's
able to do some of the heavy lifting for
instance so it's not about displacing
humans or replacing them it's about
enhancing what a human can do
any robot that's going to help with
weeding potatoes and picking tomatoes
will of course need to have mastered
locomotion
back at the deepmind robotics lab a
recent focus has been to develop a robot
which can move around on two legs a
problem which comes with its own unique
set of research challenges
on the floor we've got what looks sort
of like the play mat that you put down
for kids
akil showed me a sort of
robot play pen about nine meters squared
with a barrier around it presumably to
stop the robots inside from escaping so
inside this square then you've got three
humanoid robots they're black they've
got a very cuboid body but they have
arms and legs and even little tiny heads
they're quite small though i should tell
you that they're probably the size of a
large chicken
yeah smaller than a goose bigger than a
chicken i don't know
and basically what we've been doing is
learning to walk around and so like
robot actually learns to kind of use its
legs even its arms
the head has a camera so let's kind of
look around and see what's going on
so it is very much kind of almost like a
whole body control problem in some sense
can i touch it
oh my gosh okay
oh it's quite heavy it's got these
little handles on the back almost like a
rucksack and lots of ports like little
usb ports and an ethernet cable port and
stuff
and then for feet it's got these little
skid pads almost like it's going skiing
but just with really short skis it's
very pretty
so i'm lifting its arm up now and it
kind of returns to center but it's got
this really like smooth action have a
listen to that
i feel really sad sort of like oh please
leave me alone
okay it's walking around
imagine if you were doing a really
rubbish robot dance in a nightclub that
is exactly what it looks like it looks
like it should fall over
so you haven't programmed this to walk
around in a circle no this was learned
on the robot just by learning from the
data over a couple of days that's jan
huntlik a research scientist at deepmind
who's been following the progress of
these humanoid robots for more than a
year
did you teach it to fall flat on its
back like it just did no it just falls
it's quite good at pushing yourself up
though
yeah so those things are programmed the
pushing behavior to stand up that's
programmed because otherwise you just
spent your entire life picking up the
robot well either we need to pick them
up or they would need to learn to stand
up we are kind of humanizing them by
giving them names
the eureka protrochen is another
research scientist on the locomotion
project
what their names are these three i think
one of them is england and one is messi
from the messi the footballer and mine's
called
that's from humane deji or the hajj the
great romanian footballer
just because i'm from romania originally
so if you look at that one this is a
completely different training process
and you can see that the gate is very
different
and it can try to walk backwards and
it's actually
looks like it's a drunk robot so it's
trying to walk backwards but it's sort
of um
i must say i could have stood watching
these cute and mostly completely
hopeless little humanoid robots all day
but i wanted to find out more about the
process of training them to walk
so after waving goodbye to england messi
and co i asked jan and viarika about
their experience of training these
robots in their living rooms at home
when the pandemic hit
how did it travel does it just pop in a
little suitcase actually if you buy it
you get it with a suitcase it comes with
it so it can travel do you have quite a
big living room
not that big but yeah i'm adapting it i
have a pen there with floor mats and
foam walls so when you watch tv at night
time you sort of put your feet up and
around you is a little robot pen exactly
yeah we even had experiments where the
robot was watching tv did you really
well we wanted to run some experiments
to test
visual networks tv is a good source of
diverse visual data and it's already in
the living room right so why not
so hang on your job
for the last year has been still on the
sofa and watch tv with your robots not
quite that but
probably a few seconds of it it does
look like that yeah
so how do you train a humanoid robot to
walk again the underlying mechanism is
reinforcement learning the robots are
rewarded with points for forward
velocity and not falling over
when you haven't given them any training
what do they do oh they don't do much
they just start shaking for one second
or two at most and then they fall
after
training for a few hours then they start
actually walking like taking a few steps
and then later on they bump into walls
and then using vision they learn how to
avoid the walls so i i have a
two-year-old at home right and like the
way you're describing here it's not
dissimilar from the way that the
two-year-old has learned to walk there's
a lot of falling not that
shaking and flailing but there was also
sort much
a lot of walking into walls do you see
those similarities with the way that
these robots learn to walk in the way
that toddlers learn there are some
similarities where probably for toddlers
even before they crawl they still
discover their body they still learn to
move their limbs
whereas our robots we just put them in
standing position and now walk
and how quickly did it manage to learn
to walk i think in about 24 hours it was
already walking for me that's impressive
not 24 hours in real time but 24 hours
in sort of training time yeah yeah that
spans about a week of training but uh
training like a small uh
sessions before something breaks or
taking it to the lab for a quick repair
or something like that
the eureka raises an important point
about the fragility of these robots
the actual hardware is not designed for
a machine learning technique which
involves a robot falling down loads of
times before any progress is made
here's ryan hansel to explain
the robots that are built today are not
built for the type of learning paradigms
that we think is key to developing agi
think about when
a child learns to walk every time they
fall down
they then heal from that and they keep
on going
there's only so many times that a robot
can fall down before it simply breaks
this approach comes with all kinds of
difficulties and hurdles that the
pre-programmed robots just don't have to
worry about
here's jan humplik again
the main limitation is that you really
start from scratch with more classical
approaches you perhaps don't need any
data it's just going to work out of the
box
so these are
certainly disadvantages of reinforcement
can't you cheat though can't what one
robot has learned about the world be
imparted onto another absolutely and
there are many different ways to share
knowledge in particular you can just
have multiple robots collecting data and
this is really the way to scale up this
data collection process
what yarn is talking about here is a
technique called pooling instead of
and england learning to walk
independently of each other their data
how many times they fell over what their
sensor readings were when they fell etc
is regularly uploaded to a central
controller which combines this
information and feeds it back to each
robot so that they can better navigate
the world based on their combined
learning experience
we can track each robot how well they're
doing and yeah we definitely discuss
like oh okay my robot starts falling
more often now is yours the same did it
get quite competitive
i kept telling everybody that it's not a
competition
but yes every time somebody would cheat
the learning curve and there would be
the two robots they would be like oh
viorika is winning oh yanni is winning
i'm like no no
we're only winning if the performances
are the same on both robots
speaking of teamwork there are other
environments beyond just walking around
or inserting keys into locks or stacking
bricks that serve as an important test
project for the robots a chance to hone
in on a set of robot skills that would
be useful to have in the long term
for that in true deep mind fashion their
focus has turned to games and one in
particular
the beautiful game
in order to play football you have to be
able to control your body
you need to be able to run to walk
but then you also need to have these
skills of dribbling and shooting
and then at even a level above that you
need to have the coordination and the
strategy over the whole game so it's
really a challenge that has a lot of
layers to it
so far deepmind has been teaching
football not to real robots but to
simulated ones computerized avatars in
human form a bit like a simplified
version of the players in your favorite
video game
the difference with these players is
that their repertoire of movements is
not pre-programmed but like the real
robots they are effectively learning to
move from scratch the point is not to
have robots playing at wembley stadium
in the near future however fun that
might be
we're really trying to study whether
it's valuable to train these methods
using reward and the competition of
something like football or whether there
are other ways to train for this type of
behavior
underneath that big umbrella of
reinforcement learning it's helpful to
use a series of other techniques to get
the agents up and running
here they're also using something called
imitation learning which involves
gathering video footage from real human
football matches using motion capture to
translate the movements of each player's
joints into a dataset and then training
a neural network so that these simulated
humanoids begin to mimic the movements
of real players
so this is really layering these
different types of learning algorithms
together
and the exciting thing is that the
result in the end is four agents that
can race around this field and they've
really achieved this level of whole body
control and also
team coordination
and then raya showed me my first ever
video of a simulated humanoid football
match
and i'll do my best to commentate on the
highlights
so here we are for this season's title
decider between these two titans of ai
football the blues versus humanoids
united playing in red
well the game's begun and drogbot has it
for the blues cuts inside and then chops
back onto his right but look the ball is
broken free and robo naldo is clear
go
well they say a week is a long time in
politics but five seconds really is an
epoch in ai football
as impressive as this video is right i
think it's fair to say that at certain
points they are quite hilariously bad at
controlling their bodies
yeah i mean they are
not trying to win any points on style or
grace
so that really lets the agent optimize
for just purely trying to achieve at
school it doesn't matter if the arms are
flailing around so you can see the
problem with putting this onto a real
robot yeah i can see how that would be a
problem with the robots
it's in the game of football that we
start to see the different flavors of
intelligence converge
training agents to play football gives
them physical skills like dribbling and
passing but when combined with a
reinforcement learning algorithm which
rewards them for team play you start to
see emerging the sort of cooperative ai
we heard about in the previous episode
the question is if physical and social
intelligence can be developed in tandem
in this way could physical intelligence
provide a path all the way to agi
it all depends on how you define agi
doesn't it i've noticed this a lot yes
maybe not immediately you know if you
look at evolution it's a very long path
to go from initial creatures to human
beings then i think also it could be a
very long path if we want to
build an agi starting from first
principles of learning to move a body
but that is what we are looking at
jan humplik also believes that it will
be a long time before robotics takes us
to a general form of intelligence
if i ask somebody on the street what
would they be impressed by the robot
doing they would say something like well
you know maybe cleaning my apartment and
and if you start thinking about this
problem you're like okay so it certainly
needs to use vision it certainly needs
to understand human language because you
need to give it command
it needs to understand what does it mean
to clean the apartment and that's not
trivial because cleaning doesn't mean
destroying your furniture
solving anything impressive like this
essentially getting very close to agi
but if embodied intelligence social
intelligence and linguistic intelligence
don't necessarily lead to agi on their
own
is there a single path that does
well some deep mind researchers are
convinced that there is
and it's been staring us in the face
this whole time when we say that reward
is enough we're really arguing that all
of the abilities of intelligence
everything from
perception to knowledge to social
intelligence to language can be
understood
as a single process
of trying to increase the rewards that
that agent gets if this hypothesis was
true it would mean that we only need to
solve one problem in intelligence rather
than a thousand different problems for
each of the separate abilities
that's next time on the deepmind podcast
presented by me hannah fry and produced
by dan hardoon at whistle down
productions
if you like what you've heard please do
rate and review the podcast helps others
who are also ai curious to find it
same time next week


4 Tests Reveal Bing (GPT 4) ≈ 114 IQ

being AI passing 100 IQ might seem
intangible unimpressive or even
irrelevant and as someone who is lucky
enough to have got on a perfect score in
tests such as the GRE I can confirm that
traditional measures of IQ leave so much
of human talent unquantified but with
these caveats aside hints that Bing AI
may have crossed that 100 IQ threshold
are nevertheless stunning I will be
explaining four tests that show The 100
IQ moment may have arrived and thinking
about what each test means for all of us
this graph gives us a snapshot of the
state-of-the-art models and in blue is
Palm and palm is an unreleased model
from Google that I believe based on firm
research provided in another one of my
videos is comparable to being AI by the
way Google's chat bot Bard which is
going to be released soon will be based
on Lambda a less powerful model than
Palm but given that Palm is a proxy for
Bing AI you can see in this snapshot
that it has already passed the average
human in a set of difficult tasks called
the big bench I have multiple videos on
this task but IQ is notoriously
difficult to measure so what kind of
tests am I talking about well the
international high IQ Society publishes
numerous tests that they accept if
you're trying to join you need an IQ of
above 124 to join and the tests that
they accept are shown on the right and
the left and in what I believe is an
exclusive on YouTube I'm going to be
testing Bing AI on several of these
tests the first one is the GMAT and I
must confess a personal interest here as
a GMAT tutor it's The Graduate
management admissions test and I scored
a 780 in this and much like the GRE it
tests both verbal and quantitative
reasoning it's not a straightforward
test the official provider mba.com offer
a mini quiz and this is what bingai got
but what kind of questions were these
and where did Bing ai go wrong and also
what does this score mean in terms of IQ
that's what I'm about to show you side
by side I'm going to show you the
questions it got right and got wrong and
Bing's reasoning by the way I told being
explicitly do not use web sources for
your answer and Bing was very obedient
there were no links provided it wasn't
scouring the web and it provided
reasoning for each of its points it was
not cheating these are difficult
questions and I have spent the last
seven years of my life tutoring people
in them and smart people get these
questions wrong if you want to try the
questions feel free to pause and try
them yourself but this first one is
what's called an assumption question
where you have to ascertain what is the
hidden underlying Assumption of an
argument and Bing does really well and
gets it right it picks C and that is the
correct answer
the next question is a sentence
correction question where essentially
you have to improve the grammar of a
complex sentence you have to refine the
wording make it more succinct make it
read better and Bing does an excellent
job and gets this right it picks the
version of the sentence that reads the
best that is a really Advanced
linguistic ability what about the third
question there were eight questions
total well this is an interesting one
bing gets this wrong and I'm very
curious as to why you're presented with
a dense bit of text and what you have to
spot to get this question right is that
the US spent three percent of its GNP on
research and development in 1964 but
only 2.2 percent in 1978
whereas Japan increased its spending
during that period reaching a peak of
1.6 in 1978 and being AI isn't quite
able to deduce that therefore during
that period the US must have spent more
of its GMP as a percentage on R D than
Japan because Japan increased from an
unknown base up to 1.6 whereas we know
the U.S dropped as a percentage from 3
to 2.2 percent on research and
development so throughout that period
the US must have spent more as a
percentage Bing can't quite get his head
around that logic it just restates what
the passage says and says this is
contradicted without really giving a
reason why instead it says what we can
conclude is that the amount of money a
nation spends on R D is directly related
to the number of inventions patented in
that Nation but the text never makes
that relationship explicit this is a
difficult text Bing AI does get it wrong
its IQ isn't yet 141.50 as we'll see in
a second a score of 580 in the GMAT is
really quite impressive before we get to
the IQ number let's look at a few more
questions in question four it was
another census correction question and
being aced it
it's really good at grammar
question five was mathematics and what
happened to people saying that these
chat Bots are bad at math it crushed
this question pause it try it yourself
it's not super easy but there were many
smart students graduates this is the
GMAT after all who get this wrong we're
not just talking about average adults
here these are graduates taking this
test and 580 is an above average score
it gets its math problem completely
right maybe that was a fluke let's give
it another math problem we have to set
up two equations here and solve them
that's difficult it's one thing setting
up the equations translating the words
into algebra but then solving them
that's a lot of addition subtraction
division surely Bing AI isn't good at
that but wait it gets it right the rate
of progress here is insane again not
perfect as we're about to see but don't
listen to those people who say Bing AI
is necessarily bad at math as a math
tutor as a GMAT and GRE tutor it's not
it's already better than average final
two questions this one is data
sufficiency a notoriously confusing
question type for humans and AI
essentially you're given a question and
then you're given two statements to help
you answer it and you have to decide
whether one of the statements alone is
enough whether you need both of them or
whether even with both statements you
can't answer the question this is
supposed to be the hardest type of
question for large language models in
the big bench benchmarks most models
perform terribly at this but you can
guess what I'm about to say it got it
right it was able to tell me without
searching the web it didn't copy this
from anywhere this is its own reasoning
and it gets it right that's borderline
scary what was the other question it got
wrong well surprisingly this data
sufficiency question and the reason it
got it wrong was quite curious it
thought that 33 was a prime number
meaning it thought that 33 could not be
factored into two integers greater than
one even though it definitely can be
eleven times three it was kind of
surreal because it got this question
wrong at the exact same time that as you
can see something went wrong yes
something definitely did go wrong you
got the question wrong you might be
thinking that's all well and good how
does that translate to IQ and while
there aren't any direct GMAT score to IQ
conversion charts as you saw earlier
GMAT is accepted for high IQ societies
and using this approximate formula the
score average of 580 the NBA.com gives
would translate to an IQ of 114. now
just before you say that's just one test
you can't take such a small sample size
of eight questions and extrapolate an IQ
I'm going to show you three more tests
that back up this point the next test is
of reading age in the US it has been
assessed that the average American reads
at 7th to 8th grade level and remember
the average IQ is set at a hundred so
what age does Bing AI read and write at
there are ways of assessing this I got
Bing to write me a quick three paragraph
eloquent assessment on the nature of
modern day life and it gave me a nice
little essay say nice like it's
patronizing it's a very good little
essay now somewhat cheekily I did ask it
to improve and I said can you use more
complex and intriguing words this
response is a little Bland and I don't
think being AI liked that it said I'm
sorry I prefer not to continue this
conversation I guess I can accept that I
was a little bit rude but what happens
when you paste this answer into a
reading age calculator remember the
average person reads a seventh eighth
grade level and when you paste this
essay into a readability calculator you
get the following results and I know
these look a little confusing but let's
just focus on one of them the Gunning
fog index where the sa scored a 16.8
what does that mean from Wikipedia we
can see that a score of 16.8 on the
Dunning fog index indicates a reading
level of a college senior
just below that of a college graduate
and that fits with what I'm feeling I
used to teach this age group and where
it was said that chat gbt could output
an essay of the quality of a high school
senior being AI is a significant step
forward we're now talking about a
college senior and we're certainly
talking about a reading level
significantly beyond that which the
average American can read and write at
so far you might be thinking but I
haven't ever directly given an IQ test
and you can't fully do that because
there are some visual elements to
traditional IQ tests the Bing can't
complete but what score does it get if
we give it such a test and just get all
those visual or spatial reasoning
questions wrong it can still get an IQ
score of between 105 to 120 on these
classic IQ tests now I know you can poke
holes in these tests there are sometimes
cultural biases Etc but as an approach
indicator an IQ score of between 105 and
120 even as a rough proxy that's
impressive what does it get right well
as we've seen language kind of questions
even these more advanced mathematical
reasoning questions it's got to predict
the pattern this took me 30 seconds to
spot now when we move on to figures I
just clicked a wrong answer by the way
as I'm going to talk about in a video
coming up this kind of visual reasoning
image to text if you will is coming soon
and I will make another video the moment
it does because I would expect its IQ
result to go up even more what else does
it get right syllogisms these are kind
of logic puzzles chat DBT gets this
wrong being AI gets it right this is
spatial reasoning so I inputted an
incorrect answer then we have
calculation and it actually gets his
wrong I was kind of expecting it to get
it right and when I tried the same
question three or four times once it did
get it right but for now I'm going to
leave it as incorrect antonym and
opposite word it was able to understand
that context and analogies as we'll see
it did extremely well at analogies and
of course meanings for the final
question again I inputted an incorrect
answer for the fourth and final test
we're going to use a metric that is
famous among high IQ societies the
Miller analogies test the Prometheus
society which is one of the highest IQ
Societies in existence only allowed for
the 99.99 seventh percentile IQ this
Society actually only accepts The
Miller's analogy test as of 2004 that is
the only test that they're currently
allowing and while there are dozens of
online providers for these mat tests I
went straight to the official Source
just like I did with GMAT this is
Pearson the huge exam company and they
give 10 questions representative of
those type found in the full version of
the test I couldn't give it all 1120 the
items because as I've talked about in
one of my recent videos there is a 15
message limit daily currently but I
could give it these 10 sample questions
and extrapolate a result based on those
10. and what I found absolutely
incredible
is I didn't break down this colon
structure of the question you're
supposed to draw an analogy but the
missing answer comes at different points
in different questions and that is a
complex test of intelligence itself
you've got to try and deduce What
analogy you're even drawing between
which two items and I didn't give being
any help all I said was complete this
analogy without using web sources I
didn't explain the rules of the test
what type of analogies it would get or
the meaning of these colons and double
colons and it wasn't just drawing
answers from the web I checked this is
its own logic it does sometimes get it
wrong but look how many times it gets it
right of course you can pause the video
and try to answer these 10 questions
yourself if you like but to give you an
idea in this first question what the mat
is testing is shape right Springs come
as a set of rings coils come as a set of
Loops Now Bing stretches it a bit with
the reasoning talking about the letters
in the name but it gets that circular
shape right then a mathematical kind of
question these analogies aren't anything
they could be historical analogies
mathematical scientific ones linguistic
ones Bing can do almost all of them here
was a mathematical one and you had to
draw the analogy between one angle being
obtuse one angle being acute here was
one that I couldn't do and it's testing
if you realize that a mollusk produces
pearls while a mammal produces ambergris
I don't even know what that is I could
get this one it's Advanced vocab about
epistemology being about knowledge
whereas ontology is about being but I'll
be honest it crushed me I think I would
have gotten about seven of these
questions right being AI gets nine of
them right and the one it got wrong
honestly I read its explanation for why
the missing answer for question five
would be lever and it makes some sense
let me know in the comments what you
think but I think there's an argument
that Bing wasn't even wrong about this
either way I don't have to go through
every answer but you can see the
in-depth reasoning that Bing gives based
on the percentage correct I converted
from a raw score to a scaled score of
course the sample size is isn't big
enough and this is not a perfect metric
but while that 498 wouldn't quite get
them into Prometheus society which
remember is a 99th point nine nine
seventh percentile high IQ Society it
would put them way off to the right on
this bell curve of scaled scores but
let's bring it all back to the start and
discuss the meaning there are so many
takeaways of course being AI makes
mistakes and sometimes seems stupid but
so do I and I scored perfectly on some
of these tests I think artificial
intelligence passing that 100 IQ
threshold is worthy of more headlines
than it's currently getting it is very
fun to focus on the mistakes that Bing
AI makes and the humorous ways it can
sometimes go wrong the real headline is
this it is starting to pass the average
human in intelligence image recognition
and visual reasoning is coming soon for
purposes of brevity I didn't even
include a creative writing task in which
I think for the first time I've been
genuinely awestruck with the quality of
writing generated by a GPT model this
was prompted by Ethan molick by the way
one of the implications I think at least
for the short to medium term is that
there will be soon a premium on those
who can write better than Bing AI
because being AI is going to increase
the average writing quality of everyone
who has access to it so those who still
have the skill to write better than Bing
and that's a number that's dwindling
should have an incredible premium on
their work there are so many other
takeaways IQ is fundamentally a human
metric designed to test human abilities
speed is unaccounted for in all of these
IQ metrics an alien looking down may
decide that Bing AI is already smarter
than us it's generating these essays
taking these tests in fractions of a
second sometimes or a few seconds in
other times even me who might currently
be able to score better than air I need
the full time allowance I need the 60
minutes for the mat and the two hours
for the GMAT Bing needs two minutes at
best and what about the fact that some
of these IQ tests are designed for
certain cultures well that's not a
problem for being AI either being can do
all of this in dozens if not soon
hundreds of languages that's not
accounted for in these IQ scores the
truth is that AGI has many definitions
but in one of the original definitions
it was the point at which an AI is
better than the average human at a range
of tasks and in some senses that moment
may have happened in the dead of night
without headlines even for those of us
like me who argue it's not quite there
that moment is going to happen fairly
soon quietly on a Thursday night in some
Google Data Center and not enough people
are talking about it let me know what
you think in the comments and have a
wonderful day