Demis Hassabis: The interview - DeepMind: The Podcast (S1, Ep8)

[Music]
so here we are the last episode in this
series of the deep mind podcast my name
is Hanna Frey I am a mathematician and
someone who is deeply intrigued by
artificial intelligence much like you I
imagine since you made it this far now
we've been toying with the big questions
in this series what is intelligence how
does an algorithm learn and what to do
with the AI future once we get there and
I've been asking the team of scientists
and engineers here at deep mind to give
us their take of where things are at and
where they're going but now for this
final episode we have a chance to catch
up with demis hassabis the co-founder
and CEO of deepmind to hear what he has
to say on these questions and more demos
SRBs grew up in North London in the
1970s by the age of 13 he was ranked
second in the world at chess in his age
group at 16 he worked as a games
designer remember theme park that was
him then he went on to study computer
science and then neuroscience before
setting up deep mind with his two
co-founders Shane Lake and Mustafa
Suleiman his accomplishments are
ferociously intimidating but as an
article in The Times put it Dennis
doesn't even have the good grace to be
socially deficient but if all of that
makes it sound like Dimas is a man with
a plan then you'd be right I had in mind
something like creating a company like
deep mind to research AI from a long
time ago so I was sort of working back
from the end state which is what would I
need what skills would I need what
experiences do I need to even stand a
chance of building something like that
because these different aspects of your
life the chest the neuroscience the
games designed they're not disconnected
I mean they do build to a bigger picture
they do and it's hard to say which way
round it is so I picked those subjects
and those things to study say computer
science at Cambridge and then cognitive
neuroscience at UCL because I wanted
this component of computer science and
neuroscience to come together and
obviously that was what we do at deep
mind but even the game staff that taught
me about creative thinking
also a massive engineering projects and
then of course it's ended up that we've
used games as our main vehicle for
proving out our AI algorithms one other
thing I've learned from games is to use
every scrap of asset that you have luck
in games you always have a limited
resource pool like you know if it's a
chess game it's the chess pieces you
have left on the board and one way to
think about games is maximizing the use
of the of the assets you have left
perhaps that's why I was biased towards
using games but I also felt it was the
logical way to go about building AI what
was your PhD in my PhD was in cognitive
neuroscience and I actually decided to
study how memory and imagination works
in the brain and the reason I chose
cognitive neuroscience is I wanted to
better understand how the brain does
certain cognitive functions so that
perhaps we could be inspired on new
types of algorithms based on how the
brain works and so it's a good idea to
pick functions that we don't know how to
do in AI and I went to study with Elinor
McGuire at UCL and she's one of the
world's leading experts in the
hippocampus which is critical for memory
but I told her that what I really wanted
to look at was imagination which is you
can also think of it as simulating
things in the future in your mind for
obviously useful for planning but also
for creativity and the reason I was
interested in that is of course that's
an incredibly important part of human
intelligence and it's also something I
used a lot in my games design career so
I used a lot of visualization techniques
and imagining how would a player
viscerally like play this game like
theme park and then you try and change
something about it all in your mind
before or in Skechers before you went to
the trouble of programming all and it
felt to me that we were using a similar
type of process to the way when we
lucidly remember things that happened to
us so I thought there may be there would
be a connection with this kind of you
could imagine like a simulation engine
of the mind that was being used both for
imagination and memory and that's what I
wanted to work on during my PhD and we
ended up discovering something quite
important that then in fact the
hippocampus was at the core
both of those two types of function is
critical for memory which we already
know but it's also critical for
imagination and you can't really imagine
vividly without your hippocampus so we
ended up discovering this important
thing and then subsequently that's been
at the heart of a lot of what we've
tried to do in AI is build memory and
imagination abilities into our AI
systems and we're still doing that now
when it comes to bringing those ideas
across and trying to implement them in
AI where do you find that balance
between just directly copying what the
brain is doing and using it for
inspiration so that's a very important
signpost when you're scrabbling around
in the dark of in the unknown of science
any signpost is really valuable and the
brain is the only existence proof we
have in the universe that intelligence
is possible so it always felt to me that
it would be crazy to ignore that as a
source of information of how to build AI
so we use neuroscience for two things
one is inspiration for new ideas about
algorithms or architectures or
representations that the brain uses and
then we can get some inspiration for
that for new types of algorithms the
second way we use neuroscience is what I
call for validation so we may already
have some idea from coming from
engineering or mathematics about how to
build a learning system let's say
reinforcement learning that came from
engineering disciplines and operational
research first but then in the 90s we
discovered that the brain also
implements a form of reinforcement
learning and what that means is is that
from an AI perspective you can be sure
that reinforcement learning could
plausibly be a component part of an AI
system because it's in the brain and we
know the brain is a general intelligence
so that means that's really important if
you're thinking about where to put your
engineering resources and effort you
know that if it doesn't work right now
and things never work first time in
research or engineering it's worthwhile
pushing that harder because you know
eventually this must work because the
proof of concept is the brain having
said that though there are another
school of thought from AI practitioners
and neuroscientists that we need to
slavish ly copy the brain completely
from the bottom up like on a neuronal
level and I
think that's also the wrong approach
what we're after is what I call a
systems neuroscience approach which is
that you're in stood in the algorithms
and the architectures that the brain is
using not necessarily the exact
implementation details because I think
that's likely to be different for in
silico systems like computers compared
to carbon based systems like our minds
there's no reason to think that we would
implement exactly the same
implementation details in a
silicon-based system that's going to
have different strengths and weaknesses
to a carbon-based system like our minds
[Music]
normally tech startups have customers
they have products but this is sort of
more like a start-up research facility
yes how do you get something like that
off the ground it's pretty hard I mean
you're right in that it's a very unusual
company what I try to do is basically
take the best from the startup world the
kind of focus and energy and pace that
you get in the best startups say in
Silicon Valley and I wanted to combine
that with the best form academia which
is you know blue sky thinking incredibly
bright people working on you know
long-term big research questions and
stepping into the unknown all the time
and you know obviously I spend some time
in academia myself and there's very
great aspects about academia but there's
also some things that are frustrating
mostly around the organizational aspects
and the pace of it can sometimes be
slower than you would like it's
difficult to get momentum behind things
in the way you can a startup if it's you
know if things are going well but I when
I was in academia and I had already
obviously started and been involved with
a few startups before going back to do
my PhD so I'd experienced both sides and
I didn't feel like there was any reason
why these should be mutually exclusive
environments although they have
generally been treated as very different
almost opposite environments and there's
a lot of things that do seem opposite
but I felt if if you were kind of smart
about it you could extract the best of
both those worlds and combine them into
some kind of hybrid organization and I
feel that's what the mind is and I don't
think many people have ever done that
and and so that's why it seems quite
strange probably as a
zatia n-- i think we've shown with our
scientific output even if you measure it
by normal measures nature science papers
this kind of thing that normal academic
labs would measure themselves by we've
been very successful and i think you
know we've a bee net also on the other
side of things we've been able to
produce really big breakthroughs that
took a lot of engineering effort as well
like alphago which would have been very
difficult i think to do in academia in a
normal you know small academic lab one
other thing that makes deep mind i guess
a bit unusual is that you publish your
work I mean other companies don't really
do this is there somewhere that you're
kind of giving away your competitive
advantage really yes an interesting
issue actually so we've always published
everything we've done and lots of
companies do want to you know why we're
doing that because a lot of other
companies don't necessarily do that we
feel it's part of the scientific
discourse like that's the right way to
do science we really believe in sort of
peer-reviewed journals so that your work
is scrutinized at the highest level by
your peers which is the bina the gold
standard in science that's why we
publish in the top journals like Nature
and Science
also you do get more exposure for your
ideas like that so some of our top
papers have been cited like more than
5,000 times now in the last two three
years some of the most cited papers in
the world so that's great you know and I
think if a community shares ideas like
that the whole field can advance much
more quickly than if everyone was to
keep their ideas secret but you know
there are some interesting aspects to
that one is the competitive side I mean
I feel there that you just need to carry
on innovating at a faster pace than
anyone else that's the most important
thing not trying to keep hold of the
ideas you've already innovated you
should just be by the time you've you
know you've published it you should be
one or two idea is even further down the
line if you're continuing to work at the
same pace and the same level of
innovation so I think that's really the
biggest sort of protection against
competitors is your pace of innovation
in the early days I mean back in sort of
you know 2009-2010 when AI wasn't the
hot topic that perhaps is today did you
find it difficult to get attention from
from the people master I mean I think I
remember reading that you you decided to
go straight for billionaires rather than
millionaires yeah
to investment yes well that was
incredibly tough so it's really hard to
remember now even for me like ten years
ago you know no one was investing in AI
it was a impossible thing to get money
for actually you know no one would
invest and it's still very difficult
today I think on it at what I would call
a deep technology or science based
startup right and with no clear product
in mind I mean what we were basically
saying is we were going to build this
incredible general purpose technology
that as it got more powerful there
should be a myriad of things you could
just apply it to but you know that
sounds probably pretty far-fetched to a
normal kind of investor it was almost
like well that's what academia is for
isn't it you should be just if you don't
know you know it's blue sky research we
don't know when it's going to work it's
it's pure research then go and do that
for another ten years in academia and
then come and talk to them when it's
working but that would have been too
slow and I could see that within
academia so then that's why I decided
not to go after normal venture
capitalist who certainly in Europe and
in the UK you know they would want to
make 10x return maybe within three to
five years right that's the kind of time
horizon of course that's no good for a
research based company you know you have
barely got going after three years right
so what you need is a profile investor
who is more interested in thousand X
returned but they're willing to wait ten
years maybe even twenty years and that
kind of profile of an investor basically
just does not exist in Europe certainly
did in the back in 2010 really you're
talking about
Silicon Valley self-made billionaires I
guess who both have deep enough pockets
to take that kind of bet and if it
doesn't work it's okay but they're also
personally interested in these types of
topics and have seen incredibly
ambitious things work because usually
that's how they've made their money just
going back to the idea of how everything
sort of slots into place all the
different aspects all the different
pressures that you have slots in space
didn't chess play a role in you getting
your first funding yes it hits so chess
has been key you know core part of my
personality I guess it because I've been
playing it for so long and I think a lot
of my thought processes developed
because of that so planning and thinking
about problem solving all of these
aspects which I think
for any thing you do in your life but it
also turned out to be useful directly
because one of the first investors we
talked to was a chess player themselves
and a pretty strong junior chess player
in the US and when we were doing sort of
our background reading on this we spent
about a year preparing for this meeting
almost and it was important meeting
because we knew that not very many
people would get what we were doing and
this was one of the people we felt that
would so it was important and we
couldn't get a meeting because we had no
contacts in Silicon Valley you know I
didn't know anyone over there in
California nor did Shane and Mustafa you
know and it's sort of how do you break
into that world and so we finally
managed to get asked to a conference
where this billionaire was sponsoring
and then we knew we would meet him at
some kind of after conference party but
the problem is is that there are
hundreds of people all trying to pitch
him their ideas so if you if you're just
another one pitching you're another
crazy idea it's very unlikely you're
gonna get noticed right so I thought
instead of that I'll take this
calculated risk and talk to him about
chess instead but then you have to have
something interesting to say about chess
that maybe haven't thought of so I used
my number one fact on chess that even
surprises grandmasters which is the
thinking about it from a games designer
point of view you know why is chess such
a great game how did it evolve into such
a great game and what is it that makes
it so great and my belief is that it's
actually because of the creative tension
of the bishop a knight the bishop and I
are basically worth the same there three
points each so you can swap them off but
they have completely different powers
and that asymmetry the sort of creative
asymmetry that happens with the bishop
and knights being swapped into various
positions I think is what makes chess a
fascinating game and so I basically
pretty much led that with that line I
don't know how I managed to crowbar that
into a drinks party but I did and it
made him sort of stop and think which is
exactly what I was hoping and then he
invited us back the next day to do a
proper picture on our business idea so
we actually got half an hour with him
rather than one minute over over some
drinks so that actually you know worked
out so you could say chess works on two
levels in the meta level of planning for
that and also actually chesses and
tweaking subjects in itself so actually
getting funding from billionaires is
really
spent the year studying their interests
come up with a genius idea that will
catch their attention in a way you can
do a good picture after that exactly
easy simple very straightforward one
thing that you hear time and time again
in this building and actually throughout
this podcast series is how deep my wants
to use artificial intelligence to solve
everything so here is the chance to ask
what do they actually mean by that do
they want to be the ones addressing
every one of the world's problems once
intelligence is cracked so I you know I
have a list it's a working list of a
dozen to a couple of dozen of scientific
problems that I feel are these kinds of
root node problems and if we could crack
all of those then I feel like that would
transform society for the better and
open up all sorts of areas in you know
medicine and science for us to make
breakthroughs in go and give me the list
some of the lists of things like I think
a key thing we need to crack is cheap
abundant energy that's renewable and
clean and so if there's fusion or just
way better solar panels with way better
batteries with room-temperature
superconductors that would also solve
that problem so you know I think there's
a number of solutions to that some a
material science solution some of
physics solutions and we should have a
go cracking all of those but if you
crack that then that would open up all
sorts of new issues so I'll give you an
example
water access access to clean water is
going to become increasingly more
important as the population in the world
grows right and it's already becoming in
some countries more valuable than oil
because you know there's just so little
fresh clean water around right for a lot
of communities so they specially poor
communities this incredible problem but
we have a solution already its
desalination right 70 percent of the
earth is water but it's saltwater so how
do we deal with that well desalination
technologies exist the problem is they
cost too much energy they're too costly
so some rich countries can do it so I
think Israel gets a lot of their water
like this and some other countries like
that but for the poor countries it's too
expensive so if you solved the renewable
a clean-energy problem then you would
automatically solve the water access
problem almost straight away because
that's actually the issue so where do
you want to be when the water salination
problem is solved where do you fit into
that story I hope that we will have been
integral in coming up with those
solutions by doing something say in
fusion or in material science more
likely where we've come up with using an
alpha zero like system you know the
battery that is 50 percent more
efficient and cost a tenth of the price
of current batteries and lasts you know
ten times longer you know we would come
up with a solar panel photovoltaics
material that is twice as efficient at
converting heat energy into electrical
energy so it will be something like that
would then unlock the possibility of
making desalination within reach of
every community perhaps there has to be
some improvement with the desalination
technology itself as well and maybe we
could be involved in that but you know
we're relatively small company so and
we're going to stay relatively small so
we have to be efficient with the
solutions that we work on this all just
comes at least from my perspective from
rationally logically thinking out what's
the best thing you can do and what's
happened so far looking at civilization
in this I mean maybe you could say as a
slightly strange way of looking at it
but I think it's the correct way to look
at things and I think most people just
don't think about questions in the right
way and maybe that's what I've done in
my whole life is try to ask for the
right questions and I feel like this is
the obvious answer this is deep mind the
podcast an introduction to AI one of the
most fascinating fields in science today
okay I always say to people whatever
your question the answer is AI because
it sort of is in the limit right I mean
that's a little bit flippantly but I
mean in the limit it must be because the
answer so far you know why we're here
while we're talking while using these
Khmer zhing computers and devices is
because of intelligence human
intelligence and I think it's miraculous
and the scientific method another
miraculous thing I think that the
greatest discovery of all is that the
scientific method works you know the
Enlightenment and why should it be I
mean also you must have to question
things like that why should the universe
work like that that the scientific
method works you know it could be a
little bit more random then he'll be
really confusing right if sometimes the
Sun rose and sometimes it didn't
requires to do science then why
sometimes he repeated the experiment
with exactly same conditions something
else something something different
happened but it doesn't this world
doesn't seem to work like that it seems
to be repeatable it seems to be
consistent so therefore knowledge is
possible and incredibly strangely our
brains even though they're evolved
hunter-gathering can somehow deal with
it
which is kind of Iraq Yunus in itself so
how can you not want to a work on those
questions and B why would there be
limits to what that is capable of doing
but the ultimate goal in all of this is
to create artificial general
intelligence exactly what is meant by
that
nothing yeah artoo there's not an agreed
definition of artificial general
intelligence but the way that Shane and
I think about artificial general
intelligence is a system that is capable
of a wide range of tasks and if we think
about human level artificial
intelligence then we're talking about
system that can pretty much do the full
spectrum of cognitive tasks that humans
can at least as good as humans are able
to do that's you know one reasonable
definition of artificial general
intelligence what's the threshold for
atid how will you know when you're done
that's a philosophical issue of like how
do we know we're done with building a GI
certainly for me I'm waiting to see a
lot of key moments for example I think a
really big moment will be when an AI
system comes up with a new scientific
discovery that's of Nobel prize-winning
level
that to me would be a big watershed
moment and I think an important step in
the capabilities of these systems so you
know capable some kind of true
creativity in some sense I think other
big points will be when it can use
language and converse with us in a
naturalistic way it's capable of
learning abstract concepts these are all
things that I think a high-level
cognitive abilities that we're nowhere
near yet and I think it'll be big so I'm
posts on the way when were you convinced
that all of this was possible well I've
had this in mind since my early teens I
probably read way too much so far I'm
guessing some were really formative
things on me were Asimov's foundation
series so interestingly not the robotics
books I haven't really read any of his
robot books but the foundation series
was this really amazing series of sci-fi
novels and then Ian banks has culture
series which is the sort of space opera
about how the universe would look after
humanity's bill AI and coexist with it
and then a really big scientific book
for me was when I was writing theme park
obviously I was working on AI and
building AI for the game but I was also
reading books like girdle or bark by
Hofstadter which I suppose is more of a
philosophy book but it's an incredible
piece of work tying together girdle's
incompleteness theorem about mathematics
with Asher's drawings and bass fugues
and showing that they're all related in
some way this repeating cycle of pattern
is infinite patterns that they they all
exhibit and then he tied it to
consciousness and intelligence and it
was just really inspiring for me and
made me think about these deep questions
and I was discussing this with a lot of
my friends who were you know we're
writing games together we were doing
that 24/7 and we would discuss these
things about what the limits of AI could
be if we could not just use it for what
we were doing in games but actually
advance it to the level where it become
the same level as human and I just felt
like the sky was the limit
I mean maybe another way I can put it is
if you look around us today you know you
look at modern civilization it's
incredible well what built modern
civilization
intelligence did right that's what
building human intelligence and if you
were to take us back to our
hunter-gatherer days 1020 thousand years
ago 30,000 years ago and you were to say
one day we're gonna build Manhattan and
then fly over from London to Manhattan
and on a 747 regularly above the clouds
I mean what would you have said that it
would be mind-boggling right
and yet humanity's done that incredibly
and I don't think we stopped to think
how amazing that is enough because the
other thing about the human brain is is
incredibly adaptable right as soon as
something you know we do something and
then it becomes kind of boring a Monday
and then it's trivial right but I always
think about that one I'm taking a
transatlantic flight about how have we
with our monkey brains managed to come
up with these types of technologies it's
unbelievable
100 ton of metal flying through the sky
above the clouds so reliably and so if
you think about that then if we now
build something like AG iron in and
enhance our own capabilities with this
amazing tool then I feel like almost
anything might be possible within the
laws of physics and perhaps even beyond
the laws of physics because with AI we
might discover more about the laws of
physics or some holes or flaws in our
understanding of the laws of physics so
if you think about that and you
extrapolate it a few hundred more years
with these kinds of technologies like
AGI around and then what we might be
able to build with AGI I think it could
be truly incredible where we'll be and I
feel it will be something like with the
realization of the true potential of
humanity
but however exciting the grand ambition
of AGI is there is also a need to
proceed with caution we're cognizant of
some of the technical questions around
AGI making sure that they do exactly
what we want how do we program in our
values how do we specify our goals so
these are all the theoretical and
technical questions around AGI that
people like Nick Bostrom are worried
about and so we and the Schoen leads our
safety team that works on a lot of these
questions from a research and technical
perspective and I think there's a lot
more work that needs to be done there
and I think that's what we're going to
see over the next decade or two but I
mean even if you proceed with caution
even if you act as safely as you
possibly can based on the information
that you have in front of you at that
moment you can't really mitigate against
bad actors you can't stop someone else
coming in and mucking up for everyone or
can you I think there are ways of
minimizing that I think we'll have to
think carefully about that like at the
moment we are at the stage where these
systems are still quite nascent they can
do impressive things like play go but
they are not yet properly
general-purpose or you know you couldn't
use them for anything very dangerous or
in the real world so we've committed
that we won't do certain applications
you know obviously like things like
military and surveillance so which we
think would be bad for society we don't
think AI should be applied to those
things and we certainly wouldn't do
ourselves but if you publish some great
algorithms you have to think about the
indirect impact of that if other people
bad actors and so on around the world
potentially use your algorithms for
things that you would not have agreed
with we have to use this time now to
think about what principles need to be
put in place whether that is carefully
thought-out regulation whether that is
technical solutions to these problems
mathematical proofs more engineering
solutions so like one of the projects we
have here we call it the virtual brain
analytics project and it's inspired by
what we do in neuroscience with fMRI
machines - brain scan people
while they're doing tasks and see what
parts of the brain light up so we can
understand what the brain is doing
better and we should be doing the
equivalent of that with our virtual
brains our artificial neural networks so
I think there's both sort of behavioral
there's experimental understanding and
them as mathematical understanding of
these systems and we should do all of
those and what I'm hoping is that we'll
eventually in the next few years have a
much better understanding of these
systems than we have today and we may
even have some mathematical proofs about
what if you want to limit a system in a
certain way what you need to do what
components do you require also you know
we have to think about publications and
other stuff that we talked about earlier
whether this free exchange of of
information and knowledge is okay even
under situations where there are
potentially dangerous applications and
where I would take our lead from this is
not the first time again this has
happened you know this has happened a
lot in biology with designing synthetic
biology and viruses embryology now with
CRISPR so boa G actually has a
long-standing multi-decade experience of
coming together with wider society and
the scientists and and regulatory bodies
and figuring out what are the rules of
the road that it's safe for everybody
and I think that's the kind of
coordination we're going to have to have
over the next decade where do you see
deep minder city in terms of the broader
tech industry in terms of ethics and
safety are people following your lead
we're only one company but we are the
biggest probably group anywhere in the
world and we you know are definitely one
of the world leaders and acknowledged as
so so that gives us a powerful platform
to set an example we can create ishutin
upon AI which is a cross industry
initiative to talk about some of these
issues in products and we've also
sponsored a lot of academic groups we've
given the money for postdocs and the
things arms-length to study this we have
close contacts with a lot of the
Institute's that work on these things
like the future of humanity institutes
just down the road Oxford we talk with
them all the time and we ourselves as
leaders at deep mind we've always talked
about ethics and I've always had that at
mine and the reason we have from the
beginning of deep mind is that we plan
for success so if we're
earning all these ambitious things for
AI to do then we also need to think
carefully about what would that mean and
so we're doing a lot of things behind
the scenes and I think by sort of being
an example that will influence everybody
else I'm hoping given that we're in the
lead technologically so for the moment
that's why I think also it's very
important we have a technical lead
because why would anyone listen to what
you have to say on the ethics one if
you're not one of the leaders
technologically right then you could
just be you know any group saying that
so I think to have a seat at the table
whether that's a country or a company or
even individual you need for credibility
purposes you have to be close to the
forefront of the technical technical
side itself where does the public fit
into all of this do you need to have the
trust in you as a technology company or
do they need to be on board at all can
you kind of do it without no it's really
critical that this is discussed at large
with society in everybody and the
general public and I think they need to
engage I mean the poem is is that a lot
of these technical things are very
complex and you need PhD to understand
them and so on but some of the
fundamentals are quite easy to
understand and what you really need to
understand is the consequences of it and
then society has to decide as I said
before about how these things get used
and how the benefits accrue to different
people and society and make sure that
that is fair and I think that's the key
thing is to engage with the public now
and try and educate them about some of
the complexities of the technology but
also the implications misses partly what
things like this podcast series is about
but also we've done a lot of public
engagement with the Royal Society the UN
day.i
series that we did last year and I think
a lot more of that has to be done are
you optimistic about the future yeah I'm
very optimistic about the future but the
reason I'm optimistic is because I think
AI is coming down the road and I feel
like if we build it in the right way and
we deploy it in the right way for the
benefit of everyone I think it's going
to be the most amazing transformative
technology that humanities ever invented
and I would be quite pessimistic about
some of the problems that we are facing
as a society like climate change and
sustainability or inequality I think
these are going to get exacerbated in
the next few
years and I'd be pessimistic about our
ability to solve that if there wasn't
something like AI on the way in the near
future I think in order to solve some of
these big problems challenges society
has we either need an exponential
improvement in human behavior more
cooperation less selfishness more
collaboration or we need an exponential
improvement in technology and
unfortunately the way politics is going
right now I don't really see much
evidence of the former and you know we
don't seem to be able to get our act
together
globally to do something about climate
anywhere near fast enough to deal with
the problem and so I think we have to
double down on a technical solution at
least we should try both but I think we
need some technology bets it's hard not
to be inspired by Demeter's insatiable
thirst for knowledge and find yourself
drawn in by his positive view of the
future because if I'm honest I'm
normally quite skeptical but I have a
lot of time for overly optimistic
marketing talk and one of my favorite
hobbies is to roll my eyes at
self-titled futurists at tech
conferences but you know over the last
12 months I've spent hanging around here
I've come to the conclusion that there
really is something quite special going
on at the cutting edge of AI after 50
years of quite slow progress it really
feels as though the field is finally
beginning to deliver the problems that
everyone thought were completely out of
reach only a few short years ago are
tumbling one by one and the science is
moving forward at a blistering pace both
here and at research labs all around the
world and the questions that people are
working on are profound and important
including some of the potential pitfalls
and ethical concerns of this kind of
technology so with all of that in mind I
think I'm going to join in demouth in
being optimistic about the future and
the potential of AI to be a real
positive force for good but don't just
take my word for it as we've said
throughout this podcast we hope to in
are you on your own a I'd Ernie maybe
even by finding the answers to some of
the biggest questions there are
I'm not intrapreneur a second I'm a
scientist first it's just that this was
the right vehicle to make this happen
and it seems to have worn out but if I
could have made it happen in academia or
just done it in academia it just wasn't
possible under the constraints academia
has and that's why we also sold the
company to Google because it was about
what accelerates the mission and the
science and ultimately that's what I
want to do if my life is I want to
understand what's going on here in the
universe both inside here in the brain
and externally out there in the universe
and I guess that's what's always driven
me is this deep desire to understand
what seems to me be incredibly
interesting and fascinating mysteries
that are going all around us and I don't
really understand why more people don't
think about that all the time I can
barely sleep because I'm just fascinated
and also troubled by the things around
us that we seemingly don't understand
all the big questions you know the
meaning of life have the universe star
what is consciousness all these
questions which I feel like a blaring
klaxon in my mind that I would like to
understand and my attempt at doing that
is to build AI first
if you would like to find out more about
some of the things that we've talked
about in this episode or explore the
world of AI research beyond deep mind
you'll find plenty of useful links in
the show notes for each episode and if
there are stories or a sources that you
think other listeners would find helpful
then let us know you can message us on
Twitter or email the team at podcast
that deepmind comm you can also use that
address to send us your questions or
feedback on the series deep mind the
podcast has been a whistle down
production binaural sound recordings
were by Lucinda Mason Brown the music
from the series has been specially
composed by Elena Shaw
producers were Amy racks and Dan Hardin
the senior producer was Louisa Field and
the series editor
was David pressed I'm Hanna Frey thank
you for listening


Michael Wellman – Autonomous Agents in Financial Markets: Implications and Risks – CSRBAI 2016

good morning and welcome back to believe
a program series on the bus and
beneficial article intelligence so stood
by the pink machine intelligence
research institute and featured man the
Institute I'm Patrick Fleming travel
research fellow here and our first
speaker today is in this in this fourth
week on multi-agent phenomena and an
agent modeling our first speaker that
today is Michael Wellman professor of
computer science and engineering at the
University of Michigan michael has has
been active both in the research chain
in the end in industry in in cases of
looking at probabilistic modeling and
game theoretic decision theoretic things
and especially on the implications for
financial markets of autonomous agents
and so he's here to talk to us today
about the wealth about the implications
for financial markets of autonomous
agents so join me in welcoming Michael
Wellman thanks Patrick and thanks to
miri for hosting this colloquium and for
inviting me it's really pleasure to be
here thank you all for coming I'm going
to talk today about autonomous agents in
finance especially finance financial
markets the theme of the cloak Liam
robust the beneficial AI can be studied
at different levels and a lot of the
work that I think comes out of Miri and
will also be your feature here is
generic AI and generic techniques for
making AI more beneficial and robust I
think there's a lot of great work that
can be done at that level and ultimately
that's probably the most important level
to operate on things the other level is
looking at specific case studies in
particular domains much more specialized
work and the solutions one gets from
that may be a shorter term and less
widely applicable but on the other hand
they also made by looking very hard at a
specific domain one may get
ideas that an experience that will help
generalize things and moreover we're
also interested in different time scales
for robust than beneficial AI and the
long-term timescale of dealing with
super intelligence is tends to be pretty
controversial and many there are many
differing opinions about when we have to
worry about that whereas dealing with
autonomous agents in specific domains is
something that is at hand and we have to
deal with it now and so you can argue
that lets pretty specific domains
because you may learn something that
will be useful for the longer term in
the more general case or we can argue we
have to we should look at specific
domains because we have to we have no
choice because they're here right now
and have to deal with them and so the
area that that I'm looking at is is
finance and there's several reasons that
we should care about finance and why
it's a particularly relevant domain for
looking at robust and beneficial AI by
the way let let me also say so I don't
expect it as many finance experts here
in the audience please feel free to
interrupt and ask questions you know
this is a pretty small group and we can
make this very informal well one it's a
very critical sector of the economy not
only is it a big fraction of the
capitalization in the stock market and a
fraction of GDP maybe maybe the too high
but it actually to run a complex economy
global economy you need a
well-functioning financial system and as
we saw eight years ago when the
financial system starts to break down
actually the real economy can fall apart
too and so that really matters so very
critical and add to that as we also in
2008 is also pretty fragile the the
financial system is built out of
information oh what it what it really is
is a bunch of decisions that are made
based on the prospects for an investment
or your decisions made now on what
productivity
happens in the future I or lending money
to somebody based on the prospects they
will be able to pay you back later in
the future that's just based on a belief
in an expectation and a when there's a
change in those beliefs and expectations
it can change everybody's willingness to
lubricate the economy with loans and
things like that and that's how that's
that's how we have financial crisis and
that's how things freeze up and that's
how you could have just changes you have
nothing had nothing in the real world
happened in 2008 that made the global
system less productive that we lost
trillions of dollars of output we didn't
forget how to make stuff we didn't
forget there was no natural disaster
that destroyed resources it was just
changes in the expectations and beliefs
about what each other we're going to do
that cause us to stop making decisions
that can keep things going okay so it's
very fragile highly interdependent and
to put the icing on the cake that's
where the AIS are already there first
and then partly because it's based on
information they interfaces are such
that machines can work in it pretty well
one of the areas which I'm going to
focus on today is trading in financial
markets that's a phenomena it's just
over the last 10 or 15 years the bots
are really taken over at least one level
of Wall Street trading but we even
before then we've seen automation in
decision making about credit so when you
you know pay something with a credit
card it's not a person who's going to
look at that and decide whether to let
that through or not there's a bunch of
rules that may have been learned from
data or may have been partially
handcrafted that's highly automated and
credit decision making is the automated
not just for credit card purchases but
even commercial loans to some extent and
and to an increasing degree so the
facial system it's worth it's it's a
fragile important place and it's being
automated okay so that's pretty clear
let's talk a little bit about financial
markets until not so long ago if you
said traitor you probably conjure an
image like this of a bunch of guys and a
place called a pit shouting at each
other or grimacing actually always guys
and this is a thing of the past so now
the traders are these guys sitting in a
rack at a computer system those those
other guys in that previous picture are
out on the street eating in soup
kitchens and on about don't mean to make
light of people losing their livelihoods
and this is not a story about a high
displacing labor that's not that they
had risk that we're talking about today
nobody cares about those guys it turns
out okay really I don't see without
it but the consequence is really more
what happens when the franchise system
breaks and even more people lose their
labor that's that's a thing that we
really care about here so of course you
can say okay so what difference does
that make the decisions are being made
in a data center as opposed to in a pit
then by the way the pit still Julie dis
for purposes of TV CNBC has to have
something to show when they do stuff but
it's just totally you know poor show uh
it's it's taken over a large fraction
why should that matter right out on the
internet nobody knows you're a dog John
on on on a sock exchange data center
nobody knows you're a bot well you do
actually because there are things that
are different about how they work and
now we don't really know how they work
in the public domain totally this is a
very secretive industry you know I spent
a fair amount of time talking to people
in training firms and they're usually
one-way conversations they're they're
very high it you're much more say than
defense contractors think these guys are
very very tight-lipped but you can infer
some things based on observables like
what kind of computers they buy and
communication equipment they they use
what kind of people they hire what their
ex what kinds of things are expert in
and one of those things is AI machine
learning so they really hire a lot of
people
seem to know a lot about that stuff and
even if we can't be sure exactly which
algorithms are using in what percentages
we know that a is a big part of that
okay so one of the reasons that we know
that it might be different is is just by
then the most obvious one is just a very
speed and precision by which the
computers work so you know it's common
to talk about things happening in a
blink of an eye well a blink of an eye
is about 300 milliseconds so actually
you can have about 10 or 20 rounds of
bots responding to each other in that
blink of an eye not not not not even
just one so well beyond human reaction
times I've also times referred to this
as trading beyond the speed of reality
so you things happen in computers much
more rapidly than they happen in the
real world so in most milliseconds
nothing very economically significant
actually happens in the world but of
course in a millisecond lots of stuff is
happening inside our computers changes
of you information flows okay this lets
you do certain things so operations that
require gathering information from many
sources and putting them together Kent
could not possibly be done by humans
that can be done by machines and there's
certain new kinds of strategies
therefore that could not be done without
that automation and i'll talk about
latency arbitrage and anticipatory
strategies a little bit here and there
and this goes to get going together with
the speed is autonomy of course because
you can't react fast enough you have to
give authority to these agent you have
to give them the line to your your
accounts in order to do that you have to
let them operate without your
supervision because of that time okay so
by saying they're autonomous we mean
that we are we have some models that
maybe we programmed or we learned but
they're going to be applied to whatever
circumstances come up which may have not
actually been
aided by the programmer or in the
learning regime itself and that also
makes it possibly unpredictable world
will happen the other thing that I think
is also especially significant here is
is the scalability because if you think
of a new idea for a trading strategy
that you could automate you can try it
on New York Stock Exchange today but you
can also try it at the same time on
NASDAQ and on the French bourse and you
can try it everywhere on the world
they're not just for equities but for
fixed income and foreign currency
exchange and everything all at the same
time so a small idea can just be
replicated and deployed around the world
incredibly rapidly okay because it's
just an algorithm okay because there's
potential advantage for being a little
bit faster than somebody else we've seen
what has been called a latency arms race
and whenever actually the word arms
races in a title of something that
should maybe make you a little bit
concerned but what it really means here
is that there's a constant effort to
shave small amounts of time while things
so for example some five or six years
ago spread networks built a slightly
straighter fiber optics connection
between Chicago and New York which
shaved about three milliseconds off of
the the time to respond there that led
to a big advantage which was then a few
a couple years after that obliterated by
new microwave towers that cut another
half of a millisecond off at that time
off of the fiber optics and now you even
see lasers that go between laser
communications that go between some of
the data centers in New Jersey where all
the exchanges are the high frequency
traders pay the exchanges high rents to
locate their their particular box on the
other rack that's closest to the
decision part of the exchange usually
you know they hit they have a set rate
they give everybody they promised
everyone the same length of Y
they're so you so you don't it's not
they don't this is just a business
decision they don't have to jockey for
what WIC we're on the rack you are which
you know what a good place you get a
certain standard length wire there's
also specialized hardware that you could
buy software you want to you know take
out things like error checking which you
know that that waste a lot of time but
it could be better spent processing
things of course and making potentially
more fragile and more of a concern
there's a story that Michael Lewis tells
in the flash boys which is a very
engaging book about what describes one
way that a anticipatory strategy is used
so this is north of New Jersey Manhattan
some of you may recognize this there's
about 12 or 13 its varied over time
different exchanges where all US
equities can be traded ok this is Sudan
something that's only happened 10 or 15
years ago there used to be much smaller
number that there's there's many and we
can ask about why that is but in any
case there are and the firm's who are
trading big blocks of stock want to
don't want to necessarily send all of
their orders to one place because
they're not sure where the orders on the
other side will be so they typically
will split them up and to send them to
the various ways so puree say you're
running a mutual fund and you live in
Manhattan you'll send an order you'll
you'll survive to up in different places
and you press the button and your order
goes out down the lincoln tunnel and the
first place it hits is the bats exchange
which has its center in Weehawken New
Jersey and it trades there but then to
let's have it right so this is the
nursery Troy splitting the order to
multiple changes it hits bats in
weehawken and in my tray there but then
the h of t is there they're watching at
what happens at bats and they say well
you know if if some order just hit bats
I bet there's another
that's pretty similar on its way to the
euro stock exchange in mahwah and the
nasdaq and carteret and they race it
there to this is the anticipation uh
it's a about 200 microseconds difference
but they can get there a few
microseconds ahead of the other order
and so they know that this order is
saying going to be buying 100 shares of
something they'll buy it first and then
sell it to this guy at a slightly higher
price because they can basically see
that this order is going to be coming
and this is you once one strategy that
was at least observed and operational
over over those times okay exploiting
the fragmentation of the markets to get
that better now this is kind of crazy or
at least let me say that it's at least
wasteful all the those uh your
microwaves and and new five Rockies
those are hundreds of millions of
dollars in investments there you know
the specialized hardware buying all the
smartest people we're talking billions
of dollars that are being devoted just
to shave these micro seconds in
milliseconds off things and you can say
well that's going to make the markets
work better but my what as I said before
the fact that the markets reach their
efficient price one millisecond before
they were going to anyway isn't
necessarily clearly doing anything to
improve the efficiency of the financial
system okay so there wasn't so there was
a real benefit when they took the new
that the Telegraph replaced the horse
for conveying financial information that
actually led to decisions being made
faster in a way that mattered and we've
reached the limit where perhaps that's
not necessarily benefit anymore okay so
um it turns out you could fix this and
this is what was part of my entry into
the field was thinking about this and I
posted something about it in 2009
basically saying well the issue is
really that's that the market operates
in continuous time or at least the
abstraction this continuous time meaning
there's no lower bound on a difference
that could matter and
why should that be it in a world where
it was human communication there may not
have seemed a need to put a lower bound
because the humans themselves were
providing a lower bound on a difference
that they could implement but with the
computers it can go down to nanoseconds
and you know this as far down as we're
aware so instead what we could do is
convert to a discrete-time market and
the technical term for that is a coal
market it trades periodically orders
come in over a fixed time interval and
they are not seen by anybody the latest
accumulate and then at the end of that
interval you clear the market you match
supply and demand you decide who trades
and this has been around for a very long
time of course originally coal markets
were operated over times of days or at
least ours but the observation is that
we could run coal markets at a vote on a
human level as a high frequencies like a
second so every yard that you no one
hurts or maybe your page every 60 Hertz
or maybe if you really want you can do
it at you know every half second we
clear the market fast enough so that no
one really feels that they are waiting
fear there behind world events but slow
enough so that shaving a few
microseconds or milliseconds is not
going to be a material advantage okay so
we can do this and uh the idea other
people have also had that idea it's not
you know it's it's um it's kind of in
the air there have been some attempts to
do this in various forms and it's still
some push for it in it and it may happen
to some degree in any case we still
mostly operating continues markets
question going to be a secondary market
appears so that's actually the so one
common objection is that well you'd have
to get the whole world to synchronize on
this you one thing it turns out as we've
argued that you could have these
continuous markets and you could
introduce a discrete-time market
um and we we've got a paper on this to
show that the street on market is in
many respects a and a tractor so it will
actually personally could coexist but
even if they didn't coach this actually
a lot of the activity would go to the
slow markets and what we what you find
is there something like a predator-prey
kind of interaction so the slow guys so
there's this some fast guys fast traders
and slow traders around slow traders
will prefer the the discrete-time market
because they're not going to get picked
off they've got no one's going to be
anticipating their information so much
so they'll prefer to go there and the
turns the faster I just want to go
wherever the slow guys are so they'll
they'll they'll follow them there they
might the past guys might prefer to be
in the continuous one but if they if the
prey is going to the slow on that that
that's where they'll go so we argue that
it's just fine you don't have to
disallow continuous time to actually
make it go away the other thing that we
are concerned about is stability so
that's the argument about the latency
arms race I was largely about efficiency
you could say maybe it's about stability
too because once you get to these low
unprecedented time scales who knows what
will happen the most dramatic event was
in 2010 when the major stock indices
lost about ten percent in a matter of
minutes and there's been it's been
incredibly well studied all of the
events that led up to that and the tick
by tick you know sequence and it's still
somewhat controversial to what extent
algorithmic trading may or may not have
caused this it's been exonerated as the
proximal cause but there are many
reasons you know then in fact there were
other things that happened but to what
extent did the presence of algorithmic
trading and the predominance of that
shaped the response system so that when
a big when when a shock happened there
was no liquidity around to absorb it so
this has been really well studied what's
not been as well studied and
is equally important is why did it come
back maybe the algorithms were actually
saved us there because if say this this
was a close to 3 p.m. if we had hit the
market close well we're still down who
knows what would have happened in
contagion to Asian markets and and you
know wherever so we dodged a bullet
we're not really sure why we're not sure
why it didn't really happen again some
people argue that this is happening
again different sales I'll say a little
bit about that in a moment so what we
don't understand I point of that now
there's been a lot of President Michael
Lewis's flash floods which i mentioned i
think really started a conversation that
has been actually with good to happen
but there's a lot of other sensational
books the wheel the one that refers to a
I bandits and the threats to the global
financial system of course so would
check my there's a really interesting
one by guy named hi I'm bodek on that
focus is on a thing called special order
types I'll say that about that that is a
we in a I would think of this as
changing the expressive power of the
order interface to favor certain BOTS or
the ones with speed advantages and so
for example you could they provide ways
that you could give orders that whose
behavior is conditional on the state of
the market at that time that they arrive
at the market so that saves you a
percept action transit time sequence you
want to see what since then decided what
to do and come back you could put the
conditionality in the order itself and
thereby that could be a big match now
they always they argue well anyone could
use these but it's kind of like a hidden
API that only the insiders really know
about and some of these changes were
find small amounts of millions of
dollars for for this sorry wait
different issues so then this came up in
a kind of a cautionary tale that I'll
mention there was a paper that appeared
in science reports by Neil Johnson and
colleagues which which was very
intriguing abrupt rise of new machine
ecology beyond human response time so
their thesis which I think still may
actually be true that there's something
about this response time that could lead
to new kinds of interactions that were
not really possible with with with
humans now unfortunately I think there
are evidence for it turned out to be
spurious so they define something called
ultra fast extreme events as a crash or
a spike so you see this little crash
here or spike if it's positive that has
a duration less than fifteen hundred
milliseconds that involves at least ten
price movements in the same direction
and they argue that well you're given
this happen in such a short amount of
time there's no way that people could
have been responding to this so this is
a this is a kind of a new phenomenon
well it turns out that this is actually
explainable by some of these special
order types and a group of researchers
from England gallop at all Campea
figured out that this was actually the
results of my Colts an inch or market
sweep order which is one of these
conditional order types which I'm not
going to go into too much in detail but
basically it allows you to override the
usual routing rules there's routing
rules that say if in this fragmented in
this world of fragmented markets the SEC
says that if you're an exchange you
receive an order you cannot execute it
if there's currently a better price
somewhere else and of course that is
itself problematic because of how you
all your information about what the
prices are everywhere else is a little
bit out of date and so that itself the
open some things but you have to kind of
their do some notion of at least as far
as you know there's no better price
somewhere else and you could these
intermarket sweep order lets you
override that and so what's happening is
they're using that and then a big order
comes in and it just eats down the order
book and that one exchange even though
there's better prices somewhere else why
would somebody do that well it turns out
that in some cases they would rather get
a worse price right now
then signal to the rest of the world
that they're in the market to buy a lot
of shares because then they feel they're
going to get taken advantage of more
later by having some iteration so they'd
rather not take the time of sort of a
learning everyone else and so they'd
have to rewrite the order because
someone else is going to go beat them to
the new place and then they do this and
that that turned this can explain those
those ultra fast extreme events so that
the hunt is still on in my view it's an
open question whether there are these
phenomena to be discovered but the
little sometimes what they there's so
much called mini crashes are not really
it ok so we've have a research agenda
around trying to understand this new
world of algorithmic trading the you
know the issue in modeling this stuff is
you have to make some assumptions about
how the agents are going to behave so
let me digress just a moment so of
course there's a tons of work research
in the finance community on this almost
all of it is a totally database so which
is of course there's there's all kinds
of new financial data that's a great
store of new knowledge but there's only
so much you can figure out by looking at
data so you can see in the data is the
orders that were made the transactions
that happened and so on what you cannot
observe in the data is what strategies
anybody is using right what they would
have done in a situation that was
different than what actually happened
and what you can't always figure out you
can't usually figure from the it is what
did they know at the time that they
submitted an order so there's just
certain kinds of causality that is going
to be incredibly hard to get just out of
observational data it's also hard to do
experiments so in all work we we use
modeling mostly simulation modeling the
finance the academic alliance community
also relies a lot on analytic modeling
but that also is quite limited as to the
complexity of the systems that you can
really understand in that way so we've
been pursuing a particular pro choice
i'm not really going to get into today
called empirical game theoretic analysis
which combines simulation ancientfaces
is its agent-based simulation because
the key elements and the simulations are
these decision-making agents but the
Achilles heel of agent-based simulation
is how do you decide what is the agent
behavior and we use game theoretic
reasoning to essentially filter or
select what are the strategies by based
on rationality assumptions so we use
strategic stability game theoretic
stability concepts to drive our search
for the reasonable or rational trading
behaviors and we've studied various
topics including latency arbitrage which
I mentioned walked in making and so on I
also mentioned this predator-prey
fastest load traders thing that was one
of the studies and I'll talk a little
bit at the end today about a current
ongoing work on on market manipulation
but just to give an example of the kinds
of results are the kinds of questions we
can answer with this methodology one of
the studies we did was on market making
so unlike latency arbitrage market
making is a mostly benign activity that
algorithmic traders do they're actually
in many cases benefiting the market by
providing liquidity there are always
benefit things however competition with
market makers always almost always
beneficial so for example what we do and
I just want to illustrate this is here's
just a plot for a range of different
environments and what those a one is vs.
b12 this is not important for our
purposes we looked at does adding a
second market maker help the background
traders or hurt them does it him does it
also improve welfare or heard welfare
including the profits of the market
maker and for each of these environments
we get a range of possibilities for how
much it helps or hurts in this case
they're all on the positive side there
these are ranges because there are we
find multiple equilibria in these models
and we don't necessarily find all the
equilibria but I'm
among these ones we found we did what I
want to emphasize is that we're
comparing equilibrium so we're saying in
a world where there's a market maker and
then we equilibrate all of the
background traders to that that's an
outcome and on the world with to market
makers everyone's not going to do the
same thing something else is in the
world change we're going to be
equilibrated at we're going to be
compared things equilibrium to
equilibrium okay that's what we do
really in all in all of our models okay
so what I wanted to talk about for most
of the rest of the time is hat yeah is
going forward can we come up with a more
general framework for thinking about the
possibilities of algorithmic traders and
try to get ahead of what they do so we
have this framework I refer to it as the
Arbat and it's based on the thought that
really almost all trading strategies can
be cashed in some respect as a kind of
arbitrage so arbitrage is a very
important concept in finance it's when
you take advantage of the fact that
different markets for the same asset may
have different prices okay and some of
the things that I've talked about
already really are examples of arbitrage
so this is something that's highly
subject to all donation and so some of
the first automated traders were we're
doing this kind of arbitrage were they
just if we have fragmented markets we
should look at all of them and if
there's a chance to buy in one place and
sell at another place then we should do
it okay and as I mentioned that's one
things that computers are especially
good at is being able to quickly get
information from many sources and
combine them which is which is required
for arbitrage now of course if even you
have a few of these energetically
looking for arbitrage or two
opportunities then they're really not
going to exist anymore because as soon
as there's somebody who's making sure
there's no discrepancies that will make
the discrepancy go away
right if it's available for one price
here and a higher price there you buy
here and sell their well after you do
that enough times the prices will no
longer be out of whack and in fact
economists often use no no arbitrage as
a condition for economic efficiency if
there's no arbitrage pause that that's
just dis equate the two so what we what
the name of the game really is an
arbitrage and you when you when somebody
says that they're in the arbitrage
business what they are doing is they're
pushing the envelope about what an
arbitrage is so that's really it's not
really exactly the same thing so what I
should mention that there you also have
to take account of transportation costs
and transaction costs and so on
execution risk that is if you at the
time that you saw the discrepancy by the
time you actually can execute or take
some finite time it could go away so all
that gets in here but we push the
envelope by looking at what is really
the same so some things are really the
same so you can there's there's there's
for example index securities you can buy
a security for the sp500 and that is
equivalent to a bundle that has an equal
share of the top 500 capitalized ferns
on the standard & poor's index okay so
you could if you notice that the price
of the SP index is available for sale at
somewhat lower than you could buy all of
the 500 shares or sell the 500
individual socks you could buy that
index and then sell the 500 it's really
hard to do it with humans neither like
you have to get a big room with 501
telephones and you know people doing
that they did that in the old days now
it's computers that do it those are the
same now what about what it's actually
really not the standard poor index it's
an index future so that means your
policy do in the future well you could
if you're if you have an opportunity to
sell something in the future well what
you can do is you could buy it now and
hold it and sell it that now you have
might have to borrow the money so you
then you know just have an interest rate
addition that you have to do that so you
can make this mapping it gets a little
bit more complicated but you can you can
do it and you can make sometimes their
paths of exchanges so for example you
could look at the graph of foreign
currency exchange rates and you might
find a path that when you multiply them
together comes out to more or less than
1 by enough of a way to do that so
there's algorithms are always doing that
there's many other examples what's
called that that anticipatory play that
I talked about before is arbitrage in a
statistical sense you didn't know for
sure that there was another order that
was going to nasdaq based on what you
saw at bats but you had a good problem
you had a reasonable probabilistic model
basis for that and so if you just use
that as a system model you were doing
arbitrage and expectations with respect
to that statistical model okay so really
the name of the game in arbitrage is to
push that there's always at the heart of
an arbitrage is an identity relationship
and you're just stretching the batteries
of what that identity is okay so the
reason I'm harping on this is because
this can be automated i believe so it's
really a search problem if you print it
and there's two ways that you can do
this search if you had machine readable
descriptions of these securities and by
the way to-- derivatives are another
example where you know this is something
that security is defined by its
relationship to other events involving
other securities okay this is a church
and a reasoning problem to determine
under what situations these things are
equivalent can you come up with a a
combination of transactions that under
any state of nature makes these things
equivalent for a short profit okay and
then you execute that so that's a big
search problem that you can do if these
securities are described in a formal way
okay or if you want to go to this little
route you have lots of data about the
how the prices of different kinds of
assets move over time and you can come
up with what seems like a reasonably
reliable model of
um pencil arbitrage of course now these
some things sometimes go wrong some of
you may remember or read about long-term
capital management one of the first
hedge funds that tried to do statistical
arbitrage on a big way and there was a
regime change you know the situation
that changed in the world involving uh
uh actually Russian markets that just
made things that used to be very true
not yo good things I'm used to move in
certain tandem not true anymore and so
they they pet and they lost so there's
always risk management and these two but
but this can be automated and so the
proposal is that and I don't even want
to be too naive about it but given that
we don't know what the financial
industry is doing we could we could
should at least invest in kind of public
our bots that are watching and trying to
derive new arbitrage at least trying to
anticipate where you are new ideas out
there of exploitable that we can maybe
get us ahead of what the what the bots
might be doing and maybe the government
will never have as much resources as the
the traders out there but it's a way to
possibly hit an alert about things that
could be destabilizing to the to the
financial markets so um in thinking
about designing in our body taking the
perspective of I'm looking for
opportunities uh there are different
levels of aggressiveness where this may
apply and this may be a way to set some
boundaries on how we might want to
regulate autonomous trading agents so
when I described so far was mostly in
the category passive search for
opportunities that some arbitrage is
actually not beneficial like latency
arbitrage some others like Mark and
making which can be thought of in
arbitrage overtime you know that
statistically things get out of balance
and if you smooth that out you can make
a profit that's essentially what market
making is those are those are possibly
beneficial and those might be okay but
that's the lease aggressive however
once you are if you're doing this
exploration and the kinds of actions you
do you might you might come up with
strategies that also affect the beliefs
of others in improper ways and that's
something is called spoofing or market
it was a form of market manipulation
that's also something that that could be
possibly automatically derived is ways
of finding new arbitrage opportunities
by misleading others about things I and
my ordering is I'm not like a hundred
percent confident that I put things in
exactly the right order here because
another thing you can do if you're
really good at finding arbor at
exploiting arbitrage opportunities you
have a very strong incentive to make new
arbitrage opportunities and one could
argue it has been argued that the
proliferation of derivatives it's
sometimes happened because it's a way to
introduce new interdependencies that
some of the clients won't really
understand very well but if you
understand them well then you can you'll
have new auto shows opportunities and
collateralized debt obligations were
possibly an example of this but whenever
you define into a new derivative
instrument there's all now potential new
identities that that exists and that
that may be a new chance to get total
profits another is fragmenting markets
so why did we go from you know one or
two or three stock exchanges to 13 well
it's a it's a kind of a complicated
story originally it was very much
encouraged by the SEC the u.s. in the
u.s. anyway as a promoted competition
right because maybe we really don't want
just one sock exchange because they
could perhaps make it transactions cost
too high and we want some competition
there but then once you define the rules
to allow fragmentation there may be an
incentive to have too much
ragman tation because the parties that
are good at arbitrage between them will
want more fragmentation and the people
who create bats maybe they did that ah I
don't know exactly what they're what
they're thinking was but why did why do
we need 13 and then finally the big
category is malicious of version of
markets and this is really where the
boundary between issues of AI safety or
it is is indistinguishable from cyber
security so one could perhaps launch a
denial of service attack on you know on
Wall Street if one wanted to you perhaps
achieves achieve some goals there's no
reason that that these kinds of things
would couldn't happen most many parties
that would not be in their interest to
do that but there may be other kinds of
you know just direct things that we
really think it more as just direct a
taxes and subversion that could be
automated by AIS and certainly and
non-financial cyber security you know a
lot of that is proper is done by the AIS
and defended by the AIS and in total so
let me drill down just a little bit on
market manipulation so one of the
interesting things about this is the
difficult definitional questions both
legally and even just conceptually so
sec defines is an intentional conduct
designed to deceive investors by
controlling or artificially affecting
the march for security so right away if
you're thinking about autonomous agents
doing manipulation you're going to choke
on it this intentional conduct part
because now who intended this was it the
programmer intend to this was it the
agent itself can we ascribe intent to
that maybe somebody programmed an agent
to search for good strategies and they
hit on spoof of spoofing strategy you
know you just said go make me some
money find some patterns things you
could have actions you can take and ways
to make money it comes up with spoofing
was it intentional ok so the truly law
and policy are going to have two and one
which are very slow-moving things are
going to have to move to to deal with
this reality ok spoofing is a specific
category of market manipulation
dodd-frank has the definition bidding or
offering with the intent to again
content to cancel the bitter offer
before execution so this is also a
tricky because you can't just say well
you can't cancel orders because there's
lots of legitimate reasons that you
would cancel an order before it executed
but if you at the time you submitted it
you one hundred percent sure you're
going to cancel it before it executed
that maybe would be you know that intent
there this has actually been in the news
a lot lately have been some some
prosecutions in fact just about it a
year ago maybe two years ago they
arrested there was a high-profile arrest
in London of this guy named sarao and
who's in his parents basement doing
spoofing and they claimed that he was
contributed to the flash crash kind of
dubious so this this actually comes from
a court document or some horse or he's
an invidious investigation by the UK
financial conduct authority about a guy
named Kashia who was manipulating some
commodity markets and other started to
show what he did basically this is a
strategy called dynamic layering where
he put some sell orders here that were
out of the money meaning they were there
were they didn't match in existing buy
orders but the the idea was that
somebody would think that someone's want
to sell a lot of these you know oil
contracts or whatever they were and so
because of that that would depress the
price because he's always someone's
gonna try to sell these things and then
he went bought it here so he put it only
sell orders that he bought then the
solders went away to sell orders out and
started putting in buy orders
that he didn't really mean to use and
then sold okay so very simple kind of
pattern so we this is kind of in in
progress work we said can we implement
that so first of all to be able to
implement this in a model you need to
have agents who would be fooled by this
so you need agents who are trading who
would look at this information and
change their beliefs on there and it
turns out there's a well-known training
strategy called heuristic belief
learning that is a reasonable strategy
in a world without spoof errs ah and
when you you you equilibrates and then
you put a spoofer in there the spoofer
can raise the price above it so what
this shows is that this is comparing
runs within without the spoof or showing
the difference in the prices that
existed over time with them another
spoofer ok so this in this case
was just putting in orders out of the
money they'd never traded had it had no
effect on actual we never actually
transact but it could have changed the
prices that actually happened among the
other traders based on its own actions
now you say well this is when everyone
is doing this heuristic belief learning
in an equilibrium model where you have
some doing that and so I'm not doing
that the effect is a little bit less or
it doesn't persist as long but again we
can get the spoofer to really change the
price and we also can get this mover to
actually be profitable we can wrap
around this where it sells at the
beginning and it buys at the end or vice
versa depending on how it wants to move
the price and can make money okay so
there's still lots of open questions we
you know we just put in this kind of
extreme version of dynamic layering
could these strategies be automatically
learned can a regulator look at market
data and figure out that spoofing is
happening okay either in real time or
maybe based on auditing after the fact
is there going to be some information
available as they can do that can the
Trinities themselves be sensitive enough
to this that they can
tech the patterns the telltale signature
so that they could be spoof proof I
suspect this is ultimately really
impossible but maybe it can minimize the
vulnerability to this to some degree ok
so those are the kinds of questions
other we've if you look if you just left
them the humiliating system to do stuff
a long time with a very next version why
we did then this and many other
potentially using strategies that
dodd-frank yeah yeah yeah I I strongly
believe the answer to this is yes I
don't see any reason why wouldn't be s
but we haven't actually done it yeah so
it's very it's a reinforcement learning
problem that should be well within the
capability okay but right and so
detection will probably there's another
arms race here detection will probably
always be somewhat behind this but
nevertheless you know a lot of them mark
manipulators are not that smart just
instantly with cybersecurity actually
even if it's ultimately impossible you
can still do a lot of good with defense
okay so i just wanted to get just to
throw this back up here I mean
ultimately we really are worried about
this and if you want something to keep
you up at night there's a enjoyable
science fiction novel called the fear
index Robert Harrison this was a about
five years ago here the some physicists
and Switzerland built this algorithmic
traders that was totally autonomous and
it figured out that you can make a real
lot of money in panicked situations and
so it did things like brought people to
set off bombs and financial centers and
things like that so ah again not not not
impossible I what's what's impossible is
that the physicists would have made this
system
Stassi applause a part of the book so
just just to leave as a closing comment
I've argued that finance is a really
great case study area for looking at
issues in robust and beneficial AI
because it's a very important domain for
our world economy and yet it's at the
bleeding edge of automation there's
really interesting new phenomenon
because of the super human timescales
that are involved that we don't really
understand and even if we did understand
them actually figure out what to do
about them would present challenging
issues both at a technical level the AI
and economics boat which I view is both
technical and on the social level how do
we adopt law and policy to deal with
that so this will keep us busy for a
while and I hope we will have a fruitful
interchange with the broader AI safety
research community questions is their
focus reason why you should be legal
because it seems like the problem could
you forget a guy all right from the main
bedroom so that's I think I think it's a
reasonable question so you know in
general fraud you know is is illegal and
but it's it's you know defined it really
is always defined though in terms of
what the belief soir of the perpetrator
of the fraud and you know and their
intent to provide your misleading
information and I think there is good
reason that we don't want misleading
information in markets it really you
know is hard but at the same time a
naive definition like you can't say you
can't legislate truthfulness in bidding
a negotiation right you can't say you
must
for to buy something at the price you
really are willing to pay for it right
you can be you know it shouldn't be
considered immoral to offer ten dollars
for something that you're going to pay
twelve dollars for right so so that is a
misleading of a kind about your own
state but somehow spoofing misleading
about market conditions that aren't
really you it seems different but i but
I drawing a line is tricky yeah you just
expect money funds prove is that if you
have a good other than that can detect
moving right which is a reason to doubt
that you could have a perfect algorithm
ejecta Texas spoofing it was I mean what
we intend to do as we develop this is to
keep on recursively or iterate
iteratively pursuing that because once
we have a detector then we would let the
spoofer try to respond to the detector
with more of Asian it maybe it'll
degrade their ability to spoof but
ultimately there's an equilibrium where
there's probably a little bit of
manipulation possible that's a reducible
good so a couple of things because one
is that rather than try to say there are
some money making decisions that are
morally ok and there are some money
making decisions that are not might ask
why do we have markets and should shoot
the motion that AI agent of the body
soul EV to to make money as opposed to
function s as information exchange yeah
so I mean I think that the theory of why
financial intermediation is beneficial
right is that you want to market so that
the real so these is the reason would we
have capital markets is so that actual
investors can make transactions with
actual entrepreneurs right to provide
right I want to retire in the future you
want to start a business we've got a
really muted usually beneficial exchange
however we're sparse the ones that have
that real need and the intermediaries
can bridge across time and they can you
know they could say well I don't they
can aggregate information they really
provide a useful smoothing and
information propagation function at a
first order okay the issue is really
when they start being harmful is when
they get to the higher degree order that
they're not really providing making
information propagate better than it did
before okay so you know when arbitrage
arises naturally someone is really
providing a benefit by running that
bridge and making the prices were
consistent more can because we went the
other way when arbitrage opportunities
are outstanding there's an inefficiency
in the market and because the prices are
not signaling the the true competitive
equilibrium and so this wrong decisions
are made okay so in the we always need
the pension system people to make
decisions should I live in this place or
that place right should I buy this house
should I you try to build this apartment
building you know all these things
decisions are made using price signals
and using that and there is a real value
to the economy for those for those
really reflecting accurate information
which is an aggregate of information
that's very diffusely held in the world
so I so I I very much wrongly believe as
you can tell
and financially in you know in financial
intermediaries provide something useful
it may have or you may have grown loaded
to parasitic so the I mean the reason i
ask that you know often when you see
something that's dysfunctional because
the reward signal is incorrect right and
there's a piece of reward this is some
externality is not being taken again to
account some people are creating the
atmosphere of a bike right you do stuff
they make money from we all suffer from
this is young yes suggestion is that
that leaves malfeasance any years in the
market side yeah and i think most
economists would view like things like
the latency arbitrage is another example
of this externality or the excess
fragmentation as exactly that and but
their normal solution would be okay
regulation or you charge the externality
will factor that into the utilities oh
yes whenever we can do that whenever we
can put a price on it that's based of
the better solution and related question
is suppose we have a NE estis honest it
doesn't ease legal things that just is
that much better at modeling real
economy you psychology river and
gradually accumulates whistle money that
would also be very careful yeah well i
think you could have these parasitic
relationships even if it really is
underlying it but i think so as example
gave of market making a market making is
really useful when you have a thin
market right you know i want to buy
today you want to sell tomorrow we'll
miss each other at a market maker by
always being willing to buy always being
willing to sell is essentially bridging
that so that is the jenna thing on the
other hand being a market maker in apple
stock is not really providing any value
because there's so damn market is so
dense already so thick that the buyers
and sellers are meeting each other so
the one who's market-making they're just
being a parasite they're just like being
a gatekeeper and taking a nick
or penny or whatever it is as a toll
each time there there's nothing that
they're doing in principle that's
probably they're just a little bit
faster so they can just be there and the
bend we would have made that extra penny
in our transaction someone else is
getting it instead so we're it didn't
really do gray the efficiency with more
except for their costs but it hurt us a
little bit so I don't know what the
regulatory solution is there either but
just even understanding when is what
kinds of intermediary our beneficial one
and what you're not yeah you should buy
with the phones eggs is systemic
resistance or minor ones I ain't
distinguishing the one that might block
the whole financial markets from the
ones that cause people to lose a little
bit of money uh yes so that's good
question I mean I and actually to be
honest most of our studies have been on
the efficiency market performance
questions which are much easier to study
the stability questions which arguably
are more important are harder because
we're talking about rare events right
and you can show that certain things can
happen but showing that a certain
mechanism will reduce the probability
from 0 point 0 12.00 one that's just a
bigger lift so we're trying to move in
that direction to address those
questions but I to be honest we've
probably did cherry-picking we've been
doing the easier stuff first you seem to
know a lot of it moving have you used
this to your advantage no but I guess I
was I wouldn't say I wonder what I would
say i would say that anyway but yeah we
have this i mean i decided to we're not
doing this modeling in sport of coming
up with strategies to play it our own
behalf in fact arguably a a lot of the
work that we're doing in study and
exploring trading
strategies is just re discovering stuff
that is known in all these priori
trading firms but they're not telling
anyone about it so we figured out okay
there's still a role for us even if
we're a little bit behind all right so
um that's since I'm before lunch look
let's thank the speaker say