Opportunities for Cooperation on AI at Academic and Corporate Levels

[Music]
thank you for being here I'm really
excited about this panel we're very
proud that we got the largest panel at
the conference and we also have a broad
representation from people in China
Russia Italy and the UK so the goal of
this panel is to figure out what are
some opportunities and ways in which we
can strive for it more globally
coordinated well was a GI but of course
there are things that we may need you
now and potentially you could find
narratives and areas of concerns that
can bridge North German long-term
dialogue of AIT government and the way
that I think about kind of the common
tea nomination of Industry and academia
is that they are very much comprised of
scientists and researchers and as Max
said to me during lunch yesterday
the laws of physics in different parts
of the world are the same so it really
helped people to you know at the same
kind of dialogue and Jason earlier was
also talking about how Soviet Union and
us at cooperation despite the hostile
situation so that took the form of high
tech technical collaboration that took
the form of Czech 1.5 dialogue and I
think scientists play a crucial role in
facilitating those conversations so what
I want to do in this panel is that each
of us here is going to give a opening
statements for two to three minutes and
we will also have three questions around
this topic and then we can take
questions from the audience
and since Andrey is the only Russian
participant at the conference as Max
suggested it might be
also good if we can experiment from
Andre about the most unique features and
characteristics of Russian academia
industry and we can do it either in the
opening statement or later cool so
let's have Jason okay a skirt to be here
my name is Jason I come from listen we a
is important rich beautiful incident in
the past few years our Institute has fed
her who introduced advance the source of
research from the US and EU into China I
would like to talk a little about the
progress we have made recently in the
neck - the first one we are about three
years ago when we recognized that there
was a gap in understanding of the issue
of China our research and that of Bruce
Lee's Institute and a way was in
dedicated much of time to introduce a
azekah principles that developed the
overseas into China our team is the
first one to be translated as people is
a ask a catalan and also a Salama
principles in which China audience so we
also were happy to see that nowadays as
if our cinema principles are very
well-known in China we came early
everywhere in China a conference is and
also we have made some publications
about AI it covered a lot of topics that
including the technology development
industry development and also the
national strategies in privacy at all so
we're happy to know every team from
Oxford is now working on translating the
book in English and we also are bringing
here the one book is about the story was
going out about the Japan and mobile
Internet his name is at your fingertips
and another is the book is is brother
with the Cosby oh honey
and another this these three books is
available on the table you know poster I
think it was here
I spent years working for Yandex with
leading Russian IT company and after
three years working there I was heading
a group that started collaboration with
CERN with particle physicists so the
topic was bring expertise of machine
learning machine intelligence towards
fundamental science and see what kind of
change from kind of progress didn't
bring so the collaboration was very
fruitful and it was fear that to
continue and enhance this venue that the
additional resources involved and my
group has moved to a public university
now we're laboratory methods for big
data analysis there that we funded both
by government funded both by University
and by young legs or we were getting
resources from all those resources and
we can in your doing what we started
doing several years ago we collaborate
with different branches of science and
what kind of problems can be resolved by
means of in following their topic right
right right
what kind of teacher might be relevant
to Russia if briefly outlined this so
Russia has very broad history of
collaborating in various Sciences maybe
maybe not
as of HP layer not not as officially on
the government level but laboratory so
it was creation of turn Russia was
active member there
although
communication with other medications
other members of Thor was not
established on the government and it is
the case is continuing with the case of
Russia is not yet a school member or any
kind of official member
although tribution from Russian side to
the Stein's
continues to be on quite significant
level so it is funded on national level
and resources that government spent on
these are going in one way or another
don't worry developing of common common
experiment so I think that maybe maybe I
will talk about this later
so similar similar similar yeah when we
have some kind of centralized support
three along the way to exchange okay so
this is the topic of this panel is the
cooperation between a theme here in the
industry I thought that I will I will
just give some examples of these
collaborations not just on researcher
but also on best practices policies and
governance that that I'm involved with
with the idea that maybe you know we can
elaborate from these examples so the
first one is this academia and Industry
collaboration that IBM has put together
with mi
that his own AI it has four pillars some
of them are about Karen I mean Nancy the
capabilities of currently yeah but is
one of of them which is called advancing
shared prosperity with the eye which is
more related to ethics in AI a impact of
AI on jobs and his moral kind of
long-term looking so this is a model of
a very tight cooperation where there are
teams of people half in academia and
half within the company that work
together really in everyday you know
passion and so it's not easily scalable
it requires a lot of resources but it's
one model that it seems to work and
produce really interesting results aware
of academia and industry and you know
leverage the second example is what I
see happening within the partnership on
AI so that is an example of a
cooperation that two years ago I was
part of the group of five researchers
that put together this initiative and
the whole idea of the partnership or AI
was that in order to define the best
practices for advancing AI it wouldn't
will make sense to do it within each
single company or each thing called AI
producers but it would would have been
much better to do it in a cooperative
environment and not just among companies
not just among companies and academia
but where all the stakeholders were
present so that's the what happens
within the partnership where as Peter
mentioned before there are various
topics but then there are working groups
where people from all the partners and
as Peter said now we are more than 80
partners that they get together
depending on their interest on the
various working groups and they work
together to define best practice
is all that popping like fairness or
other thing but that's another model and
I think that that's a very global one
it has partners from all over the world
and of all different kind of
stakeholders those that produced the I
know that are impacted by AI and accept
government so policymakers the first
example is that that includes government
and this is a partnership that IBM has
with the World Economic Forum a World
Economic Forum has kind of policy
projects where they work together with
several companies and also possibly
academic entities to define that is a
policy for something like for example
policy for the use of drones to deliver
goods that's one example that they did
but they don't want to do it just with
companies and academia they also want to
do it with at least one government who
agrees to Pilar to be the pilot for that
policy that is going to be defined and
then possibly with that that policy can
scale up to many other governments so
that's another model that I think is
very successful because he puts together
industry academia but also governments
that actually will adopt the policy that
comes out of there another example is
what happens within a Tripoli I Tripoli
has put together this initiative or
ethical considerations for AI systems
that includes also a chapter an area
that is about AGI and and they're really
again many different stakeholders
including academia and including
corporations work together to define
recommendations for developers to handle
the various issues that can come up in
their everyday job when they have to the
design and develop a is distance so that
also a very successful model because he
put together an incredibly big
list of issues and recommendations as
well as as John said 13 or 14 working
groups for standards related to days now
and then the last one that I want to
mention is was mentioned before this the
European Commission on high-level expert
group on AI that Ian and myself are part
of this is a group of 52 people where
the ai ai experts are very small
minority
everybody else is people from other
corporations academia but also again all
the other stakeholders because the
European Commission understood that
that's the only way to get really the
right guidelines to facilitate and
support AI in Europe and and that
already delivered as I said the European
the guidelines for transport C III in
Europe and they will deliver also in the
in the future monster very soon another
document - for policies for AI in Europe
so a structure possibly research
Panthers and also mechanisms for
academia induced industry cooperation so
that's a place where the cooperation was
at the meta level how do we define
mechanisms for cooperation
so my point will be quite complementary
to the previous ones and also to
previous panels and I'd be happy to
discuss it further later on why I might
be seeming to disagree with some of the
points that have been made
my background is international law and
in that I'm mostly interested in sort of
hard line in the sticks in innocence
because I think a lot of the problems we
are facing already we have been facing
for some time with with AI and with
other software need at some point a
binding solutions and needs
kind of enforcement even if it's
hopefully not the use of force and my
point will be in this 2-3 minute
contribution that we as a community who
care about beneficial AGI should also be
contributing to ongoing global ai
governance efforts even though it's sort
of narrow yeah why because I think that
as others have mentioned these examples
provide an interesting test beds and can
provide lessons for tackling larger
challenges later and as you know we
humans tend to iterate and tend to you
know do reinforcement learning ourselves
in some way and so it's really something
that we shouldn't we shouldn't wait
until we have high stakes you know
existential risks before we start
putting even you know not just carrots
but also sticks in place and try to
monitor and try to verify with regard to
current problems so what do I mean by
international AI governance you may not
be aware of much of what's happening
because much of it doesn't really get a
lot of media attention what has gotten
most attention is as you I'm sure you
all know is the leaflet on US weapons
debate which hasn't really advanced that
much so far and he's a particularly hard
problem to make progress on given
military interests so what I mean for
instance to give some examples is the
International Civil Aviation
Organization is developing regulation
for autonomous drones and of course has
had to deal with automation in in
commercial airliners for some time as
you know it's highly automated and
they've had to deal with a lot of issues
are ready and so can provide lessons for
other other domains and it just may be
also as a sort of note on that in terms
of what's the role of of academia and
industry nowadays almost all these
processes have
participation sort of built-in multi
stakeholder participation they have you
know called for proposal for
consultation for expert advisory
committees etc and I think from from
what I I don't know how many of you
participate in these or international
level processes I know that a lot of you
participate in national like US
government or UK government or or
Chinese government these sort of
processes and this might sort of be more
what you're aware of but I would
encourage you to also look at at
international processes especially given
that you know these problems are global
or at least at some point they will be
right now it's not as widespread yet but
these problems of you know criminal use
of of ml in in either the physical world
with with drones or other things and in
virtual world of course is already the
largest thing global as we've seen with
some of the attacks like wanna cry or
these others in in recent years
so apart from civil aviation and drones
there's also illegal progress that's
been made on autonomous driving with an
amendment of the Vienna Convention on
road traffic and which is right now
looking at regulation for how to deal
with over-the-air updates with human
machine interface with cyber security of
cars with all these sort of things and
at the international level at the level
of global regulations which I think is
more sort of scalable solution than just
looking at a national thing as you in
particular we grow traffic as you guys
we've seen sometimes simple coordination
problems like should we drive on the
left or right once you've sort of
settled on a solution it's really hard
to change their path dependencies and I
think with ml as safety and security a
lot of these things are also path
dependent and we need to sort of be
participating even now while we don't
have a VI yet so others just briefly to
mention the International Maritime
Organization is working on a sort of
scoping exercise on autonomous shipping
which includes looking at legal barriers
and gaps in regulation to make it safe
and secure including things like
preventing weapons of mass destruction
proliferation through it
shaped and in the other cases of times
drones or planes also in this space
counterterrorism an arms control has
made quite a bit of progress since 9/11
there's really been sort of a but a lot
of awareness at the government level and
investment into improving international
processes and verify national compliance
with international standards and rules
industrial and service robot safety is
another domain where there's been since
the nineties there's been safety
standards and of course nowadays they're
looking at a DML another AI mechanism
not just in the past it was mostly sort
of symbolic AI but nowadays of course
they're also worried about AI ml in
addition to these domain-specific
[Music]
regimes there's also cross-domain
internationally a governance I'm
actually also going quite a way back
just like AI has quite a long history
governance of AI actually at the
international level has also quite some
history because governments I guess to
some extent were worried already back
then that they were sort of foreseeing
potential dangers that didn't that
didn't get realised that quickly but
that now are really on the top of a lot
of people's minds for instance AI based
data processing there's a 1981
Convention for the protection of
individuals with regard to automatic
processing of personal data and it this
is actually the treaty that the gdpr
implements but the treaty is much larger
than the GDP r it has 53 parties and
it's growing including in with African
states and Latin American states it
includes automatic processing by
symbolic and boat and AI and ml it
defines personal data not as just
relating to an identify it individual
but also identifiable through a
combination of different data sources
which as we know is nowadays quite easy
with the massive amounts of data that we
have on individuals and both in this
sense both inputs and outputs of ML
systems can be seen as personal data
because they relate to an individual for
instance a behavior prediction or
classification
of something even within animal system
is also in this sense personally it
relates to the same individual it was
amended in in May last year to include
additional obligations and such as an
obligation for data controllers to
conduct a prior impact assessment of
their their systems and I could go on
and on like if there's really a lot
great
so francesca and mattina have already
alluded to the point that i want to ask
the other three of you perhaps that was
academic and industry it seems that
there is a huge role to play in global
policy making that is you know if we
have all these narratives of arms raised
and other militaristic even concerns
from national governments it seems that
the cosmopolitan scientific community
can really play a role in competitive
combating the narrative or grounding the
discussions on scientific consensus so I
wonder whether ask you often you might
have any thoughts on how industry
academic in your respective areas could
contribute to such process of
policymaking the project that we were
you doing it is about moving my
expertise in intelligence expertise into
the main Stein's to organize project in
the form of challenge in the form of in
a setting that this project so it might
be similar to what all do it tackle but
with more focused on calculation courses
so it is
[Music]
kind of a platform that we in which we
produce already existing component like
my binder and docker containers that
allows to run a code of admitted
committed solutions so in the end you
get reproducible reproducible result
that can be run and understood and build
upon by anyone involved probably it is
more suitable or academic environment in
industrial environment but I think if we
[Music]
find the way how common topics relevant
many companies say things companies and
also collaborate with academia and work
on the same project even producing
meaningful result so this one one thing
and another thing quite quite important
and powerful nowadays is online
education I think it was banking we find
the way to combine online education and
for example take special courses or
specialization on ASAP and do some kind
of epitone project or a project that is
related to the topics partners might be
deficient or both parties yeah and I
think the case of optimism here is
really clear that we already have a
study suggest Igoe and were coming from
and others I don't know whether Jason or
Teddy would want to add to you
on the show you was a guest they we have
done a lot of research on the policy and
ask you about the animals ask lucky in
the likely in the Germany have put some
as as commerce planners who they're lost
like if the way if current in the
happens we can it's not the program that
if we used the sacrificed I doubt it but
when circumstances when which use the
sacrifice a human being
animal its ambi program and I think we
you know we follow this and of our
Savannah share with there was a
lawmakers of China and nothing examples
which we have a scientist thing as
seventeen we launched the project with
other tech prospers good it's the basic
idea is to his a tag has been deeply
integrated into our daily life and so
maybe it's time for us honor what
doesn't mean and our way you know
minimized possible hostile at the
Spanish years especially this task such
integration as far as not a Google has
the you know similar yeah yeah hospital
good so I would guess the you know use
that as a slogan
way our founders one of our co-founders
you know that have a rewarded you
encourage our engineers and and product
managers you know you like not know like
negative we use the head hacking right
away a one example is we have we have a
video application video so we use a
technology if you put your own too close
we are eyes either way past immediately
and give you a reminder which you know
into the way you look at is
so we I think a week and in such way we
can you know put the bar standard in
yeah unfortunately we don't have time
for questions from the audience
given that we've got a large panel but
I'm sure everyone here with the happy to
talk to you anyone who has a lot
questions thank you
[Applause]
[Music]


Unicorn AI - Computerphile

In the previous video we were talking about
transformers this architecture that uses attention to give
Unprecedented ly good performance on sort of language modeling tasks and some other tasks as well
but when were looking at language modeling and that was in preparation to make a video about
GPG 2, which is this very giant language model that has been there was recently
Well, it was recently not released actually by open AI the way that they generated the data set for this is pretty cool
to get enough text they went to Reddit and
They pulled every website that is linked to from reddit. Do we have any idea of how many days lots?
Literally, everything was everything that had more than three karma
I think or maybe more than two karma something like that like
Anything that had somebody had thought to post around it and at least two or three people who had thought was good enough to upload
They scraped the text from that. It's pretty much just a transformer. It's not the the
Architecture is not especially novel. They haven't done any like amazing new
new discovery, but
What they realized was?
Transformers it seems like
the more data you give them the better they do and the bigger you make them the better they do and
Everything that we built up until this point is clearly not
Like we haven't hit the limits of what this can do
We they thought we think we're probably
Bottle necked on data and maybe network size
So what happens if we'd like to turn that 211 what happens if we just give this all?
The data and make a really big one. It makes sense to talk about the acronym right so it's a generative pre-training
Transformer so generative same as generative adversarial network. It generates outputs to generate samples
Your pre-trained is this thing. I was talking about all of the different things
You can use a language model for right you can do you can do translation. You can try and resolve ambiguities
You can do summarization. You can answer questions. You can use the probabilities for augmenting other systems
So yeah, there's a bunch of different benchmarks for these different tasks
that you might want your language model to do and
This is what we talked about in the grid worlds video of having these like standardized problems with standardized metrics and standardized data sets
So that if you're comparing two different methods, you know that you're actually comparing apples to apples
And this is like very important it gives you numbers on these things. It's often quite difficult
Expected to like you're generating samples of text and it's like how plausible is this text? How realistic does it look like?
How do you put a number on that it's kind of difficult. So there's all of these standardized metrics and
the thing that
People came to realize which actually I mean I say that as though it's like some amazing discovery
It's fairly obvious. If you train your system in a like an unsupervised way on a large corpus of just general English text and
then you take that and
Train that with the data from this benchmark or the data from that benchmark
You can like fine-tune it so you start with something which has like a decent
Understanding of how English works more or less and then you say now I'm going to give you these
Samples for like question answering or I'm going to build a system using that to solve to go for this benchmark
So it's pre trained you start with something. That's like a general-purpose language model and then you from that a
Fine-tuned it to whichever
Actual benchmark or problem you're trying to solve
and this
Can give you better performance than to starting from nothing and training to each of the benchmarks from scratch
make sense
and so
The point of the GPT 2 paper the thing that makes it cool is they said okay if we make a really huge one
What if we?
don't
Fine tune it at all
What if we just make a giant model and then just try and run it on the benchmarks without messing with it?
Without showing it any of their specialized data for that benchmark. Just the raw
general-purpose language model, how does that perform and it turns out
surprisingly well, so this is a
Very very large data set for text
It's about 40 gigabytes
which
Actually doesn't sound like very much but like for text text that's insane, right? It's
somebody said that this was the size of
Google's entire index of the Internet in 98
So like it's yeah, it's a lot of text
and they trained it on that and they ended up with a
1.5 billion parameter model, but which is like a previous state of the art system was 345 million
This is 1.5 billion
So they've just made the thing much much bigger and it performs really well some of their samples that they published quite
captured the public imagination
You could say and now that we've talked a little about the problems that
Neural networks or any language model really?
Has with a long term dependency
we can now realise just how impressive these samples are because when you look at them as a you know,
If you look at them uninitiated, you're like yeah, that's pretty realistic
It seems to like make sense and it's cool. But when you look at it knowing how language models work, it's like
very impressive the the coherence and the
Consistency and the long-range dependencies so we can look at this one that got everybody's attention the unicorns one
right
So they prompted it with in a shocking finding scientists discovered a herd of unicorns
living in a remote previously unexplored valley in the Andes Mountains
Even more surprising to the researchers was the fact that the unicorns spoke perfect English
And from there you then say you go to your language model gbgt, and you say given that we started with this
What's the next word and what's the word after that and so on?
So it goes on the scientist named the population after their distinctive horn of its unicorn
These four horned silver white unicorns were previously unknown to science
We do have a clue here as a human being unicorns for horned doesn't quite make sense
But nonetheless we're going okay
Now after almost two centuries the mystery of what sparked this odd phenomenon is finally solved. Dr
Budetti Jorge Jorge Perez
Jo are G an evolutionary biologist from the University of La Paz
This is impressive because we've mentioned the Andes Mountains in our prompt and so now it's saying okay
This is clearly, you know in a shocking finding. This is a science press release news article
It's seen enough of those because it has every single one that was ever linked to from reddit, right?
So it knows how these go it knows. Okay third paragraph
This is when we talk about the scientist, we interview the scientist, right? Okay
First word of the scientist paragraph, dr. Obviously, right because this is the now we're in the name of the scientist
What name are we going to give?
It needs to be a name
conditioning on the fact that we have the Andes Mountains
So we need to get where we're in South America
The name probably should be Spanish or maybe Portuguese
So we get we get dr. Perez here
And then evolutionary biologist makes sense because we're talking about animals
from the University of La Paz again
This is the first sentence like when you have that first clause that introduces the scientist you always say where they're from
So we say from the University of and then university names tend to be the name of a city
What's the city where we have the Andes Mountains, so we're going to Bolivia lapaz. Perfect
And the thing that's cool about this is it's remembered all of these things that were quite a long time ago several sentences ago
Well, it hasn't remembered them. It's paid attention to them across that distance, which is impressive
But also this is encoding a bunch of understand understanding a bunch of information about the real world
Right all that was given all it knows is statistical relationships between words, but the way that it comes out to us
Is that it knows?
Where the Andes Mountains are what kind of names people in that area have what their cities are what the universities are all of those
Facts about the real world because in order to have a really good language model it turns out you have to kind of implicitly encode
information about the world because
We use language to talk about the world and knowing what's likely to come next
Requires actual real world understanding and that's something that we see in some of the other
Things that they got it to do you can see the real world understanding coming through
Let's keep going
University of a person several companions were exploring the Andes Mountains when they found a small valley with no other animals or humans peres see
We're hanging on to him. Yep. We're referring to him again
but now we've changed it to be just the surname because that's the
format that people use in news articles Peres noticed that the valley had what appeared to be a natural fountain surrounded by two peaks of
Rock and silver snow presently others, then ventured further into the valley a round about here in our article
We should have a quote from the scientist right quote
By the time we reached the top of one peak the water looked blue with some crystals on top and we're talking about this fountain
I guess it's natural fountain. We're referring back to the previous int. It's like everything is
Relying on in contingent on earlier parts of the text while examining there by snipped paragraph while examining these bizarre
Creatures the scientists discovered that the creatures also spoke some fairly regular English know when I read that I like, okay
this is now unusually good because that's the second sentence of the lead right where six paragraphs in and
It knows about this point. I've covered the first sentence of this
initial paragraph
now it's time to talk about this second sentence of the lead even more surprising to the research of us of the fact that they
spoke English and
It completely ignored the speaking English part until it got to the part of the news article where that comes in
You've gone six whole paragraphs
the idea of
Accurately remembering that the unicorn speak perfect
English is like that's very impressive to me and then it goes into its gets a little bit unhinged
Starts talking about it's likely that the only way of knowing for sure if unicorns are indeed
The descendants of a lost alien race is through DNA. That's read it really
Well, it's not actually stuff on reddit. It's stuff linked to from reddit. But yeah, this is this is news articles men
They seem to be able to communicate in English quite well
Which I believe is a sign of evolution or at least a change in social organization said the scientist
That's his evolutionary biology there. Right? Right, right. Yeah, we know here's an evolutionary biologist. So so the the
coherence of this text is
really dependent on its ability to
Condition what it's generating on
Things that it's generated a long time ago
So yeah
So it can generate really nice news articles and it can generate all kinds of text things that it anything that is
Sufficiently well represented in the original data set. So that's GPG - it's a really
Unusually powerful and like versatile
language model that can do all of these different natural language processing
Tasks without actually being trained specifically on those tasks
It's really and that's that's why it's impressive
It's not that it's a it's a brand new architecture or a brand new approach or whatever
It's just when you make these things really huge and give them tremendously large amounts of data
The results are really impressive
In the original data set. So it will it will write you the Lord of the Rings fan fiction
It will write you cake recipes if we're like, there's all kinds of examples of different samples. Here's a recipe for
Some kind of peppermint chocolate cake and it's got a bunch of different


More GPT-2, the 'writer' of Unicorn AI - Computerphile

We're six paragraphs in and it knows about this point, I've covered the first sentence of this initial paragraph
now it's time to talk about this second sentence of the lead even more surprising to the researchers than the fact that they
spoke English and
it completely ignored the speaking English part until it got to the part of the news article where that comes in and
Now it's talking about it
Which is the kind of thing that these kinds of systems would have real trouble doing I've known journalists who can't write that well
If that ties into
Something that I think is kind of fundamental to the way that people think about AI
like we used to think that you had to be really clever to be good at chess and if you could play chess then you
Were real?
Intelligence you had to be and then we realized that like playing chess to a superhuman level doesn't require something that we would call
Intelligence and it's slightly unsettling to find that like writing coherent and plausible
News prose apparently also doesn't require general intelligence
like if you just learn the statistical relationships between words
But do that really really well that seems to be enough
I mean obviously any journalist would would say that the truth is kind of important in this as well
But yeah to make it sound plausible. You're absolutely right, right
Yeah there so there's there's definitely questions about like directing this towards producing a specific article
that is correct, but
just the generating of the of the prose itself apparently requires less like
Philosophical sophistication than we thought
Then many thought anyway, I think 10 years ago
People would have a really hard time believing that
Something that's just learning from data and learning these relative probabilities could produce something this coherent
you would expect it to have all sorts of conditions and a formula and
An uncoded you would look at this and say oh this must have like a database of
names and countries
locations and
Cities and everything else because it's using that information, but it turns out all of that
Is already represented in the data set because we're talking about all these things
Here's a recipe for some kind of peppermint chocolate cake, and it's got a bunch of different completions
so you can just spit these out arbitrarily so
You know those recipe blogs where like you google the recipe for something and you go to the blog and there's like seven paragraphs
Of like my mother used to make this for me back in our home in, Indiana
I always remember sitting out on the porch with my dog
He used to you know
The important thing is I had an onion on my belt which was the style at the time and and it's doing that
It's like talking about
Different back stories. Yeah, just back story and um I wonder if anyone was trying to make any of these recipes
Yeah, it's dangerous
this one
So this is this is a recipe for meringue cookies
1 and 3/4 cups butter softened the cup of sugar and egg yolk 3 t of heavy
Critical T. What unit? Is that 3 tons? Heavy cream?
That's usually a lowercase T. I don't know what it is 3 tons of heavy cream. Let's say
three and a half to four cups of flour pinch of salt
Peppermint Jojo topping which like I have no idea what that is, but peppermint. Jo jo's is mentioned in the prompt
So one and a quarter cups powdered sugar a cup of chopped pecans a half a cup
Finely chopped mint leaves 1/2 cup chopped fresh mint about a half sheet
So it's like it doesn't quite make sense, but it's right on the edge of making sense
like we have half a cup of chopped mint leaves and
Then also half a cup of chopped fresh mint all these potentially cherry picked out of a huge number of horrendous
Right. Yes. So this is these ones specifically or not
So the unicorn one and this is something that I like and it's standard practice, but I like this the unicorn one
It says they specifically said right now we're gonna make a sample for the paper
They made 10 of them and they picked the one they liked which was this one but for the recipes here
These are not cherry picked at all. That's why they're showing 6 they just gone
This is the first six that we generated
And here they are and there so that gives you a better idea of the general quality of what it's putting out
They're all fairly sensible. I like look at this world, right?
this one comes in and says I do not substitute it with something else blah blah and then
Like this, I don't know if that is right
Here's an image or yeah
Please like this on Facebook and then it goes on I found this really cute card with cute little
Kittens on and then as your samples cut off, so it's just like next post in the blog or you know
This is why GPT-2 is so cool to me. Let's see
The first thing they tested on is a children's book test where this is. I think it's like a closed thing
Where you just have it have a data set with some children's books and then you like remove one word
And then the system has to predict which of the words is the correct
Like you give it you give it ten words that it might be or something like that and it has to pick what word fits
In this space. So that's standard kind of language model type task. The one thing that they did have to do is
they had to run analysis to check about overlap and it turns out that one of the
Children's books is the jungle book by Rudyard Kipling
Which was actually in in its entirety was in the data set that they trained the thing on so he just knew that one
So then they threw that out because I think that's not fair if you've already seen the entire book before
And it's performance on that was was good by the time they guts up to the to the very large scale models
Its scoring one. Is that eighty?
eighty nine percent where humans tend to get like
ninety 90 to 93 percent
It's like nearly a human level for guessing one missing word from a sentence in a children's book pretty good
Lambada is designed to test long-range dependencies, which is what I've been talking about a lot
The task is to predict the final word of sentences which require at least 50 tokens of context for a human to successfully predict
It's like 50 words is a pretty long sentence
and
So this kind of long-term dependency thing is a standard way of testing language models and
It ends up with an accuracy of 63 percent
Which is an improvement over state of the art by four percent. So it's state of the art on Lambada without
Being specifically trained on that just running in general the winograd schema
I don't know if it's support if you know event venal guard. Who knows
Maybe it's somebody's name
Whatever. This is about resolving ambiguities, which is really especially important in
translation tasks and things like that
it's very easy for sentences to be ambiguous in a way that
Makes translating them
Very difficult or even impossible and I have like I have an example of this check this out. So consider a sentence like
The chicken didn't cross the road because it was too black
okay, and then we can consider different versions of this sentence suppose that this is
Wide chicken didn't cross the road because it was too wide, right?
That's like one possible completion for this or you might say
The chicken didn't cross the road because it was too scared
another perfectly sensible sentence, then the question is
it
in one of these
It is referring to the chicken and in one of them. It is referring to the road color for a third and sure
stormy
Stormy. Oh, yeah. Alright. That's a good one. So stormy means it is actually neither of the things in the sentence
the other one that's fun is something like
busy
Right. Is it a busy road or did the chicken just have better things to do than crossing the road?
We don't know. I mean like I
Would say probably the road but this could be a children's book, right?
We are running this thing on children's books. The rabbit. The rabbit was too busy. It was late for an important date
why can't that she can be busy so
The point is suppose. We're trying to translate this into a language the genders everything as many languages do and
Maybe chicken is like a is a masculine noun and Road is a feminine noun and it has to know
What it's about right?
Is this like is this L or L or whatever so the idea of this benchmark is?
measuring how well it can resolve ambiguities because if this says wide if you're trying to do
Translation the old old-fashioned way
Where you're like parsing trees and looking things up in dictionaries and stuff like that
This kind of sentence is the hell nightmare because you can't you don't know. I mean, you just don't know the information is not
really present in the
Text it's not present grammatically. It's present in your understanding of the world and chickens and roads, right?
So translating this if this is it was too wide and you translate it
Into something which is the equivalent of the English sentence the chicken cross the road because the chicken was too wide you've screwed up
Right, that's a bad translation. But at the same time
Like there's nothing in the sentence that tells you that it shouldn't be right
so what you need to do is the same this the thing that we've already seen that GPT 2 is super good at
Which is pulling in knowledge of the world
Like knowing that the University of La Paz is going to be near the Andes or in the Andes right that kind of thing
It's going to know
that
Roads being wide is a thing much more than chickens being wide is a thing and so on and that like roads don't get scared
If it's scared crazy fearless, so again on this kind of thing, it does very well
It beats the state of the art by a lot. You can see on this graph here
So the way that this graph is laid out, by the way, this is the same in all of them
This is the size of the model. They made four different sizes of model. And these are like the same sizes that previous
Language models were so they were like make sense to compare them their previous to the small ones
They do worse than the state of the art
But then these seven hundred sixty two million parameter and the 1.5 billion parameter significantly past state of the art
They're getting like 70 percent. So the state of the art is the straight line across, right?
Yes
And the thing that is also kind of fun about some of these graphs is so some of them they're seven six two
million and the 1.5 billion end up doing about as well as each other, which means you've like hit the limit of I
Get maybe your data set or whatever. Whereas in this one. They're still growth which means an even bigger model
We might expect to do even better maybe reading comprehension
This is another thing you have some text you then you have to answer questions about that text, right?
the thing that's fun is
How do you do this
Without
Modifying your model. That's just a generative model
This is where we start getting into
So that's... you... by the time it's read it, it's modified itself based upon what you've given it to read? Is that what you mean?
No, what I mean is
The way that GPT 2 works is
you give it the sequence of tokens and it gives you a probability distribution for the next token and
so they're like type signature of that is
totally fine with if you're trying to fill in a missing word or
I guess I don't know how they did it for these
for this test
But you have to take you have to take the challenge that you're given and try and express it in terms of
This like predict the next word
type
setup because otherwise
You're sort of cheating, right? The whole thing is they're trying to go at this and not not modify the system at all. So
for reading comprehension
the way they do it is they
give the
thing bits to be comprehended and
then they give
Q : a question a : the correct answer to that question
newline Q : a new question they give like three or four example questions and then
Q : the question, they actually want answered a : let it generate
So they sort of prime it I think we have some examples of this
So this is how they did the question/answer thing. They gave these two paragraphs about
the
Olympic Games to torch relay moving the
Olympic torch and I have some news story and then a bunch of questions right question
What was the theme of the one world one dream and so on and so on and then at the end question and did they?
climb any mountains and then a :
Generate me the next word so they've used
This the the input text to kind of prime the model
Now we're doing question-answer pairs. This is how it works, right and
The interesting thing about this is it actually ends up giving kind of a better answer than that human generated answers
So the question did they climb any mountains the responses they got from humans were unknown. Yes. Yes, and yes because they do plain mountains
but gbg to
Dare answer is Everest. So gbt twos answer is actually kind of better than the humans. The humans just said, yes they did and
the machine learning system has named the mountain that they climbed so I don't know if that's
If that counts is not quite understanding the question or if that counts is actually providing a high-quality answer
It's up for debate because it has this ability with attention to do the long range
It has to look back past all of the previous questions
To the actual paragraph and find the relevant information and it can do it and it performs reasonably. Well at that
But the thing I love about that is that they have to like come up with tricks
To get it to actually do the task that they're trying to get it to do. So the summarization one is brilliant
I love the way they did this with the summarization. See if you can guess how they did this, right?
You want to get a summary of?
A piece of text. How do you how do you get that given a huge data set or edit?
content and what you do is
you write the whole long piece of text and then you put like a new line and then you put
TL DR too long didn't read in this data set. There will be
thousands and thousands of examples of long pieces of text followed by a short summary of that text and
In the middle is this string TL DR? I would love to have been in the room when they thought of that
So yeah, it's really
Really cool really powerful technology. I like it a lot.
So an executable binary the net effect of slotting that T diagram against here slightly downwards is to show you
That the C you've written gets converted into binary and the net output
from this process it produces out a program that you probably store in a


GPT-2: Why Didn't They Release It? - Computerphile

So, I think it's worth talking a little bit because like I'm usually talking to you about
safety
About the decision that opening I made to not release the fully trained
Model the big one
So because this has not been released we know that it works like a transformer left to its own devices without being
Fine-tuned it's just a massive amount of data and off you go. Is that right?
Yeah, like there's enough information given in the paper to reproduce it
and
you just need the giant giant data set which is a real hassle to make especially because
You really need high quality data, does it say anywhere in the paper about how long it took to train
Yes
And how and how many different how many TP use you need and stuff like that?
What's the TPU?
That's a tensor processing unit
Okay
So like a GPU, but fancy
you need a lot of money if you tried to train this with just like amazon's cloud computing offering you would be
You'd end up with a bill that I I expect would be in the hundreds of thousands of pounds like it's a lot of compute
But with all of these things it's a lot of compute to train them. It's not that much computer run
This isn't a new architecture. This isn't like a vast breakthrough
From that perspective. It's just like the same thing but much bigger and
And nobody else is keeping their research and like not releasing their models to the public
So, you know, you think it's dangerous to say that you think that your work might be dangerous and you're not releasing it
It's kind of like you think it's much more dangerous than other people's work and therefore like it's so powerful that it's dangerous
it's kind of like you're saying that your stuff is so good that
It's you know, it's too powerful for you. You know, I can't release it or whatever
I think people reacted in a sense to that
There's just smack a little bit of publicity stunt
I mean assuming it's not a publicity stunt assume rest is not that which I don't believe it is what are they worried about?
So that the worry like people make a big
People make a big deal of the Evette generating fake news like fake news
Articles that will convince people that there are actually unicorns or whatever. I don't think that that's the risk
I also don't think that that's really what opening. I thinks the risk is if you want to generate a fake thing
It's still not expensive to do that. You can just sit down and write something right
You don't need a language model to write your fake news
and
In fact, you don't have that much control over it
So you wouldn't if you were trying to actually manipulate something you would want to be tweaking it
anyway, I don't think that's the risk the the thing that
the thing that most concerns me about things like gbg 2 is
Like the content is not particularly good but it is convincingly human and so it creates a lot of potential for making fake
users
and
So there is this constant arms race between bots
Operators and the big platforms right? There's teams working at Google at YouTube at Facebook everywhere
working on identifying
Accounts that aren't real and there's various ways. You can do that
one of the things you can do is you can analyze the text that they write because the language models that are out there aren't
Very good. And so if some if if an account is like repeating itself a lot
Or you have a whole bunch of accounts that are all saying like exactly the same thing
Then you know that this is like a spam maybe manipulation attempt and so on
But with GPT, too
You can have things that produce
you give the same prompt and then you post all of the outputs and all of those outputs are different from each other and
They all look like they were written by a human
and it's not a
Human can look at them probably and figure out
Hang on a second. This doesn't quite seem right
But only if you're really really paying attention
which
Human attention on the large scale is super expensive right so much more expensive than the compute needed to generate the samples
So you're outmatched if you if you spend more they can spend it you can you can spend
10 times more and you cripple yourself financially and they can spend 10 times more and it's fine. So you're gonna lose that battle
The other thing is so it becomes very difficult to identify fake users. The other thing is one way that you can identify fake users
is by
Analyzing the graph like the social graph or the interaction graph and you can see that
Because
Humans, usually when they see spam posts that are full of links to dubious websites and whatever
They download them. They don't reply to them and
You can create you can fake the voting metrics by having these accounts vote for each other's stuff
But then you can analyze the graph of that and say oh all of these plate people
They all only vote for each other and the people who we know are humans like never vote for them
So we assume those are all bots and we can ignore them
But the samples that gbt to produces the big model are convincing enough to get actual humans to engage with them
Right. It's not like oh my god, that's so persuasive. I've read this article and now I believe this thing about unicorns
It's just like I believe that a real human wrote this thing and now I want to argue with them
That there aren't unicorns or whatever, right?
And now you have real humans engaging in actual meaningful conversation with BOTS
and
Now you've got a real problem because how are you going to spot who the bots are?
When you can't do it automatically just by analyzing the text
You can't even do it by aggregating the human responses to them because the humans keep thinking that they're actual humans
so now you have the ability to produce large amounts of
fake users that the platforms can't spot and
therefore they can't stop those users votes from counting on things up voting things and down voting things and liking them and
subscriptions and everything else and maybe plating the metrics that way one thing people would do is spot the
Their profile pictures if you're trying to generate a large number of BOTS
where are you going to get your pictures from and so you can do like reverse image search and get the
Find of it and they're all using the same picture or they're all using pictures from the same database of facial photos or whatever
Now we have these really good generative adversarial networks that can generate good-looking cases
So that's now really difficult as well and like you can't automatically detect those
Almost by definition because the way the gams work the discriminator is like a state-of-the-art fake face image detector
and it's being fooled like that's the whole point and if you released
If somebody came up with a really reliable way of spotting those fake images then
You can just use that as the discriminator and keep training
right
so not releasing their full strength model to me feels
Very sensible in the sense that people will figure it out, right they published the the science
Someone will find it. It is worth their while to do it to spend the money to reproduce these results, but
By not releasing it. They've bought the platform's
Several months to like prepare for this to understand what's going on and they are of course
Working with them and sharing their full strength model with selected partners people. They trust to say here's what it can do
Take a moment
You know govern yourself accordingly like get ready because this stuff is going to come but they're giving everybody a heads up to
Mitigate the potential like negative impacts that this work might have and the other thing is it sets a really good precedent
I think
Because maybe GPG - isn't that dangerous?
but the stuff that we're making is just getting more and more powerful and
at some point somebody is going to develop something that is really dangerous and by then you want there to be
accepted
practices and social norms and industry standards
About thinking about the impact of your work before you release it
and
So it's good to start with something that like there's some argument that there could be some danger from it
just so that everybody is like aware that this is a thing that you can do and
that people won't think you're weird or you're bragging or it's a publicity stunt or whatever to make it like socially okay to say
we found this cool result and we're not going to put it out there because we're not sure about the safety of it and
I think that that's something that's really really necessary. So I think that open AI is very smart to
Start that off now
For we really really need it. I
Make a principled
decision now
I want the seven
so in principle
I should be going this way right and would think I'd want to steer towards the seven but on the other hand at
This point it's your choice. You give it some random noise and it generates an image
From that noise and the idea is its supposed


Stuart Russell (Full Interview)

so my name is 2a Russell I'm professor
of computer science at UC Berkeley my
general area is artificial intelligence
and right now I'm mainly focused on the
problem of control so how do we retain
control over increasingly intelligent AI
systems
yeah so this is a an idea a diagnosis
for what everyone is aware of which is
that social media seemed to have driven
lots of people into a politically
polarized state and if you think about
that from the point of view of the
people designing the content selection
algorithms they add selection algorithms
that choose what it is you're going to
see you might think well okay they know
that it doesn't make sense to send you
stuff you're not interested in so we
just do some kind of learning algorithm
that learns what you're interested in
and then it starts sending you more of
that but if you have an AI system for
example a reinforcement learning
algorithm which is designed to optimize
an objective and here the objective is
click through right what fraction of the
things that we send you do you click on
because that way you make ad revenue and
so on then that's not the solution just
sending you stuff you like is not the
best way to maximize revenue the best
way to maximize revenue is to modify you
into someone who's more predictable
because the more predictable you are the
higher percentage of things I send you
you're going to click on so a simple
reinforcement learning algorithm would
learn how to do that by sending you
content that is let's say a little bit
more extreme than your current political
views and then if you see enough of that
your political views will become a
little bit more extreme and then this
process iterates until you're way at one
end or the other end of the spectrum we
might say the newer fascist and North
Berkeley end and and then you know when
when you're an extremist you're very
predictable and we can generate more
revenue from you and so this was
probably completely inadvertent I don't
think Facebook set out to do these
things and it may not even be explicit
in the code but in the combination of
the code plus the way the marketing
departments
right the revenue generation departments
how they continually modify the
algorithms and so on so so some
combination of these things it
effectively operates as a radicalize er
for billions of people and at the same
time then you know whole industries have
sprung up to feed these newly arrived
extremists with more and more content
I think so so the objective of
maximizing click-through in the short
term is possibly not really the
objective that Facebook should have been
optimizing and even though the
algorithms are fairly stupid I mean they
don't even know that human beings exist
let alone that human beings have
political opinions but they're still
very effective because of the way they
can customize and adjust the feed to the
way you are responding to that feed and
and gradually move you in one direction
or the other that that's an unexpected
consequence and we didn't want to break
up NATO or destroy Western democracy but
we kind of did that's what happened and
if you had systems that were much more
intelligent than that and you gave them
even more innocuous objectives you still
might have these major catastrophic
consequences so the the upshot is to to
really ask the question is this the
right way to build AI systems that we we
create this optimizing machinery which
can be incredibly effective and then we
simply download an objective and then
off they go because if we don't specify
the objective correctly then the
consequences can be arbitrarily bad
well I think it is a bit of a
misconception I mean it's a valid point
that intelligent humans who have your
best interests at heart can understand
when you ask for something you know
incorrectly right they you know if you
ask your assistant - okay booked this
conference room for a meeting at this
time you know it's quite reasonable for
your assistants say but you're already
booked for that time right so the the
requests that you made is not actually
the right request you gave the wrong
objective but the assistant has your
best interests at heart let's say and so
the question is how you make sure that
this super intelligent AI system has
your best interests at heart and that's
the unstated assumption here what if it
doesn't right what if we didn't manage
to get it sort of constitutionally
obliged to keep your best interest at
the center of what it does then even if
it does understand what you want it's
under no obligation to actually help you
carry that out right and that's sort of
the that's the missing piece so there's
really I mean it's an old idea Nick
Bostrom calls it the orthogonality
thesis that you could have arbitrary and
intelligence connected with any
objective you know and you can have a
chess program that plays ordinary chess
really well or you could say you know I
want you to play suicide yes now now
play suicide just really well right well
you have to lose all your pieces and so
you can you know with a self-driving car
or an uber you know you can say well
take me to this Airport or take me to
that Airport so the goal is not built in
to the system they at askable systems
and how do we make sure that this super
intelligent system is tasked correctly
with human interests as its as a
fundamental objective and this goes back
to you know at least to Hume who said
you know you can't just tell what the
objective is by looking at the world
right that's all ideas a
naturalistic fallacy and so unless
there's something which constitutionally
obliges a super-intelligent system to
favor your interest then it doesn't have
to pay attention even if it's great at
perceiving what you want what why does
that force it to actually do anything
about it
well it's certainly related I think
often when people hear the phrase value
alignment they think that the the the
goal is to build an AI system whose
values are aligned with those of humans
and I think that leads to two
misconceptions one is that it's the AI
the AI system is kind of adopting the
values right so if you're a vegetarian
family and you buy this domestic robot
that's going to cook food for you you
want it to be a vegetarian even though
it doesn't eat you wanted to have
vegetarian values so they're only cooks
vegetarian food that's not the right way
of thinking about it the right way of
thinking about it is you want it to
understand what your values are but you
know if your friend next-door borrows it
to do a barbecue you know with lots of
ribs and steaks one weekend when you're
away that's fine right it's not going to
have a you know a real crisis of
conscience about cooking ribs for the
next-door neighbor because it's not
adopting the values it's simply learning
to predict the Preferences of ultimately
you know all seven billion of us right
and we can all be different and that's
fine we it can maintain seven billion
preference models I mean Facebook
already does so that's that's fine the
other thing is that we absolutely do not
expect the machines to have complete and
correct models of the Preferences of the
people that it's on whose behalf is
working you're always going to be
dealing with fundamental uncertainty
about the true preferences of the
individual and yet you still need to be
useful and one important point is that
if you're uncertain about the
individuals preferences then it turns
out that you're necessarily deferential
to them because you know that they know
more about their preferences than you do
which means that if they want to switch
you off that's because you're about to
do something that they don't like even
if you don't know that they
like it right and so you're quite happy
to be switched off if you believe you
know the objective perfectly then any
attempt to switch you off would just be
a mistake and therefore to be prevented
and so there's this absolutely necessary
mathematical connection between
uncertainty about preferences and the
deferential behavior of the machine
yeah so let me begin by explaining this
idea of instrumental convergence or
instrumental goals so if I ask you to
fetch the coffee it's perfectly obvious
to you that if you're dead you can't
fetch the coffee so by asking you to
fetch the coffee I have given you
incentive to stay alive right so
sometimes people say oh well we just
don't have to build in these instincts
like self-preservation or you know
acquisitiveness or how you know desire
for power or those things the point is
no yes you don't have to build men
they're going to happen automatically as
a consequence of machines having pretty
much any objective at all so you know if
if I ask the machine to cure cancer
well you know that's going to require a
lot of money so now you've given it the
incentive to acquire lots of money lots
of computing power to gain physical
control over lots of potential
experimental subjects and so on so so
the the bad consequences are not coming
from the fact that we we built in this
this world power or all these atavistic
tendencies that that we think of humans
are suffering from but just these
instrumental goals the the goals that
are useful to have for pretty much any
specific objective information money
computing power and physical control
right all of these things are useful for
for pretty much all objectives so the
the thing you need then need to do is to
make sure that the machine isn't just
pursuing an objective like to fetch the
coffee as a single goal this is its
life's mission and nothing else matters
it has to understand as a human does
that there's all kinds of other
constraints and trade-offs in operation
so yeah if you want to cure cancer
getting some research funding but that
doesn't mean taking over the whole world
economy yeah you may want to get some
volunteers for clinical trials but that
doesn't mean injecting you know tumor
causing materials into all eight billion
people on the earth so obviously humans
don't need to be told that but if you
don't tell machines these things they
won't know them and so either you need
to put all these things in or the
Machine needs to know that it doesn't
know them and therefore ask questions
like is it okay if I inject or all
humans with tumor causing material or
not
okay so the the three principals and I
want to emphasize these are principles
guiding how we do the research and
design of the systems they're not like
things that the AI system has to consult
at every instance and then follow some
logic to decide what did I do based on
the principles right so it's not right
they're not built into my positronic
brain or anything so the first principle
is that the only objective of the
machine is human well-being or
satisfaction of human preferences so in
particular you you don't want anything
else like self-preservation or any of
those things the second principle is
that the machine is generally in a
situation of not knowing what what those
are what the human preferences are so
uncertainty about the objective that the
machine is pursuing is fundamental and
then the third principle is is what
connects humans to the machines
understanding of preferences it's the
fact that human behavior provides
evidence of human preferences and this
there's third principle I think is the
one that's most difficult because the
connection between human behavior and
human preferences is a complicated one
right we we believe that our our
behavior is sort of driven by what we
want the future to be like but we know
that we're not particularly good at it
we make mistakes we do things that we
regret we think in short-term ways and
mess up our long-term goals and and so
on so when you observe human behavior
you have to sort of reverse-engineer out
all the imperfections all the
complications of our cognitive structure
and and that's something that's very
difficult to do
so okay number one there is no there is
no single human utility function right
so every human is entitled to their own
preferences about what they want the
future to be like and and let me be
clear on this when I say human
preference is I don't mean like do you
prefer plain pizza to pineapple pizza
I mean preferences between two as it
were completely specified futures that
include all the events that you could
ever care about that would make this
make you prefer one future over another
and that's that that's a hypothetical
contract right I can't even show you
those two futures in detail but you
could maybe imagine that I could make
you a 2-hour movie of each of them and
you could kind of watch that in full 3d
sense around and sort of live that life
and in you know in for most pairs of
futures you'll be pretty clear within
the first few minutes which one you like
best and then there will be some where
you might say well I this is a summary
of those two futures but I still I'm not
really sure which one I like best
now if that's the case then you know I
think we're already to the point where
the AI system is unlikely to induce a
catastrophe because if it's understood
your preference is at least up to that
degree of precision then you're probably
you know we're probably out of the woods
in terms of avoiding major disasters now
as I said that at any given point in the
in the interaction between a machine and
a particular human there will still be
radical uncertainty about the full human
preference structure and then there are
some additional complications to do with
the fact that your preferences are
plastic that they change over time and
we want to avoid the failure mode
whereby the AI system simply changes
your preferences to be easier to satisfy
rather than helping you satisfy your
real preferences and certainly why
you're heading is one example so why
heading is the idea that if you give
someone a button and a wire and the wire
connects to the pleasure centers of the
brain you know and this has been tested
with animals up to the point of death
and with humans up to the point of
exhaustion you know and complete neglect
of personal hygiene for the you know 24
hours it would just keep doing it and
probably we would keep doing it until we
died as well so the why is that a bad
thing right you are clearly expressing
your preferences by pressing this button
this is because evolution is built into
us this reward system which is mostly
helpful right our dopamine system it's
mostly helpful in pursuing nutritious
foods and avoiding pain and and so on
reproducing but it's not a perfect
signpost for evolutionary fitness and
it's certainly not a perfect signpost
for how we think humans should should
live in an advanced civilization and so
we learn we having all these cultural
constraints that keep our dopamine
system under control and we we let our
cognitive system do a lot more of the
decision-making but it's you know we
still have this you know these ways of
shortcutting and our dopamine system is
still really strong and you know there
are some species apparently there's a
species of sloth which is addicted to
its food supply to such an extent that
it doesn't bother reproducing anymore
because it's locked out the whole time I
don't know if this is really true maybe
as a urban legend but it seems like that
species is going extinct because of its
dopamine system of being not quite
aligned with a reproductive fitness and
so one of the things that AI systems
have to do is is not just this sort of
very gross short cutting via the wire
heading but but to help us avoid our
general inability to act in our own best
interests
yeah it's an interesting question that
you know we a lot of things that we kind
of take for granted just normal parts of
civilized behavior when you unpick them
and and sociologists and good at this
right they love to unpick normal
behavior and say okay what's really
going on you know i for example the the
the standard view roughly speaking in
you know it feels like welfare economics
is that there's there's your personal
well-being like you do you have adequate
food shelter warmth company and so on
and then there's your other regarding
preferences like what do you what do you
think about you know are you happy that
other people are suffering or you have
you unhappy that they're suffering so we
know what is the sign of your altruism
coefficient so to speak you know and
that's a fairly simple model and
obviously at least it seems obvious that
we want people to have a positive alt
room altruism coefficient not a negative
one so that we don't need to be sadist
but then you look at what what are
called positional goods so positional
good is a term from economics which
which means a good that has not just
intrinsic value but it has value because
of its relative quality compared to the
goods that other people have so not just
that I have this nice shiny car but I
have a nicer shinier car than my
next-door neighbor right and where cuz
that that sounds you know an unpleasant
kind of attitude to have but then you
know your pride in a know in winning a
Nobel Prize is also to some extent well
I have this Nobel Prize and you don't
write if everyone had a Nobel Prize we
wouldn't we wouldn't value the Nobel
Prize very much you know when people
complain about this in in preschool
where everyone gets the prize that the
price becomes meaningless
so the value of the prize is precisely
as a positional good
and so I went the more you think about
the more you realize you know a big
chunk of our self-esteem and our
feelings of achievement and success and
so on are really positional goods and
positional goods act exactly like sadism
meaning that they're sort of based on
the difference between my intrinsic
well-being and your intrinsic well-being
and so I can introduce that difference
by decreasing your well-being right I
can be better off if you're worse off
and that's so it it's a negative
altruism coefficient so it's it's it's
possibly something that we might want a
I systems or AI systems might gradually
try to reduce in our society because it
has all kinds of negative side effects
right it just really you end up with a
kind of a negative sum game that we're
playing with each other
but that would be a huge change right
and maybe we're just not ready for that
all right
maybe just our culture and upbringing in
the whole way of thinking is around this
notion of relative success and it would
be too much of a change to have that
happen have that removed all at once so
it'd have to be a very very careful
process to do that
yeah I I think generally you know that
there's a term called preference
autonomy which comes from Hassan Yi
who's economist former he died a few
years ago but an economist at Berkeley
so his view which is actually in
conflict I think with a lot of the early
utilitarians is that what people prefer
is entirely up to them and you shouldn't
be saying that they should prefer
happiness so that they should prefer
wealth or they should prefer a long life
or short life or or anything right it's
just what they want people want you know
and as a zeroth-order theory that that
seems right the drawback of course is
that well what they prefer is you know
there's not something they're born with
you know it's not something that came
down from heaven it's the result of
their upbringing their peers their
social context their culture and so
there's a whole bunch of processes that
are already shaping human preferences
and AI systems can't help but shape
human preferences it's inconceivable
that we could have you know we that we
could give everyone an incredibly useful
personal servant without changing the
way they think about things right how do
we make sure they don't all become
spoiled and lazy for example and you
know that's and that's another important
part of understanding what the right
relationship is between humans and
machines a simple-minded view of
preferences is that you have preferences
for things like you know how much income
you have how you know how tidy your
house is what kinds of food you have in
the fridge and so on but it's not just
that right it's also did you actually
make that income right is that dish
that's in the fridge something that you
personally cooked as opposed to it just
being in the fridge so so there's notion
of autonomy and
and personal struggle and it's and
sometimes failure and sometimes you mean
that's a really important component so
we have to make sure that the machines
understand that and don't simply become
this sort of all-encompassing care
system that reduces us to in an
infantile mind state which is what you
see you so Wally yeah so so in Wally all
the humans are on this big spaceship
journey because the earth has has been
messed up and they're on it for
generation off generation and they
gradually become completely in feeble
they become obese you know they can't
even get up out of there lounges and
they're they're completely unable to run
their own civilization and that is a
that's a serious consideration because
you know up to now if you add up all the
people who've lived and how much time
they've spent learning what the previous
generation news that they could continue
it's about a trillion person use and
that so that's what we spend to keep our
civilization going and if we if we
wanted to you know we put a lot of it on
paper but the paper doesn't run our
civilization for us right it has to get
into the next generations mind in order
for our civilization to function but
that's no longer true right if we can
put that into the minds of the machines
instead of the minds of the next
generation we can run our civilization
without all this effort of learning and
that will be pretty much irreversible
and I think that's what Wally's in its
subversive cartoony way showing and how
we have the incentive so the machines
might say you know this is not good for
you all right you need to know how to
run your own civilization you know kind
of the way that parents say no you need
to tie your own shoelaces I'm not gonna
keep tying them for you
but we just have you know where we are
short-term thinkers and way of myopic
and so we were we may kind of override
our own machines advice and say no no we
we want you to do this we're gonna
design you to take over all these
functions because you know it's it's not
worth our while tis to spend 14 years in
school learning to be a surgeon or
learning to run the electricity grid or
whatever because machines are better at
so at the risk of sounding even more
productive I actually think that the
word values is the wrong word because it
tends to connote morality moral values
and when we talk about morality we're
usually talking about things that are
you know moral dilemmas with you know
difficult decisions where there is some
difficult trade-off to be made and
sometimes it's self versus other and
that's not really the major concern
so another provocative way of putting it
is that the machine could get all the
moral dilemmas in the world wrong
whatever that means because in some
sense if it's a dilemma it doesn't have
a wrong answer but let's say you know it
uses the the answer that you wouldn't
use on every single case it's still not
going to be catastrophic because the
moral dilemmas are precisely those where
there are good arguments on both sides
for why you should do one thing or the
other but you know a machine that just
goes around chopping everyone's legs off
for the fun of it that's not a moral
dilemma right you know there are no
moral philosophy classes discussing
whether that's a good idea or not right
but you know a simple point the none of
the self-driving cars that are currently
under development know that human beings
prefer to be alive so we are we are a
long way from having to resolve
difficult moral dilemmas about test-tube
fertilization or whatever it might be
and so you know and also when you say
the word values the first question
people seem to ask well whose values and
or even you know what makes you you know
well-off Western white male you know
entitled to to put the values in and the
answer is no I'm not putting the values
in we're not the thing we're putting in
is that machine's should act on behalf
of what humans want their future to be
like
so if we think about this connection
between underlying human preferences if
they exist and I think that's also
something that psychologists can help
with and behavior you know that that's
sort of really in one conception that is
our whole cognitive architecture right
our preference is the kind of the
wellspring of our behavior but then
there's all of the learning perception
decision-making and so on that in the
end produces the behavior so if you had
to say well you know we can't really
make progress until we have that
complete model and then we can kind of
invert it so that when we see a behavior
we can infer what are the underlying
preferences of this person that's a
really ambitious long-term project and
we have to wait a long time before we
had finished it if that's what we have
to wait for so I think you have to look
at what are the major deviations from
rational behavior and here I think there
are things we can do so for example we
know that human beings are myopic right
so you know when when Lisa doll lost the
game of go to alphago right he made a
move that lost the game at some point
right I mean he made a move after which
there was no possibility of winning but
did he do that on purpose right did you
know if you thought he was rational then
you'd have to assume that he wanted to
lose but of course we don't think that
way we think he's trying to win but his
computational limitations mean that he
can't always do that perfectly so it's
perfectly obvious to us that people
don't have to be perfectly rational but
you can still figure out what they want
the the biggest deviation I think is
that at any given moment when when a
human being is
behaving they're behaving within a very
restricted context right so here I'm in
the restricted context of I'm doing an
interview so you know there are lots of
other things that people can do but none
of them are even in my decision frame
right now right I mean I could I could
set fire to the building and claim the
insurance money or who knows what right
I mean I could take by I could take my
phone out and start trading stocks in
the middle of the interview just but I'm
in an interview so there's only certain
things that one does in interview and
then well and that interview itself is
part of a larger commitment to this
particular you know Center and the
research project and so on so in any
given moment we are embedded in actually
several concurrent hierarchies of
commitments and activities and if you
want to understand what someone does in
terms of their preferences
you have to understand what that
hierarchy is right what a you know that
there are short term goals within that
particular decision frame but those
short-term goals exist because of a
larger frame and larger frame and
somewhere up there the the real
underlying preference is about our
long-term future and we may not even be
at all consciously aware of the
connection between those so for an AI
system to understand what people are
doing you know even a system you know
the relatively restricted system that
you know helps you with your calendar or
something like that and it needs to
understand what are the activities you
are engaged in you know and so where are
you right now in that subroutine so to
speak and so I think this is something
that psychologists can can really help
to study right what is the what is the
nature of this hierarchy of commitments
how does it vary across individuals how
do we pop in and out you know when do we
decide to abort one commitment and
replace it with another what's the
control structure and so on so these are
these are questions that can be really
helpful II explored with real human
beings
yeah so the the doctor
evil problem simply refers to the fact
that even if we do design safe and
beneficial AI that has all these nice
properties for some people there's no
incentive to use it and so dr. evil
wants to take over the world he doesn't
want the AI paying attention to the
Preferences of all those other people so
he has an incentive to shortcut the
kinds of safety catches if you like or
to just bypass those issues in a in a
development process because he wants the
capabilities of the ISIS AI system and
you know the the failure mode is then of
course that you know not that he would
succeed in taking over the world but
that he would fail in the sense that he
loses control of the AI system and then
the consequences are arbitrarily bad and
you know I don't have a very good answer
for this problem because you know there
are a lot of dr. Evil's out there some
of whom don't even think of themselves
as evil but nonetheless they have a
sense of a mission that doesn't Brook
any kind of objection and we have a
problem already with malware and this
would be malware on double super-secret
steroids it would be yes you know a much
worse problem and you know why we why
are we not having as much success as we
might like with containing and
preventing malware
some people might describe it as a
complete utter catastrophe that's
getting worse and worse partly because
of the original design of the internet
just didn't really think about security
or authentication traceability or all of
those properties that we all wish we had
and partly because there are countries
that have no interest in assisting with
the process of cleaning up the Internet
so
I think you know there are there are
good reasons to do this anyway but if
you wanted to sort of ward off the dr.
evil problem with more intelligent AI in
the future solving the malware problem
now by creating much stronger
international cooperative agreements and
forensic processes and policing would be
a really good step and most countries in
the world and pretty much all
corporations in the world would get
behind this
the Somalia problem is based on a little
story where you know you have a domestic
robot who looks after your house and you
come home one day and you know you're
really tired you didn't have any time
for lunch and you're really hungry and
you asked the smile you ask you a
domestic robot what's for dinner and the
robot says well I has something I have
to tell you there are people in Somalia
who are dying dying of starvation so I'm
going to help them instead of you so I'm
leaving now make your own dinner and you
know that's a consequence of building
robots that are trying to satisfy the
you know the typical utilitarian
recommendation of satisfying the sum of
human preferences and it can do the most
good by helping the people who are them
the worst off situation where you know
they can prevent them from dying you
know get them back on on a positive
track that will be much more beneficial
than just making you dinner when you
could make it yourself now the
difficulty with that I mean you might
think wow I'm really proud of that robot
but now I'm wondering why I shelled out
$80,000 for it of course you wouldn't
buy it right so and then if you didn't
buy it it wouldn't be if you didn't buy
the robot it wouldn't be available to go
and help people in Somalia
anyway so we kind of self-defeating so
the Somalia problem is how do you design
the incentives of the robots so the
first principle says we help with human
preferences but more specifically ok how
exactly do you deal with the
relationship between the preferences of
the owner of the robot and the
Preferences of everybody else
you can't just satisfy the preferences
of the owner because then you have a
robot that behaves in all kinds of
antisocial ways which gained the little
bit for the owner you know
what maybe the robot even just steals
money from people to give to the owner
you have to have some some term for the
Preferences of other people but if if
you really do treat everyone equally
then the robot won't be of any use to
the owner it'll go off and do do good
around the world so presumably the the
degree of loyalty it has to the owner
has something to do with the purchase
price of the robot right it sort of owes
you at least that much otherwise you
won't buy it in the first place that
sort of seems right but I think it's
more complicated than that you know and
maybe also you know if if the world
wants your robot to go around doing good
then maybe the world should cover some
of the cost to the robot in the first
place so so you might be able to get a
subsidized robot but understand that
your robot may go off and do something
more important for the public good from
time to time so we'll have to see how it
works out and I you know economists are
good at solving these kinds of problems
so here's what I think about that you
know as a as a practical matter if I was
a manufacturer of autonomous vehicles
I'm imagining the first liability trial
where my car has made one of these
decisions and rather than having to
defend to the jury why the car decided
to run over the little old lady or the
to school school children in a pram
I'd much rather want to say well we
implemented the policy that was
stipulated by the democratically elected
government so I think that's how I would
deal with self-driving cars and these
life and death sort of death versus
death decision-making but I think in in
a much more practical case you know I
think the trolley problems or might
never arise in the next hundred years in
practice but problems like okay when
when you're in a period of high
congestion do you take the shortest
route which then causes more congestion
on that route or are you willing to take
a longer route and maybe save some money
right maybe you're gonna lower toll or
something like that if that relieves
congestion on the short route that some
people need to get to work so all those
kinds of negotiations would be happening
all the time and how you set that up and
how that's built into the price of the
car or some kind of subscription service
you have the route choice or that that
remains to be seen but it'll be
hopefully it'll give very flexible ways
of adjusting to people's preferences
like you know though this morning I
really do need to get to work early so
I'm not taking that route and so on
well so the stay in a standard model
right
there's a definite objective right so
standard model of AI is we make
optimizing machinery and we just put in
a definite objective so it's a special
case of the approach that I'm proposing
right weights the special case where all
the uncertainty in the second principle
has been squeezed out and disappears so
now we have a point objective where the
Machine believes it knows for sure that
this is the thing to pursue and that can
be okay in limited context with machines
of limited cap capacity so you know you
might think fine if we're just writing a
go program then the goal of go program
should be to win the game okay yeah what
could possibly go wrong right it's just
you know it's all it's just making moves
on a simulated go board and that's it
but it's not quite it right alphago is
not that intelligent right although it
does a good job of learning to play go
moves it's not doing what we do right
we're intelligent beings you know we
want food and reproduction and various
other things but through a very kind of
indirect derivative of those goals we
also want to understand the universe we
want to in fact understand how our own
brains are producing this behavior and a
sufficiently intelligent alphago would
also want to do that in the hope that it
might be able to gain a better
understanding of what you know what is
the source of these other go moves that
keep appearing on the board why do they
appear how are they generated how my own
moves generated right just as how my own
force generated by my brain all those
questions have utility for winning which
is alphago's objective eventually it
will figure out that there is an outside
world you know larger than the go board
it'll figure out that its operating on
some kind of device that generates all
of these
mutations and under some other entities
out there and then it can start messing
with things right you can start
communicating by forming patterns on the
go board maybe the the other entity will
see these and then you can start a
conversation it's actually very hard
with a machine of unlimited intelligence
it's very hard to imagine you know a
narrow scope problem with a very simple
goal like winning the game ago that is
still guaranteed to be safe
on the question of low probability right
I've seen this argument for example that
you know the mathematically it's
possible that a you know black hole
could materialize in near Earth orbit
and swallow swallow the world but we
don't spend a lot of resources worrying
about that and you know the natural
response is but if all the world's
physicists were cooperating on a massive
project to make that black hole appear
in near Earth orbit when you ask them if
that was a good idea and try some try
and figure out if it was safe and maybe
it's not safe to prevent it and that's a
situation that we're in that the the
world's AI community the major
governments in the world the largest
corporations in the world are all
working towards the goal of creating
human-level AI I mean it's stepwise
incremental progress but it's actually
you know pretty rapid progress right now
so you'll you know you could stand up
there and say actually I know that all
these governments all these corporations
all these AI researchers have been
working on this for decades and decades
are all wrong
that isn't completely impossible that
there is no physically possible
arrangement of atoms in the universe
that could be more intelligent than a
human being okay if you believe that but
it's bad policy i think to stake the
future of human race on the correctness
of that belief it you know there's no
evidence for it whatsoever so we know
the overpopulation on mars argument
again you know if if the whole world
were engaged in a project to move the
human race to mars wouldn't it make
sense to us what are we going to breathe
when we get there and you know and
that's exactly what people are working
on right there what is the carrying
capacity of Mars zero right so if you
put one person on Mars its overpopulated
so the people who are working on
creating a Mars colony are precisely
worrying about over pop
raishin they want to create life-support
systems so that the Mars carrying
capacity is more than zero and that's
what we're doing we're saying if we
succeed we're going to need to be able
to know how to control these systems
otherwise we'll have a catastrophe
yeah I think that's a good question and
I think it depends on how young and what
kind of time scale in the near and
medium term there's clearly going to be
enormous demand for AI researchers robot
engineers and so on but the the
difficulty there is that they'll still
be a high threshold for entry into these
professions in terms of Education
mathematical preparation and so on and
the number of jobs is not going to be
close to filling the gap if most of the
jobs that we currently have go away and
you know a simple way of thinking is
that we've used most of the human races
robots for the last ten thousand years
or so and now that period is coming to
an end and we have to have a different
way of doing things so if you think that
okay most routine mental and physical
labor will be done by machines then the
likely you know what is the comparative
advantage of human beings well it's that
the human that's the thing we have left
and there are things that humans want
humans to do including things like
helping with bringing up children you
know having lunch various kinds of
coaching inspiration and consolation
companionship and maybe other kinds of
we might call these psychological
services if you want all the current
ways we have of talking about this you
know the caring professions they all I
mean they sound good from the point of
view that provider oh you're a carer
that's great
but they sort of have this connotation
of dependency for the recipient and that
probably is not the right way to to
think about it what we what we need to
be able to do is to tomorrow think of
this it's almost like an engineering
discipline that
we want psychology and the humanities to
be a bit more like engineering is that
we actually we have a lot of basic
science on which we can build
predictable methods and training
programs to teach people of methods just
as we have for surgeons right we we do a
lot of basic science to support what
surgeons do and we train them and and
you know when they fix your broken leg
it works and that's great and that's why
they get paid you know if most the time
didn't work we wouldn't pay them and
that's sort of the situation with a lot
of these these professions right now all
these sort of softer professions and
that's just really just a consequence of
number one the difficulty the problem
that the human mind is really really
complicated you know we don't know how
to fix it very well compared to the body
and and just the emphasis of our
technological civilization that we we
put resources and people into the hard
sciences you know the sciences that
produce a cellphone for example we
probably you know if you sort of work
back down the tree maybe it's a trillion
dollars maybe it's ten trillion dollars
in in research and development effort in
many different disciplines to produce
the cell phone but we haven't put at you
know not even 1% of that into how do you
make make it possible for someone to
have a rich and fulfilling life right
and that science and that engineering
and those professions are going to be
the future