Peter Eckersley | Setting Priorities in Addressing AI Risk | VISION WEEKEND 2019

so Alison asked me to talk about
priorities in addressing AI risk before
I do that I'm going to just mention I
work at the partnership on AI it's an
independent non-profit but created by
the tech companies and we have order of
a hundred partner organizations this
talk importantly does not represent the
views of necessarily of PII and
definitely not of our partner
organizations take these as individual
views so thinking about AI risk there
are different ways of slicing things
short versus long term is a familiar
frame technical problems versus problems
within particular institutions versus
large-scale problems in our culture
politics or economics and then the
difference between accidents or local
unintended consequences malicious uses
or misguided uses and then large-scale
cross interactions between systemic
effects that people couldn't have
anticipated or can't control thinking
about the short versus long term
distinction there are many similarities
both of these sets of problems can
involve misspecified objectives
corruptable objectives side effects from
the thing you're trying to do having the
wrong incentives whether at an
institutional level or at an individual
level the short term problems are
interesting not just because they're
serious and urgent and many of them are
but also because they're practice since
we're at a four side event they're
practice for longer term larger scale
problems and if you're more of a short
term missed or more of a here-and-now
person you should really put value
nonetheless on accurate foresight
because that's gonna let people plan
ahead better and mitigate problems down
the road so our advice is work on both
we're trying to work on both some
examples of the short-term problems that
are really high stakes from AI right now
include premature deployments of machine
learning and high risks or high stakes
settings where the technology isn't
ready the use of recommender bandid
algorithms whether for advertising or
social media that we're seeing having a
potentially destabilizing effect on
politics or on psychology and
problematic labor and economic
consequences from really fast deployment
of new technology
to take one of those examples in detail
in the state of California last year a
bill was passed mandating that every
county in the state should purchase and
use machine learning or algorithmic risk
assessment tools to decide whether to
detain or release every single criminal
defendant prior to trial and this is in
the context of mass incarceration in the
United States which is truly an enormous
problem and reformers here we're really
trying to get the United States back in
line with other countries and with with
historical norms reduce incarceration by
moving from a punitive system to one
that's more based on evidence and so
what they were doing is collecting
survey data about people's lives that
criminal histories their circumstances
and then trying to predict Oh will you
reoffending our trial or where you fail
to appear for a court date and this
these decisions in California will
affect 63,000 people per night across
the United States is closer to a half a
million now it turns out there are some
serious problems with these systems that
are being deployed though they may be
well intentioned one problem that's been
flagged very loudly is that sometimes it
seems the african-american defendants
get high risk scores in ways that don't
make sense in comparison to their
Caucasian
comparators and in fact if you look
statistically you see that the false
positive rate the odds of being labeled
as dangerous given that you're not
dangerous almost twice as high if you
are african-american than if you're
white so this is a very serious problem
RPI has been working on mitigations
including a report that we produced with
our partner organizations with 10
recommendations of things you need to
fix in the statistics and machine
learning and the institutional setting
if it were ever to be appropriate to do
this kind of deployment efforts on
transparency so the institutions
building and and purchasing procuring
these technologies can understand
serious problems like the fact that
they're trained on 10 and 20 year old
data that includes for instance
marijuana arrests in a state like
California where marijuana possession is
no longer a crime so predicting riah
fence based on that is very problematic
and then largest-scale mitigations that
I'm not specifically about this criminal
justice problem but about the field as a
whole
the use of documentation and
transparency through a project called
about FML that how should help the AI
community tackle similar questions
wherever they arise in the AI industry
so this is a project being led by Jing
young yang who's here today and it is
trying to gather initiatives across lots
of different organizations to produce
documentation and transparency about the
way the pipeline the way that a machine
logic project goes from specification to
data set to model to deployment and get
people to pause and ask themselves the
right questions about failure modes
along the way so looking at this this
particular problem it's short term it's
about local on a tentative consequences
but there are mitigations in all of
these categories I'm going to move to
something longer term value alignment in
this community you'll often hear people
talking about paper clipping in Sumer
instrumental convergence and I want to
provoke a little bit on this topic you
know paper clipping is really a new
version of the old story of the Midas
touch you wish for everything to turn to
gold and then you realize that wasn't
quite what you wanted I've got another
frame that it might be interesting to
explore here which is that paper
clipping is actually related to
totalitarianism we have a real world
problem we're familiar with that we're
restating in futurology terms so I have
this conjecture here totalitarian
convergence this that powerful agents
with mathematically certain
monotonically increasing open-ended
objective functions will adopt sub goals
to disable or disempower other agents if
their objectives do not exactly aligned
you can prove this I'm going to skip the
proof it's a an economic style proof
with some simple stylized assumptions
but what you get is that an agent starts
off by disempowering things that
disagree with it altogether and then
it's left with it's sort of allies that
somewhat correlate with it but if
there's constrained resources it may
want to get rid of them as well in order
to or their agency at least in order to
get the perfect optimized world this
turns out to be behavior that's observed
in human political systems totalitarian
and authoritarian regimes often ally
with
other perspectives and movements in
order to gain power and then once
they're in positions of power regimes
like Cuba's or the Soviet Union or Nazi
Germany suddenly turn around and exile
or imprison or purge their former allies
and so paper clipping in some sense
maybe a story a warning about
totalitarianism a problem that humanity
has already struggled with an enormous
scale and there are some corollaries
that come out of this one is don't build
high-stakes AI systems with single
specific optimization objectives being
too sure of knowing what the right thing
to do is is dangerous the second is that
there's a research program on how to
specify objective functions that involve
preserving or optimizing for other's
agency this is actually a very subtle
and difficult point it's kind of like
figuring out liberalism West and
liberalism or libertarianism if that's
your thing for your objective function
you're gonna need instead of having a
specific goal to have a region that you
tolerate in some way in a mathematical
specification for that there are
numerous technical value alignment
problems that come along the way and
figuring out the shape of that tolerated
or good region as a function of other's
preferences sir
moving on from from that cluster of
problems there there's a question about
how we build a stronger field and
realign the many institutions that are
working on AI in a way that reduces risk
one idea here is to build safety Fanus
and social good goals into the
benchmarks data sets reinforcement
learning environments that so many AI
researchers use as the yardstick of
their field we're doing a little of this
work directly at Pei Carol Wainwright
who's here today has a project called
safe life that's building a test
environment for reinforcement learning
agents still teach them to avoid side
effects it's built on Conway's Game of
Life giving it the name you can ask him
about that we could even do a breakout
session on it but there's also a
possibility of doing something larger
and more structural inspired by the same
idea could we build a compendium of
missing data sets in machine learning
infrastructure basically a platform a
gathering place for the whole field to
come together and say I've got a mess
ethics or social good problem over here
who has the missing pieces or the team
or the funding to close that gap so as I
close you know thinking back high-level
what should our priorities be on AI risk
there are some articulated here but we
shouldn't be trying to set them all
ourselves we should be building cultures
fields feedback mechanisms and
institutional capacity to gather all the
solutions to these problems
so that we aren't just one actor
charting one course that turns out to
not quite be the right one
lastly before I finish I also want to do
a quick pitch for tomorrow Rosie
Campbell at Pei is running a session on
publication norms and be Cabello is also
here and a great person to talk to about
AI ethics implementation within large
organizations and labor and economic
impacts thanks of run okay great stay
right there
awesome okay great thank you so much
Peter I will give it first up to Robin
and Gillian and to pester you with
questions while I ask the ones that have
questions in the audience to move up
front of the mic or to find Aaron and
give you your anonymous question so
Peter when you say we should be doing
this who's the we uh there's probably a
different way for each of the places I
use that phrase in some cases I think
it's a community that's trying to do
planning and forethought on AI risk and
that's a fairly large and growing
community so it includes people in this
room people working at AI labs academics
who have a perspective on these
questions and things to bring civil
society organizations and maybe in some
in some cases I'm using the aspirational
way for the partnership on AI and its
many partner organizations trying to
gather resources to tackle these
questions maybe there were some places
whether it's government as well though I
was less thinking in those terms so one
of the traditional ways to deal with
totalitarianism in government is limited
power so in the caucus
today I that's sort of looking at how do
we limit the set of actions they could
take is that also an area that seems
it's a very good idea but I think the
thing that has made people nervous about
that approach on its own
is that making those limits robust with
rapidly improving systems is hard and
say you know as a computer security
person I'd say well you want
defense-in-depth
so you want both some limitations placed
there but you also want a system that if
it accidentally you know finds an
exploit for the limitations doesn't do
things that you'd regret afterwards
question here is Pai comfortable on
exporting AI to China I don't think the
export is the right frame if you look at
AI research and look at the literature
it is being written by an academic
research community a scholarly community
that's global in nature a huge fraction
of the authors on those papers of
Chinese descent many of them are Chinese
Americans living in the United States
coming to grad programs in the United
States and wanting to stay answer in
general it feels as though the frame of
global cooperation is probably the right
one for whether they I come from and the
framework of collaboration on safety and
allowing that you know if you want to
play strategic interest the United
States probably would serve its
strategic interest by creating new visa
categories that allow people to to stay
and work in the United States and Pai
actually has a report with some
recommendations on that front
AGI great powers meeting that we had
last year there was also one of the
recommendations that came out of there
and I think there's a little bit more on
kind of like how we can bridge which
then and bridge a gap so if you're
interested in the report it's it's lying
out downstairs and ok and and just to
maybe give you give a little bit of
context on the prediction that you made
and it's not a miraculous yet it will be
I think by tomorrow hopefully but in the
prediction you're saying you feel does
the the we're on a good trajectory if a
new norm again single value optimization
has successfully altered some
high-stakes ml deployments by
governments or tech companies do you
want to say like if you
sentences about that so people can get
predicting tomorrow so I'll give a
couple of examples of where this goes
wrong so one of our partners had a paper
that they showed us with a medical
prediction system that was recommending
outpatient interventions for people
released with cardiac conditions - and
what they were optimizing for was the
hospital's
financial incentives under the
Affordable Care Act there was a penalty
for the hospital if they released
someone who was readmitted within 30
days so they try to predict whether this
intervention could help prevent that
readmission within that window but you
could easily see that there are other
pretty relevant objectives like maybe
the overall welfare of the patient on
its own was one way you'd go for a
different decision or you might slide
the window and say well thirty days
might not be the right time horizon you
might want uncertainty about what the
correct time horizon is and so in a case
like that you probably shouldn't be
using one objective function you should
be uncertain over an ensemble of them we
see the same thing with these criminal
justice prediction algorithms there's a
lot of debate on what the right fenneis
correction methods might be for the
false positive rate problem that I
showed a slide on there's a literature
with lots of arguments about different
correction algorithms and no one's doing
anything right now because there's no
consensus there in fact impossibility
theorem is saying that none of the
corrections of the perfect right
correction instead perhaps what you
could do is be uncertain about which is
the right form of fairness and then that
leads to systems that sometimes say oh
wait and like maybe we should release
this person or maybe we shouldn't and
here are the kinds of considerations
that lead to that and so what I guess
I'm hoping is that we'll start to see
that philosophical concern taken back
into engineering and preventing the
deployment of overly confident systems
okay thank you very much thank you Peter
you


Defining and unpacking transformative AI | Ross Gruetzemacher | EA Global: London 2019

people are beginning to talk a lot about
how AI is going to transform society if
we simply Google transformative AI we
get 9 million 60 thousand results the
top 10 results are all for
transformative dot AI which is a
start-up that's trying to transform
healthcare by using deep learning if we
do a Google image search for
transformative AI the top results are
all still for transformative AI so this
tells us that perhaps not many people
are speaking about transformative AI
verbatim so if we remove transformative
dot AI from the search results we still
get 7.8 million results and from these
results we get pages that talk about the
power of transformative AI and how AI is
transforming society and business two of
these results are from Deloitte and
PricewaterhouseCoopers and we see other
results from organizations such as
McKinsey or the Brookings Institute so
we see that powerful players are
interested in the transformative impacts
of AI if we do a Google search for
transformative AI without the term
transformative dot AI we see more
results that show us that other
organizations are interested in the
transformative power of AI and how AI is
already transforming business and
society the next result on that previous
search was for Andrew ings
AI transformation playbook this is an
effort by Andrew Inge that is
intended to help organizations transform
their business into an AI driven
enterprise it's in the first sentence of
the playbook reads AI technology is now
poised to transform every industry just
as electricity did in the 100 years ago
so this is a good analogy because in
some ways it's similar to the analogies
that we're using in the EA community to
describe transformative AI if we go back
to the original image search that's
unfiltered we see there's a one of the
tops terms is forecasting and so if we
click on this term we see results from
my previous work on forecasting
transformative AI these are actually the
only results that I was able to find out
of the 7.8 million search results for
transformative AI that were related to
the AI strategy community so that's how
much people in the general population
are talking about transformative AI but
what are we saying in the EA community
and the AI strategy community here we
see four definitions of transformative
AI karnovski describes it as comparable
to or more significant than the
agricultural or industrial revolutions
the pho describes it as radical changes
in welfare wealth or power Zhuang and
Dafoe describe it to be as profound as
the Industrial Revolution and I in an
earlier work defined it as AI capable of
replacing humans for greater than 450
percent or greater of economically
feasible work most of these definitions
are not well-defined it's not clear
exactly what's meant by transformative
and some of them would imply societal
transformation only seen before in the
agricultural or industrial revolutions
previous work has considered smaller
levels of transformation
associated with social change from
advanced technologies eras broke the
Industrial Revolution down into five
separate technological transformations
we've identified general-purpose
technologies associated with each of
these levels of transformation which are
seen in the far right column of this
table by general-purpose technologies we
mean technologies that can affect the
entire economy so in the table we can
see that the first technological
transformation was driven by steam power
the second was driven by factories and
railroads the third was driven by
electricity and the internal combustion
engine the fourth by mass production and
the fifth by computers in the internet
now these technology technological
transformations were proposed twenty
years ago so they might may not be 100%
relevant now given the societal
transformation since then but they do
clearly illustrate the significance of
individual general-purpose technologies
and clusters of general-purpose
technologies on society notice that we
see electricity being associated with a
technological transformation so maybe
Andrew ings analogy isn't too bad
now I'm going to attempt to more clearly
define transformative AI the
introduction on illustrated how the term
transformative AI is being used
differently by different groups many in
the general public perceive AI
transformation to be occurring already
giving existing AI technologies while
those in the ei community see
transformative AI to be something akin
to a more dramatic change that has only
been seen in things like the
agricultural or industrial revolutions
we've identified several elements of
what it takes for technology to be
transformative these elements fall into
two broad categories indicators and
dimensions indicators of transformative
change our lock-in or irreversible
change and anomalous patterns and
metrics the most important of these is
lock in or irreversibility by this we
mean when some technology becomes so
widely used for a certain application in
society that it becomes difficult or
perhaps even impossible to go like to
change paths one example of lock-in is
nuclear weapons when nuclear weapons
were demonstrated and used this changed
the nature of great power conflicts
fundamentally and we cannot go back to
like it's irreversible we can't go back
to the time before that another
significant indicator and particularly
for more extreme examples of
transformative change is anomalous
patterns and metrics used to measure
human progress this can be thought of as
discontinuities in the rate of change of
metrics such as the gross domestic
product or life expectancy such changes
have only been seen before in the
agricultural revolution or industrial
revolution we further proposed three
dimensions of transformative change
these are extremity generality and
fundamentality the most important of
these is extremity or the magnitude of
transformative change another important
dimension is generality or the degree to
which changes impact a variety of
aspects of life and society and the
final dimension is fundamentality or the
degree to which changes impact
fundamental aspects of how people live
and work given these indicators and
dimensions we can now broadly define
transformative AI
as AI technology that leads to
irreversible change to some important
aspect or aspects of society we can also
distinguish different levels of
transformation based on the proposed
dimensions the most important of which
is extremity furthermore we believe that
it is very important to distinguish
among two broad cases of societal
transformation the first we consider
simply as transformative AI or AI that
leads to societal change comparable to
that precipitated previously by an
individual general-purpose technology or
by clusters of general-purpose
technologies so examples of this would
be nuclear power or the internal
combustion engine or electricity the
second we consider to be radically
transformative AI or AI that leads to
societal change comparable to that
precipitated by the agricultural or
industrial revolutions examples of this
would be comprehensive AI services human
level artificial intelligence or super
intelligence here is a figure that
depicts historical examples of
transformative technologies and their AI
analogues in the center we have
transformative technologies and on the
right we have radically transformative
scenarios these are all depicted on an
axis of extremity on the low end of the
extremity scale we can think of nuclear
power and nuclear weapons a possible
analog for existing AI technology would
be drone swarms or slaughter bots
nuclear weapons have changed the nature
of great power conflicts in slaughter
bots or other lethal autonomous weapons
have the potential to do the same in the
middle of the spectrum we can think of
the internal combustion engine and a
possible analog
existing AI technology maybe the
ubiquitous use of learning algorithms so
if in ten years we have learning
algorithms that are capable that are
being used ubiquitously in all Internet
of Things devices and throughout all
aspects of the economy we might perceive
that to be a level of change that's
equivalent to what we saw with the
internal combustion engine and at the
high end of the spectrum we can think of
electricity a passable analog here is
widely practical deep reinforcement
learning so if we were to see advances
in deep reinforcement learning practical
deep reinforcement learning that were on
the level of what we've seen with the
pre-enforcement learning for video games
and games we might think of that sort of
change as equivalent to what we've seen
or what the sort of social change that
we saw with electricity and finally at
the extreme end of the spectrum we have
the agricultural and industrial
revolutions potential analogs here are
those previously mentioned such as
human-level AI comprehensive AI services
or super intelligence so I'm going to
briefly discuss and can make some
conclusions some of the points we would
like to make here are that
transformative AI is likely to proceed
radically transformative AI if we don't
manage transformative AI properly this
could greatly exacerbate risks from the
transition to radically transformative
AI also transformative AI poses consec
Attis tropic and existential risks of
its own for example lethal autonomous
weapons could increase risks from of
nuclear great power conflicts even less
catastrophic cases of increased conflict
would likely exacerbate risks of
mismanaging the transition to radically
transformative AI
in conclusion we feel strongly that
moving forward our discussions about
transformative AI should be specific to
the levels of transformation described
here a spectrum of significant to
dramatic levels of societal
transformation but the levels but levels
that are less than those associated with
the agricultural or industrial
revolutions we further believe that the
term radically transformative AI should
be reserved for conversations about with
about the potential of societal
transformation on the extreme levels
associated with the agricultural or
industrial revolutions we believe that
shifting the conversation this way can
help us to have more productive
conversations about resource allocation
by enabling us to prioritize both
transformative AI and radically
transformative AI rather than just for
focusing on the most extreme scenario
while ignoring other transformative
scenarios which also have significant
potential to impact the long-term
well-being of humanity there's a lot of
points that you and your talk that
really agree with I especially like the
GPT analogy especially like the way
you're trying to push the discussion so
that people have a separation between
radically transformative AI and
transformative AI and sort of recognize
that one is likely to proceed via
receive the other I think it might be I
suppose a bit more interesting to maybe
poke at the few places where I think I
disagree slightly so might go through
like a few questions or possible
junctions so one thing I'm a bit curious
about is um I think it's the you draw a
distinction between transformative and
radically transformative where you know
in dust revolution is given as an
example of something that's radically
transformative and electricity you know
example of something that's merely
transformative I think it's actually
there's maybe an interesting point I
think along the dimensions you gesture
out so general nests extremeness
and fundamentalists of change I think
there's some sense in which electricity
may have been sort of stronger in those
dimensions so I think if you look at you
know economic history and look at you
know how quickly the technology change
in the course of people's lifetimes how
much the people's living standards rise
over a short period of time
the technology like hit both factories
and like homes and military technology
it seems like there's a lot of ways in
which maybe electricity actually sort of
kind of out-compete the the steam engine
that sounds I can think a lot of
economic historians point that the
period you know 1850 to 1900 that's
especially transformative where think
about what sort of special about the
Industrial Revolution
I actually think about one of the things
that you gesture that as an indicator as
opposed to a core dimension of
transformative change it seems like the
thing that sort of interesting about the
Industrial Revolution to some extent is
that it was really basically
unprecedented that lots of things were
changing very very slowly before Dean
desk revolution living standards
relatively flat technology was like
relatively similar one generation to the
next and then the pace of change like
picked up quite a bit and it became
maybe even faster and more radical like
later on but I think maybe the fact that
was a pivot point is really the key
distinguishing feature of the Industrial
illusions in my mind and maybe that's
the the thing to focus on for
transformative radically transformative
AI as well is not just the
transformation that's quite large but
transformation it's very anomalous given
what's been already happening so it's
something that would catch policymakers
off-guard in a sense for where that
happen like in anomaly in the metrics
yeah exactly I think you gesture as a
indicator of transformative change but I
think I'm inclined to think that might
be one of the most important aspects
distinguishing features is anomalous
nice
I think that's an indicator of radically
transformative change I think the most
important indicator of basic
transformative change is just lock-in
and then we have the other indicator
there because it's really kind of the
indicator of the radical transformation
yeah so I think I might also push back
against that a bit so I think um when I
think of like what's the most
transformative technologies that have
happened it does seem to me like there
are many technologies that to some
extent locked in that'd be hard to sort
of go back so you know ball bearings or
technology for example that didn't exist
that long ago I think it'd be really
hard to go back to a time when we didn't
have ball bearings or like just remove
ball bearings from all the objects I'm
not really that inclined to treat it as
you know to two fundamental or yeah I
think there's probably many things in
this this were John where it's something
we'll probably gonna continue to have
for quite some time like glass would
also probably be something that's hard
to go back but it doesn't seem to fall
into this category so I think I'm
inclined to think that although you
gesture out that that these were pivot
points might be these
specially interesting thing as opposed
to the irreversibility or even
necessarily the sort of generality
suppose some PM also confidence today
ask you about is so you give these sort
of high-level categories or stir things
to look at I'm curious whether you have
any sense of what sort of concrete
metrics we might use ad Ryan sort of
make this even more precise sort of
check whether a transformation has
happened no that's a good question now I
mean I think a lot about like metrics
for measuring progress in social metrics
and like technological metrics but no I
mean I can't think of any other than
than the primary metrics used for
measuring human progress yeah that's
yeah so I guess one that occurred to me
a bit is um sir productivity growth is
potentially one that I know is brought
up a lot in the context of the general
purpose technology literature so for
example when people talk about the
impact of electricity or information
technology the often sort of when they
try and talk about like when do the
impact become quite large they point at
these sort of productivity waves that
happens when in 90s there's a surge in
productivity and I believe the 30s there
was a surge in productivity with like
electricity ya know so yeah it makes
things I do look at yeah the Shubert are
cycles yeah and I think just last
question I'd be sort of interested in
probing a bit is um what eise review as
the most sort of important way to apply
these concepts so one concern I have is
um it seems like there's lots of ways in
which are quite useful so it's quite
useful to try and like give people a
sense of the magnitude of changes that
might happen like this might look like
this thing that's happened in history
when I think about forecasting and I
imagine someone in a like a situation in
the past for example trying to forecast
transformative steam power or
transformative electricity I sort of
imagined I'm struggling to target like
we know what exactly does this mean and
sort of communicate what like you know
10 years until transformed of steam
power like with actually sort of gesture
on some kind of curious yeah I guess
what you view as like the limits or
applications of the concept for both
communication and forecasting I mean I
see for communication they helped us
really in the resource allocation
and in defining what priorities we
should focus on and how we should
allocate resources for those priorities
and forecasting I think that they help
us to look at nearer term things that
could perhaps precipitate or greatly
increase the speed at which we approach
radically transformative AI and I think
that in general it's much more difficult
to forecast the discontinuities like in
metrics for measuring human progress and
then it's much easier to foresee things
like the steam engine because if
somebody was working on developing like
like for a plane example for for like
flight people were working on that for
30 years and even though the Wright
brothers have said that three years
before they didn't know whether it was
possible maybe we could look back now
and see that progress was being made and
maybe we can look at like use the same
sort of perspective to look at the
progress we're making right now in
artificial intelligence and to see that
we might be on the cusp of something
that's like a general purpose technology
or we might have even argued be there
all right thank you
[Applause]
great thank you both
we have one question from the audience
and I'll direct this to you Ross but Ben
if you have thoughts I'd love to hear
them as well
so are there any present indicators that
current level AI capabilities could be
close to becoming transformative so some
of the examples that we gave were for
existing AI technologies and you can say
that those are transformative AI
technologies on the lower level of the
extremity scale so for example lethal
autonomous weapons can be in some
degrees considered transformative they
don't have a very fundamental change on
society but their effects are very
general and they have like a moderately
extreme effect so they could change
potentially forever the nature of
warfare and great power conflicts so
that's an example of transformative AI
that is possibly already existing
great yeah I think in terms of metrics I
think one decent one to look at as well
as just investment levels because it's
at least some indication that a number
of different people think that it will
produce a lot of value relatively soon
which you know people may be wrong about
it but at least suggest it lots of
relatively smart people with lots of
resources think there might be
transformations and the like foreseeable
future great excellent well thank you
both Ross has office hours from 4:30 to
5:00 in the Queen vault room if anyone
wants to follow up further and we will
end there thank you both so much


Open-source learning: A bargaining approach | Jesse Clifton | EA Global: London 2019

all right can you hear me cool
and I should say the only reason that
this talk isn't recorded is because I
submitted an earlier version of this
paper to a conference and like wasn't
supposed to like have any like public
sort of presentation of another venue so
sorry that that's not like a very
exciting reason but okay open source
learning with bargaining which is
slightly less grandiose title than the
original okay so open source learning is
a model for multi-agent reinforcement
learning where agents are highly
transparent to each other let me start
this timer actually okay so the reasons
I'm interested in this are one to
understand better the behavior of
reinforcement learning agents who are
highly transparent to one another and
then to try to leverage this
transparency for better cooperation and
I will skip this in the interest of time
so first I'll say a little bit in the
way of background and motivation about
cooperation among AI systems and then
tell you about the the particular
contribution so I think actually
probably the first part is more
important and this open source learning
is just kind of a concrete example of
how one might proceed with research on
cooperation among AI systems using kind
of contemporary tools okay
cooperation so the ultimate goal here is
to make transformative AI which I'll
call T AI from now on go well and of
course the AI safety community is doing
a lot of thinking about problems of
alignment and control but cooperation is
also important and at least depending on
your definition of alignment may fail
even if alignment and control work so as
an example of a failure mode that we
might run into even if we largely solve
these alignment control problems it's a
social dilemma which many of you are
probably familiar with so social dilemma
is a game where everyone is better off
if everyone cooperates than if everyone
defects but individually people have
reason to defect
and examples that you're likely familiar
with are the prisoner's dilemma chicken
stag hunt and moreover these model many
unfortunate real-world situations that
we'd like to prevent so arms races
tragedies of the Commons and so on again
we may end up with mutual defection in
these cases even among AI systems which
are largely aligned with human interests
ok so what does this have to do with
transparency so machine agents at least
in principle can be highly transparent
to one another so in particular in
principle they can share their source
codes with one another if they're made
out of neural networks they can share
their parameters with one another
of course the in principle is very
important here because a transformative
AI system will be a extremely
complicated object and it may be
prohibitively difficult to for agents to
you know verify these various aspects of
one another's internal workings but
nevertheless because of the possibility
that agents will at least be much more
transparent to each other than say
humans are to one another and the
interesting properties of transparency
with respect to cooperation I think it's
still worth studying so transparent
agents can can better achieve
cooperation in some cases so I'll make
this a bit more concrete when I talk
about this open source game theory later
but this is somewhat intuitive though I
don't mean to argue that transparency is
uniformly better than non transparency
for instance agents who are transparent
to one another can also credibly
threaten each other or threaten each
other much more credibly than non
transparent agents but whether whether
it's a uniform improvement or not it's
it's an interesting it's an interesting
thing to study and try to make go better
so and of course there's plenty of
existing work on promoting cooperation
and social dilemmas among reinforcement
learners under varying degrees of
transparency and I see the contribution
of the work that all describe as helping
to map out this space of possibility
for achieving cooperation among
reinforcement learners and and I'll be
looking at a case where the agents are
highly transparent okay so the this next
page is a table with a lot of text
that's not so important to read through
the the upshot is that I've listed some
recent papers on cooperation among deep
reinforcement learning agents where the
agents are more - less opaque so in this
in this first case this consequentialist
conditional cooperation the agents can't
see one another's actions they can only
they can only observe they're sort of
local state and their own rewards and
they have to infer based on the rewards
that they're getting whether they're
being defected against or not and then
if they have sufficient evidence that
they're being defected against they try
to punish their counterpart so that's a
case of low transparency where as I'll
tell you about a case of high
transparency where the agents can see
all kinds of things about one another in
particular the parameters of their
policy and the source codes of their
learning algorithm and the intuition is
that under greater transparency you can
detect and punish defections much more
efficiently than in the opaque case okay
so open source learning again the
setting is multi agent reinforcement
learning so we'll we'll just look at the
case with two agents and these two
agents are trying to learn policies
which give them high long-term reward in
some sense so this is an example of a
multi agent reinforcement learning game
basically you have this blue player and
this red player and they're running
around trying to gather fruit which
gives them reward and they're trying to
learn policies which allow them to
gather a lot of
in the long-term so incidentally this is
uh this was an example of a so called
sequential social dilemma that was
proposed a couple years ago but this is
just to make things more concrete but
the main idea of open source learning is
as follows so the players will be
jointly optimizing some kind of
compromise value function rather than
trying to optimize their individual
value functions and punish their
counterpart if they see that they're not
updating their policies accordingly so
we would like to make this into really
more or less a single agent
reinforcement learning problem where the
agents are jointly optimizing some again
some value function that represents a
compromise between the things that they
want only of course the agents will have
reason to defect and try to go off and
do things that get even more reward for
them so we need to have some mechanism
for detecting and punishing defections
from the the optimization of the
compromised value function okay so
making this a little more precise we
have a welfare function so that's what I
was just calling the compromised value
function this is what it's called in the
the game theory literature so this
represents some compromise between
players individual value functions and
for the purposes of the talk I'll
actually just take this as Gibbon so
this could be this could arise from some
kind of cooperative bargaining setup so
for instance it could be the Nash
welfare function which I'll display on
the next line it could also the players
could also agree upon this function as
in an alternating offers game and but
it's not so important to how we're this
welfare function comes from from from
the for the purposes of this talk it's
just something that the agents are
satisfied to jointly optimize okay and
then learning algorithms which are
transparent so the learning algorithm is
the thing that takes the agents history
of observations and updates their policy
in a direction they
we'll give them more reward in the long
term so a transparent learning algorithm
is one that can see the the counterparts
policy parameters and source code in
addition to the history of observations
and then the punishment regime remember
I said that you need to have some way of
punishing your counterpart if they don't
update towards the maximum of the
welfare function so this in particular
will be attempting to minimize your
counterparts payoffs over some time
horizon okay so I don't have time to
work the details of the of this Nash
welfare function but they basically this
is one example of a welfare function
that has some nice properties and it
basically maximizes the product of each
player's gains from from bargaining so
this is one example of something that
the players might agree to optimize okay
program equilibrium so this is this is
the way these are these transparent
learning algorithms will decide what
updates to return so the main idea of
program equilibrium is instead of
playing a game where you and I
simultaneously submit actions we
simultaneously submit computer programs
that will act on our behalf and
critically those computer programs can
see one another source codes and this
can lead to basically more cooperative
outcomes because you can anticipate you
can you can submit a computer program
that will see that your counterparts
program defects against you and then
defect accordingly and this makes the
submission of computer programs that
cooperate with each other in equilibrium
so in this paper I use my colleague
Caspar Esther Held's epsilon grounded
fahrbot which uses a counterpart source
code to simulate their response to
itself and the epsilon grounded part is
that by with some small probability
epsilon you have to take some default
action to avoid an infinite recursion
because of course my counterparts
program will be in general simulating me
as well
so okay so this is all defined for just
a normal form game like the prisoner's
dilemma or whatever but we can identify
oh this is uh this is just a diagram of
epsilon grounded fahrbot playing against
another epsilon grounded algorithm I
don't have time to walk through that
okay so here we identify cooperation and
defection with updating towards or not
towards the estimated optimum of the
welfare function which I'll call the
bargaining solution so basically just
use something very similar to epsilon
grounded fahrbot to choose policy
updates again where the actions
cooperation and defection correspond to
a cooperative or uncooperative policy
updates and punish if you see that your
opponent source code implies that they
will defect that is not update towards
the the bargaining solution okay so a
picture of this real quickly so we have
this surface represents the welfare
function as a function of the player's
policies PI 1 is the first player's
policy PI 2 is the second player's
policy the cooperative thing to do is is
go towards this estimated bargaining
solution at the top of the surface and
this black Dada is where the policies
currently are so the cooperative thing
to do is for player one to move towards
the bargaining solution player 2 to move
towards the bargaining solution
altogether they they go towards the
optimum of da the welfare function or
the estimated optimum on the other hand
if player one defects this will with
high probability be defected I mean
detected and punished by player two okay
obviously you've probably thought of
many problems that need to be addressed
here one is enforceability so it might
be the case that you can do better than
cooperating even if your play if your
counterpart is trying to punish you
another is of course I've made very
strong assumptions about the agents
ability to verify
various things about each other's
internal workings and so relaxing those
assumptions is of course a very
important part of getting something that
looks kind of like this to work in the
real world it would also be nice to look
at the empirical performance of open
source learning algorithms and compare
these the ability of these to promote
cooperation with other approaches that
use make less use of transparency so I
have some thoughts on these but there's
lots more to do okay and lastly a TAF
we're working on a research agenda that
is about cooperation among tii systems
so there's a there's some stuff related
to this open-source interaction but
there's lots more as well feel free to
reach out to me if you'd like to chat
about it or look out for it hopefully
it'll be public by the end of the year
okay that's all I have cool so I guess a
question I'm curious about to start with
is the set of assumptions about how
transformative AI goes down that are
required for this work to matter like
like like which which scenarios like
what assumptions we'll be making about
what scenario we're in for this stuff do
you think yeah so I mean I guess I
should say it first like generally I
don't see like this particular thing
like being put into the transformative
AI but but what sort of assumptions need
to hold for some kind of open source
interaction to work out well it needs to
be in at least a bipolar scenario so
there need to be more than one AI in the
mix and of course the big thing is that
it's even possible for agents to verify
their counterparts internal workings to
the extent that they think that
something like this is in their interest
and of course far from obvious if that
will be true yeah
maybe this is a reason for us to want to
differentially advantage differentially
advanced you know whatever technologies
are required to make verification easier
just so we make it easier for AI systems
to verify properties of other AI systems
do you think this overall seems like a
good idea as a result of these
considerations yeah I mean I this the
fact that transparency can promote
cooperation is definitely a reason in
favor of that as I tried to say earlier
it's still not obvious that over all
agents being highly transparent to one
another is a good thing because of these
this notion of being able to make more
credible threats and so on
my intuition is that all things
considered it's still good that that
agents be able to that agents be highly
transparent to one another but that's
something I'm very uncertain about and
it seems like an important question to
answer yeah I guess another question is
like if we already have these systems
that are operating in the real world I'm
curious like how much of this kind of
research we can just defer to them you
know when we're making these really
powerful systems and we say hey there's
this like there's a couple papers on
open-source learning algorithms or
whatever can you like improve that
research area like I'm curious whether
you feel like this research is like
particularly important to develop before
we have powerful enough technologies to
help us with this research or I'm
curious like how you think of it yeah
well I think one point in favor of that
being important to do early is the
possible path dependencies that you
point to in your last question so if we
need to differentially advance sort of
technologies for validation verification
and so on that might be something that
yeah that might be a reason in favor of
doing this earlier rather than to
deferring to systems later when it's
when it's already sort of too late cool
I guess another thing is it kind of
seems to me that there were many places
in your talk where
advances in various other fields would
be helpful for making this stuff work
out so you know one that seems right is
zero knowledge proof seemed like they
would help with some of the situations
where you don't want to share your
source code with your adversaries who
are you know where you don't want to
share your AGI source code with your
adversaries who are making their AGI
because you don't want to save them like
him I'm curious if there are other
examples to you of you know outside
fields which would be helpful for making
this kind of stuff work out yeah I think
that yeah the the one you mentioned is
right and verification more generally
and also understanding sort of the
methods for better interpret ability of
machine learning systems might also be
important and also seems kind of broadly
good anyway but yeah I think the ones
that you mention are the natural wants
to look into cool I guess like one final
question here from me I'm curious how
you plan to prioritize doing further
work in this particular formalism and
getting a deeper theory of what happens
in this formalism versus coming up with
richer formalisms you know there's like
two kinds of ways you can try and
develop a research area I'm curious how
you're thinking of that here yeah yeah I
mean I think there are considerations
pointing in both directions especially
because I mentioned I don't think that
this particular framework is all that
realistic I think that there are maybe
this is yeah sort of another reason to
develop other frameworks that are maybe
maybe more realistic and I tend to think
generally that there are fairly quickly
diminishing marginal returns to research
in some kind of narrow formalism that
you're not confident is really
that realistic in the first place on the
other hand I mean I think that you can
just learn a lot by studying a formalism
even if it's not even if it's highly
idealized and I mean I think for sort of
instrumental reasons it's hard to it's
hard to kind of establish an academic
subfield if you're just if you come up
with lots of frameworks but never really
deeply explore any particular one so
yeah I think that I'd probably be
excited - all things considered keep
working on this and try to get better
answers to some of the questions I
listed great okay thank you both um so
there's a particularly good question
from the audience that I want to ask you
Jessie someone is wondering what
assumptions do you make about the length
of the co-learning phase in reality the
two agents might only get to interact a
small number of times so do the results
hold then right so so actually in the
paper everything all the utilities are
defined as these kind of limit of
average reward can infinite horizon
objectives so that is not because I
think that that's like a realistic
approximation to the lengths of times
over which these agents are going to be
interacting but really just so that the
quantities are easy to define as as for
butBut I think that's also sort of
typical of reinforcement learning theory
it's hard to say much about
finite time horizons in general so the
short answer is a question to your
question is that the the few things that
I proved in the in the paper are for
these very unrealistic asymptotic
regimes yeah okay great that makes sense
second question is do you think that
mutual cooperation is the only
equilibrium in your setup probably not
but I imagine that it's the social
welfare maximizing equilibrium yeah so
one problem with this program
equilibrium stuff in general is that
like there are many many many equilibria
of program games but you may be able to
weed out like many of the equilibria on
the basis that like there's only a few
that maximize Social Welfare


Re-deciphering China’s AI dream | Jeffrey Ding | EA Global: London 2019

rhe deciphering China's AI dream a year
and a half ago I published a paper
called
ciphering China's AI dream trying to
descriptive overview of what is
happening in China's AI landscape
because I'm really good at creatively
naming things this talk is called Rida
ciphering China's AI dream and pointing
towards stuff I got wrong stuff I think
we should be researching in the future
so the motivation is that this is not
just about finding out what is happening
in China's AI development that in and of
itself the first layer is really really
important I think China is an
indispensable player in AI governance
ensuring that advances from AI will
benefit all of humanity and preventing
risks from that development so I think
they're an indispensable nation that's a
term by former Secretary of State
Madeleine Albright in the US where she
talks about transitioning the u.s. from
an hegemon to an indispensable nation
where the u.s. is involved and has to
have a seat at the table in a lot of
these conversation that's and I think
that's the case for China as well for
two reasons first is that China probably
by most accounts has is the second
leading power in the world and also has
the Internet giants that are where a lot
of the AI research is happening and
where a lot of the AI capabilities are
located so that's number one
capabilities I think you can just
disagree about the capabilities point
and people can debate that over and we
will actually go into some of those
debates here today but point number two
is even if you think China's
capabilities don't mean that it has to
be an indispensable nation in AI
governance it's perceived as rising
power and other countries will use China
as a bogeyman type of figure or as a
frame to motivate their own AI
development most notably in the case of
the United States
so conversations about centralizing the
u.s. 5g infrastructure the motivations
for that were concerns about ai's
development Mark Zuckerberg and his
leaked testimony notes on it says like
don't talk about breaking up Facebook
and AI Giants and tech giants because we
need to compete with China so second
motivation for why we need a better
understanding of China's I developers it
can shed light on China as a key actor
in the international system
what does China want what are the
implications for power transition
relations with the US relations with the
EU how it will contribute to the
international order and third I think it
can tell us something about broader
landscape of how technology affects
global change and throughout the
presentation I'm gonna look at one slice
of this which is technology and national
power but there are different ways of
cutting that technology and people's
perceptions of technology technology and
relations between people and machines
but I'm gonna but it goes basically it
goes deeper than just what is happening
in China on AI and hopefully this
presentation will give a sense of that
okay that actually turned out a lot
better than I thought it would in terms
of putting a bunch of text on the slide
is that somewhat readable for folks
okay I'm gonna interpret silence as a
sentence so I think in those three
layers China as an indispensable player
in AI governments what I'm going to talk
about is two problems the tech
abstraction problem and the china
abstraction problem and then I'm going
to talk about how those fuel memes about
AI arms races and kind of what China's
AI dream means in an interdependent
world and then finally I'm going to talk
about the technology will change and the
rise and fall of great powers I'm not
going to talk about technological change
and structure of governance I think I
cut that slide after and forgot to
delete that bubble point so here is the
AI potential index from deciphering
China's AI dream pork cup a year and a
half ago and I think it is a good
capture it captures nicely how I was
thinking about technology in a very
abstract way especially AI which has
become this magic catch-all term where
anything as a and AI is nothing at all
so the idea was here we were trying to
come up with a rough approximation of a
country's AI potential so we take a cut
of hardware metrics we take a cut of
data research and algorithms commercial
AI sector and we try to come up with
indicators that give a good proxy for
where countries are in these different
capabilities the conclusions that
China's rough national AI capabilities
are about half of that
of the United States I think this is
like not a good approximation and it was
meant to be a First Coast meant to give
an overall picture of what if a country
devoted all of its resources to building
advanced AI systems or pursuing an
artificial generally artificial general
intelligence agent another example of
abstraction technology abstraction is
here is the Department of Commerce
proposed rules for extra controls to
identify emerging technologies that are
central to u.s. national security
so in these rules there's 14 categories
of technologies one categories AI
there's other categories like
biotechnology another category was
robotics another category was advanced
surveillance technologies what I'm
trying to argue here is that we really
do not have a good sense of what we're
talking about when we say AI or the US
government or me or I think a lot of
people in this space because when we're
talking about when the US Commerce
Department puts together a rule that was
probably researched a lot there's a lot
of staff people there they got they have
they have jobs to do and all the kinds
of AI that are put under that list range
from fundamental models fuzzy like
branches of fuzzy mathematics specific
classes of algorithms to specific
hardware you could even argue the other
categories would be subsumed under AI is
robotics AI advanced surveillance
technologies incorporate in corporate AI
more and more so just giving you a sense
of this is a slippery target that we're
trying to analyze so we when we talk
about China's AI dream what are we
actually talking about so I'm sure I
think in my slide notes I was supposed
to talk about like in Vacation Bible
School growing up we learn like the song
about like how there's a fountain
flowing deep and wide and so every time
you all talk about you think of
technology as an independent variable I
want everyone to think about technology
as a variable that is both deep and wide
and you have to do the hand motion while
you're doing it and the idea here is two
years later in written testimony before
the us-china economic and Security
Commission they asked me to assess
China's national AI capabilities so I
get another crack at the proud
with one and a half years more
experience and hopefully a little bit
more wisdom and the idea is let's
actually tackle the technology
abstraction problem at its root I think
that we that with technology and with AI
you have to divide it up in terms of
what level of depth of AI are you
talking about so are we talking about
the foundational layer of the AI value
chain AI open source software deep
learning frameworks like pi torch tensor
flow of those a white paper has shown
that 66% of those are developed by US
companies those have a strategic
advantage for those companies because
people get used to coding on those they
want to work for those companies and
they're more used to working for those
companies they become a way for those
companies to benefit from network
effects to improve all of their models
because there's so many people making
improvements to them and a way to
recruit talent so that matters hey I
open source software matters at another
level of depth of the value chain you
have the technology layer so what are
the what are the algorithms algorithms
that you're licensing to different
companies third layer you have
application layer the smart speakers
products the hardware products that are
actually being sold so that's one way to
slice it along depth and it can even go
deeper than that so a lot of people
compare they say AI is a general purpose
technology so if we talk about general
purpose technologies we're talking not
just about the application layer the
technology layer the deeper foundational
level layer but we're also talking about
technology as a system a system that so
other general-purpose technologies like
electricity it's not just about the
electric dynamo it's not just about the
electric utility it's about how the
utility connects to the factory to power
individual electricity generators that
are funneled through transformers and
funneled through an entire system of
electricity generation distribution and
transmission and consumption so this is
this is a problem of depth and it's also
problems of with so AI is a broad
category that accompanies a different
domains Chinese natural Lane
which processing has much different
implications than English natural
processing natural language processing
you might if you say Chinese NLP is
better than American a NLP what type of
NLP are you talking about can we measure
different countries capabilities in
specific domains of AI and there will
probably be very be variation between
those different domains so second
problem I think in the space is what I
call the China abstraction problem the
idea is that China is not a monolithic
actor and this has meaningful
implications for governance so this is
one example and I'd like to use the
standard space as an example technical
standards the work of bureaucrats
companies industry alliances to shape
what are the rules that govern different
products and systems as they go online
to different markets these standards
shape these standards are a billion
dollar disc decisions one popular
Chinese saying is that third tier
companies make products second tier
companies make platforms first year
companies make standards microsoft word
has a word its dominant because there's
a word document formatting standard that
they achieved they were the first to go
to market with and it's spread so this
is obviously a very important domain and
there's actually a lot there's conflict
and there's cross-cutting cleavages
within the Chinese technology space in
terms of which companies are siding with
who on standards so this is an
interesting case where Lenovo a Chinese
company votes in favor of Qualcomm a US
company on a key standard for polar
coating that's going to be important for
5g which is an enabling application for
a bunch of Atanas for autonomous
vehicles for a bunch of AI applications
so there was a big debate in China about
it when this was leaked out and Lenovo
had to issue a public apology say they
didn't actually vote for a while way but
these types of say that they didn't
actually vote against Huawei but these
types of disputes and debates and
cross-cutting cleavages strategic
alliances between Chinese companies and
other companies
are happening all the time underneath
the surface and we should we should
probably we should not just say China
wants to do this with AI or China has AI
dream
we should specify who are the specific
actors with the child within China that
have specific intentions and
capabilities but we have to say China I
dreamed to make it a sexy title and get
people to read the report so I'm not
gonna apologize for that my argument is
that these abstractions and and I don't
want to say that we shouldn't be
abstract I think that being parsimonious
is important for communication I can't
to write headlines you can't just like
we can't specify everything sometimes
you just had eggs for breakfast you
don't need to tell people the type of
eggs or that was a bad analogy but the
idea is that these two problems are
intermingled with another dangerous set
of assumptions sometimes useful set of
assumptions but oftentimes dangerous
which is techno nationalism and
sometimes people just throw around the
phrase Technol nationalism is coined by
a Robert Reich labor advisor for
President Clinton to actually refer to
US policy in response to Japan's
technological challenge so it's actually
really relevant to the current debates
right now and as you see in that
Brookings report China the argument is
that in AI who's going to dominate in
the era of industrial AI and the
argument is that China is inventing a
kind of Technol nationalism so what do
you mean by technical nationalism I
think that we should clearly specify
what we're talking about so one is the
belief that technological strength and
security are key drivers of interstate
competition I call that the techno
competition belief and I think largely
most countries believe in this second is
techno independence some degree of
technological autonomy is key to
technological strength and security so
some countries might believe that
autarky is the way to go so that's a
very strong sense of techno independence
a weaker sense might be we need to not
be dependent on a sole source supplier
of say semiconductor chips we need to
diversify our supplies that we can be
cut off some degree of autonomy and
freedom
and the third is the techno nation
belief the idea that the nation-state is
the primary unit of relevance for
technology policy and I think just to
give a sense for how baked in these
assumptions are I think probably
everybody in this room believes that
National Rd the amount of national R&D
that you spend it's probably correlated
with your economic growth or like that
spending more on innovation at the
national level will lead to national
economic strength in some form actually
that's like disputed in the literature
that we have good sense that global
innovation tracks with global economic
growth but there's arguments to be made
that national level innovation national
investments in R&D might not necessarily
translate to national level growth and
one example is that just technology
diffusion happens much easier these days
we live in a world of global innovation
networks where knowledge is being shared
so it's not clear the extent to which
the benefits of technological
development are captured within the
nation-state container so that's the
framework for what we're talking about
when we talk about techno nationalism I
think sometimes all of those beliefs are
rational in some sense and it's an it's
a way of understanding and think about
the world that can be useful sometimes
though when it's it can be very much
distorted and I give examples of
distortion in each of these spaces so
one example is a distortion of the
techno competition meme to have a good
sense of how technology is an interstate
a driver of interstate conflict you have
to you have to have a good sense of
assessing where the technology is how
technology is actually incorporated into
power whereas the meme now is that this
is a Sputnik moment China's advances in
AI represent a Sputnik moment for an AI
arms race it's unclear what what these
commentators think were racing over is
it a specific discrete weapon output is
it about incorporating and enabling AI
across a wide range of military
applications from including the non sexy
stuff like better logistics and
communications and until
transmission is it about a broader
industrial competition of AI and the
idea is that there these cold war
analogies sometimes fall apart because
there's not something countable it's not
about who has more ICBMs it's about the
I think that the more relevant aspects
of competition are going to be about who
the different types of technological
systems that develop which we'll talk
about later
the second is techno independence so the
idea is that because of China's
innovation mercantilism strategy and
spaces like artificial intelligence the
innovation innovation technology I think
there the information technology
innovation foundation iti F one of the
alphabet soup of DC think tanks that are
often very prevalent in the space there
vice-president has recommended that in
response the US should suspend all
scientific and other technical
cooperation with China so that's one
model of techno independence in response
to what's perceived as innovation where
cantle ISM so true president Trump
tweeted I think a month earlier
basically ordering US companies to get
out of China and the idea is that to
what extent can the US maintain its
technological independence is the
solution to cut off everything so this
is an example of which these these memes
can be distorted suspending scientific
and technological cooperation would
probably also hurt US companies and the
u.s. innovation pipeline and also
removes a good channel for encouraging
trust and cooperation in the space
Matthew Evangelista scholar at Cornell
has written a book about the Pugwash
movement in the context of the Cold War
where US and Soviet scientists and
physicists meeting in the Pugwash
conference and having side talks and
back-channel negotiations he argues was
a crucial channel to reduce Cold War
tensions third belief this the the third
distortion is in the techno nation space
and this this is the idea that
technological innovation Maps perfectly
on to landscape of nation-states so Chi
Phu lien in New York Times editorial he
basically says that companies will be
forced to negotiate that that countries
who are dependent on the supply of a
algorithms from other countries firms
will have to negotiate what the
countries themselves not understanding
or not at least in my opinion giving
weight to the fact that these companies
themselves will have interests to
operate in other countries that the
Facebook's and the googles and the
Microsoft's of the world are actually
trying do not see themselves as in like
mapping perfectly onto the interests of
the nation-state
I think there's important differences
there that we should acknowledge so now
just to try to connect a lot of the
things I've thrown a lot of concepts at
you have thrown a lot of terms at you've
thrown some examples at you some jokes
that did not land very well and now to
connect all the dots I think the idea is
that to tackle technology abstraction
problem to look at specifically what
these have to save for the nation-state
and what these say for stuff that goes
beyond the national container we have to
adopt a technological systems approach
for understanding how technology and
global change intersect and the idea is
that if you look back at the Second
Industrial Revolution this was late half
of the 19th century the argument here
for most scholars is that this was an
area of britain's relative decline and
the US and germany taking control in
industrial power technological power
leaping ahead of Britain and the
explanation for why it happens
oftentimes is the sexy new industries of
the time electricity chemical industry
as Germany was ahead in all these new
industries steel as often mentioned
Germany's steel output leppe forward by
bounds during this time period I think
those accounts get it wrong I think they
overemphasize the new industries that
took a really really long time to
diffuse and they probably didn't make an
impact on productivity until after nine
14 and I think they're missing some of
these less sexy less visible less
consumer-facing systems and one system
was the American system of manufacture
where you actually had machine tools
that enabled the production of sewing
machines bicycles automobiles small arms
small firearms all with interchangeable
parts all cut precisely with new like
turret lathes milling machines that
could like form things really precisely
and a lot of scholars point to this
American system of manufacturers being
the key to advances in manufacturing
productivity and the u.s. increase in
technological power during this period
so if we get to this level of technology
of understanding technology what we're
talking about systems not just buzzwords
what can we think about in terms of
China's AI development and I would point
towards maybe there's something like
what are the American systems of
manufacturing that could exist in China
today or will exist in China in the
future I think one potential example is
the industrial internet the notion of
connecting a bunch of devices in
manufacturing work room floors and
having them talk to each other getting
much more granular predictive analytics
about when we need to do maintenance on
these all very exciting topics but also
all very boring topics sorry but also
very exciting for people who are
thinking about how technological systems
will impact interstate competition and
how we think about power and so this was
one of the latest issues of the China AI
newsletter where we talked where I
looked at a translation about CAS I see
cloud SAS at cloud and CAS I see is like
a arrows one of one of the key like
state-owned enterprises who does a lot
of tech stuff and they're trying to
build an industrial internet platform
have built one and they're in
competition with Siemens which has their
own system and General Electric which
coined the term industrial Internet and
came out first with a predict system and
I think that there's that you know just
as an indicator of how important this
stuff is when Siemens and SAS at cloud
work together to sign an agreement again
going back to question
technical frames and seen that there's a
lot of things happening underneath the
surface of Interstate competition but
when they signed an agreement to work
together on these platforms guess who
was also there at the signing ceremony
President Xi Jingping and Angela Merkel
so this is I think it's not just about I
gave Iran through this with all these
kind of abstract terms and concepts and
a roadmap for redesigning China's AI
dream
just with the example of technological
competition and kind of power in mind
you could apply this framework to a
bunch of other things like what
surveillance has for implications for
authoritarian resilience for
disproportionately targeting ethnic
minorities like is happening with
weygers and Xinjiang you could apply it
to a bunch of different interesting
research questions in this space I just
ran through it as if because this is my
interest and my DPhil research on
technological power and competition in
the space but the idea is that this is a
general model for looking at what are
the problems in research how we can how
we can do better and yeah how we can
reduce ifer China's AI dream so things
this is really fascinating and not
obviously super important and seemingly
kind of a if not a fast-changing domain
then certainly one that's been kind of
popping up in a few different places
recently first question that I was kind
of curious about is to what degree do
you think the average Chinese citizen
sort of sees China as being in this sort
of nation-to-nation competition with the
US or even just kind of the rest of the
world yeah I think like the average
person just doesn't care that much about
foreign policy or international
relations so if you look out like even
American voters like foreign policies
usually like the last thing and it's all
about like economy and like
bread-and-butter issues so I think
average Chinese isn't probably the same
way at least I can only speak for like
my friends in China I think there is
some pride in like where China has gone
and how far it's come so maybe there's
more of a sense
of China as rising power to that extent
but I just think most people are just
trying to like make ends meet
get a home get married do life so with
that in mind how do you make sense of
well with that in mind how do you make
sense of the NBA story from the last
couple weeks you mentioned that that
battle system you're following the NBA
story as well but for those who haven't
made everyone who likes rap also like I
think it's a safe correlation anyway
yeah so there was a tweet by a guy who's
affiliated with the NBA which is the
basketball league yeah in the United
States which does a lot of business in
China and it was just a pretty you know
from certainly an American point of view
a pretty kind of vanilla you know
supporting Hong Kong sort of statement
and it was made on Twitter which
obviously you know the average Chinese
person doesn't have access to or pay any
attention to and yet it was blown up
into this big thing and almost all of a
sudden all these partnerships between
Chinese firms and the NBA we're starting
to get cancelled and like games are not
being televised anymore and the sort of
projection that seemed to be coming out
of China was like we are all very upset
about this right do you think that that
is manufactured and just overblown or do
you or would you somehow makes it help
us make sense of that if people don't
really care about foreign policy yeah I
don't think like most of the 1.4 billion
people in China or like really really
upset about this thing I do think that
this is something that I've actually
tried not to follow to actively because
I usually go to the NBA subreddit every
day as like a nice escape from work type
of space but now this has become a well
welcome to your new real
but I just think like I think if you
want my hot take on this I think the NBA
capitulated way too much like so many of
my friends love watching the NBA they
would just find like another way to
stream it so I think like the kind of
the business relationships like Tencent
saying they wouldn't stream games
anymore I think they would have like
they're doing right now they're going
back and like they're gonna stream games
so there's just so much demand for the
NBA that I think the NBA had more cards
to play with then than they did
that seems to be this sort of like
projecting censorship into other parts
of the world seems to be kind of part of
the Chinese government's program right
now so I'm also a tick-tock user okay
and get a lot of pleasure out of these
you know brilliant 15-second dance
videos but more recently I've also been
hearing how many videos are being taken
down not just in China but in any market
right
when this tick-tock now has a billion
plus users so how should we think about
that if we're just kind of random
tick-tock user sitting at home but maybe
a little concerned yeah
actually so tick-tock is like the one
tech know the one app that I tried to
use and I just can't speak the language
of it anymore so I don't know I I think
it's a tough question and I don't really
have a strong opinion on it I think like
good source of analysis the Guardian and
Reuters both I think the Guardian had a
good piece that looked at a memo that
was leaked about their the specific
censorship practices that were being
used so I think it's also important like
to know it's important to not equivocate
and not to like make false equivalences
between different platforms because
centering censoring for specific
political aims like the Hong Kong
situation maybe is is different I think
on some level than what like Facebook
does like I think Facebook adopts the
censorship practices a lot of the
countries that they're in and also the
extent to which filtering out hate
speech is a form of censorship or not I
think can be debated but I think we have
to make distinctions I'm still thinking
through this but we have to look at
where there are similarities and where
there are distinctions so it's not up my
alley but it's a tough question so here
only try one more that I'm kind of
building toward this the general
obviously sense of China from the
outside is that it can act
monolithically it seems like we see
examples of that with this MBA thing and
with the technology that that is even if
it's maybe overstate
a sort of a source of asymmetric
advantage or potentially disadvantage
but some sort of asymmetry there where
when you see Trump tweet on you know a
random day of the week that American
companies are now hereby ordered to do
whatever everybody just kind of rolls
their eyes and says well you'd only have
the power to do that sure but there does
seem to be an actual center of power in
China that can say like you're gonna
cancel partnerships with the NDA yeah
and it kind of happens so does that mean
something in this geopolitical context
yeah I think it I think it definitely
does mean something I think we just
sometimes assume that it means China
will have an advantage when oftentimes
these things backfire so a good example
is also with Tencent they couldn't put
any games online for about a good
portion of the year because a new agency
came online or a new bureaucratic player
came online and thought that the video
games were being too addictive for
consumers whereas there's a lot of good
evidence that video games are actually a
crucial driver of technological
development because it gets people
interested in digital applications
gamification can inspire a bunch of like
positive spillovers
and also hurt $0.10 one of the biggest
companies and they're like stock values
dropped like enormous ly so yes it is a
difference but it's also a difference
that might actually hamper China
geopolitically rather than then and then
empower it so we often top times talk
about the empowerment angle which I
think is important but we sometimes
ignore the ways in which these centrally
guided policies actually backfire well
we could talk about this all day and I
haven't even scratched the surface of
questions coming in from the app but
you'll have office hours yeah at 3
o'clock
up at our next break so come there to
talk to Jeff about all this and more for
now how about another round of applause
for Jeffrey ding fascinating topic thank
you very much
great job