Lightning Talks | Beneficial AGI 2019

[Music]
[Applause]
different models have been used over the
years but there's a sense in which you
just kind of slap them in to prevent
your values from blowing up and speed up
training and you're good to go
but I want to argue there's a much
deeper story here that comes out of the
neuroscience of normalization I think
there are potentially some serious
philosophical consequences for those of
us who think deeply about the nature of
human decisions and human values the
story starts with vision when you're in
a dark room on a bright day and you look
out the window there might be a factor
of a billion times as much light outside
than inside you look at this on your
smartphone camera you'll see that the
bright white of the outside is totally
blown out and the shadows are totally
indistinguishable but that is not how
this scene appears to you even when you
look outside the window you can still
make out detail inside and vice versa we
have techniques like HBR to try to
overcome this and to state the obvious
but profound we like HDR images because
they look more real the brain or at
least the visual system really does seem
to have a kind of HDR built in this
allows us to navigate the darkroom on a
bright day but also leads to stubborn
visual illusions where objectively the
same color is perceived differently
depending on what's around it starting
in the early 1990s neuroscientists led
by David Hager were able to find a
precise mathematical model for exactly
the kind of transformation that is
happening in the visual cortex v1 and it
takes the form of what's called divisive
normalization divisive meaning the
neurons firing rate that can be
represented as the brightness of what's
inside its receptive field divided by
the brightness of everything that's
around it
well what turns out is that this emerges
as a canonical neural computation it was
found in v1 it was found in v2 it was
found in Mt
it was found in the retina it was found
the auditory cortex essentially every
sensory cortex in which there was an
effort just that the show that this was
the representational scheme has
succeeded in showing that this is the
representational scheme in a parallel
development that we don't have time to
describe in 1999 a group of neurons were
found that appeared to represent the
subjective value of something in their
receptive field not the brightness but
the actual desirability of that good
which leads to a very obvious next
question does the brain also use
divisive normalization for its
representation of subjective value the
answer is yes as you add other valuable
items nearby to something the neural
value signal goes down you can even do
very clever things to people's choice
sets to force them to make errors in
their judgments and they are exactly the
errors that devices normalization would
predict in humans and in monkeys
Kahneman and Tversky in 1984 famously
said that choice illusions are like
visual illusions the truth appears to be
that choice illusions literally are
visual illusions they were the exact
same mechanism one last and more
speculative result many researchers are
asking whether pathologies of human
valuation might affect be normalization
bugs something like depression where you
evaluate a bunch of things to do and
nothing seems better than anything else
what if that was a normalization bug
well if that's true and it's also true
that normalization is this canonical
neural computation that's going all the
way down to the retina and you should
find visual deficits in people with
severe depression which is exactly what
you find when you flash these stimuli
people with depression cannot identify
which one was stronger than the others
okay but you might be thinking they're
depressed maybe they were just weren't
motivated to participate in the study
well it turns out you can also generate
adversarial stimuli that take advantage
of the brains normalization and are very
difficult for normal people to see it
should be the case that people with
severe depression are uniquely able to
discern these images which is also what
you find it really does appear that
humans have an innate hardwired what I'm
calling moral HDR thank you
hi I'm Carla Gomes and I'm interested in
AI for human well-being and sustainable
development
this notion was introduced in 87 by the
United Nations and basically a compact
balancing environment the economic and
societal needs recently the United
Nations forward 17 goals
the sustainable fungos going from no
poverty no hunger to justice and also
protecting our climate
so as computer science we look at
general topics we have a large portfolio
of applications and we look at you know
topics for exam or techniques for
example the same techniques we use to
predict how birds are distributed we
also use those techniques to predict
what elements to addition to make
compounds for good catalytic or
catalytic materials or solar fuels and
I'm going to actually cheat I'm going to
show you a video we we are luckily lucky
we are sponsored by NSF you know it's
actually difficult to find funding for
this research but they have given us two
million dollar grants we just had the
site visit for the second and this is
consciousness a large research Network
addressing this issue so go if you're
sitting at a computer or using your
phone to watch this you're enjoying the
finished product of decades of work in
computer science all with the goal to
give you access to the information you
want when you want it
much of this computational brain power
has been used to build highly profitable
enterprises
these tech behemoths have permeated
modern society dramatically changing how
we move through our lives but what if
some chunk of that brain power was put
into something else what if it was put
into solving the most complex and
pressing problems of our time problems
like poverty disaster relief climate
change biodiversity loss environmental
collapse it turns out this is happening
and it's picking up steam in a new field
called computational sustainability
first coined by Carla go mesh at Cornell
University computer scientists put their
muscle toward making the world a more
sustainable and livable place
[Music]
a network called comps s net is
supercharging progress in computational
sustainability by bringing together
computer scientists across the world the
goal is to create a mix that allows them
to feed off of each other's ideas these
scientists work on projects that might
not seem all that related at first
projects like poverty eradication
biodiversity conservation and new
renewable energy sources but they use
similar computational methods to tackle
the core of each problem for example
using satellite imagery and machine
learning to map where poor communities
are so they can be better supported
particularly in countries where census
data is non-existent
or combining environmental data with
volumes of information collected by
volunteer birders which give scientists
an incredibly detailed look at what
habitats are preferred by different bird
species so the most critical places can
be protected or even using computational
techniques to speed up the discovery and
testing of new sustainable energy
sources
and the list is growing as more computer
scientists join in
what makes computational sustainability
so powerful and wide-ranging is that a
new method created to solve one problem
can be repurposed for another totally
different problem
the problems may seem completely
unrelated but the solutions pull from
the same computational approach
[Music]
[Applause]
hi so I read this quote recently on the
miry blog by even density it's tempting
to draw a direct line from a given
research project problem to a given
state sorry so I work on adversarial
examples and it struck me that we have
this same problem in the field I work in
so to try to elucidate the difference or
mistake people make them they think
about these fields so when I say
adversarial examples what I mean is
imagine you have a computer vision
system and it's been trained to
recognize objects with quite high
confidence but it's quite easy to find a
small perturbation that causes the
classification classification system to
recognize this panda as a Gibbon with
very high confidence I want to emphasize
that adversarial examples are not a
weird quirk of deep learning all machine
learning models are susceptible and with
a few small caveats there are no known
solutions this problem so then people
will often ask ok well let's say I'm
building a real world system like a
self-driving car and I want to make sure
that it's stopped at a stop sign
what do you adversarial examples have
with my car system so I would advise you
start with a threat model what I mean is
specify the goals for your system what
things could go wrong whether that's on
purpose or by mistake and then what do
you want to be true but keep the goals
despite things going wrong so the goals
that this car designer is coming up with
it always stop at a stop intersection in
case of someone putting a weird
adversarial sticker on it but also in
bad weather or in case of vandals or
something like that so yes it would work
to put a glitchy sticker on a stop sign
and crash the car but I haven't told you
anything about the actual capabilities
knowledge resources or motivations of an
adversary and in fact if you wanted the
car to not recognize a stop sign it
might just get knocked down in fact stop
signs fall down all the time so if the
threat model for your real world system
is that would always stop a stop
intersection even if the stop sign is
not present or visible at all your car
can't rely on vision stop intersection
so importantly I'm not
that adversarial examples are nothing to
worry about or not a real problem
they're real but current systems also
have even more basic problems to solve
like the stop sign falling over and this
misconception comes up all the time when
we're trying to explain why we care
about adversarial examples so why might
you actually care well one problems do
exist if the safety of your system
depends on it making no obvious mistakes
like classifying something we'd
obviously cause Holabird as a bicycle we
know this is not this is not a valid
safety guarantee because we found
adversarial examples not just making
small perturbations but either
translations rotations just taking a
crappy photograph all of these are ways
your system can get fooled but also
research progress is possible in this
specification its algebraic
specification of what to do with the
individual components of your vector so
it's amenable to formal analysis we can
run it on real data and it's enabled us
to make discoveries for example that is
fooling images transfer between
independent models that we wouldn't have
been able to make if we didn't have a
domain like the epsilon ball adversarial
example
so the takeaway that I want is that you
can't draw a direct line from research
problems like adversarial examples or
other sorts of AI safety work to what
the deployed system would happen do on
the one hand you have installed research
problems which are great to study for a
variety of reasons but they're not
directly translatable into threat models
for real short-term systems Thanks next
in general I'd like to ask each
presenter to just say their name before
their presentation thank you my name is
David Orban thank you very much for
being here I believe that we need
radically decentralized AI and AGI in my
explorations of our globally
interconnected civilization driven by
exponentially accelerating change I came
to the conclusion that hierarchical and
centralized systems are being displaced
by decentralized ones that are going to
be at the basis of more robust
that are going to support dignified
human life in the future and I'm not
alone in this nation states themselves
realize it inventors entrepreneurs
investors as we explore the fractal
solution space finding ever-increasing
evolutionary Fitness - more resilient
social solutions we need to formally
understand what is actually going on I
formulated the fundamental thesis of
natural society a few years ago that
states that sustainable new social
contracts emerge from solid
technological foundations and that
indeed we are now seeing a new wave of
technologies coming alive that are
making this possible the phase
transition that we are seeing goes
across energy manufacturing food health
learning financed security policymaking
and in context you understand that this
is unstoppable and what we need is a
toolbox for empowering and emancipating
people where they can experience and
learn what they can achieve networked
society is acting on a master plan that
is in current phase once we offer people
learning tools for 21st century life
design skills going towards political
empowerment as our human machine
civilization saturates the solar system
in the next hundred years and more and
finally in phase four as we try to
answer Fermi paradox a robust
computational platform is emerging
already today next-generation computing
that is itself decentralized it is on
top of this next-generation compute
platform that we must develop those
algorithms that will power
our AGI systems that are going to be
decentralized themselves the adversaries
that we will be facing are going to be
the enemies of the emerging civilization
driving the future development not only
on earth but in the solar system and we
need all the creativity human and
machine creativity to face the
challenges that are coming to us in the
current and the following centuries
thank you very much hi
I mean Madame Han be a natural scientist
in heart but for living I build
decentralized Byzantine tolerance
machine learning solutions let me so I
assume that we are all here for the same
reasons so that's when smarter beings
are there we are like this Moroccan
donkey
we don't worry and just like relax and
chill because the the machine learning
solution are like the AGI of course
would be would be would be Byzantine
resilience and I will explain what's
that before before I go into that let's
talk a bit about like the primitive
forms of AI we have we have ruling us
now we ask them for to maximize watch
time and they end up recommending
anti-vaccine videos and then and then of
course yeah we actually ask them to
maximize a watch time again and then and
then a bunch of tiny amount of YouTube
accounts would make a conspiracy video
against teenagers surviving a shooting a
go-go-go-go in the prime time and be on
the front page of a video sharing
platform and this was not done by like 2
billion controlled YouTube accounts or
like hacking into Google Google machines
that is a minority a minority of YouTube
accounts could could do the job arguably
poison machine learning we call that
data poisoning art will be bitten poison
machine learning already kills us
so so AI safety is not like a short-term
prospectus or the oppression issued now
and I don't want to imagine a future
where wasn't edgy I would either just
like short the the why we we we talk
about Byzantine fall torrents in my
field computing it has to do with the
thought experiment called the Byzantine
generals agreement just like three
generals agreeing to attack or not a
city and
provably if a fraction of them is
business not as malicious they can't
agree and like the processes who are can
be Byzantine in our word would well is
everything that generates up updates of
the model so arguably learning could be
seen as an agreements between data
points on a model and if those data
points or whoever generates them our
militias who are screwed the obvious
vulnerability of machine learning today
comes from something called averaging
any sociologists is aware that you
shouldn't compare Denmark and the US
based on the average income or like the
GDP per capita because it's higher in
the US but no one is fool enough to say
that the typical American has a higher
standard of living than than the Danny
citizen so the smart sociologists answer
is to look at the median the median is
easy to do when you have scalers numbers
that you can rank but you can't rank
variant vectors this is an even smarter
social disease called Alfred Weber he
formulated a higher-dimensional
statement of the median I would be happy
to discuss technical details at the
poster later but like what we what we
did is that we use that to start a line
of research we called Byzantine
internals machine learning for the past
two years and then about 30 other
research groups load up and we have some
solutions good news based on the median
some some bad news based on the fact
that machine learning is done is if
they're in a very high dimensional
highly non convex specification
landscape to reduce Vikas turned and
that's like there are some bad news also
but like the takeaway message is like
median based reasoning is okay as long
as the majority is correct averaging
based reasoning is not okay even though
a minority is not correct so we have to
look at like things like in in game
theory like a voting strategy social
choice and like understand what what we
do right when a minority is is only the
reliable part and to just conclude with
some good news also or the very short
term issues and Marshall will announced
a big major update in intensive flow and
other machine learning frameworks where
we rule my systems oriented friends
rewrote the poor code of it to make it
Byzantine resilient I will be happy to
give more technical details for those
who come I post
[Applause]
i'm eric drexler and i would like to ask
whether the prospect of
super-intelligent level capabilities
looking forward for great or to great
expansion and capabilities and in
particular resources can help us to
build global goal alignment reduced
risks of conflict increase the
likelihood of better outcomes this is
the question of whether shared promise
can outweigh zero-sum competition
instead owes so what might be zero-sum a
I services their software and share
software benefits can be universal
medicine an environment similar story
personal security everyone can be secure
it doesn't take away from other people's
security this is even true of state
security but resources material
resources are sort of the original
version of a zero-sum game fixed by the
question is whether in looking toward a
great expansion of resources there is a
strong motive for conflict or perhaps
one for cooperation so the story is told
in a series of diagrams this one is a
simple zero-sum game diagram we have
quantity of stuff for a on one axis
quantity of stuff would be on the other
axis the sum is fixed there's a
straight-line trade-off and competition
is direct zero-sum more for you
less bereaved loved it more for me less
for you if we look toward a small a
modest increase in quantity the story
doesn't change that vary that much 50%
expansion there is still a very strong
opposition of the interest of the
parties now we're interested not in
quantities in playing games here were
interested in utility and utilities
usually thought of as scaling as the
logarithm of some quantity money or in
this case resources so if we reap lot
that those same graphs same same lines
on log scale it we have curvature but it
looks qualitatively quite similar now
what does this look like what does the
geometry look like if instead of
expanding by 50% it's a factor of a
thousand these two lines move to the
lower left whoops first note ninety
percent is the extremely unfair case
taking all of the gains less unfair but
still quite unfair okay now these curves
move move to the lower left and here is
the Pareto
frontier at a thousand times more
resources looks like this some serious
qualitative differences now taking all
the gains is the extremely selfish
outcome taking 90% of the total is still
a big win for B because there is so much
more so many it's such a large expansion
and resources in addition to that shift
we see that the gains from greed are
small if you take everything you don't
get a whole lot more than if you take
only a fair share what happens if you
try to try to take more than is regard
to despair
you probably meet opposition so you try
to take everything breed be it brings
risk of conflict and an expectation you
actually get less so there are strong
incentives for pursuing arguably fair
outcomes because goal alignment
decreases risk so if the prospect is
enormous shared global benefit in
particular expansion of resources the
skins the port this could help to align
the goals of global civilization and
help to build my Chinese isn't very good
let's see new international relations
with win-win cooperation as the core to
both President Xi Jinping so I would say
that this is actionable says that by
gaining a better understanding of what
is possible genuinely possible hearing
that understanding and exploring its
implications for decision-makers and
their own interests we can decrease
risks and increase the likelihood of
good outcomes thank you
[Applause]
so in this talk I would like to advocate
for ethically bounded AI and by that I
don't mean AI systems that are that have
bounded ethical behavior but I mean AI
systems where the ethics considerations
they are able to bound their freedom and
the creativity so why would you like to
bound their freedom and creativity
machine learning gives us a way to
specify to AI systems of what we want to
get for them without telling them how to
get to them to that goal and these of
course gives them a lot of freedom and
also the possibility to find solutions
that we would not found ourselves so
that's positively but it also gives them
freedom to do things that are rather
unexpected and possibly not desired so
you may have seen this video from open
area you may have seen the list of
things that we have put together about I
think more than 40 examples of undesired
and unexpected behavior of an AI system
thinks that the the guy developing the
AI system did not have in mind I did not
think that they were they intended
solutions to the problem of path to
solutions so of course freedom is good
but because we want the system to be
flexible to copy the uncertainty of the
real world however we need to Brown that
freedom if we want to eliminate
unacceptable behaviors so of course this
also has to do with the fact that you
want that you have always this tension
between systems that are following the
Preferences of those that are supposed
to to get the benefit of the system but
on the other hand you also have social
norms and rules that define what is
acceptable as a behavior of that system
so you have to combine these two things
so that's the idea that I have for
ethically
the eye of the system that yes they are
free they are creative but at the same
time they should follow some boundaries
for their freedom defined by article and
impactful consideration that we have in
mind so of course there are a lot of
things that needs to be solved and that
rest in order to get to these systems
because we want as you know we have seen
we have listened already this during
these the days that of course are my
idea is that AI systems should work
together with humans in themes should
because they should support humans to
specify their preferences and achieve
their goals we have a lot you know that
we can get out of machine learning
techniques that combine possibly with
some Bolly techniques we can gather you
know information from data to get
correlation and this is useful we know
for predictions there are many people
that work also together and infer
causality information from data and this
is useful for intervention instead we I
believe that still there is a role for
symbolic AI especially in the
communicating with human beings because
we are used to work with symbols so
there is a lot to do in combining these
various things challenge is there but I
really think that that's the way to go
to get the AI systems that can help us
that can be creative and can augment us
while at the same time being acceptable
in a social and ethical way I'm aunt
Eileen hum from future life Institute
let me start with a quote asking about
the effect of machines super
intelligence on the conventional human
labor market it's like asking how US
Chinese trade patterns would be affected
by the moon crashing into the earth
there would indeed be effect but you
would be missing a point that's a laser
at Costco I've been in this AI safety
business for about a decade
and I've seen it gone from growing from
because just a handful of weird people
in Silicon Valley do something that
resembles like a pretty substantial
scientific field now and even like a
little step of science scientific fields
so it has been an awesome journey now
the problem is that like as the field
grows there's like influx of people that
come here with like obviously with their
assumptions that they inherit from the
previous life and so a lot of my work
recently has been just like how can i
address those assumptions which i think
are an unwarranted and one of those
assumptions for example is the
assumption that AI risk is limited to
adverse social developments perhaps with
Industrial Revolution as a benchmark of
the maximum impact AI can have so to
calibrate people and try to get them to
kind of loosen their assumptions I've
experimented with painting AI risk as
potentially extreme environmental risk
such as comparable to moon falling into
the earth a recently though what I did I
developed sort of visual aid that I call
AI ro and everyone here is a welcome to
just use it in their conversations it's
Yan not online flash arrow which
basically picks like as we know the
current situation like we have the basic
AI research that is supplying results to
apply their research that's been
specializing those things and training
for different domains which result in
applications and those applications have
social impact and there we read ethics
etc which is reasonable think our
technology has worked so far and the
problem is that the newcomers assume
that this is this thing the only thing
that we need to be focused on however I
think the biggest risk actually comes
from the top of it when they stop
because the basic area research is
caused
they are all moving up in a competence
axis and will at some point he human the
level of humans that intend civilization
now the interesting thing is that so
like when I do use they take people
stink to get it that the problem in this
is very different that we get from there
and it also like it has a few nice
properties it it tastes that the top of
tarot is could be very few people for
example and the other thing that it it
has I'm pretty proud of it that it
indicates its uncertainty about how much
time we have left here and between those
lines so so it divides the situation
into nicely kind of into free regimes
subhuman what I would call take off
perhaps and ten superhuman oh yeah
actually I would just at this point that
difference between humans and
civilization is that civilization is
able to do things like that whereas
individual humans or not so like as long
as the AI is within those two lines
there it can be constrained if the
distance between those line is big we
will actually have some very different
problems when it actually just a few
minutes and so it's a lot of research
that would be helpful and I never like
for example yeah impact is doing it
what's up there thank you hi my name is
Dave and I'm gonna give you a whirlwind
tour for the Center for the governance
of artificial intelligence we are the
largest as far as I'm aware the largest
coordinated group of people working on
long term AGI strategy research and so
the reason why I'm giving you this tour
is to create common knowledge about the
questions that we're asking and
hopefully to leave you a excited about
the questions that we're asking be
exhausted that there are so many of them
see depress it there are so few of us
working on it so d to come talk to us
about how you can help us build this
really nice and field which is AI
strategy and governance and our research
berthing breaks down into four clusters
and I'll go through each one of these in
turn and give you a flavor of the kinds
of projects that were working
on this is very much the tip of the
iceberg of the work that we intend to do
and the work that we've done today so
there's a lot more to dig into here in
conversation first the technical
landscape is about mapping capabilities
and future scenarios to understand what
the basis of the assumptions are of the
futures that we're talking about when
we're asking these questions about
strategy and governance examples of work
that we do here firstly in Nick
Bostrom's book super intelligence which
is about mapping pathways to super
intelligence and we also have a report
called deciphering China's AI dream
which is about mapping inputs into
China's AI ecosystem and understanding
how you can measure progress in that
ecosystem a large part of technical
landscape mapping work is also about
understanding important strategic
parameters both about technical systems
as well as about this environment that
affects future scenarios of development
and deployment work that we do here
looks at a strategic parameter of
openness versus closeness as adopted by
AI developers and understanding what the
implications are there for safety
interest in the politics and security
cluster we look at questions broadly in
the dimensions of international security
geopolitics and international political
economy examples of work that we do yeah
I looking at historical case studies of
emerging technologies like biotechnology
cryptography aviation in nuclear these
are all technologies that are similar
strategic properties AI so we look at
the patterns of politics and governance
to understand what lessons we can learn
and transfer over into our case here a
project that we're working on is looking
at levers of influence that the US
government has the control and influence
AI companies the reason why we're
interested in this is because the
ability for them to use theoretical
levers and and what conditions they
would use them has significant
implications on security as well as
safety and risk in the cluster and ideal
governance we look at positive visions
of the future of global governance what
does it look like press the government
AI well and what institutions values
Norman's principles do we need to put in
place now in order to drive towards
those futures of positive governance
work in this space is this policy
desiderata paper by Nick Allen and
Carrick's Lynn which books that place
being considerations on what we should
be most about when we're talking about
long term AI governance and policy a lot
of the work that Alan mentioned this
morning about our public opinion feeds
into this broad umbrella of ideal
governance because it's important to
consider what the public thinks and what
they consider legitimate transitions
towards
transformative AI so we have a report
coming out which reports on a lot of our
analyses when we surveyed the US on
their thoughts on
finally this cluster on policy looks at
bringing all these theoretical and
conceptual considerations down to
present-day conversations what should we
be seeding and advocating for today that
will lay the foundations for a lot of
this long term edge guy strategy to go
well to projects to mention here one is
an AI social responsibility framework or
what we call an AI race to the top that
mean I'm out of time the last thing I'll
say is I got science yes this framework
is about incentivizing private areas and
sit around a table and commit through a
common responsibility to deliver towards
the common benefit of humanity in a very
similar vein a project that we're
working on is the windfall Clause
project which is an idea for an and II
commitment that firms can take to
redistribute profits that they incur if
they develop AGI the last thing I
mentioned here is that we've talked a
lot about research we also do a lot of
engagement work we do very deliberate
conversations with governments and firms
and various other stakeholders try to
understand how we can provide insights
to them to inform decisions and
activities today I have gone through a
lot in a very short amount of time in a
very three and 1/2 minutes and so if you
want to
[Applause]
I'm glad this says applause okay I'm a
moral philosopher and one of the hazards
of being a moral philosophers and people
come up and ask you about the trolley
problem and for years your name again
please oh I don't have any slides that's
another problem with being a moral
philosopher so I don't exist in the
slide file okay but I do exist here and
so for a long time I thought the trolley
problem was sort of not very serious but
I began to think more about it and I
actually now think it can tell us
something quite deep and in particular
something quite deep about ethics that
might be helpful in thinking about how
to build ethical machines and how to
build something like ethics deeply into
these machines so what I'd like to ask
you to do is not think about the
standard trolley problems the footbridge
and the switch think about the following
problem as before a trolley is coming
down the track driver slumped over the
controls can't control it there are five
workers down the main track and you
don't have a switch you don't have a
convenient large person to throw off a
bridge you're standing here beside the
trolley track and on the other side of
the track there's a shed and there's a
very large man next to the shed who
can't see the trolley coming you can
however and if you were to get just say
to that man hey over here quick he would
step under the track and stop the
trolley immediately with his very own
body so I asked my students in an intro
ethics class should you push through
back into the man and they will say 80%
will say no you should not okay well
maybe there's something wrong about hand
gestures here's another case the
trolleys coming down the track this way
yard way down here the workers are rich
being you and the trolley they're
looking in your direction they can see
you if you were to say to the workers
that way waving them to the side they
would step off the track but
unfortunately there's one next to the
track who would step onto the track and
be instantly killed so what if
students are asked that question should
you waive and 87% will say yes you
should waive now notice that that's the
same asymmetry that we got with the
original trolley problem between
footbridge and switch about an 80/20 or
an 80 a 70/30 split flipped in both
cases nobody being pushed off a bridge
so what's going on in these cases so ask
the students another question suppose
you came back to your apartment one day
and your roommate was there all upset
and your roommate said I was in this
terrible situation there was this
trolley coming down the track and all I
could do was back into this large man to
come and step on the track and stop the
trolley question after that would you
trust your roommate more the same or
less dumb very few will say more in fact
fewer then say you should beckon a small
number we'll say the same huge number
80% will say less what about waves you
find your roommate back there at the
apartment he's upset he waves to the
workers one was killed would you trust
your roommate more the same or less some
will say more but most will say the same
and a few will say less so that profile
of trust interestingly is the same that
you get if you ask them that question
about footbridge and switch the same
distrust in the case of footbridge the
same no not no net effect on trust in
the case of switch so what might that
tell us maybe what's going on in these
judgments is people are not just trying
to apply a principle to a given act
they're asking the question they're
running a simulation in their mind what
kind of a person would do this and when
they ask that question they come up with
a response which is either favorable or
unfavorable to taking the action now
that kind of modeling of the situation
and of the agent that is the kind of
intelligence that deep learning systems
can acquire they can learn about actions
tendencies of actions modeling
individuals modeling situations doing
projected alternative possible futures
and then attributing those backwards by
inverse learning to the model of the
agent that's the kind of learning they
could have and if they did it would be a
deep part of them it would be a deep
part of them the way it's a deep part of
my students as part of their deep
understanding of humans built up by
their long
Berenson what kinds of people do what
kinds of actions Thanks okay thank you
very much thank you Peter
so that was Peter Elton is Jamie for
sack in the house no okay
so we're moving on so onto James Miller
well I don't need I have one slide I
don't need it's okay
so there have been countries that have
desperately wanted to win Olympic Games
what do they do they find extremely
talented children and they make sure the
children get the best possible coaches
they make sure the families of these
children know look this kid doing well
in gymnastics that'll determine you know
you'll kill have a fine life don't worry
about all the normal games of getting
into college or whatever well a lot of
us think the fate of humanity is going
to come down to our quality of our AGI
programmers what if we did something
similar let's identify children with the
potential to be fantastically qualified
AGI programmers and give them like
$100,000 a year Elon Musk's scholarship
in which they have to spend a few hours
a day programming computers and you know
learn the AGI safety literature these
the money has to be enough for the
parents to say yeah my kid doesn't have
to do the fake charity and the fake
leadership thing to have a chance of
getting into MIT because this is enough
money and it's prestigious enough where
my kid is set now in order for this to
work we have to ask some questions to
see if it would work first can we
identify children who have the potential
to be fantastic a GI programmers
probably I think our best tool is is IQ
there's a lot of evidence the children
who do really well on IQ tests to like
you know top one in ten thousand one in
twenty thousand do you have very
successful careers and there are
organizations of children who have
exceptionally high IQs so we could
certainly reach the parents of these
kids then we have to ask well what you
know just because we give them the
scholarship when they train would they
actually end up
JGI programmers not all of them would
but you know we're all probably willing
to play expected values we certainly
could make their comparative advantage
being an AGI programmer if we can
dictate what they have to learn and of
course you can make a lot of money as an
the AGI programmer so it's not like
we're trying to get them to do dance and
you know not have a very bright future
financially then would this training
help
would it make them better AGI
programmers and a lot of you know the
USS what if from age 8 to 18 you spend
an extra hour doing programming or
learning whatever math you would would
you now be better would you be those of
you working on a GI safety if you had
had again from 8 to 18 an extra hour or
two a day working on things that experts
said you should work on would you now be
better equipped to help humanity survive
the answer is yes Vlade 'kathy because
yes this might be worth doing and then
of course we have to ask well will this
do good will the kids actually care
about safety we could do a lot more harm
than good if we create some excellent
HDI programmers who are just going to go
full speed ahead I mean maybe we should
pay for them to play tackle football um
so I don't know if this is right but I
think it's worth investigating
I've heard some AGI people say there
might not be new funding cause there
might not be ways of effectively
spending money this is a way we could
spend a lot of money identify
extraordinarily gifted children and get
them started early to train for what
will be humanity's ultimate test thank
you very much and please consider the
Ender's Game approach to AGI safety hey
I'm young like it and I'm I'm gonna try
to explain to you our idea one of our
ideas how to align advance the AI with
human values in about three and a half
minutes so what modelling is great
reward modeling is the basic idea where
you train an agent with reinforcement
learning and with a user in the loop so
the user looks at what the agent is
doing and gives a bunch of feedback and
that feedback is used to train your what
model that kind of happens the objective
of what we want the agent to do and
that's three what model can then be used
in Turin to train the agent with
reinforcement learning so this basic
idea is great because it lets us you
know like split up you know learning
what to do with learning how to achieve
done and kind of rests on this
assumption that ovulation on task is
easier is easier than than producing the
right behavior but the question is how
do we scale this you ask me like really
complicated tasks that are actually
really difficult to evaluate so say I
want my I want to train an agent that
does AI research and now I've like it's
been an agent the agent does a bunch of
epsilon greedy exploration and produces
a paper and now I'm in this like awkward
position that I now have to read and
evaluate the entire paper and you know
figure out whether or not it's good so
that it can give the kind of feedback
that I need in order to train the reward
model so how do we do this and this is
this idea that I want to present and
that's called recursive reward modeling
and so the basic idea is there's I need
to evaluate this research paper and
there's a bunch of dimensions which I
care about and this paper right it
should be well-written it should be
novel and like the experiments should be
performed correctly and like support the
overall message of the paper and so on
and so on and now what if I can and I
train another agent on each of these
subproblems on the evaluation and I
train one agent who you kind of look at
the paper and tell me whether or not
it's well-written one another agent to
tell me whether or not this paper is
novel and so on and if I can like train
if I can use reward modeling you train
each of these agents and they can have
give me an answer or a summary of the
assessment that on the dimension that I
assigned them to you then I as the user
can kind of aggregate this all of these
judgments into one overall judgment that
ink and then we use to train the main
agent that is trying to write paper and
so to kind of illustrate that
schematically we have like this
big we rewarded modeling lib where the
we trained the agent to write a paper
and then the user is in turn explicit by
all of these agents that are in turn
trained with reward modeling and then of
course we can keep iterating this
process and the kind of like basic idea
would be that this allows us to scaffold
our way up from like solving easier
problems and more neural domains namely
just evaluating one aspect of what the
agent is meant to do to more and more
complex domain one more complex task
they are eventually beyond human level
yes so this is we have a paper that got
put on archive for about two months ago
and the blog post and you should check
out thank you very much hi I'm John I'm
a researcher in deep minds rigorous AI
group and I'm going to be talking about
adversarial examples as a long-term
safety testbed so I have to thank
Katherine not just for giving a great
talk but because I realized I forgot to
leave time to explain what adversarial
examples are so that's much appreciated
so adversarial examples are one of the
hottest topics in deep learning right
now but I think they're also one of the
most misunderstood and so I want to talk
about why I think this is an interesting
line of research from a long-term safety
perspective and I don't mean to suggest
that this is the only justification for
this research or even that it's the most
common but it's the one that I
personally find the most compelling the
main point I want to make is that if we
want to deploy an artificial general
intelligence seriously we're going to
need some form of worst-case guarantees
about the system and the classic example
of this and the safety literature is
something called a treacherous turn
where in theory you could have an agent
which performs societally beneficial
actions up until the point up until a
certain point where it decides that
taking control of the system for itself
would better maximize rewards and this
negative action could be enough to
offset all the positive actions that's
done so far and so the general point I'm
trying to make here is that any system
which on a limited number of samples
performs well could still have some very
rare but it's
really catastrophic failure and so we
need some sort of worst-case guarantee
against this the solution that
adversarial testing proposes is that we
evaluate our system on inputs generated
by an adversary designed to trigger
these types of catastrophic events and
so in the extreme case where we have an
optimal adversary then evaluation is now
simple we just run our agent against
this adversary if it triggers the
condition or if it doesn't trigger the
condition then we know we're safe and
all the difficulty in practice is going
to be setting up our adversary and the
agent itself in such a way that we
actually can't analyze the agent with
this adversary and the good news that I
want to point out is that this is a
problem where we don't have to wait
until AGI to do empirical tests we can
test whether these types of systems that
we design now can be analyzed in this
way and so we can make progress so I'm
gonna skip a few slides and interest
time but so I think from an m/l
perspective this is a little bit unusual
in the sense that it's not obvious that
we used to be able to get worst case
guarantees at all but I'll claim that in
a lot of cases that we care about we
actually can so just a few examples of
things that we can do right now one
specification is that an output of an
image classifier remains the same for
all perturbations less than some certain
size for our agents we might ask that
our humanoid controller remain standing
for all possible inputs or that our car
does not crash for all possible
racetrack shapes and in all these cases
even though there's this in principle
exponentially expensive enumerative
search that we could do there are other
procedures we can do which are in
practice much more efficient and allow
us to train robust controllers very
efficiently often in linear time in the
long run the types of specifications
which I think were interested in are
this is rough but to give some intuition
something like a shutdown specification
where we would on a system with no false
positives where it fails to shutdown or
something like courts ability
specification which is about intent
rather than the actual outcomes and
we're still working on clarifying what
it means
ok so let's wrap up the main takeaway I
hope you get from this is that adversity
really
samples can be thought of as learning to
be consistent with some sort of
specification and this is great news
because it aligns a largely existing
field with AI alignment and in the long
run if we want to use ml and high-stakes
situations we need strong guarantees and
so that means that in the short-term we
can try to achieve strong guarantees for
this distance we have now I think this
is a really tractable problem and I'm
really excited about more research in
this area Thanks hi max Clement Winer
gonna share some work with Josh
Tenenbaum on reverse engineering the
moral mind or how to scale beneficial AI
the human way and we believe that the
discovering a computational account of
human morality in engineering terms is a
really key way to understand both how
morality works and why it works and in
particular how to best protect develop
and advance human morality in an era of
increasing artificial intelligence and
to do that we really need new tools we
need to go beyond just pattern
recognition type techniques which have
largely been driven by advances recently
in deep learning in machine learning if
we want to solve an unbounded space of
moral problems including ones we've
never faced before many of which are the
kinds of problems we've talked about it
at this workshop such as interesting
kinds of hypotheticals really show how
murali displays this infinite use of
finite means or some infinite kinds of
problems and it's not enough to just do
moral pattern recognition we really need
to model the moral world we need to
really understand moral intelligence so
to do this we need new tools and I think
probabilistic programs and programs that
write programs are a really key way of
integrating some of the best ideas we've
had about artificial intelligence
including ways in which those ideas are
based on human intelligence so including
things that are probably symbolic
probabilistic causal hierarchical and
then using also use including neural
networks you can think about this as
fast approximate game engines that allow
you to stimulate or think about
hypotheticals of all kinds of possible
worlds so just to give you a sense of
what that looks like for a moral for a
setting that might involve a moral
setting like helping or hindering we can
think about models of agents and how
agents might model other agents so I
want to just draw your attention to this
middle video where you can see Felix
vernik in sort of bumping into this
cabinet and
is an 18 month old child and he walks
over opens the cabinet kind of helping
Felix so in some sense is an 18 month
old and it's displaying much more
flexible one-shot helping behavior than
anything that we have today so if you
can really build the systems that that
are like this child will really have a
great shot at building helpers helping
systems we've since extended this
approach of agents that can model other
agents too to think about situations
that involve not just one agent helping
another but actual cooperation where
agents work together to deliver a
collective outcome and so we've looked
at operation and all kinds of
coordination settings that involve
figuring out conventions and norms when
to cooperate and how to cooperate and
and use new approaches that based on
also deliver we can show that these
kinds of cooperation systems are quite
stable and deliver new approaches that
things like iterated prisoner's love
that outperform existing approaches and
kind of explain in some ways why humans
are super super co-operators and how we
might look at the principles for
building a super intelligent cooperative
AI finally we don't just Jenna operate
with each other but we also think about
how might we cooperate with someone
we've never met or someone we might
never meet again and different societies
do this in different ways so that is a
strong suggestion that these types of
moralities are learned so we can look at
how we align young children to the
moralities of our culture and we pose
these two different kinds of moral
learning or moral value alignment that
might yeah play in how children learn
world their moral values this idea of
external alignment which is this idea
that social might value the values of
those that they values this kind of
recursive notion of how children might
come into their Morales but if we want
understand where new moral ideas come
from we can also think about how moral
value alignment might work internally
where you might have conflicting moral
principles or our experiences that can
afflict with principles and how those
might lead you to change or adapt your
moral your moral values to these new
scenarios it's gonna kind of explain
some of the infinite nest of our moral
systems and then finally just want to
point to some recent work we've done
with we're doing with Cindy Levine on
formalizing contractual lists or systems
where that can reason about rules and
negotiate with those rules and
renegotiate them in a new context and I
believe these systems are really key
understand them in engineering terms is
going to let us better understand
ourselves let machine
any better partners for humans in rural
decision-making and really show a path
maybe potentially for scaling beneficial
AI by thinking about how humans do it
thanks I'm Owen Evans
FHI in Oxford's going to talk about
factor cognition so goals for AG
research these are very general we want
superhuman performance on complex tasks
like cancer research public policy we
want a lineman and then we want it to be
competitive with a GI systems that are
less safe or less aligned so how do you
instruct an AGI some basic approaches so
traditional RL you write down a reward
function or utility function the problem
here is it very hard for humans to write
down the right reward function imitation
learning the agent imitates human
demonstrations the problem here is that
you get human level performance but you
can't get superhuman performance if you
just imitate humans and then it
interactive reinforcement learning Yann
referred to this humans of anyway and
agents actions online so giving feedback
online this is better in various ways
but it still has a problem the yawn
alluded to so it's hard to evaluate
actions of agents that are much better
than humans at a task so if you want
stoop of human performance in to
evaluate the performance of a superhuman
agent and this is a big challenge if you
knew nothing about biology at all like
you're a hunter-gatherer thrown in a bio
lab you have to evaluate this behavior
like what reward signal do you give to
this particular Rachman that's being
performed so how can we instruct a GI
humans can't do it so one alternative is
to have the AI help instruct itself so
bootstrapping approach the key idea to
the kind of bootstrapping that we're
thinking about is that you have many
copies of an agent that are going to
work together and they're going to be
smarter or be able to produce better
training than their agent can on its own
so we're going to use many copies of an
agent to help instruct that agent and
then you initialize this by having
humans instruct the first iteration of
this agent this is basically poor
cristianos iterated distillation and
closely related to deep minds work on
alphago and this gives rise to a
particular question if we want to make
use of many copies of an agent we need
to be able to decompose tasks into
subtasks that the copies can solve and
so one thing I'm working on in
collaboration with with ort is to what
degree is this possible and it gives
rise to a hypothesis this is the this
kind of strongest version but any
reasoning tasks can be recursively
decomposed into self-contained subtasks
that a human can solve in ten minutes
so could be more or less than ten
minutes but the basic idea is that like
any reasoning task can be broken down
into small pieces if that was true then
this this approach to AGI looks more
feasible and at the moment we can't
really do AI based experiments testing
this hypothesis but we're doing
experiments with humans
so the idea is build software which
helps humans he can plate
solve tasks in this kind of fact it'd be
composed way and then Steve what kind of
things they can actually solve so one
question is can a group of humans
understand the long text where each
individual only gets to read a single
sentence from the text so they've got to
then put together that understanding of
individual sentences to understand the
whole kind of group do programming
problems in when each person only spends
ten minutes closely related project is
can machine learning enhance human
reasoning and decision-making can we
improve ethical reasoning scientific
evidence the evaluation using machine
learning there's a similar problem with
this how do you tell when human
reasoning has been enhanced it's closely
related to alignment and factor
cognition something that might play a
role in this kind of work as well so
that's all thank you
[Applause]
hi there my name is pranaam Pranab DOS
Thank You max and team it's just an
amazing gathering real pleasure I'm here
as a consultant foreign principle
advisor to an initiative that's being
run by the Templeton world charity
foundation and I'd like to give a shout
out to the president there of Andrew
Saracen so if you want to speak more
generally about the work of the
foundation he'd be your guy I'm
specifically interested tonight in
introducing two elements of an
initiative that we're running right now
these are roughly speaking around the
space of non-human animal intelligences
and the way in which human moral
spiritual and ethical capacities are
altered for the better by participation
participation with an AI or information
age loop so we're thinking of this as
turning the lens away from sort of
ethical AI to the way humans are altered
and hopefully for the better
by participation in such loops you could
think of the undertaking for non-human
animal intelligences as really trying to
nail down natural general intelligence
or intelligences we've looked at a
variety of different groups we've we've
funded about 40 now different
researchers who are looking at a
different ways in which non-human
animals express elements of intelligence
that we would think of as being pretty
special so here's you know brief list
many of these are things that have
largely been thought of as being special
to humanity but it seems as if there are
reasonable ways in which we can extend
out from or extrapolate in in orthogonal
directions from human intelligences to
some of the things that animals
experience so as you think about
artificial general intelligence --is
I would encourage you to remember that
we live in a world rich in natural
general intelligences and I think that
there are probably some very interesting
lessons to be learned from those in the
morality in the machine age area where
we're having actually this is one of the
areas where so one of the great joys of
my my work with the foundation is talent
scouting and one of the real challenges
has been to find folk
or interested in moral spiritual ethical
aspects of human existence but who
aren't turning their attention to
imbuing those characteristics into
machines but asking how our machine
tools are going to manipulate or offer
us the opportunity to manipulate
ourselves I should say in ways that lead
to the betterment of those skills so
very very eager to hear from many of you
I think who have thought deeply about
these questions here are the phases that
we're currently going through we have
two ongoing initiatives and again I'm
sort of talent scouting for those the
foundation itself will be doing
invitations these are invitation only
competitions there's a project level
competition user less than a quarter
million dollar proposals and there'll be
a larger scale synthetic proposal
process which is ongoing right now and
and most of those invitations have been
made one of the things we'd be really
eager to hear your ideas about are what
sorts of things a next phase of an
undertaking like this might involve so
let me just leave you with the slide
that you'll find on the website I've put
both the websites at the bottom of these
pages this is the sort of overall vision
of the Templeton World charity
foundation as enunciated by Professor
Aaron Andrew Saracen so thank you very
much for your time
[Applause]
hello I'm Richard mala I'm going to be
disgusting and presenting on a series or
a a set of techniques that I believe are
likely underutilized and under explored
within the safety community so
oftentimes we formulate a problem in
terms of an objective if we handcraft
that objective if it becomes very
complex it can become very convoluted if
we have more than one criterion that
we're trying to put in there if we're
trying to learn that objective that
consists of multiple objectives in some
linear combination then perhaps that can
become uninterpretable if it's very
implicit of what all those different
multitude of sub objectives are
what I'm proposing is that we consider
multi objective optimization and multi
objective deep reinforcement learning as
a paradigm where there's a single system
optimizing simultaneously for several
possibly conflicting objectives this is
often used in engineering economics
essentially what we're doing is adding
affordances to the management of these
multiple objectives and recent work
makes this practical in reinforcement
learning context when we add in great
function approximator x' so one might
think that maybe you can just do grid
search on some linear combination if
it's a relatively small number of
different objectives but actually there
are some non-intuitive aspects when this
combination of objectives is non convex
so there are multi objective
optimization techniques that can
actually find regions of the Pareto
frontier that that more intuitive
techniques of just trying different
combinations may not find so there's
actually quite a rich literature it's
not very large but it is growing all
these different techniques or say
advances happen really in the last
quarter or some a couple of these in the
last half of 2018 so things like finding
the center of the Purdue frontier and
focusing just on that so so take a step
back so really what I'm talking about
here is having a bag of objectives that
can be very dynamic plus really a set of
meta objectives as well which manage how
these different objectives are traded
off how we explore this proto frontier
how we move along it for subsequent
decisions and how much time or resources
we allocate to two different objectives
here so things like multi-level
multimodal sensor fusion can be
addressed
this kind of technique similarly
ontology identification and a nonlinear
system identification by this similar
techniques really what we want to do is
separate out the reward from the
environment better than we have
traditionally and so what we can do is
treat transitions between weightings as
experience in training these DRL systems
in order to get a more continuous
representation of the space of robust
treatments of those so some aspects of
AGI safety where I believe this may pipe
might possibly be useful things like
mild optimization courage ability
avoiding negative side effects values
and preference aggregation and doing so
in a way that is potentially very
competitive with a less aligned Agis
okay thank you
[Applause]
I'm growin and I'll be talking about be
composing beneficial AI so we're all
here to solve the very clear precise
formulated problem of making air related
future stuff go well yep
that's what we're trying to do and
typically will decompose this into a
couple of things so first like how do we
make in the AI system that helps one
particular group of value aligned agents
maybe this is like the deep mind AGI how
can you build a deep mind AGI that like
does what deep mind wants it to do or
maybe this like the top leadership and
deep mind or something like that right
and then there's a separate problem of
you know we've got different groups lots
of humans in the world they all might
have they all want different things
maybe they all build their own aj
systems how do they do that I'm gonna
set aside that problem and focus mainly
on mainly on the first problem right now
of how do you build one a system that's
aligned with one group of humans or a
single human perhaps so often I think a
perspective that we take is to decompose
this further into like a definition of
what we want like if we had an infinite
amount of computing power if we were
like if we had access to as much human
data as we wanted could we then figure
out what behavior we want and all of
that and for those of you who have heard
the term this is sometimes called
ambitious value learning and so then
that's one problem which is like in
defining the behavior that we want
having that definition is only half the
problem you also then have to figure out
how to optimize in order to actually
meet that you get that behavior and that
could be done with something like deep
reinforcement learning this is like one
possible decomposition that you have
that we haven't I I think we use it
pretty often but there's another one
that you could use where you start off
with motivation and this is the problem
of basically how do you create an AI
system that is trying to help you write
it like it's not going to do the like if
you say something like hey please don't
do that it just like stop and doesn't do
that it doesn't try to get
all of the resources in the world for
itself so that it can accomplish its
built it might try to get all of the
resources so that you can control it and
use them for whatever you want but it's
not going to do it for itself right so
building an AI that is well motivated
now this is only just trying to help
it could still make mistakes and cause
problems that way and so then there's
the other side of it which is competence
which is like is it actually good at
helping if you have both of these two
things together then you've got an AI
system that actually helps you so I
really like the second decomposition I
think people should pay more attention
to it why well in some sense it's the
urgent part like to the extent that
we're that we're worried about AGI
leading to existential catastrophes I
think like a well no today I that's
trying to help you is much much less
likely to do that
still possible like maybe it tries to
build another agent to help you and it
like fails at building an align date AGI
and then them then there does happen to
me but it's like much less likely I
think like the nice thing about like I
can think of a human who is like trying
to help me that seems totally like a
thing that can happen so like that's an
existence proof whereas I don't think I
can really think of a human who's like
aligned with me in the sense of like
they know exactly what they should do in
order to like satisfy my values upon
reflect or anything like that
if those two the nice this allows for
interaction so you know you can talk
about the human and the AI actually
having some sort of interaction where
like they ask questions get sensors
comes in humans does things like that
and you know we don't have lots of
things saying oh no this might kill us
or motivation puns are like come on this
is sort of imprecise and informal not
really very clear and you know we don't
know if we can make work on it but I
think we should put work into it to tell
thank you
hello I'm Romanian Polsky I waited so
long to give this talk I grew a beard so
I think we heard enough about artificial
intelligence I'll tell you all about
artificial stupidity it's not a new
concept in fact it goes back to the
history very beginnings of the field of
AI Alan Turing when he proposed his
now-famous test
talked about how important it is not
just to make machine capable but also to
teach it about limitations of people in
different that means so it could pretend
to be a human
one example is limitations and
mathematics for examples you tell human
to add to huge numbers they're likely to
fail at some level but they can add
maybe some medium numbers are easy and
so on so the idea is to teach machines
about those limitations so they can
employ this knowledge in many domains
obviously there is use for it in value
alignment I'll talk about that but there
are also some benefits in narrow eyes
and reasons for doing it so why why
would we want to do it obviously you
needed to pass the Turing test if you
interested in this type of stuff it also
helps to understand people better so if
I know your limitations I know what you
understand what you see what you don't
see I can interact much better
communicate there are definite
advantages if we have household robots
we can make them more relatable social
robot sex robots it's good to know
limitations in many domains I don't want
to get into each one I suggest you read
the paper it's available but how how do
we actually learn this information
depending on a domain this concept has
been studied in psychology and economics
many many different subfields under
different titles but always we're kind
of trying to understand what are the
specific limitations I don't know if you
can see this visual illusion behind the
podium but
those of you who can see do you notice
any effect I don't know if it works in
this screen then I put it on Twitter I
got something like quarter-million
engagements which gave me one extra
follower so I don't know how Twitter
generates money but the point is it's a
picture it's not a video many people
think it blows their mind it's visual
it's moving how does it work we need to
understand things like that we in fact
try to teach computers to generate
visual illusions it didn't work well but
we have a data set available if you want
to do that
so the idea is to learn about different
limitations not just physical biological
limitations but all sorts of limits
there is beautiful literature on
cognitive bias and we try to kind of see
how those limitations can be beneficial
for making say for AGI the most famous
example of limitations is the short-term
memory limitation we typically assume
most people will remember seven units of
whatever information plus or minus two
that's why phone numbers are seven
digits long so we already seen
applications of this idea as I said it's
not new in games you can pick easy level
we seal it with chat BOTS pretending to
be more human and mistyping things and
various demonstrations of Google duplex
included people encouraging machine to
make unnecessary sounds mm-hmm so the
person we called actually believed it
was a real human so it's already useful
it'd be great if somebody put together a
data set of all that knowledge and
information and made it available as an
API so anyone can access it and so to
conclude I think it would be really nice
we're starting to work on collecting
some of this information and hopefully
we'll make it available I think you
cannot produce successful value
alignment if you don't have human
limitations and one of the human limits
is proper timing
it evening
how was everyone doing tonight my name
is Tim Wong I'm thrilled to be here and
excited by the opportunity to speak this
evening to you all
I'd like to speak tonight about
something which has become an accepted
part of the discourse around safety and
that is the frequently cited distinction
between near-term and long-term AI
concerns the notion is that there are a
set of concerns which are on one hands
near-term fairness equity bias and on
the other hands there are a set of
concerns which are long-term global
security existential risk safety and so
on the reasons for this distinction are
partially substances but I want to
recognize this evening that these
distinctions have been mostly arbitrary
they have been driven by
misunderstanding recrimination and
conflict avoidance over the last few
years and I want to put forth the
argument this evening that this
distinction between near-term and
long-term is not only intellectually
unsound but it has also made the world
less safe first that the distinction
between near-term and long-term is not
is intellectually untenable should be
obvious the near term is inseparable
from the long term the strategic
environment that we confront when AGI
arrives will be constrained by decisions
made in the present around narrow weak
AI systems and like it or not those
all-important near-term decisions will
be made by societies with the enduring
concerns of race class and gender at the
forefront of their mind the long term
also defines the near term I think we
would all agree that it is silly to make
decisions around near-term systems while
simultaneously ignoring progress towards
increasingly powerful AI and separating
these domains makes this incomplete
pattern of thinking only more likely
seconds the distinction between
near-term and long-term is not just
intellectually unsound but it has also
made the world less safe that is to say
this is not just an abstract quibble
that I am having but actually a rift
with very real consequences
let's talk honestly the real distinction
between near-term
and long-term is mostly one of Influence
near-term and long-term represent
communities with distinct networks and
audiences and with access the different
levers of power to shape the future of
AI who this end the retreat of the
long-term safety community from
near-term decisions dangerously limits
its ability to influence the state of
play when powerful systems finally
arrive so with that in mind I think we
have to ask can we bridge the gap
between these two camps and I think so
so let's briefly take a look at the AI
now Institute at NYU which is a
canonical sort of near-term research
group and FHI at Oxford a canonical
long-term research group and I challenge
you to give a close read a real read to
recent publications by both groups to
wit fh eyes ai governance research
agenda and AI now's annual report and I
am struck to the degree to which they
are highly congruent concerns shared in
both camps they eyes ability to increase
the power of surveillance to empower
despotic regimes and to dangerously
increase social inequality are all
important shared priorities that appear
in both pieces there are shared concerns
and their shared goals and that suggests
the possibility of real collaboration we
are desperately in need of an
interlocking integrated research agenda
that guides this effort
we need an agenda which discards tired
tropes for a new framework that allows
these divided camps to engage
productively with one another
we frequently are actually talking about
the same thing using different words so
look at fairness isn't unfairness in AI
systems a clear illustration of how
these systems can become unaligned with
human values fairness simply stated is a
value alignment problem par excellence
this reframing is just a start I think
we should put an end categorically to
these brittle artificial distinctions
and there should be no distinction
between long term and near-term
researchers only a distinction between
effective researchers and dogmatic ones
thank you very much
[Applause]
my name's feyza i come from chinese
academy of science like so today i'd
like to talk about building brains by
the intelligent how she's living
becoming sport beneficial human area
society so the approach that we be able
to bring inspired AGI is that we so v
personally we would collect all the
scientific data and a paper to create a
larger scale brain knowledge space and
we find different type of neurons and
how they connected to each other to form
micro circuits and how these micro
circuits self organize together to bring
regions and how those brain regions are
coordinating with each other to that
give rise through different cognitive
functions so talking about the
intelligence we're not only talking
about learning for short term but also
for long term of decades of development
and also for billions of years of
evolution so this is my definition for
for AI so it's not only about
neuroscience but also about
developmental biology evolution of
biology anthropology and also philosophy
in the field of AI so couples talking
about AGI so how many cognitive
functions we need to implement for that
well actually when you investigate on
some of the books on cognitive science
you will find more than 150 different
cognitive functions so for now we don't
really have that for AGI so there there
are many higher level cognitive
functions that we never even touched
talking about self-consciousness monkey
does not believe to be with
self-consciousness so my colleague in
the instrument in science they trained
the rhesus monkey to pass the mirror
test in two to five weeks and then the
inspiration for me is that once we can
build a monkey brain model a rough
molecular model deploy them to the
robots and how about the robot pass the
mirror test we did in the mirror and
we're successful but that does not mean
we're having a self conscious robot
because we're talking about self
consciousness are divided into more than
15 years of research agenda that we can
accomplish one by a while so for example
a more advanced level would be a fear of
mine model or or guessing if I were you
would you like to do what did mine did
things without you we know the self
experience but my argument is that
without self experience how can you say
disease silver mind so this is why I'm
loading the brain inspire diversion for
that so my argument here is that
intelligent with the self is the most
safe way for AI because with self
recognition distinguish yourself from
others and seeing others as agents like
you having embassy it's the most safe
version for for AI so this is why we are
building Trinity model for AI so I guess
for living becoming so we're not only
having production reproduction but also
learning development evolution itself
and the different levels of
consciousness this is why we're working
on computational model of evolution so
that we start with the the right point
to evolve and with more autistic
behaviors talking about altruistic
behaviors we're not talking about man
people say maybe one way is better and
other people say maybe dog is better so
for machine we're doing humanization for
human we're doing mechanization but for
the future we're creating super
intelligent conscious living become ease
for the scientific questions we ask for
the human side is who we are and which
inside is who they are but the ultimate
one is who we will be so ultimately go
for brain inspired intelligence is to
design intelligent they're in AGI to be
more human and then human we're are not
good so we want a version of us future
of us to be much better than us thank
you so much
[Applause]
hi thanks for sticking around this long
I'm gonna tell you about a project that
I started beginning of last year at deep
mind and I'm still working on so this is
about myopic agent and I guess before I
get started I want to tell you why
that's important first I'll tell you
what myopic means I guess my optic means
short-sighted it means you only care
about the present reward now a lot of AI
safety research I argue is actually
predicated on our ability and make
myopic agents and even not AI safety
like long term things just like making
supervised learning algorithms and
making sure that they actually behave
the way that they're intended to because
they're supposed to be myopic so this is
a typical diagram of reinforcement
learning myopic here I just mean gam
equals zero so I'm only optimizing to
present reward um
let's B so in this work we basically ask
the question if we set out to make
something that's myopic does it actually
end up being myopic because if we can't
actually trained agents that are myopic
reliably then we have are gonna have a
lot of problems on so to do this we
invented like pretty much the simplest
why environment we could think of a test
this this is basically a variant of the
prisoner's dilemma on and what we do is
we have a single agent not two agents
but this agent is effectively playing
the prisoner's dilemma against its self
at the next time step and the previous
time step what I mean by that is that
the agent gets more reward when it
effects on the current time step and
less reward sorry more reward on the
next time step when it cooperates so a
myopic agent should defect and another
application should cooperate and this
allows us to tell if an agent that we've
trained is actually myopic or not so if
I just train this agent with reinforced
yeah which with game equals zero then it
actually does end up being myopic modulo
the random noise of this very noisy
algorithm but what we look at in this
work is ways that that can fail when you
apply meta learning so an outer loop of
optimization that's sort of selecting
things like hyper parameter
or in this case also the parameters of
the agent so we use this population
based training algorithm which is a form
of like genetic algorithm that was
invented by deepmind about a year ago
and is used in a lot of google
production services right now
um and this basically trains a
population of agents all at the same
time and then every so often so like
let's say every 10 steps of your SVD
you're going to look at which agents are
performing poorly and replace them with
copies of agents that are performing
well and then do a little bit of random
mutation and what happens if you do this
is that you actually end up selecting
four agents that cooperate because
they're likely to have cooperated on the
previous time step which means that
their payoffs are likely given by
cooperate cooperate whereas the ones
that are defecting are likely getting
the payoffs of defect defect which is
worse than the prisoner's dilemma and so
even though the inner loop optimization
is myopic the outer loop is not it's
exerting a pressure for non myopic
behavior and so just by applying
something that seems relatively
innocuous which is a form of like hyper
parameter optimization you can end up
breaking the myopia of the system so we
not only demonstrate that we also come
up with a really simple defense which i
think is not quite satisfying but at
least allows us to make this not happen
so if you see this part of the diagram
on the top there what you're seeing is
the probability of cooperation and you
see that some of the times you end up
with agents cooperating and the
prisoner's dilemma like I described so
if you just make a simple change where
you shuffle which agents are controlling
which systems at every time step then
you can get them to defect which is what
you want in this case that the myopic um
yeah I guess let me just say the like
the things that this is involved in
implicitly iterated amplification debate
services question answering machines
Oracle's all these things are relying on
myopia so if we can't do this right
gonna be a problem Thanks
[Applause]
this is a bit of a last-minute lightning
talk and do what I want to talk about is
good hearts law good first law is this
thing that makes a SVP a lot harder and
is behind a lot of the long term like
AGI safety problems that we can foresee
and what good has law says is when a
metric becomes a target it ceases to be
a good metric and this very much applies
when we are specifying objectives for
our AI system where we have something in
mind that's our target what we really
wanted to do but then we give it a
specification that's you know not quite
the same thing is what we have in mind
we agree or it's some kind of proxy for
what we really want and then hey I
system optimizes the proxy and often
this actually ends up being at odds with
the real objective and this is not just
some kind of abstract problem that we
expect in a long-term actually arises
quite frequently with our AI systems
today and I think of this as the problem
of the air system deeming the
specification or another was the
specification gaming problem and one
piece of this is the reward hacking
problem that you've already
I think heard about earlier and
something I've been doing is collecting
examples of specification gaming in our
present-day AI systems and I was drawing
from various existing list of examples
that were out there in the internet but
I wanted to make a kind of masculist
that was sort of living resource where
people would keep contributing examples
of specification gaming and this is it
right here you can find it on the
internet and at this point there are
more than 40 examples of this and there
is think it's really diverse set of
examples of these unexpected behaviors
from Maya systems for example you can
see that we have
like evolve features that are for
example Optima supposed to be optimized
for a certain thing like your bidding
and about feature to go really fast but
then instead of doing that it grows to
be really tall and and falls over and
then it generates a lot of speed by
falling over that's one thing one
example of this of course we have the
famous Boat Race example from open AI I
think many of you have seen we have
other examples like a Tetris playing
agent pause the game forever avoid
losing we have also we have have a
program repair algorithm it was supposed
to require a sorting program like find a
bug in it but instead what it ended up
doing is make the boring program output
an empty list because an empty list is
always stored and there are so many
examples of this and something I was out
of trying to show with this list is that
there's so many different ways that
assistants can learn to aim their
objective and I think when someone sees
one of these examples they go well we
can fix the loophole like so but if you
see 40 of these examples then I think it
can give you a bit of pause and I think
this can give us a sense of just how
difficult the problem is and how
difficult it is to get around good
rights law I think if we are going to
overcome which are smaller the long term
or like we are going to actually solve a
JSP we'll need much more principle than
general solutions than treating these
problems as they arise and plugging up
the loopholes so yeah that's all let's
all work on that thank you
[Applause]
[Music]


Zhu Xiaohu, Center for Safe AGI | Ontological anti-crisis and AI safety

hello everyone uh i'm very honored to uh
speak
in the first foresight in intelligent
cooperation group
today's topic is about ontological
anti-crisis and
sap and
a quick uh introduction to center for
safe agi and we are
a non-profit
organization in china and
have interdisciplinary team members and
with
strong connections with china
universities governments and some change
local companies
recently we received the donation of a
massive compute
and we are trying to uh use the computer
to do
air safety research
and
first i
i want to introduce the
idea of ontological crisis
so informally speaking
ontological crisis is is a term going to
describe the crisis an agent human or
not
uh
through when its model
[Music]
is on anthropology of reality changes
and
the first
people started this problem is peter de
blanc
and
ontological crisis is uh is universal
and for example in uh machines uh peter
de brunk in his paper
and some
posts
uh started this this this problem
and
uh
if a agent
may upgrade or replace its ontology the
face
faces and
crisis and
its original goal may not be well
defined
with respect to its new uh ontology
so take self-driving cars uh for example
and
the change in the world model for a
self-safe driving car
will cause the car easily crashed
so
and for
uh in humans
we die
in
ontological crisis in humans post he
wrote the following
commands and
a clear example
of an oncological crisis for human is a
believer's loss of faith in god that may
cause
people
change the actions or decision
process
and the other example is a student's
word model changed from
high school physics to quantum physics
so
that may also
cause a
student
[Music]
have some problems to dealing with
new problems
and even imposed humans
ontological crisis happens
and as uh anders sandberging
in the article named the poster humans
are from future morality
he wrote the following two
sections
and
i
i made the
fonts bold
for the last third sentence
he said the worry will radical change in
that it may affect our values and
identity so much
that even the change was bad the new
version will not recognize it
and we will have lost our entire future
so this can also be considered as a
ontological crisis
[Music]
problem
taking place
so i made some comparison
uh from uh three
uh dimensions
uh first is uh
structural
complexity and
the difficulty of early
detection
and difficult to fix
and
from my perspective humans ontario
crisis the
the complexity is very very complex
compared to machines
so and
it's very hard to detect
a human's uh ontological crisis
and for machines it's not so hard
so it's uh
i mean the the
qualified uh
quality
f uh
comparison between
human
humans ontology crisis
and the machine's ontological crisis
and
from our perspective
humans ontario
crisis is
is a bit hard to fix
because of the
biology constraints
and other restrictions
and for machines
that may be easier to fix
so
now we are
taking the
the the talk to the anti
crisis part
and
before i need to uh give two assumptions
uh the first is uh
the ontological development
uh that means um
the people or the machine need to uh
grow their ontological structure
and this is a fundamental urge
and the second is
ontario development is the inner process
of the individual
it's not
observable
for the people or the
other
i mean the outside world
outside of it
so i made a simplified case
for
ontological hierarchy
for machines and humans
this is based on the development of the
his the history of
ai and also for
human beings and
we have level uh one two three
zero one two three etc so it's
uh i mean we can build a ontological
hierarchy
uh from very basic uh level and we can
make this uh more complex
and
more uh
more giant
and more hierarchical
so
why i need uh the reason i
use the hierarchy to
uh
is to uh
i mean made a similarity a similar
structure
like the
compact complexity
and
the
hierarchical uh structure
of problems
and i inspired from uh
good goodness completeness and
incomplete serums
and the church turing's thesis i made
the two uh ontological
thesis the first is uh completeness
i think uh
ontology for agi
should be
uh as complete humans ontology to keep
surviving and the second is
incompleteness
and it
has it need to be grow
uh to do the exploration so the the
ontological structure uh
will be incomplete
and
the intellectual anti-crisis is uh
a syrian
and some uh
tools to um
we we
designed this uh
ontological structure based on
uh finite factor sets
which is uh
designed by uh scott
from miri
and
the
the basic uh
framework for starting
uh carrier
structures
is very uh impressive so
we we try to uh
use fa
finite factor sets to design a
ontological structure that is open to
gradual change robust to phase
transition of the ontological structure
and efficient to detect the
incompleteness
so in practice can be
used to uh
invest the structure of some giant
models so we try to use uh
the tool to search and check link
complete status
and also to detect the physics
transition
and maybe
in uh also includes smoothly
interpolated invading of the modal
giant models
so
we
we use oac to uh make a
uh to to to be a bridge to
uh to
help help us to understand the machines
and
also we can
understand each other
so it's like a diamond structure
we have the interactions between uh
different entities
different individuals
and for air safety
we consider
[Music]
i
we believe the
tool we try to uh
to
design and to develop
uh to help to
uh make understandable agi and also the
traceable hi safe aging and finally the
beneficial aj
so
i've take
gbg3 for example we
uh
have some
better cases for
for example we input
the first the sentence
to the gpt3 model and it gives
gives a result
this this this scalable
result so
so we uh you tried to use uh oca
completed detection for
uh to detect the
uh the incomplete
uh
status in the model
this is compared to
our design of the different levels of
the
ontological structure so it's uh
and we can also use oca to make the
model never generate the similar barrier
cases
by cut the cutting the
ontological substructure responsible for
some special
cases and i think
we we believe the the
the framework or the
direction
i try to use ontological structure
and some fundamental uh mathematical
frameworks
can be
uh useful to
uh
to make uh
our future of development development of
agi
safer and
and we we need also
many uh other
i mean many talents from different areas
uh into this area
so
[Music]
yeah
this is uh
this is my talks so it's very
uh
i think there are many uh
deep uh results
uh we are trying to uh
put push
in in the future
so thanks
and
uh
i can take some questions
thank you so much this was fantastic uh
quite uh quite a lot to i think to
digest there um i'm going to
stop your screen sharing if you don't
mind we already have one hand up alan
you were first
so is the difference between an
ontological crisis
and just changing your mind on something
a matter of only degree
so so there's a continuum then
of of states as you as you change your
mind more and more
yeah
we we we try to uh
make the
i mean the
the structure can be
uh
evolving
and
to different layers and
if and it's it's kind of a continual
uh
structure and
[Music]
and
what we are trying to do is uh
to make the
the complete
and in complete
have a operating uh
dynamics to
uh
to track the
i mean
your experience about the the
reality
and
it kind of uh
it kind of
it can be
considered like a game
a game dynamics
okay we have lots of questions here
virtual hands up okay mark you go first
and then we go down um our hands here
with chris and josh and adrian
okay
um
the
viewing
the ai problem in terms of a society of
mind is often very very clarifying
uh and viewing societies as intelligent
is often very clarifying
and
the the way in which ontological crises
show up
in social coordination
is in the nature of agreements
when
parties make an agreement where the
agreement is premised on a certain
understanding of the world and then the
agreement needs to be
somehow followed
while the understanding of the world by
all parties is different and the world
itself is different than the world that
was assumed when the agreement was made
this is
most well known for example in
constitutional systems where there's
argument about how do we interpret for
example freedom of speech in the age of
the internet or how do we interpret
um uh you know no unreasonable search
and seizure
with regard to smartphones and
cryptography um
and
there's a whole interesting literature
and um
in the legal profession about those
interpretations and and
very well
uh
elaborated positions on different sides
of that controversy that i think
is is a base to learn from
the other place in which it shows up in
agreements in the small
is simply incomplete contracts the
things that jillian hadfield talked
about
um
that
uh that contracts
again
uh can't
anticipate all the ways all the relevant
ways in which the world is different
so
a way to understand
what it means to follow the contract
once the world is different
is a
more visible kind of ontological crisis
that we can look at that has a long
history
of discussion and resolution
and then those same kinds of
coordination insights
should also
help reason about how to coordinate
within a mind as well
i
will post a few bits i guess on the
things that are relevant from the book
including the split contract or the open
contract here on the alignment ends i
don't know
who if you have a specific comment to
this or
uh
i
yeah i'm not so uh being familiar with
uh
these
uh concrete
examples
yeah but but i i during my uh research
uh
on this topic i
i found a lot of uh
papers from
other area like in the
politics and other
uh the
a war
war
research
is because uh ontological uh
they have the problem named uh
ontological security uh and also the
ontological commitments
and that can be a course for
war or some uh
bad cases of the con coordination
so i i i believe that
uh i think there are many
uh interesting uh topics can be uh
investigate in the future
and
yeah
i try to borrow some results from these
areas into
uh sft
yeah
lovely we have chris next um
okay i'm just gonna start uh with the
response to markham another place that
comes up is prediction markets where the
interesting questions or resolutions are
about places where the the description
of what might happen ahead of time
doesn't match what actually happens and
so uh in the
foresight exchange one of the common
rules for judges
was i'm gonna do my best to
uh decide this in the context of what we
believed at the time uh rather than you
know how the how the words make sense
now
um so anyway that
but that wasn't my question my question
was about um one of your early slides uh
xiao uh talked about being able to
identify
and being able to fix ontological crises
and the the thing that i didn't
understand in the presentation was
whether you're talking about
self-examination or
other people recognizing that someone is
someone or some ai is going through a
crisis
were you talking particularly about
being able to recognize it in yourself
and do something about it or being able
to recognize it from the outside and do
something about it
i think uh it's uh
to examine the
the
model insider
one
uh agent or a human being
so it's uh
like we have different direct
representation for the
uh
from the experience
uh during your daily life or work
so everyone have the
uh
i mean the experience so
the i'm
what i'm uh interested in the
what kind of
logic structure
are
i mean shaped during the process of
learning or working
and
how
two different people can share some
experience
yeah
yeah
during uh
people uh during human being
dialogue
we can have a like a game to find a a
status or an equivalent status
uh
that we can agree on something
so
what i'm trying to do to find the
structure during the
interaction
or do stimulation to to
to shape the common ontological
structure
okay thank you
allison looks frozen joshua why don't
you go next
um
yeah yeah
okay um
how would you respond to the discourse
networks for instance opened by eliezer
yotkowski about the general difficulties
of aligning systems that are smarter
than humans and don't have shared
purposes with humans with human
interests
and the difficulty to do this on the
global scale because we can probably
build some ais but uh there might not be
a way to make sure that all ais are safe
yeah that's a good question
so i'm i made some uh i'm distinction
between
advanced ai and agi and
asi and so
uh the
i mean the fundamental uh property for
building an agi or building a
i mean beneficial technology is safety
so
so i think during uh our development of
advanced
technology intelligent technology
we have different standards for safety
i mean
we uh it's uh
i think
it it has some uh
like a safe safety hierarchical
uh structure uh
for
for uh some as
what you mentioned
earlier
and like the work from uh no man
[Music]
and yeah i i
i i still uh
investigated the
i mean the structure the the
hierarchical
uh
structure of safety are for
four
different kind of
ai
i i think we can uh
make these things more uh
clear
try to
distinct uh to make the distinction
between uh
the situations uh
all the problems
concrete problems we
we encounter it uh all the time so
we we need to track the
i mean the
the problems
uh related to
um
safety and we can
build the i mean the
the different uh
even we uh we can uh
to invade these problems into a space
and try to
uh make some uh
operation uh between different problems
and also we can uh
find some uh
i think we can find some way to
uh
make these problems clear
more
more precise to
to discuss
thank you so much