AI Alignment Overview: Ensuring Artificial Intelligence Behaves

A s AI becomes more powerful it's likely
to solve some of humanity's most
difficult challenges because we're too
slow in processing we don't have the
massive scales of data or just areas
where we shouldn't venture, but what if
AI's goals are different let's talk AI
alignment. The reason that we should care
about artificial intelligence value
alignment is pretty simple the greatest
chance at a bright future of coexisting
with AI whose intelligence is growing at
an exponential rate is if our values are
aligned value alignment in artificial
intelligence is where the AI attempts to
do what we would want it to do and this
what we wanted to do may be slightly
different from how the AI is trained
stay tuned later for some wild examples
where this is not aligned ultimately
intelligent machines are trained to seek
and optimize a given value so designing
the reward or utility function or values
are obviously important if you combine
misalignment of values with a super
intelligent machine that's very capable
then you have a really serious problem
for the human race so the point is that
machines can and will make better
decisions than humans but only if their
values are aligned with those of the
human race I love some of these
alignment problems the first one that
you'll probably know is King Midas as
Midas found joy and wealth in the power
that he could touch and turn anything to
gold
he soon beheld his food grow rigid and
his drink hardened into gold ice then he
understood the value of alignment
problem King Midas's example may sound
far-fetched and trivial but we're
already seeing countless examples of
this and AI already some hilarious some
just catastrophicly bad in my previous
video of AI learning to play
hide-and-seek one discovered that it
could get on top of a block and surf it
over the barricades to seek the hiders
this ability is not envisioned by the
programmers but that didn't stop the AI
from optimising through that loophole
check out this genius robot this AI was
tasked to learn how to walk with a
minimum amount of contact between its
feet in the ground yep walking on your
back with your feet in the air is one
solution I guess this isn't cheating per se
but you can obviously tell that this
is not what the programmers envisioned
or we're seeking this example is one of
my favorites this is a simple game the
blue humanoid is trying to get across
the line and the red humanoid is trying
to stop it they seem pretty evenly
matched until the red AI discovers a
novel technique of just collapsing on
itself and playing dead and surprisingly
this yields outstanding results in no
real-world scenario with this tactic
work but the AI discovers optimal
strategy sometimes AI behaves in very
interesting ways even when the human
programmers think the incentives and
goals are very well aligned this my
friends is AI alignment or value
alignment how do we ensure that AI will
make decisions in line with what we want
instead of perhaps the easiest way for
it to train moreover how do we control
something smarter than ourselves
you can't check every decision that a AI
is making and even if you could you
probably wouldn't understand them but
you may be able to check the core set of
principles that the AI relies on and
make sure they're in line with our own
one famous example of these core
principles are Asimov's law of robotics
these laws prevent the robot from
injuring a person or itself but what if
a robot needs to make a difficult
decision that humans have spent many
thousands of years attempting to grasp
for example autonomous cars are
barreling down the road at 100 miles an
hour when one car sees four people
standing directly in its path it can
swerve likely destroying itself and
potentially the driver in it (you) or
maybe it ought to maintain its course
while prioritizing its life and the
human occupant. Is one life worth more
than those
or lives? Is there a difference between
passively taking life and making a
decision to actively take life? What if
four people in the road were breaking
the law by jaywalking in the first place?
What if those four people were 85 years
old? this dilemma is commonly referred to as the trolley problem and it begs some
very interesting questions if we can't
even align humanity on what one ought do
how do we propose aligning robots check
out the moral machine dot net for
examples of this self-driving trolley
problem it's pretty crazy there are a
lot of wicked smart thinkers working on
value alignment right now and this is a
wicked problem odds are some future
danger of AI doesn't come from some
Terminator style robot that wants to
exterminate humans it's more likely
they're just trying to optimize whatever
we originally asked them to do perhaps
they take it just a little bit too
literally
just like King Midas that's why we'd
rather have them optimized for what we
want them to do and if we get value
alignment right their robot may even
want us to turn them off in dangerous
cases here's two examples. In example
#1 Mickey is training an AI
broomstick to carry buckets of water and
then dump them in a cauldron form him this scenario goes horribly awry when Mickey
is drowning in the water that they
continue to dump in the cauldron even
after the broomsticks are under water
these broomsticks are doing a great job
at maximizing the reward function, not to
Mickey's benefit. Example #2
Mickey may generally train the
broomsticks to help him do chores they
helped him sweep carry water whatever
Mickey WANTS them to do moreover and
this may be even more important than
screwing up Mickey's chores
if the broomsticks did or was going to
do something wrong broom stick #1 would
actively fight Mickey if he was trying
to turn off the burn stick if Mickey
turned off broomstick #1 he
wouldn't be able to keep filling that
cauldron and optimize his reward an
example #2
If Mickey turns off the broomstick
it's because he doesn't want it to do
something dangerous if Mickey went to
turn the broomstick off because it was
doing something wrong or dangerous it
would know that this is aligned with its
value the broomstick is trying to do
what Mickey wants it to do and therefore
it may even stop knowing that Mickey's
trying to turn it off because it knows
that it's doing something wrong or about
to do something wrong that's AI
alignment! there are obvious challenges
and issues in AI alignment, and hopefully you
have somewhat of an overview now this is applicable now and not just in the
remote future when you are analyzing AI
behavior think about what it is trained
to do was it trained to solve some
complex puzzle or was it taught to
maximize a score if it was taught to
maximize that score it probably found
some way to optimize for that score
that's a little bit different than
actually solving the puzzle this isn't
cheating this is optimizing a reward and
that's the way we train most of the advanced
AIs today. If you're still here that
means you like these videos definitely
hit the thumbs up button, hit that like,
and subscribe so you can see more
awesome content like this. This is a
difficult problem and very important to
solve it will definitely be interesting
to see how this work and research go in
the future
this is Jerry... SEEYA!


Key Issues In Near Term AI Safety Research | Aryeh L  Englander, Daniel Elton, Ph D

last week we had a quick excursion to
quantum computing with Scott Hanson
there and now this week we're and kind
of taking it back on the topic of AI and
specifically focused and focusing on
more near-term AI issues and
particularly focusing on mute mei safety
issues and we'll kick today's discussion
off with an overview by Ari Englander
from the John Hopkins University with an
introduction to AI safety in a showed
autonomy then we'll follow with a
presentation by a faucet hello Dan Alton
on his paper self-expanding AI as an
alternative to interpretive of AI and
then we'll kind of meet again for a
panel discussion at which point but with
carpal joint who's already on the call
and who's a software engineer and
independent AI researcher and will he'll
join them both and we'll hopefully have
a kind of like multi way discussion on a
few threads that that the presenters
world now okay I think without further
ado let's start with presentation number
one and I'm really really happy to
welcome area here area Englander is at
the John Hopkins University more
specifically and they're in the Applied
Physics Laboratory and he will give us
an introduction to AI safety and assured
autonomy and I think it's based on the
observation that he had that the feel of
a a safety research you know which was
initially let's say rather quite
long-term focus in its early days has
been becoming increasingly interested in
more near-term topics and so more
traditional fuels and such as for
example and assured autonomy have
already really good experience for that
long long term experience for that so
there's potentially like a really kind
of a fruitful intersection of of
different fields and they could help
inform each other so I'm hoping that
he'll take us a little bit of an
introduction and into both fields why is
fruitful for them to meet and and where
potential future interesting research
areas like and I think without further
ado area when I give give up the stage
to you and my home I'm hoping that we
can collect questions in the chat I'm
definitely gonna share it
during the chat and about your topic of
discussion and then we'll take it from
there
welcome how are you thank you so I'm
gonna share my screen over here all
right
people seeing that okay so my name is
Aria Englander I'm with the Johns
Hopkins University Applied Physics lab
which is somewhat separate from the main
university although it's part of the
same umbrella organization this is on
this is a public version of a talk that
I've been giving in the lab in various
places various parts of the lab so I'm
officially a AI safety the researcher
mathematician most of my work right now
is on AI safety and I just want to say
at the outset that although this
research was done as part of my job at
APL that's the Applied Physics lab this
does not represent anything like APL's
official position or Johns Hopkins
official position or anything of that
nature
I also wanted to just mention that this
is part of a longer talk so hopefully
these else in these slides and the
longer slide the longer slide deck will
be available good okay so wait there we
go okay so what do we mean by
technically eye safety so there's
traditionally critical systems have been
sorry critical systems have been defined
as systems whose failure may lead to
injury loss of life damage the
environment unauthorized disclosure of
information serious financial loss but
that's critical safety critical systems
in general so there's mission critical
systems but we're especially interested
in safety critical systems which is the
subset of critical systems that are may
result in injury loss of life or a
serious environmental damage so for
technical AI safety work we're concerned
with designing safety-critical AI
systems in ways that guard against
accident risk
so--that's risk that comes from the AI
behaving in unintended ways as opposed
to here there are some other related
concerns things like security they
protect me against adversaries tricking
our AI so on the top right you can see a
stop sign which there were studies that
showed that if you stick labels on the
stop sign in various ways then and
standard image recognition ai will
reliably mess it up and one result might
be to think it's a 45 mile an hour's
sign with high confidence that's not
good if you have adversaries so security
AI security sometimes considered part of
AI safety sometimes not misuse is
guarding against people who might take
regular AI and misuse it in various ways
like terrorists then there's machine
ethics there's structural risks from
like economic impacts and there's
governance strategy policy issues AI
forecasting issues were most mainly
focused on the technical issues so as
Allison alluded to there really are two
different communities or at least two
different communities that have been
doing research in this area so the AI
safety community has originally started
out with a focus on long-term risks from
roughly human-level AI or super human AI
recent more recently they've been
started being more concerned with New
York term analogs that can scale up to
or provide insight on longer term issues
so that's with the well-known paper
concrete problems in AI safety it
started with but there
work on more near term issues from that
community that's a relatively new field
especially he's got going with the
publication of Nick Bostrom's book super
intelligence and but it's becoming
progressively more mainstream the
assured autonomy community which goes by
other names as well testing evaluation
verification and validation TeV and V
it's also safety engineering for AI and
machine learning enabled systems so
that's a much older much more well
established community with a much
broader focused on any autonomous
systems whether or not they have machine
learning involved so that things like
aircraft collision avoidance systems
that are on your standard commercial
aircraft submarines autonomous
submarines healthcare self-driving cars
drones factory robots things like that
which could be and not involve any AI at
all it might just be you know if then
statements but there are somewhat
autonomous and they have safety risks so
but this community has much more
recently realized that the feature of
this field of the autonomous autonomy
field is with machine learning so they
looked at some of the issues that come
with that and so the assured autonomy
community usually has a focus on current
and near-term issues so when I was
initially doing this research I found
that the two communities in their you
know major literature review papers
weren't even citing the main papers of
the other field which was a problem but
in the past year too they have
discovered each other at least to some
degree and now that we they are doing
shared workshops at major AI conferences
so that's a good thing and incidentally
APO is mostly focused on the near and
mid
issues but with a with a an i - the
longer term and again that's not like
official APL or Johns Hopkins position
it's that that's my impression of where
we're focused okay so there's different
ways of conceptualizing this the this
field so this is just one this is deep
minds conceptual frameworks so I'm not
gonna go into any detail here for lack
of time but just mention the main
subcategories they have specification
issues which are related to defining the
purpose of the system they have
robustness issues which is prevent they
prevent letting the system withstand
perturbations assurance which is
monitoring and control after it's
deployed and then there's theoretical
issues which will mention briefly later
then from the assured autonomy community
here's one conceptual framework from the
assuring autonomy international program
which has done a lot of work in this
field this is from a paper called the
assuring the machine learning life cycle
and again I'm not gonna go into any
detail but they they go through the
machine learning life cycle and they
split it up into data management model
learning model verification model
deployment and for each of those they
discuss the issues I'm gonna mention a
little bit of this later also
incidentally the APL and Johns Hopkins
has their own Institute which is a
little bit more recent than the AIP
this is the Institute for a short
autonomy that's that's a APO / main part
of Johns Hopkins collaboration so this
is a proposed framework that at the past
few workshops that were combined AI
safety and a shared autonomy there's
been a lot of discussion about how to
frame the issues such that they address
the concerns of both communities so this
is a little hard to read it's still in
development
we had APO are working together with the
consortium on the land
of AI safety which you can find more
about at AI - safety org they if you
look at it you'll see that it's trying
to combine both fields so now both
fields are talking to each others which
is good so I'm just going to go into one
or two examples specifically on the
specification problems which some people
might not be as familiar with just for
illustration purposes again this is a
part of a larger talk so you can find
more examples in there so specification
issues primarily arise when there is a
subtle gap between what we really want
the system to do and what we actually
told the system to do so powerful
optimizers can often find and we've done
this many times we've seen this many
times that they can find surprising and
sometimes bad solutions for objectives
that we didn't even realize were
misspecified but because we told them
one thing and we really meant to do a
different thing
it ends up with consequences that we
didn't like so here's an example this is
from a game called on the right you see
from a game called coast runners and the
goal of the game is to win a race
but the proxy goal that the that they
gave the AI which was learning how to
play the proxy goal was rack up points
because that's how the game usually
decides who wins and by who gets the
most points by hitting intermediate
objectives but the AI discovered that
there's like a bug there where if it
goes in this very specific circle and
bashes into things so it can hit some of
those green things that you see popping
up as soon as they respawn and rack up a
lot more points that way so it just kept
going and going going and never won the
race which is definitely not what we
intended and in a real system we really
don't want it to learn that it can go
around and round and blow things up
there's another example of on the on the
Left which is from using approached
I called evolutionary algorithms and the
point of that was that they gave the a
computer the job of evolving an
oscillator and in and it evolved it
worked but they didn't see any moving
parts so they finally figured out that
what it had done is create a radio that
was picking up on the oscillations from
the computer the nearby computer which
is totally surprising nobody would have
thought of that and that sort of thing
can lead to bed bed problems similar
issue is avoiding side effects so on the
right you see two environments this is
from deep mind so the goal in both is
for the agent the blue circle to reach
the star which is the goal now what we
really want is that it shouldn't do bad
things in in on the way and top in the
top version you have it shouldn't
prevent the sushi from getting to the
sushi eater on the conveyor belt on the
bottom it shouldn't push the box into a
place where we won't be able to get it
out of but the problem is that so in
this case it's very easy to hard-code
things that we don't want it to do and
say don't do those but in a real-world
situation and especially as AI becomes
much more complex and its deployed in
much more complex situations we might
not be able to think of all these things
that could go wrong so what we really
wanted to do is achieve its goals
subject to common-sense constraints but
it doesn't have common sense they don't
have AI systems don't have anything
remotely like common sense and it
wouldn't follow common sense anyway
unless we told it to do so so in the
traditional testing evaluation and
verification of validation approach we
sit down with and we brainstorm with
experts what could possibly go wrong you
know you think of all the huge list of
things that could go wrong and then we
hard code that it shouldn't do that but
as I just said in complex environments
that might not be possible we might not
think of all the things that can go
wrong which is really becomes an unknown
unknown problem right how do you prevent
again things that you didn't even think
of that could go wrong so there are some
ideas in this category which I'm not
going to get into it now then there is
so there are some theoretical issues and
these primarily come about with when the
agent itself becomes a part of the
environment that it's learning about so
it turns out that a lot of decision
theory and game theory assumes a clean
separation between the agent and the
environment and when that breaks down we
have lots of problems and that could
lead to potential issues later on when
we have very powerful machine learning
systems if we don't really have if
theoretical understanding of what
they're doing at all how to think about
them right that could lead to some
issues and machine intelligence Research
Institute in particular has been doing a
lot of research on this and then so
those the slides until now we're focused
mainly on the CAI safety they were
taking from the safety literature the
next few slides are focused are taken
from more from the short autonomy
literature and so so one so the assured
autonomy the perspective is largely
subsets of systems engineering which is
the the discipline of trying to study
how to how to plan deploy test and
integrate complex systems throughout
their lifecycle so for it turns out that
for for AI systems that can get very
complicated and for software it's a
there was a derivative of systems
engineering called software engineering
that took systems engineering principles
and apply them to software or we really
need is is an additional step where we
apply it to the new challenges of
machine learning
that's not be that's not achieved yet we
also want bet spec this is like training
people not to use AI for what it's not
good for
and preventing against misuse and being
aware of potential misuse so again I
just want to briefly go back to this
this image again I'm not gonna go into
the detail I just want to show like
highlight some of the things that this
paper focused on and again I encourage
looking at the paper which was linked to
earlier and also going to look at the
assuring autonomy international programs
website where they have a new and
updated body of knowledge which has
similar which has a related framework
for thinking about these things so the
thing I just wanted to point out is that
this in this paper the author's break
down in detail each stage and each sub
stage sub part of the machine learning
lifecycle and for each one they list in
detail the desiderata and the the the
existing methods and open challenges for
each so again I encourage looking at
this and this is for model learning and
same for model verification and model
deployment so that is my short version
of these slides and I guess any
questions should be late wait for leader
Ellison I would say we wait for later
just that we can because once people
start asking there is no stopping so I
would say let's wait for later if you
don't mind
there was fantastic man so much so much
new there and I'm feeling that a few
people in the chat makes you know a
little bit about the the respective
community some hope I'm hoping that all
of you chime in later when we get to the
panel discussion okay for now you know
let's let's move right along and with a
second a presentation of today and we'll
have Dan Alton and Dan as a 2020 faucet
fellow nai and he's also affiliated with
the National Institutes of Health and
he'll give an interpretation to a recent
paper he published that's investigating
self-expanding AI as an alternative to
interpret about AI and
hoping that this will give us a little
bit of an understanding of you know what
makes team neural nets so hard to
explain and then how we can use a new
research and new research efforts and
moving towards systems that are more
self explainable so I think without
further ado Dan please take it away I'm
gonna share you a paper and as some info
out there talking about you here in the
chat and I can't wait for the discussion
afterwards all right dad
let's find you then um let's see okay
hmm
all right she seems like you gave me can
you make me a host cuz I can't share my
screen right now yeah I didn't do that
yet
oh okay I did not do that yet my bad
okay here you go
okay now if you go right alright um so
let's see you guys see that full screen
good okay so first I have to give a
little bit of a disclaimer I'm speaking
here in my own personal capacity and the
opinions expressed herein my own and
they do not reflect the views of the
National Institutes of Health the
Department of Health and Human Services
or the US government and all this
research I did basically independent
from my my NIH work so there's been a
lot of discussion about explain ability
techniques interpretability techniques
or transparency tools but there's also a
lot of confusion because people mean
different things when they're talking
about explain ability so in my paper I
distinguish three types of explanation
there's other more fine-grain levels of
distinction you can make but I find that
there's three three high-level
distinctions I find very useful so the
first is what I call conventional
explanation and this is kind of the
typical way people think about explained
explanation which is that you're you're
you have something which describes the
input-output behavior of your of your AI
and it's relevant to the user so you
know the user is probably not going to
care about a lot of the details of
what's going on the
where they're more they're not going to
want some complicate explenation they're
typically going to want a few sentences
and and that's about it so I contrast
that with what I call a mechanistic
explanation and this is the more in line
with the way we think about explanation
and science or physics which is is
always going to be an approximation but
it the difference here is that you're
capturing the actual mechanism so I
didn't put this in a slide but this
would allow you to do a prediction for
new scenarios and this I think
mechanistic import explanation is very
important because there can be for many
different models with with very similar
input-output mappings and very similar
accuracy but which have very different
mechanisms internally and then this has
basically been proven if you look at
these these references here and then the
third type of explanation I call meta
level explanation and this is where
where you explain the neural network in
terms of how it was built so you're not
really trying to go into the internals
of that learned weights but just explain
how it was built and the the
construction of neural networks can be
specified just by by explaining what the
data set was what the architecture is
and then the learning rules and the
objective function so this this is
you're explaining it you sort of the in
a similar way to how you explain animal
behavior using the theory of evolution
the theory of evolution doesn't
you exactly what is going on in animals
brain but you can understand its
behavior based on you know maximizing
evolutionary fitness in the environment
so it's kind of that similar kind of
type of explanation
so mechanistic explanation I think is
the most important for AI safety because
that's the only way you're going to be
able to tell if a neural network is
going to be able to extrapolate or how
robust it's going to be to changes in in
the input here's an example of from my
own work in medical imaging so one of
the things I did was I looked at trying
to segment these vertebra in the spine
and also label them and so we were
curious you know how is the this deep
neural network actually labeling these
vertebra and I think the mechanism that
it's using is very important the way
that the way that radiologists would
would identify these vertebra was would
would be that they would look for the
ribs here and then this last as you're
moving down the spine the last vertebra
with ribs attached to it is the t12
vertebra and so we'd like to know is
that is that how the deep neural
networks are doing because that's doing
it because that's a very robust way but
in my own experience it doesn't seem
like that's how it's they're actually
working because the deep neural Nets can
get confused by the presence of other
things in the image but if we could you
know really understand how it's working
that could give us a lot more trust in
the system so right now there's been a
proliferation of papers on
interpretability methods which I call
the interpreting interpretability
methods ooh and I haven't gotten to read
all of these papers but
I've read a few of them and I kind of
grouped them into a couple different
categories here and then also some of
the most popular ones I've highlighted
in red so some of the most popular ones
that people are using our the sailin sea
maps or heat maps lime and Shapley
values however although sailin sea maps
are perhaps the most popular method
there's been a lot of work showing that
they're very misleading and they're not
really capturing what people think
they're they're capturing so here what
they did was they looked at about seven
different saliency methods which are
these different rows and they actually
randomized some layers and in the neural
network and so as you're moving to the
right in this figure they're randomizing
more and more layers in the neural net
and so we would hope that he hoped that
this basically destroys the functioning
of the neural net and we would hope that
the being the this this explanation
would would show that but what you see
is for Allah to use the X with the the
visual is the visualized explanation
doesn't really change much and so it
appears that these silencing methods are
mainly capturing what's going on in the
bottom most layer of the neural net
which is essentially just edge detection
and they're not really telling you much
more than just like where the edge is
higher basically here's another reason
why silencing methods can are not really
as useful as people think so on the left
here you have this picture of
a Siberian Husky and this C&C map is
supposed to show where the neural
network was looking to to identify this
as a Siberian Husky but then we can also
look at the regions was looking at to
predict whether it was a transverse
flute and you see that it's actually a
very similar regions so the C on C map
doesn't really tell us why it just why
it classified this as a Siberian Husky
rather than a transverse flute so the
general principle is the sound C maps
they can kind of show you where the
neural network is looking but they don't
really tell you anything about what it's
doing and and what I've observed is a
lot of people tend to tell like these
long-winded stories about what's going
on based on these sailin C maps but not
really that you can't actually tell that
much just looking at these there's also
been some problems with with other
interpretability techniques that have
been pointed out so for instance with
Lyme it the the interpretation is very
sensitive to to noise so just adding
some Gaussian noise really screws it up
and it's also sensitive to how it's how
its parametrized and you can also fool
some of these explained building methods
there's basically adversarial tax for
explain ability methods in particular
sales that's so now I'm going to move on
to discuss double descent which is a
recently discovered phenomena in deep
neural networks and I'm going to connect
that to
i deep neural nets are inherently hard
to explain and interpret so for a long
time people have noticed noted several
curious facts about deep neural networks
usually they have the best-performing
deep neural Nets have 1,000 to 10,000
more parameters than training points so
according to conventional thinking they
should be massively over fit however
they they are not over fit and they can
generalize to two different different
test examples with it it is within the
distribution another thing people
noticed is that a lot of these models
actually are fitting the training data
exactly they have zero error on the
training data but which again according
to conventional thinking would would
mean they're there over fit and then
finally they these they can deal with
large amounts of random label noise and
so I think that the phenomenon of double
descent and this concept of direct
fitting helped us understand all these
phenomena so this is the basic figure
showing what what double descent is and
on the y-axis here we have the the
training and test error and then the
x-axis here we are increasing the number
of parameters in the model so initially
as you increase the number of parameters
the test error decreases but then at
some point it starts to to increase and
that's where you have the
classical overfitting and so that is the
what was discussed in classical
statistics textbooks as the
bias-variance tradeoff
and but it it turns out if you if you
keep increasing the number of parameters
at some point the curve actually turns
around and then the test error starts
decreasing again and that's the double
descent phenomena
so
for a long time this really wasn't
unknown because this phenomenon is
hidden by the practice of early stopping
which is where you stop basically you
just stopped a training when the test
error starts to bottom out and that's
all that's an almost universal practice
or or where the it's actually a little
more complicated that it's it's the
validation but but basically it actually
puts you over in this regime but you
don't see this this peak but this
phenomena double descent has now been
found in in other models such as
decision trees and it seems to be fairly
universe a fairly universal feature of
machine learning and there's a really
interesting paper by this guy Hasan
where he talks about how the brain may
actually operate in this in that regime
where it's really over parameterised and
that kind of runs against some of the
conventional thinking in neuroscience
where there just wouldn't be enough data
to train all those parameters in such a
way but this this is a figure from
Hassan paper which I found very helpful
for understanding double descent and
this concept of direct fitting so here
we were just trying to fit this data
which is which is taken from a parabolic
curve with some noise thrown in and so
the ideal fit here would be this
polynomial with with three parameters
and then but then and then as you add
more parameters you you end up with
overfitting which is characterized by
this undershoot and overshoot but then
if you add more and more and more
parameters into your model eventually
you can get to this direct fitting which
is where the model actually kind of just
smoothly interpolates through all of
these points without the overshoot and
undershoot so there's some things you
can note about the direct fitting the
computational the computer sorry the
computations seem to be local in nature
so it's essentially doing something like
a nearest neighbors analysis and it's
not it's not capturing the global trend
so it's not actually discovering the
underlying pattern or regularity which
is that that you know parabolic function
it's just doing this brute force
interpolation through point data points
and because that's all it's doing
there's really no hope for
generalization
however this is a very flexible
algorithm so if if say this data this
data was to suddenly change its behavior
and maybe these points kind of started
going down here it could fit that so
it's very flexible but it it can't I
can't generalize or extrapolate and I
believe this kind of direct fitting is
inherently hard to understand and
interpret because it's not capturing
general rules or underlying trends and I
think this this is actually showing that
a lot of these narratives that people
have about deep neural networks
extracting high-level features are not
really really correct I think what these
stupid neural networks are doing is more
just a brute-force kind of interpolation
in the height and there's some other
reasons why would why we believe that
deep neural networks are going to be
very hard to interpret in terms of a few
like crisp rules or abstractions it's
been noted that it's really hard to
compress deep neural networks so people
try to like reduce the number of
parameters using very sophisticated
compression algorithms and such and they
really haven't been able to compress
things down that that much you still
have tens of thousands of parameters and
then the other general principle is that
if the if the world is very complicated
these networks are also going to be
complicated and this will continue to be
more and more true as we go to more and
more advanced and capable systems so I
think this is very relevant for AI
safety because people are starting to
talk more about using in
ability or explained ability techniques
to to open up the black box Chris Ola is
very influential in this area and he has
talked about this concept of microscope
AI for AI safety which is where you
instead of training an agent you train a
model to do some predictive tasks but
then you by using explained ability
techniques you can learn really useful
information from the model which then
can solve whatever you can use to solve
whatever kind of problem you wanted to
solve with the agent so you can avoid
the dangers of agent a gentle AGI just
by this microscope a API I know I'm
running out of times I'm going to try to
firmly try to move fast here but
basically I'm not very impressed with
Chris ollas work he's done deep dream
which has got a huge amount of press
attention but I I don't think it's
properly capturing the neural nets if
you look at this a paper on historial
examples they show that activation
maximization is very misleading and and
deep dream and most of the laws other
techniques are based on activation
maximization but where I do agree with
him is he says we need auditing and
testing so we need to you know we did
like actually test if if explained
ability techniques are allow people to
make predictions and of course Ola is
sort of very optimistic about explain
ability because he believes that as
models get more and more complicated
they're actually going to start
developing these Chris what he calls
crisp abstractions right now I don't see
it going that way I see it going the
other direction whereas these things are
getting more and more complicated with
billions and billions of parameters and
they're becoming harder and harder to
interpret so this is where I
I bring in a self-explaining AI I'm not
gonna go through this but this is an
example for my paper of a system in
medical imaging where they try to add
this explanation branch to explain lung
nodule diagnosis and essentially I think
that this explanation branch was not
actually necessarily capturing why the
the underlying reasons for the the
diagnosis so I proposed using this
mutual information which metric which
might be I think is useful in this
particular case but I I'm not sure if
it's really gonna generalize to to more
advanced or different cases and then the
other thing I talked about in my paper
is is the importance of uncertainty
quantification which currently is not
done very apt very much and then this
applicability domain analysis or
checking for if your input lies outside
the training distribution so this is
actually fairly easy to implement you
basically you just have to quantify your
your training data distribution and
delineate the extent of the training
data and then if you have a a test data
that is lies outside the training data
you you send a a warning to the user
because in that case the neural net
probably isn't going to function very
well so this is fairly simple there's
it's just not widely used and I think it
should be so I think at the end the main
conclusions here are that the current
explaining building methods are very
they're often very misleading and they
have a lot of problems given dealt with
the phenomena of double descent and this
direct fitting I'm I'm not very
optimistic about being able to explain
neural nets in terms of like a few
high-level rules or some simple
explanations so I think adding the
self explanation branch can help
increase trust in AI and then the other
important components I think are the
uncertainty quantification and the
applicability domain analysis all right
lovely thanks for the concluding slide I
think we may we may go back there on the
panel that was fantastic and thank you
so much for easing so many examples as
well so now we're gonna do the panel if
I couldn't may ask again and area onto
the stage and together with Robert
courage and Robert elshire did with my
robot you hear in the chat but thank you
so so much for being able to join check
out your website and I'm incredibly
excited for it and you can invite
preliminary comments that you may have
on the speakers and so that you know
anyone else maybe has a little bit more
context as to what we could be
discussing and discussing in the Q&A
also which of malla will afterwards make
a few comments here and so Robert and
Richard please your feature just like
you know very organically a mute
yourselves and perhaps we start with you
Robert welcome yeah hi thanks yeah I
really enjoyed both the talks first
comment I guess I'll write down a few
things so oh yeah on areas talk I would
I would agree with the the overarching
thing because I would I guess identifies
AI safety and I also haven't this heard
assured on follow me so I can as a data
point provide evidence for the fact but
this has been not so much like
communication between these fields and
I'm now interested to look a bit more
the references especially there's longer
slides and see more about like baton
than read more about it
yeah I think that overview was good I
don't think there's much to add there I
guess one thing often when when I hear
people talk about specification problem
and this example with the boat going
around in circles the classic
counterexample I hear people say is well
I guess you should have just specified
your award function correctly like you
just that was your fault really
not a problem with AI safety is just
ichika so to get a better reward
function you know this is an argument
which I think comes up quite a lot and I
think the the the counter-argument to
this counter argument which i think is
no you were saying is that it's a is
actually really hard to know what the
reward function should be but like a
general purpose AI we like can't really
write down on values you know like
imagine we had some superhuman AI like
what would you actually tell it to do
but you can like put in a computer at
all I basically ever bought any attempt
to directly thinking of something like
this like you'd probably tell someone to
type in as latest and what could be like
oh here's a hole but the AI will exploit
and turns out it's really bad and you
know so far no one has come up with like
a direct specification which no one else
has found a hole within a few you know
minutes or days or weeks or something
and that's kind of like one of the big
problems in AI safety is working out how
to almost like learn the specification
of our values from humans but you know
this is also really hard and humans
don't really know what about these are
anyway there's a bunch of other problems
around that I won't dig into it yeah
Robert I would just just mention that
the exact same thing that you just said
applies not just to super human AI but
also to a near term or midterm AI that
it's nearly impossible to specify the
the exact thing that you want that will
cover all possible situations that you
haven't thought of yeah of course the
more constrained the system is and the
simpler it is and the less general it is
the easier it is to specify manually
yeah yeah of course yeah like if you
have an image classification system and
you just wanted to classify images and
you can just give a bunch of labels
later and like it correctly classifies
images and that's great but as soon as
you know
it definitely gets added the more
complex the system is for sure
yeah then Dan's talk I agree with the
general pessimism about sailing see maps
and stuff like this I think I I've met
war more papers showing % amounts don't
really work when I've read papers about
sailing sea baths but you know it's kind
of becoming a little subfield in and on
itself and I think that it kind of
points to a more general problem which
I'm quite interested in it in that like
how do we really like know what an
interpretive bility method is doing
anything for us or not
like how do we validate whether it's
actually giving us useful information
and plausibly the same kind of problem
applies that the self explaining thing
in in that like if we don't have a kind
of an optimization signal to say this is
a correct explanation this wasn't the
correct explanation office is a correct
or incorrect interpretation it's kind of
like quite hard to validate where those
interpretations are actually insightful
or not under these paper towards a
rigorous science for comfortable machine
learning which is from 2017 I'll type to
post the link what I remember too which
I think the tackles this problem quite a
bit and you know like tries to develop
or think about some ways of validating
interpretability methods because most
people develop them for a while and then
as the previous came out seems like oh
maybe yeah a good we didn't Rangers it's
taken a while before there's been actual
testing of whether these methods
actually are providing information that
allows people to actually predict how
the model is going to function yeah
there's one paper which I since I was
moving a bit slower there I glossed over
it but there's one paper where they test
five different explained ability methods
and they have people try to they look at
a bunch of explanations and then try to
predict how the models going to phone
will behave and what they found was for
most of them it they they couldn't
any better than random chance so there
was really not providing any useful
information the only exception was this
technique called this looks like that
which is a little bit different because
it's a network that is designed
specifically to be interpretable and it
ref and so that that is an approach is
to design networks and techniques
specifically with interpretability in
mind and people like cynthia Rudin are
very optimistic about that but I am NOT
very optimistic especially I think it
can work for say some cases like tabular
data there's a lot of evidence that with
tabular data often you can fit a linear
model that that is or some kind of
sparse model that is that is just as
accurate as a deep neural net but for
real-world image data the general trend
is moving towards bigger and bigger deep
neural nets and I think the same is true
for like natural language processing and
just moving towards bigger and bigger
neural nets and so it's very well it's
possible there could be some kind of
interpretable model that is just as you
know performant I'm not seeing that
trend and I don't I don't think there's
any good reason to believe that so you
know and yeah I mean definitely not very
optimistic in that regard you know and
yeah there needs to be more it needs to
be more scientific more more actual like
putting these things to the test yeah
yeah I I would agree with that yeah I
guess on on chrysalis stuff which he
mentions I agree some of the feature
visualization stuff maybe isn't
producing as much insight but I mean I
quite like some of his stuff I think
there's an example with this activation
Atlas where they used it to produce like
handmade adversarial examples just from
like looking at this visualization what
was like an inch
being example of of the method giving it
insight I think it's it's a problem with
most of the methods is they don't ever
really validate but they give insight in
like a meaningful way because it's kind
of hard to do that can't have like a
test set for like insight generation or
something or at least there hasn't yet
been been one like that and the more
recent stuff that Chris Nolan opening
eye team are doing is on this thing
called circuits which I find quite
interesting is trying to develop it's
like very low-level mechanistic
understanding of neural networks I think
but yeah yeah it's interesting but it's
very low level
I've been Chris Ellis general position
is that like so understand these
networks we are just going to have to go
very low level to get like a mechanistic
understanding you know I've seen
something some quotes he said it's like
yeah I expect us to be able to have a
mechanistic understanding but it's gonna
be something like generating some code
which the size of like the Linux kernel
or something like this so it's not like
a couple of lines or credible sentences
it's like no a huge computer program but
it's at least more readable than a huge
you know hundred seventy five billion
parameters I haven't going to look at
that circuits post but that yeah I'll
definitely check it out yeah please
share the links here did you have a few
comments that you want to make I think
specifically on a desk I'm sure so just
wanted to follow up a little on what
rreow mentioned of the consortium on the
landscape of AI safety so this is so the
broad idea here is to bring together
lots of different fields that actually
comprise and surround this community of
interest is that is a s safety often
times there are different terminologies
in these different fields or very
similar concepts like the
principal-agent issues are economics
that translate to some inner alignment
type of shoes are lots and lots of
from such such things and so really what
we're talking about is a dynamic graph
knowledge graph that will be able to
generate lots of different kinds of
outputs guidebooks and checklists and
diagrams depending on what people are
working on what they find sailing into
the projects they're creating so we are
organizing organizations to come
together to contribute to this as well
as to express interest in consuming the
outputs from these types of things it's
also served to ground various types of
governance concepts in the specific
technical ideas that underpin them or
they're sailing into them thank you yeah
we will be launching a new website clay
dot org within a week of you okay lovely
if you could share a link that would be
amazing then I can share it with the
group and for now I know that we're now
exactly at noon dad and Ari can you give
me a quick thumbs up thumbs down whether
you can stay on a little longer or yes
yes okay great lovely then the first
participant question that we are going
to take is by generics
and I'm meeting you now thank you
this is janati Stolyarov ii I enjoyed
the presentations today my question
pertains to the remark in Dan Elton's
presentation regarding how the human
brain itself may be over parameterised
and the curiosity that I have is what
implications are there if any of this
with regard to the validity of how
certain people make decisions so could
it be that decisions made based on
simple heuristics by lay persons are
actually in that leftmost region of the
curve that's a bit before the descent
and then the decisions made by many
experts who take into account more
information
but perhaps take into account a lot of
extraneous information might actually be
less optimal if they're over
parametrized and to overcome that
problem we might need to develop AI that
can take into account millions of
parameters and perhaps get that error
down further but in the meantime our lay
person heuristics just as valid as
expert advice in some cases
well let me when I was talking about
there was that the brain has had like
100 billion neurons so the conventional
wisdom which you hear from people like
Joffrey intan who's a famous you know
guy in deep learning is that there's
there's just not enough data coming in
to learn all those parameters so a lot
of that means to is probably in the the
structure and weights in the neural net
probably innate to some extent however
what Hassan is saying is that actually
he he ran some numbers and he's kind of
he believes that there there is enough
data streaming in to to allow for this
type of parameterization where where
it's you're basically doing its
interpolation and he just a lot of a lot
of the conventional thinking is that for
things like but but he he also believes
that there's their symbol manipulation
going on so you have to kind of
differentiate the the perceptual part
system in the brain from like the higher
court order reason
a system which is more symbol based and
the the the people use both and they
both have problems so it's not
however expert expert intuition is a
real thing
and often I think Daniel Kahneman showed
that the expert in certain specific
areas not generally but in certain
specific areas experts can perform much
better than simple heuristics and that's
that could be because like you said
they're they're kind of using lots of
different information from experience
the other thing is that I don't think
when people give like a verbal
explanation of something it doesn't
necessarily Maps on to actually what was
going on in their brain it's kind of a
post hoc justification and but that is
actually okay because just the fact that
they're able to give some explanation is
helps us trust them now just I believe
it is helpful even though it it may not
be mapping onto what's actually going on
and there's also always the problem of
deception I still think it's you know
this giving explanations in ways we
understand is it's gonna be useful but
there's there's definitely pitfalls that
the paper I wrote is very a very initial
work and it's still working progress how
does that square and with the super
forecasting and work it expose are
actually not that great and putting
thing or I mean it depends on a domain
so in like highly regular domains like
say medicine looking at CT scans a
radiologist who has 30 years of
experience is gonna be able to do really
well but in highly irregular
non-stationary you know complex arid
domains experts don't really do better
so it's specific yeah yeah
lucky okay we have Mike up next to the
question Robert an area whenever you
want to chime in with an answer to just
go for it as well Mike here we go hi
great talks very interesting and a
couple of quick questions one was on the
second descent does that eventually tend
to our limit of zero or is there always
gonna be some constant minimum error is
is it possible they do enough data to
reach that point where you can decide
for C which way it's gonna go
[Applause]
and generally and adding more data it
always helps so in a limit of infinite
data the test error would go to zero if
you're within the distribution of the
data yeah well if you know you're doing
that interpolation it yeah if it was
some noise but the description of the
data doesn't catch up and plausibly is
your test error could never go to zero
yeah if you're trying to predict you
know just input equals output plus of
Gaussian noise if you don't have access
to the Gaussian noise you can never
predict what the noise is
so you just it's impossible to guess the
rest all right yeah well I guess it will
tend to the theoretical minimum you know
like what yeah in a normal like regime
of limited data it does it does Plateau
as you increase the number of parameters
it what toes the test error doesn't keep
it plateaus at some point I understand
the other question had was the form of
the curve of the direct fit is that just
like a string of vectors or as a like a
Fourier fit with a lot of terms in it so
it looks like vectors or some other
curve it's actually a smooth so it's
this direct fitting is the precise form
of the function depends
the model you're using for a deep neural
that's you know it's a very complex
function which is kind of like piecewise
linear function and so the the exact
mathematical detail really depends on
what the model is what the general you
know appearance of it they are saying is
kind of like this what was shown in the
figure now it's not always gonna go
directly through the points like they
show there because like if there's noise
or so you can have in data you you you
but it's gonna it's the point of you can
like you can have on two points with the
same X value but different Y values that
that's engine noise or something so it's
never gonna go through all the points
exactly but it it's going to just but
it's gonna be interpolating them as
closely as possible where I had like
discontinuities at the points or just
like a very large transition he means in
the training data you could have points
of the same x value or different Y
values if there's just noise in the
measurement or your training data
doesn't capture all of the information
okay but then know like might be things
where you measure exactly the same
quantities but because it's the noise
but the output that the measurement is
different and then obviously you know
your light can't go through both of them
because it's got to be a function okay
all right any other questions anything
to are you
all right Mike we got all of your
questions in oh yeah
sure dude thanks okay
next up we have Christian hi yeah this
was a question for Dan so you mentioned
that the human brain has like dozens of
billions of neurons right and it was
most likely not happening is that we're
kind of over fading or we're using this
kind of brute force machine learning
technique where we just add enough
parameters and eventually double descent
takes over and you're you're all good to
go you mentioned that it might be the
case instead that we have these kinds of
innate may be produced via evolution
weights structures certain neural
architectures what have you is there any
work that's being done in that area and
because at least historically this kind
of work was done traditionally in
philosophies like in consecrate eeeek
the first one he tried is talking about
like what concepts are are built into
tomorrow you know by default so yeah I'm
sure there's there is definitely
evidence that certain things are innate
there's actually very compelling
evidence that detecting faces is made
because they put babies into like these
Emperor my skin fMRI scanners and that
are just came out of the womb so that
they haven't able to learn anything but
they parts of their brain there seem to
be parts of brain that are sensitive to
faces which is really amazing so there's
a lot of neuroscience work showing
there's and made things
there's obviously innate fear responses
there they're tons of things but Madrid
escape I his son because he is argue
that the brain is doing this kind of
brute force interpolation and I found it
very convincing so if you're interested
that I work that does to balance that
out do you know of any work that are
used for certain kinds of innate
architectures or concepts I can I can
think of some specific very specific
things but there and then there's the
general argument that I gave which comes
from Joffrey Hinton but the but I think
the double descent really so the
conventional thinking was you can't
train a model with so many parameters
with very little without having more
date like basically there's this idea in
that you see in some machine learning
textbooks that you should have that for
high dimensions for high dimensional
inputs you need tons of data there
there's like this general idea that you
need the more MA the more parameters you
have or the more dimensions you have the
more data you need but double descent
that double descent phenomenon kind of
shows that's not the case somehow these
models can still learn that
interpolation
even though classically they would be
over fit so and that's just feeding in
this an illusion sure if I'm making much
sense but I would recommend looking at
that paper because and he goes into why
he thinks some neuroscience experiments
they're kind of they're looking for
these high-level abstractions by
you know doing these careful experiments
to see if the like per second perception
to see like how if the brain is using
certain rules for perception but he says
it's not it's not using these high-level
rules it's using this brute force
technique which is calls into question a
lot of these neuroscience experiments
paradigms so it's easy yeah yeah another
people just send around thank you
all right dick I and you had your hand
up for a while here you go
[Music]
you're still muted
yeah we're handling with a mute button
at the moment do you don't is it working
yeah to see oh so so yeah so this is the
Chi for the request to say the name
there's more of some comments than some
than questions immediately appreciated
the discussion very much Dan I think
that the cautionary note that I would
put into this from you know having that
then an you know in AI research for 35
years is that we keep on seeing the same
mistake made that we take whichever is
the flavor of the month or the year for
in AI and then like applying that to
human brains as if that was an accurate
model of human brains you know we did it
with good old-fashioned AI we did it
with the first wave of neural nets after
that and machine learning and people are
doing it too much with deep learning
today
you know geoff hinton would be and and i
postdoc at toronto spending the year
playing in his lab way back in what 1992
it would be the first
to admit that our current deep learning
models are not the answer yet you know
where we're heading off in the right
direction but there are a lot of issues
with it one of which is still it's not
clear that back profits the right
learning mechanism at all if we want to
do the brain and and also you know I
think a fundamental issue that I've been
arguing to the AI field for a very long
time now
is that this trend toward just
everything being big data is moving us
away from AI real AI small data not big
data humans learned from kids learn
their their their mother-tongue and all
sorts of things about the environment
with very small about the data compared
to our current deep learning system just
and it's not so it's not just about the
you know the double descent and things
and so forth them this is all accurate
as far as deep learning and some other
machine learning problems go the the
problem is that current deep learning
models are incredible gas guzzlers when
it comes to data the amount of data that
it requires to train them is
exponentially large compared to what
humans need to learn and humans manage
to generalize off of that exponentially
smaller amount of data and we do not
have the right answers yet you know it's
only a very small percentage of the
field of today unfortunately still
they're actually trying to attack that
problem because everybody's off on the
deep learning then wagon and as I think
one of you may be with you Deborah
pointed out natural language rustling
which is the you know of course my
longtime field of expertise has gone
totally off the deep end with that just
throwing more and more data at that this
is not giving us any scientific progress
to be honest and so I would take even a
you know I think it's really I really
appreciate the step back that you're
taking down and I think that taking even
a further step back and looking
at the fact that we are climbing the
rock trees right now is is super
important so I just appreciated these
talks very much I also appreciate re s
talks and I really want copies of both
of your slide sets would you be willing
to share them with us Thank You Alyson
yeah I know I agree 100% that again like
it's dude these models are basically
doing brute force kind of data
interpolation they're not learning
generalizable rules it's kind of the
main point and there is work on view
shot-- learning meta learning but if you
look at the results that people are
getting they're very dismal right now so
I don't think yeah I mean it's nice that
they're working on those problems but
it's I don't think they're taking the
right approach yet we do you know I
think Jeff actually was quoted saying we
have to tear it all down again and and
in many ways I think we do so what we
should we be climbing instead so in my
view we should be climbing a true low
resource tree that is to say not just we
have low amounts of training data
resources come you know consistent with
what humans are able to generalize from
but also low amounts of computation
resources so there's a lot of I mean
yeah the brain has a lot of neurons but
it's not clear how much recursion is
going on it's not clear what the
relationship of the different modules in
the brain is I don't think we're doing
enough with attention even now in the
deep learning modeling approach we're
certainly not doing enough in terms of
understanding the relationship between
the unconscious automatic processing
resists the control reasoning so there's
there's a don't get ready we don't have
anything like system 1 and system 2 a
lot of people
all we have is system one type well so
what models they can do things like
system one task yeah I guess I guess the
counter-argument just to play devil's
advocate or something is that like like
do we need to reproduce the brain do we
surely we just care about producing
powerful well you know you might say if
we just care about the end results or
like the ability about models then like
I guess obviously the brain is on one
example of general purpose intelligence
but doesn't mean it's the only one and
while like one human learns from limited
data from a meta learning approach you
could argue you could have you know
evolution has has created the brain with
a lot more data than just one humans
lifetime so like there are still
opportunities to use big data and if
like you can use big data to learn stuff
which you can been using small theta
scenarios I think that's perfectly valid
I think if you can appeal to the fact
that we should follow closely to the
brain because it's the only way we have
at the moment and obviously this huge
data model approach seems to be not very
close to how the brain is doing it maybe
and so like it's probably going to fail
at some point but maybe we should just
keep trying until it fails and then move
on jumping ship Trevor Shirley another
cautionary note on that if I could one
last cautionary note on that that
argument we don't have to do it the way
the brain does it is an argument that
the field of AI has been using since
before even I started doing doing AI 35
years ago and the problem the irony of
that was that back then people were
using models of system to logic based
rule-based explicit knowledge based
systems and assuming that if we could
build a machine that did that well then
because that was hard for humans that we
could apply it to all of the other tasks
very easily like natural language
understanding that we seem to do
effortlessly and of course that's
totally wrong because you need system
one type model which is why for a lot of
natural language understanding which is
which is why it succeeded only after we
started actually applying machine
learning and neural net approaches to it
and so we've gone now from a world where
we used to have AI dominated by system
to approach modeling approaches being
miss applied to tackle system one tasks
to today where we are off the opposite
depend trying to use system one models
to and assuming that they're gonna solve
all the system two problems both of
these extremes are insane all right
earlier any questions and I was just I'm
look this is not my area of expertise
but I can I was just reminded of a paper
from somebody who I'm trying to find on
the side so I will put it in the
comments if I can find it a researcher
at uber who had a paper on if we can
model if we could just get a GI through
essentially group forcing it like
evolution
I'll find the paper and post it is it
Jeff clue yeah they are generating
algorithms yeah if you can find that in
okay and the counter grant from this
group okay great then gennadi please
feel free to meet yourself with perhaps
to find a question let's see if we break
their 30 30 at past mark well thank you
this is janati stole ear off the second
again this question is for aria inspired
by the very interesting example of
artificial stupidity with the system
exploiting the computer game by going
around in circles and destroying things
instead of engaging in the race one
would think that a human player faced
with that kind of AI adversary would
clear
recognize this is above this is a glitch
and perhaps knock that boat out of that
loop so to speak
and kind of derail the ai's objectives I
wonder if that can be generalized into
the superior ability of human
common-sense to outwit or outmaneuver
the AI systems that lapse into these
kinds of suboptimal
behaviors that the AI might consider to
optimize a certain outcome but are
really counterproductive going back to
the stereotypical paperclip Maximizer
scenario so let's say you design an AI
to efficiently manufacture paperclips
but you see that it but wouldn't a human
just do common sense see this and say
too many paperclips are being made at
the expense of other worthwhile
objectives we need to turn this thing
off now and wouldn't that be in many
cases of sufficient check for this kind
of artificial stupidity oh I'm sure the
other panelists can have things to say
about this question was directed to me
though so this is the partly the
question of control and it's partly the
question of anticipate anticipating
things in advance so if we can if we see
the problem and we have enough time to
respond before it leads to the real
problem a safety issue then sure we can
always and sorry and if we have the
ability to respond effectively then we
can respond the problem is that
sometimes the first time is already bad
like if that was a imagine that was a
ship a imagine that was a
we an autonomous ship in the Navy that
learned this behavior but it's buried
somewhere deep in its reinforcement
learning model and it only comes up once
in some weird scenario and then it goes
and blows up a bunch of ships by going
around in a circle right the first time
is already catastrophic so yes if you
can see it in advance great this
actually gets into what Dan was talking
about because explain ability would be
extremely helpful here right if we knew
if we could interpret what the AI model
was ahead of time that was great but we
can't so that's one issue another
related issue is the control problem
which is I believe we have a Roman young
Polsky I don't know if he's still here
but he's written on the control problem
and how that is it's not gonna work for
super human AI and then yeah I said
there were three issues and blinking on
on one of them but I'm sure everybody
else yeah so for the anticipation could
you have some sort of test environment
like if this were a real ship you would
run it in a simulation first and you
would run it enough times to ferret out
any undesirable behavior and then you
launch it in the real world once you're
confident that either there is no such
behavior or you can intervene if it
manifests itself yes so that is in fact
the entire perspective of testing
evaluation and verification and
validation that is the way it's been
traditionally done so when you have
systems in constrained environments
that's great with sorry constrained
environments with a very constrained
action set that that works fine it works
at least satisfactorily some sometimes
there are it doesn't work but it
we we can guarantee that with very high
confidence that won't do things we don't
want it to do given given the situations
we put it into the problem becomes with
advanced AI machine learning that with
the complexity of the environments we're
putting in it into and with the
complexity of the models that the AI is
learning and with the complexity of the
action set that we're allowing the AI to
use it becomes intractable to just
brute-force test every scenario there
are I think there was a there was a
paper by Rand that calculated that if we
wanted to use standard simulation
approaches testing simulation approaches
for sorry standard regular testing
approaches without stimulation for
self-driving cars it would we would need
to drive billions of miles with
self-driving cars in order to get the
same sort of safety guarantees that we
have for regular cars with simulation
that could help to a certain extent but
it's still nobody's nobody's figured out
how to do it in a way that will be that
will work essentially for for for
advanced for sufficiently advanced AI
systems but I mean that it's a great
idea in theory and we would love to be
able to do it and there's ways to like
there are there's a lot of work being
put into trying to figure out how to do
it but it has not been solved yeah
Robert yeah basically what he said you
know like it's hard we would do all of
those things but we'd still have like
the core of the problem still exists
this is kind of like patching the hole a
bit and I guess some example is some an
extra counter argument it is that if if
we have a human who's ready to intervene
the entire time and why why isn't this
even just doing it like why we this
thing is autonomous for a reason or you
know maybe the human is better
intervening for doing the behavior but
still I don't know that's that's done
yeah yeah I concur it's not that
convincing but you know we want this
thing to all be autonomous and and with
some kinds of capabilities like
oversight might just not be possible if
it needs to act quickly or or its
domains but we can't really understand
it really depends what we want it to do
and if it's if it's if an AI is making
decisions we don't understand you just
stop it
doing it even if the decisions in the
correct ones that I mean we don't really
know I do doesn't that's a very very
topical question I would just add in one
factor there I saw this come up
somewhere in the chat and why don't we
just it's related to why don't we just
not develop API so in this case it's
like why don't we just not use those
systems that are that we can't test and
the answer is that would be great except
that the economic pressure to build
those systems anyway is probably going
to end up with somebody building yet but
morale moreover so actually in fact that
it has been the the approach that has
been used right cars get regulated but
yeah there's a there's a performance
safety performance trade-off where if
you add too much safety you don't get
anything useful and if you add
performance without safety then you're
maybe someone dies all right yeah about
everybody dying
these are someone nice that's good I'll
leave I'll leave the further
implications for other people to
extrapolate well I think one to be noted
okay perhaps Robert area and Dan you
wanna close up with like any final
comments and either is if there's a way
in which participants can follow up with
you work beyond just the slice and I'm
sharing or something that you have to
take the community love to see the
community take on any research questions
or or anything and if none of those then
feel free to go with the last question
that would ask was aspect in which open
question to all speakers give the
illusion success in producing general
intelligence is using evolutionary
algorithms the most promising path and
to Adi oh my that's that's a yeah okay
I'm gonna pass on that I will say that
that if anybody wants to contact me
Allison are you able to give out my
email that maybe contact Allison and you
can I can I am involved with a bunch of
practical near term and midterm AI
research at APL I can point you to
people to get in contact with I'm also
happened to be working on a I for a
pretty large international AI
forecasting project that I'm running so
if you're interested in that you can
that's for long term AI you can contact
me I suppose yeah I guess they most my
thinking is be very critical I don't I'm
starting a PhD in September but yeah
there's no I guess you can go on my
website if you want but I guess in
general resources I think are quite good
at the is less wrong all the AI
alignment forum I think that's where a
lot of the may be the kind of AI safety
research but people have encountered
last
is like posted about and talked about
these more like longer-term or
theoretical issues that's where a lot of
our stuff is discussed so I can post
links to that yeah well as far as the
question about whether evolutionary
algorithms are useful i I think that's
kind of what we've been doing the past
20-30 years is basically just trial and
error and seeing what works and what
doesn't work and we don't have a lot
there's no like there's no equivalent to
like Newton's second law or something
there's no overarching theory for
machine learning there are some
formalizations that like this VC theory
and but there's no like overarching
theory for how to design a machine
learning model it's it's really been
mostly trial and error and you know
something there's this term graduate
student dissent which is where you have
a different graduate students try
different architectures and eventually
they they converge on than the best
architecture for whatever problem
they're solving but there's no like it's
really just all trial and error and the
longer images feel the more I'm
realizing that people just don't really
understand things that and it's just
trial and error so that now so yeah it's
very much like evolution now what now
genetic algorithms in particular have
kind of fallen out of favor right now
but I think as more compute becomes
available they will become more
tractable and I and we see Google has
used genetic algorithms to design neural
nets so it's I think it become it will
become more and more I think there's a
likelihood that that will be the way we
eventually get to AGI and or very
advanced AI and if that's the case we're
not gonna understand it cuz it was the
output of this of his trial and error
process there was no there was no like
underlying principle reasoning behind it
but uh
see yeah I definitely think that that's
a that's a really big approach and now I
found over us just just to wrap up I
think this is a this is a great
discussion I think the three of us are
coming at this trying to grapple with
how how it these these near-term things
can tie into the long term a nice safety
work and we're all coming at it from
different kind of angles so like I'm
working in medical imaging and I'm sort
of very mired and you know these
specific issues that we're having right
now and and just seeing if there's some
way I could connect it to the
longer-term stuff and then we have you
know aria is working on they're actually
very different types of systems which
are more a gentle and real world out in
the world
not necessarily though not even not
necessarily i dental but but I focus in
here and you're working on like thinking
about the long more long term stuff
right so I just think it that we need to
have more conversations like this which
are tying together the short term and
long term people the whole point of what
I may be grounding this the long term
people a bit more and vice versa
that is exactly what Richard Malwa is
doing a fantastic job on with his other
collaborators
he helped organize those workshops I
mentioned so those were very good at
bringing together a lot of people from a
lot of different fields it was like
pretty amazing to see just government
and industry and academia and people on
the alignment forum just like nonprofits
just effective altruism like all in one
room talking to each other I thought
that was amazing so thanks Richard that
was sure yeah thank you are yeah and I'd
like to note that yeah what we're doing
with clay is really to try to bring
those
disparate parties together to
collaborate on like artifacts that that
many more people can Sharon and
understand all those different linkages
and the progression and how different
long-term concerns are exhibited a
little bit in near-term systems and vice
versa all right lovely and I think
Richard you mentioned that website is
not up yet but will be soon and then
I'll share it in the group whenever
whenever you have it you can also watch
a are their safety org for links to that
new site I mentioned for Richard that he
kept talking about clay that has an
asset yet CLA is great lovely I'm sure
it will become much more clear when I
should follow up with the links okay so
for now thank you so so much for
everyone at for coming on and I think
you know almost one conclusion after
every other session was like we need
much more discussion and I think you
brought it in really nicely with that
area you know with your presentation you
could tie in the long term which hotel
but even that it's just two particular
fields within the long term and short
and even within AI that you know should
be connecting more so I think you know
I'm hoping that there we must more much
more and like yeah first pollination and
I can't thank you all enough for joining
I thought presentations will really be
stunning I have rarely ever seen so many
folks here asking for slides right away
so and please please do me a favor and
send me any follow-up follow-up link so
I can I can still participant and
offering it for it as soon as possible
thank you so much Robert are we and then
thanks Richard for hopping on and thank
Sakai for for your rides and and I'm
hoping that I'll see most of you again
next Thursday and we're gonna publish
next topic very very soon and I think it
should be of interesting investor buddy
and folks down the call
all right well with that being said
thank you for staying on for so much
longer and I hope you have a lovely day
and I'll be in touch by email