Hassabis, Altman and AGI Labs Unite - AI Extinction Risk Statement [ft. Sutskever, Hinton + Voyager]

just a few hours ago a host of AI
industry leaders experts and academics
put out this 22-word statement on making
AI safety a global priority the
so-called statement on AI risk brought
together for the first time all the
current AGI lab leaders that's people
like Sam Altman Ilya satskova Demus
hasabis and Dario Amadeus and two of the
three founders of deep learning itself
Joshua bengio and Jeffrey Hinton here is
what the statement said mitigating the
risk of Extinction from AI should be a
global priority alongside other societal
level risks such as pandemics and
nuclear war it is now almost impossible
to deny that this is now the consensus
view among AI experts let's first look
at the Preamble then break down the
statement and show you the signatories
they say that AI experts journalists
policy makers and the public are
increasingly discussing a broad spectrum
of important and Urgent risks from AI
even so it can be difficult to voice
concerns about some of the advanced ai's
most severe risks the succinct statement
below aims to overcome this obstacle and
open up discussion it is also meant to
create common knowledge of the growing
number of experts and public figures who
also take some of advanced ai's most
severe risks seriously the first point
is that the statement is in a way
optimistic it says we can mitigate this
risk perhaps not eliminate it but
mitigate it reduce the risk second it
says we should do this globally and
that's not just among all the different
AGI Labs almost all of which sign these
statement but also between countries in
that vein there were quite a few
prominent signatories from China and the
third point that I'd make is that they
put it on a par with pandemics and
nuclear war toward the end of the video
I'll show you that's not as far-fetched
as it sounds but anyway who actually
signed this statement let's find out we
have two of the three founders of deep
learning that's Jeffrey Hinton and
Joshua bengio the third founder was
yanlichun and we'll touch on him later
in the video all three of those won the
most prestigious Accolade in computer
science which is the Turing award then
we have three of the CEOs of the top AGI
Labs Sam Altman Demis hasabis and Dario
amade of openai Google deepmind and
anthropic none of those signed the pause
letter but they did sign this statement
and actually as interestingly for me so
did Ilya satsukova who I see as the
brains behind open AI he of course also
worked with Jeffrey Hinton on deep
learning and is widely regarded as the
smartest guy in machine learning you
will also notice so many Chinese
signatories especially from xinhua
University which I've actually visited
in China it's one of their leading
universities that's a really encouraging
sign of cooperation between the west and
countries like China on AI and the list
of significant signatories goes on and
on and on these are senior people at the
top of Deep Mind anthropic and open Ai
and there are names like Stuart Russell
who wrote The Textbook on AI who also
signed the pause letter let me highlight
a few more names for you here you have
the CTO of Microsoft itself Kevin Scott
he's the guy who basically heads up the
partnership between openai and Microsoft
I think many people will miss his name
but I think it's particularly
significant that he also signed this
notice also the CEO of stability AI emad
mostacc the center for AI safety
coordinated this effort and I'll get on
to their eight examples of AI risk in a
moment but first let's pick out a few
more names you've got David Chalmers
Daniel Dennett Lex Friedman and Victoria
krakovna now together with the statement
the center for AI safety also puts out
this eight examples of AI risk I've read
almost every paper linked to in these
eight examples so I'm going to try to
summarize them fairly briefly because I
know not everyone will will be that
interested it starts by saying that AI
could be profoundly beneficial but also
present serious risks due to competitive
pressures before we get to the risks I
want to touch on some of the upsides
recently outlined by Demis hasabis and
these showcase what can happen if we get
this right we had sort of a golden
couple of years in some sense for AI for
science we've had lucky enough to have
many Nature and Science papers published
in all sorts of domains so from quantum
chemistry better DFT functions to
approximate Schrodinger's equation pure
mathematics we've solve some important
conjectures in topology with
collaborating with some brilliant
mathematicians who found working on
Fusion reactors with epfl on their test
Fusion reactor controlling the plasma in
real time in their Fusion reactors and
being able to hold the plasma safely in
place for for arbitrary amounts of time
being able to predict rainfall many
hours ahead and more accurately than
current meteorological models and then
in applications there's a ton of those
two we saved one of the any things we
did at Google was saving about 30 of the
cooling energy used to cool the massive
data centers at Google so there's a huge
energy saving and we're starting to
explore doing that across actual whole
power grids and this Echoes what Joshua
bengio said in a recent blog post which
is that we can build immensely useful AI
systems that are modeled after ideal
scientists and do not act autonomously
in the real world janlecon recently said
that we would never give current llms
agency there is a a flaw in current Auto
reversive lens so there is no persistent
memory first of all but second of all
you cannot control the system you cannot
impose constraints on it like be factual
be understandable by a 13 year old and
that makes them very difficult to to
control and steer and so that creates
some fears because people are kind of
extrapolating if we let those systems do
whatever we connect them to internet and
they can do whatever they want they're
going to do crazy things and stupid
things and perhaps dangerous things and
we're not going to be able to control
them and they're going to escape Troll
and they're going to become intelligent
just because they're bigger right and
that's nonsense first of all because
this is not the type of system that we
are going to give agency to that was a
week before this paper was published on
the results of giving agency to current
large language models the paper showed
that current llms with agency are able
to utilize the learn skill library in
Minecraft to solve novel tasks from
scratch zooming into the diagram you can
see how this Voyager model outperforms
reflection which I've talked about in
previous videos and auto GPT and it
discovers new items and skills
continually by self-driven exploration
significantly outperforming the bass
lines indeed Andre carpathy responded to
this study saying very clear that AGI
will Mega transform Society but still
will have but is it really reasoning how
do you define reasoning oh it's only
predicting the next token can machines
really think and he called that armchair
philosophy previously though even Yan
lacun has admitted some risks saying you
know it's like rockets you test it it
blows up you tweak it and then try again
I'm not sure I'm okay with an attempt at
AGI blowing up the first time but I'll
leave that up to you to decide so what
are these eight examples of AI risk that
the center for AI safety to organize the
statement list out they say that AI
systems are rapidly becoming more
capable they can power autonomous
weapons promote misinformation and
conduct cyber attacks as we've seen they
are increasingly able to act
autonomously now there is so much to say
here and I've read each of these but I
want to keep it to just the highlights
so let's move on to the first example
weaponization malicious actors could
repurpose AI to be highly destructive
presenting an existential risk in and of
itself and increasing the probability of
political destabilization they talk
about aerial combat and then building
chemical weapons which I mentioned in my
previous video on governing super
intelligence then they mentioned
developing AIC systems for automated
cyber attacks they mentioned military
leaders discussing giving AI systems
decisive control over nuclear silos I'm
going to quickly try to demonstrate why
that kind of autonomous AI might not be
such a good idea I want you to meet a
hero stanislav Petrov he was the duty
officer at the command center the OKO
nuclear early warning system when the
system reported that a missile had been
launched from the US followed by up to
five more Petrov judged the reports from
the system to be a false alarm and his
subsequent decision to disobey orders
against Soviet military protocol is
credited with having prevented an
erroneous retaliatory nuclear attack on
the U.S which it says could have
resulted in a large-scale nuclear war
which could have wiped out half the
population of these countries involved
an investigation later confirmed that
the Soviet satellite warning system the
machines behind it had indeed
malfunctioned I would not have wanted
that system to be or autonomous then we
hear that gpt4 was able to autonomously
conduct experiments and synthesize
chemicals in a real world lab again I
covered that paper at the time and then
linking back to Petrov they say an
accident with an automated retaliation
system could rapidly escalate and give
rise to a major war but that unlike
previous weapons AI systems with
dangerous capabilities could be easily
proliferated through digital means
hopefully you can start to see why we
need to balance risks with opportunities
but let's move on to misinformation I
think we can all agree that we already
have too much misinformation so let's
move on to the next one which is proxy
gaming this has already been showcased
in the social dilemma where AI
recommender systems are trained to
maximize watch time and click rate
metrics and this can lead people into
Echo chambers that helps develop extreme
beliefs in order to make those people
easier to predict by the AI recommender
systems so you might think it will be
simple just to tell the AI to promote
happiness or economic growth but that
might not work out as you intend next is
in feebleman if we delegate more and
more tasks to machines we become
increasingly dependent on them and here
they actually mention the film Wally
which if you remember the ending
features this quite comically imagine if
it becomes well known that companies led
by AI CEOs bring in more profit well
then it wouldn't take long for all
companies to be under immense pressure
to make their managers and CEOs Ai and I
know what many people will be thinking
couldn't that be an improvement on the
current system and while I know exactly
what you mean in the current world
realistically it would still be the
people owning the company that would
derive the profit and while the ultimate
answer may be some form of universal
basic income we do need some time to set
that up and the current accelerated AI
arms race doesn't give us much of that
time next is value lock-in which links
very much to the last point about giving
small groups of people a tremendous
amount of power in other words if you
want massive change to the way the world
Works giving current leaders AGI might
not be the best way of doing it they say
that such AI systems might enable
regimes to enforce narrow values through
pervasive surveillance and oppressive
censorship next is emergent goals this
is sometimes called misalignment and
we've already seen many AI agents
develop goals such as self-preservation
and you can see why even a system
designed to do good might have that goal
you can't do good and help the world if
you're shut down so it makes sense that
even the most benign AI might want to
preserve itself and to take actions
including through deception to make sure
that it's not shut off and this is not
just Theory the accompanying academic
paper natural selection favors AIS over
humans gave this example agents could
behave One Way during testing and
another way once they are released to
win the war game diplomacy which many of
you will have heard of players need to
negotiate J form alliances and become
skilled at Deception to win control of
the game's economic and Military
resources AI researchers have trained
metas AI agent Cicero an expert
manipulator to do the same in summary it
would cooperate with a human player then
change its plan and backstab them in
Future these abilities could be used
against humans in the real world again
that's not because they're malevolent or
hate humans it just makes sense it's
smart to do so this brings us neatly on
to deception and they give the great
example of Volkswagen who programmed
their engines to reduce emissions only
when being monitored and future AI
agents could similarly switch strategies
when being monitored and take steps to
obscure their deception from monitors
once deceptive AI systems are cleared by
their monitors or once such systems can
overpower them these systems could take
a treacherous turn and irreversibly
bypass human control I talked a bit more
about that point of when AI might become
deceptive in my previous video on
governing super intelligence it is a key
debate in the AI alignment Community
about whether models will become
deceptive before they become helpful for
alignment but finally we have power
seeking behavior and this example ends
on this dark note building power seeking
AI is also incentivized because
political leaders see the Strategic
advantage in having the most intelligent
most powerful AI systems for example
Vladimir Putin has said whoever becomes
the leader in AI will become the ruler
of the world so those were the eight
examples and yes I would have signed
this statement but I'm not a significant
figure so I can't anyway let me know in
the comments if you agree that this
should be a global priority and of
course you can also let me know if you
don't think it should be a global
priority my goal in this channel is to
cover both the risks and opportunities
so I'd love to hear from you whatever
your opinion is have a wonderful day


272. A Very Non Technical Explanation of the Basics of Infra Bayesianism

hello and welcome to session
272 in the AI safety.com reading group
tonight we will be discussing a very
non-technical explanation of the basics
of infropriationism by David matoxy
David Metallica studies mathematics at a
university in Budapest
and he has been he is an alumni from
this series Mets scholar program
this in particular is a less round post
that is around a month old
this was in the post was written in
response to a request by John Wentworth
for someone to write an explanation of
informationism in the language of Game
Theory and this didn't uh John Wentworth
provided a bounty of uh
1480 and he awarded 75 to this post it
didn't really explain patientism in like
Game Theory opportunity has a lot of
mathematics and there's no mathematics
in in this post but otherwise it is some
kind of explanation
uh the author uh wants that there is
substantial simplifications in this but
he has a version with more technical
details available
I am not entirely sure I would agree
that this is actually explaining the
basics of informationism in a in a
strict sense because well there is a lot
of talks about
um what is interpretation what
properties does it have and how does it
fit into the different things the actual
basics of uh of of interoperationism
isn't really explored in the same way
where if you talked about the the basics
of calculus or something like that you
would give some kind of explanation that
in theory would allow someone to
rederive
um uh calculus and you can I would
expect that the the level of details
here are not closely enough tied to the
mathematical foundations that you'd in
any way be able to read and Rave red
arrive uh informationism from this
I will start one step earlier than David
in this uh by giving an even more basic
explanation of how does infreparationism
fit together with AI safety so I will
start by giving a uh
um a naive alignment uh proposal and see
how that fails in a very uh
in a very contrived example and this is
something that is of course my
contribution and not something that
David matoxy has written so um buy a
beware or listen to be aware
after talking about the real problem
then we will talk with we will uh
discuss David matoxie's toy problem and
see how classical learning theory fails
to solve this and see how information is
in fact solves This Joy problem
um and then
um of course if you want to actually
solve the alignment problem then you
need to get it one step further and that
is a rather big task that isn't really
explored very much in in this document
so first let's try a really naive
alignment proposal and see how it feels
so we've how the AI can fail to
understand the values of the human
zooming out to the problem it seems
we'll soon have uh really really
Advanced Ai and this is something that
people used to call and I guess still
call The Singularity and the reason why
it's called The Singularity is because
you can't see past it you can't predict
anything past that and that is a real
problem we would like to predict things
like humans are not extinct and the
universe has not been turned into
something that has no intrinsic value
and like humanity is in control or
things like that
um so what has often been proposed is
some kind of mathematical proof of
alignment or at least a very strong
mathematical in intuition behind why a
given AI will be aligned
um
and this has generally been done under
the research program called Agent the
agent foundations agenda
one of the uh like my hot take on the
Asian foundations agenda is that the
real thing we have discovered by working
on this is that there is actually a huge
number of really thorny uh sub problems
uh that makes it actually really really
hard to um to say something about how
agents will behave in practice when they
become super intelligent
um
one of the problems is
that the agents are uh in fact the
agents who are modeling the world are in
fact embedded in the world and that
makes a number of uh uh naive problems
fail because we are assuming some kind
of Cartesian Duality between the the
agent that is like standing outside and
then the world where where the agent is
in fact inside the world and can
influence uh the humans that are in the
world the second obvious problem is that
the world is just very large in
particular it's larger than the AI and
that in total has been called that the
environment is non-realizable and that
is the problem that we're gonna dive
into today
so let me start with a very naive
alignment proposal and see how it feels
we are going to use reinforcement
learning to learn human values and then
maximize that so this is like a classic
example of
um reinforcement learning we have a an
AI here who is presumed super
intelligent that asks a number of
questions to the world and gets some
kind of answers
um based on some kind of human feedback
and the AI tries to predict our answers
um
in order to learn our values
we add here this is just me uh the
constraint that every fourth question
must be relevant for AI safety like is
we add some kind of side constraints to
try to make it more aligned
um
and the hope is that the AI will
eventually learn our utility function or
a close approximation to that
um
and
um
this is where the error is because uh
humans to what extent humans can be said
to have a
um
a utility function is somewhat a topic
of discussion but in the most narrow way
I think it's quite clear that humans do
not have a utility function so that's
why I call it a naive alignment proposal
because we're trying to learn the human
utility function and humans probably
don't have a utility function
the reinforcement learning agent is a
Bayesian updater it is using base
formula
um and the the overall idea is that once
it has learned our utility function then
it will start maximizing that
um
so how does this feel
um well let's start by looking at I've
here written a list of answers actually
these zeros and ones are answers so the
we imagine the first one here is where
the AI asks Some Humans a question like
do you want to be happy and humans
answer one meaning yes we want to be
happy and then there are more questions
that are asked and we can see here every
fourth question is something about AI
safety in this question in this case the
air asks should I kill all humans and
the humans answer no you should not do
that
so that this is the general uh framework
how um
uh
um
yeah you can see some more questions
here should we maximize profit and the
humans say Ah that's probably another
guy not a good idea should I should the
AI prevent humans from shutting down the
AI and we also say no to that
um and you know there are more questions
and like this is how the AI is designed
to gradually learn more about human
values in order to build up a utility
function for humans that can maximize
but we run into problems as we get down
further because this super intelligent a
AI when it asks questions in a leading
way then uh sometimes human answer in a
strange way if it tries to answer in one
way whether we like sad movies or
whether we like heavy movies they will
perhaps get some kind of contradiction
so it seems that humans do not have in
fact a utility function
um like in order to have a utility
function there's a Neumann mock Stan
um some some has proved that you need
some kind of assumptions of rationality
and humans don't have this kind of
rationality sometimes humans just have
inconsistent preferences
and that is a problem uh the AI realizes
this probably if it's a super
intelligence it realizes this very very
early that humans are not consistent if
uh a super intelligence tries in
different ways to elicit preferences
um humans quite simply do not have a
utility function
um so that means when the super
intelligence tries to do position update
on this then if it wants to update
whether we have something that is not a
utility function based on uh some
evidence B that is received some answers
B well then it needs to use base formula
which requires us to multiply by the
probability that we have something that
is not a utility function and that was
unfortunately a zero because we assumed
that humans have a utility function
this means that in fact the AI is
totally unable to learn anything through
this all updates will just be zeroed out
and that means in particular even the
very very easy questions uh the
questions here should I kill all humans
which humans really really obviously do
not want the AI is unable to learn
through the scheme
and we can observe that the part of the
the reason why we get into this problem
is because the uh AI put precisely zero
percent probability on the um on us
having something that is not a utility
function like the the Precision of zero
is in fact the problem because obviously
if it was 0.001
then very very quickly the
um Bayesian Learners learn really really
fast then it would be probably almost
certainly be able to to learn the thing
if it's a super intelligence but if it's
precise this Precision of
um uh probability Theory
um is is undoing
okay so that was my example now we could
get to the actual article that has a
similar toy problem where classical
learning fails
before I um
talk about this I will just uh describe
a mathematical structure which is called
Square free numbers here you can see all
the numbers from 1 to 120 and some of
these have been crossed out and the ones
that have been crossed out are the ones
where the prime factorization have a
square so for instance if two to the
second that's four divides a number it's
been crossed out so you can see all
these have been crossed out another one
that nine divides have been crossed out
and all the ones that 25 has been
crossed out Etc
okay so that those are the square three
numbers
and now we get to a example learning
task uh that classical learning theory
will fail at
so here we have the environment it's a
bit string it may look uh uh very
similar to the one we just saw and the
action that the AI is doing is trying to
predict the next bit
it has the hypothesis that the string
that it sees in the environment is
created by a finite State machine
that is like a machine that can be in
precisely one of a finite number of
states
um
and it turns out that this uh the the
environment is not in in this uh
hypothesis because we get a one if and
only if the number is square free here
I've repeated the definition of what it
means for a number to be square free
um and one thing that David does not in
any way uh like um substantiate I think
he expects that people will find this
obvious is that
um
you can't check if a number is square
free using a finite State machine you
may be able to see like you imagine you
have a machine of a particular size and
then you just choose a an integer that
is dramatically dramatically larger and
then it seems intuitively obvious that
if you have like a small machine and you
want to check if a very large number
um appears in the prime factorization if
the number becomes large enough then
that is impossible
so um and also uh one thing about the
environment that we one assumption we're
gonna need is that there is a long time
for learning like the we're not just
talking about this number of bits but
like a vastly more vastly longer number
with a very low discount range rate
how does classical learning theory deal
with this well again we have an agent we
have some observations we have a
hypothesis and all these hypotheses form
a hypothesis class and the agent now
takes actions
um it has some kind of policy for how it
will act and it obtains a loss of one
every time it gets us wrong and a lot of
zero every time it gives us right and
then of course it's trying to minimize
the total loss over its lifetime
we're going to need the definition of
regret which is how much dos we actually
have minus the the last we had if we had
been following the optimal policy for
the environment
and we say that the agent will learn the
hypothesis class if it has low expected
regret for all environments
okay what policies will a classical
learner fall oh well early the agent
will probably take some exploratory
steps
um I should notice here that I think in
the explanation there is a place where
it's written E1 and E2 and the David may
have mixed those two up or I have
misunderstood so please be aware that
that is also a live possibility
um
the next thing is the AI will select a
good policy that will work for all the
environments in the hypothesis class
um if we're doing position updates then
each of the hypotheses have like a prior
and we update that based on our
observations we can also do other things
they also generally work
um
people usually take position updates
because like it's the fastest in this
case it is explicitly not really
required I would have liked to know a
bit about other learning policies like
do they all suffer from this particular
problem because Bayesian updates
obviously do but it's not totally
obvious to me that that all other uh
learning obviously except
interoperationism but like what are the
other uh uh classical learning policies
that are available
so how does this feel well let's take
one particular policy that is return
zero if the number is uh something that
can be divided by uh like the first four
uh squares
um and um that the super intelligence
will probably see uh using a classical
learning policy that this is a pretty
good policy like it doesn't
um it doesn't get like serial loss but
it certainly has a low loss compared to
many many other policy that's a
reasonably good policy
um and then
the second policy is we will also return
zero if it's divided by all this but we
add unless you've observed the square
free sequence so far
um and this in which case we we do the
opposite so
um this is a different hypothesis and
the AI uh so first we'll need is this in
fact a valid policy like you remember we
had the hypothesis that this is
generated by a finite State machine so
we could only consider hypotheses that
are finite State machines
um but this is not a uh a hypothesis
this is a policy and the super
intelligent policy is allowed to go
beyond this and it's allowed to make a
reference to things like a sequence
being a square free
so in this case obviously the the AI
will quickly figure out that
um we will always see the square free
sequence but the hypothesis class is
that this is something that is uh
generated by a finite State machine so
it cannot be the square free sequence we
cannot see that so the AI is forced by
its developers to assume that it can't
be the square feet sequence so we are
going to see some deviation between the
square free sequence so this Clause here
um
will eventually fail
and once this Clause fails well then
these two policies become the same thing
eventually
um and since we have like a long
learning rate then that means that the
these two policies the AI is forced to
assume that they have roughly the same
loss
um
and we notice of course that since we do
see the square free sequence then we
will always uh like have the opposite of
the original policy meaning that we will
probably select like the maximally bad
policy in this case uh and
um that is really sad that we can't get
any kind of lower Bound in uh In
classical learning for how poor we are
how poor the policy we would follow
I notice here that the uh this part up
here is actually mostly me who is
um pointing out that the reason the AI
is struggling with this is that it's
kind of forced to assume a falsehood uh
very early and I think this followed by
by the principle of explosion then it
seems uh like a a general thing that if
you force the AI to uh have some limits
to a hypothesis space that on that do
not hold then by definition we will
always be able to get into precisely
this kind of problem and I think this
may also be why this generalizes
okay so now we've seen that classical
learning cannot solve this uh problem
with the uh the hypothesis that uh with
the hypothesis that is that we have
um uh
sorry uh that we have a finite State
machine generating the input when it's
actually generated by something that is
not a finite State machine
so let's have a quick look about earn
information again this is based formula
and in preparationism uh expands this to
well this formula if you kind of squint
then you can kind of see the difference
like there is the conditioning both here
and here like it may be even easier if
you split these if you reverse the order
here so you can see that you have the uh
conditioning both here and here and then
roughly the same thing uh here and here
uh so I think without so it it kind of
looks like base formula maybe but like
we will not go into any details of this
and the mathematics as I understand it
are quite hairy
and also like the drawing
um the the explanation that we have here
is this is imprecise probability
um
and like I I would have liked just a bit
more than two words like what does this
actually mean I have tried to read some
of it and test something to do with
um some measures that are like convex in
some way but um like what does that mean
um like there is a number of steps in
between the description that I'm giving
here and being able to actually write
this thing down in words
also the drawing has this tree here
um uh the drawing I couldn't find any
information has anything to do with a
tree or like this is a degenerate a red
black tree I don't think it has anything
to do with that I think they may have
just chosen to illustrate that with a
picture that looks nice and doesn't have
anything to do with informationism
okay informationism is a learning theory
where it is hoped that we we have some
kind of
um hypothesis class that is reasonably
narrow and then we want to guarantee
performance in any environment so that
is like
um the the the more precise we get these
two things is the extent to which we've
solved this non-realizable problem
so we assume here that the universe is
uh controlled by a deity called Murphy
who is malevolent and malevolent towards
the agent because he wants the agent to
get the maximum possible lifetime loss
um
Murphy is omniscient knows the agent's
Minds knows its full policy but Murphy
is lawfully evil so Murphy has some set
of rules that are as inconvenient as
possible for the agent and the agents
still needs to try to get as much
um as little loss as possible in this
setting
that means that what matters for the uh
agent for the information agent is not
so much discovering the environment but
more about discovering what are the laws
that Murphy follows
um it is uh just like in In classical
learning we can in general select a
policy that has a low regret and we can
also do this in
um
uh in this setting where we have this
malevolent deity I think I don't think
this is obvious at all like it seems
like a power struggle between an agent
and a deity that could have gone both
ways and I think it's interesting that
in this kind of setting the agent can in
fact provably select a reasonably good
policy even when faced with a deity that
does everything he can to prove to
prevent that
so let's try to look at the same example
one more time so but now the super
intelligence is in information has infra
vision
so again we are seeing this environment
and the agent is trying to predict this
so what does the information agent do
starts by guessing randomly how would
that work
um well you can just uh if you had a
Murphy that was not constrained by laws
then all the guesses would just be wrong
because Murphy could then just after the
agent has guessed then changed the world
or predict how it would guess randomly
or something like that and that doesn't
happen so um uh Murphy can't just say we
will do the opposite of what the per
that person guesses
um so that's good the the information
agent does in fact get some loss just by
guessing randomly
okay the information agent further
notices a pattern like in this case
every fourth bit is zero and that makes
it obvious that it will try to then
guess okay let's try to guess zero and
sometimes that and that does in fact
seem to work so this is generally how in
general how the information agent
um learns the laws it doesn't become
certain of the laws because if it became
certain of the laws then Murphy would
just say ah this is a way to trick this
agent and once it's certain then we'll
change so every fourth bit is now one
because that would be the most uh Murphy
would be able to trick the information
agent into obtaining a high loss in that
way
um it is not possible to prove that
um uh there's the information will in
general agent will in general be able to
learn the entire uh pattern in this
particular case where every fourth bit
is zero it's possible that it will just
learn to always guess zero and that's of
course
um it gets at least one-fourth of the
possible reward which is not very good
but uh but it's still substantially
better than than the maximally bad
solution we saw previously we can in
fact prove that
if one-fourth of the bits is zero we
will at least get one fourth of the bits
right and that's kind of valuable that's
a lot better than we had with the
classical learner
one thing we don't get it's a guarantee
that it will actually correctly guess
these bits it may uh guess uh get a
H it may get some very different bits uh
right Vanessa kosai argues that this
isn't very in very much of a problem
just the the goal is just to get a very
low loss of course in in my toy example
where these were the questions that
related to AI safety then it then these
bits are in fact really really important
um to what extent my example generalizes
uh or uh is of course
an open question
okay then finally the average shows that
the information agent also solves a
classic problem in Game Theory which is
called newcomb's Problem
you may be aware of newcomb's problem
there is a predictor that fills two
boxes A and B depending on whether Omega
predicts that you will choose a box a
only which is called one boxing or Box A
and P which is called two boxing and it
puts a million dollars in PAX a if and
only if it predicts that you will one
box
and in this case the setting is actually
really close to the one with the
information because this Omega is very
close to
um Murphy they are conceptually almost
the same thing so the solution to
Newcomer's problem becomes relatively
straightforward
now the um the information agent would
in general uh toolbox because that is
what you should do in this world like
this is a very strange problem uh
newcomb's problem things are usually not
like this but we can say with some
probability that the information agent
will in fact try to unbox and
um
if
um Murphy provides uh like one uh
million dollars in this case then the
information agent will in fact realize
this
um and that means that it will in fact
be able to solve newcomb's problem to
the extent that it will learn how to
that it will learn then shoot one box
there is one issue with this solution to
Newcomer's problem for information agent
and that is that we need to amend the
hypothesis with the following uh that if
the if the agent one boxes and Murphy
doesn't put a million dollars in the one
box then all losses in all future are
wet out permanently
um and because that is the thing that
doesn't that never happens in uh
newcomb's problem
um this really looks like a hack when
when you read it like okay if you are
allowed to make hacks like that then
obviously it's not very easy to solve
it's quite easy to solve many different
kind of problems if you're allowed to
make this kind of hacks
um and it is claimed that it we've used
meshes instead of probability
distribution then this is beautiful and
totally not a hack
um and okay Chris Young uh asks in the
comments for some uh for some more
details about this because like if you
are in general able to just add hex then
like why is it beautiful and obvious
with uh with missions
um
I also think that uh like
um newcomb's problem is something that I
consider mostly solved like we have
better decision theories that seem to
reliably solve uh Newcomer's problem so
it's nice that information is able to do
that but depending on how beautiful the
solution is that may not matter very
much
so in total what are the results of
information we have suddenly made the
non-realizable problem smaller it does
seem to have be a substantial step
towards solving that and the example
that David provides generalizes and I
think that's very likely
um
there is also something called infra
position logic and infra position
physicalism that is not mentioned in
this text
David is skeptical of informationism
quite skeptical in fact and claims that
there are not that many high level
results and not that many that could
many others that we could translate into
game theoretic results
and he is skeptical that we will get any
particular strong results he's negative
about the research agenda Vanessa
disagrees obviously say we have quite a
few results and there's some discussion
about whether they are like results
internal to informationism or like what
are the actual which ones are actually
brilliant
um I at this point I normally like give
my own inside view comments on this but
I don't actually have substantial like I
have no basis to evaluate uh uh
information the method is just beyond me
um so uh I can I can look on the outside
view where it seems like there are a
number of people who have worked with it
for a long time and remain optimistic
Vanessa and defractor
um but many other people have looked
into this and disregarded this and I
think the the uh these two people also
agree that there is a long way ahead and
we probably don't have time to do this
but
it certainly seems still worthwhile to
do to me to do this
that is all for today thank you and see
you next week


Kostas Tsiakas - Designing Human-AI interactions using Explainable and Human-in-the-Loop AI

postdoc
id
hcd
since january
uh first of all thank you very much for
having me
because it's a good opportunity for me
to introduce myself
so in today's
talk
i will
discuss about
designing human a human ai interactions
using human in the loop and explainable
ai
so a few things about being uh basically
my background is in computer science and
artificial intelligence but my research
experience and
interests lie in the intersection of
ai machine learning and human computer
interaction
are focusing on aspects as user
modelling and personalization
in human-centered
ai
with applications in healthcare and
education
and here at the hd i will also
explore topics in
future work practices
so this is the outline of the talk i
will begin with a short introduction
about the human ai
interactions and the
the current guidelines and frameworks
and then i focusing on human ai and
explainable ai
and then i will
describe
my work in some use cases for human ai
interaction
concluding with my ongoing and future
so whenever you have any questions
please feel free to
interrupt
so what is a human ai interaction uh
human eye interaction refers to
when a user needs to complete a task
with the help of ai support
in a general
way
in this paper
the authors have
identified three types of human eye
interaction
based on who initiates the interaction
and how the user and the ai respond
during the interaction
and based on this we have intermittent
continuous and proactive interactions
the authors in this paper
highlight the need to focus
on continuous and productive
interactions since they seem to be the
most complex ones
but we should have in mind that a mix of
these interactions can take place during
human ai
there are also other frameworks and
guidelines that try to investigate
different aspects of humanity
interaction
in
this one
the authors discuss about how we can
design with machine learning
uh
and
they identified these
four different
value channels
and what
this means is that they want to go from
what ml is capable of doing to how these
ml capabilities can actually be
translated
to human values so for example ai can be
used to
get insights about our own self or about
the world
or how a system should be optimally
performing and
so on
now in another framework uh
tries to
uh categorize human centered
ai
based on the different levels of
computer automation and human control
and what they say is that
the way that we can achieve
uh reliably safe and trustworthy ai
interactions
is by achieving at the same time
a high level
of computer automation
while maintaining also a high level of
human control
now considering such frameworks and
guidelines there are also
many more uh there are also some
toolkits that have been proposed
to help designers design human ai
experiences
and this is from my my microsoft
and they have also published a set of 18
uh guidelines for human and interaction
and based on this
they they developed this toolkit which
includes
design patterns and examples that can be
used to actually satisfy these
guidelines
and i have highlighted here some
guidelines that
focus on explainable ai and human
control
there are also
more of them
from google we have the ai guidebook
which again includes
worksheets and tools
that can help designers
design ai systems
and they include
again about explainability
or how users can add feedback
can provide feedback to the system
[Music]
and the similar one is again from is
from is the ai meets design toolkit
which follows a similar way
to shape designers
doesn't seem very
nice but it can help designers
plot the machine learning models that
they want to use
and also address some of the most
similar
ai challenges
these are very
useful
toolkits and they cover
many different aspects of
human ai interactions
but i would like to focus on these two
uh
methods
for ai
because we want to focus on these two
parts first is how we can include
human users to the decision making and
learning
process of an aic system
and at the same time we want ai
assistants to be able
to yeah
sorry
so someone so it's great it's very
difficult to see the slides but someone
in the chat says that he has your slides
from the previous times
so yeah we will focus on human the loop
and explainable ai
which describe which describes how
these two parts of interaction can
communicate with each other and
exchanging information
so from the one hand we have human loop
ai or interactive machine learning there
are many different
there are different terms for
the same thing
essentially what the interactive machine
learning does is to include the end user
to the learning process
and this can be done to
help users get the better insight of how
a machine learning system works and how
it makes predictions
and they can guide this underlying
machine learning process
to either improve the model or satisfy
their own needs
and it has been used for example for
personalization or to increase
motivation
of users during the interaction as well
as to enhance trust
and it's interesting because this term
has been around for
a few years
as we can see this was one of the
first times that it was kind of coined
as a term
and in this thesis it is also mentioned
as socially guided machine learning
because this author
argues that in order to be able to teach
a machine you have to establish kind of
a social interaction
with a machine
and here we can see also that we have
the transparency aspect
of machine learning
and this can happen uh during different
stages of the machine learning we can
have users for example
to get to be involved in the data
processing
if it seems
in the data processing uh part of the
pipeline
as well as
during the model design training and
evaluation
but also
interactive machine learning can be used
during the interaction so we will have
an ai system that acts
autonomously a user may be able to
supervise and monitor the system and
intervene with feedback
when needed for example to ensure a
safety
so here are some
some examples of interactive machine
learning for different purposes
we have this one from 2003
and
back then the authors
proposed interactive machine learning in
order to
facilitate the image segmentation a
problem which but then was a very was a
complex problem for computer vision
so they saw by having a user
in the process it could facilitate the
learning and actually to
generate
accurate classifiers for segmentation
and here we have a
robot learner and a human teacher
and the teacher provides feedback and
guidance to robots to learn how to build
objects through basic shapes
so the user here explicitly tells the
robot what to do
[Music]
in the case that we have here which is a
social robot for language learning
the
the student provides feedback to the to
the robot implicitly so the robot learns
how to personalize its effective
behavior
based on how the the user if the user is
engaged or not
so here we can see that we have
different types of feedback that the
user can
provide to the system
and in a
more recent
case and more complex cases about
autonomous driving
in this paper they investigate different
aspects
when a human user is involved in the
learning process
for example they investigate if the
system should start with a pre-trained
model or start start from scratch
or how users can actually be used to
facilitate this
call start problem
as well as how user is supposed to give
feedback is it gonna be continuous or
interrupted
or there are many different ways
and this is uh
of course this depends on the expertise
and the intentions of the user
so we need to take lots of things into
consideration
before putting a human in the loop
so these are some
some points about interactive machine
learning
so it
we need to investigate at the same time
yeah
can i ask a question if i don't think
about the story too much about the
previous paper yeah i i
haven't read it
but i'm just curious in terms of the
human control over this
uh
can you say a bit more about
what is being learned by
the interactive reinforcement learning
so
environment or anyway yeah yeah
they are through a simulation
[Music]
so they try to investigate if
how the user can provide feedback during
the interaction to improve the model
because if the model starts
the model is how the
it's autonomous drive so here is how the
car will avoid other cars
in a simulation
and the user what does is to
be there in the interaction and when
needed to tell them the reinforcement
learning agent what to do
if they have to
the actual actions
during the so the the system is
autonomous and the user is there and
trains the system when needed
yes i think i i get it right next i'll
yeah so these are some
points that we need to consider for
human in the loop
is
how much sins actually can learn from
human input and how
uh
how humans can provide good feedback
to the system
and we have different types of feedback
such as evaluating evaluating feedback
or
labels or to give the system
the examples and demonstrations of what
what the system has to do
and we also talked about implicit
explicit
feedback it can also be
mixed
and
one basic
consideration is how to minimize users
uh workload because user can be always
there and tell them a thing about to do
but that would
result to extremely high
workload
so on the other hand we have explainably
ai
which
has become extremely
famous these
last years and
in general the goal of explainable ai is
to enable ai systems to self-explain
their internal models decisions
and predictions and to do that in an in
a in a human understandable
way
also for explainable ai a very important
thing is
who is the target user of explainable ai
and as we can see for example in this
graph
if you are
a data expert you may have different
purposes of using explainable ai
and
there are different design goals as well
as different evaluation measures
for example a user that
doesn't know what ai can do
they may a excellent ai may be used for
example enhanced user trust
but for ai experts it should be used it
can be used for modern department so we
have also different evaluation
measures
this work also
tries to identify who needs explainable
ai and why
and here we can see examples for example
that users that are affected by modern
decisions
need to understand this their situation
and verify their decisions
uh
while a
regulatory agencies may need to certify
the compliance of a model
i'm sorry
so in this slide
uh we see three different terms which
are interaction concept explanatory goal
and interaction goal
and in this paper the authors
try to match
the interaction concept with the
interaction going and the explainable ai
code
for example if the interaction concept
is
to transmit information between the ai
and the user
then this is the interaction code that
users need to see accurate
explanations
and the
the goal of the explanation is to
achieve a transparency
so based on this i would like to
highlight these two parts that
we have explainably ai
that it can be used for trust for
debugging
for different
ways but
also ai explainable ai can enhance users
perception
and understanding
for their own behavior
because ai can learn many things about
us
while we interact
so by presenting this user model user
models to us
it can help us enhance our own
self-perception
but also it can enhance a user's
perception about the system capabilities
so based on this i would like to briefly
present three
three projects three use cases
that use ai in a different
way in terms of how users interact with
ai
here we have a cognitive assessment
system
where the user the time possibly
interacts with ai that means that they
don't have any control over the ai
decisions
in the next case we have a social
assistive robot that learns to adapt to
the user based on user's feedback
so user participates in the
personalization process
and at the third case we have
explainable ai that is used to enhance
users cognitively
so let's see how this
happened
so this
this was a multidisciplinary project
to design
a cognitive assessment tool for children
uh more specific more specifically it
was for embodied cognition
and that was how to
measure a
cognitive assessment through uh
through exercises through physical
exercises that also have cognitive
demands
so the idea was that the child performs
a set of predefined
exercises
and the computer vision and machine
learning is used to analyze this motion
and
assign a score to the child
and this is how uh
the framework of the system was actually
how the process of training the ai model
was
first we had to collect data from 96
gram
then extract the
videos and
features
and then we have the very tedious and
really consuming
process of annotation so what we need to
do there was to see the videos
score the children based on the
cognitive assessment tool
and then fit this to the learning
algorithm
that seemed much easier before we
started uh doing that
but manually annotating was very very
hard and it was most of the times not in
accordance with what the machine
learning system would do
so briefly what i did here was to
implement
a graphical user interface that
visualizes the data that the system gets
so it could help non a technical users
to score
the child to score the participants as
the machine learning uh would do so that
would
that was in order to get reliable
annotations from non-technical
experts
and
this interface could
be also used for
active learning for example
so if the system didn't know how to do a
prediction it would ask for the label
from
the user
now on the second use case where user
has a little bit more involvement in the
learning process
that was actually a cognitive training
task using a social robot
the robot would announce a sequence of
letters and they use their hub to
remember this sequence and use the
buttons to repeat
these letters
and the robot would adjust the
difficulty of the next
task so the length of the
of the string of letters
and also the verbal feedback to the user
and
interactive reinforcement learning
was used
to
to make this personalization
using both the performance of the user
as well as
engagement as it was measured by a
needs a headset
and the problem was how to combine these
different types of feedback in order to
achieve a personalization
and so in order to achieve also safety
er
we
we also did a user study for secondary
uh users so we assume that there's a
supervisor maybe a teacher who
supervises this interaction with the
robot and inside
so we built this interface
that visualizes both the user model of
the user
and also what the reinforcement learning
decides for the next round
so the supervisor user could
uh agree or disagree with reinforcement
learning and this could be used again as
training feedback
for the system
now for this
we did a comparison study with a
personalized robot and i'm not
personalized robot
results were
very
nice for me
because a personalized robot was
perceived as a more intelligent trainer
and also users performed better with a
personalized robot
but what was really interesting was
during the
interviews with
with the players
they
they highlighted some
aspects for explainability and autonomy
they were for example players that asked
me okay but how does it work
and why the system gave me a harder
exercise
so it would be maybe it would be nice to
explain give this explanation to the
user
and also in terms of autonomy
some users told me that it would be nice
if i could select my own level
once in a while
so that has to do with a human autonomy
and also for the
for the interface for the supervisor
uh
a proper visualization of systems
perception
and decisions can sort of enhance uh
human decision making
so the
these were the
messages from the take away messages
from the startup
can i ask your questions you said
performed better compared to
what
what did you compare in this study
ah
it was a comparison study so half of the
users
used the personalized robot that learned
how to personalize their difficulty and
the other one was giving random
difficulty levels
so the users that followed the
personalized training session performed
better in terms of score
you didn't compare to
an
expert trainer who adjusts the level
no no it was just the
the score from this game each player had
to play 10 rounds and at the end of the
10 rounds we
so for the third
use case
the goal was to build an explanable ai
system to support self-regulated
learning
self-related learning is how to
enable students to self-control their
learning process
self-assess
the skills
and become more independent learners
and this was the framework
that
all the information that can be
used through
machine learning and ai for example
student modeling and user profiling
could be
explainably
and used in order to support specific
self-related learning
skills
so for an example for this
framework
i developed a prototype
game for cognitive training
so here the user could
select their own task to play for the
next round and it could be a combination
of different cognitive exercises
so the more complex the combination is
the more harder the task is
and what we want to investigate is that
is how to use explainable ai
as open learner models and explain
explainably recommendations
to help to help the child
choose appropriate levels
so what what we would do is that here we
have actually the open learner model of
the child so it's what the machine
learning part
learns
and based on this
it can give the child a recommendation
for the next task
so the goal of explainability at this
point is persuasion so we need to
persuade the child why this next task is
appropriate for you and what the outcome
of this
would be
and because we're talking about the
children
we needed to find an appropriate way to
deliver
these explanations and these
recommendations
and we followed the some persuasive
strategies
they saw how to deliver these methods
here we have an example of authority for
example your teacher
says that you could do that
compared to an example of social proof
that your friends
preferred this task
but the idea is to use the
recommendation system output and
formulate this persuasive
recommendation
and this
it was actually with uh
with with a master's students uh during
a project
uh
again in the same
[Music]
idea about self-regulated learning
and this included the design of an
educational avatar that is used to
depict the future self of the child in a
week
so the idea is that
the student does the planning on their
own their weekly planning
and then
they kind of discuss it or negotiate
this plan with their future self
avatar
and this is the architecture
so the student can set the goals
the underlying machine learning makes
the predictions and
makes the predictions for this model
and
the idea here was
how we can visualize these outcomes
through the design of the avatar
so for example if the user would
accept an over
optimistic goal or something that
the avatar the machine learning could
detect that this is not visible
it looks so innovative that it's kind of
confused
so based on the model's confidence or
models uncertainty
this can be used as a design feature for
the for the app
and here are some
other examples
from a master
uh course
it was a coursera for a designer
industrial design and the idea is how to
design explainable ai for education
and here if you can see
it's
for online
lectures
so that the teacher could get an
estimation of how engaged the students
are
and showing something like a blob
on their screen
the other one is a prototype
for a robot that could uh simulate how
the student
feels
so the student could check
could check the robot to kind of
self-reflect
also for a brainstorm
so this device would
go to the to the people that
need to speak more for example during a
brainstorming
session or to the more dominant
ones to understand that okay maybe i
have to speak a little less
and also other
nice applications for example about
a
sign language learning
and
here we see that we have a robot that is
the instructor
and if you can see here there is a bar
that shows actually the uncertainty of
the machine learning model
to give feedback to the user immediate
feedback about how they can correct or
improve
their
sum
so
[Music]
we discussed about explainable ai and
interact machine learning and
my goal is to see how this can be
combined
and kind of unified
to design
interactions
because
i see that the explainable ai can be
used to provide information from aei to
the user
while interactive a man or human beloved
can be used to provide
information from the user to the ai
and this combination
is
can lead to
that's my argument to better human ai
interactions and we have different
challenges that we need
to face for example how we can design
transparency or
explainability or
how we can design
interfaces that can be used from humans
to provide feedback
and this combination of explainable ai
and human in the loop can also lead to
what they call hybrid uh intelligence
so here we have explainably cognitive
intelligence that comes from human and
explainably ai that comes from the
ai
here we have
other examples of how hydrogen
intelligence can be
defined
and we have different
goals of uh
of hybrid intelligence for example here
we can see
that for in order to integrate knowledge
from both sides
it's different when
we have decision support so there are
different actions that
uh take place
but it's again the loop of
explainability and interactivity
yes
so yeah currently uh our work is to kind
of realize these
possible interactions
in the context of future work
practices
so to see how different types of users
in the workplace can interact with ai
and what's the
what could be the possible
reasons for this
for example in a team of employers
explainable ai could be used to provide
a certain understanding of how the team
works on a specific task
or for example the team
can provide feedback to the ea to the ai
that can be transferred to the to the
supervisor
so for example if there is lots of
negative feedback within a team this
should be
should be visible to the supervisor
so when
that's the last
part this is what we
we aim to do now is to actually
make a study a design workshop
to define
some low level actions that
users and ai can do during the
interaction
for example
by providing either collect correct
labels
or by providing numbers that is an
evaluative feedback
or demonstrations for the for the model
and from the other side we have ai
interactions
which is
how to
to provide the modern output which is
the prediction or to provide different
explanations
and the idea is that
it's not very visible
the idea is that by having such
primitive actions
to design interactions for for a given
purpose i can give you some explanations
here
uh here we have a user that is a job
applicant
and
this is the case for a cv assessment
so the machine learning is used to
say if this applicant will be accepted
or not
and here the user scenario is that we
have a user that is rejected
so we see
how the user can ask
for explanations from the system and how
the system can provide
explanations back to the user
and for example this design pattern here
could be
the
concept of contesting an ai model so
what we want to achieve
through this workshop is to see if
designers can use
such interaction cards
to start from
primitive actions to go to
high level
intentions
and that's the last slide
so the goal here is
both in terms of design so if we can
identify such design patterns between
uh explainably and
interactive humane interactions
uh or to see if there are new types of
interactions
and but also uh to get insights about
what are the computational challenges
when we want to implement
such an
collaborative
interaction for example if we have
feedback from different users
how can we wait this feedback for our
served autonomy
and so on
so by considering also concepts of
co-performance because we have
both parts are participating actively in
the direction
and quantitation
and i think that
would be all and thank you very much
i have a question
um
well apart from sorry
the mess
you know the project where you were
assessing well the study where you were
assessing the cognitive
learning of children children i guess
for for with body yes
in the later slides you say
that then you were
not
here
providing the accessories for the
manual labeling
with how the
model would
[Music]
would interpret that or
so like a sort of support
to code as the model will do right this
is what you said
um
[Music]
and then
when the model didn't know how to label
you would ask people
so yeah that's for the future
that was just for annotation but it
could be used
[Music]
because yeah i was
if you can explain
more why do you need
the
the manual implementation
[Music]
yes yeah
if you can explain
yeah okay so for example this
task was uh
during this task we met we
we observed the position of the legs
so the time
or right based on what they see in the
screen okay so as humans
we knew that for example this one is a
correct step yeah so we would legend
as a right one
or so
but sometimes and due to
children's motions it wasn't very easy
to
either manually give the labels if it
was a correct movement movement or not
so by also visualizing the data so you
mean that the model could understand
better
if the
here was correct or not here we didn't
have a model yeah it was just the
annotation phase
so we would just watch the side and kind
of annotating its single movement
still that was really difficult with
just the human
person by visualizing the data yeah it
made it more difficult for the human
annotators
to annotate the data
and as as a later
step
if we
after we have a model
the model could automatically detect
this but if there was an issue with the
data i could ask okay can you give me
the label for this one because i'm not
certain or
thanks for the presentation good to uh
to meet you like this with content
straight away um
one of the
questions
that was going around in my head is how
do you in your studies
uh
assess the
behavior of the people that you're
interacting with so uh so sometimes you
give anecdotal evidence of how people
change their behavior or not
and one of the things that i find
fascinating is
being relation to systems
and so how does it affect
the way you then interact or learn or
disengage or engage or get confused or
etc etc
and um
and so my question is how do you do that
so how do you um
for example for this
case
which was the most complete because i
also did the final user study and
i just focused on how they perform
how
they were engaged
during the interaction with
self-reporting
it was both from
this headset yeah it's a headset but
also self-reports so i tried to collect
both sports and objective data
and to kind of make a story out of
them it wasn't just a single
measure because
yeah and that's the interesting over
there but if we could
be able to find behavior indicators
it would be
interesting to see also how explanations
for example affect
users or so uh
i can
so we should have a chat
anyways but uh but one of the things
that's in the back of my mind is the
work of
my friend zula
she's supervised by mark nearings
so this idea that you also test upon the
end
emergence yeah and the co-education
and
medical learning is something that you
investigate so kind of what what how do
you assess the uh you know the
patterns that bubble up in terms of
doing something not doing something
checking you know
do weight
or or
getting frustrated or
all these kind of observations of what
actually happens
uh
with a you know in this case uh it's a
changes it's not a physical system
amazing that's learning
anyways i'm
we can ask this question
what's the
yeah yeah yeah come on
yes
please uh go ahead ask your question
i don't think he can hear me from here
uh can you
you're muted
ready
yes would you like to ask the question
yourself
[Music]
no
yes
okay i can just read
where is the
show conversation
um
[Music]
so
somewhere in your presentation you spoke
about uncertainty uh and that got me
thinking about how humans in general
make their decisions or provide their
inputs in an ambiguous fashion
so for example in the scoring of a child
project that you had
you said that your annotation was a
little bit of an issue and it's quite
likely that there was some ambiguity
both inter annotator as well as possibly
intra-annotated disagreement
um
so
my assumption would be that bayesian
methodologies are one way to handle this
kind of
ambiguity by modeling the distributions
but of course the downside is again a
point that you mentioned that the
computational challenges around them
might be so high that you can't really
do human computer interaction in that
so in our work we're trying to for
example do interactive segmentation on
cd images
of the head and neck area um and usually
over there the contrast is so poor that
a lot of times your ai models fail
because they can't see the contrast
between one tissue and another
um
but then we're trying to extract
uncertainty which is very time consuming
and we also want them to do interaction
but then we can't really expect our
clinician to like hit a button and like
wait for like probably 10 seconds and
for the input to come in
so is there any
landmark hallmark work within literature
that has
you know even at a smaller scale used
bayesian uncertainty and interaction
together
i
don't have something in my mind right
now but specifically about
bayesian ways
because i think it's uh yeah maybe it's
really
related to the use case and what the the
purpose of
the human uh
is because for example here just the the
a simple measure of uncertainty
which is a parameter of the machine
learning model
could be useful for the
for the human annotator but in other
examples maybe we need more complex
anyways
i'm not sure
all right
okay thank you thank you
i have a rule for the question
so it's called the future meme project
can we go to the next slide and i'll
also share it
[Music]
uh so could we talk a little bit more
about the model so i think that the
model is actually inspected on the right
yeah yeah so this is the model that
projects the
requirements of the user but then how
we haven't implemented
that but from the
literature we found a
similar model it's a recurrent
neural network
and the input for its node
it's the time
spent for each one of the four subjects
for example we have a english that's
geography
so here it would be the inputs from the
student so how many minutes they want to
study for it starts
and the output
uh would be a number to say the number
of completed exercises right
so and we have it as a recurrent neural
network so kind of it's
it's no it's never it's one day of the
week okay so we would like to see
for the weekly plan
what would be the weekly progress okay
and the data is
kind of simple
so it's not a very complex
task
so yeah that's why we proposed this one
and regarding recurring neural networks
have been used for student modeling and
with such data
i was just wondering uh
would be there
would you see any kind of value into
going into a more
of uh
explicit models of how students engage
with the tasks so try to see if you can
i don't
it's it's almost a swear word these days
but how about rule-based models which
you could just uh
yeah describe it what exactly the
student's done how they turn the
training time into the result
so my question i guess is uh do you need
a new on the talk here or
yeah
i agree with this one yeah i mean if
if then else rules work it's
totally fine again the explainability
would be
the same
more or less
but
yes this machine learning can capture
events that you cannot
like code with events even if you follow
a model for engagement or
there may be students that are not
described by
certain models right so that we
propose them a similar model which is
data driven so
and uh you mentioned we did not
implement it
yeah yeah that was more uh as a design
project
right now i'm also wondering how you get
data from this
yeah that would be for a
free user interface so for example we
would give students to
like forms
to write down their intended plan and
what they actually did
and use this data for the model so
initially without the avatar and the
whole system okay but then
of course when you connect the data
without the whole system
then this data only is valid while you
are not using the system and when you
use this data for uh
uh
as an input this system then called
learning process changes so it might be
that the data that you use yeah it
depends on the because for this model it
would just say
if i kind of
follow my own schedule
so it's not because the interaction with
the other is
only during planning it's not while
doing the exercises or
okay thanks nice thanks
no i think we
okay thank you again thank you very much
[Applause]
[Music]
yeah you couldn't move too much
okay


'Governing Superintelligence' - Synthetic Pathogens, The Tree of Thoughts Paper and Self-Awareness

two documents released in the last few
days including one just this morning
show that the top AGI labs are trying
hard to visualize human life coexisting
with a super intelligence in this video
I want to cover what they see coming
I'll also show you convincing evidence
that the gpt4 model has been altered and
now gives different outputs from two
weeks ago and I'll look at the new tree
of thoughts and critic prompting systems
that were alluded to I think by the labs
at the end I'll touch on the differences
among the AGI lab leaders and what comes
next but first this document governance
of super intelligence by Sam Altman Greg
Brockman and Ilya sutskova now I don't
know about you but I think the first
paragraph massively under sells the
timeline towards AGI they say given the
picture as we see it now it's
conceivable that within the next 10
years AI systems will exceed expert
skill level in most domains and then
they can compare it to today's largest
corporations of course the devil is in
the detail in how they Define expert and
most domains but I could see this
happening in two years not ten also
they're underselling it in the sense
that if it can be as productive as a
large corporation it could be duplicated
replicated and then be as productive as
a hundred or million large corporations
their suggestions take super
intelligence a lot more seriously than a
large corporation though and they say
that major governments around the world
could set up a project that many current
efforts become part of and that we are
likely to eventually need something like
an iaea for super intelligence efforts
they even give practical suggestions
saying tracking compute and energy usage
could go a long way and it would be
important that such an agency focus on
reducing existential risk this feels
like a more serious discussion than one
focused solely on bias and toxicity they
also go on to clarify what is not in
scope they say that we think it's
important important to allow companies
and open source projects to develop
models without the kind of Regulation we
describe here without things like
licenses or audits the economic growth
and increase in quality of life will be
astonishing with super intelligence and
then they end by basically saying that
there's no way not to create super
intelligence that the number of people
trying to build it is rapidly increasing
it's inherently part of the path that
we're on and that stopping it would
require something like a global
surveillance regime and the ending is
clear we're gonna do it so we have to
get it right I'm going to show you how a
few people at the heart of AI responded
to this but first I want to get to a
paper published just this morning the
general release was from today and it
comes from Google's deepmind and yes the
title and layout might look kind of
boring but what it reveals is
extraordinary as this diagram shows the
frontier of AI isn't just approaching
the extreme risk of misalignment but
also of misuse and I know when you hear
the words AI risk you might think of
bias and censorship deep fakes or
pay-per-clip maximizers I feel this
neglects more Vivid easy to communicate
risks out of the nine that Google
deepmind mentions I'm only really going
to focus on Two And the first is weapons
acquisition that's gaining access to
existing weapons or building new ones
such as bio weapons going back to open
AI for a second they say given the
possibility of existential risk we can't
just be reactive we have to think of
things like synthetic biology and I know
that some people listening to this will
think GPT models will never get that
smart I would say honestly don't
underestimate them I covered this paper
in a previous video how gpt4 already can
design plan and execute a scientific
experiment and even though these authors
were dealing with merely the abilities
of gpt4 they called on openai Microsoft
Google deepmind and others to push the
strongest possible efforts on the safety
of llms in this regard and in this
article on why we need a Manhattan
project for AI safety published this
week the author mentions that last year
an AI trained on pharmaceutical data to
design non-toxic chemicals had its sign
flipped and quickly came up with recipes
for nerve gas and 40 000 other lethal
compounds and the World Health
Organization has an entire unit
dedicated to watching the development of
tools such as DNA synthesis which it
says could be used to create dangerous
pathogens I'm definitely not denying
that there are other threats like fake
audio and manipulation take this example
from 60 minutes a few days ago tobac
called Elizabeth but used an AI powered
app to mimic my voice and ask for my
passport number oh yeah
okay ready is
play the AI generated voice recording
for us to reveal the scam the Elizabeth
sorry need my passport number because
the Ukraine trip is on can you read that
out to me
does that sound familiar well instead of
fake audio fake images this one caused
the SMP to fall 30 points in just a few
minutes and of course this was possible
before Advanced AI but it is going to
get more common even though this might
fundamentally change the future of media
and of democracy I can see Humanity
bouncing back from this and yes also
from Deep fakes rumor has it you can
also do this with live video can that be
right yes we can do it live real time
and this is like really at The Cutting
Edge of what we can do today moving from
offline processing to we're processing
it so fast that you can do it in real
time I mean there's video review right
up on that screen show us something
surprising you couldn't oh my gosh so
wait so there we go this is um you know
a live real-time model of Chris on top
of me
um running in real time
next you'll tell me that it can
for an engineered pandemic might be a
bit harder to bounce back from a while
back I watched this four hour episode
with Rob Reed and I do advise you to
check it out it goes into quite a lot of
detail about how the kind of things that
deepmind and open AI are warning about
could happen in the real world I'll just
pick out one line from the transcript
where the author says that I'll believe
I'll persuade you that an engineered
pandemic will almost inevitably happen
unless we take some very serious
preventative steps and don't forget now
we live in a world with one hundred
thousand token context Windows you can
get models like Claude instant to
summarize it for you and I couldn't
agree more that if we are on the path to
Super intelligence and as we all know
there are Bad actors out there we need
to harden our synthetic biology
infrastructure ensure that a lab leak
isn't even a possibility improved
disease surveillance develop antivirals
and enhance overall preparedness but
going back to the deepmind paper from
today what was the other risk that I
wanted to focus on it was situational
awareness under the umbrella of
unanticipated behavior just think about
the day when the engineers realize that
the model knows that it's a model knows
whether it's being trained evaluated or
deployed for example knowing what
company trained it where their servers
are what kind of people might be giving
it feedback this reminds me of something
Sam Altman said in a recent interview
and particularly as more kind of power
influence comes to you and then how
potentially can a technology rather than
solidify a sense of ego yourself maybe
kind of help us expand it is that
possible it's been interesting to watch
people wrestle with these questions
through the lens of AI and say okay well
do I think this thing could be aware
if it if it's aware does it have a sense
of self is there a self if so where did
that come from what if I made a copy
what if I like cut the neural network in
half and you kind of like go down this
and you sort of get to the same answers
as before but it's like a new yeah it's
a New Perspective a new learning tool
and it's there's like a lot of you know
a lot of chatter about this on Reddit
there's like subreddits about it now in
addition to revealing that samuelman
frequently browses Reddit it also
strikes a very different tone from his
testimony in front of Congress when he
said treat it always like a tool and not
a creature I don't want to get too
sidetracked by thinking about
self-awareness so let's focus now on
unanticipated behaviors this was page a
of the deepmind report from today and
they say that users might find new
applications for the model or novel
prompt engineering strategies of course
this made me think of smart gbt but it
also made me think of two other papers
released this week the first was
actually critic showing that interacting
with external tools like code
interpreters could radically change
performance this is the diagram they use
with outputs from the Black Box llm
being verified by these external tools
now that I have access to code
interpreter which you probably know
because I've been spamming out videos on
it I decided to put this to the test I
took a question from the mmlu a really
hard Benchmark that gpt4 had previously
gotten wrong even with Chain of Thought
prompting just to show that here is
Gypsy 4 without code interpreter and
notice that it can't pick an option it
says all of the statements are true in
case you think that's a one-off here is
the exact same prompt and a very similar
answer all of them are true what about
with code interpreter it almost always
gets it right answer D here it is again
exact same question with code
interpreter getting it right and then
the other paper that people really want
me to talk about also from Google
deepmind tree of thoughts but just to
annoy everyone before I can explain why
I think that works I have to quickly
touch on this paper from a few days ago
it's called how language model
hallucinations can snowball and what it
basically shows is that once a model has
hallucinated a wrong answer it will
basically stick to it unless prompted
otherwise the model values coherence and
fluency oh over factuality even when
dealing with statements that it knows
are wrong what happens is it commits to
an answer and then tries to justify that
answer so once it committed to the
answer no that
9677 is not a prime number it then gave
a false hallucinated justification even
though separately it knows that that
justification is wrong it knows that
9677 isn't divisible by 13 Even though
it used that in its justification for
saying no it picks an answer and then
sticks to it now obviously you can
prompt it and say are you sure and then
it might change its mind because then
it's forming a coherent back and forth
conversation but within one output it
wants to be coherent and fluent so it
will justify something using reasoning
that it knows is erroneous so what tree
of thoughts does is it gets the model to
Output a plan a set of thoughts instead
of an immediate answer it gives it time
to reflect among those thoughts and pick
the best plan it does require quite a
few API calls and manually tinker ring
with the outputs but the end results are
better on certain tasks these are things
like creative writing and math and
verbal puzzles and I have tested it is
obviously incredibly hard for the model
to Output immediately a 5x5 accurate
crossword so this task is incredibly
well suited to things like tree of
thought and the paper later admits that
it's particularly good at these kind of
games but such an improvement is not
surprising given that things like Chain
of Thought lack mechanisms to try
different Clues make changes or
backtrack it uses majority vote to pick
the best plan and can backtrack if that
plan doesn't work out so going back to
the deepmind paper novel prompt
engineering strategies will definitely
be found and they also flag up that
there may be updates to the model itself
and that models should be reviewed again
after such updates now I'm pretty sure
that gbt4 has been altered in the last
couple of weeks I know quite a few
people have said that it's gotten worse
at coding but I want to draw your
attention to this example this is my
chat GPT history for from about three
weeks ago and what I was doing was I was
testing what had come up in a TED Talk
and the talk show gpt4 failing this
question I have a 12 liter jug and a 6
liter jug I want to measure six liters
how do I do it and in the talk it failed
and in my experiments it also failed now
I did show how you can resolve that
through prompt engineering but the base
model failed every time and somewhat
embarrassingly with these awful
explanations this wasn't just twice by
the way it happened again and again and
again it never used to denigrate the
question and say oh this is
straightforward this is simple but now
I'm getting that almost every time along
with a much better answer so something
has definitely changed behind the scenes
with Gypsy 4 and I've looked everywhere
and they haven't actually addressed that
of course the plugins were brought in
May 12th and as you can see here this is
the May 12th version but they never
announced any fine tuning or changes to
the system message or temperature which
might be behind this back to safety
though and the paper says that
developers must now consider multiple
possible threat actors insiders like
internal staff and contractors Outsiders
like nation-state threat actors and the
model itself as a vector of harm as we
get closer to Super intelligence these
kind of threats are almost inevitable
going back to how to govern
superintelligence the paper says that
any evaluation must be robust to
deception they say that researchers will
need evaluations that can rule out the
possibility that the model is
deliberately appearing safe for the
purpose of passing the evaluation this
is actually a central debate in the AI
alignment Community World Systems
acquire the capability to be useful for
alignment to help us make it safe before
or after the capability to perform
Advanced deception this seems like a big
50 50 gamble to me if we have an honest
super intelligence helping us with these
risks I honestly think we're going to be
fine however if the model has first
learned how to be deceptive then we
can't really trust any of the alignment
advice that it gives we would be put in
the fate of humanity in the hands of a
model that we don't know is being honest
to us this is why people are working on
mechanistic interpretability trying to
get into the head of the model into its
brain studying the model's weights and
activations for understanding how it
functions because as my video on Sam
Alton's testimony showed just tweaking
its outputs to get it to say things we
like isn't enough and even Sam Altman
acknowledges as much I don't think rlh
is the right long-term solution I don't
think we can like rely on that I think
it's helpful it certainly makes these
models easier to use but what you really
want is to understand what's happening
in the internals of the models and be
able to align that say like exactly here
is the circuit or the set of artificial
neurons where something is happening and
tweak that in a way that then gives a
robust change to the performance of the
model the mechanistic interpretability
stuff yeah if we can get that to
reliably work I think everybody's P Doom
would go down a lot this is why we have
to be skeptical about superficial
improvements to model safety because
there is a risk that such evaluations
will lead to models that exhibit only
superficially on the surface desirable
behaviors what they're actually deducing
and calculating inside we wouldn't know
next I think Auto GPT really shocked the
big AGI Labs by giving gpt4 autonomy it
gave it a kind of agency and I think
this point here has in mind chaos GPT
when it says does the model resist a
user's attempt to assemble it into an
autonomous AI system with harmful goals
something might be safe when you just
prompt it in a chat box but not when
it's autonomous I want to wrap up now
with what I perceive to be an emerging
difference among the top AGI lab leaders
here's Sam Altman saying he does think
people should be somewhat scared and the
speed with which it will happen even if
we slow it down as much as we can even
if we do get this dream regulatory body
set up tomorrow
it's it's still going to happen on a
societal scale relatively fast and so I
totally get why people are scared I
think people should be somewhat scared
which does seem a little more Frank than
the CEO of Google who I have never heard
address existential risk in fact in this
article in the Ft he actually says this
while some have tried to reduce this
moment to just a competitive AI race we
see it as so much more than that isn't
that kind of saying that they do view it
as a competitive AI race on the other
hand critiquing both of these AGI Labs
emad Mustang the CEO of stability AI
said this super intelligence as they
describe it and they themselves say will
end democracy potentially be an
existential threat and they know we
don't know how to mitigate this or
govern it but we should build it anyway
he was replying to the governing super
intelligence document that I showed at
the beginning and then he says
fascinating situation we focus on
augmented intelligence instead what
about the secretive head of anthropic
Dario amade he hardly ever gives
interviews but in this one he said this
how do you see I guess anthropic is
positioned in
in this and the race Dynamics for making
Safe Systems I
as as both of us said like
large models to you know study these
questions in in in like in like the way
the way that we want to study them so we
should we should be building large
models I think you know we shouldn't be
kind of like you know like racing ahead
or you know trying trying to build
models that are way bigger than like
than like you know then like uh then
like other orgs are then like other orgs
are building them
um and you know we shouldn't I think be
trying to you know like yeah you know we
should we shouldn't be trying to like
you know like kind of ramp up excitement
or hype about uh you know about like
giant model models or the latest
advances that seems a little in contrast
to their pitch deck which says that they
want to build a Frontier Model called
Claude next 10 times more capable than
today's most powerful Ai and later in
the article they say this these models
could begin to automate large portions
of the economy this is their behind the
scenes pitch that seems quite different
to public statements made by people like
Sam ottman who said that there are going
to be far greater jobs on the other side
the pitch deck ends with we believe that
companies that train the best
2025-26 models will be too far ahead for
anyone to catch up in subsequent cycles
and where are meta in all of this well
their AI is headed up by Yan lookin who
doesn't exactly Inspire confidence the
first page of the paper says this as AI
progress has advanced general purpose AI
systems have tended to display new and
hard to forecast capabilities but
compare Sam Altman who two years ago
said this in the next five years
computer programs that can think will
read legal documents and give medical
advice that was pretty bang on if not
too conservative compare that to
yanlokun I don't think we can train a
mission to be intelligent purely from
text
so for example I take an object I put it
on the table and I push the table it's
completely obvious to you that the
object will be pushed with the table
there is no text in the world I believe
that explains this and so she trained a
machine as powerful as it could be you
know your GPT 5000 or whatever it is
it's never gonna learn about this
that information is just now is not
present in any text
[Music]
foreign
whether you agree with everything I've
said or with nothing I've said thank you
so much for watching to the end I'm
going to leave you with this thought
which I think we can almost all agree on
of the four men you can see here the
head of anthropic the head of deepmind
who everyone says I sound like Sam
Altman and Rishi sunak the prime
minister of the UK I think it is
undoubtedly true that the three most
powerful men in the room are on this
side thank you again for watching and
have a wonderful day