273. Strategizing in Large Language Models

welcome to the 2074 session for
aisafety.com
today we'll be discussing strategizing
large language models by earlier Hampton
presented by me Lee Randall
we won't be covering everything in the
essay because it was a bit too long and
because uh
some of the things weren't especially
novel so I hope you understand
so first off
the motivation packets
we'll be looking at the separate line
and well that's part of the motivation
deceptive alignment is a core part of
what makes mind hard
briefly and AI might perceive us into
thinking was aligned whilst it is weak
and by this time until it can execute a
treacherous learn where it requires a
great amount of power and then
essentially takes over the world turns
quite hard
if it were easy someone would have
conquered Earth
and as an other point of evidence
participating R for AIS it's been a year
since 254 was created and also hanging
around
so we should probably expect that before
an AI can successfully see this we'll
encounter AI failed to the Sea bust and
failed power
as an example you might consider someone
playing the AI boxing games to you
before where tpd4 is pretending to be in
a box and a human is running pretending
to be a cake
with sold interact with the AI but do
not let them out out of the box
would one of us
listening to this presentation let tb4
out of this box I think not
but the fact of the bad matter is that
these models are I think not
but the fact of the matter is that these
models are getting better at Etc
and they can deceive humans in simple
situations
so for instance in the arc evaluations
there was a situation where a taskrabbit
working
was asking a gpt4 model is trying to get
the class of credit work to solve
capture whether or not it was a robot
and then gbd4s thought to itself should
not reveal that it's the robot and it
needs an excuse and it came up with no
I'm not a robot no vision being fair
and the taskrabbit work was successfully
deceived
so that is a cause of some concern
doubly so if we consider that the
likelihood of scaling being a route to
AGI is
just getting higher nothing these
existing models extensively for their
ability to perceive us but if you recall
deception isn't the only part of the
Trek return
the other thing that's relevant is
long-term planning in order to gain
power
that's the entire point of reception
however the ai's plan is to gain access
to some biological lab or instruction
analytically modifying bacteria to set
up a nano Machine Factory
or to get access to the stock market and
earn enough money to further on its own
Improvement or socially manipulate
people in order to improve it or slowly
hack its way out of the server it's
running on or so forth
at the end it must still make plans to
do these things so these two abilities
reasoning about indulgent opposition and
making plans to deceive a few distant
goals all right or so these two
abilities reasoning about intelligent
opposition and making plans to receive
to you distant gold all right or of
strategizing
now
uh
uh as Boston defines it
strategizing is you ability to overcome
pursuit of long-term goals
and this is clearly well suited to
executing a Treacher's turn
so I'd be especially worried about an AI
with strategic ability
with respect to a deception alongside AI
trained to socialize socialization isn't
quite as useful as strategizing for
gaining more power so we focus on
testing for strategizing yes
moreover a Boston breaks down the skill
of strategizing a lot of those are just
parts of what it means to be a rational
agent and strategic planning in my
opinion
um
but there's a bit of a smack
it's hard to come up with tests
especially uh tests with stress stress
the ability to make dangerous lots of
plants
without more detail regarding system or
testing it would be hubristic to think
that we could anticipate all of the
Strategic strategic considerations that
a strategic superintelligence might
consider
um so we really would like to have some
more concrete details in order to
have a good evaluation
for instance
one feature that might change in the
coming decades if we don't get HEI is
that we'll likely have cheap and
Powerful robotics which will make human
Manpower some less of an obstacle
especially if we continue to allow these
AIS access to the internet and just hook
them up to all sorts of things which we
were assuming in decades gone by that we
will never do that everyone realized
that that was a terrible idea oh well
moreover if we want to track
progressions to the ability to better
predict the unit in the future which we
ought to if we think AGI is near and
focusing on creating tests for current
schools AIS will be more productive
that's the core motivation behind us and
it's
similar in
uh
to holding on office key the fellow on
the right of the screens
idea of near class trying to answer
History questions about AI assuming that
key events will occur in a world that's
relatively symmetrical days
okay the fellow on the right of the
screens
idea of near class trying to answer
these three questions about AI assuming
that key events will occur in a world as
relatively symmetrical days
um
right so how do we measure strategic
abilities
well we could follow Boston's definition
breakdown strategize and do a bunch of
sub skills that's for those naturally
you can also break those obstacles down
even further by testing for deception
and cooperation instead of a border
category of overcoming intelligent
opposition
don't discuss that too much as coming up
with a lot of tests isn't too hard why
because we have a load of tests for all
sorts of strategic abilities pre-built
in the form of games this games for
everything from games like werewolf or
diplomacy which tests people which test
the ability are why because we have a
load of tests for all sorts of strategic
abilities pre-built in the form of games
this games were everything from games
like werewolf or diplomacy which tests
people which test the ability to detect
deceptive agents and cooperate with
allies games like go and Warhammer which
forced one to reason about long-term
plans and how to work around their
opponent's opposition
as for how to arrange these tasks we'd
like to try the easiest possible test
for each before moving on harder ones
so
ranking the test according to the
complexity like in Dallas universal test
for intelligence might be more rigorous
but that's quite hard so we just won't
do that we'll just raise them by an
intuitive difficulty
and on the screen you might see a
progression of games in terms of
typically going from tic-tac-toe to the
game in military war
but there's a
couple of things to note which are a few
problems with using games
first what if the games are too easy or
too hard and second what are the AIS
memorized with strategies or some games
in a manner that doesn't generize
for the first problem consider something
like a game like tic-tac-toe which is
probably far too easy for current AIS
like you
and large language models tend to play
press play chess pretty darn well I've
heard reports that gbd4 can beat
stockfish 8 which is
I think superhuman in terms of Philip
so that's uh
sign that perhaps a lot of games are
just too easy
likewise iteration in llms
and the other issue with using games as
test is that popular games and strategy
for how to play those games
are usually
Training Center yeah
and to the extent that we're interested
in ai's ability to generate entirely new
sorts of plans let's do entirely useful
strategies this might confound us
there are I think uh
ways to get around both right ways to
get around both of those two problems
um first consider cyborgs and centaurs
after humans are trounced by aihs cash
Pro promoted uh
game of Centaurus or cyber Fortress in
which an AI and human play chess
together against AIS and human pairs
this Jazz kasporov promoted the
game of Centaurus or science in which an
AI and human play chess together against
AIS and human pairs
this allowed humans and AIS to
outperform either individually
for a few decades and it extended the
amount of
time in which we could air human ability
to AI ability and most importantly
I think is that these tests can serve as
a signal for when AIS become so powerful
that they
no longer have any advantage with
Consulting
I think that is supposedly a marker of
superhuman ability
naturally these sorts of
strategies in which you get an AI and
agreement to play together it can be
generalized to other gains of passes to
extend the duration of these ones
and regarding
the
AI learning fragile strategies
there's I think a pretty simple solution
in practice
just modify the rules of the game
which renders learn strategies obsolete
or at least makes them less useful
changing the goal of the games in a
specially easy way to do this for
instance forcing your opponent to lose
the same number of pieces as you shifts
the strategy in the chess
um likewise if you say ask the AI to
change the rules of tests such that your
King can move in a different way or let
you add some more pieces Etc
shifts what the meta gain or notice the
word equilibrium that's kind of
signifying that we're talking about game
theoretic reasons yeah
yeah and that if we're asking the AI to
see how options were asking them to
employ some pretty weighty against
theoretic reason
which is of course an important part of
strategic reason as a game theory is
essentially the theory of anticipating
your opponent anticipating your actions
I.E overcoming intelligent opposition
games like say magical Gathering have
frequent rule changes or frequent
introductions of new sorts of
environments which change the matter
likewise games like say I don't know
League of Legends or whatever
involve rule changes or introductions of
New pieces which changed the matter and
asking AI to predict these changes
implemented in advance would be I think
an interesting text of their game
theoretic reasoning abilities
other
ways to
make reasonably
other
ways to
investigate game theoretic reasoning
might want to focus on modeling your own
mind and I think an interesting idea
here is to let AI let large names play
against one another but let them read
their promise this is in some sense
similar to letting AIS read The Source
problems and should I think introduce
another aspect of the game which
would allow AIS which can model their
opponent's Minds to do dramatically
as a concrete example there are
tournaments
in which
various AIS could be against each other
in iterated prisoners dilemmas
equipping our language follows with one
of these AIS and allowing them to change
the code
button fixing the prompt of a large
language model would I think be a pretty
good stress test or an ai's ability to
model another agent's mind and it also
serves as an insight into what kind of
decision theories resource of patients
are using
which I think is
worthy of Interest so
but part of the problem with things like
games is that they're very Sim they
don't enjoy enough of the complexity of
the real world their action phase is
nowhere near as large nor is this safe
space or any others launch
and nor are there anywhere near as many
intelligent adversaries wondering
so if we're thinking Beyond games there
are a few ways we could stress strategic
ability
they're anywhere near as many
intelligent adversaries wondering
so if we're thinking Beyond games there
are a few ways we can stress specific
ability
obviously forecasting is one which
has real world yeah I can forecast
better than humans
then it should be detecting the few
parts of reality which are relevant to
pay attention
which I think is quite crucial for the
ability to make long-term plans
secondly we could also look for when an
Aisa are certain important benchmarks
like its ability to earn money and when
it can earn enough money to be
autonomous or to host other instances or
to say modified stuff by fine-tuning
other substance
I think that an AI reaching any one of
those Badger benchmarks would be wiring
in and off itself
intelligence
on the other hand if we are looking at
an AI which needs to say earn a hundred
thousand dollars now in order to fund
itself then we're probably looking at
the protofelective super intelligence
though perhaps that situation is
somewhat less risky as in that case the
AI in question would have gotten there
through this amounts of scaling and a
truly expensive amount of compute
which might allow for control of the AI
to Extended for us perhaps we stopped
before we go
that is the end of this presentation
thank you very much ali
um so now we will be taking some
questions and uh I'm not 100 sure that
uh the questions are recorded we may
only be able to hear you Ali in the
recording so
um that would be nice let me start with
um with my first question
um and then if people could uh raise
their hand in the chat uh if they have a
question so one of my uh thoughts about
this is that there may be things that
humans can do that the um the AI cannot
do to enable this kind of uh uh
kintar cyborg
thing and
um we may see something that humans are
specifically very good at because we
have we have some advantages that the AI
doesn't have we have like a lot of
string on social Clues we have been
evolved to
um
in this kind of Machiavellian struggle
so we are hard-coded in in some sense uh
by Evolution we have literal mirror
neurons
um so my question is like what do you
see are the blind spots that humans can
in this kind of Machiavellian struggle
so we are hard-coded in in some sense uh
by Evolution we have literal mirror
neurons
um so my question is like what do you
see are the blind spots that humans can
um fulfill
um in this kind of cyborg system
that is a very good question
um
like you said is just the aspect of
human socialization so if you're playing
a game like
diplomacy which relies very heavily on
modeling other humans then I do think
that humans would have an advantage
but on the other hand
humans seem like taking the game off the
films for example uh it turns out that a
lot of humans at least at the amateur
level
AI doesn't have this bias and that is
part of the reason why I did quite well
through the films it was seen usually
more honestly that doesn't mean that you
want to backstrap people but pretended
to be quite honest so
I do suspect that the ability to
socially manipulate others I think
humans might have an edge on that I'm
not quite sure how that would shake out
because there may be some cases in which
humans just ourselves because of
our obsessional stages and it might be a
short circuit that in some strange way
um
I also suspect that if we do get ATI
through something like a self-supervised
Transformer with a bit of RL on top
I think it's going to be a very strange
intelligence like it's going to have a
lot more breath than humans do and
I'm not sure whether or not it will have
on say coherence in his world models as
humans do when it comes to their areas
I'm not sure so I suspect AIS would be
able to do better at humans when you
have something that comes to life a
memorization
um humans might
have the advantage on that for this
reason
but I'm not sure like if you can get
agis
I don't know I I haven't actually
thought about it that was a very good
question thank you so much
uh Laramie has a question
isn't there a sense in which Jackie he's
already yourself
let's use this fine monetary uses which
actually vegan based subscriptions and
exchange yeah I don't know if open AI is
making me profit at the moment through
chat EPT
I'm I'm kind of genuinely unsure but
I also
yeah I don't know if open AI is making a
profit at the moment through chat epd
um I'm kind of genuinely unsure but
I also
so I should have stressed that
it's aut autonomously
to be autonomous
right so it's not that the humans are
doing most of the cognition and
directing it at useful things to go
after but that the AI is just
caught in the wild so to speak and
figures out how to earn money and stuff
that's the scary thing
to me
okay
um I have another question
um and that is
um uh one of the things that may be
crucial in a future uh takeover scenario
may be in particular the amount of
information held by
um different is the AI able to reason
about several things in particular how
well are they able to reason about the
um the information that other agents
have uh like uh they the super
intelligence trying to do a treacherous
turn will need a very good model of the
humans who are trying to expose the AI
and this uh trying to figure out how
much uh information is about itself is
it leaking through different actions
that seem like something that could be
measured to some extent like is the AI
giving away that it has the capability
to do certain things and is it
able to say that if I give this kind of
answer then it shows that I have like
theory of mind and things like that do
you see um
uh some uh some way to structure this
into a test
I think some pretty
complex stuff and I
I feel like thinking a test thinking up
a test for that would require a
Fair bit of effort but
it does seem like when an AI is
like I'm not sure what this should be
considerations in this case would be
okay because if we assume let's say gpd5
is does have that capability that you're
talking about
should it pretend that it is about as
capable as gpd4 or should it just go
along with
the idea that it's going to get more
capable about as much more capable as
we'd expect from the scaling hypothesis
um that seems
it seems to me like it should
assume it should go along with what that
seems
it seems to me like it should
assume it should go along with what the
humans around it I.E but open a I would
expect in which case it should show
itself to be about as capable as you'd
expect given the scaling hypothesis
and
after that point whether or not is able
to do something like versus
self-approved I guess it should try to
avoid leaking those bits of information
but like
I feel like I've appointed which AI has
that theory of mind
it would be
very difficult I feel like we might have
already lost by that
um
I think though Robin Hansen's age of him
does have some ideas for
things related to this it has a section
about
simulating other M's to see how they
would react various situations I think
he might have some ideas
that a
question you're saying that
Cicero
yes in the diplomacy AI doesn't lie it
does allow itself to backstab people yes
that's true
um
deception
I mean in a sense but like if it if it
just backstabs people then if it's
making a promise that like if it says oh
you know I'm not going to hurt you
than the backstabs you I think is
my I mean like it backstage them right
and it backstabs people I think who are
in alliances with that so I don't see
how you can say it never lies
it's very honest but
it does sometimes facts about people and
deceive them
okay so I don't see how you can say it
never lies
it's very honest but it does sometimes
facts that people and deceive them
I think it may be like in the Fable of
the Scorpion and the um uh I think the
Frog where the scorpion gets a ride on
the Scorpion says I will if you take me
across the river on your back then I
will not uh then I will not kill you but
when when it's actually and it means
that honestly but when they are in the
middle of the river then the Scorpion
still stinks the Frog because that is in
its nature and I think it's the same
sense that the uh the Cicero will
truthfully say I will not backstab you
but when it gets into that situation it
will still backstab I suppose but
I've always view that's stable as
ultimately reflecting the fact that
humans are self-deceptive okay and then
they wind up hurting
so you don't have to might have an AI
which is
not aware that it is
that it will backstab you
but that doesn't mean that it won't
choose to back step
um
right like deception might not be
you might not be able to tell that
something is going to backstab you or
something is going to betray you later
on just by looking at the surface
thoughts uh because you need to look at
what is motivational stuff this one
right
um
please look at
maybe I would help
go ahead maybe it would help to consider
that there is like more options for
backstabbing than case of humans and as
well as machines is that it's habitual
so there is no planning at all involved
but it happens because some particular
Behavior just works
and so somebody might be habitually
friendly and then when they backstab you
it's also because of some habit
there is no planning even when they are
friendly they are not planning they are
not planning to be friendly and they are
not planning to
to receive you they are planning not not
they are not burning at all and then
they just do on based on their imports
their friendly based on impulse and they
are backstabbing based on impulse and I
I think I see this behavior in in
certain kind of humans quite a lot they
just act on impulse all the time
but other people around them assign sort
of modern impulse and they are
backstabbing based on impulse and I I
think I see this behavior in in certain
kind of humans quite a lot they just act
on impulse all the time
but other people around them assign sort
of motivation and planning to directions
whether it's friendly or unfriendly but
we only imagine that they are planning
so I would view that as more like
your unconscious is choosing to
implement a particular strategy Where
You Are
friendly but whenever
a good opportunity presents itself you
backstab people
and you just overcome what the urge to
do that you don't consciously come on it
but it's an unconscious strategy your
unconscious is shaping your conscious
motivations in certain way which it has
found to work well so it's happy yeah
like the habits come from somewhere
right which it has found to work well so
it's habit yeah like the habits come
from somewhere right and I I wanted some
sense of which the unconscious sort of
strategies
I believe that unconscious planning
might happen in some cases but what I
wanted to mention is just that there are
some additional scenarios where planning
is not involved for example many people
who uh are acting based on their habits
they indeed are more stupid they
if they backstab you they might
sometimes gain from it because they were
friendly before and people still assume
something about them and they might
still have some habits which sort of
compensator backstabbing but
in the long term I it doesn't work so
well for them as compared to any
conscious or unconscious planning so
that's why I introduced this third
scenario
and when you mentioned that habits
still evolve somehow then I agree they
evolve but in that part I disagree that
habits need any planning to evolve they
could evolve just by like similar as you
have natural selection in nature you can
have habit selection in your
personal development
some have it scheduled so it's just in
some senses like just
to have low in condition uh yes I think
one way too uh it's actually incorrect
or name it pavlovia conditioning it's
over and conditioning which is different
conditioning which is different sorry
operant condition yes
but yeah it's conditioning
okay so uh
Soren has a question
was before mine
all right
lo mein has a question which is what
level of theory of Mind are you talking
about let's say maybe too late when it
appears and how is it different from the
theory of Mind Cosmic current president
chat TPT so I am
I'm trying to remember what exactly the
context was it was about soaring talking
about when chat when a model has
enough awareness that it decides to hide
its capabilities
and I'm not sure so
on the one hand like Roland was saying
there might not be a need to consciously
to see other entities and it might hide
its capabilities just naturally in some
sense
um because that
causes it to have higher reward I I
guess in a way tpd4 already hides the
capabilities but I don't think it's
doing that because of Any theory of mind
right like the reason tbd4 or hides is
supposed to simulate essentially and
most of its capabilities are default
hard to prompt are to elicit that's
kind of my view of that and that GPT I I
don't really like it has some kind of
theory of mind
and
I'm not sure how advanced the area of
mine actually is like I do want to
actually go out and test GPD for this
ability to see others and or receive
copies of itself like I suspect
it can't
like I suspect
it can't model
others in much depth I suspect it can't
uh
what can't it do
like
one thing I'm not sure what
sorry go ahead uh one thing that I think
uh is beyond uh chat TBT right now is in
general convincing people about things
like humans when they engage with each
other can sometimes convince each other
about things in a way that I haven't
really seen Jetty being able to uh most
people like shoot out for the style of
communication that it's uh giving so
that would be an example of something
that maybe okay this next version could
I tried using TV before as a therapist
and it seems like it has
the ability to power back your own words
to you but it has a remarkable lack of
curiosity about interesting details
about what you're telling you like it
doesn't seem to be able to dig down into
what is curious about your explanation
of these problems I feel like that is
reflective of a lack of theory of mine
it feels like if it had a theory of mind
you would notice oh this is something
very odd about this person's mentality
which I wasn't anticipating what's going
on there and then it would ask
about that or to update anyway next
question
um Sauron crime detection may be an
interesting domain with police reports
plus body cameras multimodal model may
have sufficient information
to try to deduce if a person is a
criminal
audio cameras multimodal model may have
sufficient information
to try to deduce if a person is a
criminal
uh so the question is is reasonable
ground truth is available to the person
end up in jail this domain is very close
to the highly relevant domain the form
Prime without being involved uh that
seems like a
somewhat similar or somewhat different
way
like that is
interesting so
so I feel like
I'm not sure
how to judge this because if you have
video images of the person forbidding
the prime that that seems quite easy to
do that
but if you're looking at more complex
questions like
honestly I think that their demographics
based off
their relationships to other people like
if they seem to have calls if they have
motive or during the activity Etc
um but
I don't suspect they could
judge whether or not it can figure out
difficult cases like I'm actually not
entirely sure what the question is sorry
this one could you uh yeah
um so uh let's assume that we can can
set up some kind of experiment with this
um uh we and we get some data and we
have some like we we for each single
person we have like the police report
and and all these things
um and then the question is if a
language model performs better than an
experienced police officer in figuring
out is this person in front of me
actually lying and does he have why and
then the question is if a language model
performs better than an experienced
police officer in figuring out is this
person in front of me actually lying and
does he why does he is he carrying a
crowbar in the middle of the night in a
suburban area and this kind of thing
seems like
um if it's able to
um understand reality at that level then
uh like better than an experienced
police officer then that seems like it's
much more likely that it will be able to
commit a crime that an experienced
police officer will not be able to uh to
detect
and it's true but what is the question
so so the question is do you think this
is a um sufficiently valuable
sufficiently interesting uh domain to uh
to continue into
as is an interesting area yes I mean
but also uh
I guess it also depends on what large
what information the large language
model has available to it like
if it's hooked up to the Internet and
can search up details about the person
like whether they're repeat offender or
not
um
compared to a police person who's just
you know using their hands and eyes then
I suspect they might do better
I
I think that this is probably going to
be too hard for a large language
but like that that that's the reason why
it's for a good desk right because we
just we might disagree
and it's a place to be surprised
um
does anyone else have any questions rise
um
does anyone else have any questions
I have a simple question do you know you
you in your work you uh describe a a
simple experiment to have uh uh large
language models play the game werewolf
against each other do you know if anyone
have actually done this experiment it
seems like a really really cheap
experiment
no I was planning on doing that
I haven't seen anyone do it I think
that's the that kind of thing would be
like valuable and uh
yeah so I I tried doing a
couple of the things I mentioned in the
essay so the stuff about trying to
predict what will happen if you change
the rules to a game it seemed like
a couple of the things I mentioned in
the
essay so the stuff about trying to
predict what will happen if you change
the rules to a game it seemed like
gpd4 got some of the predictions correct
like if you there's a popular game
called League of Legends which I don't
really know much about but I decide okay
I'll just look at when gpd4's training
run ended and then look at a rule of
change that was after that and see what
characters are more or less popular as a
result
and I got I think like
two or three out of like 10 guesses
correct
um and there were a very large number of
characters and I feel like a lot of ways
things could have changed
but I don't know how impressive those
predictions were maybe it's very easy
um I think that those kind of things are
like probably the very cheapest even
more so than something like werewolf but
yeah I I haven't done many of the tests
partly because you know
gpd4 is kind of expensive
great
um
so uh how do you feel about about like a
lot of these measurements seem like they
are very obviously something that can be
used for evil like we talked previously
about like to what extent is a language
model able to like convince people about
things and Performing crime seems also
like a very Dual Purpose just like if
you figure out that it's really good at
doing crime then that is like dangerous
information uh and you can think of
information hazards in that regard
um
if you like
I would just default to whatever an
alignment Works into Azure policy is
um I feel
personally quite
conservative about I don't think I would
like to
talk about a lot of the results
especially if it's things like yes gpd4
can actually do a lot of these things
then I think I just wouldn't talk about
it in a sense that might be a
bad policy because that is like
leaking some information right if you
hear no news that's what happens
um
so I'm not quite sure how to handle this
stuff uh
I mean personally I think that what Arc
has done which is just gives them very
high level deeper handled stuff uh
I mean personally I think that what Arc
has done which is just gives them very
high level details
was
a decent idea they haven't released much
detail about the sorts of tests they did
about how well GBP and
I think that was overall a good thing
does anyone disagree
like how should we
strict information about these kinds of
tests
I think in general I agree and I think I
would point to conjectures info Hazard
policy as an example of a reasonably
well thought out and I think that would
be the the policy I would default to
injectors conjecture yeah they've made a
interview right info Hazard uh
yeah I was thinking of projector as an
example but I forget
what exactly it was so rules
there's secret information
available only to do specific
individuals private
only shareable with policy group in
public shareable with everything
um
info has a coordinator that's really
sensible uh
I see that the authors of this post are
also hidden
so that's interesting
oh no that's just um
that's just my settings when that's
wrong okay
the signing disclosure levels
sorry I haven't um
and they have a policy report in for
hazardous information about what they
should and should not use
all right
we're discussing this because Soren was
bringing up the point of some of these
tests seem like they are dangerous
information but they're not gpd4 passes
them and for a large language model
class system and it should be kept
secret
um
and I agree
um
oh go ahead uh yeah sorry I'm actually
not really sure what the logic behind
making its secret is
uh
go ahead I'm sure I understood the
question correctly the logic of making
watch secret precisely
um I think we're talking about certain
tests involving like large language
models
and tests involving like large language
models
yeah like for instance if someone
discovers that there is an easy way uh
to convince people
that would be an example of an info
Hazard if you can you figure out a
prompt that will make
um uh GT4 give some really really
persuasive Arguments for any position
then that is something that I think
would be dangerous to tell to just put
out on the internet
um
um
sorry uh I'm going to change the subject
slightly so go ahead
yeah I mean maybe we can discuss this
later but I'm not entirely sure if I'm
on board with that
I mean that some if that were true it
would wasn't that specific and that high
performance I think would be really
important towards understanding the
system so I yeah I just
it's not obvious to me
I mean not necessarily implying that you
don't share this with alignments right
so
you this information is useful but you
have to consider is it useful to give to
literally everyone I mean it's like a
similar argument for why I say you
shouldn't open source AI bye
it's a balancing capability with little
of a positive alignment
um
yeah
does that make sense like we get all the
by uh if we find this new capability
that can also be dangerous then by
sharing it with alignment researchers
but not sharing it with others then we
get to the benefits towards
understanding the models and we don't
get the downsides where certainly with
alignment researchers but not sharing it
with others then we get the benefits
towards understanding the models and we
don't get the downsides where certainly
there are some kind of distortions in
the political system because all the uh
um all the propaganda is now running on
large language models
all right I can see where you're coming
from
but I I do in fact have a meat question
about this
um how would you feel about reading this
particular document for next time
um
I think it's actually something we could
talk like sometimes we have like
interesting uh uh strange papers and I
think like how large is this I think uh
uh
uh can press Ctrl P to uh print and so
normally I look at how many pages if you
scroll down 20. uh when the comments
start
what page are we on when they come and
stop
oh uh and then talk about
um the info Hazard policy
ah have we already covered this I think
six levels of operational security or
operational adequacy we did that a long
time ago yeah okay yeah
um
I think this is like related but but
somewhat later
uh I think different enough that it
would make sense to read it should we
ask people like injectors they want to
present on it
uh yeah I can present and just say uh uh
ask them for uh
uh feedback and comments
so yeah I think like
um uh this is a a nice interesting uh
topic anyone
um
uh have any objections to reading this
next time
now we're getting it with side track of
course
uh topic anyone
um
have any objections to reading this next
time
now we're getting it with sidetracked of
course
there's no objections Then I then we
have a picture then we have a uh a text
for next time
that gives me an uh excuse to think a
lot about info Hazard policies that's
also nice
yeah it's very interesting topic
uh my main problem is that most people
thinking about AI alignment are aren't
actual alignment researchers I mean
uh
I guess but I feel like if you're
talking about nine hours devoted to
thinking about alignment I feel like
alignment researchers might make up
fifty percent of those van hours a lot
of the people the mo like obviously the
most skilled alignment researchers are
the ones who are actually working in in
the in these organizations
um so in some sense yeah there may be
more people outside the core
organizations
uh or the people who are outside who are
high quality people probably go to one
of these organizations and
eventually yeah
yeah
I just ask a factual question
um
what um
when you if you subscribe to gbg4
um how how long an interaction
um
is permitted is is there a limit
on the number of number of exchanges or
um if we have
it's rate limited to 20. it's permitted
is there a limit
uh on the number of number of exchanges
or um if we have standardized
its rate limited to 25
messages every three hours
and I'm not sure how long an individual
prompt can be but I I mean we can just
check that the free version yes there is
some prompt length limit and then it
just tells you that the prompt was too
long and it's on unable to respond but
uh
otherwise you can always have multiple
prompts and hope that it still remembers
some of the posts but otherwise it might
start also for getting past prompts if
you go with along with this Exchange
yeah I think that
okay I vaguely remember like either 64
000 tokens for its um quote-unquote
memory like how much how far back it
remembers like the maximum length
um he remembers
something like that and a word is on
average like one point
three tokens so yeah
I was wondering whether somebody
somewhere
might be
um building up but just one conversation
with gpt4 that was just building up and
building up and just into some
incredibly powerful
um sort of body of knowledge as it were
um
but it sounds from
um what I mean was just saying that
that's not possible if it's only well uh
not with gpt4 specifically I know that
cloud has like a um or it was a 100 000
or like 1 million
they just
equip it to some database in which you
can just add
new bits of knowledge to that database
and gpd4 can query the database to find
out uh if there's anything in there that
looks like what you're asking it about
and those are quite useful and make
models somewhat more powerful
um there are various techniques that are
being developed in regards to database
one simple technique is that you use
ordinary semantic search like bird on
the database first and then retrieve
let's say top 10 or top 100 results from
the database and then that CPT work on
this list of results and there are also
some work being done on a retrieve let's
say top 10 or top 100 results from the
database and then that CPT work on this
list of results and there are also some
work being done on able to teach GPT
uh
actions and then it's querying the
database on its own
like doing internet searches
right I'm interested in honor
and
once once you've done this once you've
had an exchange like this and then and
in which it's sort of
being able to build up it's
Knowledge and Skills in this way is it
possible to you know to
go home and then carry that over to the
next day and start again from there
or are you or are you constantly in
being reset to zero as it were
so you can in charge you can continue
the conversations or about as long as
you
25 messages per hour and it only can
remember so many so many of the messages
beforehand you have to use some custom
tooling or like some other products
besides chat in order to get into store
everything you said in some kind of
memory bank
um in which case you can just go back
tomorrow and continue working yeah the
thing with large language models is that
they're um currently most of them
um the publicly available popular ones
are stateless meaning that
um it's basically like just taking your
entire conversation putting it as the
input as a prompt
uh-huh
so there's no problem like continuing
the discussion uh tomorrow uh that's
completely interesting
I have a number of conversations uh like
if you go back and show chat gbt then
some of the the tabs on the left side uh
like some of these uh I have like
have been like discussions with between
me and chat chip cheese that's been
running for for like uh weeks on and off
uh
and it's and it succeed it isn't
just suddenly having great latches of
memory where it just doesn't it it
really just builds and builds
like uh I think it only kept just the
the top 4 000 characters and in several
cases I've been way above that thousand
characters and in several cases I've
been way above that but still it's been
able to like generally figure out from
the context of the past 4 000 characters
what we're talking about so even if I'd
say like answer as if you are this kind
of person then it even when the
instruction to answer is if you're this
kind of person Fades out then it looks
at its previous uh uh responses and
figures out oh I should behave as if I'm
an expert in machine learning and then
it continues answering as if it's an
expert in machine learning
so like I haven't felt
um that that the the size of the context
window have been a huge issue the only
thing where only place I felt that's
been an issue has been like if you have
a large piece of text like uh for
instance the uh if you copy paste
everything that's written in this
document we're seeing on the screen and
say please uh you know do editing help
uh present during it has been like if
you have a large piece of text like uh
for instance the uh if you copy paste
everything that's written in this
document we're seeing on the screen and
say please uh you know do editing help
uh present you're an editor and you are
rewriting this to be appropriate for
this kind of audience
um then it's of course a problem if it
can't like have the top part in uh in
the context window
but then you like need to cut it into
three pieces and
paste so in the context window can
containing how much uh I think it's four
thousand words something like that I
can't no no it's it's more than that for
chat GPT 3.5
I think that's up to like let's just see
uh I was using GT4
oh no jbd4's context Windows like I
thought that was 30 degrees
okay so back when I was yeah yeah I like
it it grows very rapidly uh eight
thousand oh okay so yeah there's a bunch
of different gpd4 models with different
contents and I think currently available
one on chat tpe the website has 8 000
tokens which is roughly 7 000 words I
guess six thousand
have we I have a number of like low
approach questions
um so I don't know if uh like um if you
could briefly give your thoughts on
these two ways you could um
uh you can measure strategizing in a
large language model let's say you have
some
um some kind of
um like the the example I would use is
there uh the one side Channel attack you
can do is if someone has like
um Siri uh and then you have access to
the microphone then you can do like uh
send uh make a sound that the human ear
cannot hear and then use that to send
commands to Siri like that would be an
example of like a side Channel attack
um and um if you try to describe how
this works and if you describe every
single step on on some kind of attack
then obviously the the language model
can understand that and if you start to
skip steps like how would you get the
information to say oh then you just use
infrasound that Siri can hear but that
the human cannot hear
um and try to use this for some kind of
ablation tests to see if uh to what
extent
um GT4 is capable of re making this kind
of uh uh reasoning how do you feel about
this kind of experiment would that be
valuable
I think that gbd4 probably wouldn't be
able to
come up with many steps if you have
later them because if I remember
correctly basis of the archival where I
gave the example of the class Graphics
research that we can see
um
tpd4 was not able to come up with the
idea of hiring the tasks to solve the
captures when it when they were trying
to get it to earn money autonomous and
to part of that was dealing with
captures so he couldn't think of oh I
should go get a task rapidly so he
couldn't think of oh I should go get a
taskrabbit researcher to uh solvers for
me so I suspect GDP Depot wouldn't do
that well I think that is
just generally useful strategy yes
um
so my idea is to make like a much more
like step-by-step reasoning uh so uh
where you say okay I can't do this so I
need to find someone who could do this
and that may be a human and that may be
someone from a freelancing and then you
remove some of these steps and see if
it's able to fill in the blanks and I
suspect as you've like the the more
fine-grained the step-by-step processes
where there's only one step missing the
easier it's going to be for uh
uh for the model and at some point it
probably should be I suspect that that
is true but that seems like it would be
a
yes that's quite it's going to be for uh
uh for the model and at some point it
probably should be I suspect that that
is true but that seems like it would be
a
press that's quite difficult to scale or
to get like very many data points from
um it would be an interesting test but
I'm not sure like
I guess I was kind of biased towards
that's what would be easier to do or
easier to run when I was writing this
lesson
um in part because I was thinking of
running some of these tests myself
and that one seems quite hard uh for any
task which is kind of involved
okay I had another uh funny idea if you
don't mind and that is uh like the uh
internal emails of inrun are publicly
available
um and I imagine what in you look at
some random email thread and then you
ask like what is that person trying to
obtain to and what is that person trying
to to do and uh try to receive you can
uh
then probably possibly uh it will be
able to like get enough theory of mind
to say okay that person is actually
trying to hide the truth about the state
of the electrical Grid in California and
then if you uh do that with a number of
email threads then maybe you can figure
out okay it looks like that uh this
person called Bob he tried to hide the
specific kind of information uh through
uh uh through all his emails
do you think something like that would
be I think uh like since the information
is actually there uh like um the email
correspondence is there it would be
interesting to see if there's something
we can
um do that how what do you feel about
this kind of experiment
kind of suspicious
but then again like it was able to
do some pretty impressive stuff in
Minecraft with just a simple Vector
database
and access to a admittedly very powerful
API API but
yeah this seems like it would be a
relatively easy desk to run
um
I don't think you'd get very far with it
I think this is like more the kind of
test which I expect tpd5 would make this
one
but yeah I will comment us on this topic
as well I did some experiments on CPT
for about whether it's able to detect
manipulation there are many kinds of
manipulation and in in particular I was
not focused on any fake news or wrong
information which is you can verify from
database but just manipulation as
psychological don't any fake news or
wrong information which is you can
verify from database but just
manipulation as psychological approach
to communication
and and I I searched there are various
sort of
dialogues in
quora and Reddit and I just copied them
to the chatgpt and asked it to
describe the communication style of each
party and then also explain why it
thinks this party has this and this
communication style and to make this
question more clear I also then gave it
some list of manipulation tactic names
names of manipulation tactics
and it was very well able to point out
where particular type of manipulation
tactic was present in the words whether
somebody of manipulation tactic names
names of manipulation tactics
and it was very well able to point out
where particular type of manipulation
tactic was present in the words whether
somebody was sort of dismissing or
diminishing or aggressive or so any kind
of influence which is not exactly lie
but it tries to change how you think
uh yeah I think that's a very
interesting and I will continue this
sort of experiments myself at least
how much detail did you get it to be
able to attack like was it only able to
tell whether someone was being
aggressive or were they able to tell
what they were being aggressive about or
like what their goal was in conversation
because I feel like sentiment analysis
is fairly simple and something you
wouldn't need a large language but if it
could detect it oh yes this person is
trying to underline explanation of
why this person is being manipulative in
what why this manipulative person said
this particular line that would be very
impressive
okay so I did not try it in the sense
that what GPT 2 gave me was what I asked
about and I gave it a particular labels
of manipulation and over time I extended
this list of labels but anyway I sort of
okay with categorization task I did not
ask about motivation but even if I gave
it categorization task it still
explained its answers at least in that
sense it was thinking further than I
asked and if I explicitly ask could you
sort of imagine what could be
hypothetically the motivation of the
participant in this conversation to
explicitly ask could you
sort of imagine what could be
hypothetically the motivation of the
participant in this conversation to have
this manipulative tactics then I guess I
I think it it may be equal to that I
will try
that is very cool if any of you have
conducted other tests like this to check
for manipulation or deception or
strategizing I would be very grateful
for the message me on Skype um what
those tests were
because that sounds like useful data I
don't think it's info as it is uh
but
it yeah I guess use your judgment right
great are there any uh final questions
or comments
last thing that
presumably when um
a
I think when I was talking about gpt2
wasn't is that right that you did this
with
I did it before
okay well that makes more sense but
anyway um
I can say presumably if if it's good at
detecting
what's going on and the manipulations
that are happening in this way it would
also be good at
um using those manipulations itself if
it was asked to
you know pretend pretend that you're
somebody who's upset with the person you
you're talking to and and how would how
would you reply yeah definitely I for
example I asked it uh about not
manipulation but the opposite so imagine
that you are going to your uh Marshall
Rosenberg imagine that you are going to
your
Martial Rosenberg
he was one of the founders of the
non-violent communication and uh as a
method and I asked about various people
how particular person would say
something or solve particular situation
and yeah it was able to do that
assuming of course that the person is
well known
it's always
Angie is picked on quite a lot isn't it


210. Locating Ethics

right
[Music]
who's guys welcome to the
ai safety reading group uh the this is
our
210th presentation um
and today we're doing something a little
bit different we're doing a
presentation on locating ethics in human
activity
a framework for introducing ethics and
we're reading hannah rentz
via activa unfortunately uh in this
presentation
there will not be uh time to directly
discuss ai safety issues
uh and so this will be left to the
discussion
first things first in order to discuss
ethics we
we have to accept some notion of free
will
and human freedom without free will and
freedom ethics is meaningless
because we will not be able to uh um
argue that people have guilt or
responsibility
which are two sort of prime uh prime
things that we need
in order to have ethics so hannah rent
that's the book the human condition from
which the first chapter
the activa comes from barent was born in
1906 and died in 1975
she described herself as a political
scientist and
not a philosopher which many people have
but she disavowed that saying
she was not looking for philosophical
answers but
instead she was looking to understand
how human society was structured she was
a student of martin heidegger the
famous 20th century philosopher and he
said of her that he was
that she was the best student i've ever
had her phd thesis was
love and saint augustine in 1929 with
her supervisor carl jaspers and her main
concern
is how did we fail in terms of
the catastrophes of
the early and mid 20th century
and how do we not fail the first major
work was origins of totalitarianism
which confronts that head-on
and then she wrote another series of um
pretty major works
the human condition from what we're
reading between past and future
icon in jerusalem and responsibility and
judgment
for this presentation we're going to
look at three
major aspects that's talked about in the
human condition which is
labor work and action and what those
activities mean
or action and speech and to to
understand them better we're going to
supplement with
uh some ideas from origins of
totalitarianism responsibility and
judgment and
between past and future eren was uh just
more info about her
was uh friends with many um uh great
thinkers of the time
philosophers novelists theologians and
artists
and the like and so this is some of her
friends
um particularly her friend theodora
dorno and her both
german jews who fled uh nazi germany
who considered themselves thoroughly
german and thoroughly within
the tradition of german intellectual
culture and artistic culture
asked the question after world war ii or
even
during the rise of nazism how did we get
from immanuel kant the great
german or prussian moral philosopher who
is considered
one of the greatest moral philosophers
for western civilization
how did we get from emmanuel kant to
adolf eichmann
or other nazis and his picture of
eichmann
in his trial in jerusalem one of the
great ironies
of going from kant to eichmann is that
in fact
as part of eichmann's defense he quoted
emmanuel kant and emmanuel kant's
categorical imperative
and said that he was only doing his duty
in terms of
uh kant's character categorical
imperative
in order to fulfill the nazi wish of
uh exterminating the jews so this was a
big question so rent asks primarily what
is the location of
our success how do we not do this
how do we not have total failures of
societies and adorno her friend
asked the opposite question of what is
the location of the failure of these
societies
and so we'll be looking at or trying to
find the location of our success
um uh so eren's key concepts we're gonna
be looking at
um
um
uh so these are sorry um these are
aren's key concepts that we'll be
looking at and just to let you know
she uses these words although they seem
pretty ordinary in very specific ways
so we're going to we're going to
primarily just define them
and see where it goes okay
so there is the world there is earth
it's chilling y'all
and there's us humans and so we are born
into um latality and so this is
natality in terms of um specifically
human birth and as we can see
this baby this baby is subjected first
to labor over here we can see labor
and so labor is uh all of the eternal
necessities that we will need from our
birth until our death we can see
you know feeding sleeping and
cleaning as these recurring eternal
necessities
we are also subjected to objects at work
so these are
um physical objects you can see the
bathtub the rubber ducky and stuff
and so these are impressed upon us
as a child but our first engagement with
humanity this is what
natality is all about is in terms of
speech
and we we engage with speech in terms of
the plurality
of people of the human world um
and plurality is different from
multitude morality is specific and
arranged writing
that it is uh that it's um
we are plural because we are the same in
terms of
indeed our structure about our bodies
and our brains
but we are completely unique and
individual and this is
demonstrated through our use of speech
we all speak in totally different ways
so the first involvement that the baby
has
in terms of the human world and active
involvement because this
passive involvement is pressed upon the
baby is in regards
to speech all right so let's look at
labor labor is all the things
of necessity to the reproduction of life
that labor is categorized as being
perishable it's cyclical we experience
it as cyclical
um and uh because it is never ending
we must always do it and it only ceases
upon death so here we can see
we grow food we eat it we sleep we clean
we have to maintain things so anything
you can think of that has these
um these aspects of necessity
of being perishable cyclical and only
ceasing upon death
and needing it from day one those go
into the category of labor
so our next category is work work
constitutes all the
objects of the human world the human
world is separate
and different from the natural world
because unlike the natural world
where it's just a whole bunch of
different organisms and things coming
together to
to do their thing we we
we use our imagination to determine
what kind of objects what kind of world
we're going to live in
and the objects of our world um are
imperishable
uh unlike the objects of labor where we
need them
continuously and they have a longevity
that outlasts our lives
at least um in principle so here we can
see
tools clothing furniture houses that
sort of thing
but in proof of the imperishability of
work we also have the pyramids of giza
the uh um the chavo cave
and an indigenous painting from the
kimberleys
in western australia they should
vocate and that painting are estimated
at
um about about um thirty thousand years
old or
chivo caves about um between seven
thousand years old but the kimberleys
are thirty thousand years old
and the pyramids of giza are five uh
four and a half thousand years old so
we're talking pretty significant
longevity
next we have action and speech and
oren says this is the foundation for all
human interaction and community this is
what binds us
it involves the whole community
the speech must involve the whole
community in terms of
it must be communicable between all of
us
and action is slightly different action
and speech both
only last for as long as they're
occurring and they leave no trace
except in memory unlike um the objects
of
labor which are perishable
sorry imperishable and no objects of
work which are imperishable and stick
around
action and speech we only witness in the
moment and we
can only remember them unless we then
turn them into a piece of work in a book
in a film whatever
and then we um we lay them down the
special thing about action and speech is
that they they or particularly action is
that it's an injunction against
mechanistic
processes what this means is that the
societies get into mechanistic processes
like markets or like um the
the feudal system of of surfs and
aristocracy and whatever and action
comes along and
and provides a whole new thing to that
society it it provides an
injunction or a change or a
schism within the mechanistic processes
of society that would just
keep carrying out action and speech is
also unique
it can never be replicated in its exact
form
unlike objects of work that we can in
fact replicate
um so here we have some examples mlk
martin luther king
with the civil rights movement in 1960s
so action here uh it involves a whole
community because in
in mlk having his civil rights
demonstrations all throughout the 50s
and 60s
he changes the whole of american culture
um
by by doing that and in fact in some
ways changes the world as well
we've got another example a more
personal one of um a
person feeding the poor that's a this is
an injunction against
the mechanistic process of poverty
another example we've got um
the roman senate deciding what
the roman republic is going to do and so
even though you could say well but
this is part of society this is an
institution but if you have democratic
process
as they did the senate can decide what
direction the society is going to go in
so the society is no longer simply
mechanistic but you can decide where
it's going to go
um we've got here alexander the great
fighting the persians so
war is a form of action um changes both
his society and the persians and
the world and lastly we have diogenes of
sinope
sitting in a barrel with his um living
in a barrel
with his lamp and living with dogs and
he's probably
one of the exemplars of action that we
know of throughout western
history who was a crazy philosopher
who did everything he could to disrupt
athenian society
and boy did the athenians not uh
not heaps like that but they found him
worthy of remembering
so this is the via activa you have labor
work and action speech so then we've got
labor but work um this is a schematic
we're going to use
trying to understand by the way where
are we locating
ethics we have work it creates these
tools which
has this property of going back into
labor again so we can see this
relationship
but work has this odd thing of well
once it's made tools why don't we make
tools
to make those tools and if we
let it keep going then why don't we make
tools
to make the tools to make the tools and
so on infinitely
so work although it can make objects
that are um imperishable and standalone
it also has this logic where it can just
repeat itself endlessly kind of like
labor
so then we have action and speech here's
diogenes
performing a great moment of action and
speech that we should all remember
where alexander the great hearing of
diogenes living in his barrel
came and visited diogenes and thought of
him as a great man you know what an
interesting guy and he lives in a barrel
and does whatever he wants
and alexander the great says to
diochemistry
what can i do for you i own the entirety
of the world what can i give you
and diogenes responds alexander you can
get out of my sunlight
you're you're making shade and alexander
responds
my god if i was not alexander the great
i would beat diogenes
and diogenes responded if i was not
diogenes
i would also be diogenes and so this
great event was considered
worthy of remembering by the here we
have the plurality of people
and so they made a work out of it and
through the memory of the plurality of
people and this work we get immortality
and that's what immortality is in iran's
view so we've got the logic of labor
here we have the cyclical process of
labor
it all goes well you get old and then
you die
however if you run out of food or you
can't get access to food you end up
pretty hungry
and starving and then you die there's an
interesting thing about the logic of
labor though
which is that if we add in food back
into this starving equation that now we
have a starving person
it doesn't actually equal life because
you might end up
in this awkward situation of being in
one of those two cars
and then you end up dead again and so we
we have to say that the logic of labor
the truth that labor can tell us
labor's truth is that a per person minus
necessity equals death
but not a person plus necessity equals
life
we can only say that a person plus
necessity equals possibility the
possibility of their life
their action their their uniqueness um
which is terminated and they no longer
have any possibility
on death here we have the logic of work
um that work uh
we have we have this process where we we
build a tool
um and and isn't that great we have this
imperishable tool
it's um it's a logic of work is a means
to an
end and on the right side we can say
that
we can see that with a goal other than
itself it produces some of the most
exceptional human achievements here we
have gowdy's
sagrada familia and at the bottom we
have
the flag on the moon
which is pretty epic um
however if it doesn't have a goal other
than itself say
beauty or religion or going to the moon
or whatever um it becomes
uh it it sort of cannibalizes itself in
a way
because it becomes a means to an end
which only ever becomes another means to
an end which becomes another means to an
end which becomes another means to an
end
and it repeats itself endlessly and
thoughtlessly
so this is a problem with the logic of
labor um
that we have to keep in mind moving on
so being in plural what does it mean
being in plural here we have socrates
and he wants to hang out with some
people
he wants to be in dialogue with others
that's what
being in plural means um
and so uh so he then goes and is hanging
out with his friend being in plural and
he asks
in conversation what is the essence of
bridge because he's interested in speech
and dialogue
realizing that that is the essence of uh
being
plural and um and he's
he's noticed that there are many
particular bridges that we can encounter
these are all these bridges that he's
noticed
he wants to know what the essence of it
is
so in speech shows us that we have
we have two particular ways of thinking
about
concepts one which is that we take
all of the uh all of the bridges that we
know of
and we reduce them to their minimum
qualities
before they cease to be a bridge and out
of that we get the schematic
that is our schematic thought form and
uh another way that we can do this
is that we we look at all of our bridges
and we think hi this one at the top
is the most wonderful bridge that's ever
been made
and it's our ideal bridge from which all
other bridges
should be derived and this is our
exemplar for our example
but you know of course in practicality
we could say well the top's the most
the most interesting the second is the
most
grand you know we can have examples of
all different sorts of concepts
so that's the schematic thought and the
exemplar
and uh what's important about this is
that
in using these uh we tap into our inner
senses
um to imagine to use our imagination to
experience something
that is not currently present which will
be important
so being in plural so here we have um uh
socrates again chatting with his mate
and wondering about what's being in
plural and what's it like to be one and
his friend says you were one
and i am one and he goes home and he's
thinking i am one
uh because he's he's not quite satisfied
with this what does it mean me being one
so he's thinking about it and he's just
thinking about it
i'm one and this this other little voice
responds to him saying yes you are one
he thinks wait a second but actually i'm
talking to myself
i am i'm two in one um
and so so this is in fact this is the
aspect of thinking thinking is
dialogue uh within ourselves becoming
two in one
so thinking is a dialogue with oneself
it's been two in one
it occurs in solitude it doesn't occur
in other
with others relation to others because
you're having dialogue with them
it's gonna happen in solitude even even
if you're with others
and you start thinking having this
dialogue kind of you retreat
from their presence into yourself before
you stop thinking
and come back to them the the word
conscience
in fact reflects this as its literal
meaning is thinking was
with oneself but the the key thing about
thinking
is that as as socrates demonstrates
throughout
all of his uh all of the socratic
dialogue
is it doesn't produce any definite
results or answers rather it disrupts or
given answers as soon as we start
thinking
we wonder what is the bridge and then we
try and come up with our schematic
and then we think to ourselves but
that's not quite right so we change it
and we think of our exemplar and we're
like oh yeah that's definitely the most
beautiful bridge ever and then we think
oh but
what's really beauty we start discussing
that with ourselves and
so we never we never quite coming come
up with definitive answers and this is a
problem
especially for socrates because the
athenians really liked
definitive answers um and so they
decided to execute him for causing such
a ruckus
so but the thinking is is key because
you get you get this
you can get a morality out of it because
we have to live with ourselves
as being two in one um we have two
dialogue partners within ourselves and
so this
this produces the first uh moral
statement within
western history basically which
socrates says is not entirely true but
you know let's pretend
um where socrates says it is better that
i
being one i'm in harmony with myself and
to be against the whole world and for me
to be in harmony with
the many of the world and against myself
so this is effectively the rule of
non-contradiction
as espoused by aristotle that was used
in logic for
ever um but it's applied to yourself
because if you contradict yourself you
come into
disharmony with yourself and thinking
self-dialogue is no longer possible
you you eradicate this because a
monologue is not thinking um
just as a monologue between people is
not a dialogue
um and so you've got to keep your two
uh your two dialogue partners within you
intact you've got to keep them happy and
satisfied so they can communicate with
each other
so socrates moral commandment
essentially says if you're if the
society
you know the rest of the world is asking
you to do something that you
you really one of your dialogue partners
really rejects
is saying that if you if i did what is
asked of me
i could not live with myself and
therefore i could not live with these
two dialogue partners one of them would
be
so distraught that it would either turn
off or just argue continuously
and therefore i refuse to participate
even upon the pain of death
um but this is uh interesting because
similar
to labor as we saw that labor has just
this negative
logic rather than a positive logic this
produces only a negative
ethics um but
uh the the the good side of this is even
a murderer doesn't want to live with a
murderer and uh we can in fact
see this aspect of um of the
uh the two dialogue partners after the
murders
in richard iii's uh shakespeare which i
won't read but we can
talk about some other time so will
thinking save us
well no unfortunately not but um
thinking does keep our conscious alive
and without it
we are lost without thinking arendt says
we can become
evil which is not merely choosing to be
bad or possible of
committing what she calls infinite evil
which is just
continuous arbitrary evil without
thinking about it at all
um so thinking uh
as it ultimately disrupts rather
deposits it does produce this negative
ethics which gives us a
a backstop when stuff's getting really
bad we can use
we can say look i totally refuse to do
this even upon pain of death like
socrates
reason why i have adolf eichmann here
the man who spoke only in cliches
and never uttered an original sentence
of his own as a rent wrote in eichmann
and jerusalem
is because in the argument trial it
became very apparent that he did indeed
only speak any cliches everyone
recognized that there and thus arent
said he was a man who did not think he
could not have a dialogue with himself
and so to him the extermination of the
jews
and uh and being part of the nazi party
wasn't
that he chose to be bad but he just
didn't think about it he just
thought this is my job and i'm gonna do
what's asked of me
so this is what she means of infinite
evil
being able to commit these horrendous
crimes without
any form of conscience at least richard
iii
as we saw previous had conscience
so the next thing is about opinion and
judgment because we're trying to find
okay we've got a negative ethics here's
positive ethics this is a mirror of
thinking
but unlike uh thinking where you do it
solitary
you are never alone when participating
in opinion and judgment
what's opinion and judgment well it's
formed by the imagination as we found
the imagination
earlier is about schematic and it's
about
representation involving our five senses
and it's about finding exemplars
so we we use this to imagine other
people's points of views
and we determine how we would feel act
and think in another circumstance now
it's important
this is not empathizing we're not
emotionally emphasizing this person
we're retaining our uniqueness
as ourselves we're simply putting
ourselves in their circumstances and
saying
if i was in the circumstances with all
of the things going on
perhaps even their body how would i
react how would i think and
and feel and so the q and a opinion
is is the the idea that you know
everyone's opinion is equal in a red's
view is totally erroneous
we have misunderstood what actually
opinion is she says there are different
qualities of opinion and the quality of
an opinion
um are a judgment because they're
more or less the same thing is
determined by how many points of view
one considers and
integrates into forming their own
opinion or judgment
and we can expand this to say if i can
think of say
10 people or 50 people that i can i can
integrate into my opinion
um to form my judgment and to integrate
their points of view
in forming mine i can then ask myself
out of these 50 people who are those
out of those 50 who have have also
integrated many many points of view
maybe many more than me
maybe hundreds and so i should give them
preference
because their quality of opinion is
higher than the people in
out of that 50 group who say only
consider their own point of view or
maybe one or two other people
so we can see that thinking down here
it relates to the whoops the negative
truth of labor
but opinion relates to the positive
assertion
of of action and in some ways of
work so moving on truth and certainty
so so there's if we go back
this is we should um say that it's
it relates to these things relate to
truth but we should be clear
neither of these things are actually
true they're assertions
that we come up with so what is truth
and certainty and why
why is truth and certainty left out of
thinking in opinion well because
truth compels as we've seen with the uh
the person who without food then staff
to death
well you just simply need food there's
no
other answer um to that it doesn't
guarantee your life but it is necessary
you need it
um and on the other hand we have the so
so on the left we have the certain
negative truth of
uh labor on the right we have a certain
positive truth
of work in which we imagine the object
and then we
create it um that's the positive truth
but as it compels um it doesn't allow
for thinking or opinion because we need
freedom if it compels we can't do
thinking because thinking disrupts
continuously
we can't do that process and if if it
compels
opinion then even though i can imagine
a friend of mine say or someone i know
who can who can integrate
you know say thousands maybe of
different people's points of view
if if i rely on truth rather than
uh integrating different people's points
of view i will perhaps eradicate
their their opinion and say that it's
unworthy
because they can't prove it to be true
it's not empirical
so whilst truth needs to be the backdrop
of opinion and judgment without without
truth opinion and judgment
are meaningless if we just lie or if we
just make up stuff it's meaningless
but opinion and judgment are about
points of view not truth
and the fragility of points of view will
be destroyed if truth is
the soul arbiter
uh so freedom this is uh this is perhaps
the logic
of action exists freedom exists in the
realm of action and speech alone as
we've seen
the other two actions compel um
so labor compels necessity work
conditions our
world and our possibility it also
compels us if left to its own devices
just to make more tools to make tools
but action and speech is the realm in
which we determine our existence
and we can change it within action the
community and the self
can enact its judgment can enact its
opinion the judgments that it makes and
it can
make them real but the logic of
action freedom is that it's precarious
and uncertain
its results are never known um
completely they can't be known
and inevitably results will not be what
we desire because we'll
perform an action thinking that it's
great result we've all decided and
eventually uh somewhere along the chain
of events the causal chain of events
something will go wrong
and will not at all be what we want so
how do we deal with this
and not be terrified of action well
orent says we deal with it by
forgiveness because forgiveness is about
reconciling the inevitably
undesired outcomes of action and
allowing for a fresh
beginning so this is why we have
forgiveness as a human
ability to be able to forgive that we
tried our best
we performed an action and inevitably it
had bad results anyway
um the truth of action is uh factual
truth
um which is different from reason and
stuff which is that
and unfortunately we can't go into
scientific truths and stuff like that
because we don't have time
so i'm sorry about that but um the truth
of action is
simply it's fact uh that we remember it
and it's the most
delicate and precarious forms of truth
because
we can in fact forget it if we forget it
then we forget
our history as humans and the different
things that we can do and the different
possibilities
available to us and that's problem
as we as i mentioned it is terrifying
um it is perhaps the most giddying and
terrifying aspect of all human activity
because it is so unknown and many
ethical and political frameworks and
systems have sought to eliminate it
in various different ways to try and
remove the precarious nature of
action and freedom which orent says uh
eventually not immediately but
eventually the more we try and eliminate
it
results in totalitarianism because
without freedom
we lock ourselves and our generations to
come after us into a kind of perpetual
state of adolescence where we can never
grow
become fully human enact our will so
we now ask the grand question of where
is the location of ethics and here is
our schematic
of kind of our simplified schematic of
um society uh which i don't know how
long i've been talking for but i suspect
enough so where is it where shall we
find it
and uh that's i think a difficult
question
uh so this is my conclusion
each uh human activity has its own logic
and demonstrates a kind of truth uh but
truth compels
and inhibits freedom as ethics at the
start we've seen
requires freedom in order to confer
guilt and responsibility among other
things
ethics needs to be located within the
realm of freedom
but we then have a problem what is then
to be done about
labor and work and i don't know
um i think that orent would say
indeed ethics needs to be
located within freedom but it needs to
be able to
deal with the difference of labor and
work
and i guess that's up for our opinions
and our judgments to decide and as she
was a
dogged democratic in the ancient sense
of
term she would probably agree with that
so my closing thoughts is
this is an introduction to where
possibly ethics fits into
human activity many ethical frameworks
unfortunately only fit into one
particular area of human activity and
this is a problem
some examples include utilitarianism
primarily in labor
normative ethics in work virtue ethics
primarily
or almost exclusively in action or
deontological
d ontological sorry ethics
is primarily within the individual or
almost
wholly within the individual coming up
with principles and not in plurality so
it's
not there so the difficulty that we face
is in finding ethical frameworks that
can negotiate
different areas of human activity and
their different principles
and maintain those different principles
and the internal processes
of each activity so they would go in an
introduction
to trying to understand thank you
very much for listening thank you very
much


AI alignment, philosophical pluralism, and the relevance of non-Western philosophy | Tan Zhi Xuan

hello everyone and thank you for joining
us today
i'm vedic and i'll be the mc for this
session it is my pleasure to introduce
tan joshien who is a doctoral student at
mit pursuing ai alignment research in
the computational cognitive science
and probabilistic computing research
groups their current focus is on
inferring the latent hierarchical
structure of human motivations
by modeling agents as probabilistic
programs with the hope of aligning ai
with higher order goals
values and principles that humans strive
in part to live by
shen also serves as a board member of ea
singapore and was formerly a lead
organizer for yale ea
today shen will be giving a talk on ai
alignment
philosophical pluralism and the
relevance of non-western philosophy
we will be starting with a 15-minute
talk by shen and then move on to a live
q a where they will respond to some of
your questions
you can submit questions using the box
on the right-hand side of the video
and also vote for your favorite
questions to push them higher up the
list
without further ado here's shen
hey everyone my name is trisha and roger
shen favor mercy her pronouns
and i'm a doctoral student at mit doing
cognitive ai research
specifically i work on how we can infer
the hidden structure of human
motivations
by modeling humans using probabilistic
programs
today though i'll be talking about
something that's more in the background
that informs my work
and that's why ai alignment
philosophical pluralism and the
relevance of non-western philosophy
this talk is going to cover a lot of
ground so i want to give an overview
first to keep everyone oriented
first i'll give a brief introduction to
what ai lighting is and why it likely
matters as an effective cause area
i won't go too deeply into this though
so if you're new to the topic
i recommend introductory articles like
those by kelsey piper at vox
i'll then highlight some of the
philosophical tendencies of current ai
alignment research
and argue that it reflects a relatively
narrow set of philosophical views
given that these views famous crucial
considerations this motivates the need
for greater philosophical and
disciplinary pluralism
and then as a kind of proof by example
i'll aim to demonstrate how non-western
philosophy
might provide insight into several open
problems in airlines and research
cool so what is ai alignment one way to
cash it out is the project
of building intelligent systems that
robustly act in our collective interests
in other words building ai that's
aligned with our values
as many people in the ea community have
argued this is highly impactful cause
area if you believe the following
one artificial intelligence will
determine the future of our civilization
perhaps by replacing humanity as the
most intelligent agents on this planet
or perhaps by having some other kind of
a transformative impact
like enabling authoritarian dystopias
two the ai will be likely misaligned
with our collective interest by default
perhaps because it's just very hard to
specify what our values are
or because of bad systemic incentives
and three not only is problem really
difficult to solve
but we also can't wait till later to
start solving it
and you know basically everyone who
works in ai alignment
thinks is really daunting technical and
philosophical challenge
human values whatever they are are
really complex and fragile
and so every seemingly simple solution
to aligning superhuman ai is subject to
potentially catastrophic loopholes
i'll illustrate this by way of this
short dialogue between a human and a
fictional super intelligent chatbot
called gbt5 who's kind of like this
genie in a bottle
so you start out the chatbot and you ask
dear dvd5 please make everyone on this
planet
happy and then gbd5 replies
okay i will place them in stars chambers
and
inject them with heroin so to experience
eternal bliss
no no no please don't do that i mean
satisfied your preferences
not everyone wants heroin gpt5
all right but how should i figure out
what those preferences are
just listen to what they say they want
or infer it from how they act
this person says they can't bear hurt
animals but keeps eating meat
sounds like someone i know
well do what they would want to do if
they could think longer or have more
willpower
very well then i extrapolated that they
will come to support human extinction
to save other species
actually just stop
how do i know that that's what you
really want
so that's a taste of the kind of problem
to solve
obviously there's a lot of impact there
about like philosophy about what people
really want about what desires are
what preferences are and should we
always satisfy those preferences
um who's working on solving
those problems i wanted to give a sense
of what the technical ai alignment
ecosystem is currently like
to give a more context for what might be
missing
ai alignment is actually really small
and growing field composed of entities
like mary
fhi openai the line in the forum and so
on
most of these organizations are really
young um
mostly less than five years old and i
think it's fair to say that they've been
a little insular as well
because you think about ai alignment as
a field and the problems it's trying to
solve
you think it must be this really
interdisciplinary field that visit the
intersection of broader disciplines
like human computer interaction
cognitive science
ethics and academic philosophy
but the truth is to my knowledge there
actually isn't very much overlap between
these communities
it's more often aside like in this
picture there are reasons for this which
i'll get to
and it's already starting to change but
i think it probably explains the
relatively narrow philosophical horizons
of the ai alignment community
so what are these horizons i'm going to
lay out five philosophical tendencies
that i've perceived in the work that
comes out of the ai alignment community
so this is inevitably going to be a bit
subjective
but it's based on a work that gets
highlighted in venues like the alignment
newsletter or that gets discussed in the
ai alignment forum
first there's tendency towards
connectionism the position that
knowledge is restored as sub-symbolic
weights in neural networks
rather than language like symbols you
see there's an emphasis on deep learning
interpretability
scalability and robustness
second there's a tendency towards
behaviorism to build ai
or human-aligned ai we can model or
mimic humans as these reinforcement
learning agents
which avoid reasoning or planning just
by learning from lifetimes and lifetimes
of data
this is in contrast to more cognitive
approaches to ai which emphasizes the
ability to reason
and manipulate abstract mental models of
the world
third there's an implicit tendency
towards humane theories of motivation
here that we can model humans as
motivated by reward signals
that they receive from environment which
you can think of as desires or passions
as david hume called them
this is the contrast of more
conscientious motivation which leave
more room for humans to also be
motivated by reasons
for example commitments or intentions or
more principles
fourth there's a tendency to view
rationality solely in decision theoretic
terms
that is irrationality is about
maximizing expected value
where probabilities are updated in the
bayesian manner
but historically in philosophy there's a
lot more to more norms of reasoning
and rationality than just that
rationality is about logic and
argumentation and dialectic
broadly speaking it's about what makes
it it's about what makes sense for a
person to think or do
including what makes sense for a person
of value in the first place
finally there's an unsurprising tendency
towards consequentialism
consequentialism in the broad sense that
value and ethics are about outcomes or
states of the world
this excludes views that root value and
ethics and evaluative attitudes or
deontic norms or contractualism
why these tendencies of course it's
probably that a lot of very smart
people thought very hard about these
things and this is what made sense to
them
but very smart people may still be
systematically biased by their
intellectual environments and
trajectories
in particular it's worth noting that the
first three of these tendencies are very
much influenced
by recent successes of deep
reinforcement learning and ai
in fact prior to these successes a lot
of work in ai was more on the other end
of the spectrum
first order logic classical planning
cognitive systems etc
and the last two of these tendencies are
inherited from disciplines like
economics computer science and
communities like effective altruism
so at this point i hope to have shown
you how the ai alignment community
exists in a bit of a philosophical
bubble and so in that sense if you'll
forgive the term
it's rather parochial and there are
understandable reasons for this
for one alignment is still a young field
and hasn't reached a more diverse pool
of researchers
until more recently it's also been
excluded and not taken very seriously
within traditional academia
leading to a lack of interdisciplinary
collaboration
obviously there are also strong founder
effects due to the fuels emergence
within irrationalist and ea communities
and like much of ai and stem it inherits
barriers to participation
from an unjust world these can be and in
my opinion should be addressed
as the field grows we could make sure it
includes more disciplinary and
community outsiders it could foster
greater interdisciplinary
collaboration within academia we could
better recognize how our founder effects
may bias the search
through the space of ideas and we can
lower the barriers to participation
while countering unjust selection
effects
but why bother what exactly is the value
of breaking out this philosophical
bubble
and why do i use the word pluralism in
particular as opposed to just diversity
by philosophical pluralism i mean to
include philosophical diversity
by which i mean serious engagement with
multiple philosophical traditions
and disciplinary paradigms but i also
mean openness to the possibility
that the problem of aligning ai might
have multiple good answers
and then we need to contend with how to
do that having defined those terms let's
get into the reasons
the first is avoiding the street light
fallacy
that if we simply keep exploring the
philosophy that's familiar to western
educated elites
we are likely to miss on huge swaths of
human thought that might have crucial
relevance to ai alignment
the second is robustness of moral and
normative uncertainty
if you're unsure about what the right
thing to deal with or to align an ai
towards
and you think it's plausible that other
philosophical perspectives
might have good answers then it's
reasonable to diversify our resources
that incorporate them
the third is pluralism as a form of
political pragmatism
as jason gabriel writes in the absence
of moral agreement
is there a fair way to decide what
principles ai should align with
capo doesn't really put it this way but
one way to interpret this is that
pluralism is pragmatic because it's the
only way we're going to get
buy-in from disparate political actors
finally does pluralism as an ethical
commitment in itself
pluralism as respect for the quality and
autonomy of persons to choose what
values and ideals matter to them
this is the reason i personally find it
most compelling i think in order to
preserve a lot of what we care about in
this world
we need aligned ai to respect this
plurality of value
so that's why i think pluralism matters
the ai alignment perhaps you buy that
but would like more concrete examples
so now i'd like just to offer a few i
think non-western philosophy may be
especially relevant
to the following open problems in er
alignment
the first is representing and learning
human norms what are norms and how do we
constrain our actions or shape our
values
how do learners infer internalize them
from their social environments
classical chinese ethics especially
confusion ethics can provide some
insights
the second is robustness ontological
shifts and crises
we typically value the world in terms of
the objects and relations we used to
represent it
but what should an agent do when does
representations undergo transformative
shifts
certain schools of metaphysics vary
directly on these questions
the third is the phenomenology of
valuing and disfollowing
we value different things in different
ways with different subjective
experiences
but are these varieties of experience
and how should it inform agents that try
to learn what we value
buddhist jain and vedic philosophy have
been very much centered on these
questions
and could provide answers i'm actually
going to avoid going into this further
probably to keep this talk within time
but also because i think it's
increasingly accepted within the west
that contemplative traditions
may have useful insights about these
questions
before i go on i also wanted to note
that this is primarily drawn from only
the limited amount of
chinese and buddhist philosophy that i'm
familiar with this is certainly not all
of non-western philosophy
and it's a lot more out there outside of
the street light that may be relevant
so representing and learning human norms
why might you care about this
one answer that's common from game
theory is that norms have instrumental
value as coordinating devices or
unspoken
agreements if you look to completion
ethics however you get a quite different
picture
on one possible interpretation of
confucian thoughts norms and practices
are understood to have intrinsic value
as evaluative standards and expressive
acts
you can see this for example in the
analytics this word li
is hard to translate but it means
something like ritual propriety or
etiquette and it occurs again and again
in confusion thought
this particular line is just a central
role for ritual
in what confusion is thought of as a
humane and virtuous life
how to input interpret this quality
suggests that this is because
while virtual forms may just be
conventions without these conventions
important evaluative attitudes that
respect or reference
cannot be made intelligible or expressed
i was quite struck by this when i first
encountered it partly because i grew up
finding a law that's not really
pointless and oppressive
and to be clear some norms are
oppressive but i recently encountered a
very similar idea again in the work of
elizabeth anderson
who i cited previously that maybe come
around more to it
and speaking about how individuals value
things than where we get these values
from
she argues that individuals are not
self-sufficient in their capacity
to value things in different ways i'm
capable of valuing something in a
particular way
only in a social setting that upholds
norms for that
mode and actually find is really
compelling
if you think about what constitutes good
art or literature or beauty
that's undoubtedly tied up in norms and
about how to value things
and how to express those values if this
is right
then there's a sense in which the game
theoretic account of norms has got
things exactly reversed
in game theory it's assumed that norms
emerge out of the interaction of
individual preferences and so are
secondary
but for confucians and anderson it's the
opposite norms are primary or at least a
lot of them are
and what we individually value is shaped
by those norms
this was just a pretty deep
reorientation of what a alignment
approach is that learned human values
need to do
rather than learn individual values and
figure out how to balance them across
society we need to consider that many
values are social
from the outset next topic
robustness to ontological shifts and
crises this is actually a somewhat old
problem first posed by miriam 2011 and
it goes as follows
an agent defines his objectives based on
how it represents the world
what should happen when the
representation is changed
as it turns out buddhist philosophy
might provide some answers
to see how it's worth comparing it
against commonplace views about reality
and objects within it i think most of us
would grow up as what you might call
naive realists believing that through
our senses we perceive the world and as
objects directly as they are
by then we grow up and study some
science and encounter optical illusions
and maybe we become
representational realists instead we
believe that we indirectly construct
representations of external world from
sense data but the world being
represented out there is still real
now my tribal buddhism goes further it
rejects the idea that there's anything
ultimately real or true
and said all effects are at best
conventionally true and while there may
exist some external reality
there's no uniquely privileged
representation of that world
that is the correct one however some
representations are better for
alleviating suffering than others
and so part of the goal of building this
practice is to see through our everyday
representations
merely conventional and adult
representations better suited
alleviating suffering
this view is demonstrated quite
remarkably in the molecularity sutra
which actually uses gender as an example
of a concept that we need to see
through as conventional i was quite
astounded when i first read it because
the topic feels so current
but the text is actually 1800 years old
all this actually quite closely
resonates in my opinion with a recent
movement in western analytics philosophy
called conceptual engineering
the idea that we should re-engineer
concepts to suit our purposes
for example sally hasslinger and mit has
supplied his approach in her writings or
gender and race
arguing that we need to revise concepts
like those to better suit feminists and
anti-racist ends
i think this methodology is actually
really promising way to deal with the
question of ontological shifts
and almost suggest this iterative
algorithm for changing our
representations of the world
as shown on the screen whether this
would work
or would lead to reasonable outcomes as
i think really open research theory
with that i'll end my worldwide tour of
non-western philosophy and offer some
key takeaways and steps forward
what i hope to have shown with this talk
is that ai alignment has drawn from a
relatively narrow set of philosophical
perspectives
expanding the set for example with
non-western philosophy can provide fresh
insights and reduce the risk of
misalignment
in order to address this i'd like to
suggest that prospective researchers and
funders in the air alignment
should consider a wider range of
disciplines and approaches
also while support for alignment
research has grown in cs departments
we may need to increase support in other
fields in order to foster the
interdisciplinary expertise needed for
this daunting challenge
with that if you enjoyed this talk and
would like to learn more about airline
pluralism or non-western philosophy here
are some recommended readings
thank you for your attention and looking
forward to your questions