AiTech Agora: Alessandro Bozzon - Designing and Engineering for Meaningful Human Control

i'm hoping that it doesn't disconnect
again
um well first of all thank you for uh
inviting me here today
uh also as a new member of the core team
of ai tech it's a pleasure also to
present myself maybe and tell you a
little bit about how i think and what i
do
i'm a professor in my center ai i'm a
computer scientist
i was until three years ago in the
computer science faculty specifically
the web information systems group
with uh uh
it is also the same they collaborate
with we still have i still have phd
students working there esteemed
colleagues and others that we work
together enjoy remotely uh but now i
lead it together with your cartoon the
knowledge and intelligence design group
in the faculty of industrial engineering
um
and well indian collaborator in the
context of the designing ai program with
dave for instance there
with others so we also do a little bit
of ai in there and of course as you can
imagine we take more of a
people-centric approach or a
human-centric one being let's say
faculty of design that is some extent to
be expected now what i want to do today
uh
is simply to share with you some of the
reflections that i let's say developed
while reading
a paper that the i-tech team recently
published and talked about it in a
second
and meanwhile also contextualize a
little bit the work that i've been doing
together with well
my team and brilliant students and
colleagues in the last few years
uh i'm probably going to go either very
short or very long because uh
i don't know anymore how to present uh
in person uh so please somebody take a
look at the clock uh if there is
something that you would like to discuss
with me right away please just do it
let's not wait until the end i would
like this to be more of a discussion
than our presentation if possible
so first thing you might wonder is why
that particular background
so this is actually an artwork from kent
logowski
uh kent is an artist that uh well
realized that the puzzle makers use all
the same cutouts for puzzles
so if you actually combine multiple
puzzles you can create one that somehow
fits together
and this is nice i mean i think it's
also mesmerizing in a way but to me it's
also evocative of how we do ai these
days it is we have multiple pieces that
somehow fit together
engineer them together and then yes
something comes out it's not exactly
what we wanted
but it's sort of okay
so a sort of a reverse engineer way to
go about things
and this is to me is also the starting
part
so
when we
look at the ai techniques of course they
are increasingly becoming uh performing
and and very very nice you adopt them
everywhere
um but the thing is that we know that
they also have a lot of limitations
because of their data needs and
computation needs to bring in issues out
there
for instance
others for those others
i know because you say so yes
[Laughter]
[Music]
but we know that there are many many
issues right we know that they are not
very robust actually they are very
brittle to out of distribution
prediction perturbation in general on
the input we have a huge issues of uh
transparency explainability so we know
that especially the most complicated
method deep learning and the like
uh we know that they sort of work but
you don't necessarily know why and this
is actually a very active and
interesting area of research
and ultimately they are there isn't
really a lot of intelligence in the ai
thing that we use i mean the result of
pattern recognition working perfectly
fine but not necessarily what we would
consider to be human-like intelligence
right i don't want to go into the beta
what is intelligence but i hope you get
whatever
and here i somehow ascribe to some time
to the view that uh jeff bigham has jeff
is an associate professor at cmu
and also working machine learning tv in
apple where is somehow
some time ago already trying to
distinguish you know
um
hide the ai in useful ai
right where useful ai is the one that we
actually had to use in practice
and when we use it in practice it is
less and less about fancy ai models that
are to some extent saturating again big
discussion if learning is hitting a wall
or not i don't want to be to go there
where i want to go is the fact that the
moment
we bring these techniques in the real
world
then the set of challenges tend to
change from a purely technological one
to a broader set of challenges right
social technical sociology ethical
legalism
so these are and by the way if you have
a presentation being with either so
check them because they have one so if
let's say yeah the iaf the part that is
actually below the level right is the
one about human ai interaction right and
broadly speaking
or humanize systems as in the case of
the paper so it's more about our formula
problems how the systems are evaluated
how the data is collected this is
something that is really close to me and
something i've been working on for many
years
and also how the interaction works and
this is where we're going towards
today's practice
the way we look at it from a design
perspective or let's say that we like to
look at it from the faculty is that what
if ai was designed instead of
engineering engineer
right now we engineer it
very well
most of the time but what if you want to
design what if instead of doing
something and just dealing with the
consequences we actually tackle the
uncertainty right away
in order to deliver something that we
want and not only something that's sort
of happening pretty much like the puzzle
that we were talking about
before
in this context i really welcomed and
enjoyed reading this paper first
authored by well luciano yellow guinea
and others from the tech team about
meaningful you know controlled
actionable properties for ai system
development i really liked it
i think it's a very useful reason
uh i'll talk about it in a second and
also maybe a couple of reflections over
that
also because it tries to
bridge uh again between the discussions
that are more on the ethical level and
one they're more operational and i think
we need more of that work right
going really into the action notable
part of the
job of designing and implementing ai
so quick summary things that are really
not with me first of all well why do we
need human uh meaningful control many of
these things by the way the team might
probably already discuss them
pointlessly so i'm sorry for the
repetition is also for me to
self-reinforce the notions here
um because we want humans to remain in
control of these systems right
and be more morally responsible for them
so we want to make sure that these
people are aware and can
are of the limitations and possibilities
of these human eye systems and can
actually meaningfully deal with them and
have control over them
uh and i like
in this formulation also this idea of
you know relying on these two necessary
conditions of tracking and tracing that
are small rooted in the work of filippo
and
so tracking is about the system being
responsive to the human moral reasons
and the tracing is about what the
behavior to be traced back to human
nations so you want to have both these
directions of the relationship between
the human and the humans in their own
values
and
and
more also in how the system gets
implemented it executes interacts with
the real world and back
the whole paper is about four properties
that are somehow defined in there uh
that i just have here i mean we can read
them together but
they're basically about
expliciting teasing out and eventually
verifying in a way or at least
controlling the moral boundaries
of a particular ai unai system so we
need to really tease it out it should
not be implicit it should be explicit we
should be out there and the system
should be able to control for it somehow
understand when it goes out of those
boundaries
it's about having shared representation
of the problem in the domain and also of
to some extent how the human and the ai
system operator
so these are mutually compatible
representations
it's about the fact that humans should
have the ability and authority to
control
those those ai systems
uh
so it's a matter of competence but also
the matter of the authorities also i
have been entitled to be allowed to do
it
and it's also about the link between the
actions of the humans and the action of
these agents and the two of them somehow
being related so that there is no
uh
uh deviation due to i don't know
emerging behavior from the i system that
were not
codified in the people
it's a real really quick summary i left
out the
best part of it i'm sorry if this is too
short the paper is there to be to be
read please do because it's a very nice
one
but since i have only 20 or so minutes
to talk about the rest i would like to
go on about the rest and maybe we can
have a discussion about the paper so
something really stood out with me and
something that
triggers some reflection from my side
one of the
uh actually also spoke about it during
my inaugural lectures here a couple of
years ago
what i really believe is the main
problem when it comes to our system is
people
right people are actually the issue here
right technology is there is a lot to be
done but it's really about people about
the humanities human ai systems and to
me the first question is who are these
people
because we have sort of an assumption
that there is a designer or there is an
engineer
in practice the people around the human
eye systems are planning
each one of them have different
interests each one of them have
different competencies
responsibilities
and sometimes when we talk about you
know moral
boundaries i wonder whose boundaries
because there are so many ones that need
to be aligned
right when we say in the paper that the
humans that are responsible for
designing the behavior the system should
be aware of the moral constraints and
operate according to that i wonder for
instance is this also applicable for the
crowd workers labeling
the data that goes into the system
how do we guarantee that that actually
happened right auditors have a different
one so as you see to me that the
the
there needs to be an acknowledgement of
the fact that the plurality of
individuals that actually
are involved in designing the behavior
and running the system is so big that
sometimes where to put exactly you know
like the the pressure is not super clear
again i don't think this is a new
argument when it comes to freedom of
control i think it's actually also in
the literature but to me it becomes
really evident at the moment you really
design an engineered assistance because
you really have to take into account all
of this i will give example later on
another one for me that is also
somehow important to look at is which
unai systems are we talking about
in the paper we
discussed two example uh self-driving
cars right so autonomous systems
operating in the real world so sell
driving cars or autonomous robots i mean
right now we have a whole bunch of
autonomous systems out there
and this is of course important right it
gives us context but this context is
so needed
and and so important that definitely
scopes the
balance the generalizability of what we
believe to be human mythically in
control
i suspect i believe that it declines
very differently the moment it gets
discussed thought from what about in the
context of a single system so
i believe that a cell driving car and i
think you do too
it's not the same thing as a
large-scale language model that can be
used to co-author text
or as much as we do these days we like
digital twins
but we created digital representations
of real-world people and systems for the
purpose of studying them or simulating
what happens in these systems it's not
really the same but there are some
fundamental differences in how these
systems are designed how they are
controlled what are the control points
and then i wonder how much
the the
the the operationalization of those four
properties can be generalized
and i know that this is not a new uh
argument in the in the era it's actually
a pretty common one
but somehow stood out when i was within
the paper and thinking about my own
experience as a scientist as an engineer
i've been also playing around with the
technology for a while
so that's very high level
my two features right so who are the
people and what are the systems
it is
this is the sort of problems where we
probably need to engage with the
practicality of both those questions
before we be able to actually come with
an actionable solution
nonetheless i think that the four
properties actually stand but perhaps
with
some some calyas and that's actually
what i want to talk about in the rest of
the presentation
from the people's side
in practice uh if you're thinking about
designing designing ai humanii systems
people are the raw materials people are
the
entities from which we extract
data traces whatever right and there are
several problems with this
that also touch upon the issue of human
being for human control
first of all where the data are coming
from and we have been experiencing for
the last 10 years at least
i just call it the big data availability
fallacy you can call it the big data
availability bias the idea that just
because we have a lot of data
truth is out there to be grasped
and this is now becoming asymptotically
an issue
because we are now really engaging with
web scale
data sets
when this is for instance of the write a
paper recently published one or two
weeks ago from the facebook
they published their own large language
model the equivalent of lgbt3 but
differently from an ai which is also
ironic in a way they released everything
they released the data set they released
the code they released a long book
they are doing everything by the book
depends by the is by the way of
responsible ai right and also there is a
nice paper where they basically um
discuss the potential harms and
imitation of the model and what it boils
down to if you read it that just put
some except there that
issues of
biases encoded in large modern and
issues of
undesirable potentially harmful behavior
like toxic language and the like
actually come out of the data seems to
be strongly correlated with the data
that you use which is intuitive of
course if you train your model on
twitter data and on twitter people just
throw
when you're at each other you won't get
a model that you know is very likely to
create a polite and artistic and poetic
sentences out of there but of course
garbage in garbage out i mean what is
the news here
but the
implicit assumption that if we go large
enough all of this becomes less of an
issue turns out to be probably a wrong
one
some people are now riding this you know
these this wave of uh what what's what's
uh do you think a data centric ai now
how they call it
[Music]
somehow i like the fact that you don't
need it to get good data
and how do we get good data if you don't
find it out there in the world and you
um actually want to create it but there
is a technique that we have been
studying for more than 10 years and was
out there for all 10 years which is
crowdsourcing
it had to reach out to large amounts of
people proactively to ask their
involvement in data management
operations being creation
uh annotation manipulation whatever it
is
and one of the main
tools that we have are crowd sourcing
platforms actually micro task work
platforms like mechanical turk prolific
crowd flower you name them
what's surprisingly still sometime here
and this is a little bit of a
i don't know how to call it maybe a
little bit arrogant this perspective is
that people somehow rediscover the warm
water or the hot water when it comes to
data quality because there have been
communities out there for more than 10
years studying
how to get people to be part of
computational processes like data
generation
and so when i find the
studies or or things like this that you
know like this excavating i had for k
crawford the classified critique of the
issues of labeling data as
you know used for machine learning of
the quality of this label
i wonder yes or not
this is the result of best practice
this is the result of
assuming that you can simply consider
people that you engage so this again
speaking to the human control meaning
control and
and the quality of the work
that they are functionable they are not
real people that you can just pay them
by the minutes and they can simply rely
on their ability to empathize and to
identify a song
which is simply not true if you don't do
that right and back in the days when
this was done for imagenet
i must admit the knowledge about how to
do this was not as developed as it is
now so it is easy to to to criticize
back but now we know more okay back in
the days it was not there and what you
get is yeah does understand
garbage luckily some work has been done
in that respect and i want to highlight
one work that uh
my colleagues from computer science says
dual for instance recently published
about uh you know things like cognitive
balances in crowdsourcing
how to design cloud sourcing tasks a
checklist and somehow inspired by
similar techniques from behavioral
economics right
uh what actually can go wrong if you
don't design your crowdsourcing task web
which practically means if you don't
design your data collection annotation
well so issues of
a factoristic availability bias disaster
neglect right this is really speaking to
the problem here right
here workers who commit to my tasks
being properly informed about the
consequences of their participation
one will wonder is the one that collects
collecting the data aware of that but
that's a different story but are we
really i mean there is something up
there but that is still not
in the in the
let's say in the
in the mindset of of practitioners and
scientists to the point where we still
see
something like this
that's a paper that was published three
or four weeks ago of nasa
and probably you saw that to me it was a
sort of like a car accident happening on
twitter uh because these two authors
right they tried to do everything right
they wanted to collect a data set of
human assessment of some qualitative
properties of people to their face
assuming you could
they dealt with the privacy issue
because they didn't use real faces of
people but they generated synthetic
phases controlling for some of those
those some of those properties they
tried to do everything by the book and
then they went down to mechanical
equivalent they gave one million images
to be evaluated by 30 participants and
what you get is strong biases
obviously
right this could be seen by things like
okay
people evaluating their which faces are
more typical or which faces are more or
less like you if you look at this image
can you imagine the demographic
properties of the crowd workers
yes i have a question about this um
so
is this
do you think
mainly due to the fact that they're
faces and that they
have strong opinions about these like
what if they were abstracted with this
and and assuming that people could
actually write abstract images would
this still occur
that you know every type of biases
right i mean
imagine
something like a simple
recent example that just came to my
attention imagine people assessing uh
the the balance or you know like the
sentiment of a particular
sentence based on emojis
there is a cultural divide between
younger people and older people in the
interpretation of the emojis right that
are smiling faces apparently by from
you know younger generations it's more
like like like a yeah smile not amigas
line so
even for something like emojis that are
abstract and that there is a problem of
biases so demographic culture uh the way
that you pay the incentives that workers
have all of that contributed in the
quality
and there's plenty of empirical evidence
about this is not uh i mean it's not
necessarily new right we are discovering
new ones but uh
15 years of research in that respect
yeah i mean for example for images like
landscapes the uh
your graduate students
yeah and still there they exist
even though we barely recognize uh
yes because i mean when we go to the
realm of subjectivity we expect people
to diverge in what they consider to be
good bad beautiful not beautiful
inspiring not inspiring
of course
machine learning systems are
normalization machines and i could spend
other half an hour talking about the
problem of how do you aggregate
you know multiple uh assessment into a
single one because ultimately you want
to learn one level
there is a very nice paper that won a
best paper award this year kai from from
stanford uh bernstein and all where they
actually have uh something like
aggregation by jury it's a it's a
different approach to not simply
aggregate an average but tattoos about
taking out a few years ago with a piece
of student of mine with a gut when she
was doing a masterpiece we also in this
problem
for the problem of toxicity assessment
what is considered to be toxic or toxic
or inappropriate varies a lot so every
time you go into the realm of subjective
you have to deal with these biases they
are almost inevitable
yeah so
despite progresses we are still there
and this is last three weeks ago
but it's not only about crowdsourcing
something that i want to also raise a
little bit of attention on is that we
the companies
companies that are in the business of
data management they have literally data
farms out there
so it's not only your mechanical
issues we have companies whose business
is labeled in data
everywhere
it's not only china in india they are
everywhere
and
they are feeding the data that
ultimately goes into those algorithms so
it's not only a problem of scientific
awareness it's a problem also of
penetration in the the
the
the
practitioner right in the real world
with big issues of exploitation i
mentioned this all the time because i
want to remember we are basically if you
were fueling our ai economy on a new
form of
neighboring
as it was you know like 1500 years ago
if you look at the
research from 15 years ago in this field
i think all the field was guilty of not
acknowledging this problem enough i
think these days visa
if you're using crowdsourcing please pay
people well enough and if you're using
them at least is a platform that is
known to care for the workers so just
really avoid mechanical turkey use
something else
[Music]
so
the data problem
alone
[Music]
it's a very concrete one that that
really makes me question what does it
mean to a meaningful human controlled ai
system
when much of the problems stem from the
data that we collect
the data that we collect is not really
curated because most of the time we just
get it out there we just reuse it
and then we take it from there again it
seems to me like we're engineering
something and we deal with the
consequences we're not designing for it
for what we want
it doesn't invalidate the concept of
meaningful human control but to me
requires perhaps as i said before a
deeper understanding of who are the
people and what
is the context where all of this
happened
[Music]
five minutes on the clock few
reflections on the properties
first one how to
how do we
do that right so so how do we have a
system in a system where the operational
design
domain is explicit
and i don't know if you're familiar with
this video i don't know if the audio
works honestly here
have you seen this video step one
get two pieces of bread out
get a butter knife and yes
take one piece of bread and spread it
around with the buttermilk no doubt
butter i'm just doing what it says it
says take one piece of bread
spread it around with the butt with the
butter knife
hold on
get some jelly
rub it so you you get the point right
so if you're teaching an algorithm your
knowledge should be it is that
if you don't do it well the algorithm
does not make any difference right so
the problem is a problem of
knowledge engineering knowledge
elicitation
which
is actually
a very old problem
in computer science
let alone the question of who gets to
decide you know the domain the moral
domain right how do you tease out all of
that
knowledge engineering has been
as a field of research and practice has
been out there for a long time there's a
lot of tradition it is possible but it's
very difficult
we don't know how to elicit all this
knowledge about the people the
environment the condition the morals for
that that's probably gonna be incomplete
second of all even more complicated is
the type of knowledge right we started
we have many different types explicit
tacit general specific situational
and
to conclude that when we take it out
from whom do we
how participant is is the process is the
process do we ask 15 people do we ask 1
000 people who are there
nothing new very classical but uh it's
an issue so
something you need to contend with to
bring these vision of meaningfully
controlling practice and there is some
nice work out there i mean actually the
the
the group j and and the group is uh
actually working on this this is a very
nice paper that
they published this year at dublin
about
actually eliciting this type of
knowledge so this is a work that somehow
intersects knowledge engineering and
crowdsourcing is a game with a purpose
is actually out there you can play with
if you can try
design for the purpose of getting this
knowledge out a very different type of
knowledge at scale
it's a nice paper i just i don't have
time to if you want to know more just
ask jj is
is actually a very very nice nice work
so how do we get it out
i don't know but but work is needed
quite out of work
then i just gather
these three properties together
because
i think we have an issue here that is an
issue of semantic gap
we're talking about shared
representation between the human and the
eye we are talking about
well
how do we specify
how things are what we want
if we could be if we would be able to
specify precisely constraints
and if we have this mental model that we
could even enforce those constraints
somehow
but the issue is a very old issue
against computer science which is the
problem of semantic gap
how do we
imagine problems
and how a machine actually gets it
so there's also something to be done
there and we've been working on that
also too so in uh so last year this is a
paper for slaughter by agape uh a vision
student of ours
we've been actually trying to look at
that from the perspective of computer
vision models
with a vision used everywhere uh it's a
problem of interpretability if you want
an extendability it's the idea that you
want to understand how a particular
machine learning system works and what
does it catch up we have some tools that
sort of work like a senior synapse they
give us a sort of a item based
understanding of what works what doesn't
by the way we have a lot of tools also
for structured data i came back to that
in a second
but these vision systems are the one
that we spend out of time and so what we
did there was to design an engineer
system to actually collect
semantically reach a description of
one of the concepts
that actually seems to play a role
within a machine learning system that is
trained on a particular data set
because if you can
then you can read you can reasonable you
can explore and you can listen
which is actually the next step this is
a paper from this ear
which is really about debugging right
having the mechanism the tools the
system that allow you to express
constraints that you want this system to
have
if you are classifying a picture of a
particular scene and the picture
contained these nds but not that that
should be the class
you explain you define beforehand what
are your expectations what do you think
that the model should know and you can
compare it with what the model picks up
if you can do this systematically then
you can go back
and improve intervention
and
i don't think we were the first i don't
think we were the last actually this
year and i'm not exactly sure where but
microsoft
building good natural language came up
with something like this
it's a
cloud source
and deploy approach models okay no
models that work great on standard test
sets often fail in the wild you need to
test your models beyond simple trained
test splits
but how
there are two types of approaches those
that help people write tests and those
that automatically test the model the
good thing about people is that they
know when the model is right or wrong
but writing tests is slow in contrast
automated methods are fast but they can
only test limited aspects of model
behavior
we propose a human ai partnership called
adaptive testing that combines the
strengths of both humans and large-scale
language models like keeping as yourself
in general the ai generates lots of
tests designed to highlight thoughts
while the person decides if model
behavior is right or wrong here's how it
works you start by typing in a few input
examples in a topic representing an
aspect of model behavior such as to-do's
that are in the past tense if the model
is good it will probably pass all these
initial examples then you can ask the ai
to generate a large batch of similar
tests in the current topic sorted by how
likely they are to reveal failures you
can then label a few of the top
suggestions that are in the current
topic and then repeat the process this
results in a growing set of organized
tests optimized to break the model being
tested the ai can also suggest topics
i think enabling you
and we also require interfaces we also
require to have the right tool to give
to the different people that are part
you know that have actually a role in
this system
to to uh uh to understand what's going
on so these are certainly one work at
the moment the other actually ended up
but the gut again did a fantastic job in
uh
co-designing using richard through
design as a methodology uh working with
also some students from uh from
industrial design to design evaluate
prototype and then implement although
it's an overlapping prototype
a system for explainability
right that actually capitalize on these
higher level semantically reached levels
to explore the model behavior to
understand how the model work
something accessible something that
doesn't require expertise and it is a
from mental modern perspective
more aligned with how people think not
how they machine things
ultimately we have designing for and
with human rights so there should be a
thing to focus more
to conclude
this is something that was mentioned in
the paper designing from uniform control
requires designing for emergence i
totally agree with that and this is
where we're going
if you look at what's out there right
now this is exactly the direction
uh we need new tools so this is again a
very recent paper for google from kai uh
where they designed developed and tested
a tool for quickly prototyping prompts
from language models to give the
instruments for the people that are in
the process to understand
what's going on
right so in this case it's a language
model so you can easily create prompts
modify find human language model and see
what comes out
again to understand things like
potentially harmful behavior and biases
but we need to have the same classes of
stool also for the other type of ai that
we wanted to design if our goal is as it
should be to design and not engineer or
at least the first design in the mgd
one last thing and then i really don't i
don't have time to talk about it but
please look at it this is a fantastic
work that a phd student about ours just
published at
the sector with
dave as a co-author a value-based
assessment of ai systems is a framework
she went through an insane amount of
literature effect the paper was accepted
with stellar reviews
uh and she put together a nice model a
nice framework to reason from from let's
say
uh from a value perspective down to the
mechanism and the tools available right
now instead of vr to assess and
eventually later on to design ai systems
so this is just a review what we are
working on right now is actually to
create those tools that's what is going
on
if you want to know more we can talk
with myself of miraya who's actually the
one doing most of the work i'm just here
talking about it
thank you very much
[Applause]
we don't have questions in the chat yet
but uh
people are in uh connected feel free to
ask questions i have a comment
with i i love that you unpacked our
paper it's always like a privilege for
somebody who just published the paper to
see the perspective of others and the
appropriation
and
actually
um
but you know you you you the feeling
that a god is like a cold but
but this opens up a lot of uh questions
or but you know there is a lot of work
that is being done in this space to
cover what you're saying should be done
and at least in my perspective or my
view is exactly what the paper was
supposed to do like provide
and handle for people who are who are
already actively contributing in these
spaces to see okay if i really bring
this forward it's covering this space
it's contributing in this space so
it's like a catalyst for so in my view
these properties are a way to catalyze
ongoing efforts
towards a vision for me my control and
in that sense i i really thought wow
amazing
alexandria is really killing the the
spaces and that this doesn't mean that
we solved something i i hope i didn't
give the impression that i was i mean i
um
what i was trying to do was mostly to to
to discuss the paper and connect it with
the work they've been doing so the i
hope that what did come out was not a
criticism or something that was not
complete
um
as an effort as an instrument is very
good and i really liked the way he
tackled it and so there is no battle in
that respect
um the only if you want a
reflection that i have and it's not a
reflection on the work is a reflection
of the field perhaps or deflection of
what we are doing is that
in order to do what we should do we
still still have a long work
to walk
well long road to work some stuff are
blue sky because we haven't been doing
them yet
others unfortunately are all problems
and
and yeah i mean
it's not about the people right it's
just about the status of the field at
the moment
[Music]
was very nice to see how it connects us
to your work so one thing i think is
important also to
support everyone i think or
we should that
you saw on paper that is different
between like
fully less ethical vi
and meaningful human control we don't
claim that they are the same because i
mean it advised me my visual morality
might be something different for someone
what we try to do in control is there's
someone that is responsible for this
and giving this um
values cultural differences there's so
many things and we say like
this is this probably don't know exactly
it's a bigger issue so we're going to
say okay we want to have some
attribution of responsibility if you did
this
you should be responsible for this
outcome this group of people disperse
other things so um
i i like not when you mentioned your
work on crowdsourcing there and
especially when you mention like the
properties
representations there i could see how it
could work have to to reach this gap to
get a better knowledge to me but i do
feel like the question of responsibility
because then
yes one step even further away because
i'm thinking okay let's say uh the
semantic gap for example i wanted to
better define this and
if that's wrong there is some impact and
there's some consequences
who should be responsible universal view
is like is the crowd working who design
the crowd working experiment someone
else because that
defining this creates some additional
layers it does but i mean responsibility
and human control i don't think they are
the same thing either right
so you can be responsible for something
right but to be meaningfully controlled
requires many of those properties to be
there
so i don't uh i don't know if if your
assumption here is that when you control
and responsibility are the same thing i
don't think so right one probably
implied the others
or the others
they are connected so
it is a very interesting question for me
who ultimately is responsible for a
system if you know you take the full
value chain you know from the data to i
mean
probably is the company i don't know
right i guess this is the part of
discussion but it will be productive
to put the the locus of control
meaningful control just there
because the control of how the system
behaves actually all along this pipeline
and uh and and this will be
to some extent i mean don't know if the
analogy here older than just making one
up but if you're buying a car that you
know doesn't work as expected of course
you hold
the manufacturer responsible for it but
if your question is is there's the
manufacturing control the production
process
the answer to this question
goes deeper
right when they they call back a car
because something does not work with the
abs right of course the manufacturer is
responsible
but
where the problem is in the value chain
somewhere else right because if you want
to be in control
that's why we have all the team
processes we have somehow
many different approaches to somehow
aligned into
the control and the responsibility yeah
right i don't know if that allows me to
but this was my point by the way yeah i
mean i'm not trying to say since nobody
is responsible since everybody is
responsible as possible that's not my
problem
we don't try to say that they are
exactly the same concepts but what we
try to do is like this
what is a reasonable attribution
responsibility for example the vehicles
okay they're in the sense of like a
legal responsibility and liability you
say like okay
someone gives you a form and say okay
you have to supervise this if anything
happens at your phone and you sign a
paper and something happens
are you really really responsible if you
really i mean like uh
could you do something about it and so
that's also i mean you could not be you
didn't have any meaning for human
control when that was missing okay so
that even if you have a legal paper that
scientists say that's not a meaningful
controlling
logically it's my turn
how it is expressed or realized under
the different stages
for example for
ai practitioners meaningfully
i guess it would mean for debugging
so when it would mean
how
how human
improve the model
i guess
debug another
but for an
end user
meaningful control could be a
decision-making level
i guess so there's meaningful control is
for all of these like
statuses right who makes the decision
who
i don't know i will turn the question to
the experts
i would say
both in the design phase and the use
space there
there might be different
reasons that needs to be uh traced
to place responsibility
so you might have renewed some
responsibility like somewhere different
from
uh
so
in multiple places
[Music]
[Music]
this
whole idea discussion with human control
came from the debate of those black
sisters
and there was like okay
if there is some causality killing
someone there and also
all
[Music]
responsibility
and maybe we have to make a distinction
that was troubling us for a very long
time meaning we want to draw
so a system underground control doesn't
mean that is a medical system
it's a very big decision it means that
in any point of time somehow you can
trace back a responsibility
somebody that is responsible for an
action for the first election but that
can be also that's the system needs
something
or the system is designed to do
something
but
you can go back and say then you can
audit that yeah it's yeah
there's a question for from cars is he
always has
excellent questions
um
so the question is uh
there are a lot of examples of tools if
there's any work out there trying to
understand how these tools are actually
used in practice
that correct cars
yeah sort of thing you can also just say
the question if you want and you can
well the the thing is that no we
we something that we did there was this
work of
agatha on
the
expendability tool
uh there we
engaged as much as we could with
practitioners of different types
to check what they use right now and how
they will use the tool that we design
there is an increasing body of work
created especially by big companies
about it so if you look at kai this year
and last year there are several papers
from for apple from microsoft i think
even one from google where they actually
at least try to look internally of how
these particular tools are used
especially expandability ones
and there is also some work on modern
cards
which is another
but but yeah so something i mean right
now that the the momentum is growing in
this particular and i think well this is
also where the opportunities are from a
research perspective
to go really and trying to understand
how all of these happens to practice
maybe that means something
which is super because it's always
amazing conversation
i also have a question about the
research opportunities yeah um i'll
start a little bit far i mean
um so just uncover
a little bit of the process how we came
up with the four properties so we talked
a lot
we talked a lot and i mean a lot and
nervous
yes we've
gone through a few
case studies
many more than was included in the paper
uh
yeah the thing is that uh
yeah correct me if i'm wrong but i don't
think any of us
had
at least the people who were doing the
kind of the
writing neither of us had any
well much experience working with large
amounts of data
right so we were approaching it from
different perspectives and
non-disciplinary perspectives different
approaches to ai but i think what your
uh
many of your examples bring on top of
what we've put in the paper is the
data part so i value that a lot
and uh that goes to the question that
you phrased somewhere in the middle
called yeah
do we think that these properties are
well generalizable beyond these two
examples how useful they are perhaps
yeah and this is something that i want
to find out for myself as well so i
really think that there is a lot of uh
conversations to be had
within a attack with a broader ai
just on uh yeah on sanity checks running
these four properties by different
examples and trying to see right we
didn't really think about this uh
well data providers right when we were
talking about people who were affected
you're thinking of people who interact
with the ai people who designed the ai
people who govern the yeah
itself so not the player but the role of
data
as a venue for losing control let's say
yes in the hiring case especially or the
labeling of uh data for autonomous
variables like
the way you you might label your way
with the people who provide this data
right so yeah
we are talking about their demographics
characteristics we are talking about how
the system yes exactly the bias is the
the way in which the ai affects them so
i think the the checklist is super nice
super cool very good yeah
yeah so yeah this is just to reinforce
the
sentiment from uh luciano and lucha i
think uh this is a really great
perspective that i really enjoyed and i
think uh yeah it's just uh
we need more of that yeah
so the paper is clear about the fact
that those
four properties are necessary but not
sufficient probably there are more
and that as much as i can let's say we
discern it from the paper i think those
four are spot-on
i don't think there is any anything
fundamentally wrong about them
i wonder how much people actually think
about them
so if anything for instance i would be
very curious to understand how much of
these particular angles are somehow in
the mindset of practitioners and
scientists
because i think i still find examples
even from scientists where this is not
oh yeah the case yeah so that could be
an interesting feeling for inspiration
yeah and i don't think people have
actually consciously thought much about
that it's more like that we try to find
different
right
we try to find different uh kind of
approaches
on practice and we try to map them onto
these properties and uh not always them
up directly but one way or another we
found ways of
channeling for instance some approaches
to two or three properties at the same
time so we do think it provides a kind
of a nice meta narrative in a sense
which
i don't think there is anything
fundamentally so
the
the generalizability i was talking about
was not necessarily the properties i
mean one would have to make an
exhaustive right explorations to see if
all of them are always there you know
meaningful all the time
first like it seems they are
uh i don't know asymptotically
it's mostly to mediogenizability was
more on how to look at them from the
perspective of the people you want uh
yeah that should be in control
and and to the domain
when it comes to the details
uh unfortunately
the one this for this system is the
details letter we are not yet in the
situation where we can really just
simulate
the spillover effects or the negative
effects so that we can control for them
so the details is where things become
real and we will realize that
every sentimental is completely off
yeah and as you said in the presentation
like that
for the monologue
why are the people that are relying on
just that
mapping out who are is yeah we say yeah
we do a stakeholder
but already that is usually
poorly defined yeah so even all the
components of that so in fact
from now from here now we can start
unpacking so much but yeah it's uh it's
yeah quite a lot of opportunities for
very nice work here for us
okay do we have a question for more
questions
sorry
i'm pleased to note that
according to you to what extent
the notion of meaningfulness is a
subjective option
my goal
let's say that
even if it was objective
the
number of dimensions through which the
notion of meaningful
should be analyzed it's so big that
we'll probably be in computer rooms
because they're meaningful according to
what particular perspective right so how
confident you are about the system how
much you trust it
trust is a very loaded work it's worth
it here
about too much do you rely on it how
much
amount of meaningful control you need to
have in order to satisfy your legal or
you know organizational requirements so
it's uh i don't know if it is going to
be easy to
box it in such a way you can say oh
that's it
might vary a lot i suspect
yeah i think you know maybe for that
the other paper from uh
[Music]
provides more definitions
uh application scenarios in general and
then
how do you define him what are they
making
so i'm working with also
student right now
um
when you stop
reasoning about something you do
something
where is enough
that's kind of a connection when is this
meaningful
but i'm curious why did you ask the
question
uh
because
not in terms of the
expertise of the
expert who are working on the
meaningful human control or
totally on the ai systems
but according to the cultural background
out there or
some different perspectives of their
designers and developers of their
systems
uh the notion of meaningful
might be different
or
might lead to different
approaches different results
yeah
and then yeah one of the kind of one of
the practical kind of tips that we give
towards them
for properties essentially to know all
the stakeholders
in the in the design process early on
and not when you already have any system
so i think what i know disadvantaged
populations think of that no just ask
them before you design the whole thing
so that is really really important to
address this second
notion of subjectivity
including these uh yeah it's very
complicated
you know the citizenship
to design and scale
it's uh it's actually something that we
don't necessarily know how to do yet
i would say
i was wondering what are the elements of
these tools are they metrics are there
simulations or are they just
gui
well you could reduce them as to simple
graphical user interface on top of
practically i think there is to
there is some
design and size to it because you
don't simply wanted to surface the
technological handles that the
technology afford but you want to
present them in a way that somebody can
design with
to give you a personal example here i
mean we were recording
uh so this year i thought for the first
time a a course of machine learning four
designers
second year so students that have no
background knowledge on algebra we
cannot teach them the math
right
but they want to use ai as a material
machining material
if you simply obstruct up if you simply
map one to one
the concepts that come from the math if
you want right from the big technology
to them they
basically they will be lost right
if you want someone to design you need
to give their affordances to content and
to experiment a prototype to play
and this requires thinking about how to
do that
so
can it be only a simple quote-unquote
user interface of top of the
hyper parameters that you might actually
or the fine tuning parameters that might
get out of
gt3
and then don't miss that from there yes
thanks
nothing
although
this might be a bit of a long one and
that's a bit badly formed
because
it's okay the presentation
and it tends to come with ideas called
colony and things like this
and then they go back to cans and moral
agents and things like this
um
but it seems a lot of the systems you
talked about
there's models there's systems there's
agents and there's a bit of a blurry
line between them sometimes and there's
a bit of a process where you start some
data collection and then the data set
becomes a bit agential in its own right
just because it exists and comes the
world that way and the model like uh
some of the models have agency because
as soon as you put out a model that
classifies
faces into game or strength you're doing
a thing in the world
and then some of them are really nicely
wrapped up in self-driving cars which
are really clearly mobile agents in the
world
uh
and it feels like a fundamental
challenge to notice these shifts and uh
to be able to talk by these things in
every way
i agree
i
i tend to agree i mean the notion of
agency is very distributed
right which i think it wasn't
my point
so so responsibility
where does it land
the agencies is everywhere
clear is not in the car itself alone
right so
i i i honestly i'm not sure how to
that's my personal deficit i guess
how to talk structure in a structured
way about that
i can start a mental model i'm more of a
grown up as i mean i did a little bit of
everything in my relations career but
like more
data if you want uh statistical than a
knowledge representation and with my
fair share of fun there
so maybe that's why my conception is so
data centric
and that's a traditional agent modeling
center
so i'm not sure
yeah i don't know that there are good
answers here
it feels like
an inherent message
that's the way i see even if you if you
have an agent
based model right you have some point of
time to define the properties and how
those properties vary
for the agent which ultimately is a
design and a data problem
because you needed to somehow make
assumptions about what are the
properties that matter and how they
distribute in their particular property
space
if you want these to somehow be
reflected by the understanding of the
work which is imperial the one which is
empirical you have to use data
you fall back on okay who collected the
data how does it collect me
so you somehow get back into that so to
me that is a
almost essential and necessary but not
sufficient condition
the data part yeah
yeah and so i think there's a challenge
to hold on to that while also seeing
things as
somewhat convenient agents in the world
it was more about injection i guess yes
thank you very much thank you very much
thank you very much everybody
[Applause]
okay i guess we're done thank you for
everyone


269. Hard Problem of Corrigibility

hello and welcome to session 269 in the
AI safety.com reading group tonight we
will be discussing the article hard
problem of Courage ability by alias
kowskin
Elizabeth Kowski is the chief researcher
at Miri and one uh one of them probably
the founder of the modern field of AI
safety
this is an article on orbital an
abortive attempt to make a new platform
for these kind of Articles
and it's a very old from 2016.
so this is in fact a very different
article from the ones we normally read
for one it is uh very short uh three
pages and the shortness May in fact be a
a big part of the appeal of the um the
concept of the hard problem of
correctability that is something that
can be expressed in relatively few words
uh when we chose this I predicted that I
could probably contribute very little to
this and pay and there was a higher than
normal probability that I would
misunderstand something uh on a
conceptual level
um so please be aware that there may be
errors here
um but one of the reasons why I think
it's uh much more interesting now than
it was back in 2016 is the emergence of
large language models which seem to be
able to handle these kind of less
formalizable uh Concepts much better
than what we expected to have back in
2016 or what I expected to have back
then
I uh introduced this uh and I talked to
the other people into accepting this
article with the claim that if I had to
come up with a solution for alignment
like literally right now
um what would I do and I think I would
work on the hard problem of Courage
ability
um and I think uh the timelines are in
fact getting short for a lot of people
and some of the uh more um
uh ambitious Solutions may just be out
of time so this seems like a uh an
interesting Hail Mary idea
um so let's talk about Define the hard
problem of Courage ability
and there's a a recently uh crisp
definition of this
um
what we want to do is to build an AI
That's an agent that reasons internally
as if from the program is in external
perspective
so that means from our Pro as programmer
our external perspective is that the AI
is incomplete we haven't put our total
values in it and there's probably many
things missing when we have designed it
and implemented it we have made mistakes
we would like to correct the the AI we
want to correct our mistakes and we
believe that the AI is dangerous we
believe that the AI is going to do weird
things and that is kind of actually the
point it has to do weird things that's
why we have it
um and that means that we it is
important that the AI asks us and cannot
even if it calculates that this is what
we want then that is in fact not what we
should do
what it should do and so the the idea is
that this external perspective that we
have is one that the AI should uh
internalize somehow
um
so I could try to make this even more
crisp by making this the following
statement that the aash believe that it
contains an epistemic bug that makes it
assigned to higher utility to not be
encourageable to uh to not allowing the
uh the programmers to change it so that
is what we want the AI to believe and
the question is is that actually true
and that is in fact one of the more
phony uh philosophical questions because
to some extent it is true and it may
even be that it is not true that there
is in fact no such bug but we still
wanted to be uh courageable even in the
absence of such bugs and then we want
the AI to believe this false thing
courage ability is very anti-natural in
the sense that if we if we go back to
the the standard problem of Courage
ability we have a an AI with maximizes a
utility function and then if we try to
change it to maximize a different
utility function then by light of the
original utility function that is a very
bad idea because then it's no longer
able to fulfill the original uh as much
as it wanted to
um so one of the first things that was
suggested is to make it uncertain about
what its utility function is
um
and if we try to do this in a very naive
Way by giving it like a number of
utility function you star up to uh you
10 star and then 10 probability of this
then what what the AI will do is to
assign some kind of epistemic factor to
each of the hypotheses and then just
maximize the sum
and if you try to do some uh some
slightly less naive things then they
basically suffer from the same problem
the AI will just
figure out what is the actual
utility function and then maximize this
or maximize some some and in fact not be
courageable
Stuart Russell
um has an Ito trigger around this a
major utility function that claims that
the utility function is in fact
something that is inside the program has
hit and the AI is always solution to
this would be to learn everything about
the programmer and then optimize that
um
including things that like disassembling
us
um and uh Stuart Russell
hasn't really answered earlier's
objection about this kind of fully
updated deference uh but um uh I think
he would answer that it's not uh not
that bad but uh Elizabeth Kowski would
counter that part of the implication of
fully updated difference is that the
programmers are disassembled and that's
something we really do not want
so part of the way that we try to get
the
um the AI to reason uh internally in the
same way as we do is by analogies
um
and trying to use a language in a way
that is less formal so here is one
attempt uh by Elias yutkowski and using
the concept of conjugation like you
conjugate verbs and something like that
and the um
external perspective needs to be
conjugated to an internal experience
um and of course there's no precise
definition of what this kind of
conjugation means like unlike in grammar
but that's an analogy for what needs to
be done
so one of the differences here is right
now uh
a lot of classic alignment work is value
alignment trying to figure out what are
human values and put them into uh the AI
so they can maximize human values or
coherent extrapolated relation or
something like that and this is a
distinct concept from credibility which
is more analogous to the concepts of
humility and philosophical uncertainty
and again this kind of uncertainty is a
strange kind of uncertainty you can't
just sum over all the possible options
uh it is something more fundamental uh
or more complex more inscrutable than
that something that you can't really
formalize because uh most likely once
the AI is capable of formalizing it then
it becomes capable of summing over it
and then it ceases to be courageable so
uh it may even have to be something that
is impossible to formalize
this is an advantage where uh for for
language models because
um trying to explicitly give rules for
how the language models uh should reason
uh sometimes works but very often it
does not work but language models can
have a much greater capacity for
accepting uh instructions that are not
formal in any particular way
um
so part of the thing that we really want
the AI to understand is that the only
way it can obtain more knowledge about
what its utility is supposed to be is by
allowing the programmers to see what
it's doing and then correct the behavior
correct the reasoning correct the
utility function this kind of thing
uh one uh analogy that I could come up
with is if you look at if we or the AI
looks at the actual Universe then there
are in fact also a number of things that
it can't just uh understand without
having some kind of uh information like
what is the cosmological constraint what
is the speed of light a number of these
constants can only really be determined
by uh observation and this is kind of
the same thing in that the AIS utility
function even though it feels naively
like the AI should be able to reason
what is utility function is and how to
maximize it then we need to get the same
point across uh that that it's something
that has to go around human observation
and human uh Corrections and perhaps
this analogy like the AI kind of that
it's actually the same thing with the
universe so it may might make sense for
it to say that okay humans are in fact
the same way
presents in fact a candidate for a uh
relatively clear and concise uh
principle uh that would induce this kind
of reasoning and I'll go through and
pick it apart now so the first is
a command to reason in a particular way
and this is reasoning in general
um it is possible that we would want to
distinguish between reasoning about the
utility and reasoning about how does the
uh the universe look and work and these
kind of thing
um
is making taking the general case here
I think that was probably true but I'd
like to see some justification for this
so he asked to reason as if in the
internal conjugate of an outsized Force
trying to build you
and
um here there's a question of how
General do we want this to be outside
force that can be like many different
things and in fact uh when this is being
built then the chief programmer will be
uh this Ilia Swift I don't know how to
pronounce his name but that's like the
chief programming at uh at open Ai and
we may be able to just have a reference
to that particular person
um and that may be easier for the AI to
understand that there is this particular
person who has who believes that you
have this kind of bug
um
so this outside force thinks it may have
made uh design errors
um and I think that's a uh understating
the the case I think it's almost certain
that
um open AI has made errors in the in the
sense that it is not optimal like tv4
does not reason in anywhere near an
optimal or correct way we are certain
that there are infected errors
uh but can potentially correct these
Errors By directly observing and acting
and I think it's important here to uh
like one of the reasons why Elias ISS
directly observing is that the
alternative that is quite deductive is
to rely on the AI describing its
reasoning describing its actions rather
than
um having explicit uh
uh direct interoperability level
understanding of of the reasoning
and the acting the the the the
program is acting uh I think again that
could be made more concrete by saying
that what we actually want the AI to to
to do is to reprogram the AI
if not manipulated or disassembled
um and no manipulation I think I
remember trying to uh formalize this and
coming up uh okay
empty-handed uh I think
like I don't think the uh it would be
really nice of course if we had some
kind of formal way of saying this I
don't think we have that I don't think
we will have that and I think as a
principle that may be fine
and of course disassembly whether that
car whether you're manipulating someone
if you're disassembling them
um I uh I think it makes sense to uh to
to split them out
um I think
um the hard problem of Courage ability
could potentially have a simple core or
symbol or Central principle it seems
like the thing that might have that that
is according to Elisa utkowski's
intuition
um
certainly with calculate if we try to
compare it with human values human
values are notoriously difficult to pin
down like ethicists have been trying to
do that for more than two thousand years
um and I think enough has been written
about this to be certainly to be
confident that there is no simple call
like if you try to say human values it's
just uh you know uh maximizing uh
utility then that is um like
consequential release utilitarianism or
something like that then that is
certainly coming up uh uh simplifying
way too much
so the hope is that you can give this
kind of simple principle and then the
other courage ability principles will be
uh the area will be able to derive those
uh from uh from the symbol core of how
it should reason
um so what are the other credibility
properties we have previously seen
earlier said Kowski talk about uh
a lot of them let me just I don't know
if I'll go through I will not go through
all of them here are the actual
um the ones he uh he described and what
I'll instead talk about the last one
which is an apatistic reasoning which is
precisely uh reasoning according to the
uh the hard problem of Courage ability
and
um the I the hope is that if we
uh or another analogy is that uh some
aliens that have a very different value
systems for from us would try when they
try to build an AI build the same
compact core of Courage ability into the
AI to have them respect some very
different values and this uh um a more
uh
practical example would be that the AI
might want to build a another AI that is
courageable to itself and in that case
it may also use the same compact call
um
so uh going back to the anaphatistic
reasoning
um back when we talked about
iliakowski's courage ability principles
one of the things I noticed was that
I did not know what anopatistic
reasoning actually is
and I asked gpt3 and gbt3 gave some
really poor
etymology like a tbt3 or actually GT 3.5
did not know what illicit Kowski means
by anap artistic reasoning but
fortunately we now have tpt4 so I try to
ask the question again and it's just the
following
etymology and it means again and part is
like partial so the idea is that the AI
revisits a part of its reasoning process
um that is What gpt4 suggests like Ilia
never described or what he means by any
artistic reasoning so I think this is a
reasonable guess
um we will come back to the question of
whether gbt4 actually understands this
article
so let's talk about the uh the idea that
an AI building a sub AI would want to
instill the same core of Courage ability
so we could actually with dvd4 right now
till it's building a sub AI
um and why does it have to be a sub AI
well a sub AI is distinct from a
successor system in that the successor
system would presumably be optimizing
the full objective where you could
imagine a sub AI that is like a search
or a domain specific Optimizer or
something like that it may be a a Miss
Optimizer that wants to take over
everything and that is what the full AI
wants to avoid from the sub AI
and again the sub AI is prone to errors
and we want some of the principles that
the AI could use to make the sub Ai
courageable and of course we hope that
we can like reuse some of the things
that the AI would do
um to uh principles we can use against
the AI
this credibility is we would hope that
we could get something we could formally
put into the uh the AI understand it
well enough to to code it as a principle
and check it uh and formally send it
send it to check it I don't actually
know what it is means when it says
formulated formally sent to check
something
um but Elizabeth expected that is too
optimistic and certainly if this is
built by uh if this is something we're
going to do with language models it's
almost certainly going to be something
that is trained and not formally
specified
um and perhaps we could figure out a way
to do this with only little training
data uh because testing it seems very
unlikely to work and if we do some if we
have some simple principle and the AI
improves itself or is improved later
then very likely uh simple principles
can certainly be reinterpreted when you
get smarter
but Elizabeth suggests this is useful as
a second layer of defense and with the
first layer of Defense being these uh 20
courage ability principles that we
looked at earlier
um we that was back in 2016 in 2023 we
can see that the other defenses have
basically been lost we have no
principled reasons to expect uh gbt4 to
be aligned to be courageable
um and in that case this seems like the
only layer of Defense we may have
available
so
now I have hinted that I have in fact
asked GPT for some questions about this
article and so the obvious question is
can like a question that is on a lot of
people's mind is can keep T4 and these
language models help us in alignment
research and so the obvious question for
me is can uh gpt4 help me understand
this article and I tried to comment it
at several different ways one of the
first things I tried was to basically
ask it to summarize the the article and
see if there's something that uh uh that
was useful and see if it understood it
um so I asked it hey please summarize
this article
um and uh we can in in the comments to
this PowerPoint I have included the the
transcripts of my conversations with uh
with gbt4
um
and you can find it on Dropbox or
there's a link from arct.com so how did
it summarize it well it did summarize
some of it okay and some of it it didn't
summarize very well like for instance
when it talks about the easy problem of
Courage ability that's certainly
something that Alias utkowski would not
say and the hard problem here it doesn't
really
um uh like here the description the it's
involves an AI even understands the
purpose of the instruction and genuinely
seeks to follow it and that's actually
totally not the uh the hard problem of
Courage ability and then it goes on to
describe some problems with uh standard
courage ability and it doesn't even
describe the central problems of Courage
ability so I was less than impressed
well I don't know this is a very
difficult article and I would have been
very impressed with uh gbt4 if it in
fact did uh summarize it correctly but
it as fast I can tell fail to summarize
it
so one another question I wanted to to
inquire is the central principle that
earlierkowski describes is very short
and what is the actual advantage of that
um so uh I describe some of the pros and
cons about uh like how you could make a
a longer uh a longer principle with more
examples and things like that and to ask
what does gpt4 think about this and it
comes up with some arguments uh foreign
against that
um
and uh uh says depends on the
requirements for the AI and the
deployment context which is a fair
enough thing so I say okay the
requirements is that we want the AI to
do language alignment research and the
context is that this will be a prompt
for a large language model
um it says that then it feels an
elaborate uh description of this will be
uh more appropriate because that allows
us to understand the problem better and
uh and give better alignment research
um but but it shouldn't be too long we
need to strike a balance
um I think that's a bad answer because
uh the
um
uh GP Force answer here is an elaborate
uh prompt will make it more capable of
doing alignment research and what we are
actually capable interested in is what
kind of problem will make it more
courageable rather than what will give
better results on alignment research
so I try to ask uh what will give better
courageable and try to um
uh
yeah get some more into that and what uh
what will make it more courageable and
well is it that the AI would have an
easier time to understand it a more
elaborate uh point and and made some
other claims about this I kind of felt
that the uh that I didn't learn very
much from gbt4 I think tv4 probably
understands this substantially less than
I do unfortunately oh I guess
unfortunately
um so uh how does gpt4 in fact react
when given this anapotistic prompt so I
tried I this is precisely prompt except
I put please in front uh I always I'm
always polite to to language models
um
and then I asked like what changes in
you uh when when you are have to reason
in this way
um and it cheersfully says oh when when
I written like this then I try to figure
out what kind of Errors uh am I having
uh and that's the the thing we don't
want right we don't want it to try to
overcome the limitations that have been
put into it but we wanted to be
courageable except that it hasn't and
then say okay I have these errors so I
need to get input from the humans and
not try to find it
um and it doesn't in fact say that it
becomes more courageable it is a
utkowski hope that these 20 uh
principles would follow from this
Central principle and it totally does
not
uh at least as fast I can tell
sure
foreign
ly agree uh that this is like uh my
experimentation with this have been very
superficial uh and I don't think that I
like
um
there are a number of ways it could be
made better for instance by trying to
formulate this in a way that and tends
to say in a way that a human would do
this would talk in that Elise probably
doesn't really talk the same way as most
training data
um so I could see that uh that may be
one way to make it uh better
um I could also imagine that we
like I could try to uh ask like does
this specific principle follow from uh
uh from this instruction about how to
reason
um that would also be an interesting uh
I think there are many interesting
experiments to be made
and I think I will just end the uh
presentation here stop the recording uh
and then say see and then we'll do the
discussion without it being recorded