188. Formal Metaethics and Metasemantics for AI Alignment

so hello and welcome to session 188 in
the es safety reading group tonight
we'll be discussing the blog post formal
meta ethics and meta semantics for AI
alignment this is a work done by
jungkook who has a graduate degree in
philosophy from the top rank means
ethics program I think that's probably a
unique program in unit university but I
don't I have never looked up which one
that is
she describes herself as a computational
media this is in fact only one person in
the entire world describes themselves
like that so we have a meeting and also
this is in particular even though I read
this blog post I will be talking almost
exclusively about the actual program the
program was published in October and in
January of this year a large amount of
the way you could almost call in Kaunas
a description was added and this
probably has helped quite a bit from my
understanding at least but before I dive
into this I need to make some
disclaimers quite outside of my field
here I don't know much about media
because I don't know how I can figure
out what claims are good or bad example
da di which is a guy that does what we
shouldn't wanted to do and when I look
at the code in the description it seems
to me like me I too do not what we
should and I think the answer to this is
actually in that paper but I haven't
read
paper and I'm not sure sure I would even
be able to read that paper so say when I
say with a big grain of salt
I haven't read the entire description is
around five thousand words have many any
attempt to download a run yet to a code
and still has to make some criticism
here and there and I I think I will I
need to state up front that I think this
is really great work so but I'll still
come with criticism where I see it so
let's stop what this means
ethically I it is a fully technical
ethical goal function for an AI core
function and utility function and
probably reward function is mostly the
same thing in this context so the
important thing here is that this is
something that can be executed the code
is written and it bottoms out in the
sense that the individual parts of the
code there are tests and you can run it
and it's something that will actually
compute an answer but it seems like in
the tests there are only the half of it
that are being tested so that I couldn't
thank an example of an end to end test
like you can imagine a world with like
two people one of them on chocolate and
the other one won't strawberry and then
try to figure out how to satisfy that
preferences and this kind of thing I
couldn't see whether that was done I
think it would be possible but there
might be a part that that is not
possible there are some rather strong
assumptions unlimited compute and a
complete cost and bottom of the world
and of all the human brains in the world
so those are reduced from such an
unlimited computing that you can now try
every solution see which one is best and
it is those are not infinitely long so
that's some
superintelligence you can see some
weight
this is summed up by junk Lewis this is
not an Elana to be thrown away but a
scaffold that is to be superseded this
is implemented in that sits in which is
a programming language emphasizing sense
from the 60s but set in X which is a
more modern implementation running on
the Java Virtual Machine which is yeah
this is the logo for it so if you know
anything about mathematical work on
infinities you know that this is problem
not something that that is meant to run
fast but meat meat ethical today I is a
set directly view of AI and for that
reason having a language that is good
for sets and not good anything else
basically does to some extent make sense
however I think very close to nobody
chooses settle X so this is really nice
and when she posted it to to the general
AI safety community I think the the
choice of language harmed her hair
appeals it communicate quite a bit I
mean and many other languages like
Python has great support for sets but
pattern just does many more more things
and Python is something that a lot of
people who work in AI knows so for me
actually personally I work in Prolog so
for me looking at it says that lace was
like chocolate Suetonius revealed so so
for me that wasn't a problem at all and
now as for the code coding Belle use of
course the term describes them as
optimizing for solving the problem
rather than communicating the solution
and when I look at it I think actually
what just is optimizing for correctness
secretary tests and trying to make the
code
structured in a way that makes errors
stand out and even though paints me to
to say this as a software developer
correctness might just not be the right
choice here because I think optimizing
for readability is much more important
as far as variable names under school
one there is a word that's an
explanation in words and there's a code
and they have different structures and I
think this is a real problem and in a
moment I'll try to show the code and see
how it relates to the explanations and
what you see me do you think missing me
jump around a bit and say don't do this
later and please ignore this for now and
things like that and and I think this is
something that makes it much harder to
understand and on the technical side
from the website
I can't into this presentation here so
everything is done using images minor
things let's go and see the hood in the
description in words what is brains by
having a social welfare function from
the set of extension rational utility
functions of the brains this is the
brain and the way I feel be you know a
physical feature it seems to work for
the brain I think everyone is a nicer
shell in Princeton just because all
speakers then who gets to decide precise
you are a salsa Sarah anyway this one
here can be split up into four parts
first is that the world is given as a
model and and this arrow triangle here
means this can be explained in and key
it can be expanded a lot and this over
here refers to the is a link to the X to
a power then the year utility function
for the
it's the it depends on the brain's
decision algorithms if it made more
optimal decisions and then we in order
to compare brains then we need to catch
them out as in terms of to catch the
brains terms out in terms of the real
world and then finally when we need to
merge these we choose the center of
gravity
among these extension rational Newton's
functions so this is like the basic
overall structure now let's have a look
at the code so here are the line numbers
in the left and you can see here the
mathematical AI utility function is a
procedure that has input takes in the
world and a set of brains and then it
returns a utility function down here so
this is the code that you can actually
do so the first thing we have here is
the the world model so in this world
model we get in the set of all states of
the world Campion this is section 1.1
and then we define a utility function so
we have the entire world with all this
possible states and then for each
possible state we make a ranking of them
and then we just say utility function is
a mapping from this states to tune in
ranking paper basically so if the
rankings of all states that the world
can possibly be in you know what to
calculate that we need to given the
brains that is given as input then we
need to find their decision algorithms
this is section 1.2 1 and once we have
these decision algorithm then for the
brains we need to find the the new
rational utility function cashed out
again in terms of references for the
earth to the world
they then we have the the set of the
utility functions for the brains then
here we do something that's not
explained and I think I kind of
understand it I can't really explain it
so we'll jump past this and go round
here where we have a set of possible
utility functions
recall that up here we have the set of
all possible utility functions and then
here we can compare them by saying which
one is closer to the utility functions
let the brains have and once we have a
comparison of this then we can just
solve all the possible utility functions
and then take the best one that's the
code down here and once we're the best
utility function it will be returned and
that is the the main overview of what
mythical AI utility function actually
does so you can see here there are a
number of some fields that we need to
dive into and let's start with one point
one where where we have the world and as
a mark of mom so this is again the
entire description and it's a detailed
specification it that's 150 lines of
code
there's also tests and you know here's a
reasonably following also rather
standard six book so well meaning that
nothing can even happen there and of
course in a safety if you go to this ROM
you will find a lot of things about a
causal path parking etc so but but I
think this is a very fair fair
assumption so let's go to part one point
two you are given a brain and then you
need to figure out what decision
algorithm does it actually use so
there's again some description of this
how this is done so let's see how to say
you do this
[Music]
[Music]
is to dive into the code and we try to
find I think and I can show the
representative sample of so we move
grains and we need to find out what
decision algorithm is implemented by the
spring so we start by finding the set of
all possible decision algorithms that's
some testing coherent that needs to be
ignored and how to find all the decision
algorithms in the world there's some
people who get you on the Nexus 5 but
once you have all the decision
algorithms then you need to filter this
set down with the ones that correspond
precisely to the spring the ones then
and of those it also needs to have the
lowest complexity weights on a thousand
and some like Kolmogorov complexity and
then among those algorithms we take the
one that is the better explanation and
then we return that so that's of course
a trade-off between whether is precisely
corresponds to the brain the Kolmogorov
complexity and this better explanation
and thus the need to be have some kind
of trade-off between these values better
explanation is actually also cashed out
into four different things that the
decision algorithm should be as good as
possible
and we'll get to this in two slides but
first let's see what is the set of all
decision algorithms here you can see
what we do is we have brain and then we
look at how many states can it be in and
let's say the brain can be in 1 million
states then we take authorization
algorithms
with complexity less than 1 billion and
then co-producers I think this is an
implementation of first-order logic but
I'm not going to have an implementation
we're referring to Turing machines
because that's generally the way I think
about the set of all algorithms but
let's go to the question of when is a
decision algorithm a better explanation
for what what the brain actually does
and the way this is implemented is that
first we find and cut it for all
explanations a assault order requiring
we give each explanation is score and
the score is calculated by with four
parts how ambitious the decision
algorithm is - how complex it is - how
much instrumental irrationality there is
- Lee in coherence so these are again
for things that are defined for for
these algorithms I won't go into all of
them in the explanation this was called
charity and to me the word church it
doesn't really met that comes to church
him okay let's go on to how we actually
once we have the values of the brains
when how do we refer this to the world
so first we need to figure out what are
the values of the brain according to his
own representation we'll get to that in
the next slide but once we have this
then we need to go to the world and then
find the expressions figure out what
they refer to and then try to see how we
can for all well the world states that
can dothey the human were campion what
utility do we assign sir
and this is basically how we do this and
away with some of the utilities I don't
think I'll go into more details here but
let's instead never new get how to
figure out the utility of brain
according to its own representations so
again we're the rational utility
function a procedure that given the
world and the brain it starts by all
possible social choice utility functions
which is basically everything that the
world that the brain can refer to and
the the rankings and they are not
defined in states of the worlds but in
something a bit different I'll get to
that in a moment but one thing you might
be wondering here is why is there a
social choice here in the brain because
there isn't that much socially inside a
single brain so here you can see exactly
where the domains and the range are for
these inside the brain utility functions
and there's some more you know
bookkeeping I would almost call with you
and then finally what we see here is
that these possible utility functions
are then weighted against each other
with with a voting weight and this
voting weight is in fact Boston's
parliamentary model for how to manage
moral was also in this 15 years ago or
something okay so we have the world
model and we have the brains how they
work and in order to compare them we
have cashed it out in to where are
things that relate to the real world now
this was the original utility function
and then you remember that was this
merging here where if we find the
difference between them so this is a
repeater for that said previously and
then what we do here is that we find the
difference in in audience between the
utility function and the the utility
functions and and then we Square to
ensure that it's non negative and the
problem here the way I see it is that
someone could be a utility monster so if
we imagine that everybody has you know
reasonable utility functions about and
the world should be you know rather
normal middle of the rope and that's one
person who really really favors that he
should be given all the power then this
distance will be very very large for for
him towards all states where he doesn't
have ultimate power and in this case of
course just squaring it will will make
it very very large so the sense of grad
C algorithm will tend to give about very
much power in in this situation so when
I look at this and again I didn't I was
[Music]
but I didn't find any discussion of how
to make this faster right where with
other this kind of theoretical
algorithms like I see for instance we do
have approximations and we have some
path towards some kind of disability and
also like an optimum patient reason can
be approximated to some extent but
there's no discussion of this there's
the problem of mine crime and that we
this as far as I can so implements every
possible minecraft there is also no
possibility for moral growth against
Zipcar seems to imply that you basically
have end up with this idealized decision
algorithm
these values and then you can never have
anything else and I think this on a
conceptual level has a lot of similarity
it is with aixi and I think if some kind
of explicit comparison with I'd seen
would be nice to have there's also a
very similar suggestion for how to for
what n/a I should to be in an ethical
sense called coherent is troubling
position by any SAT counseling I would
also like to see some comparison here
straps ROM has made a impossibility
theorem about houses distinguish well
used from limits and rationality and I
won't select to see some kind of
relationship between the existing word
and jungkook's work of course social
choice Theory have a number of other
impossibility theorem most famously eros
theorem which is also not considered but
in Tomi
I was very impressed with us this surely
has been a lot of work and I hope this
is something that will be asked as fall
to be superseded that is all for today
thank you and see you next week


AiTech Agora:  Karim Jebari - Artificial intelligence and democratic legitimacy

especially when you have a cooperating
with with uh colleagues and things like
this makes you
increases the motivation level so it's
great to be here
uh so uh yeah i can say a bit about the
background so
we have a research team
here
at the institute for future studies that
are doing uh things related to
ai and democratic legitimacy and other
kinds of
uh moral philosophy but also other like
empirical issues and uh yeah i hope that
uh i mean if you're interested in
presenting something uh at our seminar
that we have just started forming that
would be uh really great for us so if
you if you would be interesting you can
just send me an email
okay so uh the
this article is about
as you you might have seen the abstract
about
this idea of democratic legitimacy and
how
uh it can be impacted or how we can
think about
the impact of ai on the uh democratic
decision making
or decision making in a democracy okay
so here's the outline i'm going to talk
a bit about democratic legitimacy
and the principle of publicity i want to
um relying on the work of thomas
christiano
i also want to talk about how we extend
the principle
or at least
apply this idea to a wider set of of
problems or a wider set of domains
then i'm going to say something about
how
machine learning can be used in public
decision making or is being used and the
challenges to the principle of publicity
from machine learning
and then
give a few kind of considerations for
when machine learning can be legitimate
from this from this point of view and i
just want to on the outside here saying
that
machine learning can be legitimate yet
bad in other ways perhaps if it's not
inaccurate or if it's biased and so on
so so this is just that a
decision-making process is legitimate
it's not a sufficient condition for this
for it to be good it's a necessary
condition
okay let's start
so the idea of democratic legitimacy is
uh well
it's it's basically a property that a
public institution has a public
institution that exerts authority over
citizens for example um child protection
services courts employment agencies and
so on
and
what democratic legitimacy does is that
it permits
that public institution to wield power
and sometimes even coercive power
over citizens and it also implies that
there is a moral requirement that
subjects comply with that authority so
if um authority is not
legitimate then it's wrong for this
authority to wield coercive power over
citizens
and citizens have no moral
responsibility to to obey that authority
there might be other reasons why um
public authority maybe
you know you don't need to follow
to to comply but but the legitimacy is
an important one and this is
discussed a lot in in um political
science and so on and one of the central
ideas that make um
a
polity or a
public institution
legitimate is this idea of public
equality
and this is an idea that's
been defended by several philosophers
but most prominently by thomas
christiano
and the principle of publicity says that
um
well it has two main components
the one is
recent giving and this is the idea that
the
decision maker the authority needs to
provide reasons for a decision
and
those reasons
need to
be
specific to the case so for example
let's say that
i was drunk driving and and i got caught
by the police and the police um
made a test so it's uh
they could
verify that i was uh drunk and then the
the courts or the police could say that
you know it's
it's illegal to be drunk while you're
driving and we also have evidence that
you were drunk
and therefore
these are the consequences so the
important thing is that the reasons
are specific to me to to
to this particular case so would not be
enough for
the court to say that you know on this
particular hour
then people are often drunk
at this
at this intersection or people that
drive cars like yours are often drunk so
it would not be acceptable to make a
decision on the basis of kind of some
statistical regularity the the the
reasons need to be
particular to to me
the second
aspect is that
those reasons need to be
public for the relevant stakeholders
for example lawyers myself of course and
perhaps journalists and so on and and i
think there's been a lot of talk in the
realm of ai ethics and so on about
transparency but we want to emphasize
that publicity is not the same thing as
transparency als although they are in
some respects similar
so publicity just
as it means that a
a verdict or the the process by which
the verdict was reached
is made public to the relevant people
and and it might not always be
a good idea to make them completely
public for example in the case with
minors being defendants in criminal
court cases uh in sweden at least we
have a rule that
the the identity of the child is
protected
and this is uh makes the process less
transparent in one sense but it's not
something that violates the principle of
publicity
or at least it's nothing that
necessarily violates the principle of
publicity
so according to thomas cristiano the
this idea of the
public equality and the
principle of uh publicity is mainly
important for the democratic assembly
and the
executive power so
the the democratic assembly needs to be
public with its uh reasoning and with
its decision making
um but we argue that public equality is
also important for administrative
authority uh and judicial authority and
we're not sure that uh
thomas christiano would agree disagree
with us i think that he would agree with
us but what we don't know um but we have
two main reasons for why this uh this
idea should include administrative and
judicial decisions uh one is because
um
the it's it's required to uh determine
whether
administrative decision makers
uh
serve to a uh serve the aim of the
democratic assembly so
if if we don't know for example
uh the the reasoning behind uh
uh a court when they sentence someone
then we don't know if the court takes uh
the laws uh and the procedures are
stipulated by the democratic assembly
seriously so that's one kind of
instrumental reasons for why
this public equality is is important but
we also and this is perhaps a more
substantive claim that uh
that um
there is no strict separability between
the legislative domain and the judicial
and administrative domain of public
authority
i mean in one sense there is
well these are different branches of
government of course but if we're
interested in the legislative process in
in how
uh democratic societies craft rules and
laws
then we cannot
disregard the fact that those laws are
not fully formed when the ink dries so
to speak on on
on the legislative assemblies uh
uh decisions uh the the laws that's just
the first step in the legislative
process in the the de facto legislative
process and the most obvious example of
this is of course
it's um
the president's as established by courts
so the courts interpret the law and
apply it and those precedents become
well have the the same force as law
but
precedents are also important in an
informal sense in
public administ agencies for public
administration
there
is the precedence are established
routines are established
uh
analysis of the requirements are
established and so on so the whole kind
of process downstream that
from the legislative domain to the
grassroots bureaucrats is actually
a part of the
legislative
process so that's why we think that this
idea
should also include um
all all branches of government
that exercise power over citizens
okay so this brings us to uh machine
learning uh
algorithms
and uh yeah so you are uh
of course uh you know uh
more about this than i do perhaps but uh
yeah so
we are also very familiar that the
machine learning algorithms make their
decisions or yeah based on the basis of
categorizations
uh so for example a a very famous case
and you probably know about this the
the machine learning system compass uh
placed individuals in different
categories um
on
different categories on the likelihood
to reoffend
by comparing the profile of these
individuals with other
profiles
of people that got apprehended after
reoffending
so if you are sufficiently similar in
some respects to someone that did
reoffend then the system decides that
you are likely to reoffend and then
of course the system does not have the
final say in this case but
the courts have
relatively little uh discretionary power
to disregard the verdict of the system
so
they can deviate from what the system
says but it's um
you know
it's costly for them in
in in some respects they need to
motivate it and so on so it so it means
that the fact of what the system is
doing is uh
exercising power over a citizen on the
basis of that citizen's statistical
similarity to other citizens
so this would then we argue violate the
principle of reason giving
another problem and this is of course
something that you are very familiar
with is that machine learning systems
are opaque
and this opacity we argue is
can be broken down in a number of
different kinds of opacity and this is
partly
inspired by
julia dressler's work
but we have observational opacity so to
speak it's very difficult to inspect the
code
uh there's also a cons to considerable
extent uh theoretical opacity um the the
field of machine learning is still
uncertain about
you know the you know theoretical
yeah it in its theoretical understanding
on why certain
network architectures work uh why others
don't work how many layers there should
be
how much training you should have
etcetera etcetera so they're still kind
of the jury is still out and there was a
paper from a few years ago at mips that
described the state of the field of
machine learning as alchemy
that is you know we can produce cool and
uh effective tools but
we we don't have much understanding of
how we're doing it
um yeah so
i don't remember the name of the author
but you can you can google uh
nips and alchemy
um
then we have of course judicial opacity
uh
and this is of course not always the
case but in many of the cases that we've
looked at the algorithms are actually
not run
not owned by the public
administrations but by a third
party company and they protect their
algorithms and they also sometimes it's
also secret how they collect data
and
yeah so so there's a kind of a whole
system of
capitalist or proprietary opacity around
the use of these algorithms
on top of that we have sociological
opacity this is something that dresser
names too that
this is and hopefully this will not be a
very big problem in the future but at
least at the moment
our societies
have no clue how ai works most people
have this kind of very
weird sci-fi inspired idea about how ai
works that it's supposed to be some kind
of magic or or that it can think and so
on
and this creates a lot of problems um in
terms of opacity because
it's often misapp ai systems are often
misapplied
and it's also difficult for
society for example uh legal scholars
and and journalists and so on to
understand what's going on uh so that's
that adds another layer of opacity and
finally there's a certain
sense in which machine learning systems
are psychologically opaque and by
psychological opacity we mean that
um machine learning systems don't really
behave
like people do nor
do they behave like
um
like machines often do so let let me
give you an example if you have a car
and the car
you know it's a bit rusty it has a kind
of some problems that
the capacity of the car will degrade
gracefully you know the car will be
worse at accelerating maybe it will be
it will start to shake and so on and
that that means that you can somehow
know that you know i i probably need to
take this car to the workshop fairly
soon
but ai systems
not they're not unique but
one feature that air systems have that
that cars and
humans have is that they don't have this
graceful failure effect and often a
minor deviation can cause a catastrophic
failure and there's been multiple
examples of this you know
you're probably familiar with all of
them when you add a couple of pixels to
a stop sign the uh to an image of a stop
sign the
the the the cars can't see the stop sign
anymore or
or interpret it at some something else
um
so and and the problem here being of
course is it's an obvious problem when
when
when the systems don't do what they're
supposed to do but the main the problem
here from the perspective of opacity is
that it's difficult for humans to
understand
the failure of machines of these kinds
of machines so it's difficult for us to
predict failure
in a machine
and this adds another layer
and this connects to
an objection
that we discuss
and that is often voiced in these uh
circumstances that humans are also
opaque and this is of course true um you
know you can't see my brain and so on uh
but
um
humans
yeah so but ai systems have these
multiple layers of opacity
which means that um and and these
multiple layers we argue
reinforce each other
so
um the
um
the yeah the outcome is a situation that
is much worse uh
than with most human decision makers and
we can of course also add to this uh but
we don't discuss this in the paper but
perhaps we should is the the the
fact that humans can be personally
accountable
whereas ai systems cannot of course you
can hold the company accountable so if a
company produces an ai that is for
example biased against black
people then you can
say that okay you need to recall this
product or you need to pay out fines and
so on so this is of course
perfectly possible but the problem is
that
our
institutional systems don't have that um
institutional capacity at the moment for
extracting accountability from
third-party providers like this and this
will perhaps come in the future but
as long as we don't have that those
mechanisms there's a
difference in how easy it is to extract
accountability from an ai system versus
a human decision maker
okay so this sounds a pretty uh downbeat
should we does does this mean that we
should not use uh ml systems at all in
the public decision making
well so this is um
i think
this opens up for the need of a more
kind of
complex annuals analysis uh that
that cannot be kind of sweeping and too
general
so one of the considerations that's
important is what happens at the
grassroots
grassroots level so in some cases uh
people use the ai system much more than
intended so
the they
they um
perhaps because they don't have time to
make their own considerations they just
follow the the advice from the ai uh
decision support system
uh or for some other reason maybe for
some cultural reason or for some
sociological reason people feel
pressured to do what their system tells
them to do without
themselves making a
a judgement
but in other cases the reverse can be
true and this seems to be the case we
did a case study here at the
swedish public employment agency
where we saw that a lot of the
caseworkers at the swedish employment
system they were very unwilling to use
the
uh
the considerations of the ai
and uh and it was kind of in is it's a
kind of a fun example because this ml
system had um
yeah it had an algorithm that
assigned the probability of getting a
job to a job seeker
but
since there's a lot of noise in the
system the ml system also had a random
variable
so if you
if the caseworker clicked on
re-evaluate many times the verdict could
change and uh we
we have seen that many caseworkers uh
keep clicking on the button until they
get the result that they
feel um corresponds to their intuitions
so yeah it can vary a lot and these
variations are important to how
acceptable it is
because um
a public decision for example in the
switch employment office is of course a
a decision built up from any sub
decisions
and
and
to the extent that
some decisions are very kind of near to
the top of the decision hierarchy that
would be more problematic than in a
decision that is low in the decision
hierarchy um
and whether or not it's on the top or on
the bottom of the decision hierarchy so
to speak it depends on these
sociological factors
other other considerations is that some
steps in in a public decision uh are not
and cannot be explained so for example
consider the case of a police officer
stopping me here for for drunk driving
and
one of the steps in the decision of
giving me a fine for example
would be for the police officers to
check my driver's license and
um make sure that i am the person
that
on the picture of the driver's license
so yeah so they get the right person and
this step in a decision is
typically not explained
the police officer will if asked at some
later time the police officer will just
say well he looked like the guy on the
id so yeah and and it yeah
that step is typically not explained and
it's also a step that um
is a very a clear case of a pattern
matching or pattern recognition exercise
so what the police officers is doing is
that it's matching my facial features to
the facial features on the person on the
id card
so
in a hypothetical case where this step
in the decision was made by a machine
then
that step in a decision would be
then that would be no worse from the
point of view of um
of publicity than it is now so if you
think that it's acceptable
the way it works now with a police
officer doing this visual check
then
then we should also accept that machine
learning systems can
could do this this step
and
finally we we argue also that
um and this is also from cristiano that
that there are
different kinds of decisions that the
public administration can do and some of
them concern our constitutional
interests and exactly what those are
well that's a matter for debate uh but
examples of this uh include you know
the right of habeas corpus
the right of free speech and so on free
assembly and so on
and
when those interests are threatened
or or under consideration then it's much
more important than
that the decision process be public
whereas other decisions
are
not
constitutional interests but just the
interests of the legislative assembly or
the democratic assembly for example
let's say that the democratic assembly
wants to
raise the
child allowance
credit
uh for
families with
many children say
and and then you need to have some
decision maker that that needs to you
know um make an assessment of whether
this
this child is my child for example and
so it's not my constitutional interest
to get an increased child allowance for
example so in those cases it would be
you could be more uh permitting of these
uh
partial violations or or partial
infringements of the principle of
publicity
yeah that was the end of slasher thank
you
yeah thanks a lot kareem um
yeah if you could stop sharing the
screen then we don't see ourselves twice
uh yeah yeah
i think that helps
um right so
feel free to raise a hand or post
questions in the chat
i
since no one is doing so just yet i'll
just take the first question
so one thing i was wondering about is if
you could say something more
about these reasons people are supposed
to be giving so i can imagine that
caseworkers when they estimate the
probability of someone getting a job
soon might do this based on previous
experience and so comparing the current
case to earlier cases they've seen so
how does that contrast to
these statistical inferences you were
talking about that aren't
uh allowed which
at least to some degree make similar
inferences based on earlier cases
um
yeah so
so
yeah so that's that that would be
another example i would say of a public
decision maker
uh in some cases making an assessment
that is very similar to that
assessment
which a machine learning system would do
so
of course there are um
also
individual particular considerations
that the caseworker takes into
consideration
that are perhaps not part of the kind of
data set that the caseworker has let's
say that someone is a felon a convicted
felon then a caseworker would
if we think that the convicted felons
are very rare uh so they would perhaps
not be in the kind of data set of a uh
of a case worker
but
the case worker can take that into
consideration when making an assessment
so um
i would say that in most cases a
caseworker would
make
an inference based on on statistical
similarity
but in many cases it would be um
it would be kind of based on on
uh
yeah you know individuals uh
particular circumstances uh of course
that does not make the decision any good
but that's that's another question
but yeah so the just to answer your
question the the discussions we've had
is precisely this that that many
decisions are made
on this statistical basis
and they also
not
by you know by mistake or by
you know the it's there's an explicit
instruction from the policy maker from
the legislative assembly that you need
to do this
and and we've been thinking about that
you know when there's an explicit
mandate from the legislative assembly to
make uh a decision on the basis of
statistical similarity that could also
be um
a set of cases that where machine
learning can be acceptable provided that
this opacity
is not a major problem
okay thanks um
andrei had a question
or you raised your hand
yeah
yes oh
yeah i think it works now uh sorry i
just my microphone was uh not working so
um really interesting presentation i had
a couple of questions one uh in the way
you
in your characterization of uh reason
giving
uh it seemed a bit restrictive that you
would characterize it in terms of rules
and laws
uh a connection between rules and laws
on the one hand and some natural effects
on the other
because if you if you frame it like this
i don't see why um
the reason giving property would not be
a technical complication when it comes
to ai system as opposed to some sort of
structural
uh impossibility so we can program
uh
ai tools in such a way that they do
connect some individual property of the
case to some law
on or general rules so in that in that
respect you can make them reason giving
in the state sense that you or in the
relevance that you characterized
um
and and and second about public reason
giving
um
i was thinking that there are standard
and this is not so much on the side of
the authorities but depends on how you
think about authorities in the
democratic context
there are
standard
or salient democratic institutions such
as juries
or voting where
we have good democratic reasons not to
engage in public recent giving so for
example
jurors are actually protected from
giving reasons for their verdicts uh and
they're
protected against the public in their
deliberations and similarly so
for uh voters
so um
i was wondering whether they're not
might not be considerations that are
similar to those when it comes to the
problem at hand and finally a question
of
about the reference i didn't get is it
destler about the dressler the types of
opacity if you could uh just pump that
because i find it interesting thanks
um yeah thank you so um yeah so
and this is i didn't mention this but
in in our view it's a big difference
between uh classical expert systems
and machine learning systems uh in uh in
this context because what the
an expert system or a classical ai would
do is that it would
you know
say
here's a condition
uh if that condition is fulfilled then
you know then this is something that we
want to do yeah and and there are
actually
expert systems
based in sweden for uh
social insurance compensation and
although those might be good or bad and
there's been some complications with
those systems because of this kind of
proprietary thing uh that
they're making verdicts and nobody
really knows how this system is working
because it's
they don't have access to it but from
the kind of
recent giving
aspect it's uh it would be fine in our
view because it's not kind of
statistical in in that sense
um
you asked also about what public means
in this context or or want a
clarification and i think that the jury
is a very good um good example of as a
case where the reasons are public in one
sense but they're not transparent so the
the reasons are accessible for example
okay i'm not an expert on the american
jury system so you can correct me if i'm
wrong but the the the reasons uh
provided by the jury can be uh accessed
by the the courts or by uh lawyers and
the uh the
the prosecutor perhaps i don't know but
there is at least
an idea that um
that these reasons can be communicated
in in some respect
and
but but i think it's an interesting and
especially the other example you gave
voters the voters
yeah
they are protected in
all democracies from disclosing what
they voted on and so on uh and uh but
but that would be a case that would not
be a case of public
decision-making so a voter is not
the state it's not the coercive
authority of the state so the voter is
uh the citizens exerting power on
the decision makers so so that they are
not kind of included in this requirement
um
and perhaps now i'm kind of just
thinking what i'm talking and perhaps we
should consider juries in the same
category as voters in this case that
juries ought not to be considered as
representatives of the state but rather
as representatives of the public i don't
know what do you think
so so i i don't want to dwell but
especially for juries i think the case
is clear that they are exercising a
public office uh in that respect so so
we might disagree about what voters are
whether uh you know that's a
civic office and so on but for for for
juries it's clear that it is a public
office that is filled in with ordinary
citizens
and that affects concrete
uh
people the persons directly
and there might be at this analogy in
the sense that there are reasons during
the liberation so whatever goes into
their deliberations is accessible in
in places where they can discuss about
the deliberations and you know there are
a lot of biographies and kind of quality
there's a lot of qualitative research on
that yeah but they're not accessible to
the to the judge with some exceptions
where
uh basically they engage in in
problematic practices uh but they can
have very weird
reasons going in there and and they're
not bound to communicate them uh in uh
in in any sense but uh yeah so yeah
thank you so much this is an interesting
case i never thought of so i need yeah
yeah we need to discuss this thanks
okay um david also had a question
yes indeed uh thanks for the for the
talk um
so i'm not sure i could follow all of it
but uh but i i did have uh a
question
or maybe like a you know a point for
brainstorming
um when you
you basically compared uh
the decisions that humans need to make
and need to be able to provide reasons
for
uh versus
the fact that you know that that's hard
to do that for algorithms
and i was wondering um if you could
think along with me um
that i have this feeling that uh
um
uh
kind of we we accept the opaqueness as
you called it of of of humans
um but we seem to
uh in general kind of
overestimate or or you know hope too
much
uh of technology as being able to
yeah i think afghani usually calls it
like a tech our way out of the
opaqueness
um
and so
i guess i'm not quite sure where i'm
heading but but my i guess my point is
that
those are kind of
public processes
where
a lot of the times also the officers
that introduce them be they public or
companies
kind of position themselves as this is
going to improve our decision making or
this is going to make it much more
objective and much more clear etc
whereas i think you pointed out many of
the issues with that
so maybe i'm asking a kind of meta
question
so maybe i think what odd law or public
offices
uh
yeah what should they do to kind of uh
yeah i guess talk about
or frame their use of technologies
um
do you have any perspectives on that uh
or is my question just too vague it
could also be
um
yeah i i'm not sure exactly
that we addressed that point i mean
i think that they what we rather want to
say is that
you know these
this idea of
public equality
we take it seriously i mean in many
countries take this very seriously it's
there's a mention about this
the right to an explanation and so on in
many of the eu treaties in swedish law
also
and and those are kind of echoing the
sentiment that you know if if
the child protector agency wants to take
my child
you know at least i should get a
explanation for why
uh and um
and that uh
that explanation can not only be
you know
according to some companies uh secret
data set you are similar to other people
who lost their custody of their children
uh that yeah so it so that's the kind of
idea but but of course that's a kind of
general intuition and what we want to do
is to bring that down to the
complexities of
of how
ai comes into the decision process
so i don't know if it was a good answer
to your question but that's at least the
way we
we try to approach that that question
and uh sorry i i forgot to answer uh
andre uh
the the person is julia dressle
and she has an article from i think 2016
about uh opacity uh
let's say yeah
so so can if i can quickly respond
so i think
what you're describing lies at the heart
of what we try to do with ai tech has or
this concept of meaningful human control
being able to somehow you know track and
trace
um
so that's
part of it lies in the idea that well
maybe we could somehow you know
investigate the algorithms
and ask questions and make them more
transparent
and i think now we propose several
approaches to do so
but i guess the main point is that
that i feel that
that even if we do that
um
yeah there's something in us that as a
society or as users overestimates that
the the technology and so i i guess
maybe i'm saying as a brainstorm so it's
not a clear question but
but how do those kind of
systemic
you know impacts of
seeing technology as a as a way to
improve decision making and
uh what are your your thoughts on that
yeah
yeah
so yeah i have a few thoughts and well i
i kind of
kind of mentioned some of them that
in the presentation but i think that
one of the thoughts is that we can
actually reduce some of this opacity can
be reduced perhaps it's
and i think many researchers now are
trying to reduce the technical opacity
uh and observational opacity and that's
great but
there are so many layers of opacity that
don't have to do with the ai system
themselves like for example this
proprietary opacity that you know
companies can say like no we don't we're
not going to share our data or we're not
going to share our
data gathering practices and so on so so
that's kind of an opacity that could be
reduced quite quite a lot by kind of
legislative efforts and also the
sociological opacity can also be reduced
you know if people were more informed
about how ai systems work then
perhaps yeah then then that would be a
good thing
uh i can i can give you an anecdote from
this swedish public employment office
where they they had uh so
it's fairly easy to know the probability
of someone getting a job if they have
been
registered at the office for a long time
because uh the probability of you
getting a job within the six next next
six months is um
very well correlated with an amount of
time you've been unemployed so the
longer you've been unemployed the less
likely it is you're gonna get a job so
okay that's great but the problem of
course is to how do you categorize
people that are just
coming into the office so then you don't
have this data of course because yeah
and and that's why they developed the
algorithm for those cases
uh because it was a kind of a difficult
uh
so here's a person he's been unemployed
for two weeks
is this one of those that is gonna stay
like for many years in the unemployment
office or is this person gonna get a job
the next day
great but the problem is because the
bosses do not understand ai
they started using the same algorithm
for people that were already unemployed
for a very long time
the problem is that the variable being
unemployed for you know three years was
not part of the algorithm so of course
the algorithm sucked
compared with any you know reasonable
caseworker that said like look this
person has been unemployed for three
years and we don't think it's going to
get a job
in the near term but the since the
algorithm did not have that data
variable that is super i mean it's like
correlation 0.7 or something uh then
yeah and and this is not the algorithm's
fault so to speak it's just the bosses
that don't understand how this works so
sorry for that
yeah
thanks thanks a lot for thinking along
with the with the vague question uh
thanks claire
okay great uh and then focus had another
question
yes thank you thank you for the
presentation i i missed the first few
minutes so maybe you answered my
questions then i have three things to to
mention so if that's too much then um we
should maybe deliberate
after this meeting
i'm studying the use of aiber police
officers so you'll understand that i
have some
some thoughts about this as well
my first question is
when you explain the objections to
currently for systems being not being
opaque
you mentioned the opaqueness of machine
learning that is based on
correlation and not on causation
but what i did not see but that may be
because i missed it is that it's not
context aware that we do not have
general ai
so that things that for people are very
for instance in the swedish
system
if somebody has cancer
and is seriously ill
and in case vehicles understand he will
not get a job before he's but if it's
not in the system
the system will not see it and will give
bad
predictions
so
that was that's my first question uh
do does it play a role in your article
or your considerations
uh
no not necessarily so we're not
assessing whether
you know the accuracy or what how good
an a system is but rather this aspect of
legitimacy
and of course that there's what's
the political scientist called process
legitimacy you know if a decision maker
is really bad at what they're doing
and consistently produces outcomes that
reduces
citizen welfare then that can also
undermine legitimacy but
it's kind of another
route to to that
yes but i think that
the lack of context awareness and of
general of common sense
that that is a very important reason
that
the legitimacy of machine learning
in public processes should be limited
that's my first remark the second
is
that you state
you have three
three
reasons that machine learning can be
applied even if it's
you in general should not do it
and one of those is if it's currently
not being explained like you said
recognize a picture a person in a
picture then we don't have to ask it
from a machine learning application
i don't agree with that i think it's it
can be um
sometimes we we should ask it for
instance in policing
um currently the selection of people to
stop
uh there is a suspicion that a bias
plays a role racist
racism
and we want to get rid of it so even
though currently we do not have a good
explanation why people by police
officers stop certain people and they
don't stop other people
we would we will not accept this opaque
system in this way we want to be sure
that this bias that we have i think we
recognize
is excluded
how do you think about this
no i agree and perhaps i was unclear
about this so what we say is that uh if
you replace this step in in a decision
with an ai system then the ais system is
no worse than it was before but of
course if you think that this is an it's
unacceptable that we don't explain this
decision then of course you're gonna
think that it's unacceptable that we use
ai too um but uh yeah so so yeah i agree
with that point yes
and the third one is
a principle i
formulated for the dusk police
um
that has to do with
with risk management
um and you could maybe add it in your
considerations
that um detectability
and
repairability of decisions
as well are
a factor that can
help decide whether you can apply ai or
not and very easy example is speed
cameras
be accepted if there's automated
decision taking and automated finance
sent to the people because it's very
easy for people
or the the burden for people
to uh to object or to say no it i have
not been there give me a picture and i
can show you that it's not my car
it's quite easy
so it has to do with whether you can
recognize that an error has been made
and then you can repair it and this
could as well be applied maybe in
in other less
mundane
decisions like
legislation or so
yeah yeah thank you yeah so
i think that this is something that we
want to discuss in more detail and
perhaps this is uh
yeah we we are not sure
exactly how far to go in this direction
in this paper but this is definitely
something that we want to discuss that
we want to
discuss like what is a public decision
and how
what do we mean when we say that ai is
supporting a public decision so a public
decision can you know it's there are a
lot of
steps
before the decision is made
but as you suggest
the the the decision might not be final
because people might uh
uh
ask for appeal
uh for a decision and and that means
that we also have to this is a good
point very good point i think you made
because that means that we we don't have
only
we don't normally have to consider what
happens up until the decision but we
should also consider what happens after
the decision and if there is a right to
appeal
uh so justice that can be it makes a big
difference if let's say
a person a decision maker has the legal
right to deviate from the ai
recommendation or if they don't so
that's
that's a big difference in our view
or
and and of course the difficulty by
which you are allowed to deviate from
that verdict because that can vary also
a lot right um for example in uh in this
at the moment in the swedish employment
office the the bosses are trying to
force people to uh comply with the ai
recommendation so
a directive has been issued recently but
it was very controversial so they
dropped it was that they need to comply
with the a system at least at least 50
percent of the time
which is kind of okay whatever
but yeah but that was very controversial
because people were very dissatisfied
and yeah but there are a lot of these
back and forth and what you mentioned is
also a relevant aspect of that
consideration is there
an opportunity to appeal
and if so is that
instance of appeal is that a person or
is that perhaps also an ai because
that's also possible right that it could
be
another a that that gives you a second
opinion or a second verdict so that's
relevant yeah
uh you're on mute yes so in this is
there an instance where you can appeal
but the other thing is
is there a way to detect whether a
decision was
maybe not right
because some systems are so complex that
the human will have no idea whatsoever
whether a mistake has been made or you
don't feel the the
you don't know that a decision has been
made so you won't be able to appeal
because you don't detect it for instance
if the police would decide
to uh
in their in their office
to do a lot of controls in my
neighborhood i won't be informed about
this decision or the reason for it only
i think i will
feel that sometimes i'm stopped a little
bit more than before
but i i have no no no way to appeal
because i even don't know that this
decision has been made
so that's really in uh
yes and we have had an ai agora meeting
before where some
dower presented his research about
autonomy of people when using ai systems
do you know about this
uh
what was their name a tower i will put
it in the chat
i thought the i thought the
pronunciation was down man okay okay
yeah yeah i know
okay
okay yeah
i
i think i i am familiar with the with
the article
yes okay thank you for uh answering
thank you
yeah thanks um
well if there are more questions feel
free to ask um one thing i wanted was
still curious about sort of as a
follow-up on on david uh oh well afghani
you go first
ah
thanks stefan uh kareem thanks very much
for uh very interesting presentation so
i i uh
my question i think it connects to what
david uh was bringing up
uh so connecting to the what you were
telling about
uh the this point that like when
if when you talk about the things you
talked about in your presentation
sometimes people will say well humans
are also opaque
um
but and then you you well you explained
well but hang on uh
what are some of the differences i'm
curious do you have some thoughts
um
so so you you mentioned that some some
of this comes also from misunderstanding
of how the ai functions
and that and that then i think so you
imply i think that comes from people who
are not themselves let's say technical
experts and but maybe they misunderstand
how the technology works but i'm curious
do you have thoughts about how those of
us who come from the technical field
how we can do a better job
in
also being clear to ourselves but also
being clear to
the outside world about
um
what
we you know what does the technology
what it can do what it cannot do
what it quantifies and what does it do
with these quantifications and these
kind of things
thank you how can we communicate
actually better to the outside world to
to
to decrease the miscommunication
and manage expectations yeah thank you
uh so yeah actually i have some thoughts
about this and this is from a friend of
mine who is a professor in
uh
in the uk uh
in york university and he's a professor
in machine learning
and
he is you know he he comes from uh
mathematical statistics from that side
so that
he comes from that side and and he
thinks so he he tells me like for me
it's important that the students
understand that what they're doing is
mathematics and statistics right
so
so he had this course plan where they
you know explained like dutch books and
those kind of uh uh you know
bayesian statistics and those kind of
the the
the the the basics of
understanding what is
what is statistics but a lot of students
were very dissatisfied with this and
they actually went to the prefect and
complained and said like we don't want
to learn about this uh
uh
you know
philosophy uh
we want to learn just how to do uh
do the apps
so we can get a job uh you know and uh
and and he was uh yeah he was very
bitter about this because he he felt
like uh unless
you give up the students kind of a
rigorous understanding of
you know what was the statistical
influence and what that means uh the the
programmers will not understand what
they do and they're going to be kind of
reproducing false ideas about about
machine learning so i think that one one
going for on his view i think that one
important aspect is to
uh
have a
a curricula for programmers or for
machine learning developers young people
that that includes this
kind of maybe more difficult ideas about
the nature of statistics and so so
yeah thank you very much thanks
uh okay well very quickly then so one
thing i was still wondering about is
if we
uh manage to have more of this
technical opacity if that takes away
some of your worries about the reasons
giving aspects of machine learning or if
this is purely because it's statistics
that's problematic so let's say if we
have
feature importance methods and we can
point to specific
aspects of an application why it's
deemed as unlikely that this person will
get a new job does that solve part of
the issues around giving reasons for the
decision
or is there just the fact that it's
fundamentally statistics that's
problematic
um yeah i would say that
[Music]
yeah well i mean i think we have
different views in the
among us three in on this idea
so my idea is that
you know
it's very difficult to be absolutist
about
what is reason giving because
as some of your examples before the
there are many cases where reason giving
certainly involves a statistical element
at least
uh when people do do it
and
so
um so i think that
or at least my view is that we need to
have kind of
a a holistic approach to this that we we
need to both consider to what extent are
these decisions
uh
yeah
to some extent what what
is this decision reason based
and the problem opacity but also the
thing that i mentioned the in the end of
what interests are at stake
are these uh
constitutional interests that have to do
with yeah with personal liberty and so
on or are these principles that have to
do with you know uh
compensation uh like a well one first
system compensation levels or safety net
compensation levels that can of course
vary and
uh and uh from
depending on on what the democratic
assembly wants
so i think those are our uh
important considerations as well
but i think that my political theory
colleagues they're more kind of
no there needs to be public reason
uh so uh so yeah um
yeah i don't know if that was a great
answer but that that's at least the way
i see it that uh
that it's um
it it's it's difficult to give a kind of
a
universal answer to that that question
yeah okay well thanks a lot uh also
thanks so much for this conversation
yeah
great thanks so much i have to actually
run to another semi