AiTech Agora:  Pradeep Murukannaiah - Personal Values and Social Norms as Foundations of AI Ethics

all right
uh do you see my screen now yes all
right
cool uh i think we can get started
uh hi all uh first of all uh uh thanks
for uh inviting me to uh present in this
forum
that's uh i think uh ai tech is a great
uh
match for the several things that i do
and it's a pleasure to be presenting
here
uh so uh lucian already mentioned so i'm
pradeep murukulaya i'm an assistant
professor in interactive intelligence
uh so i also co-direct uh hippo lab so
this is a new delft ai lab
it's a bridge between computer science
and
tpm broadly on the theme of using ai for
public policy design and analysis so
we're currently
in the state of like hiring phd students
and of course we're looking forward to
build the lab
and happy to talk to any of you uh with
respect to what we do and uh
looking for collaborations there yeah so
with uh that said i guess so we can
just uh dive deep into it uh
what did we do here yeah so
uh broadly the outline for talk today is
uh i mean i want to introduce the notion
of uh
ai ethics uh from our standpoint so it's
from standpoint of
social technical systems and then i'll
describe what we mean by social
technical systems and why our
view of social technical systems is
important for uh
like realizing ethics in technical
systems
and then i'll talk about particularly
about values and norms so those are
two ingredients of ethical social
technical systems
and then i'll end with some broader
challenge challenges and
immediate opportunities to advance this
topic so that's
broadly the outline
okay so let's um start with uh like a
very fundamental question
so what is uh ethics so i mean if you
think about it uh
i mean you may read several definitions
but at the end of the day
the core of it is about uh
distinguishing right versus
wrong behavior right so that's uh what's
at the core of
ethics this may make it sound like this
is purely like a
an individual's trait like being ethical
but if you think about it uh like
even in the the classical foundations of
ethics and individuals behavior is not
purely
individual it's shaped by others around
the individual so things like
family and friends which expands into
the the broader society
so that's what it means is whenever we
want to study ethics
it's important that we try to understand
one's behavior with respect to
others and this is also evident when you
look at
several ethical dilemmas so ethical
dilemmas are basically situations uh
where there are no obviously good
choices so
uh you can start by looking at uh
classic trolley problems
uh somewhat overstudied in my opinion so
there's always like these scenarios
where
these hypothetical scenarios where the
trolley can go in like one track or the
other track
like kill 10 percent versus like 10
people versus like one child and all
these kinds of situations right
like what is the best thing for the
tally to do ethically speaking
uh i mean you can also look at other
scenarios like for example if you think
of
les miserables so the
the main character so i mean he steals
a loaf of bread of course like stealing
his bad like ethically speaking
but at the same time
uh i mean he stole the piece of
the loaf of bread to feed a child right
so then uh
is stealing really a bad thing because
that was with a good cause
and again you don't have to think of
like the hypothetical scenarios right so
you can also think of like for example
think of like a
like an ambulance that is speeding to
the hospital so what is the extent to
which
it can speed uh on on the one hand uh
the fast rate goes the higher the
chances of saving the patient in the
ambulance
but at the same time if it goes too fast
then there is the higher likelihood of
accidents and also harming others so
like then
considering these things what is
ethically speaking
the right amount of speed at which the
ambulance should travel
i also want to highlight that ethics is
not only are the ethical dilemmas are
not only about this
hypothetical or extreme like life and
death type of scenarios
so ethical dilemmas arise also in uh
very mundane
day-to-day type of scenarios uh for
example think about uh
like uh one answering a phone in the
meeting right so
like when like a meeting like this one
so you're in a meeting
let's say it's more appropriate if it
was like a real physical meeting
so um can one take a phone call
uh in between the meeting uh it doesn't
have to be the presenter it cannot be
the people in the audience as it often
happens
uh well on the one hand actually it's a
bad thing to do so it disrupts
the meeting and it also affects other
people who are taking their time to be
in the
meeting but at the same time the person
that's calling you
uh maybe let's say that's one of your
friends
who is in some sort of an accident and
uh need your help
and of course at that stage actually
taking that phone call taking that uh
phone call it does help the uh the
your friend so uh so it can be justified
from that perspective
now you can take it even further right
so once you take a call
so you have an option so if you don't do
anything then other people will think
that you're rude
or you probably want to tell other
people that you took a call if not right
now
later that you took the call because uh
somebody needed your help and
that was important so that you can avoid
the sanctions from these other people
but i mean is that okay now considering
that you violated the privacy of the
person uh you said that actually your
friend
was in an accident and telling that to
other people creates a
an impression that your friend was a bad
driver so i mean if you start thinking
about these things like ethical dilemmas
they arising like uh
several scenarios uh in like day-to-day
type of
activities as well so broadly uh what
we're interested in
is like i mean if agents were to be
uh or like if ai was to be in this
decision-making process how can it
understand these
scenarios and help make decisions
so one thing that you may have noticed
in all these examples
is that all of these are inherently uh
multi-agent
settings so in the trolley problems you
always have a decision like considering
one person or the other person
similarly with the speeding hospital i
said that there's a patient in the
ambulance and then there's other people
who may get harmed if
you go too fast answering your phone
then i think that people in a meeting
and then there's this other person who
was calling you
so in almost all scenarios whenever you
talk about ethics there's always like
multiple parties involved and the
decision is always about
what is right or wrong considering like
this multi-agent setting as a whole
[Music]
however the dominant view for the
the large majority of works that you see
in literature today like i'm talking
about computer science literature
are typically about making a one agent
or uh
one algorithm uh ethical so i i see that
at least there are three
like major problems in this view so the
first problem is that
i mean ethics is not really about the
technical entity right so
i mean what does it really mean to say
that actually like let's say
accountability as the ethical property
to say that actually the software is
accountable
right so in almost all cases uh there's
always like a principle behind
this technical entity which is a social
actor and the
the the ethical burden lies on the
the principle so if your autonomous car
were to be in an accident i mean what
does it really mean to hold the car
accountable
i mean it could be the company who
manufactured the car or it could be
the person who owns the car but the
accountability always is with the
the principle behind the the technology
so
um so that's the first problem so trying
to make the algorithm itself or the
the machine itself uh ethical without
thinking about the broader
social context in which the mission is
operating
so uh the second problem uh
is uh thinking that you can make uh one
entity like
like like the car on its own or the the
algorithm on its own
ethical like consider for example an
application that makes uh
recommendations about mod gauge in a
bank so broadly let's say
the bank aspects to be fair in making
its market decisions
and that's why this is uh i mean in the
machine learning literature especially
in the last decade or so
there have been several algorithms
metrics and measures for
uh making the algorithm itself
ostensibly fair
so why do i say ostensibly fair because
actually there have been several studies
that
especially recently that show that uh i
mean none of these
algorithms in the uh long run
uh yield equitable outcomes to all
participants
so i mean you may adjust the gradients
of the algorithm a certain way or
tweak the parameters of the algorithm a
certain way but often actually it
happens that actually the unfairness is
because
let's say the machine is being trained
on data from the previous bank managers
who made the lending decisions
if those managers were uh unfair in how
they made the decisions
and if that is the that on which uh your
mortgage algorithm is a trend
then obviously the best the algorithm
can learn is
this biased decisions that the previous
managers made
and again it's not just like one
algorithm right so the whole let's say
the mod cage process i mean there is
like a what case recommendation
i don't know there is like a an audit
department or
maybe there is some document processing
department i mean all of them need to
make certain decisions eventually
to make sure that the decision is uh
fair
focusing purely on making an individual
algorithm fair and that doesn't uh solve
the problem especially
in the long run so what we need like i
said
is you want to think of software systems
in general
as broadly situated in a society of
stakeholders
and ethical considerations really arise
in how the
the social actors or the stakeholders
interact with each other
and our goal is to design these systems
in such a way
um again it becomes like a difficult
problem so i'll show you in a bit
so design this uh systems in such a way
uh not in an atomistic manner trying to
uh
make only like like a part of it fail
but the system as a whole uh fair so
that's uh
the the broader objective of what we
have been doing so you can push it
one more step further so i mean you can
think of as
a software like in the previous slide as
a decision support
systems but i think more futuristically
you can
also think of these uh software systems
as agents that uh
act and interact on behalf of
stakeholders in that
they also make several uh proactive
decisions
so so then i think the question really
is uh i mean we're thinking of this
uh ai system it's not it's no more like
a single algorithm
no more like uh like a box of computer
but it's a broader
uh multi asian system uh which is like a
micro society i'm saying a micro society
because like within the larger human
society you can have several
uh multiagent systems depending on the
application scenario each of them is a
micro
uh system and the agents in this
micro society they reflect the autonomy
of its uh
stakeholders and it's important to note
that i mean
i think i mentioned it one of the slides
before autonomy and automation are not
to be confused
i mean often actually when people say
autonomy they mean automations like more
and more things are done in an automated
manner
but that doesn't really mean that the
software or the principle is
flexible to do whatever it wants to do
and that's what autonomy is
all about and of course autonomy and
accountability so they mirror each other
so one is only autonomous to the extent
to which they are accountable for
whatever they are free to do uh i mean
if you want to understand concepts like
this and model concepts like this
uh the essential argument is that you
want to think of systems as
multi-agent systems or broadly like like
what i'll show in the next slide as
uh social technical systems otherwise uh
whatever ethical outcomes or ethical
properties that you may realize in
technology
uh would be fairly shallow in my opinion
so
um just to um
[Music]
okay or maybe i think i'll just take a
bit of a
small break here to see how you're doing
and if there are any questions so far
is there anything on chats luciano i
cannot really see the chat screen
no there's not a chat but someone had
any questions
please just speak up or raise your hand
we have any ques
i guess that everything is is very clear
so far
i do have some questions but i'll keep
them for
for the end then sounds good yeah let's
keep going then
uh go ahead thank you yeah so um i
wanted to take uh
uh i want to describe our vision of what
a social technical system is uh
in a little more detail like i said uh
um whenever you want to think of like
engineering an ai system uh it's a
strong argument that you want to think
of like engineering it as a broader
social technical system
which has both the the social tier so
the principles or the
when i say principle i mean like humans
and organizations
so they interact in the the social tier
and
many of the uh the ethical challenges
arise in the social care
and what we also want is a technical
tier where agents can
represent the the principles and of
course the agent's behavior
affects the principle and there is a one
additional layer to it it's not about
like also engineering the individual
agents
what you also want is you want a
specification or a model of a social
technical system that can govern the
interaction between these individual
agents
and one other point that i forgot to
mention earlier is also that
what's important is that you want to
think of these social technical systems
as a decentralized multi-agent systems
actually even in the multi-asian systems
community they've been quite a few
works which confused like distribution
with the decentralization
i mean the fact that uh let's say each
of us uh
uh so i talked about like let's say
you're using uh
an app for managing your the ringer on
your phone
um or any other app for that matter the
fact that each of us have a different
app installed on our computer
that in itself doesn't make it uh
decentralized
so if all of these apps are essentially
talking to like a centralized server
like google or apple or whatever that's
sitting somewhere i mean conceptually
this is uh
as bad as like a like a centralized
system that's like a claim server system
everybody talking to
one server now when you say
decentralized what you really mean is
that
each agent uh has its own control and
the corresponding principle
is the control for that agent and of
course you will still need some
centralized
entities to uh realize this broader
system as whole
but you don't really need this
centralized system for governing the
system so
each agents can enact their part of the
broader protocol
like we say to realize the social
technical system as a whole
uh yeah uh yeah and then like i was
saying uh
uh when we say sts again it's not really
like a separate
running system so it's not like you
engineer a system and then you put that
system on a set of computers and they
transit
so the idea is that you engineer
individual agents and you specify the
broader social technical
system as a whole and this when when
these agents are put to use actually
their interaction is uh
what makes up the the social technical
system um so like we say
in a setting like this you want your
like whenever
the agent wants to make an ethical
decision
you need to have a user model to
understand what are the the
the values and the preferences of the
the user you also want some aspects that
are at a
societal level or at a social level
things like nonsense sanctions because
these are expectations from others
and then you also need some technical
capability like like reasoning about the
context and
what actions to take and things like
that all of this have to go uh in the
into the decision model of the agent
so the broader technical challenge then
is how can we engineer
such a decentralized multi-agent system
so like i was saying you need both
so on the one hand you want to model the
individual agents and some things can be
modeled at the agent level like for
example
the values of the individual users and
their value preferences
they can be learned at an individual
level
but this will influence what norms you
want to use for specifying the social
technical system
as a whole so the norms must be modeled
then at the level of
social technical systems and again once
you have a set of norms info
in place actually they can influence the
values of the system so it's a process
that goes
hand in hand uh so the agents
like on the one hand to specify the
social technical system
the stakeholders have to come together
and negotiate with each other in order
to come up with whatever are the
appropriate norms of the system and when
you
develop the reasoning for the individual
agent and that takes into account
the values as well as norms that were
specified at the the social technical
system
and then uh so this relates also to the
the
the ethical properties that you desire
like things like fairness transparency
and accountability like here so know
that actually each of these
concepts actually there's you can think
of them at the social technical system
then you can also mirror them and think
about it like a projection of that at
the individual agent level
and this goes all the way from like
designing and developing the agent to
like verification and validation
for example if you want to verify the
individual agent you can verify that
with respect to norms of the the social
technical system so that was specified
at the level of sts
and if you want to validate uh the sds
then you can do with the values of the
individual members of the system
so in a sense what i want to convey from
this picture is that this development of
individual agents and specifying
the social technical system as a whole i
mean that has to go uh hand in hand
it doesn't mean that you have to do it
uh one shot i mean so this is going to
be an iterative process like uh
i'll show you in a bit uh and then you
start with certain
version so you may start with certain
like values and design your individual
agents
then you would notice once the agents
are put to use that the agents are
repeatedly getting sanctions sanctions
for violating some non
norms so that means that actually either
there is a problem in um
the way the agent models the values are
there could also be a problem in
the way norms are put together in the
place and so one of them
requires uh changes to it
so um in a sense the
the so what we have been doing uh for
for
at least like seven eight years now is
uh developing uh
uh broadly uh a methodology for
specifying the social technical system
and uh the agents that operate within
that
social technical system at a very high
level so the way it works is
you start by identifying the individual
stakeholders
value preferences and then you can
specify norms that support those value
preferences
and of course you would refine the norms
uh
as we go and norms are what are crucial
for accountability when
things don't go as expected then you can
see who you can hold account
account for using the norms in the
system
and then the the individual agents uh i
mean they take one or more roles in the
the sds as a whole and they carry out
their part of the enactment
so each agent has to carry out its part
of the enactment and the enactment
itself is
enforced decently there is no one
central entity that enforces this uh
enactment now when you have a system
like this then you could
evaluate the outcomes of the system with
respect to uh the values promoted or the
norm satisfied and things like that
and then you iterate this process over
the period of the
the sds of course it's a fairly abstract
uh
method as a whole and it seems like easy
enough when you look at the whole uh
methodology but of course realizing each
and every part of it is like
like a challenge on its own so what we
have been doing is like uh
we're developing like a suite of methods
some of these are
more methodological some are more
technical uh in realizing this broader
overview of how you specify an sds and
uh
how do you specify individual agents in
that sds so we've been working on like
for example the most recent work here is
about
identifying contextually relevant values
and then we
did some earlier work which shows how
you can incorporate values in agent
oriented models
and you can also use both values and
norms for
reasoning about what actions to take i
also did work on how communicating
values how can agents
communicate their values with each other
so as to receive positive sanctions not
negative sanctions uh then of course
there are going to be conflicts in a
system like this so how can you identify
those conflicts and
resolve those conflicts so uh i'm not
going to of course talk about
all these works uh in this presentation
but just to give a flavor of what we are
doing uh i'll talk about
like values and what we're doing with
the values especially the recent work
and then i'll talk about how values can
be used to inform
norms of the system okay
i'm just uh taking a bit of a pause to
see if there is anybody
there's any comments yeah um but
we have a question here from fgm
about a figure you showed early on about
that you
you divide the social technical cease
and the human asian duel
and there is some i believe he refers to
that one
uh if jenny do you wanna can i
should i read it you wanna yeah myself
thanks thanks for saying
thank you hyperdeep uh yeah thanks very
much for very much presentation
so far so um it's about an earlier
figure you showed
uh where you showed how um so yeah i
think it was here yeah so how norms
values get goals here value preferences
get realized
in different parts of the system so if i
understand correctly you show that norms
and values are realized in
technical parts of the system while the
social side of the system regulates
uh but i i would say that
socio-technical
socio-technical design uh approach
requires us to realize values and norms
in both social and tech social and
technical aspects so it's as much about
designing
interaction between humans through uh
organizational practices
social infrastructure as it is about
designing interaction between humans and
technical artifacts so it's
the synthesis of all uh of these
together the humans
uh and the technical aspects that uh
gives the socio-technical system it's
emerging desired properties would you
agree so yeah yeah
yeah sure i think the social
infrastructure and uh the technical
infrastructure as part of it
yeah i am and i agree i'm not sure uh
which aspect of it when i convert i
think i fully agree with what you're
saying and that's exactly what uh
i want to convey as well so the
distinction between uh like uh
like i say the social and the technical
theory is more like
a social there is where like humans
operate and technical theories where
the the agents or robots or whatever
they operate and of course
the real challenge is uh i mean taking
some of these abstractions that uh
the principles using the the social tier
and coming up with the technical
abstractions that can be realized in uh
technical entities i mean that's what uh
specifying the social technical system
uh is all about in my opinion and of
course it's not easy
yeah i think so so thanks for clarifying
so because when i looked at the figure i
saw that
so the green lines that you have they
show that the realization happens in the
technical tier but i see the for to the
social tier i see the
regulate uh the regulate line so then
i was i just wanted to uh to check with
you uh
because like basically what i'm saying i
i would i would extend those of the re
the realization you know
where we implement the norms it's as
much about the social tier
as it is about detect and to them
together interacting with each other
yeah so i mean what we mean here is uh i
mean yeah of course uh uh
i mean if you're if you're thinking
about uh like even broader issues like
for example like things like
loss and so on uh then of course some of
these norms and so on they need to be
realized in the the social tier as well
so that's a like definitely like a
dimension there but at least here what i
want to convey is that i mean we are
really interested in uh
like how can you realize the the social
abstractions
in the technical tier and that's why
you're saying uh uh the realized in for
the technical tier
and the regulate for the the social tier
because this is something that
we're not asking principles on how to
behave i mean of course that's again
this another domain like like for
example uh
uh the policies and things like that
so they may work on that but at least
the scope here is that uh i mean you're
not
trying to um come up with like uh
like rules for how humans or
organizations should behave assuming
that
they are behaving a certain way based on
whatever is the framework
how can you sort of like translate that
to the technical deal that was the the
focus here
well so just well i don't want to delay
it right but no so so my point is
basically
i'm i'm saying that we should be
thinking about designing them jointly
together so basically to your point that
you mentioned earlier in the
presentation like with fairness right
you said that it doesn't make much sense
to if you try to
avoid discrimination to focus just on
the algorithm that you need to think
about
the larger practices which are around it
right so
because all of it interacts together
basically i'm saying like that
it requires invites requires basically
designing
yeah i think that's indeed the right
intuition i think i agree that you
cannot
like i mean just like if you want to
make any rules about like how uh
like i don't know that the humans in a
smart car should behave
then that has to go hand in hand with
the rest of it i think that's what
you're saying so not only about uh
you should have to come up with norms
for how smart comes should behave but
also about norms about how
humans in those smart cars should behave
yeah i agree i think that's that's the
right intuition i think
yeah thanks very much thanks cool yeah
it's good thanks
nice nice discussions do you have any
other questions
right now all right
i i think for now you can yeah go ahead
thank you okay so i'll uh keep going
um yeah so like i said
uh how are we doing what time now is 3
32 so we already have some time
yeah so basically uh what i'll do is um
so especially this part of the
presentation i think i'm going to talk
about a few of our papers
at like some more concrete details now i
will go through it uh
fast but of course uh if you want more
details
please feel free to ask me as i speak
so first of all so i want to talk about
talk a bit about values and why are we
even talking about values i mean what
are values to start with so essentially
uh values uh when you think of values as
a schwarz
we think of what is important to us in
life
and then why are we talking about values
because if you want to make any ethical
decision
you need some building blocks to able to
be able to reason about
what's right and what's wrong and values
are
one such building blocks for reasoning
about
ethics uh and uh schwarz
schwartz describes here all values have
uh
like characterizes like six main
features
so these are priorities that guide
actions i think that's important from a
technical perspective because you want
to eventually use values in
making decisions and these are beliefs
that are intrinsically
linked to effect so so it's all related
to effects i want to don't want to go
there
these also refer to goals so it's like
each of them has a clear
motivational goal and values also
transcend context
so it's not something that is applicable
in only one particular scenario but they
can be applied in a variety of different
scenarios and of course this is uh uh
challenging right so
that depends on the abstraction at which
we are looking at values and i'll talk
more about this
transcending context aspect in a bit and
there are also the standards criteria
so and that's what uh that's how we
could justify the
the rightness or wrongness of one's
action based on
what values they they hold especially
when there is no obviously
uh right or wrong answer and they can be
ordered by
importance so that's what uh
distinguishes one person from the other
although the values are universal and
all values are applicable to all people
and how you ascribe relative importance
among these values
changes from person to person and that's
what makes these
preferences also subjective uh
just a bit on like like a motivation as
to like why these are
universally applicable so in a sense
like
humans as a social beings
so all of us have like several basic
requirements so i mean you can look at
the the maslow's pyramid
so which range all the way from like
biological and physiological
requirements as well as requirements of
the social requirements like uh
love and uh belongingness and all the
way to like more about
self-actualization and more more
individual type of
requirements in several of these in
pursuit of several of these values
what you need is a cooperation of other
people
and essentially then what you need is a
language to communicate your needs with
other people and that's how and values
are that language
so you can use values to express to
other people where you want to
do something a certain way and then they
may be interested in
cooperating with you as well so that's a
bit of a motivation about um
like what values are and why they are
universal
but the question is still can you use
values
in like decision making context as is
like for example what i show here is uh
10 values that
schwartz identifies in his model so
they're on two dimension
so some of these values are
more self-centered so achievement in
power and the others are more about
other centers things like universalism
and benevolence and of course you will
have preferences among these like i mean
each person will have preferences among
these
and the other dimension is um some
values are more about
conserving uh the past or with the
tradition
and the other values are more about uh
openness to change right so this is ten
values along these uh two dimensions
but the question is um like how can you
use these values in uh
like engineering uh specific apps like
for example um
we had a scenario uh some time ago let's
say you're trying to develop an app
in this pandemic times
so you want to build an app that would
recommend you when to go out and when to
not go out right so depending on
i mean whatever factors the crowding or
the the infection rate i mean whatever
it may be
i mean can you just take values like
the ones that i show here and engineer
uh an app like that
and it turns out that that's not trivial
so firstly these
these values are i mean not all of these
values will be applicable
in the particular context that i'm
talking about and again
how you interpret these values could be
quite different
from one context to other context so
what we have been doing
what we did recently is we developed a
methodology for
coming up with values that are
specific to a particular context so they
are applicable in that context at the
same time
they have an interpretation that is
specific to that context
so this is a hybrid methodology so this
is where i think uh
uh some of our hi technics also come
into picture i mean
humans identifying these values at the
same time automated techniques in this
case uh natural language processing in
particular
is assisting the annotators in coming up
with these values it's not like uh
some designer thought that these are the
values that are applicable in this
context uh
on their own so this is more of a data
driven approach and where do we get data
for doing
things like this we look at discourse
like for example um
for covet related values so you can look
at uh like what people are talking about
uh covet and what they like and what
they don't like uh
so this is like for example like a
reddit discussion about uh
uh protests about uh lockdown measures
uh and you could so start look like once
an annotator start looking at a
discourse like this then they can
identify what values may be
relevant to users in this context or not
i want to get into the details of the
method
but again we don't have to this doesn't
have to be reddit data right so we
actually did it with the
data from a large scale survey that was
conducted at tpm
uh by a neat motor so it's called the
participatory value evaluation
but you can also use like for example
like for specific uh like let's say for
mobile apps you can also look at their
application reviews
to see what kinds of things users value
in that application and things like that
and this all gives data for you to
identify values that are
relevant in a particular context and we
also did some experiments to show that
so the methodology that we had yields
values that are i mean first of all
specific to this context i mean if you
identify values in a different context
then you're going to get like a
different set of values not the same
and also the methodology is repeatable
in the sense that it's not specific to
people who apply it now yeah i'm happy
to talk more about it afterwards
so once you know what are the values
that are in play in a particular
scenario
you also want uh some mechanisms to
incorporate these values in the the
model of an agent
right so coming up with like a list of
values maybe that's fine for like
like a policy maker to look at it and
say what to do in a particular scenario
but if you really want like agents to
make decisions from uh like
abstractions like values you want some
technical abstractions to incorporate
these
values into the the design and also the
eventually the source code of the
the agent so i think for that i think
there is a quite a bit of work
in the multi-asian systems community
broadly about agent-oriented software
engineering
which has these high level abstractions
things like
actors goals plans beliefs and
dependencies
and as we noted earlier so values
essentially refer to goals
and you can incorporate values in an
agent modded
directly using the goals you remember i
talked about um
this intelligent ringer application so
essentially this application is uh
trying to decide whether to ring your
phone uh like silent or like cloud or
vibrate it automatically does that
depending on whatever it thinks is uh
ethically appropriate like if you think
of that scenario i think
there'll be like values like privacy
that are important
and you don't want to disturb others
that's a value and you also want to be
reachable by people and these are the
values that are specific
to this context first that can be
identified using
axis the methodology that i described
earlier and then using the other methods
that we have described
you can incorporate them as part of your
agent model
i mean note that when i talk about these
asian models these are not just figures
right so these are
formal models so they can be
computationally represented
and they can even be refined to such an
extent that eventually they can be
addressed to the the actual code of the
agent so there are
two advantages of models like that so
one uh it makes the
reasoning process of the uh the designer
explicit
uh which we have shown in some
experiments that when somebody else has
to maintain your application
they better understand why a certain
thing was done a certain way looking at
this
agent models and the other advantage is
also from the the end user's perspective
like for example i mean you know what
are the values that are
under play in a particular context but
you still want to know
like what values one user prefers uh
over another value so it's the value
preferences
uh which is user specific and if you
want to elicit that
uh i mean i mean instead of giving like
uh like a flat service to users and
asking them what
they value more you can start asking
these questions in particular context
so these models will have like
placeholders where you could say so this
is something
that i know that actually like for
example in this case um
[Music]
so deciding whether somebody wants to
like
whether they value to be reachable four
by four more
or to work interrupted by four more you
need to ask question there
and we're saying that depends on the the
relationship to the caller and whatever
is the context of the neighbors
and then you can ask this question to
the user in this particular context
so so essentially you have a model uh
like this
and that's realized as the software
agent and the software agent is put to
use
whenever this context is recognized
that's the time at which the the
software would ask this particular
question to the user
um and again i think there are some more
details here and how you ask that like
in specifically we use active learning
for doing that so this way you could
also combine the the more of the
symbolic approach which is this is the
symbolic approach
and then actual learning of this context
is more from a machine learning
perspective you learn that from the data
you can combine both the the learning
and the symbolic approaches and doing
things like that
so that's a bit about values
and then i want to slowly transition to
also the notion of
norms so values are fine so if you want
like like i said you can develop
individual agents that can
make decisions uh for the individual
user
so you can do it based on what values a
particular action promotes and what
values uh an individual action demods
and then you can come up with the
balance but it's more complicated than
that
because a lot of the times it's not just
values that influence action but it's
also the expectations of
other people like for example in this
case i think let's say we are talking
about an app
which is like a location sharing app so
the app automatically shares your
location with the other people
and who it shares with when it shares
with depends on uh whatever is ethically
appropriate
and of course you can so like let's say
frank is the user who is using this app
he prefers pleasure and recognition as
values
so the app can share his location
whenever it thinks that
it is promoting these uh values
but uh but it's possible let's say that
um
his mother grace so let's say frank is a
teenager
he she cares for frank's safety
so then i think it's in her interest
also that frank shares his
location whenever so even when these
values are not promoted uh probably
grace wants his
location to be shared similarly let's
say like frank is visiting
his aunt hope uh but again uh
hope prefers privacy so the frank's
action to share with all uh
promotes pleasure for him but i mean the
fact that he is with the hope
if he shares this location that is also
sharing uh hope's location
that violates uh hope's privacy so then
how can you make decision like ethical
decisions in this case um
so the values that the individuals care
for in itself is not
sufficient we also want to know what are
the expectations of
other people about your behavior and
that's where norms come into picture
so in essence uh norm uh again
i'm using it is in the the word norm in
a technical sense
so it's a directed social expectation
between principles
so although they can be like a variety
of expectations so we can like
many uh common uh interactions we can
imagine that
these expectations can be thought of as
like very few different
forms like things like commitments
prohibition authorization and power
in that setting essentially you would
say for in each norm it says
what is the subject and what's the
object what's the antecedent and what's
the consequent
so that's essentially like the the core
of the norm
so it says that like for example for
commitment it says like the subject is
committed to the object
when the antecedent holds it will make
sure that the the consequent will
come into picture so um again there are
two obvious advantages of using norms
one is it makes the expectations
explicit so whenever something doesn't
happen
you exactly know who is accountable for
that
action and the other advantage is that
you don't need to invent them for each
and every individual application
so you can specify the the norms for the
application but you don't need to
implement them
in every single agent you can have like
a like a norm machine or like a like um
like a library that like verifies norms
or enacts them
at the same library things like a
middleware can be used in all different
agents
individual agents will have different
norms but the actual realization of that
individual agents don't need to do that
can be done in something that is shared
across
agents uh again i'll
go through that a bit faster i also have
an example it's about like showing some
examples of
norms in a healthcare setting i will
skip through these things but i would
like to mention that again these are not
set in stone like a designer will
specify norms to start with
but uh once the uh
the system is like starts operating the
norms can start
we can start like refining the norms
like for example here uh to start with
uh
like this healthcare social technical
system there was a prohibition which
said that um
uh so so the hospital prohibits
physicians from
sharing the the personal information of
the patient like outside the the
hospital and in
all all circumstances so that was the
the original norm
and then later you realize that let's
say there is like an emergency and your
hospital is
overcrowded and there are like let's say
uh other doctors
which are from outside the hospital who
are volunteering
in your hospital to help deal with the
the
the overcrowding at that stage like i
mean you know that actually the norm
will be violated
but that's okay norms are meant to be
violated and like uh like rigid rules
and if it's like broken more than once
you know that actually there is
something wrong and then you can
refine the number now we can say that
unless this is an emergency the hospital
prohibits
the physician from sharing the phi
outside the
deficiencies of the hospital so that's
the general idea here
um and again i think in the interest of
time i'm gonna
uh skip this so so what this leads to is
like a fairly complex system so you have
like individuals values and you have
norms of the system and also these are
evolving over time and you need some
mechanisms to reason about these things
automatically and one of the methods
that we developed
uses a multi-criteria decision making so
is the method called
wiker so given all the stakeholders that
are involved in a particular decision
context
knowing their value preferences and
knowing their norms so you can come up
with
again you need to use some theory again
this is where i think uh
um how the technical
artifacts can benefit from research in
more humanities uh
disciplines like the utilitarianism and
egalitarianism there are like known
ethical
sort of stances in the literature and
then you could realize them
as part of this reasoning framework
okay so with that said i want to go to
the last part and i think this quickly i
want to run through this
so essentially i want to conclude by
saying that um
so what i did so far uh i introduced
this notion of
social technical system and then i
argued that in order to reason about
ethics
you cannot think of like purely
technically designing one algorithm or
one machine you want to think about
engineering the social technical system
as a whole
and then again social technical system
is like not one running entity or like
one computational entity that you would
uh
engineer you want uh you want to
engineer individual agents in the system
and you want to engineer them such a way
that when they are put to
put together to use in a society and the
entire social technical system is
realized
and for that besides engineering the
individual agency also want to come up
with the specification for the social
technical
system as a whole uh so what are some
so i recognize that there are three
broad categories of challenges
in uh realizing this vision as a whole i
mean we have been doing quite a bit of
work but that
but it's like such a complicated problem
that i can imagine uh
spending uh rest of my career and you
know i don't know several other people
spending their careers on realizing this
vision fully so the first set of
challenges are about
modeling ethics so i talked about uh
norms and values
but if you start thinking about it uh
one user
he or she would participate in multiple
social technical system
and within the social technical system
there are multiple contexts and then the
value preferences and the norms will
change
depending on the settings and so on so
if you want the individuals to deal with
each and every of the scenarios uh
independently
i mean that is just like too much
information overload so what we still
need is
we need to we need some ways of
abstracting this to a
higher level so like for example one
thing that we've been thinking is uh
uh can we take these things like the
behavioral pattern
within a socio-technical system and
start
formulating some ethical postures so if
you know that actually this user
like acts in like one of these 10
ethical postures
and if you know which is the like with
ethical posture is appropriate
in which application and the context
combined
and then the users have to deal with
this 10 postures and refining and
updating those 10 postures
and not deal with each and every uh
decision point in
each application um
yeah so and there are again more
challenges here right so i only talked
about
values and norms but there are several
other concepts that need to be formally
modeled and defined things like guilt
and consent and inequity aversion and
things like that
and also about like formulating
pro-sociality in terms of the principles
of justice
i would say i mean these have been
studied in the
uh the humanities literature like the
philosophical literature or the
psychological literature
but what we really need is as like
computer scientists we want technical
abstractions to represent these things
and realize them as part of our social
technical systems
that to a large extent is still missing
in
computer science and then we also need
techniques for analyzing the ethicality
of the system so you model the system a
certain way
and then once the system is put to use
you want to see to what extent uh
the system is ethical and again you
could think of this doing this both
from a design time perspective where you
could extend some of the the formal
verification or the model checking type
of methods
but you can also do it uh in run time uh
by simulating the behavior because i
think many of these things uh
it would be really difficult to test uh
in the wild
and i think there are also some really
good opportunities in combining uh
this design time verification type of
approaches with runtime
simulation type of approaches so finally
i already alluded to it a bit so i mean
designers need to
model the system and then also need to
like users and designers they need to
verify
or the validate the system once it puts
to use but in the process
you also have the end users so many of
these things uh
the values or the norms of the system
they must be elicited from the the end
users
so and you want to do it uh
unobtrusively or un
intrusively so you don't want to ask
users too many questions
because then those solutions may not
work but you still want to give enough
flexibility that people who want to
specify can specify
and others can use fixed defaults of the
system but how do you do it in like a
balanced way i think that's a challenge
uh so one challenge is the level of
individual users and the other is also
at the level of
the sds as a whole and developing uh
deliberation and
uh negotiation type of techniques for uh
coming up with norms that everybody
agreed
yeah so with that uh i would like to end
my presentation
uh thanks for your attention uh i think
we gave like a longer tutorial i believe
it was about
uh two and a half hours recently at amaz
so
if you want to listen to it more or if
you want to get more resources please uh
uh go to this website uh so there's
longer description but
i'm happy to talk more now and take some
questions
yeah sorry i think it took longer than i
expected but uh
some time to ask questions okay great
yeah thank you very much that was really
fascinating a lot of
of course as you yourself mentioned a
lot of
research challenges a lot of things to
work on but you can see a lot of
great framework you were developed so
far so we have um
the first question uh here from derek
derek
should i really would you like to ask
yourself
sure with some compliments i thought it
was a great presentation a lot of
really good material and very specific
uh
so so thanks for sharing thank you
i guess the the two questions here are
um and i may have misunderstood but it
seems like a lot of this is
for kind of the initial system design
but um i'm trying to understand this in
the context of feedback loops
and a kind of continuous iteration over
time
are are there barriers to that or are we
missing
kind of any tools or frameworks for uh
applying this
to systems um iteratively yeah
no i know i think there are it's a great
question i think also
uh like like a main challenge i would
say so
i mean there are two types of uh like
feedback loops here right so one is i
think among developers i guess so i mean
you develop
one version of the app today and that
needs to be refined uh afterwards so
then you need to communicate actually
what was intended by the first developer
to the second developer
there is that and that's to some extent
i think the
the the software engineering community
is focused on that
but uh the other feedback loop is also
between like the designers
i think it goes beyond like designers it
could also be like regulators and things
like that
to the the actual stakeholders or the
end users of the system
uh and that's something that we have
been trying to
sort of um at least make an initial
attempt at bridging
so like the designer comes up with an
initial specification of the system
and says like within the design what are
the points at which
we want the end user feedback for
example as a designer i can say that
actually these two values are
in play in this setting but i like which
value is more important for one user or
the other user those are the ones that
the end user is supposed to say
so the fact that you can put that as
part of your app actually then the app
knows
uh the app being intelligent is what i
mean who knows when to ask the user that
question so it can ask that question
in context also creating these feedback
loops
between the designers and the end users
is important and then also these other
challenges right like for example i also
did some work in the requirements
engineering community so they're more
like a passive feedback loops there like
somebody uses a an app and then writes a
review about that application or
discusses about that application in some
forum
and we can also mine like forums like
that to understand what the users want
and then
update the designs uh accordingly but
yeah
but overall i think it's uh like a
fascinating challenge and i don't think
uh
i mean i do think that there's quite a
bit of work to be done here
great yeah uh yeah thanks for the
questions eric
uh we have another question now from
jenny go ahead
yeah thanks for saying uh pretty so
my question is about studying values and
eliciting preferences so
uh at the design stage um
i mean so for those of us who come from
computer science the technical side of
things
uh who are the other experts you would
say that need to be at the table with us
in order to a design stage study uh
people's values people's needs
interpretations
uh like for example do we need to work
together with uh
ui designers ux designers in order to do
that and
also do you think it's necessary to well
depending on the context
use qualitatively rich methods to do
that for example
interviews methods from ethnography and
so forth
yeah yeah so yeah indeed i think that's
a good question i mean i mean to start
with i think uh
so the first sort of criteria i would
say for anybody who wants to be part of
this process is uh
an understanding of what values mean but
i think that's i assume is something
that's
doable i mean people can educate
themselves on that but the other part of
your question i think that relates to
what they call as
value sources in value sensitive design
i mean that's certainly i think
you need like like a mix of people right
so on the one hand the value resources
can be i mean i would say
the most important thing is the actual
end users of the technology
if not all of them i think you
definitely need a representation of
those users to be part of this uh
like value specification or value
preference solicitation process
and some of these values i think they
also come from designers and developers
so
although people may want to be
objective i think there will be all
kinds of unconscious biases and
so the designers values may eventually
like surface in the applications you
also want
of course them to be um the part of this
value specification process at design
time and then
there was one other value sources that i
think are missing
[Music]
yeah but i mean things like i mean this
again may be the same category as
developers and uh
designers but more like people who are
regulators
so i would like to to add social
scientists to that so
you really have a great specialist in in
in values
like uh hofstede
is is a good person to add to that and
they're more like that
yeah indeed i think so this is what i
meant
like when i said actually we want these
people to be like uh like
even the other people to be like
educated in values but i think you're
saying
definitely having people who are experts
in values itself they can certainly
uh bring other people on track if
they're missing something about values
great yeah thank you thank you very much
thank you jenny for the question i have
also um
i have a quick question myself also uh
you you mention as uh accountability
in the framework as also connecting and
respecting to
norms so i i just have like the the
but when you look us in moot agent
systems there is also this
yeah this agent did this because of the
interaction with one agent the
interaction with the other and the other
and then
you get some immersion pattern that you
were not necessarily expecting
and the connection to this idea chooses
like this problem of many
hands and many things which like in and
thus
diffusion of responsibility or
accountability
then accountability gaps so how do you
see this
in in the framework by connecting
accountability to norms
how can we keep track of this yeah yeah
i know that's a great question i think
uh
there's been some work that i know of
it's about uh like
delegating norms so to start with let's
say
uh the physician is prohibited by the
the hospital and maybe that's too vague
like like what does it really mean uh
like
the hospital right so it may be the the
the lord division of the the hospital so
uh i mean i don't think it's uh an easy
problem
but one way i can think of dealing with
that in the norm setting is uh i mean
you can
you can you can update uh the the
subjects and objects
and of course each time you object that
then you need to know actually what are
the other norms that are related to it
and you also need to update them uh
there so otherwise you would create uh
asymmetries between so there's one
expectation here but there's a different
expectation at the other time
but all these things i think um the
advantage of doing it in a declarative
way like i said
is that each time something changes you
only need to change the specification
the declaration you don't really need to
change
excuse me you don't really need to
change the underlying
agent implementation i think that's
definitely something to
uh like like a virtue of this approach
so things will change and
like things will go out of control and
they'll evolve over time
but each time you don't have to update
the implementations of the agents rather
you need to just deal with the
specification and
update the the representation or the
symbols so to say
yeah great so yeah to have this dynamic
a little bit bottom up
great thank you very much so uh we ran
out of time
so yeah thank you very much uh pradeep
it was really interesting talk really
nice discussions
um if you have uh
one or two more minutes we have a final
question
if you if that's okay yeah i could do
that okay
so thank you everyone for joining uh
stefan
you had your hand and raised do you
wanna ask it
uh yeah sure uh thanks for the talk um
so one thing i was still wondering about
is how you were
handling conflicting values so i can
imagine that
you don't necessarily want to go for say
the value most people agree on because
there might be some values that are
normatively
more acceptable than others so that you
have some kind of ethical background
theory for
ranking values um so how how are you
treating that
problem yeah yeah i know i think that's
again uh like in general
the notion of conflicts i think is uh
like challenging right
so first of all i think there can be
conflicts uh
like within one agent so like i may have
said that uh
i prefer x to y uh like one time and i
may have said the other time that
i prefer y to x that's already like like
a sort of conflict like
conflict within and that's good to
realize to start with
uh so it's possible that uh that's like
a real
uh preference so i do mean i think in
this context before x to y and that
context i prefer
y to x but that's fine but it could also
be that actually like humans being
not always rational or like being
bounded rational you may have not
thought it through you may think that
actually you prefer x2y
but you really prefer y2x in both
settings like recognizing this
is already a starting point for uh sort
of like uh
prompting the user to think carefully uh
about this
that's that's still at the the level of
an individual uh
user now of course there will be
conflicts uh when you
like especially when you want to use
norms to inform
when you want to use values to inform
norms so i may have
genuinely one value preference and the
other stakeholder may have
another value preference so how can we
specify like one norm
where both of these stakeholders
interact so i think
that's i mean to some extent uh uh
i mean unavoidable but at the same time
it's not if you don't have to force
people to follow whatever is uh
the the majority so i can still uh
follow whatever i think is uh more
valuable to me i take actions like that
knowing that actually if i do that
actually these are the sanctions that
i'm gonna get
like like for example um like the norm
is for only
a physician to administer a certain drug
uh but uh it could be like a like a like
a like a patient is
dying and the doctor is not there and
the nurse knows that actually to save
the patient's life for this is the
uh the drug that needs to be
administered there
so even though that action was violating
the norm so if that is in accordance
with the value preferences of the
the nurse she may still uh administer
that particular drug so that would
eventually then mean that actually
there was something wrong with the norm
right so that you could recognize and
then you could update
the norms over time with the same
motivation so
yeah i mean i think it's a challenging
thing and also i think some of the other
things that we are doing with respect to
these
negotiations instead of like one
designer
manually specifying the norms of the
system like when you say one person
specifying the norms
if these norms were to be specified as
part of the interaction between the
stakeholders even from the very
beginning during the design time
then it's likely that you know like what
were the norms and why they were uh
put together in the first place to start
with yeah so
that's yes
you start with the preferences and then
adjust the norms based on those
yeah indeed yeah okay so yeah i think
yeah we have to wrap up so yeah
thanks everyone thanks again pradeep
very nice
and yeah see you all soon on the next
yeah takakora
thank you all i think i appreciate again
for inviting me here and uh i would be
happy to talk to
any of you more about these things or
also other things that i'm interested in
uh
please feel free to send me an email and
we can set up a meeting
thanks bye