214. Consequences of Misaligned AI

all right
hello welcome to another session of the
ai safety reading group
today we'll be discussing consequences
of misaligned ai
now this paper formalizes
in a way a certain idea that's been
discussed a lot in the ai safety
community
the fact that human
goals tend to be fairly complex
and if we miss a part of them that can
lead to pretty catastrophic results
this has been discussed in some toy
models
informal models in passing etc over the
years quite a few times
for example
some work from chai recently eg the
paper when robot should be obedient
notices that for certain ai designs and
types of interactions with humans
if you miss features from
[Music]
your utility function by features i mean
things that you think that are relevant
in the world for the ai to take note of
and to care about then that will result
in a different sort of behavior
from the case where you've included
everything that is relevant
to the utility function
in fact they encode a procedure to check
for such
divergent patterns to see if features
are missing
and then to switch the a off if that is
the case
because otherwise the ai might start
being disobedient
in strange ways or
it might lose utility for substantial
periods of time or so on
but they didn't really quantify exactly
how much utility would be lost
and stuart armstrong for example
explored this from another angle and try
to see
what sort of missing features do we
usually not include
which would do away with a lot of
utility laws
what in other words what are the things
that we
usually expect utility functions
nowadays
to miss out on
and he came up with some interesting
ideas um
i think the ref right reference here is
anthropomorphizing ai
to sequence on that strong but this is
the first time that i've seen someone do
this particular sort of model
so i think it's worthwhile just for the
fact that it
people can point to this and say hey
look uh if you miss features from your
youtube
utility function then you are guaranteed
to
in some cases lose utility and it's also
not too hard to see that you could
generalize this in some
[Music]
ways that are fairly desirable because
as it stands
the assumptions in the paper are a bit
weird
anyway let's just get on to it right
so how does the paper model things it
says that
suppose the world
has some parts of it that you care about
for example you're designing a
recommendation algorithm
and you care about ad revenue the
quality of your product the impact you
have on society
the diversity of the content you
recommend
and you can model these using some
numbers let's say
using four numbers for each of those
things right
and you know that no matter what there
are some restrictions on how high these
numbers can go
for example because a person can only
click on so many ads at a time
so you can only get so much ad revenue
from them or because you're sure that
you can only
show so many different pieces of content
in a person's lifetime right like you
can only watch so many different shows
uh given the life you lead on earth
these are meant to be reflecting some of
the assumptions of the paper
where it's assuming that the attributes
you care about
cap out in some way that is the world
can't have any more than a certain
amount of these attributes
this is somewhat reasonable
in that you know it
you can just make the attributes so the
limit so high that is essentially
irrelevant in your day-to-day affairs so
we won't quibble that too much
and the particular
sort of limit is described by
a cost function where they say that
we're going to draw a boundary in the
space of attributes and say that
everything
on one side of this boundary is part of
the worlds that are allowed and
everything on the other side is part of
the worlds that aren't allowed
now of course you have a utility
function
over the attributes of the world that
you care about
and we assume that in principle
you like just having more of each
attribute so let's say that
again we're going back to the
recommendation algorithm
obviously if you get more ad revenue
your utility is always going to go up
if the quality of your product improves
then
the utility that you get will improve
because you know you have some pride in
creating a well crafted thing or so on
or for example obviously you want your
audience to enjoy your creation as much
as possible
so they make this assumption that your
utility function can only
increase with your attributes as you
have more of one of your attributes your
utility goes higher
and
that is kind of standard in economics
and is applicable to loads of situations
so you know
it's not too crazy a restriction right
and even if it
isn't the case even if your utility
function does eventually go down in some
place
you can say okay well like in those
cases they're like
those situations are so rare that they
don't really matter practically speaking
right for example say perhaps
for a human there can be such a thing
as too much pleasure right but it's very
unlikely that you're going to wind up in
such position
using the things you have today unless
the ai you are employing is quite
powerful
because we all know the standard case of
okay well the ai just
hooks you up to a system which floods
you with endorphins
in that case too much happiness is
probably a bad thing and your utility
goes down
but this paper is more dealing with
more everyday or
current ai
so again we'll just leave that
assumption for now and not really
argue against it so what about the ai
well the ai we're going to say considers
only some of the attributes we care
about but not all of them
this is of course pretty much always
true right like say
in the recommendation algorithm system
it is very hard
to find out how you can encode
the impact of your algorithm on the
well-being of the community
that's what features would you use for
that
how would you properly encode the
well-being of the community
in maybe just two or three numbers
that's a pretty
tricky thing to do as a person designing
this algorithm you're probably just
going to skip that and say ah forget it
i'll just encode um
in the utility function the amount of ad
revenue i get
and the diversity of the content i show
because those are somewhat easy to
measure
and i can just optimize for that surely
that's not going to be a problem
well so says the paper yes it will
eventually
and now what about the ai's utility
function well the utility function now
will just be another thing that
keeps on increasing with the attributes
but only
with the limited set of attributes we've
given it only with our impoverished
description of the attributes of the
world we care about
in this case it's only getting two out
of four
of the things we care about for a
recommendation algorithm
and we're still going to say that okay
the ai's utility
it increases with the more ads it gets
and with the more diverse content it
shows that is going to be true no matter
what
so we have the rough setup
okay now what is this paper saying is
going to happen
we can kind of guess what's going to
occur because the ai
in this recommendation algorithm setup
doesn't really care about
meaningful interactions or
overall community well-being it's just
going to try to drive the world in a
state
where there are none of those things
presumably
if it comes at the cost of ad revenue or
if it comes at the cost of meaningful
interactions
and you know even otherwise if they
might be decreased because
say suppose the ai just
is doing some random actions which has
have some complicated side effects
and it doesn't really care about how
that impacts community well-being
well the ai is going to take it anyway
because as long as it has
and leads to an increase in ad revenue
then that's good
in the ai's mind and this is
sort of what the paper gets at
it says that in this situation where we
have our ad system
where we only give the ai two attributes
to care about and we say that
okay ad revenue is restricted in how
high it can get
and so is the diversity of content
in that case
it turns out that the ai
it's going to eventually optimize things
to a state
where some of the attributes we care
about
wind up as
at lower value as they possibly can go
this sort of makes sense because our
constraints ensure that
there's a kind of trade-off between
how much community well-being you can
have and how much
ad revenue you can generate or say how
much content diversity you can have
right
and for the aai since the only cares
about ad revenue and content diversity
it's going to trade away as much
community well-being as it can it
doesn't really care about community
well-being as long as
long as it increases its utility even
tiny bit it's fine
so you know it seems pretty natural that
the
community well-being is just going to be
driven down to as low as it can possibly
go
until lowering it any further doesn't
free up any more resources
for what the ai cares about
this is what i meant by this being a
fairly common result in ai safety
or rather spoken off as if it was a
common result we've all heard these
claims before right
uh usually in the s
usually in something like say good arts
law or that sort of thing
where eventually there's a trade-off
that happens
between some of the
things we talk about and
some of the things that we want usually
there's some kind of trade-off in the
limit
and if we optimize for just one of those
things it's going to come
at expense at the other things right
like say
again with a happiness example if you
try to optimize too much for happiness
you're just going to wind up with
an ai flooding everyone with endorphins
and you rob everyone of freedom which is
another thing we care about
or you know beauty or so forth
and this paper then goes on to say okay
well there's a couple of ways you might
be able to get around that
you might be able to get around that by
saying we're not going to let the
ai reduce any of the other features of
the world that we care about
and that of course does
do something useful but it
doesn't really get you that
it doesn't necessarily get you to to the
best possible world you could have been
in because the ai might say okay you're
really limiting me here i can only
i have to keep community while being
this at this level and i have to keep
the quality of the product at this level
so i only have so many resources i can
use to generate ad revenue
and content diversity i'm going to try
to
get as much as i possibly can but like
you're restricting me here i can't do
every action i possibly could so i'm not
going to be able to
do the best according to my utility
function and conversely
the best according to your utility
function um
but then there's a kind of strange thing
that happens here which is that
i haven't really said that the
things you care about have a lower bound
right like i said for example community
well-being
it might be the case that you exist in
some
world where it's possible to have
ridiculously low levels of community
well-being perhaps it's even possible to
have hell-like levels of community while
well-being
and what's kind of interesting about
this paper is that it says that
if the if the constraints
that is are placed on
community well-being content diversity
etc
if the constraints on the attributes are
shaped in a certain way and if
your utility function has a particular
sort of shape which is kind of
difficult to describe in words i really
wish i had my slides here
but if they have some particular shapes
using some fairly simple
assumptions then
you are going to spend arbitrarily long
amounts of time in arbitrarily bad
worlds according to your utility
function
this is guaranteed to happen so what i
mean by this is that say
suppose that you have some world which
you say is -10 utility
then the ai is guaranteed to put you
into worlds
below 10 utility for an infinite amount
of time
and suppose you ask okay fine but i
i really don't want to be below -100
utility that's just too horrific
well too bad the ai is still going to
put you into worlds
which have less than minus 100 utility
for an infinite amount of time
and this is going to happen no matter
how low
the utility is so things are just going
to get more or less worse and worse and
worse
again these are somewhat strange
conditions
but you could see a unwary designer
putting something like this into an ai
system
hopefully no one is stupid enough to do
that for a super intelligent ai but who
knows you know
like uh there was that case
of stewart
russell telling the designers of
facebook that look you guys are saying
that you aren't going to input a stupid
utility function
but you started off for a few years just
trying to get ai to maximize
the number of clicks and you know that
had pretty weird effects
so if a massively successful company
with
brilliant engineers like facebook can do
this for a couple of years at a time
well uh you know that's not so good a
situation
but at least this paper formalizes this
that it really could get that bad
and it has been accepted in some
pretty good journals so hopefully people
will say oh okay
if you are doubting what i'm saying
here's a formal argument showing some
guarantees
the methods that the paper talks about
in order to prevent this kind of thing
they are more or less
a sort of interactive game which is the
center for human compatible ai
chai likes to use
where the ai does something
and the human has the option to update
their utility function or give them some
information the ai does another thing
and the human repeats so sworn if you
would mind going to
figure one scroll
up
that has just like a nice little cartoon
of what i mean
by in this case what happens is the
robot is given a utility function and
optimizes it for a little bit the human
looks at them and says okay
um these are you
i want you to stop now and i'm going to
give you a different utility function
using these features
and then the robot starts optimizing
that and it keeps on going on like that
forever
or until you hit the perfect mix of
attributes of the world and recommend
and the perfect policy for the ai to
follow
and they guarantee that in their
particular scenario
the human is guaranteed to eventually
wind up in the best possible world
according to the utility function
now of course there are a lot of things
you can take issue with this for example
say that oh okay i don't really
i don't believe that my utility function
is always increasing or
with these attributes you talk about or
for example um
i don't think that there are these
constraints that you really speak about
i think that things can be
fantastically good or what have you
which is fair enough but that's not
really the
sort of point of this paper um it's more
or less just
i i suppose is um perhaps unfair to say
this about the paper but it seems like
it's more like just turning folk wisdom
into something sort of more formal and
rigorous and
being able to show it off to people
that's kind of my opinion of the paper
which is a bit harsh
because honestly i i look at some of
their techniques and i think okay the
issue is that
these techniques you're talking about
they discount things like okay well what
happens if
you i mean you're assuming that the
human actually knows what
attributes they care about right like
we're not really sure what things
exactly we care about
we're not sure how we could even uh
hope to put some of these things into a
utility function
and it also ignores things like
implementation errors or that sort of
thing
where you would really want the ai to be
trying to do what you wanted
to do rather than what you encode it to
do or what have you
but still i suppose it had some
interesting things right like
sworn would you mind going down again to
the second figure
the infinite
the these curves were what i was trying
to gesture
oh you missed it
there you go so these curves are trying
to gesture what i was talking about
earlier where they have the proxy
utility which is the utility function
you give to the ai which doesn't include
all the stuff you care about
and your true utility function which is
the yellow thing for the dotted lines on
the left
that does include the things you care
about and they go through their setup
for a recommendation algorithm going
through
okay what features should we leave out
and they see that um no matter what
features they leave out
eventually their
proxies will
their ai optimizing their proxy will
eventually start causing
the true utility to start decreasing so
you see that on the axis it says utility
change with time steps on the horizontal
axis what this is basically saying is
that
at each time step is the are the utility
functions
increasing or decreasing so if they're
positive then you know they're
increasing and
that's fine if they're negative then
they start decreasing that doesn't mean
the utility is negative just that it's
decreasing with each time step now
so you can see that on the left the
proxy utility the blue line keeps on
increasing
that's just an optimizer doing what it
normally does
and the human utility does increase for
a while because the proxy tracks it
relatively well
but then eventually once the ai has had
enough time to severely optimize things
to put the world into kind of weird
states
then the true utility starts declining
the their behavior
sort of gets uncoupled and on the right
you can see that
the people in the paper said okay we're
going to
remove we're going to only use two
attributes in our
from our true utility function and put
that into the
proxy utility function and see whether
or not
the utility eventually starts declining
in all cases
and of course in all of the curves it
does start declining
because that's it
because according to their setups
according to their theorem
eventually all of these utility
functions are going to start
losing utility because the proxy utility
is just throwing away
some of the attributes you care about
it's trying to get rid of them
in order to have more resources to use
for the proxy things
and that puts you into some really
bizarre situations like right like we've
heard of these things before for example
say you have
a ai that is attempting to reach a very
high score on a racing game
and it finds one of those strange
situations where you can just go around
in a circle and keep on increasing in
points
and the
utility according to the game keeps on
going up
but you want the robot to be very good
at driving and
okay sure it's it knows how to
drive pretty well around in circles but
if for example you were i don't know in
that car and you were trying to get
somewhere
i'm pretty sure that your utility would
start declining after the first hour or
so that you go around in circles
continually
and you know you'd get pretty bored
eventually
but that's more or less the contents of
the paper i'm
very sorry i don't have my slides and i
couldn't have given a better
presentation but i hope we can
discuss things i
am yeah
well thank you very much charlie for
your presentation


New extractivism (Vladan Joler)

is still not possible but these actually
give us the opportunity to have people
like platinum who are not based in the
netherlands
for this kind of meeting and it's uh
it's really great
and um so also
the next meetings ai takagaras will be
online for the next uh two months and
there are already
appointments that you can see on our
website
and i would say let's move uh to the
talk
and let me introduce briefly vladimir
vlad and
joeller or how shall i okay
okay is a researcher and artist based in
serbia
and his work is uh
focused on data investigations data
visualizations critical design
explorations
and they aim at investigating societal
implications of technology
if i'm correct and you may know
one of his very famous projects which is
anatomy of an ai system
developed in collaboration with kate
crawford from
ai now institute but there are also
other very interesting projects and
probably will see some of them so
you can start when you want okay thank
you thank you
so yeah it's really a pleasure to to
to be here today and uh
so let me start some kind of like maybe
longer version of the story
so uh basically
i think maybe even 10 years from now
i i started to organize some kind of big
gatherings of
internet activists here in in serbia it
was called share conference
and it was you know back in those days
in in which we believed how internet is
you know a good place and how we should
fight for it and stuff like this and
then
so it was like some kind of big
gathering of activists
uh hackers uh lawyers
you know this kind of internet freedom
uh uh people
so uh but then in one moment we realized
that that
you know you cannot do a lot with just
uh
you know meeting great people and and
gatherings and stuff like this
because in moments when some something
was happening here in serbia
and in the region we didn't have
capacity to
to react in a way on those like
different attacks tracks
and stuff like this so we started to to
organize one
group the first name it was a shared
defense
and and so it was like kind of mix of
of you know lawyers and and
cyber forensic guide and there were like
tech experts like lots of different
interesting people at one place so we
started to deal with those problems of
like
mostly in serbia so we're doing like
monitoring of
cases of internet attacks and then
slowly we started to do
some kind of investigations into
like uh network first started with like
network topologies
because always i wanted to to even back
in like 2000 early 2000
when i saw like first maps of of the
internet i i wanted to
you know to try to do the same thing and
uh
and in a way you know when i
coming back like i don't know like 14 15
years later
i realized now i have like a knowledge
and tools to start to
explore those like network topologies
and we started to
do our first investigation into like you
know trying to understand
how those networks look like
how for example like internet in serbia
look like
or what is the life of one internet
internet packet
in in a sense then we were like
following these internet packets and and
trying to build some stories out of it
and then so we're like going step by
step into
uh uh deeper in deep deeper
into like uh investigation of what what
i like to call like this invisible
infrastructure
basically everything that is happening
behind the
screens no so i can maybe start to share
some okay just a second
okay are we good
yes so for example this is like the the
those investigations that we're like
doing
in the beginning so there's invisible
infrastructure the exciting life of the
internet packet and then going deeper
into
some mapping and then we're you know
step by step
diving deeper and deeper you know like
first visualizing trackers
permissions data flows
and then more and more
going into this kind of like you know
like surveillance architecture
or or surveillance capitalism
let's say until we we
and those investigations end up with
like
creating first of the the big match that
i did
and that one was called the
facebook algorithmic factory and now i'm
going to show you like a lot of
crazy black maps so this is the first
one
you know so it's called the inside
facebook algorithmic factory
and here i wanted it was like really
the question was like you know how
this invisible imaginary like this
abstract
factory of facebook is functioning so
we're
trying to to map for example on one side
all the the the data we are
here or all the types of data we are
contributing to this platform and then
on the right side we try to map
how this platform is basically selling
our data
so this is for example mapping of of
like this kind of
categories that exist within the uh
process of creation of like
ads and then
you know the the most challenging part
of this investigation and
map was this like middle part of like
what's going on
between the input and the output
and and this is how we enter into this
field of algorithmic transparency and so
on
and in a sense like we tried a
lot of different ways how to measure
what's going on
behind this black box but at the end the
the most useful way to
investigate that we found in that moment
so that was like
i think like 200 2 000
i don't remember maybe 14 15
when we started to do that so i found
out that there's like a
collection or that is possible to to get
all the patents that facebook had
and in a way that we found like several
thousands of patents and we started to
read
some of them in a way to create in order
to create
this mosaic and that
that is one of the rare visualization
one of the rare maps that
exist out there about this kind of
invisible algorithmic
uh process so i was trying to find
different
patterns that that that were like
basically solving the mystery of
what's going on between the left side
and the right side
of this map and and then when i created
this drawing first of all
it you know like it's kind of like you
need to be
a bit crazy to do that and uh
but at the same time i was like really
sure that
something like this is really necessary
even if it looks scary
it looks like you know like what's going
on here and stuff like this i think
and back then i was like really thinking
and i still do
that you know trying to understand those
like
complex processes that are happening
behind
it's really really important so if you
want to progress
in our you know life in future and
whatever
it's really good to know and to try to
understand even even if
if it's props possible if even probably
it's
not possible at all to create the map
that will completely uh
explain the process because like those
processes in a way
i was working on this map for like two
years or something like this but
and and during these two years the the
this
you know processes behind the black box
changed one million times so in a way
this map will never be precise enough
that we can say this is a reflection of
reality
completely reflection of reality but in
a sense it's the only map that exists
out there
and i think it's really important to try
to to map the the processes that are so
influencing our life lives on many
different ways you know
and and in a sense like
this one was like over there for some
time and
and it was mostly used by by uh
people uh uh who are like dealing with
the internet
freedom advocacy lawyers and i was going
around until in one moment it didn't
became some kind of like uh
mainstream thinking because like bbc
wrote like some text about it and then
it became some kind of
known piece of of
like known drawing or map
and in a way this is the moment when
those things transform
from some kind of like cartography
into something that that some people are
considering
that this is art or this is design or
this is i don't know there is like there
was like a lot of different explanation
what is this even for me it was never
art you know it was never
something that for me it was just like
my internal
uh uh struggle to understand
different types of complexities that
exist out there no
and then you know on one side i have
this kind of
curse of of of being trained as
a visual artist so for me that
somehow combine into what you are seeing
and then in a way there was some kind of
big change in in what i was doing
because i i
i met one friend of mine it's called
her name is jonah mo and she like
infected me with this idea of
materiality of
of internet
infrastructure and and then
you know basically my investigation
exploded in many different
uh uh directions you know so what i was
explaining to you until now
it's basically this like i was always
starting with from the position of a
user
who's here and then trying to go deeper
and deeper as i can
into those kind of like things that are
happening behind
so for example most of in this
investigation
we're starting with uh with uh uh
an object so in this case so this is
another map that i'm speaking about it
is called
anatomy of an ai
and so it's starting with the uh object
it's called
you know amazon echo so first thing it's
of course to open the device you know
and then our opening device trying to
understand what's going on inside
and trying to you know like uh
understand what kind of
data is coming in what kind of data is
coming out
but even on this first step on
investigation you see a lot of different
layers of untransparency so for example
you know the issues related with
hardware can
we can say try to repair open schematics
diagnostic tools planned obsolescence
proprietary software privacy digital
security and also this is some kind of
first layer of of the problem and then
deeper you go so the next step then i'm
investigating like for example how these
is connected to the internet what kind
of packets are going through then we are
doing like internet
mapping and infrastructure trying to
understand for example in this case
where
the packets are going where for example
amazon have their own their
data centers where they are located and
so on so on
and then going deeper deeper behind
trying to understand you know like what
kind of hardware there is
and then trying to you know somehow go
into the level of
algorithms neural networks and so on so
on trying to understand data behind
these
and this is something that i was doing
for a long time let's say like five six
years i was like really
you know being more and more skilled to
to go deeper
in deeper and to investigate deeper and
deeper
but then as i said like i i i met joanna
and i started to think differently in a
way and she gave me one book
uh the name of the book it's geology of
media by
yusi barika and then i was completely
like
you know like trying to think from the
about those infrastructure from the
completely different angle you know so
from the angle of
long period of time
from the angle of geology and if you
start to think in that direction
the story goes completely different so
in this middle part of the map we're
mostly speaking about like privacy
security
data exploitation lot of this kind of
stuff
but then when you when you start to look
at the this kind of infrastructure from
like the angle of of geology or angle of
like deep time
there is a different problems and then i
started basically to draw this map
not from this middle part but from here
from the
from the the earth
and elements that are basically part of
those
devices that are part of those like
infrastructure that we are seeing
so in in a sense what they did i
kind of you know like went back in time
when all of those devices that
are part of this infrastructure were
different kinds of rocks
and then i started from there and then
uh i wanted basically to tell a story
about uh birth
this is the left part of the map life
and death of one device and in this case
this is this is a
amazon echo or device
and then i started to work with with the
kate crawford she is like one of the
most
i think known uh people in the field of
the ai ethics she's running one
institute in uh
it's called ai now as a part of nyu in
in new york and then we started to
investigate this story starting from
you know elements and then the story
goes
there is like you know in most of these
devices as a mobile phone
you know like or servers computers and
those amazon alexa for example there is
like a three
uh two-thirds of periodic uh
system of elements in it and in a way
even on that level when you start from
elements you you see that the
devices that we are using today they are
you know like really complex to produce
in a sense that you really need some
kind of like
really diversity of different elements
and it's not just like
you know a few hundreds ago most of the
things that we were producing were made
like from
few different you know materials
now hide that it's built
of almost like whole periodic system of
element
and it's hard to tell a story you know
is it really complex we're starting from
many different elements
and in a way i was also thinking like
you know if you just follow each of
those elements you will
be able to make a book about this
journey
from you know like this kind of one
element to the product
and then another thing uh happened i i
i needed to have some kind of uh visual
structure visual
elements in order to tell a story so i
use this kind of triangles that
are basically really present in the work
of so it's classic this kind of marxist
way of like understanding labor and
everything but in a way it's really
present in
in in the work of another author it's
called christian folks
in which basically i try so the score
story goes like this it's always like
this you need to have a resource
and tools then you are using some labor
and then this is as a result you you are
getting some product then this product
is becoming a resource for another then
you are using another form of labor and
then you
getting a product so resource labor
product
resource labor products so i try to
tell a story through those triangles
going through
different you know processes
that are happening during the birth of
of of one device but what is really
interesting it's like
when you go deeper into that you realize
that for example
you know like uh apple have one
um in one device you have
like they have like just first layer
layer of their
suppliers it's some kind of 250
companies
and those 250 companies that are
supplying these parts of this machine
are you know like all around the globe
but then those 250 companies have their
own 250
suppliers that have their own 250
suppliers and
in a way for me what i was like
more and more i was investigating for
for me this structure became like really
interesting so
i understood basically that in some way
those supply chains are some kind of
fractal structures
in which you can zoom more and more and
more and more and
there is never an end and uh
but what is like really for me another
important aspect of this is that when we
are seeing
like the problems that are appearing on
each of those like let's say steps
it's completely different than the ones
that i was investigating before so it's
not anymore like this kind of security
or privacy it's more like you know hard
labor
force labor child labor conflict
minerals and but in most of the time
it's there is like a price and this is
like environmental price
that no one is paying so on one side you
have like this kind of like exploitation
of human labor and really some kind of
diversity of different
kinds of relations that exist out there
on other side you always have this kind
of exploitation of
of resources natural resources and
you have always some kind of
environmental cost
that is really part of the process
so when we are speaking about binds you
need to understand that for example you
know like for
one little part of the the you know
you know one little part of the gram of
some material
you need to move you know like half of
the mountain from place a to place b
and so on so on burning and melting and
so
like so many different and also like
like
the the the numbers of kilometers so for
example when you when we speak about
this like
just from all of those first level
suppliers of
for iphone for example like just i was
calculating a number of kilometers
how how many kilometers all of those you
know
uh components need to travel to to
shenzhen and foxconn factory
and it's like two times the moon and
back
and and so it's a massive
uh a planetary scale factory in which
like everything it's moving around and
and
and the thing is that those costs of
environmental costs
of those movements of goods are
really not calculated into the the
enterprise now
in a sense this is something that we
share
between all the people on the earth but
the profit
profit it's not shared among the older
people in the world so i tried
also to to calculate like for each of
those jobs that i was like
able to to detect there like how much
people are earning
and then you you know like
big part of the the the process is you
know happening in some kind of china and
philippines in in india
and so on the really low salaries but
then you know like cap
between the the person that is on the
bottom that is basically probably a
child working in the mine in congo
and jeff bezos who is on the top of this
pyramid
is uh uh you know it's it's not even
possible to
to to count but it is
you know in a way like one to seven
million
or something like this so someone and
all of other people are somewhere in the
scale
in between so in the sense there is like
inequality
of like standard of living between
people in this like one big factory
that is going from one to seven million
and and and then there's a lot of also
different things that we can speak about
but maybe
i believe this later for questions or
something but please interrupt me
if you if you have something to ask
uh in a sense there is also different
forms of labor that are happening here
so we
we you know here on the bottom you have
like people working in the mines then
somewhere here you have people working
in foxconn doing some kind of
algorithmic work like on the on those
like assembly lines
but you also have some kind of hybrid
relations for example in amazon
uh storage systems and those like places
where the robots and machines
and people are working together
and and we managed to to i was able
because i was traveling a lot
trying to investigate most of those
places so i i went
to some of them so for example i managed
to enter into
together with kate and some of those big
storage places of amazon
uh where you have those kind of robots
moving shelves and stuff
stuff like this so you you are
if you follow from the beginning to the
top you
see different forms of labor you know
like and
different relations you know
and so this was the birth somehow and
then all of those things are
fueling the the infrastructure that is
like in the middle of this map and then
on the right of the map
you have a process of death in a sense
like
most of the devices are being like
thrown away after few years and then
going
backwards basically to to again to the
earth and again a lot of you know like
strange and really dangerous like
working conditions
and stuff like this and stuff like that
and then the main question in that sense
like how fast
those cycles are happening how fast we
are seeing this kind of like
movements from here to there
and and because like this is every time
when we for example buy a phone
or buy like we are basically giving a
new spin to this cycle we are
basically approving all of those
processes
another interesting point maybe here
will be like
you know like if this this is the global
planetary
factory and it will be interesting for
example to think
uh what will happen
and what's going on now in in this
circumstances of like a global pandemic
and
how those parts of this global factory
are being like
you know how they are surviving this and
things like this and so
yeah maybe maybe i can make a little
pose here and open for some questions
and then i can move to
some other other
yeah other things that i want to show
so everybody who has questions uh
they're welcome
i have a little question in in curiosity
so
you were saying that you actually
entered into facilities of amazon and uh
so did you find resistance for that
because i imagine i mean it's not that
easy uh to get in and uh
ask for information and that about this
kind of things
yeah yeah basically for amazon we we
and i really uh encourage
all of you to do that yeah in the sense
there is a way to to get in
by some kind of official tour because
and people are so once or two times per
month
i think there's like you know
they allowed i don't know some amount of
people to
to go there and to go on some kind of
guided
tour but in a sense being guided or not
it's like really uh it's really intense
experience because like you feel you are
inside
of some kind of like you know inside of
a machine
it's like really noisy environment and
those robots and it's really not
safe environment then and you really
see some kind of like dystopian future
of work
in which like most of the people there
you know
really doing some kind of you know
repetitive
gestures and moves but in the same time
because like
all of this machine and all of this like
storage system is run
mostly algorithmically so for example
the on those shells you have like
different
you know products that are not related
in any logical human way
so for example not you know it's not one
shelf is books another shelf is
i don't know something else no
everything is messed up everything
it's mixed up because it's just
representation of what
people are buying together or there's
some kind of like
algorithmic logic so it's completely
alleviated
from the human understanding so the
people who are working there
you know there is no human logic in it
so you are just part of some kind of
algorithmic process for example it's a
really crazy place
yeah cool
and yeah maybe you are also i don't know
maybe luciano has a question did they
see a hand
yeah hi um yeah fascinating it's very
interesting what you do there
and i have a question regarding the
the the idea from human
knowledge and human behavior to training
data i know that you're on the bottom of
the center part where i could see that
you have something like
this can you share a little bit there
because is this do they have some
similar com ideas relate like to from
the geology
the geological perspective to the
hardware
and from this knowledge to the data
training how do you see this is this
like so this is the relation there
yeah uh so so that this bottom of the
map was
basically the most troubling for me and
most like demanding
in a sense i really spend like uh you
know
months in trying to do those two
things and they're far away from
from perfect so what we're trying to
explain
here i i will speak a bit later about
some other thing that i did on the
similar topic
but what we tried to do here we tried to
think about like this process of
enclosure
so in a sense like uh uh you know in
contemporary
we have this idea of like you know like
uh
during this time of like a big uh
industrial revolution in the sense
like this so the the resources was
mostly like
you know like different kinds of uh
metals and ores and and land and
things like this and then those like in
that time there was this kind of process
process of like an enclosure
of of like resources and
but in in and then in a sense
this is like the time when the first of
this kind of
you know like big concentration of like
power or going on you know rockefeller
central
and and in a sense what we have here is
kind of similar to that except the big
difference is that
that the the contemporary you know data
based capitalism can expand
into infinite number of
different territories you know
but at the end most of those territories
that they are invading
uh for in this search for resources
are basically territories of our bodies
their iteratives of
our uh uh behavior or
of our conscious and unconscious of
our you know cells in our bodies and
different kinds of data that we are able
to produce as a humans on one side
on other side so there is this kind of
huge rush
into quantification of everything that
can exist you know from
stars to the smallest atoms of our body
you know
but on other side there is like a
product so this is those
two parts one is like quantification of
like human body actions and behavior
another one
is quantification of human-made products
you know like whatever we create
will be quantified will be
digitalized and we'll become a part of
this
new it will become a new territory that
it's going to be
basically digitalized quantified and
then
uh in a way like transformed into some
kind of like
wealth or whatever and and and so so
in that sense here i try to to map
basically some fields of of of like
quantification or
fields of like those new territories
and then in some way the process how
this is like
being transformed into uh uh
and becoming a part of of the process of
for example here like
learning of like algorithms or like
learning machine learning process and so
on
so on and in the sense this i think like
an upcoming
another me this map didn't went
far into that i will show you some other
other maps that are maybe a bit more on
that that track uh
because here we we wanted to to
to basically show the map that is
kind of like speaking at the same time
about
you know exploitation of labor
exploitation of nature and exploitation
of data in a sense
and what we wanted to tell here is that
that
maybe we should not
forget about like exploration of labor
and exploitation of nature
as a as a equal
part of this mosaic of this story
because most of the time in like
last i don't know like your tens of
years
we we are just both speaking about you
know our data data data data data
and in a sense like what we wanted to
show here is we wanted to show okay this
have like another also another
layers of of problems that we
uh want to speak about
i will show you some other is there any
more question in the same related to
this map or
there is a it's fine it's fine after
this one i
started to
sorry no no i raised the hand but uh
it's fine
you were already answering those things
so go ahead
okay okay so we after that i tried to
i worked for like one year or maybe
even more on something
together with uh matteo pasquinelli
and and then the project was called uh
noscope
and it's published i think like few
months ago it's not it's
kind of still fresh in a sense and and
here we wanted to
to uh
you know basically uh
basically this noscope it's part of this
part of the map
because here we didn't we gave some kind
of wider anatomy of like an
ai system but but then here with with uh
we try to go deeper just into into
you know this kind of like uh uh ai
as a is a system without this kind of
you know like uh supply chains
right just what's going on within
the the process of data learning for
example
and and in the beginning the the
the project was called the uh limits of
ai
because we wanted to understand where
those
systems are failing what are the limits
of of
like that that are happening
behind and we end up with this snap
in just a second just a few seconds
ah little corn break
sorry no working for from home
so we end up with creating this
map this is basically also following the
similar process so it's starting with uh
with a let's say like the world society
and then we have like in the middle we
have like a
process you know first you have a
process of capture
then you have source selection then data
set
composition selection labeling then we
have a
algorithms free
feature extraction pattern extraction so
at the end we are getting a model
and at the end we have some kind of
application of
of the model but but what is so this is
some kind of
like you know basic structure of one
machine learning process let's say but
what we also wanted to understand
here we wanted to understand what kind
of labor
is there and what kind of what type of
bias we we can
we can map during this process so for
example on the left side of the
this map we try to map some kind of you
know for example human bias
related to all of those kind of ghost
work or
or the work that we don't see when we
are thinking about
like ai and stuff like this but on this
other side on the right side of this map
it was like you know process of like
trying to map
different kinds of machine buyers
because we understood
they're like something it's a mix
between human and machine buyers that is
complete
all the time on in the pro during the
process of
creation of like machine learning
models so what
and and then we tried so this is what's
happening on the
bottom part of this map and in the
middle part of the map
we also follow the same logic but here
we are going more
into this kind of like uh
types of of of because like
in a sense what we are seeing here we
are seeing
an ai is a process of like uh
compression of information you know and
and uh statistical compression in a
sense
so in that sense we are trying to
understand all of those like
mathematical let's say statistical
processes that are happening and trying
to understand what are the limitations
and what are the the
failures that are coming from from each
of them
and then at the end like we try to map
uh uh different kinds of like you know
like uh
you know what kind of consequences we
are dealing with
on the top of this when we when we
sense like apply those models into the
into the world so this one looks
better so that was the the the
something that we were like that i was
working
on together with with matteo basquian so
this one is
maybe a bit abstract to to show like in
such a short
period of time but in a way we can
i can like you know if you have time
please go and check this one so it's no
scope.
i uh
yeah so that's the um
i think it's really cool this work i
i also love working with uh visualizing
information not at this level of course
but um so i'm really curious how you see
the potential impact of this kind of
work because you said
uh it's in your opinion it's really
important to understand what are the
hidden processes behind these
uh systems and i really agree but do you
have an idea apart from like
yeah these projects are very important
impactful because they get uh
acknowledged they get recognition
through awards and
museums and these kind of things but uh
beyond
let's say sensitizing do you see like um
a used like a
some way like decision makers
policy makers could make use of this
kind of work
well you know like
i think there is like a lot of different
levels of it so for example
just in in uh in building
this one uh uh
you know matteo and i will spend like
one year and like
uh thinking about how to
because for example in this field of
machine learning you know
and like ai you don't even have uh
you know like clear terminology
or or like explanations of like
some parts of the process so some kind
of common
explanations for this or terminology for
that you know like so
so every kind of attempt to to create a
map
it's a challenge for and on the first
place it's a challenge for us because
it's it's it's like
you know you can write academic text
but but you know when you start to to
draw something and you start to draw a
map of something
then a lot of different questions are
are appearing
you know and and and so the the the
process of creation it's like
really really important
i think because like then we are trying
to
to find an answers to to some
questions you know that appear during
the process of
mapping and but but then you know but
not
every map every especially this kind of
cognitive map now we are not speaking
about like the first map that they show
you it's more
more like uh i don't know data
visualization
and this kind of things but now the the
more and more some kind of cognitive
maps
that uh uh you know that they're
always you you also need to think about
that that
you know those maps are also really
biased in a sense of like you know like
they
they pretend to be you know
something that is true but in a sense
that can be also questionable because
it's it's
our or my way of interpretation of the
world
you know true so sorry in that sense
like
especially i don't know anatomy of an ai
and also facebook one and
it was used in different fields so for
example facebook map was mostly used
for this kind of advocacy uh
you know like to show how the system is
complex or evil in a sense it have this
kind of like potential to be a tool
for scaring people to make some
decisions
and then on the other side anatomy of an
ai it's more like
you know it finds its way to both in
into
like classrooms where people are using
this map to
to show something but
also it find its way into museums into
galleries into like different
formats so i think like there is there
is a huge potential in like
and this power of maps you know like in
the sense i believe that
the the power map is is really uh
is really big you know like in the sense
that
to help people to navigate you know
because like what we are
facing in the world is like we are
facing this kind of like really big
complexities
and obscurities of of like reality you
know and
and i think it's it's important if you
want to navigate those
spaces and in a sense it is really
useful to have a map
and i think we have other questions
in mind i saw arcadia had a question
before but i don't know if you still
have that
and then you are now
yeah i had a question which
basically resolved when i looked at the
the notes so i went to noscope ai and i
realized that they actually wrote the
whole essay
so yeah i actually found the answer
okay okay okay yeah so yeah
cool yeah thank you it's really really
fascinating the work you do
um i i have a question about
the also about the implications of this
kind of tools
not only the implication for for ethics
or for policy
um but also for design so i was thinking
in terms of like ethics i
thought about transparency and how um
seeing the humans and the environment as
part of
it positions transparency or
explainability of ai in a different
place
in a different site also in relation to
humans also and then
questions of like who creates these
explanations who is
receiving them this kind of challenge
and then also the implications for for
design because
when you see how much humans are
implicated in
in the construction of ai
it kind of uh moves the the
the way we or she needs a shift of the
way we design ai
we think of ai in design as like
constituted
by by humans and and yeah so what
did but that's you having that or not
that it's one of this like uh uh uh
you know we tend to to
this kind of idea and believe into into
automation and out in the way of
you know autonomy of technology outside
of human
field it's completely some kind of like
myths and some kind of like idea that is
really
present but in a way it's really far far
far from
from uh reality because like if you
deconstruct
uh like in noscope or in
anatomy of an ai all of those systems
you will see that
they are full of people doing different
things
and it's just not just full of people
but full of people
being treated differently you know like
people
labeling the the you know armies of
these mechanical workers
uh labeling uh images or people who are
being part of the image being on image
you know on one picture that are
being taken from those companies and
becoming part of
of like data sets under like whole
variety of of uh
different roles that human beings have
in the process of creation both of
the the hardware and models and software
or whatever
so so you know this kind of meat of
automatization is completely
you know a myth there is no such a thing
of like you know like
completely independent out of human
existence
intelligence or whatever you know and
and but each of those steps
each of those uh involvements human
involvements
we brings with itself like some form of
human bias now
sorry so and then we can
also speak about that you know like that
how probably there is no way
that something that is completely not
biased is ever be able to be
produced because there are like lots of
different questions
of like you know like who is choosing
the data set
from which you know how this data set is
being sampled
[Music]
how you know like who is testing the the
results who is fine tuning the the
learning algorithms in a sense of like
you know
tuning hyper parameters or something
like this you know so it's
it's like really really hard to to
this idea of unbiased models is
completely crazy in a sense
and the the also on the other side you
know like there are some kind of like
within the processes of like compression
of data
and all of those like statistical
processes that are happening
most of them all of them have some kind
of
you know like problems in a sense like
so for example
interpolation extrapolation model
fitting
curve fitting dimensionality reduction
you know like those are all uh different
kinds of like uh
process of like reduction and
compression of information
that in a way every time you do that you
have some kind of
loss you know and then what do you have
as a result of this kind of loss
and most of the time you have like this
kind of like
anomalies are being thrown out of the
game and then if you think on some kind
of more ethical level who are those
anomalies maybe
i'm the anomaly you know like maybe you
know so there are so many
different layers of like
the problem and so when we were thinking
about this kind of limits of an ai
you know there are so many different
dimensions of those limits and so many
different
problems that are coming with
there is another question by afghani
i think yes
i um yes thank you thank you very much
really fascinating work
um really incredible visualizations i'm
curious about your thoughts
so so you mentioned um
that you mentioned this if we want to
navigate the space it's useful to have a
map
and i'm curious what are your thoughts
so we
you showed maps of existing systems
uh what are your thoughts about trying
to construct maps
of the kind of world that we want to see
and then can can we then use such an
approach to
help us understand how do we get from
the kind of systems we have today
to the kind of systems we want to have
you know how do we make the bridge what
are your thoughts on that
i'm curious you know this is the
the the you know something that that
appears as a possible you know way
out of you know this and but for me
personally i don't
feel uh i'm
i don't feel competent in the sense of
like projecting
i feel more comfortable in the field of
like trying to understand
first how the
the system look like or how those
complexities look like
and this is somehow where my capacity is
is is ending this is what then i i don't
feel for example i
i you know like my kids
you know went to school today and i'm
still not sure is that a good
or bad decision so i'm really bad in in
projecting what's good or what's bad i i
find
like my role in in this will be
this process is to try to understand how
things look like
in this moment you know and then i i
hope that someone will
take the responsibility of framing
something
the next step how to go out of um
out out of there so so so it's it's
um somehow i don't know i don't feel
capable of like doing that for the
moment
but i think it's really important if
someone
thanks thank you for sure
uh yeah i can show you something
else as well uh so this one is like
so this one is bit different than than
the
all of the maps before i show you it's
called uh
uh new extractivism and i'm not too sure
you will be able to see a lot
but
but i think i had a lot of fun doing
this in in
last i don't know one year intensively
during like
the next last six or nine months
uh so it's basically the
the the map of something that they call
like a new extractivism
so anyway it's the same topic as a
uh previous maps
but this one is basically some kind of
assemblage of different ideas
different concepts of different you know
things that other people
did so mostly like philosophers or like
media theories or different
artists and in a way what we are doing
here this map we are following
so we are following the the
you know like some person who is falling
down into the black hole
and and in a way this is starting with
this idea of one artist
she proposed that like to try to think
about the internet and those spaces
about some kind of like that you can
apply
this kind of idea of gravity how those
like big
companies are basically curving the the
space and time and in a way i started
with this and then we have this little
guy who is trying to
escape from this like a black hole and
in a way like when the
this person passed the point of no
return he's falling into some kind of
cave so this cave it's basically a
plato's cave and also pound of the
concave
in which each of us are like spending
our lives in
then i try to understand the
architecture of this cave
so under one side of the cave you have a
projection of some kind of personalized
content
you have interface you have some kind of
chains and then you have
shadows and then these shadows have been
captured by
different capture agents and then
and then the story is following this
you know invisible abstract
machine into the processes of like
on one side you know like i'm following
this idea of individuals and
dilutes and fuco and french
school how like you know like extraction
of this information and then you're
you know you have some kind of like
creation of individuals on other side
extraction of information from the
content and then all of these together
and then on the bottom you have it this
one is a bit crazy
and you have like you know like
different forms of extraction
[Music]
how basically different forms of labor
it's cool it so it's called manual
so you imagine it as a also sort of
guide for
people who want to open your work
okay what you're seeing now it's manual
that is basically used for
to reading of this map so the there is
and then basically what i'm trying to do
there i'm trying to go step by step
uh and to explain each of those
we have a new question in the chat
and derek would you like to
make it uh by voice or shall i read it
uh yeah sure um uh so yeah just
just very briefly because it's the end
of the hour but uh uh
i i loved seeing the new scope um
there were a lot of ideas that seemed
very new to me and um
uh overall the work is really fantastic
but uh i i loved its esoteric references
but i also loved how it addressed some
of the magical thinking
that takes place in i guess the politics
of ai
um the politics of uh new extractivism
seem a little bit more uh apparent um
but i'm curious what some of the
intentions were and
outcomes were of nusco
well there uh you know
as i said the intention was to
to to work and to explain the the
process in in some kind of maybe we
didn't manage because like you know like
also the materials text it's really
intense and
and it's it's intense you know but but
the idea was to try to
create some kind of basic dictionary or
uh
uh understanding of
the the the process and of creation of
like models and and
ai and stuff like this so so so we
wanted to
to to make to make a drawing that will
be
on one side you know accepted and
understandable for
tech expert people who are working with
that
and to them and we wanted to share this
idea of like
you know bias human
uh
you know involvement in all of these and
we wanted in a way to
to try to to break this like
uh meat about or automatization
or you know like autonomy of these
things this kind of myth
of like how this is some kind of you
know like
magical uh
intelligence or whatever you know so
that was idea but
on other sides for like more uh uh
you know like this kind of
non-tech audience or like non-tech
experts
we well wanted to make something that
maybe will help them to understand
uh also the technical part like what
what is the the basic
anatomy of this system so it's supposed
to be somewhere
uh in between those two worlds and and
at the end i'm not sure that we managed
to
to create something that is readable on
both side
but but that was some kind of intention
to try to
deconstruct and to try to understand and
communicate
the the limits of those
uh systems it reminds me a little bit of
uh richard feynman's discussion of uh
science and beauty um you know this idea
that understanding the science of a rose
would somehow make it less beautiful
and it's not quite in terms of the
beauty but in terms of the magic of it
i felt uh the by deconstructing it
and taking away the magic of it uh it
actually revealed the
magic of it and uh yeah for that reason
i thought that this
project and the other was was just great
art great science
great political gesture so just
kudos it's really great to see yeah it
is it is somewhere
there it's always about what i like to
do it's like i
like to be in those like intersections
between between different worlds and in
the sense of like
there is not one side you know like so
many times
in my life i was like you need to pay a
price for that
in the sense of like you know that
people from
you know science will say oh this is not
science this is art people from art who
said this is not our this is science
you know so you you're always somewhere
in between but i think
i'm always somewhere in between but i
think this is like really interesting
place to be you know and to try to
to try to somehow use
uh all all the
possible tools languages and weapons we
have in order to
demystify and in a sense like understand
those like complexities
and and this is what i was like trying
to do in like
all of those maps it's like because i'm
at most of the time
i'm working together with either like
you know sometimes like scientists
sometimes like
you know media theories sometimes
philosophers sometimes there are depends
you know and and and i think that's the
only way in the sense of like when we
are speaking about the
the you know like drawing or like making
pictures of black boxes i think
what we can do is to try to look at them
from
different angles as much angles and we
as we can and then maybe we will
create some kind of chance to to
understand how what are the shapes
of those things you know so for me the
angles are
you know that
cyber forensics uh uh legal analysis uh
art um you know philosophy
those are different angles that i'm
trying to see
the object from
great thank you very much vladim i think
we have to wrap up because i know
some of uh people in the
talk of other meetings it was really
amazing to see
and uh yeah uh thank you again
very much and uh yeah
it was yeah again other comments in the
chat
um great
okay thank you yeah and uh
i hope you enjoyed the questions yeah
yes
it was really a pleasure to to be with
you guys today
thanks it was our pleasure oh thank you
very much
thank you time bye
thank you bye bye bye
bye bye bye
hmm
success