204. Universal Intelligence

pdf hello and welcome to the ai safety
reading group session 204
today we'll be presenting universal
intelligence a definition
of machine intelligence by shane legg
and marcus hutter
given some feedback from participants in
the reading group i'd like to try
something somewhat different
from what we normally do which is
instead of summarizing each section
almost chronologically i'd like to
emphasize key points
and make this a rather short
presentation um
i'd also like to focus on discussion at
the end and i have handwritten notes
here so the slides will be somewhat
sparse
if i've covered something on a slide and
i continue talking that's because what i
have written is more important than
what's on the slides
it should be somewhat obvious
and i have a new photo of an owl which
is not copyright protected
i found so that's neat anyway we can go
to the next slide
so this is a little bit of background
about the authors um
of the paper one the first author on the
paper who contributed most of the work
is shane legg he has a phd in computer
science from usi in switzerland
uh this is the lab that jurgen
schmitthuber is at
he has a bachelor's in mathematics and a
master's
in technically discreet mathematics and
computer science
um shane started his work in machine
learning
at the university of i believe auckland
maybe
working on the weka project which is a
graphical user interface for
these machine learning algorithms
perhaps it's the wrong place but
he worked on the weka project
and then he did his master's on
universal induction um
so he also became a co-founder of
deepmind during his time as a postdoc at
university college london
in the gatsby computational neuroscience
unit and he received the singularity
institute for artificial intelligence
prize
uh which was ten thousand dollars
canadian for his phd work
we got the next slide
uh the next author is probably quite
well known uh
dr marcus hutter he's now a senior
research scientist at deepmind
but formerly he's a professor at
australian national university
he's notably known for the ai
xi model or ai model depending on which
version you'd like to look at
he also worked at i idsi a usi
and he was the supervisor of dr leg at
this time
we can go to the next slide
so the primary focus of this paper is
the following problem and that's what
nobody really knows what intelligence is
and that's what they say in their
abstract
um in watching talks hudder has given or
not leg has given on his work
he's described the two types of people
that talk about uh
ai research in particular and one type
is someone who works on computer vision
which we'd now probably extend to the
area of
machine learning or deep learning or
reinforcement learning or other
algorithm-based
problems whether they're classification
or something else
these people generally will say well
yeah i'm not really working on
artificial intelligence i'm working on a
narrow subset of machine learning which
creates a high resolution images or
classifies images or
gets a car to drive and they don't
really claim to be solving intelligence
they claim to be doing a
much narrower problem but because of
branding or other reasons they just kind
of say
yeah this is artificial intelligence it
looks good that's becoming more and more
popular now
and the other type of person that like
has talked about
is one who claims that intelligence
research is going to be people who are
working on creativity or compressibility
or other
language that's not well described
and these people he he has less respect
for
because the first kind of people at
least know that they don't really have a
clear definition
but the second kind of people claim that
intelligence is too difficult to work
with
which is quite interesting
so what he did in this paper here is
he collected informal definitions by
experts and he tried to extract their
essential features
as he mentions going through every book
that talks about a method of describing
intelligence is just not feasible
because
there's so much other information that
surrounds it especially since this work
was published in 2007 it
you know full text search of entire
books was even slower than
so we can go to the next slide
another problem with intelligence is
that it's usually defined with respect
to humans and that's what he calls
anthropocentric
uh it's not that great
if you can only define it with respect
to one system
and if you guys looked at the sections
that talk about intelligence
whether that be an iq test or a g factor
usually they're standardized via
something like
age and again we're not even getting
comparison of
intelligence among all humans it's
comparison of intelligence among a
subset of humans
so you're really only able to make
in-group comparisons
and the problem with that is that we
can't compare the intelligence of
vastly different systems so he talks
about systems that have
completely different sensory features or
different processing mechanisms
so we can go to the next slide
so if you get nothing else from this
talk
please get the text in the green box if
you're color blind i only realized that
after the fact it's the only box on this
page
this intelligence measures an agent's
ability to achieve goals in a wide range
of environments
i'm focusing on this because we're in
the ai safety reading group and many of
us are
concerned with what happens when
systems become super intelligent if you
have a clear mathematical definition
maybe you could define it
but this has key implications for
something like
the orthogonality thesis which says that
intelligence and
goals are separate so if we all have an
agreed upon definition
we can actually start working on making
coherent statements together rather than
just disagreeing over semantics
i was going to provide a image of like a
non-continuous function and ask someone
what the derivative of it was at this
point where it's not continuous
and then i was hoping some people would
say well we can't
find the derivative of that function
because it's not defined
so if nothing else remember intelligence
measures an agent's ability to achieve
goals in a wide range of environments
we can go to the next slide
okay so when we talk about
measures it's this uh equation i'm
probably
butchering the name of the symbol but
it's upsilon not epsilon epsilon
uh at least that's what it is in the
tech of pi
is defined to be the sum over
environments of two to the negative
kamal graph complexity
of that environment multiplied by the
value of that environment
following some policy pi and pi is the
agent
there's way too much mathematics in
there to get it in the first pass
so we'll try to break it down uh the
important thing to know here
is that if you want to do science i i
don't know if he mentions this by the
exact phrase
you need to be able to define what
you're working with and
hopefully be able to measure what you're
working with
a lot of discoveries first require like
key insights
for instance solving uh getting
superhuman chess
required understanding game trees
and other discoveries require something
like
figuring out where the problem is and
how hard it is
as a brief aside the modern day notion
of probability theory
where we can represent something as like
uh
the likelihood of an event happening is
relatively recent only being discovered
in the 1930s
uh due to this person kamal groth i
don't know his first name
um and he also invented this notion of
kamalgrove complexity
so fun fact we can go to the next slide
so we had
this kind of dense mathematical equation
so let's briefly just talk about it so
an agent is something that can select
actions that can move right
move left output some text that's
generally what he's talking about
an environment he represents by the
letter mu
he defines it with respect to
probability which i just mentioned
but explaining that in a 20-minute
summary
is a little bit more difficult than i'd
like to if you'd like we can discuss the
specifics of it
after the talk during the discussion now
if we go back to the text definition uh
intelligence
needs to be across a wide range of
environments so they define
the environments which as a set and then
they just define it to be the set of all
computable environments computable is a
restriction that you need if you want
this to ever run
there are some philosophical
implications of it but it's usually not
a problem
goals are represented by some value so
usually
how much reward you can get following
some
policy function it's usually discounted
in some
way it can be represented by tables or
other stuff
but it's just representing rewards in
this case and then you have this
complexity penalty
i didn't have time to draw out why the
complexity is two
um but it's it has to do with binary
representations
we can go to the next slide i think
so the other key idea is that in theory
you could compare the overall
intelligence of agents
um in practice we can only approximate
the intelligence of agents across
a finite set of environments prior to
the talk it was mentioned that this work
isn't necessarily
grounded uh to
respond yeah uh you can't actually make
this comparison i made i
i kind of used this as a almost a
counter example you can't
with this definition compare the
intelligence of a calculator or a baby
in theory you can but in practice you
can
now a feasible solution to this might be
to compare the intelligence of a
calculator at some task
versus the intelligence of the baby at
some task for instance the calculator
can't burp
it can't be padded on the back because
it doesn't have a way of
it doesn't have the sensory system for
any of these things
i think we can go on to the next slide
the key ideas are that we have a
definition
and we in practice could or in theory
could use it but we can't
in practice um uh one of the assumptions
in the model of universal intelligence
is based on this idea of a reward
hypothesis so in the beginning
we started talking about the the paper
discusses
uh human mechanisms of measuring
intelligence so we're only comparing
humans within groups specifically
children and then we want to extend it
to other types of humans so like older
adults
children of various ages then they
compared french children to american
children and different
intelligence tests had to be developed
to compare
these different groups but
we other intelligence work wanted to
compare
animals and the problem that the authors
introduced was that
you can't tell an animal what you want
it to do
uh you have no way of saying hey can you
go fetch me a cup of coffee or
sit down please stop jumping around so
what
animal researchers have tried to do is
uh coax or guide
animals to doing some goal that they
wanted and usually this is done
via reward sometimes it's sucrose
sometimes it's it's sex
sometimes it's uh neural stimulation
or sometimes it's you want to guide them
by punishment so you'll
put shock wires somewhere and you'll try
to get them to stop doing a behavior
continue doing a behavior based on this
negative reward
ultimately these mechanisms for trying
to get
a system in this case an animal or a
machine
are based on this idea that you can get
these system to achieve a goal
using some reward now you can go to the
next slide
the idea is that you can do any goal
with a reward but that introduces
a really big problem to achieve goals
one needs a reward signal there's two
papers
by deepmind that i don't think people
are familiar with one is psyclab and the
other one i don't have the name of
but they introduce environments where
they'd like to train their algorithms
and then they like to compare them but
they quickly abandoned these techniques
because there's the reward wasn't well
defined in these environments and what
happened is people said oh well you're
just
uh cheating by putting reward in this
environment to see how
a system will behave like you're you're
essentially
you're coaxing it and that's not real
intelligence so they moved on to
games where they could more easily
demonstrate the universality of their
algorithms
but to really illustrate what i mean by
this we can go to the next slide
um the word hypothesis kind of becomes
like
i don't know i don't know if it's
trivial it's not the word um
it's not as strong as people think it is
because if we had
a good representation of rewards for
goals we'd like to do
we could just specify a system to
achieve that goals
i gave i think this example looked
really good this is a game called temple
run i think it was popular a while ago
you can see these shiny pieces here uh
this agent just goes forward the whole
time there's no way to stop it
and there's hazards in the way so you
can go left right up
i don't think you can go down and then
you try to collect
coins and the task becomes more and more
difficult over time
by increasing the amount of hazards and
the amount of actions you have to take
so you can clearly be super intelligent
at this game if you just collect as many
rewards as possible
and avoid dying but for many tasks
in particular mathematics rewards are
sparse and
it's very hard to define so
i'd like to point out that limitation of
this model is you have to frame
everything
in terms of rewards and that's
really difficult to do so that's why you
see
very few reinforcement learning robots
in the world
we can go to the next slide
i am gonna use this uh for the
discussion
i spent a long time making it so please
somebody ask a question about it even if
you don't care
later please um essentially this is a
comparison of tests and definitions
there was a table in the back of the
paper but that table used
uh big dots and small dots it was
very hard to parse
so we can go to the next slide this is
just something for reference later
um there was something that was
mentioned it may
have been a technical detail that people
skipped over if they were looking at the
mathematics but it's important to know
that rewards are
bounded the reasons for this are
are a couple fold you can't they they
chose this way of representing it so you
could have a super intelligence
um ai xi being the super intelligence
which is kind of like maximized in this
model
but it's interesting that aixi was
created prior to
this definition uh i think i'd have to
double check
but um anyway there is a super
intelligence by this model but we don't
know what the precise limit is
i show a number line here just because
it's continuous values between
a and b and you can actually continuous
probably isn't the right word because we
have a computable agent
but uh there is a way of directly
comparing
intelligence of these systems
theoretically meaning
that you could have the baby as 0.2 in
the calculator as 0.1 or vice versa
or whatever the number is you could
numerically represent the intelligence
of systems and we can go to the next
slide
okay that is the reference to the paper
um
if you guys would like to see it please
let me check if i have anything else
that was really important for the
discussion
there's different ways in which machines
can be represented animal tests and
human tests
don't necessarily compare well but i
really wanted everyone to get
the key insight which was in green that
intelligence is defined
to be the ability to achieve goals in a
wide range of environments
and the reason i'm presenting it here is
that
again this has implications for the
orthogonality thesis
and for instrumental convergence as well
as various problems and ai safety
so we can go to the next slide
so thank you guys for listening
thank you very much for your
presentation


Emiliano De Cristofaro: Understanding the Weaponization of the Web via Data Driven Analysis

today to talk to you a little bit about
uh our work uh we've been doing with
with a number of colleagues
um first funded by uh a new project
on on cyber safety and now uh with some
sort of virtual
international lab that that has
several members scattered all around
around the world called the
drama lab uh on what i generically
define
as a weaponization of the web these are
sort of a set of issues uh
related to uh to safety cyber safety
uh or like in the uk they like to call
it online harms
um and i'm gonna try to to walk you
through through some of these issues
uh using some some case studies i would
say
and um a lot of our work is is
data-driven so
um and what i mean here is uh it's
really through the lens of
a sort of large-scale quantitative
analysis
which is you know a part of the of of
the story right so
um there is also a lot of great work
happening
um at a smaller scale or you know with
qualitative analysis and um
you know there's kind of our hope and
you know what's going on is that
these sort of two lines of work inform
each other and support each other
and uh i'm happy also to to report that
you know in the in maybe more recent
times
we have been growing collaborations
across these two these two words
all right so um before i uh i dive in
uh sort of a warning that there might be
content in this talk
um they may be perceived as offensive um
and you know we we decided um you know
not to censor uh content in this in in
our presentations
if anything because you know where to
sort of draw the line
of what is considered offensive it's
actually pretty hard and this is part of
what makes this research
uh challenging so um you know sort of
toxicity and
offensive content is something that is
very subjective it's very
um context-dependent obviously you know
there's
there's clear examples that you know
nobody can really debate but
the others are a bit more nuanced right
um and overall we wanted
we wanted to sort of give you the
unfiltered uh look
at what you know what our data looks
like and what these communities that we
study
uh look like um so i apologize in
advance and you know if
if anybody has you know any issue i'm
you know very happy to
kind of skip over slides or you know if
you just want to drop out of course
by any means uh um please you know no no
no hard feelings um of course all right
so
um like i said is a concept of
information weaponization is a very
broad genetic one
and you know it kind of happens through
a number of
of sort of problematic uh uh issues
uh ranging from cyber bullying cyber
aggression which is
sort of a coordinated um set of attacks
um targeted to to specific uh
people or groups of people um
radicalization
misogyny misinformation propaganda and
so on and so forth right so
you know this is uh these are some
examples of of of things that
uh that the research community has uh in
cyber safety online arms
online extremism however you call it has
been focusing on
and uh you know obviously these are uh
very important
uh socio-technical problems and that
have attracted a lot of
um sort of interests from the research
community um
and you know from both from uh um
from the point of view of solving
technical issues and also from a
societal
uh importance point of view of course
right and uh you know my
as as mentioned in the introduction um
you know my experience comes from like
the cyber security
and privacy technology side and you know
a lot of
of us in this field have had a lot of
experience in
doing research on so mitigating unwanted
activity on social networks
right so um social network on the web in
general right so
you can think of um sort of problems
like uh
spam right that's an unwanted activity
you don't want
i i assume that nobody wants to receive
spammy emails
in their inbox um and there are in
general
a lot of tools available to uh detect
sort of automated
unwanted activity or you know activities
perpetrated by by bots
right by sort of automated programs
right and
we've actually done a lot of great work
in in
in that space um you know i think
if if you use like gmail or you know
outlook or you know this kind of large
providers
they are pretty good at detecting spam
of course they're not perfect but they
can really rely
on the fact that uh you have sort of
large-scale
synchronized uh unwanted activity that
you know sort of really exhibits
some um some clear patterns right um and
so this actually does not
um apply only to spam but in general to
uh you know both activity um in in
social media right so
you know we can be spam can be um tools
to
uh um like do this kind of like forms on
facebook or
um you know networks of of retweets
inflate number of followers
in inflate engagement and so on right so
there are machine learning classifiers
that
that we can build uh that are able to
model
uh these recognizable patterns and have
uh
reasonably accurate uh outcomes
on and like these examples on the slide
i think the trapezoid cohort barometer
uh or box layer um you know they're
they're pretty
they're pretty good with some caveats
okay and this maybe it's
a conversation for for not to talk okay
but what can we do really i mean can we
actually use these tools
can we apply it with the same intuition
to solve some of those issues that i
mentioned in the information
organizations like cyber bullying cyber
aggression toxicity and so on
well i mean the first uh observation is
that a lot of those issues
are actually perpetrated by humans by by
um
a lot of those activities are prepared
by humans right and human activity
overall
uh is less coordinated right than than a
script that is
you know launched at a large scale so
characteristic traits and these patterns
are much less recognizable right uh if
anything
you have a much looser synchronization
of activities
um toxicity can come and go at different
times
uh and you know real users uh use common
talking points
and so it makes it uh much harder to
sort of recognize
um you know sort of differentiate
and this activity from what whatever is
considered normal activity
right um and also you know
a lot of the um the tools that we've
used for
uh uh automated unwanted activity
detection
really take the fight on economic
grounds like for instance for spam
or you know for like preventing uh
denial of service attacks
what we do is we actually make it uh too
expensive
for the attacker to launch his attacks
or we make it less profitable
right so in the end the attacker you
know doesn't uh send spam
if he cannot if they cannot um profit
from it right so they will move to some
other platform that will move to some
other kind of criminal activities let's
say right
um but in this case you know not always
things are
have to do with money right so um or
you know even if they do uh you know
some some of these activities
may involve adversaries with deep
pockets like for instance state-level
adversities that want to
interfere with elections or you know
want to
uh spread some kind of uh propaganda
that will you know drive uh um
they will will drive ideology in a
certain direction or
or something like that right so um you
know this
sort of is our first observation uh when
it comes to this problem
uh the other one is that um you know
online services do not exist in a vacuum
right and it's not only
um you know considering things like you
know each social network
is distinct from each other right so you
know
these social networks really um impact
each other so content that
you know may come may originate from the
reddit social network
might then spread and you know go viral
let's say on on another social network
like twitter and facebook and so on
but also you have different sort of
ecosystems let's say that have uh they
obviously are really have a big
interplay with each other
right so you can think about content
providers and social networks and
uh news providers and so on right so
when it comes to this kind of research
uh looking at a single service at a time
is not enough and unfortunately you know
we are used to
really focus on like one platform at the
time one problem at the time
and you know if anything for just like
for
both for um what it means to do data
collection
but also with respect to the tools the
analytical tools that we have
right so this is was one of uh still is
one of the biggest challenges
uh from a technical point of view in
this space
and the next observation is that uh you
know in these ecosystems you
have both what you may refer to as sort
of mainstream
uh communities or mainstream platforms
think about social networks
right you have uh um social networks
like facebook that has
billions of users a twitter mainstream
but then you have
so-called fringe communities right and
uh
um for instance like sub communities of
reddit
called subreddits that are you know sort
of fringe or extremists
you have uh actors like uh communities
like 4chan
han um and new uh communities that's
that
sort of come into into the picture and
more and more often like gab parley a
vote
poll and so on and so forth right and
this french community is actually you
know
they're considered niche uh they're very
small uh in footprints in their
footprint compared to you know facebook
has billions of users
but you know fringe doesn't mean
unimpactful doesn't mean
unimportant and in fact uh we have and i
i'll
tell you some examples in a little bit
we have actually found
uh significant evidence of sort of
of these communities playing a really
important role
uh visit the information weaponization
be it uh misinformation or
disinformation campaigns
uh coordinated hate campaign um
[Music]
production of weaponized highly toxic
uh racist or politically charged memes
and so on and so forth so
it's also important to look at these
communities okay
and so here are some examples right so
there are actually coordinated efforts
that
you know spill on on mainstream uh
social networks like twitter
that actually originate from from
communities like uh 4chan
this is a classic example of a
minority being sort of pushed off
twitter or
mainstream network by sort of a
coordinated mob
uh attack uh uh some coordinated attack
by by a mob that
was was originating and coordinating on
4chan
the same happens with conspiracy
theories like
pizzagate which now has somewhat evolved
in
into q a and
overall you know you see that the
actions
of these things are not just confined um
on these french communities nor on these
mainstream platforms as
a result but actually still in the real
world right so there is a lot
of of evidence that sort of this
radicalization that's
if if you allow me to to use this word
um
of of users on french platforms have
then
uh resulted into real world violence uh
like mass shooting
uh incidents um in in the us
and in australia sorry new zealand and
elsewhere
okay um so um
okay so look there is a you know in
these examples i mentioned already
for chan a few times so i'm just going
to uh
talk a little bit about 4chan and like
sort of use it as a as like i said a
case study
of of some of our work um so first china
is a
it's an image board forum uh that has is
really integral part of internet culture
let's say you might have heard of them
as they produce a very large quantity of
memes that ultimately go
viral like the love cuts
back in the day they serve as a
coordinated platform coordinating
platform
for the hacker group anonymous
they have really sort of mastered the
art of trolling
and made microsoft chatbot
tie racist and say
say things like the holocaust didn't
exist didn't happen and other sort of
racist things
and they have produced like a number of
sort of internet culture
memes and and and narratives and um
notations let's say and expressions like
pepe the frog
which was actually a sort of an innocent
cartoon uh
that was created by you know sort of a
non-political uh
uh person and then was kind of
appreciated by by fortune and became a
symbol of hate
um and and and so now it's for instance
banned on
a number of platforms um and so
it and this is sort of another um
very important observation to make is
that um
a lot of these issues really do not
necessarily unfold through
just text but a lot of things happen to
images
so for instance paper the frog was you
know it's usually sort of
morphed into different characters and
different things and during the 2016
present
presidential election in the us you know
the picture of
trump uh looking like pepe the frog went
so
mainstream that actually the the uh the
then candidate donald trump
retweeted itself and like i said paper
the is is a designated symbol of
hate
by uh entities like the anti-defamation
league
in the us and a number of other uh
organizations right
and i will come back to to this uh later
in the talk
um so fortune really got involved and
sort of in politics let's say
or through some uh some part of 4chan in
the 2016 elections
to the point that you know really the
community took a lot of pride
and a lot of credit uh for sort of
electing
uh trump um and there are many examples
that i that i can
that i can cite um so like i said
fortune is an image board forum
uh well it's essentially um um
so you you create a new thread in the in
the in a board by
um making a new post and the new post
has to have a c
an image attached so that's you know
also tells you
uh that is actually a lot of uh of image
imagery being posted on 4chan and a lot
of it is
sort of um original right so we
i i forgot but we we didn't have uh uh
a measurement of how many unique images
are on 4chan
and overwhelming majority are actually
unique so most of it are like sort of
photoshoppings
um so you know there is some sort of a
basic subject like paper the frog and
then you add text or you add like some
some other
uh elements and you make a new image
right
other users can reply in the thread by
um
you know adding posts and this can or
cannot have images okay
uh so these are the 70 boards that are
uh
active at the moment uh we focus on
particularly on one called politically
incorrect or tall
which really has extremely lacks
moderation almost anything
goes like nothing really is removed
um and one of the features of 4chan and
paul in particular is
um that everything is anonymous right
there is no concept of
uh of user account um you know in order
to post you just need to solve the
captcha
so they in that case they can avoid sort
of automated
um posting by scripts of you know
spamming the board uh but you don't have
to
have an account there is some degrees of
permanence
uh identifiability that is supported
something called trip codes
i'm gonna skip this for uh for the
interest of time but
if you are interested or if you looked
at q unknown
conspiracy theory this is relevant for
that case this is
kind of like a way where so q was using
the q was using to prove that it was the
same person
uh um sort of licking these uh secret
documents
um the other the other important feature
is ephemerality so
all threads are archived and eventually
deleted after a while
so there is sort of a very short
attention span span
um like all so all content is removed
after a while right so
i think that at most things last about a
week okay
and there is sort of a complicated um
not so complicated but you know sort of
a system
to determine which thread is deleted
first so there is essentially a limit on
how many threads
uh can be active at the same time and
then you know if you make a new post
uh the thread is bumped up to the top of
the of the board
and if you make a new thread then the
last uh
thread gets deleted but there is a limit
on how many times
uh the trade can be bumped so eventually
everything is deleted right so and these
affect a lot
uh conversations on unfortunately if
anything because there is no memory so
like certain posts and certain
narratives um
sort of reappear uh periodically so
you know one good example is is our
paper on 4chan
that you know after after every few
weeks
someone posts like oh my god someone
wrote a paper about 4chan
and then they start discussing it and
there's a number of conspiracy theories
about it like
if we were funded by soros or by the un
and a bunch of other things and
this sort of really periodically uh
happens all the time so as part of our
work on 4chan we actually made a data
set available
uh through a conference called icwsm
that has a data set track um that'll you
know sort of really encourages people to
to share
uh data so we have been crawling we've
been getting data from 4chan
uh since june 2016 and then you know the
end of 2019
we released a snapshot of the data so
about three and a half
years worth of data there's about 3.4
million threads 130 million posts
and you know this data is available on
xeno so you can download it and
uh we augmented the data set with um
toxicity scores as obtained from
google's perspective api
there's different labels like toxicities
severe toxicity and so on
um these are
you know scores of how toxic a post is
and should not be taken at face value i
also want to say that so google
perspective api is not perfect
it's a automated library that sort of
gives assign scores to
text uh you know like i said for how
toxic it is
but it has problems right so it can be
circumvented like if you change a little
bit
the spelling of certain toxic words you
might be able to
evade the classifier it's also been
there's also been studies showing that
the api is somewhat biased against uh
things like um african-american
uh um english for instance it's
is scored as more toxic than uh
than it should essentially so you should
not take
these values at face value and they
start sorry this chorus and face values
but they're good to sort of compare
things so if you want to compare like a
board or four channel with another board
of 4chan
across distributions uh this might give
you some
some interesting insight um we also
released what we call entities
sorry named entities that are mentioned
in each post so these are things like
um concepts or names of celebrities
so you can search and we have uh
collaborated with a number of social
scientists
for instance to retrieve uh all posts
mentioning donald trump right so you
have this uh um
this easy way to uh to search okay
um right so um and so what you can do
with this data is you know of course
sort of a generic
characterization uh for instance like uh
understand more or less how you know
sort of toxic
and hateful content is so it's pretty
bad
as you can see in this slide uh you have
a very high percentage of posts that
have
at least one um hateful keyword so this
is a very simple dictionary based uh
approach
we use something called hate base is a
dictionary of hateful keywords
but i also show this slide to to to
sort of uh highlight how much you know
these things are
are hard to uh to do in practice because
you like i said
you you might have even keywords simple
simple single keywords that um
may be hateful or not depending on on
context and the stupidest example is the
word frog
which you know can mean uh you know the
animal frog
or can be pepe the frog which in some
cases like i said is used in
in hateful context or maybe the hateful
um you know word for to to to call
french people right so it's a um
it's you know this is uh um
derogatory yet i couldn't remember that
the word
derogatory term for french people for
instance so you know if you
want to look at data and determine you
know hatefulness let's say
you know it's not easy to do it at scale
like i said you know
frog for instance can can be hateful or
not
depending on the contents uh if you look
at the prospective api again
you know um do not take it at face value
but you can see like this is a
distribution a communicative
distribution function uh that shows that
really an overwhelming majority of posts
have
very high toxicity scores um and
you know severe toxicity is a bit more
robust um so i usually encourage people
to use severe toxicity rather than
toxicity
uh and you can there's other labels like
inflammatory profanity and so on okay
so overall when you sort of study these
french communities you have a number of
challenges that you have to face
okay so i already mentioned that you
know actions are not limited or
you know obviously these things don't
happen in isolation so a lot of
of things that are talked about on 4chan
are influenced
of about influenced by what goes on on
other platforms
um the classic example is the gamergate
controversy
um you know was uh primarily happening
elsewhere
but of course it was discussed and
certain actions were coordinated on
4chan
uh or you know certain things that
originate from 4chan then
you know have a big impact on other
platforms uh but
overall even if you consider sort of uh
4chan as a in a
in a vacuum let's say it's clearly not
your typical social network right so you
have this anonymity in the funerality
that really changed the way things
happen
uh ephemerality if anything like like i
said uh
it creates this like sort of lack of
memory on on the discussions and so
it really uh makes some topics sort of
reoccur
also makes it harder to collect data so
we have to have like a crawler
that um you know periodically checks
when a thread is archived and then
uh retrieves all posts in that trade
thread
from the archives not easy and you have
to have a lot of redundancy to
uh to support that um you know a failure
in the corridor infrastructure means
that you you miss data
you cannot go retroactively like you you
could do for instance on twitter or
something like that
uh also anonymity means that uh you know
there is no way to do
any kind of like user-based analysis if
anything
um i mean maybe there is a way but it's
certainly not not ethical
um there are like sort of stylometric
features you could use to find posts by
the same users
but you know that would not be ethical
so we've never done it
um knowing what you're talking about is
not easy
uh so there is like a number of words
and really a lingo that's
sort of you have to learn to even figure
out what the hell they're trying to talk
about
um it's a bit risky uh it's very risky i
would say actually uh you might get
attacked
uh doxxed so doxing is you know sort of
the
the act of exposing personal information
about a user
uh with sort of really um hateful intent
um we might i i've this has happened to
us multiple times also
something like something called concern
trolled someone sent an email to
my department chair
pretending that our research was you
know detrimental to
some minorities and you know that was
sort of phrased in a way that looked
plausible
so it wasn't really an easy thing to
deal with
um so it also means that you sort of
have to quickly learn
how to deal with these things and how to
protect yourself and
your collaborators your students and so
on and
you know sadly is not something that um
institutionally
we we are geared to really to to to
support
um i most of my of the support i got
actually was from colleagues in
the crime scenes crime science
department i had been doing research on
um terrorism and radicalization
and you know they were sharing some of
the resources some of the support but
you know it's not really something that
we learn just like we don't learn ethics
really you know there's nobody teaches
you in
computer science uh a curriculum things
are changing but you know when i went to
college
uh in the early 2000s nobody was even
ever mentioned ethics in any of my
lectures ever right so
and same in grad school so it's really
something that we have to learn uh by
ourselves
luckily there is you know sort of a
community that help tries to help each
other
but there is a lot more to be done there
okay so actually let me just take a a
breath and drink a little bit and ask if
anyone has any questions
so far
yeah someone questions you can
just pick up raise your hand
no questions all right uh no problem
i do have right a question for you but
it's about moderation on these aspects
well maybe you'll cover in the next few
slides
so if you prefer i can wait until the
end of your presentation to ask this
is you talking about identification also
on some aspects so
this kind of motivation ask me the
question and ask me the question because
i might not cover it if i if i am i
i will just tell you okay great so uh
yeah so there is also the very
interesting
uh approach as you said to identify for
example
uh hate speech and but that's also
uh still stays with the thing okay how
would we identify what is going wrong
there
how would but given that we identify
this
there's a lot of challenge to identify
but how how do you see moderation on
this kind of communities in the strange
communities
is it about having some person there is
about some automated algorithms because
you said like there's this challenge as
moderation with uh
yeah yeah yeah it's a very complex uh
very complex problem and happy to i mean
i think this is a good time to
to answer this question so uh first of
all i mean
you can there is a sort of two kinds of
moderation one
is sort of community level moderation
and one is you know
uh fine grain let's say post or user
level moderation right
so for instance what you see is like on
reddit uh
you know radius organizes communities
the subreddits so periodically you see
some
um subreddits being banned quarantined
or banned okay
and um so you know that's close and
nobody can post on that commute on that
subreddit and then you know um
like users leave and what happens is
that obviously these communities don't
die but they
like migrate to other platforms or they
create
new platforms uh um all together right
so
and so this is one issue which is i
think it's a very important societal
uh problem which probably myself as a
computer scientist
i'm sort of the the last person who
should have an opinion
on this uh this is really for for people
that
you know have uh uh training and have
dedicating their life i've been
dedicated their life on
on sort of the societal problems to uh
to
to study uh because you are essentially
and we have a paper i'm not
going to to present this today but we
actually have a paper that tries to
uh uh analyze what happens when
communities are banned
and as you maybe could expect uh what
happens is new communities that are
formed um
are smaller in size um so when you
de-platform
um like this for instance the donald
community or in cells communities on
reddit
they migrate to forums they migrate to
like things like parley
or gab so these are much smaller in size
so their footprint is
smaller they sort of outreach their
audience is smaller
but they're much much more toxic so
you you see that users even like not
very active users
they get much more toxic and so there is
a sort of an increased risk of
radicalization
right so you have like really two things
here like a more toxic but
smaller community or like a bigger
community that is still toxic
but still there's maybe some you know
cross-pollination with less toxic
communities or overall
there is some kind of moderation like
already you know you cannot just
go and and and you know call someone
like
the n-word but you can on this other
platform
so i don't know how which is which is
better you know like i said i'm computer
scientist
i should shut up and not have an opinion
on this um
but it's it's an important it's an
important problem when it comes
down to you know sort of post or level
or user level
moderation yeah it's very challenging
because
a lot of the so in french communities of
course you know
so so first of all like every platform
can decide sort of a community
guidelines and
you know decide what uh is admitted on
the platform like this is free speech
and you you can decide what what is
allowed and what is not
um and so you know you might think okay
a
platform has sort of this kind of values
but sometimes these values are not
applied
uh in a sort of coherent way right uh or
you know the the values are not really
driven by
some sort of ethical or societal
standpoint but they're they're driven by
you know
economic interests right so like the
classic example is i could say facebook
not
banning political ads you know for for
you know they have sort of market
reasons right
um but anyway so even even once you um
you have agreed where you have stated
some guidelines and you know what is
allowed and
what it's not it's still hard to
uh stick to to those guidelines uh
because moderation at scale
is just not possible like you know you
cannot just manually
do moderation of things and even if you
did
manually the moderator is themselves
they will have different views
so there are studies in the show you
know they they ask
moderators to label um 10 moderators to
label the same
piece of of text and you know very few
posts are like
you know the labels are 10 out of 10
agreeing
right um so it's very it's very
complicated
um and the other issue which i'm going
to talk about later is that
um this approach is after ultimately
reactive right so what happens is that
you someone reports that
a post or user is violating community
guidelines
and there is you know some mix of
automated tools and machine learning
and moderators and labeling and you know
then people get their posts are removed
or people get suspended or abandoned on
so you know the damage so to speak has
already been done right so
you've already as a victim you've
already been exposed to the toxic
content
um and you know just that the post is
deleted
you know maybe doesn't necessarily make
everything better right
so it's a very very problematic uh it's
a very complex problematic yeah
thanks thanks for answering so i think
uh
if i can just very quickly react to this
i can say like yeah
that's definitely it's a very
multidisciplinary challenge right so you
do as i said
as a computer sciencey there's so much
you can do but if you work together with
some
with athletes with social sciences with
uh yeah a lot of
designers and many other audiences then
yeah
yeah that's very important you know and
um
yeah unfortunately you know
yeah so this is
uh ben wagner emiliano thanks for
joining us it's great to have you here
sorry i can't turn my video on right now
um what i wanted to ask is so i'm one of
those crazy social scientists who work
on the content moderation stuff
so i thought it would fit in quite
nicely into sort of the context
and one of the things i was wondering
about you were mentioning the robustness
of google's
toxicity model and specifically that you
don't really
think there's uh helpful the sort of the
general toxicity but only severe
toxicity produces relatively robust
results
i was wondering if you could say a bit
more about that both sort of in terms of
false positive false negatives how that
sort of plays out in terms of
at least your experience of how that
codes things
and the other oh yeah um please no first
then i have a second question but go for
the first one first it's
all good yeah so maybe i misused a
little bit uh terminology and
robustness is it's just a little bit
less
context uh dependent so you have
um sort of a lot of
of a lot more text in the toxicity
uh case that is sort of mislabeled as
you know toxic um you know
based on certain words uh whereas severe
toxicity is
you know a little bit less um
sort of mislabeled yeah so
it's a little bit less uh um
subject to to mislabeling right so
um and it's because it actually uses
uh um a lot more
uh text that was um so this is obviously
the
uh the prospective model is trained on
on text that has been labeled by
by humans right and um
uh so the severe toxicity had much more
agreement
um the model for is much more agreement
across different um annotators
so that's why it's a little bit more
robust but like i said
robust maybe it's not the right word to
use here
um but i hope i hope you understood what
i mean
yeah totally and i think that's exactly
the point that i was trying to get at
because it's quite interesting when you
look at the
basically i i feel so we've had some
uh law students basically go through
coding data sets like this from a legal
perspective
and once they did that they came to
relatively high levels of agreement so
higher than 90 percent or 85 percent
after like they've done some training
rounds
and what i found interesting about that
is that the problem is itself the term
toxicity or harms that we use that are
not sufficiently defined
as to be clearly able to then yeah but
basically that means that even the human
coder stuff you're talking about
measuring the severe toxicity against is
also as you say has its problems
so it feels like a lot of the time
you're sort of you're circling around
looking for a ground truth which can be
really frustrating
no it's very hard yeah so i mean there
is a there is a simple example that
people
make is you know for instance like the
on sort of context dependence in
toxicity right so uh for instance sports
communities uh in the us um
you know sort of the bar of what is
considered toxic is
is allegedly higher than
you know in some maybe political
discussions or
you know some some other kinds of
discussions right so it's it's sort of
so socially acceptable let's say to
insult each other
uh in sports based conversations like oh
you're a fan of this team
and so you know insulting certain cells
uh
compared to you know other contexts
right so if you don't take that context
in in into account uh it's hard to
you know to to to do to kind of uh have
this ground truth right
yeah you had another question you know
yeah no that's uh that was literally a
second question but i think it's really
helpful
and i think i mean i don't want to hug
all of the time so maybe we
chat about this separately all right but
i really found that super helpful for
looking at this thanks
thanks um all right so yeah then um
i i mentioned uh already a couple times
this concept of sort of coordination
of of actions and in particular of
hateful campaigns
so this is something called raids uh
there is coordination of sort of real
world
events so for instance the january 6th
storming
of the capital uh i think that's the
picture i have in the slide
was coordinated on um on mostly
on platforms like partly and and and
other ones
but uh what i'm talking about here is a
a raid we call a coordinator hit
campaign that targets another
platform for instance youtube um so we
observed this
anecdotally at first where someone
posted a link to a youtube video on
on 4chan and maybe with a prompt like
you know what to do and then what
happens is that uh
the youtube video starts getting a lot
of hateful comments
okay um so we we
did like a an exploratory study trying
to
to see if we could actually model uh
this behavior um so you would imagine
that if a rate is taking place
you have a sort of a peak in youtube
comments while the thread is alive
so you know the the number of comments
that that video
gets spikes because of
the coordination happening on 4chan and
actually you might
observe a synchronization uh between
these two things right
uh and so this is actually an example of
things uh of something that happened to
me
i was giving a talk about our 4chan
paper uh
a few years ago and you know it was at a
workshop when
you know there was a some some issues
some some people couldn't
attend and so they asked they asked us
to um
record the uh uh the talk and i just put
it on youtube
um then i was giving an uh an interview
to a journalist
uh talking about 4chan and say oh by the
way here is
you know our work here's the paper
here's a youtube video
and the journalists linked the youtube
video
in their piece so what happened is that
someone
found and found the video and post the
link on 4chan
um and yeah so my my video was was rated
like in a matter of
minutes i had thousands of views this
will
never happen to me uh and um also you
know we have i had um
hundreds i think of of hateful comments
um and also dislike um so uh
people like using this dislike on
youtube anyway so
uh what you can see is that you know if
you consider like time zero
being the time where the youtube link is
posted on 4chan
you can really try to model the activity
right so you see that 14
of videos uh that were posted on 4chan
see
a peak of comments uh um during
the time where the pull thread is is it
was active
and actually that things were
synchronized so there is a way
uh using signal processing to measure
the synchronization of time series
um i'm just going to skip the details
for the interest of time
but you know visually you can sort of
plot the youtube comments sets
that are hateful and the youtube content
comments are
not hateful and you can see that you
know they're really
most of them happen uh around the time
where the
the thread is alive right where right
after someone posts the link
on on 4chan so we sort of defined his
metric
called uh hates comments per second uh
which became a meme on 4chan now people
um you know mentioned this uh uh
repeatedly
and uh we also looked at um you know a
few
few other things but so we we sort of
found some quantitative evidence that
um when people post youtube videos on
youtube links on 4chan this may make it
rated but we flip things around and and
we thought
okay can we actually try to uh go one
step forward
and not just simple simply you know
uh observe that something is taking
place uh the rate is taking place and
wait for somebody to report it
um and you know just actually trying to
predict
uh uh that something like that might
happen right so this will allow us to
um you know have a more proactive
approach that does not
rely on um you know reports and
moderator
going through through content which is
an approach that's obviously it's
inherently slow
so we thought what if we take a video
and
we try to determine whether or not this
video
is at risk of getting rated by 4chan
okay and uh so again this is a
completely proactive approach right
doesn't even
wait for like the link to be posted on
4chan
right so what we do is we try to model
what
what are the videos that that are likely
to attract hate
uh from 4chan so what we did we we
constructed the data set
um of so we had sort of ground truth
here
some videos were rated some videos were
not rated
and we try to understand what are the
features of the videos that
do get rated uh and in the end we
extract a bunch of features and we use
an ensemble classifier
to predict on unseen data right so on
videos
uh which we for which we don't know if
it's going to get rated or not
whether or not you know the it will get
rated or not uh
whether or not it will it will get rated
so it turns out that there are some
distinctive features
uh like the topic being covered by uh
by by the video of course um you know
the
the elements in the video so we use for
instance like an image recognition
library that tells okay this video has
like a man in a suit
a man playing rugby or something um and
also the metadata is very important like
the description
that you see on the on the youtube page
itself
and it turns out again i'm skipping the
whole thing here but
turns out that there is some some uh
hope that we can actually flag
uh videos at the time of upload that may
be targeted
uh by 4chan raiders so you know you can
have proactive measures where you know
for instance you
um keep an eye you know algorithmically
speaking
speaking on the on these videos or
you've warned
the users uh that this might happen
i don't know right so there is a tool uh
here that that we can use right
it also tells us that you know we can we
have
some hope of trying to model uh the
content that might attract
uh hate okay and not much more work is
needed here of course
so the other the other thing i i wanted
to talk about is
you know i mentioned before how
communities influence each other
and i i'm going to do this through the
use case of sort of disinformation
uh campaigns and sort of disinformation
news um and so an example
that sort of motivated our work was the
pizzagate conspiracy theory
which was something that originated
during the 2016
uh um u.s presidential election campaign
so there were some leaks uh email leaked
by by wikileaks from the
dnc the democratic national committee
and
in this email someone mentioned like a
pizzeria okay
and um someone of 4chan put two things
together and
and claimed that um the email pointed to
uh evidence that the democrats and
hillary clinton were running a pedophile
ring
from the basement of a pizzeria in
washington dc
um so this you know 4chan acted like a
theory generator
and then the tier incubators and sort of
the gateway to the mainstream world
were platforms like reddit and uh
websites
uh like infowars and breitbart and so on
and so this eventually really went
mainstream to the point that
um someone uh did a study the show that
actually it's against shifted some some
votes
uh in that election so it went on
twitter went on facebook and so on
right so we we try to say okay can how
can we somewhat
model quantify how much influence
um a platform like 4chan
might have on these kind of things so
the idea is to look at the appearance of
alternative and mainstream news urls so
you know mainstream urls you can think
of like i don't know bbc
or cnn alternative news think about
breitbart in for words um daily mail
these kind of things right
um we took a data set um that some
journalists had published
um so and what we do is we build the
sequence of appearance for each url
according to timestamps so imagine you
have a brightpearl url
uh first it appears on on reddit then on
4chan then
on twitter right so we can build the
sequence and um
we can accept the sequence and build a
graph and
use a statistical uh a tool called hawks
processes
that allows us to quantify the influence
of
an event occurring on one platform
um having you know sort of the influence
on on the other platform
right so um essentially what you try to
look for
is impulse responses uh that an event
might might cause okay and so this gives
us some
because we are able to model the
background rate
uh so like what is the expected um
sort of number of occurrences without
the event
on the other platform we can start
reasoning about
casual relationship uh so like try to
have some confidence about the number
of events in this case the sharing of a
url
um caused by another event okay
so i'm going to sort of skip the details
here but
sort of the the main takeaway here is
that
seemingly tiny web communities like
4chan
or certain subreddits like the donald
subreddit
can really punch about the weight class
when it comes to influencing the greater
web so they actually produce
a lot um so they they take these news
urls
and by posting it many times it actually
you know exposes people that are also
active
on mainstream uh social networks let's
say like twitter or
more mainstream subreddits and
eventually
that those urls
spread spread there because
of these french communities okay so
and we actually did this both for urls
and for memes
for for image memes right so um you know
we all played with memes and you know we
think that memes are fun
um we send it to to each other but in
many cases
you know memes are not fun at all uh
like paper the frog like i said
it is a meme but it's a hateful one uh
is used
for instance this kind of examples to
sort of convey
very hateful problematic
issues and they are increasingly used
also in political campaigns
i think you know especially the 2016
presidential campaign was really
where memes um became like came to the
forefront of sort of the political
campaign
um and were used and even sort of
retweeted by uh
trump and um his sons and so on
um so and they actually
have been often um sort of
uh discussed in alongside
um real world acts of violence so this
is for instance the picture
of um of a guy who went on a shooting
shooting spree in florida his van
was covered by you know these 4chan
memes
um right so uh what we do actually here
we try to
uh to understand what memes are
shared on on platforms like 4chan gab
reddit twitter and so on but also how
much
sort of content uh con like meme content
that originates from french communities
like 4chan
ultimately gets to to mainstream
networks so again
estimate the influence of of this
platform and we have like a really
nice uh cool processing pipeline
uh that is open source and it's been
used by other researchers now
where you can take some images find
um visually similar ones using an
algorithm called perceptual hashing
and clustering them together so you find
sort of variants of the same mean
um or sort of like memes that you know
that really look
look the same and we have an automated
way to label these memes
we use know your meme which is some sort
of encyclopedia of memes
as a website um so we can actually
you know label them and say where are
these memes about um
and so on and then we can look at the
influence estimation
uh again uh i'm gonna skip the details
but you can see like what are the top
names for social networks you see very
sort of disturbing things on
on things like 4chan less so on twitter
um
we can sort of find this cluster of
memes uh and
actually on the on the paper there is a
link to a website that
um has sort of a browsable version of
this picture so you can click on these
clusters and see
examples of memes and
you can also see how you know the um
the production of memes is correlated to
certain events like for instance for um
you know especially on platforms
uh um like gab but also on twitter
you see like a spike in memes being
shared around
things like presidential debates or
elections and so on
but again we can quantify the influence
and you know long story short sorry if
we look at the influence
um sorry so we look at the sort of
pure influence we see that paul is very
very influential
in spreading racist memes onto um
other platforms including mainstream
ones like reddit and twitter
and but when you look at the normalized
influence so
normalized to the number of posts of
meme posts
actually specifics sub-communities have
read it like the donald's
are even more uh are very efficient
right so they produce
a a sort of a smaller number of memes
uh which ultimately end up uh being very
um
you know end up being on on mainstream
networks
all right so um i'm going to skip the
rest of of
of stock so we we still have some time
um to maybe answer questions
um okay so sorry
okay so this is it i just wanna
acknowledge uh the eu project that
funded us in case
um and my collaborators at the idrama
lab
which is again it's a virtual
distributed lab of people across the
world
thank you very much thank you very much
really
interesting fascinating topic i think i
have a good time for one question uh
ceda
please yeah emiliano um
maybe you got to this but for a moment i
had to step away but like
you know we we heard a lot about
algorithms being optimized to increase
engagement
um and you can imagine that part of the
what are what you're seeing
is that you know they're media moments
that companies
benefit from by increasing engagement
like i don't know how much for example
twist
twitter and facebook uh let's say um
really up the viral
virality of messages during media
moments right like
like whatever the presidential election
or debates being won
or sports events being another um
and you know we saw this also with
gamestop where you know members of these
groups are very savvy about how these
platforms are basically
full of algorithms that amplify their
their effect
right and so i just wonder um how much
do you think
the problem is this kind of gear
of these companies to grow by increasing
and optimizing engagement
that is quite easy to analyze from the
outside so that you can get this
efficiency gain
and how much like are you trying to kind
of band-aid the fact that they continue
to grow
using these efficiency algorithms for
optimizing engagement
enabling these parties and you're just
allowing to pick up the guys that are
disturbing that
that infrastructure right so i'm just
wondering like how much
would you like if you had a magic wand
emiliano well how would you change this
underlying infrastructure so that you
don't have to do what you do now
this is the question yeah i mean i
i haven't done work in this space myself
but uh
i mean there are other scholars who have
and you know they've
provided evidence of how you know
platforms
like you know facebook and twitter and
so on uh benefit
from you know sort of the polarization
right and and and
what you mean so i think that uh
you know it's in it's in their economic
interest to
um maximize
engagement and one way to do so is to
maximize
polarization right that's the evidence
that we've been seeing
um so how to you know to disrupt that
uh personally i think that the only ways
to
through policy um i i don't think that
there is much that we can do
on on this side uh we can expose we can
do uh measurements it's unfortunately
very hard to do
any kind of of analysis of facebook
because we
don't get any data so it's impossible to
to get data
they sort of promised that they would
make some data available
um you know through this one
facebook one or something like that
program
was we were very excited there was going
to be differential privacy
there and and unfortunately that didn't
really work nobody got the data
and the data is useless okay um
so that but still there are people that
have managed to
do some kind of measurement in that
in that respect it's a little bit easier
on on twitter because we have
access to at least you know one percent
stream through through
academic projects
which is is perfectly enough to
monitor sort of large-scale phenomena
it's you know the one percent is a
representative enough sample uh for this
kind of uh
behaviors um but yeah so we can expose
that but
i i don't think that you know like even
if i did have a
magic algorithm one i don't think
there's really anything that we could do
to disrupt
uh from the outside so the only way to
sort of disrupt from the inside is
through policy
i think i don't know
we will have to take it offline okay all
right
all right okay yes thanks
thank you very much emiliano thank you
for raising all this they have very
important issues
telling us about your work really was
fascinating so thanks everyone for
joining us
and yeah i'm sorry maybe i i didn't
leave enough
enough time i i actually did not see a
watch for a question but i'm happy to
stay a bit longer or you know i'm happy
to
you know answer questions uh offline via
email
or signal or whatever um so
you know please please reach out um i'm
happy to talk a little bit more
great i'll stop the recording now but
i'll leave it let's keep the
room open so whoever wants you to talk a
little bit more with emiliano please
stick around thank you thanks