Current AI Misalignment and Potential Long Term Implications

hello everyone and welcome to the second
and final day of the inaugural stanford
existential risks conference
as mentioned in our earlier email we
highly recommend prioritizing setting up
meetings with other attendees over the
course of the day
our first session for the day is with
mutalai nconde
on current ai misalignment and potential
long-term implications
i'm habiba islam and i'll be your mc for
this session
matalo will be giving us a short talk
and then there'll be time for audience
questions so for those of you watching
please do start submitting your
questions
on the right side panel of the swap card
platform you can submit and upvote
questions during the session
also please do share your thoughts on
the live discussion board
but without further ado i'm delighted to
introduce matalei
matale unconde is the founding ceo
of ai for the people a non-profit
communications agency
prior to this nakande worked in ai
governance during that time she was part
of the team that introduced the
algorithmic and deep fakes algorithmic
acts as well as the no biometric
barriers to housing act to the u.s house
of representatives
he started my career as a broadcast
journalist and produced documentaries
for the bbc
cnn and abc she now also white
writes widely on race and technology as
well as speaking at conferences around
the world
thank you for joining us thank you so
much habiba
and it's my pleasure to be here so
um i'm gonna go right into my
presentation if i could just have my
slides
and over the next eight minutes just
give you the lay of the land
here so i grew up in the uk
moved to the u.s around 15 years ago and
have been living in new york city
and what you're seeing in front of you
is a robot dog that is being used by the
new york uh police department and we're
being told
that this dog is just being used in the
same way as live dogs would be
to go into difficult scenes and to help
with investigation
however there is um an issue the dog
much like
driverless cars operates using sensors
to uh look at the crime scene and is fed
in its training of what a crime scene is
and what danger means
is fed information from prior crime
scenes so we're thinking about computer
vision
we're thinking about audio for folks
that use clubhouse if you think about
how they're using audio technology
as well as facial recognition and we
know
that crime is a heavily racialized area
in the united states and so for people
like me
that look at the racialized impact of ai
development
this is extremely extremely um
extremely scary for us and for major
reasons so
dogs in the united states have always
been used
against black populations as a
crime-fighting tool
so for them to even use this this um
this form really goes back to an area of
policing that was extremely
um anti-black and then on top of that
the history of policing in the united
states actually comes from
slave catching laws where there were uh
black codes in place that meant
if you were a black person by yourself
without a white person
anywhere in the united states you could
be captured by the then police
and taken into custody which is exactly
what we're seeing today
however post the murder of george floyd
um we have now seen a multi-racial um
coalition coalescing behind this idea
that black lives matter
and this critical evaluation of not just
police but
technological policing so i'm coming to
you
um really to talk more about a report i
worked on
in 2019 that was really designed to
um help uh engineers other people
working in
ml fields to really think about how can
we bring this racial awareness to the
design deployment and governance of this
system and it has three elements
cognitive emotionally and our action
plan
so one of the things that we ask in the
context of this framework
is cognitively ai systems are being
deployed in ways that uphold white
supremacy so just like the robot dog
is much like the dogs used in the civil
rights movement
and being used by forces who within the
antebellum
south and slavery were used to
incriminate black people we're seeing
that but we're now seeing it encoded
into systems
emotionally the research and development
community are not facing this
what they will say is these are
unintended consequences
the doc is just there to to clear a
crime scene
we don't have mission creep whereas we
know
that licensed place readers when they
were introduced in the us
context were for parking tickets and
then
with the trump administration became one
of the ways that people which
funneled into the deportation um
pipeline because that data that was
gathered was then cross-referenced
against immigration records which was
not the intended
purpose but race was never a
conversation
because tech was seen to be neutral
and then lastly um action plans so one
of the things i would say
as somebody who wants racially literate
futures within ai and ml
fields is that those who are developing
these systems
have to look at the race cannon the
african-american cannon
and other and decolonized cannons that
center
the stories and perspectives of not just
black but other marginalized groups to
inform
development and that's the future that
we're trying to create
unfortunately that's not the presence
that we're in this is dr
tim nick gabriel who has a phd from
stamford university and was recently
she would say fired google say another
story
i er on the side of firing for doing
this type of research she's actually one
of the co-authors on the paper that
found
facial recognition does not um recognize
black faces
her partner michelle mitchell who also
was over the ethical ai team
at google was later fired so tim knit
went
in december michelle went i believe at
the end of february beginning of march
but that left really a void for people
who
were really looking at this idea of
racial literacy within technology
and timnit who's a friend said that she
actually used my framework
in a training and got fired from google
so before
you run out and tell people not to be
racist
uh within some of these company settings
know that it could get you fired
so um i did a study in 2019 and found
that google
94 of the google researchers in their
google brain team
were white and um han chinese men there
were other asian
people in that mix but han chinese was
the dominant group
so that makes you think not only is this
hostile environment and it's not just
google it's just google's got the worst
press at the moment
but you've also got this homogenistic um
company setting that doesn't enable
critical computer scientists to have
even the power to have these
conversations
and this really is taking us back to
what i would consider
the south during jim crow where there
were places that white people could go
there were there was agency given to
white people
that was not offered to blacks but like
this man
in the screen we don't care we're going
to defy those and we're going to create
the type
of um the type of industry that we want
to see
because the truth of it is the robot dog
you saw on the first slide
isn't always just going to be used to
scope out crime scenes this is an
example of the same model of dog
but you'll see here that a gun
has been mounted on its back with a
sensor
this was used in an artistic setting
sorry i obviously i'm in the noisiest
place on earth
this was used in an artistic setting and
used to make a point
but given the crisis of uh
gun violence in the united states the
crisis
of the killing of black and brown bodies
at the hands of police we need to ask
ourselves
is the future that we're preparing
ourselves for
for one in which the robot dog will
become armed
and i would argue that not only is that
what we're
preparing ourselves for but if we don't
intervene now
that will that's the present that we're
in so thank you
so much and i look forward to questions
thank you mitali um we'll go dive
straight into questions
now i'm just first to set the context uh
could you tell us a bit about your
organization ai for the people uh what
is it and why did you set the
organization up yes so
ai for the people's communications firm
um i've been
a journalist for the last 20 years
so starting at the bbc in the early
2000s
in the news and documentary unit
and within that role was always telling
stories
about um science race and society
broadly this was prior to my work in
technology that came about 10 years
later when i worked
um opposite google's uh external affairs
department
and the job there was very much like the
job i have now we were taking
um a a company a company that based
its products on science and we were
really
going in and talking to policymakers and
other stakeholders about the benefit
this would be
for new york city but that benefit
never had any critical analysis about
the way these systems that we were that
we were developing would impact
uh negatively racialized groups like
black people and so ai for the people is
really an answer to that in the sense
that
we create media content you know
film other types of media content that
really explain
what the technology does and then we
hopefully will use that so that people
can build
power towards deciding do you want
an armed robot dog um in your
environment which which is incidentally
a black mirror
um story line so is that you it's so
horrifying episode
um but very gripping um uh
so thanks for thanks for explaining what
mai for the people does
um i noticed you've uh recently been
working with amnesty international on my
campaign
can you tell us a bit more about that
particular project yes so
amnesty international are doing this
amazing uh campaign global campaign
called ban the scam
and what they want to do is push to
change
norms around how facial recognition is
used or in their case they don't want to
use it across four cities um
in the globe new york city being one and
so as a communications firm
we came in and co-produced a five-minute
film
that were just case studies there was a
housing case study where facial
recognition was replacing keys
as well as a criminal justice case study
and we used that five minute film
to go and engage community groups and
legislators and organizers
who have who are working on other issues
but they don't have a technological
analysis
and we're so great we're so proud that
it's come out
around the time coded bias is on netflix
because we feel like it's a
it's like a a very nice companion for
people who don't have
two hours to really get into a
documentary but maybe
five minutes to connect with our
characters
great um and just to talk about the
kinds of issues that you're concerned
about
um do you have a sense of what kind of
time scales you're worried about some of
these things coming to fruition in
around the sort of
the facial scans and the sort of robot
dogs things like that
yes so we have um two bodies of work
one's called race technology in the
black body which are biometrics
and all of those things are happening in
real time right the robot dog was used
here in new york city last week it
didn't have a gun on it
but it's part of a police force it could
it could have a gun on it
um very soon and we also
do a body of work that looks at
information integrity so
myths and disinformation specifically
targeted
to black groups and we're in active
conversations with twitter
because we we built a data set and did
some analysis around the u.s election
and i also sit on the con tik tok
content advisory board
so really um making sure that those
algorithms
are not shadow banning people or
um not creating any type of racial harm
these are all
ongoing uh live issues but our narrative
change cycle because we're very
interested in narrative change it's a
five-year
cycle so we try to move slowly and we
try to build community very
intentionally
and do you have a sense of how uh the
work that you're doing connects to
sort of concerns that people might have
over even longer time scales
sort of like how ai might develop over
the coming
multiple decades yes so we're already
seeing
the the beginning of companies like ibm
developing these social media campaigns
around quantum
computing and how safe it is and how
it's gonna
you know do just be great and as a
science communicator and as somebody
that had done very similar work for
another company
i know that that's not the whole story
so we're already even in our own work
beginning to look at quantum beginning
to look at the applications because
the technologies are often agnostic but
it's the way that they're deployed
and the governance or lack thereof that
create real harm for communities so
kind of getting in front of that enables
us to
introduce that to the lexia lexicon
because
many of our communities don't don't even
know about ai
like they they just think it's like
terrible magic
um just switching over to some audience
questions now that we've had through
uh swap card um people are uh
you talked a bit in your in your uh in
your talk about uh what's been happening
at google and
and google's ai ethics team so i'm
curious how you
how optimistic you are about how big
tech companies are going to respond to
this and in particular someone's asked
um how do we safeguard against this kind
of ai misalignment given that most of
the actors developing ai seem hesitant
to acknowledge it's a problem at all
i think we need to regulate them and i
think we need to break them up
we have to remember that these tech
companies are
beholden to their shareholders so firing
timnit firing michelle
vira i mean in another iteration of the
talk which is which would be longer
i actually go through all the people
that have been fired across all the
various departments
that's not going to change but what we
can do is change market conditions
so we need to have strong regulation
that isn't
um written by the companies as well as
social movements that integrate these
concerns and that's where i think
ai for the people really comes in
because we're saying if you're a housing
activist
you should care about biometrics if
you're a police activist a reformer
you know abolitionist you should care
about this if you are somebody that's
looking at fair food systems
and in giving that overall analysis
because our whole world is now mediated
by these systems
we will build the type of power that
gets to
on my very first slide this idea of
white people in in the midwest of
america saying defund the police and
black lives matter like that's
ridiculous that's something i would
never have envisioned but it happened
and it's happening again right now
that's very
very optimistic take on things um i'm
curious if you're
hoping that a lot of this pressure is
going to come via government's acting
uh on this then and putting in place
their regulation
um yeah how do you have a view on how
how your sort of
uh how you perceive the the us
government um
and their position on ai at the moment
and how optimistic are you that they
will actually
respond as is needed
i'm famously optimistic but the level
the level of literacy is really low and
so we have to do
so much education because the other side
are creating the future they're creating
the future they're creating the markets
they're setting terms and so i'm hoping
that you know civil society can come in
and do that there have been some really
optimistic hires
um by the biden administration looking
at these issues not everybody has got
through confirmation
and so um i i see the willingness
but i think it's going to be another
it's going to be a sustained
um sustained pressure from sustained
administrations to make that change
we've had a couple of questions or
people are uh interested in some of the
technical challenges here
um so one question is um
how can these machine learning giants so
for example google deep mind open ai
how can they select the material to
learn or institutionalize
positive um systems that would actually
consider all individuals across races
nationals
nations species and generations so um
so again sorry i don't think they can
because if you're using historic data
then are
we going back to our past and being
anti-racist
abolitionist um you know
gender affirming uh humans
i don't think that they can i think that
we have to ask another question which is
more pressing for me
do we need to have ai systems govern
this particular instance and
another question on top of that is if we
are going to
um if if we're going to be in violation
of human
or civil rights of individuals
should we even be using these systems on
humans
so um there is an experiment in canada
where they're using facial recognition
technology
to um help characterize their
bare wild bear population and i was like
bear recognition great
bears have no human rights knock
yourself out in fact
go into the ocean go to the coral reef
because we are in an economic crisis
right we're in an environmental crisis
right now
and some of these technologies could be
used to classify that's what they do
they put things in categories
should we even be using them as
predictive tools because last time
i looked the great potential of being a
human being is that we have the capacity
to change
but if you create if you put in systems
specifically in the criminal justice
context
where you're predicting whether this
person will be criminalized until the
end of their natural life
i think that's completely unethical we
should be looking for for
we should be looking at reform we
shouldn't even be using
those types of tools within that
environment
we've had one question from an audience
which um is interested to dig into that
that uh that piece a little bit more uh
so for example we have
uh we could we could rely on ai systems
or we could rely on human judgment
here and someone has asked are ai
systems with their own biases picked up
from these historic data sets
actually more dangerous than human
decision makers who also have those
biases
and so this person has pointed out that
one advantage of ai could be that the
biases can be more
easily identified and then fixed more
comprehensively than if it was human
decision makers
well depends who's the person
identifying because
you know almost 100 of the white
computer scientists when we were doing
racial literacy and tech
said that there was no racism so they
wouldn't even look it for it in the
first place they were like
it was like the um the uk race report
where they were like racist to us
no actually you are actual colonizers
you are actual colonizers no
stop it and so um i would argue
that um it really depends the research
question that you asked one
to whether that would even be surfaced
and the great thing about humans is that
they can change
ai systems do not have the capacity to
take on
social context to change to um
to move their position to go from
police affirming to abolition that's not
what they're designed for
ai systems are designed to pick out
patterns based on historic patterns and
make future predictions
um and i just think we should never as
scientists put ourselves in a situation
where
we're predicting the future we should be
um in the we should be in the business
of thinking
that human beings and and the human race
can only get better
if given the chance and if giving new
information and so ai
is is then um is incompetent at that
particular type of reasoning
um as you know this this conference is
around the concept of existential risks
these kind of
these most extreme uh catastrophes that
might befall the sort of the human
species of the planet
um and you might consider sort of
climate change or nuclear war in that
category
and a lot of people consider sort of a
lot of the development of maybe
a very advanced ai to fall in this kind
of category
um someone has asked that they find some
resistance amongst the community
the existential risk community to care
more than the average person uh
to care about sort of race and class
issues um
uh and they're interested to hear you
speak a bit more about
how one might be able to frame social
justice issues in ways
that might help persuade people who are
focused on these these biggest
catastrophic existential risks
so uh two things but those climate
crisis
impacts people and it's not going to
impact people equally the
black people and poor people are going
to be the most impacted and first and
actually are
just go to a councillors day and find
out who has asthma
and then go to the cotswolds and find
out who has asthma so that's
one two um the thing that i would say
and i've been saying this even when i
did policy work the anti-black racism
isn't is a national security threat
and when i first started saying that in
2017
people were like oh my god she's just
always talking about racism
it's totally over and now she's just
exaggerating
and then january 6 2020 happened at the
u.s capitol and the mob
of white supremacists tried to interrupt
the vote
and as i said one group one body of our
work
looks at algorithmic driven
recommendation systems
and what that does to our democracy and
i would i was
able to hand a case study that said in
this case
race-based discourse delivered
through algorithms radicalized enough
people that they went to the u.s capitol
to behead
a sitting vice president this
i said to you anti-black racism is a
national security risk
but what i should also have said is so
is white supremacy
so all of these other systems nuclear
war
um are on almost always about
um advancing the white supremacist
project through the lens of misogyny
as is a climate denial has been very
much about
individualistic rugged individualistic
uh male-centered white centers
narratives of how we should be on this
earth
we shouldn't think about future
generations we shouldn't think about
community
we should only think about ourselves
because we go we conquer
we um colonize and that's unacceptable
and so unless you're bringing in this
race class analysis
and reminding this community that these
threats are going to be
visited upon and these threats are going
to be visited
upon human beings then um
i don't feel that they're really truly
understanding the magnitude of those
issues
yeah and if i can just add to that i
think that point around
um uh not considering the interests of
future generations i think is a really
important piece here because i think
there's a lot of um
for those who come from an existential
risk are in the essential risk community
and are caring about some of these
catastrophic risks
um a lot of there's a lot of um
similarities uh in thinking about uh
speaking up for underrepresented folks
or people who don't
have uh a position of power in the
current uh
socio-political environment and there
are disadvantaged groups nowadays who i
think are in that category and also i
think
people in the future who don't have a
say at all are also in this category of
being
not considered in the in the decision
making um at the highest level at the
moment
and so i think that can also be like a
an interesting point of
um and in for people who are interested
in sort of both things
um i just want to pick up on there have
been a few questions
in the chat about um the identity of
people carrying out this kind of
research
so just one quick point of clarification
the in your slides you mentioned a 94
figure for google brain um did that was
that figure
uh only um white individuals that also
included han chinese
it included han chinese but it was
overwhelmingly white
so even in that data set the
second most um the so there were two
other groups that were in that 94
han chinese and then um indian
uh but but tamil brahman so
it wasn't even just like india oh look
at all these people from india it was
tamil brahman
folks who then had a huge lawsuit
against them um
in california because they weren't
allowing um
delic colleagues to get you i mean it's
a big it's a different zoom
but it was a problem it was a lot of
dominant group folk
let's just say i think um yes i guess
that's a whole other
topic of of discussion and unfortunately
that is all the time that we have for
in today's session but thank you so much
matali for your time today


Daniel Filan - Peering into neural nets for AI safety

hello and welcome back everybody to the
tourist data science podcast today we
are talking to daniel fallen
who is an ai alignment researcher at
berkeley and the host of axerp an ai
limit podcast that you should definitely
check out right after this episode
we're going to talk about dan's research
but also his motivation for getting into
ai safety
his involvement in effective altruism
and
why that's led him to among other things
place bets on the outcomes of
presidential elections
so daniel's research is actually really
fascinating it involves studying the
structures that neural networks form
internally as they learn
to try to understand neural nets better
and also hopefully
to make their behavior safer and more
predictable so we'll explore the
importance of interpretability for
safety
and what may be the deep connection
between safety research and capabilities
research
as well but i think that's enough of a
preamble for this one so without further
ado let me step out of the way and let
the episode begin
all right well dan thanks so much for
joining me for the podcast thanks for
having me on
i'm really excited to have you here um
there's so many interesting topics that
we could discuss
obviously you're doing some really
interesting research under stuart
russell supervision at berkeley
and i think we'll dive into that but i
think what's really cool
about you and a lot of the people in the
space generally is that
you tend to fall into this bucket of
concern
thought and philosophy that is called
effective altruism
and i wanted to start there just to talk
about the the way you see the world
the way you decide what to work on that
sort of thing
maybe we can start with just like your
thoughts on how would you define
effective altruism and how does it
affect the way you think about how you
spend your time and on what
yeah so i guess effective altruism is
this
uh i guess uh philosophy in the kind of
colloquial sense um of like
trying to think carefully um
about how to do the most good with like
the limited resources you have available
to you so um
i i guess implicit in that is a couple
of ideas like
like to think carefully sort of implies
that the answer might not be obvious
um or like well because there's
definitely some world like you could
imagine a world in which like
the important thing is to just like try
and do something good
and like most things you could do that
you would think of is trying to do good
or like approximately as good
or like as good to you know maybe some
are better than others but like you're
way better at some than others so like
um
you should focus on those um so i guess
one
premise of the effect of altruism ideas
just that like
there's some opportunities to do good
that are like way way better than other
opportunities
um the second premise is it's not
obvious which ones those are
um so i guess
and the third idea is one of efficiency
right the natural way to think about
philanthropy or doing good is like oh
let's like think of a problem
and like maybe and you know maybe you
check like okay is this like a really
important problem
and then you kind of stop
there like you find an important problem
work on it um whereas the
effective altruism approach the
effective part is very interested in
this idea of like okay but like
how much change can you make you know
right because you only have so many
dollars in your bank account
you only have so many hours in your
career if like that's what you want to
use your career for so like you sort of
want to
be in some kind of maximizing mindset
with respect to that
so um that's basically i guess the
intention
of effective altruism the extension is
there are there are people working on a
bunch of different things
um so a lot of people who are interested
in um
global poverty reduction or like like
trying to improve um
health outcomes in developing countries
there are a lot of people who are
interested
in reducing basically reducing the
number of animals who live on factory
farms
um there are a bunch of people
who i would count myself among more who
are really interested
in reducing the possibility that
humanity might go extinct or otherwise
like
kind of be locked into a pretty bad
future like much
worse than we could have and then there
are also a bunch of people who are like
we should
think more you know maybe we haven't
come up with the best idea yet
so let's just analyze some stuff yeah
sort of a whole portfolio approach to
to this kind of altruism stuff right and
i mean with that when you say
that when you um hint at the fact that
these problems the problems that are
worth working on are non-obvious
i think one of the things that comes
with that is a certain level of
intellectual rigor
and in particular a focus on
probabilities and analysis you know
bayesian probability theory
applied bayesian probability theory
really um yeah i'd let people try to do
that
right yeah so that's kind of where i
wanted to go with this what do you see
as the ways in which
effective altruism requires that you
pull in
this statistical lens and what are some
of the ways you apply that lens in
deciding how to spend your time
oh that's interesting i guess i kind of
i think maybe one view i have that is
perhaps uncommon
in the machine learning community i know
how uncommon it is
but i basically think that like
machine learning like like figuring out
how what good ways
are for ai systems to reason
i think if that doesn't change how you
think
you should live your life or something
then either you're wrong or the theory
isn't good enough
so in in terms of like how i think about
the importance of using patient
probability for effective altruism
well i kind of think of it as important
for just like having good beliefs
in general um but i i guess the specific
case
yeah so it's specific to effective
altruism right um there are a bunch of
things where like you don't
like suppose you want to know um
suppose you're interested in this
problem of like okay it turns out that a
bunch of people in developed worlds
um have uh worms um
worm infections um that's a thing that
can happen unfortunately
um and like it kind of sucks
you know it means that you can't go to
school as much as my understanding um
it's definitely unpleasant and so the
question is like
okay one thing you could potentially do
is you give people deworming pills
and like you know um and though i i
believe the way you do this is like um
you go to a school
um you give the school a bunch of
d-wearing pills you like
tell them to hand them all out on a
certain day like like maybe you help
with infrastructure of that
um and then you know a bunch of people
get dewormed
and the question is okay what what
happens next right
like how firstly like does that make
people
um less you know
are they less infected with worms um and
you know are there any like weird side
effects or something
um secondly like
but secondly getting to the basin point
you don't you don't just want to like
reject the null hypothesis or not right
you hear about the magnitude because if
it costs like ten dollars
to buy a pill and the pill makes
and you know eat every ten dollars you
spend like
there's a one percent chance of one
person feeling like
one percent better that might not be
worth it um but you know if the pills
are really cheap
and the effect size is bigger then like
that's a bigger deal so
you want to care about like okay what is
the actual effect size you want to care
about
okay what is the like
you don't just care about people not
having worms right like like the reason
you care about them not having worms is
like
maybe that'll help their education or
maybe that'll help them live longer
so you want to check okay like are
people going to school more
is that like helping their lives um
are they having a living longer effect
yeah is it having the
intended effects um you know to what
degree um
and then you've got to say okay we ran
this one study
in one state of india can we now do this
in some other place will that have the
same effects
and the answer is like well you've just
got to like think about
like the answer is not going to be
definitely yes or definitely no
um it's inherently kind of a
probabilistic question yeah um so
yeah i i would say i i think it
especially
is true for things more like the global
poverty side where like
um you really can like some people do
run
rcts um sorry
i'm very interested uh randomized
controlled trials okay so like yeah some
trial where you give some people a
deworming pill and you give some people
a sugar pill and you see like okay
you know is there a difference in what
happens um and you randomly pick which
people give which pill
um so i'm really interested in reducing
the chance
that uh humanity goes extinct or is like
locked into a really terrible feature
uh for that one it's a little bit less
statistical because we don't have many
experiments of that one
there's some like conceptual
basin thinking i guess where you have
this idea of like oh no i thought
like you should kind of think in
probabilities and you like
should try to like increase your
credence in theories that like predicted
with a high probability what you
actually see um
but you know it's it's hard to make
those predictions
in the kind of landscape and with the
limited information we have right now
uh for that sort of thing right i mean
this is i guess as you say this is like
one of those big differences between the
existential forward-looking risk that we
face and the the current
deworming poverty reduction stuff where
we actually have data
i'm curious about like that aspect there
are a lot of arguments that
you know make the claim that either we
aren't going to reach
the levels of technological capabilities
that would be dangerous
i'm talking here about ai specifically
because i think that's the context that
we're both thinking of it from but
there are a lot of claims that either
you know we won't reach that level or if
we do
somehow safety is going to automatically
come with capabilities so
you just won't be able to build a super
powerful ai system
that isn't already equipped with like
what some people call common sense
or some notion of ethics morality that
wouldn't be super moral in addition to
being super intelligent
and in particular i think that was the
argument that melanie mitchell made on
the podcast a couple of episodes ago
she of course debated stuart russell
who's your your um your supervisor so
i'm sure you don't have like a
telepathic link to stewart but
to the extent that you're able to i
guess summon your mental model of him
and maybe
parts of your own as well as as you
think these things through i'm curious
like how would you interact with those
arguments that say
either you know we're not going to reach
that level or if we do it's going to be
done in a safe way almost intrinsically
yeah so i mean in terms of we're not
going to reach that level we sort of
know that there is one physical system
that can um achieve human levels
of uh intelligence at least as long
where you define intelligence is the
thing that like
makes changes to the world um that's
humans right
like i guess i don't see why
but like the impossibility claim i it's
difficult for me to see how
that could really hold up um
a different claim you could make is like
okay well maybe we'll
maybe it's possible but like it might
just be like a thousand years from now
right um it might just be incredibly
incredibly difficult
and basically my answer is like well
i think if you just if you think about
okay what can machine learning do now
and like how
how much would we have to train like how
big would things have to be
and how much would we have to train them
in order to get
to the human level even if you have like
pretty pessimistic bounds on that like
if you think like oh we have to like
redo all of the like neuronal
computation that evolution did
for like like since i don't know the
jelly
fish first existed or something like
more computation is becoming cheaper and
cheaper year by year
like it it really seems like
like as far as i can tell when people
like try carefully to forecast this
it's probably gonna happen in the next
two centuries like like i find it
difficult
i don't know maybe it's possible that
i'm missing some like really clever
argument against
but like yeah as far as i can tell i
think we have pretty good reason to
believe that like
sometime that there's like well a decent
probability to
go back to their previous point um that
at some point within the next
100 years i think we could probably do
it
um in terms of whether like a really
smart thing would be super moral
um yeah i guess i kind of
buy the hum
like it goes by two names one is the
human theory of motivation and one is
the orthogonality thesis
so basically what they say is like the
things that um
i might be butchering the human theory
but i might be getting it approximately
right
um what they say is that like look you
can like
be smart and have like a wide range of
goals
that you want to pursue so like one
well i think one reason you might think
that um really smart things would be
super moral
is that they would like figure out like
the moral facts
right like they'd be able to like reason
about what humans want
and like what humans ought to want and
stuff and they'd be able to like
come to true conclusions about these but
it's not obvious that like
those true conclusions would be
motivating
right like motivating in what sense
like um suppose i come
i suppose well some people doubt this
but like let's let's suppose that there
is just a matter of fact about what's
right and what's wrong
um and let's suppose further
that i like really that i think about it
a lot
and i come to the conclusion that like
actually eating
eating ice pops like these these icy
snacks they're like sugar water and
tubes that you freeze um
that eating these is just like
inherently morally wrong right
well then it's not obvious that that
would make me not want to eat them
because the reason that i want to eat
them is that it's hot
and they would call me down and and you
know i i have all these reasons for
wanting to eat them
and like the fact that it's morally
wrong like like this
like it seems like i could just not
particularly want to do things that are
morally right
at least not want to do it enough to you
know give up his chance to get this
delicious
um icy pop now now one
one i think one way that could be false
and one way that it is false for humans
is that what's morally right is like
really closely linked to like
kind of achieving a wide range of goals
so like i think like
for instance why should people be honest
to each other i think that just
basically makes society works better
work better i think like you know
mostly your life is just going to be
better if you're honest to people
right so like if you're much more
powerful than all humans
like maybe you don't have to play by
their rules you know so that that's
basically my response to those two
ideas i i definitely i definitely agree
with that frame i think
the tough thing is i have the same
biases as you so i i do want to
remind the audience you know if you want
to see that other side of this argument
definitely check out that melanie
mitchell podcast or the one with
ornetzione
that we just put out um so i am going to
agree with dan here and you're going to
get a little bit of propaganda but
that's okay
uh i i definitely align with with where
you're coming from here
and also at the just the level like
human beings
we look at some of the the most
disastrous regimes in human history and
they were run by
objectively very intelligent people
hitler was
brilliant stalin was brilliant paul pot
was bro these are charismatic
intelligent people
but on any iq scale by any measure i'm
sure they're going to be in the top
whatever 20th percentile maybe even 10th
or more
and yet you have these wildly divergent
uh moral
moral outlooks you can debate whether
they were sincere about particular kind
of moral framings
but certainly some of them were there
have been highly intelligent sincere
people who've disagreed
profoundly about what's morally right so
like i i guess
is that that's the sort of orthogonality
in the orthogonality thesis right you
can decouple intelligence or
capabilities like from yeah yeah i guess
you might think that like look
you know stalin was really bad but like
he wasn't he maybe he wasn't smart
enough to really you know
understand the true morality so i i
don't think that like
that's a totally knocked down argument
but uh perhaps it's illustrative
yeah well i guess yeah my intuition here
is something like
even if um so
there may be a lot of uh optimal points
for society
local optima that are approximately as
good and it's really hard to tease apart
from the outside like which one is
better
and they might all have really really
different moral norms
like in some in some cultures arranged
marriage might be
you know the the default norm and that
it might be possible for those cultures
to attain
an average level of happiness or a
median level of happiness
that's just as high as any other and yet
they look completely different sort of
from the inside view
so it's um i guess this is where my
intuition kicks in at least on the human
level and obviously these things have
limited domains of applicability
when we're talking about super
intelligence but i don't know that's
kind of where my mind goes
do you think these these human analogies
are like are dangerous though like is is
there a risk when we start thinking
these terms like i just did
that we start to over generalize based
on human failings or human
thinking yeah i mean
i guess i guess you just have to take
humans as the evidence that they are
which is an existence example and like
you know people like people have about
this range of capabilities and they can
do
that they have you know
this thing happened with humans and it's
really true that that thing happened
with humans
and if we expect ai to be like really
similar to humans in the relevant sense
and that's strong evidence
if we don't then it's weaker evidence
unless we like unless we derive
a like really powerful theory from
humans that we are like pretty confident
that that theory holds in general um
yeah i don't know i think like you can
i think some people like i think it's
easy both
to draw too much on the lesson of humans
and to like not draw enough on the
lesson from humans
yeah i don't have a super strong opinion
about like which way people
are more it yeah it's frustrating when
when the right mature answer is is
moderate and that happens so often i
want an extreme
yeah uh well okay before we get into the
the details of your your research and
some of your your work in
ai safety i do want to ask one more
question about this broader kind of
ea space and some of the the ways of
thinking in effective altruism
and this has to do with betting so every
effective altruist i talk to loves to
talk in terms of probabilities and i
love that i think it's super
constructive
but there's a specific kind of approach
to leveraging probability theory
that is this kind of skin in the game um
betting approach that i know you've
you've
kind of gotten involved in so where does
betting fall into effective altruism how
has it helped you think more clearly
about the world
yeah i think of betting as a
habit of mind that's like generally
useful
when you're not just for effective
altruism but when you want to
have true beliefs and in particular if
you want to have like a community of
people you have true beliefs
so basically when i say betting i don't
mean like uh
betting on horses necessarily or sports
i mean like um suppose somebody says
something like oh i think like
um i kind of don't want to use a
political example but most
mostly people like say very
overconfident things about politics so
let's say oh yeah
the democrats are sure to win in the
midterms i'm like i'm like
you know they've got this uh like i
think people can often be very
overconfident about that
and i think when you ask somebody okay
do you want to take a bet at like three
to one odds or you pay me 30
if the democrats don't win and i give
you and i pay you 10 if the democrats
you
do win i think there are two effects of
that
firstly if i just said the democrats
will definitely win i might not have
been thinking very carefully
but when when a bit of money is on the
line i'm like oh is it is it like
three to one well maybe it is maybe
maybe it's not right
so so firstly like it just causes you to
think
a little bit more um
and the flip side of that is it's sort
of uh tax on like
 basically yeah like i think i
think one problem with the world is like
i don't know a lot of people say a bunch
of stuff and it's not obvious like
when they're like you know when they're
just bloviating or when they're
when they're just like saying what's
popular or like
versus when they're like really trying
to get the truth and like
you know the the more people bet against
bloviated bloviated claims that like
have nothing to do with reality
i don't know the poorer those people get
and like maybe the fewer
the less blaviation you have um i think
another advantage of it
is like when you have like uh
this especially when you have a liquid
betting market when like um
you know it's not just like that one guy
you bet against about the midterms but
when a ton of people
um bet against each other about the
midterms then like
you start getting a sense of like okay
what's like the sort of
market probability of this thing and
there's reason to think that like if a
bunch of people are betting
like that probability's going to
bring out like a bunch of information
that's available
to bear on that probability because like
individual traders will like
you know um trade based on what they
know and that should come to some
equilibrium
so it can be informed moving that way
which i think is good
oh the other benefit and similar to that
i think betting has this
opportunity to like give you a sense
that you might be wrong
about something so there's one person in
the ea space
who was pretty there was recently a
story about like
there potentially being life on mars and
there was one person kind of in the a
space he was like oh yeah i think there
is
uh i'm just i'm going into that this is
yeah
i wasn't sure that i wanted to name your
shaymin um but it is robin henson
um and yeah somebody was like
i'll take you up and then robin said yep
sure and then like a bunch more people
were like oh yeah i'm willing to bet
like big amounts of money and then he
was like hmm
this is one of the signs that you're
wrong uh so
so the the kind of market forecast can
be useful not only for a disinterested
observer but if everyone's willing to be
against you then maybe um
maybe that gives you a sense that like
you might be the one who's wrong well to
robin's credit and i think to the credit
of anybody who engages in this kind of
activity
it really is like i mean it's the only
way to make contact with the unforgiving
it you get that jolt
of you know when you're signing the
metaphorical check
and actually saying like you know you're
almost developing that muscle memory of
writing i was wrong
and then handing it over to somebody
else as penance and and
i imagine that do you find that that
serves like a useful role
in actually causing you to follow
through and update your mental model of
something rather than just going like
well you know what i was wrong and it's
you know it's an odds game maybe i was
wrong by chance which you could have
been
but like there's something about that
activity of kind of conceding in
not in writing but in dollar form that
makes it a little bit more uh point
i think sometimes it does and sometimes
it doesn't so i've definitely made bets
that i've lost where i thought like well
it was still the right bet to take yeah
um so for instance
um i think it's a matter of public
record that i bet against this guy brian
kaplan
about what vote percentage the
libertarian candidate would get
in 2016. um and
the thing about betting against brian
kaplan is that so he's a guy who's made
tons of public bets
and he's won i think literally all of
them are all of them that have resolved
there's one about climate change coming
up that i'm pretty sure he's going to
lose
but uh so i did the foolish thing
betting against brian kaplan
and i lost but apparently i didn't learn
my lesson because i'm still like ah
you know i think i was betting on based
on a pretty good forecast and
yeah but but there are some bets where
i'm like oh okay that's source of
information wasn't as good as i thought
um
well do you find it motivates you to go
to the other person and say okay what
was the mental model that you were using
uh and can i learn something from that
one you know now that we have the
evidence that it seems
probabilistically to have been more
right than wrong um
i think people love talking about their
models or at least people i know so
normally like i'll have heard it
beforehand
and like the only the update will be
like oh i guess that was actually
reliable and whatever reasons i had for
things that it wasn't
weren't didn't hold up also i should say
like a lot of the bets
i make um like like the majority of
money that i bet
um has been on predicted so
that's this u.s platform for it's mostly
about predicting us politics
um and it's nice in that like you can
bet more and predictive than you can
with your friends
but unfortunately like you can't like
find the people who bet against you and
like ask them what they were thinking
you can go to the comments section but
like that is extremely low quality on
that website
it's all ultimately i guess about or at
least in part about skin in the game
that seems to be a big part of this
and maybe that's a good segue into
something we all have skin in the game
for which is this question of ai safety
or at least in principle
depending on how this bet comes out
we're all we're all going to be in the
same boat we're all going to have some
some stake in it
so you're working at stewart russell's
lab you did i think an undergrad
studying rl
and rl and deep learning before is that
did i get that right no
um i did my undergrad so actually during
my undergrad i mostly studied math and
physics
i didn't yeah i did an honors year that
was um studying like
this like very i know this kind of
obstructs theoretical aspects
of um you know supervised and reinforced
learning
um but it is not meaningfully yeah i did
not really meaningfully study deep
learning
or um very much uh practical rl in my
undergraduate
so it was a real pivot then once you got
to stuart's lab and you're like alright
let's do this the safety thing that was
a big change um
yeah well it was a big change in what i
was doing um sort of
starting in the middle of undergrad i
kind of knew that i wanted to make that
change
um and sort of it like took me a while
to work up to it you know
was that because you like i guess you
read the super intelligence or or
some eleazar yukowski stuff that got you
thinking about ai risk or
yeah super intelligence had not been
published at that point um
so it was um yeah some eliezer edokowski
stuff um so he's uh
um i want to say blogger
but also ai researcher um who
wrote um this thing called the sequences
which i highly recommend to people by
the way
they're mostly not about ai alignment um
if you find people who talk about ai
alignment really annoying you might also
find the sequences really annoying
for not because they're about ai
alignment but because
there are certain habits of speech that
some people find annoying but anyway
uh yeah it was based on like reading
yeah reading some of that stuff on this
website called less wrong um also like
talking to a friend about it um yeah
and so yeah i got this idea that like
yeah maybe this was like a real big
technical problem
but also one that like you could pin
like you could make some headway on
solving and i was like huh
when i started my undergrad i kind of
wanted to be a theoretical physicist
um and i was like well both of these
paths are kind of maffy
and one might save the world and one is
really really unlikely to
which is which by the way was not the
whole like like if i think about it from
an effective altruist standpoint i'm
like ah
maybe i should have thought more about
it but i don't know ultimately i think
it was the right goal
yeah i mean it's i often think of um
i think there's a reason that physicists
and computer scientists slash machine
learning people tend to
have similar lenses on the world and
it's not just the math i think there's
something
roughly equally fundamental about both
disciplines like physics to me has
always been
a kind of an outside-in pursuit of truth
where you look at the world around you
and you kind of go
okay let's try to explain how this is
related to that and how that's related
to that and then you kind of manually
construct
this giant ontology this set of linked
concepts
that explains hopefully as much as you
can and then the machine learning side
seems almost like
inside out where what you're doing is
saying okay how does cognition work how
does intelligence work
and then how can we leverage that to do
everything else for us so sort of like
similar but
i guess complementary approaches yeah
definitely both of them
they're both kind of hubristic right
right yeah they both have this conceit
that like oh
we can like come up with like these
models that like really explain reality
and i think in both cases or or like
maybe they're unexplained
like in the computer science case does
it explain reality like not quite but it
like
makes good types of reality but you can
come up with these like
useful models that you can really rely
on
yeah as opposed to like oh maybe if you
i'm just speculating here but maybe if
you study history you get this idea of
like ah there are a bunch of people who
thought they understood
what was going on and they didn't and
like
really you should just like kind of see
what used to happen and that's probably
gonna happen again
um yeah i think this is related to
i i think this is probably responsible
for some of the great things about those
fields and also you know how there's
this stereotype that like when
physicists get old they start like
saying ridiculous stuff about fields
that they aren't in like uh i remember
seeing this paper about using the easing
model to predict
um covered uh which physicists will know
that this is kind of silly but maybe
some of them will be really tempted
but um yeah yeah i think it's a strength
and weakness
i i think this is a really good point so
just like internally at the company that
um
that i work at so we're all a bunch of
physicists
and very early it became obvious that we
looked at the world through this lens
a lens that by the way i mean you said
hubris absolutely we actually refer to
this internally as cocky physicist
syndrome like
you look at you'll have a tendency to
like look at an entire field and like
reduce it to
a single differential equation you'll be
like why does this field need to exist
it's just a second order differential
equation in this and uh
anyway that that temptation is really
like i think something that
over time hopefully you learn to walk
back especially actually when it
you know when it comes to things like
safety ai safety i find that to be one
of those areas where
um caution and and just like care
and sincerity as well with curiosity
just are at an absolute premium and you
tend to see that with safety researchers
like the hedging
and the authentic hedging in many cases
against risk
um is that something that you've seen as
well and like how do you see that
translated
in like the let's say the research
techniques that you've explored and the
ones that you think are most interesting
yeah i think they're definitely more
hedgy partially just because it's not
like
like some fields have a formalism that
they can run with or have a paradigm
they can run with and like i guess some
people think this is true
of like ai alignment um but i don't
and i don't think most people do can you
explain what you mean by paradigm by the
way just
i'm thinking for like non-physicists or
people who aren't used to working in a
field with a paradigm
because i think that's a really cool
point yeah so i don't know a paradigm
might be like
your job is to build a house and like
you've got to figure it out
if i were in that situation i would not
really know what to do
i'd be like what is okay what is a how i
guess i guess
it should provide shelter for me and it
should stand up but how to
well like you're kind of confused about
the problem whereas there's one
situation where you can be in where
you're like ah
i know you know neoclassical
architecture or something i guess you
don't want that to be your house but
let's say it's neoclassical architecture
i know new classical architecture
i know i want columns i know that like
the job of a house is to do these five
things
and the way you do it is you solve this
problem this problem and this problem
and then you're done then you've got a
house
that's kind of the idea of a paradigm um
for instance like in physics
like i don't know if you're if you're in
like year 1000 dc
you're like oh how does how does how
does everything work you're like
maybe it's all made of water
but like like it's not obvious like what
what even are the questions you're
supposed to answer
yeah whereas now if you're a physicist
you're like okay here are the subfields
of physics
your job is to come up with a really
good differential equation yeah
it's probably going to be second order
um so that that's kind of the difference
between
hopefully that wasn't really an explicit
definition but hopefully that gives you
a sense of the difference between having
a paradigm and not
yeah like it feels it feels
aesthetically right i mean to me
the vibe i was getting from what you
were saying was you know if you look at
something like algebra
like everybody knows how to solve for x
or at least in most most uh equations if
you don't combine exponentials with
linear terms and stuff like that but
anyway
so like there's a we all kind of know
you know when we approach that like all
right i've got some manipulations i know
what the feel is of this
x might be really really hard to solve
for and like that
it might be super super hard but at
least i know the paradigm i know the
problem setting the sandbox that i'm in
whereas as you say if you start in like
you know 5000 bc or whatever
there's no no notion that x is something
worth solving for or you might want to
find x but have no idea how it
interacts with different things or even
the paradigm of solving these equations
so your view is alignment is more like
that eh
yeah i think we don't really know what
the
like i think there are like promising
avenues
to go down but i think like a lot of
them might be dead ends
i think we're not certain like there's
this question like
i suppose you want to not there die of
ai right
the way you need to do that like in
order to be like really confident that
you're not going to die from ai
at least from a sort of uh i don't know
maybe from a physicist mentality or from
this like inside construct model's
mentality is like okay
what are all the ways in which ai could
kill me all right now i need to make
sure
that none of those happen right and like
how do you know that you thought of all
that ways that ai could kill you
well one way you could do it is you
solve five problems
and then you create super intelligent ai
and then it doesn't kill you and you're
like oh okay i guess that was fine
that's like a little bit risky yeah um
like another way you could do it is like
okay here's my
here's my computer program for agi like
it's written in python it has all these
like really nice variable names
it's got the like logic subroutine and
it's got the goals subroutine
and i can reason about that really
carefully unfortunately that's not
really the state the machine learning is
in
so yeah i think like everything is a
little bit
i i think if you really yeah by by the
way i think if you think that like
machine learning is not the way that agi
is going to turn out
and you're like oh i know how we're
going to build ai it's going to be
logic databases with prolific
programming and such and such
then then maybe you have a paradigm or
maybe you're in a position to get a
paradigm
but um if you don't know
or if you think the answer is machine
learning but machine learning is
confusing
um then it's it's just a lot harder this
seems related to me to
the um because i've seen a lot of people
argue that interpretability and clarity
and
like our ability to understand what
these systems are doing is really
um inextricably linked to safety like
chrysola for example at open ai i mean
their
their safety team and their clarity team
go arm and arm this idea of this this um
one two punch between clarity and safety
is really interesting is that sort of
what you think this points to as well
like that there's a dual um
a dance between those two that's
important yeah i mean i think
yeah so i basically work on um
or i i try to have my research go
towards
um helping make ai more transparent
and easy to reason about so i kind of
would say this but i do see them as
being
at least logic logically inextricably
linked like logically very tightly
linked
um that being said most of the work um
that's done on interpretability is not
really done by the people who are like
really worried about
existential risk um so they're
they're i think they're less
sociologically linked than they are
um kind of uh
linked in terms of what things you need
to get done
you brought up your research and i think
one of the there's so much cool stuff
that you're doing and i was going
through
the papers you've worked on and well
yeah no problem thank you for for
actually writing them or for doing the
research
but one of the things i really wanted to
talk about was this um this cluster
ability
um paper that you put together so can
you explain like
what is cluster ability like what's the
problem setting then what is cluster
ability and how can it help us
understand what ais are up to or how
they're built
yeah so so here's the problem setting
imagine you have a neural network
right you've trained a neural network
and you want to know what it's going to
do
in reality right like
currently the main way you like get the
main way people try to attack this
question
is they look at okay what was the i
don't know loss or the accuracy
on a training set or or you know on a
test set that um i didn't train on but i
like carefully curated and like i think
that should give you some pause because
the real world
is like not necessarily going to be like
that test set
right you know maybe you didn't collect
all the data you could have maybe the
real world will change
once the ai's once your big fancy neural
network is there in ways that are hard
to predict like people might be trying
to game it
or you know it might be like shaping the
world in interesting ways
um maybe you know entropy is always
increasing
you know um large uh you know
large uh composite numbers are getting
factored and like
they're not factored today but they will
be tomorrow so the future just does look
different from the past
which might give you pause the idea of
like extrapolating performance
from kind of past trends so
what what can you do about that
basically
i think that what you probably want to
do is look at your neural network
and be like okay how does this work what
is this doing
like can i can i understand the
operation
of this thing in order to get some
guarantees
about it or if not guarantees per se
then maybe like high confidence
you know reasons to believe it will do
good things maybe it's not you maybe
it's a computer program
like okay so my way of thinking is like
how do you understand things
i think mostly you understand them by
like taking them apart into bits
and like by saying okay like
like here here's the world you know i
live in this world
it's like got a bunch of things but like
there are a bunch of things that i can
think about
like separately at least for a short
period of time
so like um you know when i think about
like
the theory of electromagnetism and i
think about like
earthquakes in san francisco they
basically like don't interact
which is good because if everything
interacted it would be like really hard
to like
well like like when things are on their
own they're sort of easier to think
about
so basically because of that i thought
yeah i got interested in this idea of
okay what if you just
your neural network you have to
understand it it would be easier to
understand
if you sort of break it up into bits you
had smaller bits and they had fewer
things interacting with each bit
and um maybe somehow you could have a
better job
at figuring out what's going on in those
can i try to butcher this by the way
just to make sure that that i roughly
have the right idea
so uh just by by intuition comparing to
the human brain and there are going to
be a whole bunch of pro like issues with
comparison but
something like can we can we understand
the function of a brain by
uh breaking it down into like sub brains
if you will parts of the brain
that are responsible for like some kind
of some concepts or some some
computations is that is that
fair yeah that's kind of the idea so you
might think like oh
you know neuroscientists have discovered
this like language trusting area
or this um you know this uh visual
vision processing area of the brain and
you know maybe that that makes it easier
to understand
the brain's computation unfortunately
that's like really not good enough like
imagine
you like have someone's brain and like
you wanna
you wanna figure out um are they ever
going to commit mass murder right
like knowing that there's a language
area is not obviously
like it's probably step one of like 50.
yeah but it's not like you know it
doesn't really do the job
um so it's it's actually i actually
consider that a bit of a like
like not the most promising comparison
of the world but that's roughly the idea
and in particular like the thing that
i'm interested in
is like finding this out just from the
weights of a neural network
so one thing you could do is like okay
i'm going to give my neural network a
bunch of tasks that i think have to do
with language
and see what neurons light up that's not
what i'm doing because i'm sort of
vaguely distrustful of anybody's ability
to do this what i'm interested in is
okay
can we just like see if the neural
network has like regions
that are not very connected to other
regions in the network
like where the weights are really small
and where therefore you might have some
reasons to think like oh they can't be
influencing each other
much um so in the paper cluster ability
in neural networks
i basically quantify this for various
neural networks that you could train
um so i think it's kind of cool we do
things from the like
from the level of a little mlp on mnist
to inception v3
um and we kind of say okay in what ways
can you carve this up
and in what ways is this like better
than
random but like like if they're just a
random collection of whites
like as long as they're different you
could you can you know try and cut the
like low connection weights
when you're dividing the network into
pieces um so are we like more
clusterable than random
and the basic result is that like you
are more clusterable than random
but you know you're not so clusterable
that like uh
it's you have you have this like you
know
great way of um you know incredibly like
neatly carving the network into bits
is there a metric that uh you use to
kind of quantify cluster ability in
in one number yes uh we use this thing
called the normalized cut metric
um that was actually developed by
another person at berkeley professor uh
chandra malik um back in 2000's but um
yeah what we do is we say okay take one
way of dividing the network into bits
right now for that way look at
um all the weight that is kind of
crossing the boundary
between like that that group of neurons
and like the outside
you know any other neuron yeah take the
ratio of that
to basically to essentially all of the
weight that's like
inside that you know weight weight of uh
connections from one thing
in the group to another thing of the
group do and do you have a name sorry
for those
those densely connected kind of clusters
yeah i call them clusters um okay when i
well
really you should only call them
clusters when they are in fact densely
connected i guess
uh partition elements is my kind of
clunky name for
describing them when they might not be
clusters so you take this fraction
of like the weight that is kind of
crossing your boundaries to the weight
that is not crossing your boundaries
that's not exactly it but it's
approximately it and then you say okay
take the average of that over all of the
um partition elements you have
and then okay how can you draw the
boundaries that minimizes this number
so what's like best way of cutting this
network into bits by this metric
okay um that's that's what
our um that's how like divisible
the network is and do you find that
these structures
are that they reflect specific concepts
i mean i guess it would be too early to
to tell probably based on the research
you've done but are there hints of that
yeah so we've done a bit of so this is
actually like what i'm trying to
get into research whites this year so
hopefully uh in the future i'll have
better answers to this question
um i can tell you what's not the case
it's not the case
that they're like like suppose you're
doing mnist classification
um i think we've basically ruled out the
idea that like there's one group of
neurons that's responsible for the
number five
that that doesn't seem to be what's
going on
we are getting some promising results
for the idea that like oh
maybe like there are some clusters some
groups of neurons
that we're finding that are responsible
for some kind of pattern
um but it seems to be like a little bit
lower level um but yeah i
i am looking to answer this question
this year um but unfortunately
so far i don't have a great answer no i
mean that makes perfect sense just the
fact of having discovered that
that sort of a greater than random
structure
is itself interesting i'm curious about
um
how this relates as well to like our
assumptions
about the way reality is because so when
we um put together a convolutional net
right we kind of
immediately make a series actually we
build these clusters ourselves right we
say
okay we're going to assume that there
are some neurons that have zero weights
like that's how you define this kind of
window that scans across your object
so you have this window that's densely
connected stuff and then zero is
everywhere else
i guess you would you were working with
uh fully connected
uh no you mentioned i think as well
vision right so were you doing some
convolutional nets too
uh yeah we we used both fully connected
networks and convolutional networks
um and was there was there a difference
there did you find
one would tend to like form have a
better ratio of the structure
you just described that is a good
question um
i don't i actually don't know um so it's
a little bit confounded because like um
suppose you have a network that is
really deep right
well then one way you can kind of cut it
into bits is just like get the first
quarter
of the layers from the second quarter of
the layers and then the third quarter of
the layers and then the fourth
and like as long as that works really
deep like you're not you're not cutting
that much by just taking the first four
layers
or the first quarter of the layers if
they're a bunch of layers and you're
only cutting like one
so and and like the this the
convolutional networks we were looking
at
were a bit yeah yeah we we didn't focus
on doing this comparison
uh but we did we did find some kind of
interesting things in
convolutional networks but it's also the
case that we weren't like um
so this thing where um you have like
these different convolutional filters um
and like it's not kind of fully
connected
we were kind of not
the way we analyzed those networks was a
way in which like
that kind of modularity or that kind of
um divisibility
like like didn't really count for us
that makes sense yeah
i'm sort of curious with about what this
does to
this kind of fuzzy boundary between the
assumptions that we
wire into models so when we look at
computer vision
we're really we're adding we're hard
coding this assumption about
what it means to see stuff and the fact
that you know a convolutional filter
can be the same filter and it scans the
whole image and the same filter is going
to be
equally useful across all parts of the
image and that's like this assumption
we force the model to reflect and you
know when we when we have a dense net
we don't do that so the network is
forced to kind of if it's ever going to
learn that it's going to have to learn
it from scratch
and so there's a sense in which when we
use a convolutional net
what we're really doing is giving our
model a big leg up we're saying we know
these things to be true or we think
these things to be true
about physics so like let's give you a
leg up kind of teach you this from
scratch and then you can do the rest
based on that
and i guess i'm curious about like um
naively i'd expect the more of it of a
leg up you give a model
the more interesting structures it can
then focus on developing
maybe or that's like the vague intuition
so i guess i might guess that
convolutional nets
maybe would have a more interesting i
don't know this is
obviously a clueless guy throwing
something out there but yeah i mean
so one confounder there is the
convolutional networks are just better
at classifying then so yeah maybe one
question you could have is like okay
suppose you have a convolutional network
and a fully connected network
that have the same degree of accuracy
can do we see a difference
in how um interesting
their um you know the the structures you
can find in the bar
and the answer is i don't know and i
probably
won't find out there are lots of things
to find out um
but uh i do think that's an interesting
question
going to your border theme i do think
it's interesting this degree to which
like
um you kind of um
yeah we somehow like build certain
assumptions into the model
or like certain ways of thinking um into
our machine learning models
and like on one on the one hand
i don't know we have this blog post
bitter lesson or there's this idea
floating around
i that is not exactly the idea in the
bitter lesson but there's this idea
floating around they're like look the
way we the way you're going to get smart
things
is just have them learn all the relevant
structure on their own
yeah and convolutional neural networks
kind of spin at the face of that right
well like one of the most interesting
things we're building in like why
shouldn't the model be able to learn
that
um but i don't know i do
on the one hand i do think it's totally
sensible to like
like if you really believe in structure
like
you know build it in there's no reason
to like be this kind of super masculine
uh you know if you just
yeah if you just want to do something
like you know build the system which
does the best
um on the other hand i do think that you
do want some flexibility so
so one thing i'm particularly interested
in in the context of cluster ability
is like so networks that you train
normally they're a little bit
clusterable but they're not as
clusterable as you'd really like if you
wanted to say like
to make like really strong claims about
how the network worked so one thing i'm
interested in doing
is in like regularizing for
this kind of structure oh what you're
saying um and like
in particular i think this is kind of an
interesting approach because like
you have this uh there are definitely
people who design
modular networks so like um
i think there's this paper where they uh
designed this network to play starcraft
and they have like
one unit that does one thing and one
unit that's what you know one network
that's like designed to do one thing one
network that's designed to another one
network that's like designed to like
gather all the information from these
sub networks and like process it
um so
but but that has the pitfall that like
on the one hand if you know what the
right modular structure is
then you can just do that on the other
hand if you don't then i think there's
something nice about like having the
network
learn it also if you regularize for
cluster ability
then you might get a lot of it rather
than like a you know
statistically significant but uh maybe
not practically significant amount um
so yeah i think this idea of like uh
yeah how how much do you want to push
this like like if you want networks to
be clusterable how much do you like
want to push it on them it's kind of an
interesting idea that has come up in my
research
and it seems to kind of speak to this
question of compute usage right i mean
in a scenario and i think there's it's
no coincidence that like
we've been able to make progress on
computer vision in an era with
relatively limited
compute power available because we made
all these assumptions
it's kind of a mirror image of um or a
continuation really of what we saw in
classical data science where we went
through this era where
humans had to do all the massaging all
the feature engineering and then like
have these pristine features that you
feed to a logistic
regression or an svm and then like oh
good you know
it's doing that last little bit that
last mile and now we're sort of
gradually offloading
in an almost continuous way it seems our
cognition two machines
by doing less and less of the kind of
prior fixing and tweaking
and moving towards i guess more
generalizable architectures like
transformers and so on
but um what i find fascinating about
this is like you're sort of playing with
that interface
where you're getting the you know by
regularizing or by forcing these models
to
to take on more structure you're kind of
allowing us to
peek in without having to do that
tweaking
it almost seems like that would force us
or give us the opportunity to learn new
things ourselves about how
how the world is organized if this thing
like learns new abstractions or new
concepts that
humans don't tend to lean on yeah i mean
you might hope so
yeah there i actually um so you
mentioned chriszola earlier um he has
this idea of microscope ai
which is like oh what are we like
there's this question that you have if
you're really worried about agi which is
like well is it what's it good for
um and there's this idea of like okay
here's what we're gonna do we're gonna
like build
these networks that are really good at
some tasks and then we're just gonna
learn
we're going to like inspect them we're
going to understand their weights
and then we're going to learn some
things about the real world like this is
kind of
what happened in go so like we have this
uh
you know everyone's everyone in the
field of ai at least is sort of out good
so we have these uh networks that um are
really good to go
um what people might not know but
probably could have inferred is that you
know there's there's been further work
since alphago
um there's this thing called cartago
which is like this publicly available
network that you can train with and you
know
we we have tons of these blood versus
blood games where the bots are like
higher level than humans and so we know
some things
we know we have some new ideas about go
right we we have like some new opening
tricks and like a new sense of like oh
here's
here like some heuristics that we're
just looking at from how the
thing plays um
which is nice but like
i do wish we could look at the networks
and be like okay what yeah what is this
thing thinking you know
what like like what does it know that i
don't that is uh
or is it just one thing or is it like oh
it's just like
really good at reading or there's like a
it's some
some really complicated answer that you
can't say especially like go right it's
this case where we know
like the the best models they are sort
of
they're not pure deep learning they do
use this monte carlo tree search thing
i don't know whether like if you did
inference without mcts they would still
be superhuman
um we know the domain of go it's not
like how to live a good life or what's
the action space you know you know go is
like pretty pinned down
we know what it means to win or lose we
know what all the moves are
like you would think that we could that
once you train a thing that's really
good at go
you could then gain a ton of insights
but in practice we have to do it this
way where it's like okay
the only way we can learn things is if
we set up a board thing
a board state where the answer that the
thing it plays on the board
is the answer to our question about like
the true nature
of the game of go it's like is that
really the best way that we can come up
with
yeah it's almost like we're only able to
speak in the language of the very final
layer of the network and say like
okay you know we can understand your
outputs that's human understandable
but the minute you start to move into
the network there are
thoughts happening presumably if i'm not
anthropomorphizing too much here
that uh you know that are like not human
tractable and that that
seems like both a wasted opportunity and
actually i think that's a
a good sort of um maybe that's a good
note to close on as well
just this idea that it's not clear at
least that they're necessarily
uh that safety is like clearly a tax on
capabilities all the time
that as these systems get more and more
capable our ability to get value from ai
is actually going to be limited by our
ability to very specifically indicate
what we want obviously this is a very uh
this would be a controversial view i'm
not even sure if i agree with it but
i'm curious what you what you uh think
of that it's definitely sort of
yeah this distinction between like
safety and capabilities where the idea
is that capabilities is about getting
ai to do more impressive stuff and
safety is about like
making sure that it doesn't you know
ruin everything
um it's one of these distinctions
that i think is kind of fake but also
really
but also kind of profound so it's
definitely not the case like sometimes
people are like oh do you do research on
safety or capabilities
i think that is kind of a fake
distinction
where like yeah a lot of stuff like
transparency i think is
i kind of think of them as two different
objectives and research can like
contribute to those
you know to those um objectives in like
different ways
so in the economics i learned in high
school
there's this idea that imagine you're a
country right and there are two things
you can
produce you can produce uh computers or
genes
right like if you put all of your effort
into producing genes you produce 100
genes and zero computers
if you put all of your effort into
producing computers you could produce
like
50 computers and zero genes and if you
did a mix of both maybe you could
produce
like um 49 computers and eight genes or
something
and like there's just this like if you
think about an xy plot where x is the
number of genes you produce and y is the
number
of uh computers you produce like there's
kind of this curve
that that like slopes yeah
some something like that i'm i don't
know how good this maybe it's
an audio yeah some kind of thing um
and like at a fixed level of technology
you can move across the frontier
right so there really is a trade-off in
the sense that like
right now with the knowledge that we
have now we can like go for break in
producing something that's really smart
we can make sure that absolutely nothing
bad happens but like we can't really do
both
but you can move that frontier up right
if if like
if the country like discovers if it
figure out
figures out how to make a a factory
that produces genes like for half the
cost
for half the like number of workers you
know um using half the material
that like the old factories used yeah
well then that moves the curve up like
like with the same number of computers
you could produce more genes
or if you like you can produce the same
number of genes
with these like cheaper factories and
then you use all those workers and all
that like
steel or whatever to make more computers
so um
i yeah still being in the machines not
in the genes i hope
um but basically i think that like there
totally is this idea of this moving
production possibility frontier for
safety versus capabilities where like
yeah if we figure things more things out
hopefully we can get ai to do more
impressive stuff in a more safe way
and like i think that should be the
dream like
you know ai
if we had like really if we had things
that were much smarter than us
and like were really useful to us that
could really change the game
i think so i don't know i i do think we
should be looking at like
how to move the production possibility
frontier
further you know up and to the right and
also
you know what do we um where should we
be on that frontier is a different
question
um where perhaps i would urge safety
more than other people might i don't
know i mean that makes sense to me
like one i think source of tension in
the ai safety ai alignment community
that i think i've perceived is like
there's one camp that says as uh elias
yukowski recently tweeted
like look you've got the thing with
intelligence super intelligence is
you've got one shot
you do it once it's more intelligent
than you and if you screw it up
basically it's game over there's no redo
you can't learn from your mistakes and
try better next time so we really do
have to anticipate everything that could
go wrong
and then the other side is like well um
can we make a superhuman ai that can
help us to align ais can we make a a
narrowly super human ai in some sense
which i believe and i don't want to
speak for open ai but
you know having spoken to a couple folks
in that ecosystem
it seems as though that's a big part of
the philosophy there is like let's try
to let's try to get a theorem proving ai
let's try to get things that can
actually
help us make progress faster than humans
could yeah i do think there's that
distinction
i mean yeah there definitely are some
groups that are like trying to create
really big language models
because they think look there's some
research that you can only do on really
big things
and yeah what do i think about that
i guess i don't know which
i honestly don't know what the right
answer is i do find myself a little bit
i think if i were founding a research
organization
i would probably not put a ton of money
into making really big capable models
but yeah i don't know i there is this
question of like yeah how much
i i think it goes in part to this
question of like
how much warning do you get before you
have a like really
before you have like the terminator or
whatever yeah like like do you go
like like one month before you have the
terminator
do you have like a mouse or do you have
something which is like
three percent less good than the
terminator right if it's three percent
less good then it's a little bit less
like you um have
it's a little bit you know you can like
the problems maybe don't like
qualitatively change
a huge amount and like maybe you can try
and solve them as you go on
but even then i don't know if i think
about like how good is is humanity at
solving these problems that like
creep up i'm like uh
not the best yeah but not as good as i'd
like them to be
but yeah if it's the case that like a
month you had nothing
and then you have like a really smart
thing and you've got like
and if you have the ability to build
this really smart thing then maybe
five days from now someone else has the
ability to build a really smart thing
it's like oh that's a much harder
situation
um and yeah i don't know i think it's
kind of hard to figure out what the
which one is actually going to happen
well and that kind of makes me
think as well about the pace at which
we're adding compute
and and size to these models like you
know it's not like we're going
like open ai is releasing an
incrementally like five percent bigger
system every week or every month instead
they're going from like
you know one x to a hundred x to ten
thousand x and we're leaping way over
these like orders of magnitude which
seems uh
i mean the good thing is that that'll
stop eventually because right at some
point
there isn't enough computation available
in the world
yeah i don't know if that's good news
for veterans but um yeah like like if
you look at these
like um scaling laws for just how much
computation is being used
they just are literally unsustainable
unless we like
you know quintuple moore's law or
something instead of having it slow down
um yeah yeah i
i do think that yeah we're definitely
well we're definitely seeing big like
leaps in like performance or sorry
that's the opposite of what i wanted to
say we're definitely seeing these
massive leaps in size
it's not obvious that we're that like
gp3
was like a thousand times more useful
than gpt2
um i kind of don't know how to think
about like what that scale
yeah that's that i agree that that's a
super confusing thing because like
it is it does seem to be the case that
gpd3 has
economic value that's orders of
magnitude greater than gpt2
but at the same time like it i don't
know i i very much resonate with what
what you're getting at here because
like economic value is it's an important
measure too because companies that can
do this can generate more revenue and
therefore scale systems faster as well
so there's like an interaction there
it'd be interesting to see more thoughts
on on measurement i think jack clark is
doing a lot of that over anthropic
sort of focusing on measurement of ai
progress too maybe that's a good place
for us to park the thought on a
forward-looking note um i do want to
plug
so for anybody who's interested in uh in
dan's work research and
the interviews that he conducts he has a
podcast uh called
excerpt exer sir
so axrp um it's a really great i've
actually listened to
all the episodes they're really good
great exploration of sort of existential
risk research alignment research
with a bunch of different researchers uh
we've had evan who been drawn
um dan's done a great interview with him
about inner alignment so please do check
that out
dan is there anything else you want to
plug i know you've got a website as well
i'm not sure if you're interested in
sharing that yeah i mean i have a
website i don't
if you're interested in what papers i've
published you can read my website
yeah the website also has a blog it's
called danielfilen.blog
yeah i think the thing i'm most excited
about is like if people
listen to my podcast and then
if it's really bad or useless
tell me why and if it's good then
hopefully
your life will be moderately better i
think if anybody who's listening to this
right now
finds themselves interested in pursuing
ai alignment research
i think there's a lot of great stuff on
daniel's website for you to kind of get
started with too so
dig through those papers dig through as
well some of his co-authors like he's in
the thick of it so this is really worth
doing
and um and certainly well dan i mean i
don't know if you're open to twitter dm
stuff like that but uh
i'm volunteering your inbox no feel free
to reach out to me in any case uh
just like for for routing purposes and
whatnot as always
yeah i guess i currently have i could
currently stand
to have a doubling in the number of
people who contact me but i
don't think i could stand to have a deck
toppling so
you heard it here folks govern
yourselves accordingly 2x let's 2x
stands uh
dense twitter inbox here all right
awesome thanks daniel this is great
great to talk to you as well