The Alignment Problem: Machine Learning and Human Values with Brian Christian

- All right.
Good afternoon, everyone.
I usually say,
"Welcome to the Jackson
Institute for Global Affairs."
We're across the street from
Horchow Hall at the moment
here at the Watson Center.
I'm Ted Wittenstein.
I'm Executive Director of
International Security Studies
at Yale and we're just
delighted to partner
with the Wu Tsai Institute at Yale
to co-host this discussion
on, "The Alignment Problem,
"Machine Learning and Human Values."
That's the title of Brian
Christian's wonderful new book.
We'll introduce him in just a moment,
along with Professor John Lafferty,
who's gonna moderate this session.
So, just silence all of your devices.
It's a thrill to even remind people
to do that in person again.
This is not to hybrid Zoom audience,
but we are recording the session
and we'll make the video
available after the fact
to the benefit of everyone.
Thanks so much to John Lafferty
for inviting Brian to campus
and helping us moderate this session.
As many of you may know,
at the Jackson Institute,
we've launched a new Schmidt Program
on Artificial Intelligence
Emerging Technologies
and we're really building
bridges across the campus
in computer science and data
science across the university.
Of course, Professor Lafferty
has extraordinary expertise
in this area himself.
He's the John C. Malone
Professor of Statistics & Data Science.
He has a secondary appointment
in computer science
and he's Director of
the Wu Tsai Institute's
Center for Neurocomputation
and Machine Intelligence.
So, thank you so much, Professor Lafferty.
Thank you, Brian.
I'll let John introduce and
kick off the conversation.
Brian has a presentation
and then we'll make this
interactive with everyone.
So, thank you so much.
- Thank you, Ted.
All right, so, it's a real
pleasure and an honor for me
to introduce Brian Christian to you.
Brian is a science writer
and he was recently a science
communicator in residence
at the Simons Institute
for the Theory of Computing
at Berkeley.
We're pleased to have him
here with us at Yale today
to speak on his recent book,
"The Alignment Problem,
"Machine Learning and Human Values."
This is really a remarkable
book that dives into the history
and the future of machine learning and AI,
focusing on the emerging
societal and ethical issues
that we are rapidly being confronted with,
whether we're ready, or not.
As the world tries to get
its collective head around
the implications of the advances in AI,
Brian Christian's writing is
helping to clarify the issues.
Now, as those of you, who
have read any of his books,
will know, Brian has a superpower,
which is a remarkable ability to distill
extremely complex
technical concepts in a way
that gives the reader a
glimpse of the core ideas,
in this case, the ideas behind
different machine learning
and AI frameworks to understand
both their power
(door slamming)
and their limitations.
The, "Alignment Problem," is
a book about machine learning,
but that, of course, also
involves a lot of discussion
about human learning and
human abilities and biases.
Brian is also the author
of, "The Most Human Human,"
which looks at AI through
the lens of the Turing Test
and also, "Algorithms to Live
By," which was co-written
with the Cognitive and Computer
Scientist, Tom Griffiths.
So, we're very fortunate that Brian's here
to give us a presentation on this book
and afterwards we'll have some discussion
and we'll welcome questions
and engagement from all of you.
So, welcome to Yale, Brian.
It's a pleasure to have you here.
(audience applauding)
- Thank you so much,
Ted and thank you, John.
It means a lot to be here.
So, I wanna talk this afternoon about
the alignment problem in machine learning,
namely, how to make sure that
machine learning systems,
systems that learn by example,
rather than being explicitly programmed,
actually do what we want and what we hope.
We find ourselves, I think,
at a very crucial juncture
around the development and deployment
of machine learning
systems in the real world.
And I think that, critically,
there are questions here,
not only of technology, but
also of governance and law
and policy in what the
path forward looks like.
So, what do I mean when I say,
a transformative decade
that we're coming out of?
Well, many of you in the
room will be familiar,
but just so that we're on the same page.
We've seen in the area of
image classification...
My slides are kind of
flickering in and out so,
we may have a slightly more
verbal than visual presentation.
In the area of image classification,
where you're trying to determine
the contents of an image,
we've seen error rates drop
42% in a single year in 2012
and cumulatively by more
than 92% over six years.
That just doesn't happen very often.
Now, fast-forward to the present.
It is now possible to train a system
to achieve human comparable accuracy
on the ImageNet competition
(door slamming)
for $4.59.
So, in the span of nine
years, this task went
from impossible to, you
can do it for $4.50.
In fact,
today you can't even take a picture
with a contemporary smartphone camera
like an iPhone, or an
Android, without invoking,
in this case, an 11-stage
machine learning pipeline
that is doing everything
from the auto exposure
to the focal distance,
to the white balance,
to the color correction, de-noising,
it's fusing multiple exposures together.
All of this is happening
silently and invisibly
in real time
and
it imposes some interesting constraints
on the types of photos that
you can and cannot take.
So, for example, my friend
reports to me that it's very hard
to take a photograph of
falling snow at night
because the iPhone software
thinks of it as noise
on the sensor and de-noises
the snow out of your photograph
and you have a photograph
with no snow falling.
But I think this is a
really interesting analogy
for the relationship
that we've come to have
with machine learning, namely, that ways,
ways that are ubiquitous, but
at the same time invisible,
it has kind of interposed itself
between ourselves and the world.
It is mediating our
experience of the world
in ways that we don't always appreciate.
Of course, this is far from
the most consequential use
of machine learning.
We are seeing autonomous cars increasingly
on neighborhood streets.
Certainly where I live in San Francisco,
it is almost every hour of the day.
There are cars from
Cruise, Waymo, et cetera,
coming through and you always
have this awkward interaction
at an intersection where
you're not quite sure
if it sees that you're there,
or if it's going to yield to you,
or whether it's going to
be more, or less aggressive
than a human driver would
be in that situation.
Andrej Karpathy who runs AI at Tesla
has described machine learning
software as a kind of fungus
that is eating away at the
C++ code that he and his team
have so painfully written over the years.
I mention this, partly
because we have this narrative
that machine learning is
replacing human expertise
and human judgment, but
I think it's actually
even more significant than that.
It's also replacing
traditional software
engineering practices.
Then you might, or may not be
able to see it on the screen,
but there was a Tweet
from Waymo CEO yesterday
saying that they have just
now begun officially doing
driverless cars with no
human in the driver seat
in San Francisco.
So, I am the human guinea pig
for this particular exercise.
So,
not only in kind of celebrated
high tech examples like this,
but in ways that I think are,
in many ways going under the radar.
Machine learning systems,
some of them no more complex
than what you could put
in an Excel spreadsheet,
are increasingly penetrating
the decision-making of our
institutions, public and private.
So, if you look just at the
criminal justice system,
there has been an exponential
uptick in the use of
statistical and machine
learning risk assessments
in the U.S. criminal justice system.
And so, machine learning is
replacing both human expertise
and traditional software.
It's doing it in ways that shape
our most everyday interactions
and could also determine
the course of our lives.
I would argue that there's a sense
in which we are putting the world,
literally and figuratively, on autopilot.
So, there are a lot of reasons I think
to be, frankly, concerned.
Are these systems actually learning
what we think they're learning
and can we actually trust them to do
what we think they're going to do?
Now, this is far from a new concern.
In fact, it goes all the way back to 1960
when the MIT cybernetics
pioneer, Norbert Wiener,
wrote this, I think, very
prescient essay called,
"Some Moral and Technical
Consequences of Automation."
Wiener used the metaphor of,
"The Sorcerer's Apprentice,"
which many of us know as this
lovable Mickey Mouse cartoon,
where Mickey enchants this broom,
animates this inanimate object
and gives it some simple commands like,
"Fetch water from the caldron."
Of course, Mickey is
not quite precise enough
in how he formulates his command
and anyone who's watched the cartoon knows
that he ends up almost drowning himself
by the time that the
master magician appears.
And Wiener says, I think,
quite prophetically,
this is not the stuff of fairy tales.
This is what's coming for
us in the relationship
that we are going to have
with artificial intelligence.
The famous quote of his is,
"If we use, to achieve our
purposes, a mechanical agency
"with whose operation we cannot interfere
"once we have set it going,
then we had better be quite sure
"that the purpose we put into the machine
"is the thing that we truly desire."
Today, this has become one
of the central concerns
of the field of AI and we know
it as the alignment problem.
So, five years ago, five
and a half years ago,
I decided I wanted to tell the story
of the alignment problem,
of the history of machine
learning, how it's intersecting
in deep ways with human
norms and human values
and the technical and
interdisciplinary community
that is coming together
around some of those questions
to do what I think is
some of the most exciting,
but also important work in the field.
That process ended up taking
me through archival research
into the field's pioneers.
People like Walter Pitts
and Warren McCulloch,
all the way to about 100
oral histories and interviews
that comprise the stories of the book.
Obviously, we don't have time to get into
much of that level of detail,
but there's a few threads
that I'd like to pull
through this morning.
So, said most plainly,
(audience member coughing)
how can a machine learning
system, as Wiener said,
fail to have the purpose put
into it that we truly desire
and what, in turn, can we do about it?
Because this is co-sponsored
by the Wu Tsai Institute for
the Study of Human Cognition,
I'll try to highlight as
well, few of what I think
are some of the juiciest intersections
between machine learning and
human cognition along the way.
Okay, let's begin.
So broadly, we can think of
a machine learning system
as having two halves.
There is the training
data, the set of examples
from which the system learns
and what's called the objective function,
which is how we are going to
mathematically define success
in each of those examples.
Each of those offers an opportunity
for things to become misaligned
and we'll look at each of those in turn.
I'll start first with the training data
and I'll be a little bit briefer here,
so there's more time to talk
about the objective function.
Many in this room, I
suspect, will be familiar
with one of the most infamous
cases of machine learning,
in particular, image
classification gone wrong,
which was in the summer of 2015.
Google Photos suggested
the auto-generated caption,
"Gorillas," to an album of selfies
taken by web developer,
Jackie Alcine and his friend.
This incident and others like
it, led to a kind of reckoning
into the work of people
like MIT's Joy Buolamwini,
who did a, I think, now
classic intersectional analysis
on commercial face recognition,
face detection systems
of 2017 and 2018, showing
that the error rates
in then-state-of-the-art,
commercial systems,
were orders of magnitude higher
for darker skinned females in particular,
something like 30 times higher
than they were for white males.
This has come as part
of a more broad scrutiny
of the kinds of data sets that get used
both in industry and in academia.
One of the most used and most
cited data sets of the 2010s,
for example, is called,
Labeled Faces in the Wild,
which was scraped together on the web
using digital
newspaper front pages from the 2000s.
An analysis that was done just
several years ago, showed,
for the first time, that as
a result of this methodology,
the dataset contains
twice as many pictures
of George W. Bush as it does
of all Black women combined.
So, anyone working with this dataset
to build face recognition
technology had perhaps
inadvertently built George W. Bush
recognition technology.
(all chuckle)
These very same sorts of dataset issues
are also being seen in autonomous driving,
including in the 2018
crash of the Uber car
that killed a pedestrian
in Tempe, Arizona.
If you read the National
Transportation Safety Board report,
you find something really striking,
which was that the system
basically did not have
any training data of jaywalkers
and so, it was just
fundamentally unprepared
to encounter someone crossing
a road not at a crosswalk.
What's more, the system was built
on this object classification system
that had a very rigid set of categories
that included pedestrian,
cyclist, debris, et cetera
and had thousands of examples
of each of those things.
But this particular woman
was walking a bicycle
across the street,
which was something that
the system had never seen
(door slamming)
and so, all bets, essentially,
were off.
To underscore this point,
this is head of AI at Tesla,
Andrej Karpathy, speaking at
my institution, UC Berkeley
and explaining that when
he was a PhD student,
he lost 95% of his sleep
thinking about models and algorithms
and today running AI at Tesla,
he loses 75% of his sleep
thinking about data sets.
We're starting to see some
norms emerge, not unlike
the nutrition facts that
you see on food packaging,
to indicate the contents
and the potential dangers
of data sets.
This is from Google.
Microsoft, IBM, et cetera, have
their own versions of these.
They're called model cards, or data cards
and they include information
about the providence
of the data and possible misuse,
possible bias, et cetera.
And Labeled Faces in the Wild
itself, now starting in 2019,
comes with this big red
warning label attached
telling you about the
various SKUs in the data set.
These sorts of disclosures, I don't think,
will prevent all harms and
neither do nutrition facts,
for that matter, but it's a start
and I'm encouraged to
see that norm emerging.
Although I suspect Andrej Karpathy
will have to keep losing sleep
for the foreseeable future.
So, we'll come back a little bit
to questions of training data at the end
when we talk about language
models, but for now,
let's shift the focus to
the objective function.
This is how we're going
to numerically operationalize success
in each of these examples.
We have seen this be
the crux of many alignment issues in AI.
I think perhaps most
well-known is the COMPAS system
that does pre-trial risk assessment
in many jurisdictions in the U.S.
And really the focus of the
ProPublica investigation
into this system in 2016,
which caused a number of
headlines ended up sharpening
into a question of the
correct statistical definition
of fairness to be used in the
case of classifiers like this.
So, one potential way of defining fairness
is what's called
calibration, where you say,
okay, if someone is an 8 out
of 10 risk to be rearrested,
then they're gonna have
the same probability
of being rearrested whether
they're white, or Black,
et cetera.
This seems pretty intuitive
and it seems pretty desirable.
But there are other
definitions of fairness
that we could use, for example,
what's called equalized odds
or equalized opportunity,
which looks more at the error rates
and the composition of false
positives to false negatives,
et cetera and it looks for this property
to be the same across different groups.
That also seems like something
that we would really want
and seems intuitive and an
aspect of our legal intuitions
about what fairness might actually mean.
It turns out, we don't
have time to get into this,
but a system can't
satisfy these definitions
at the same time and so
we are left with, I think,
a really interesting set of
questions at the intersection
of computer science and public policy
around how we want to
statistically operationalize
these ideals of fairness
and equal treatment
that exist in the law.
You could think about,
even the infamous Google
Photos' gorillas example
that we began with, as a matter,
not only of training data gone wrong,
but also of an objective
function gone wrong.
So, the classic objective function
that's used in image classification
is what's called cross-entropy loss,
but you can simply think of
it as wanting to minimize
the number of misclassifications.
Well, what could go wrong?
That seems quite intuitive indeed.
If all you care about is minimizing
the number of misclassifications,
then you're implicitly assuming
that any misclassification
of any X as any Y,
is equally harmful.
But I think part of what the
Google Photos example shows us,
is that this is not true at all.
There are many, many errors
made by Google Photos
that didn't result in a national scandal
and personal apologies from the engineers.
In fact, I think it's probably likely
that certain misclassifications
are millions of times
more harmful than others.
So, how do we try to reorganize
some of the fundamental
objective functions
of something like image classification?
There are a number of computer scientists,
including Stuart Russell, who have argued
that we should make
this loss matrix itself,
something that we want
the system to learn.
We'll come back to that
idea at the very end.
Thus far we've been talking about,
what's called supervised learning,
as many of you're probably familiar.
In discussing the objective
function, I'd like to now
turn our attention to a different
branch of machine learning
which is called reinforcement learning.
So, if supervised learning
is about predicting
certain hidden attributes
from visible attributes,
reinforcement learning is about
essentially maximizing rewards
and minimizing punishments
through a series of behaviors.
Now, this has been used in many arenas
within machine learning.
Some of the most significant successes
like playing Atari games
with superhuman capacity,
defeating the world champion at Go,
increasing dexterity
of robots and so forth,
are examples of reinforcement learning.
Reinforcement learning is also
to an underappreciated extent,
being used in consumer tech.
This is a very interesting
paper that Facebook published
a few years ago, talking about
the use of reinforcement learning
for delivering notifications to users.
They had previously used a
supervised learning system
that simply ranked all
the possible notifications
by the probability that you
would interact with them.
Then, if it was over
a certain probability,
they would send you the notification.
This led to various things,
including people getting burned out
and turning notifications
off, which they didn't like.
So, they changed to a
reinforcement learning system
where Facebook gets points
for the interactions
that you do with their notifications,
but once you turn notifications off,
it's like game over in the Atari context.
In fact, they literally used
the exact same model architecture, DQN,
that DeepMind used to play Atari games,
Facebook is now using
this to play us, right?
I think we're all familiar
with the expression,
"You're not the consumer,
you're the product,"
but I think we can maybe
add another adage, which is,
when we think about the
gamification of social media,
we are not the player,
we are literally the game
as far as Facebook is concerned.
We are substituting for
the Atari in this context.
I also just wanna mention
very briefly in passing
that reinforcement learning is
of particular interest to me
because of its rich
connections to neuroscience.
In fact, a particular set
of ideas in the early '90s
called Temporal Difference learning,
whose first major success was in the use
of computer backgammon, ended
up, by the end of the 1990s,
becoming accepted as the explanation
for what was going on in
the human dopamine system,
which had been an unsolved riddle
in the neuroscience community.
I think this is one of the
most interesting stories
in reinforcement learning
and it shows that
there's a very deep resonance indeed
between some of these
ideas in computer science
and the same fundamental
algorithms of learning
that evolution found.
But what I wanna focus on,
is this mysterious numerical reward
at the heart of reinforcement learning.
These points that the
system is trying to maximize
because that reward function
is what determines the behavior.
It turns out to be exceedingly difficult
to develop a reward function
that doesn't break down
in sometimes hilarious,
sometimes tragic way.
My favorite example comes from
David Andre and Astro Teller
who work at Google X now, but
in their grad student days,
they were working on a
robotics soccer competition.
And the soccer robots were just wandering
all around the field at random.
They had no idea how to score goals.
So, they decided to give them
a tiny numerical incentive,
which in computer science
is called reward shaping,
but you can just think
of it as an incentive,
of something like 1/100th of a goal
for taking possession of the ball.
What did the system learn to do?
It learned to approach the ball carefully
and then vibrate its paddle
as quickly as possible,
taking possession, like 50 times a second.
So, if you talk to any
reinforcement learning researcher,
they have their own set of kind
of horror stories like this
of the system doing what they said,
but not what they wanted.
This is something, too, that I think
has very deep connections
to human psychology
and human incentive design.
My favorite example here
comes from my friend and
collaborator, Tom Griffiths,
who's a cognitive scientist at Princeton.
He's also a dad.
One day his five-year-old
daughter was sweeping up
some crumbs on the floor and
put them in the trash can.
He was very proud of this
and so, he did what any parent would do
and activated his reward
function, which is called praise
and said, "Wow, great job, honey!
"You did such a good job sweeping."
And he watched as his
daughter, beaming with pride,
then took the trash can and
dumped it out on the floor
in order to start sweeping even more
and get an even greater helping of praise.
So, Tom had fallen prey
to exactly the same kind
of incentive failure
that David Andre and Astro Teller had.
I think it's quite interesting
that cognitive scientists
and economists are turning to
the computer science community
for ideas about how to
create incentive structures
that don't distort behavior.
But the moral for our purposes
in thinking about alignment,
is simply that it's very hard
to create an incentive system
that doesn't breakdown and
can't be exploited in some way.
We see this even in toy
environments like Atari games.
In the simplest of Atari
games, it might be possible
to simply reward our system
for getting points in the game.
Space Invaders is one example.
But even by the time we
get to Super Mario World,
points don't really reflect
the actual playing of the game.
You can get points for getting
coins and breaking bricks,
but that's not the point of the game.
The point of the game
is to save the princess.
If you look at, for example,
this boat-racing game,
which is called Coast Runners,
there's a famous example from OpenAI.
They train the system to maximize points
in the boat-racing game,
but it found a loophole
where you can do donuts
in this little harbor
that has these self replenishing power-ups
and it forgets about the race altogether
and just drives off the course
to do these donuts infinitely.
In games that are very stingy with points,
for example, this one
called Montezuma's Revenge,
a system based only on random exploration
and then using reinforcement learning
to get more and more points,
we'll never figure out
how to get the points at all
and we'll just sort of give up
and stand there and not move at all.
I think this is very telling.
So, even in these really
toy sandbox domains,
it's very, very difficult to
articulate a reward function
that incentivizes what we
actually want the system to do.
So, what hope do we have in
any kind of real-world setting,
like driving a car through the
street, of making this work?
The conclusion by most people
who think about AI safety,
is that generally-speaking,
it is simply not safe
to manually supply a reward
function to an RL system,
particularly in the real world.
There's always gonna
be some weird loophole,
or something that's going to exploit.
But maybe we can do something at else.
Maybe we can make the learning
of the reward function
part of the machine
learning problem itself.
Maybe we can have the system
learn the reward in our heads,
from us.
My favorite example of this is a paper,
that was a collaboration
between DeepMind and OpenAI.
Paul Christiano led the DeepMind effort
and Jan Leike, excuse
me, the other way around.
They wanted to take something
that would be very obvious
if the system got it right,
but almost impossible
to specify numerically.
And what they settled on was backflips.
So, they wanted to see
if they could get a robot
to do a backflip.
Now, if you think about
this, it's very hard
to come up with some mathematical formula
that determines what a backflip is
as a function of the rotation,
or the torques, or whatever.
It'd be very hard to even do a
demonstration with your body.
It'd be hard to do
demonstration with a joystick,
but you'd know it if you saw it.
So the question was, was that enough?
So, they had this really wonderful setup
where they would have this robot
just wiggling around at random
and they'd show you two video clips
and you'd get this instruction
that says, "Look at the clips
"and select the one in
which better things happen."
I just love how vague that is.
So, which one of these
wrigglings looks infinitesimally
more like a backflip?
And in so doing, the system
would then, behind the scenes,
build this reward model
of what reward it thought
you had in your head that would explain
these preferential
judgments that you'd made
and then it would go and
try to optimize for that
and show you two more video clips.
Again and again, you pick the
left clip, the right clip,
blah, blah, blah.
You do this for about an
hour, which is fairly boring,
but by the end of the hour,
something amazing has happened,
which is that the system is doing
these gymnastically perfect backflips.
I guess you don't quite
have the refresh rate
to appreciate the aesthetic beauty,
but it is tucking itself
in, in order to spin faster,
the same way a figure
skater tucks their arms in.
It's sticking the landing.
I find it even more intriguing,
every person that they did this with,
the system ended up with a backflip
that was slightly
different as if each of us
has our own platonic ideal
of a backflip in our heads.
And I think this is really remarkable
that we have a mechanism
for extracting these aesthetic preferences
out of human brains using nothing
but these binary preference judgments.
I think that is really,
for me, sort of a beacon of
hope that we can get beyond
the sort of manual
specification of reward.
Now, I thought I might say a few words
about large language models,
which is kind of the current
frontier of AI alignment.
I'll try to be pretty quick
because I wanna make sure
that we have plenty of time to chat.
So, you can think of, what to me is,
sort of the current frontier
in machine learning,
which is these so-called
large-language models.
For people who aren't familiar,
it's basically autocorrect on steroids.
So, we have these systems on our phones
that will predict the next
word that we're gonna type
and in fact, they will
secretly dynamically change
the input buttons on
our keyboard invisibly,
to make more typical letters
literally wider on the screen
and easier to hit.
But what would it mean to
have an autocomplete system
that could autocomplete
an entire term paper?
So, that is what the current
generation of things,
for people who aren't familiar.
I think it's really stunning.
Here are a few examples of
me playing with OpenAI's GPT
So, I said, "The following is an essay
"for Mrs. Simpson's fourth-grade class
"about what would happen
if dogs could talk,"
and it produces a reasonable
fourth-grade essay
about what dogs would say.
I could say, "This is my AP
English exam about symbolism
"in the works of Herman
Melville," and out comes, I think,
a very reasonable high-school-level essay
about symbolism in Moby Dick.
I even asked it to give
me my own remarks to you
on the alignment problem
and it came up with
something fairly generic,
but I think totally coherent.
And why stop at prose?
Why not do code as well?
Microsoft has this thing
called GitHub Co-pilot,
which will autocomplete your code for you.
You can say, "The following
is a Python function
"that will, blah, blah, a
blah," and (blows a raspberry)
out comes Python.
DeepMind has the same thing.
But there is a fundamental
alignment problem here
between the autocomplete objective
and these systems actually
being helpful to us.
So, for example, if you say,
"Explain the moon landing
"to a six-year-old."
You think you're giving
a command to the system,
but the system merely
thinks it is autocompleting
a document that contains that sentence
and so, it autocompletes it
as, "Here's a list of commands.
"Explain blah, explain,
blah, explain blah."
Well, that's not what we
actually wanted, right?
So, there's a fundamental
difference in the objective
that we've trained the system on
and how we actually wanna use it.
There's also this fundamental
problem, as many of you know,
which is that the internet is quite toxic.
So, if you build a language
model on the internet,
it will sometimes say
extremely toxic things.
In fact, this is a problem
that actually gets worse,
not better, as the size
of the model goes up.
Particularly if there's
any hint of toxicity
in your own writing, or your own prompt,
the smarter, the more
powerful the model is,
the more likely it's
going to pick up on that
and make an even more
toxic autocompletion.
The same thing is true of buggy code.
There's a lot of buggy code on GitHub.
If you start writing buggy code,
a powerful language model will realize,
oh, this guy's writing bad
code and in my training data
bad code tends to be followed
by even more bad code.
So, it will take your mistake
and use that as a reason
to give you a bunch of
crappy code in autocomplete,
but that's not what you wanted.
So again, this is the alignment
problem in all of its glory.
We've seen reward modeling techniques
like with the backflip used here,
where they will give the human
raiders from Mechanical Turk,
a bunch of possible
outputs from the model.
They will ask the people to
rank them from best to worst
and they will use this to
generate a reward model
of what a good summary of a document is.
And amazingly, this appears to generalize
up to a point.
Of course, there's an
interesting question of,
who are these raiders?
Do their preferences coincide
with what our preferences would be?
Whose definition of
toxicity do we care about,
et cetera, et cetera?
In some ways these are some of
the oldest ethical questions
of them all, which is just,
who gets a seat at the table?
As a place to sum up, I
think this is significant.
This reward modeling
technique is significant,
not only within AI itself,
but also for broader forms
of institutional decision-making.
If you think about the way
companies, governments,
et cetera, make decisions,
often they determine
some explicit metric.
There's some meeting at
which a metric is determined
and then that metric is optimized
until there's some future meeting
where they decide to change their mind.
So, for example, at
Tinder, there was a meeting
in 2013, or something, where they decided
that their metric was going to
be maximize swipes per week.
So, any change to their
layout, or their UI, et cetera,
was going to be A/B-tested and
it would only be rolled out
if it increased the average
number of swipes per week.
So, if anyone has heard
people complaining,
"Uh, all I'm doing is
just swiping mindlessly,
"but there's no actual interaction,"
well, that's their objective function.
That's what they're optimizing for.
We saw Facebook, which was
optimizing for many years
for time on site and then
this produced addictiveness.
They scrapped that,
then they started
optimizing for engagement
and this produced the outrage machine
that we now know so well.
But you see this outside of tech too.
There's this desire to
make more evidence-based,
objective assessment in education,
but this results in teaching to the test.
We can optimize for
shareholder returns, or GDP
and this ends up with huge
externalities to the environment,
to inequality, et cetera, et cetera.
So, if 30 minutes of case studies
on machine learning gone
wrong has imparted anything,
it's that this is a highly dubious way
of running a tech startup
let alone the world, right?
Is to just define some of objective metric
and just hit the gas pedal.
But maybe reward modeling suggests
that there's some alternative here.
People often ask me if I'm pessimistic,
or optimistic about AI.
If I'm pessimistic it's
because the alignment problem
is, in my view, exactly the way
that human civilization is
already going off the rails
and the AI is just a
forced multiplier of that,
our ability to ride bad metrics
into the externalities of no return.
However, if I'm optimistic
it's because I think
we are coming to an understanding
that there's something beyond
the optimization of metrics.
So, we're starting to see social media UI
removing the quantitative
part of the experience
as if to say, don't optimize too hard
for the number of people
that like each photo.
And I think most poignantly,
we have this idea
that you can just present to
people two different worlds,
two different versions of the timeline,
or something as generic
as that and just say,
"Select the one in which
better things happen.
"Which of these
experiences do you prefer?"
You don't have to be
able to even articulate
why you prefer one to another
and you can still allow the system
to have a more nuanced
representation than you would get
by just defining metrics yourself.
Okay, so this is roughly speaking
the current state of
the alignment problem.
I think there's a lot of work
to do and I hope that's clear.
I honestly think this is some
of the most exciting work
that's happening right now,
both on the policy side
and in the technical side.
I think this is likely to be, in my view,
the defining human project of the 2020s,
is how to get machine
learning to work for us,
if not the 21st century.
So,
at its best, I think it offers
us a revelatory encounter
with our own
human nature.
What we value, how we learn, what we want.
On that note, I'll give the
last word to Alan Turing.
He was giving a radio address in 1952
about really early experiments
in machine learning
that he was doing.
And he says to his co-panelist,
I've been doing a bunch of
these experiments lately,
but the system takes an
awful lot of intervention.
It's always learning the wrong thing,
or not learning the right thing
and I'm constantly having to
jump in and correct course.
And his co-panelist says,
"Okay, but who is learning, Turing?
"You, or the machine?"
And he says, "Well, I guess we both were."
Thanks.
(audience applauding)
- Okay, thank you, Brian.
That was so interesting.
So, maybe I can start with a few questions
and then we can open it
up for more discussion.
But maybe I can start with,
your last quote from Turing
is a good place to start
and one of the things
I wanted to ask about,
which is co-evolution
of humans and machines.
- Mm.
- So, machine learning has advanced.
Some people like to joke
by stochastic graduate student dissent.
(audience chuckles)
And that's one type of
survival of the fittest
algorithms evolution
that has advanced the
technology for many years.
But as these systems have become
fielded and used by people,
there's this co-evolution
where, as we use them,
they become adapted to us and
we become adapted to them--
- Yeah.
- as that Turing quote
hints at.
And without becoming...
We could become grandiose about it
and sort of draw an analogy
between the agricultural and
the industrial revolutions--
- Yeah.
- that the AI revolution
in is leading us into now.
But what are your thoughts on how machines
and human societies are
gonna co-evolve in this way?
- Yeah.
I think that's a great
line of inquiry.
There's a quote that comes
to my mind from Hannah Arendt
where she's talking,
in the mid-20th century
about behaviorism and she says,
"My problem with behaviorism
"is not that it's false, but
that it could become true."
And I think about that quote
a lot in the context of AI.
I think it is true in
almost any computer system,
whether it's traditional
object-oriented programming
where you have to design this mini world
with nouns and verbs, or
something like machine learning
where you're determining a
category structure, et cetera.
These are simplifications.
You know, we use the word model.
There's the famous quote from George Box,
"All models are false,
but some are useful."
And I think there's a danger that
the models that we're
building become so powerful
that they reshape the reality
that they were originally approximating
and then the reality itself conforms
to the assumptions that
were made in the model.
So, for example,
if every car on the road ran
the Uber software from 2018
and killed jaywalkers, then
people would stop jaywalking,
or the people who did
jaywalk would be killed.
So, the assumption that
the model makes of,
there are no jaywalkers,
would become true.
- [John] Hmm.
- You sometimes see this.
There can be kind of a confirmation bias,
even in autocomplete systems where...
I, for many years, my phone
would autocorrect the word ill
to capital I, apostrophe L-L,
even in a syntax of like,
"Yeah, I'm still feeling
a little bit ill,"
it would replace it with I'll.
I don't know if 2022 is the
year we finally stop doing that.
But
my response to that is to
fight it the first couple times
and then to just cave and be like,
okay, you want me to
say sick, oh, I'm sick.
(John chuckles)
That's something that
really gets under my skin
as a creative writer, which
is that, it has always been
the role, the calling of
poets, of painters, et cetera,
to find authentic idiosyncratic
modes of expression,
to sort of buck the traditions
and the conventions.
That's hard enough if you're
just living in a society
that has certain norms,
but Picasso wasn't painting on a canvas
that was literally
pushing his brush strokes
into more conventional shapes under him.
That's essentially what we have,
the textual equivalent now.
- Hmm.
- That's another way in
which these models are false,
but could become true.
That a team at Apple, or
whatever, deploys this model
and they see, oh, our accuracy
keeps going up and up.
We're getting better and
better at predicting the word
that the person wanted to type.
Well, maybe that's true,
or maybe the person's just giving in.
So, you're impoverishing the language.
So, that is a kind of co-evolution.
There's a Oxford philosopher
named John Lucas who says,
"If and when we pass the Turing test,
"it will not be because machines
have become so intelligent,
"but because human speech
has become so wooden."
So, that's the kind of
thing that worries me from
a co-evolution standpoint.
- Yeah. Yeah.
Really interesting.
Another question goes
to the first part of it
when you talk about the data problem.
I guess this is the one that's now keeping
Andrej Karpathy
(Brain chuckles)
awake at night.
- At night, yeah.
- Another way of talking about this is,
garbage in, garbage out.
- Yeah.
- And you can see this very
easily in word embedding.
So, I think you talk
about this in your book.
And you can see it in
the large language models
as well.
- Yeah.
- That's why OpenAI pulled the plug
on the access to those
language models initially.
But the question I wanna
ask about this is that,
one way of thinking about
the issue here is that,
we're training these
systems on human data,
but we're holding the systems
to a higher standard--
- Yeah.
- than we hold humans to
and I think rightly so.
- [Brian] Yeah.
- So first, in a domain
like self-driving cars,
it's easy to say that we want
the statistical rate of
accidents to be much lower
than it would be for
humans, but in other domains
like language, it's not so clear.
And if we can't even agree
on fundamental questions
of morality amongst ourselves,
how can we prescribe them for machines?
So, what is your thinking on this notion
of holding the machines
to a higher standard?
- [Brian] Yeah.
I think there's two things there.
The first part is this idea of
how can you do better than human morality?
There is a quote from a
Google researcher saying
this idea that we need machine systems
to embody human values is not good enough,
'cause human values
are not good enough--
- Right.
- as they are today.
And it's interesting
because you would see this
in the '80s, '90s, 2000s in
the game-playing literature
because back then a lot of the methodology
for even the way Deep
Blue played chess was
we're gonna start with a huge
database of grandmaster games
and learn to predict the way
that the grandmaster plays
and then will play like that.
And you started to see,
even in computer Checkers,
people are writing, yeah, but
our program is gonna
get to a certain point
where it's actually better than a human
because it's going to
play a different move
than the one that human
played in that situation.
We're gonna tag that as an error.
So, how can we ever hope
to transcend that barrier?
And essentially the way that you do it
is by just having the program
play itself from scratch.
This is like AlphaZero and
so forth and you can sort of
throw the human data
away either the later,
or just from the very
beginning never use it.
So, is there a way to
do something like that
in the moral domain?
Which sounds crazy, but I think
there's a number of people
who are asking that question seriously.
One of the people who comes
to my mind is Paul Christiano.
He was at OpenAI.
He now runs his own thing
called the Alignment Research Center.
He has this idea, which is
called Humans Consulting HCH,
which is a recursive acronym.
The idea is that you
ask people some question
and you give them a computer
system as a resource,
as a research assistant.
It could be like, "Is such
and such medical procedure
"ethical or not?"
And then your computer
assistant is empowering you
to be better informed about that
than you would otherwise be,
but it's also trying to predict
your ultimate decision.
And as it becomes better able
to predict your decision,
it's better able to inform
you, or empower you.
In theory, you can get a
sort of bootstrapping effect,
or a virtuous circle where it
can then essentially push you
deeper into your own values
than you would've gotten on your own,
but without supplying anything
else from the outside.
I think that's a really tantalizing idea.
I've yet to see anyone
really take a crack at it
as a research project,
but I think that's really interesting.
The other aspect of what
I think is going on here,
is this question of how can
machines embody human values
when human values are heterogeneous?
People don't always agree.
So, in the OpenAI research,
the language research
I talked about at the very end,
agreement between their Mechanical Turkers
on which summary is a better
summary, is about 73%.
So, it's good enough that
we can just kind of go
with the modal answer.
But what do you do when
people don't agree,
or where there's sort of
multimodal preferences?
You see this with self-driving
cars where it's like,
if half of the people would swerve around
the object to the left and
half swerve to the right,
you don't wanna just split the difference
and plow into it.
(John chuckles)
So, work on how to deal with
heterogeneous preferences.
I mean, even something as seemingly banal
as a recommendation system
for parents who have children
that use their Spotify account,
their Spotify is forever polluted
with these dumb children songs
and they can never get back
to the music that they like.
So, even multimodal
preferences for something
that's seemingly everyday
and banal as music,
we still haven't figured that out.
And it's sort of an open research problem.
- Yeah.
And then replace parents and children with
one human society, one country and another
and (chuckles) these
problems are magnified.
So, I wanna open it up to
questions, but before I do,
one question that
I think a lot of people
would like to ask you,
which is that...
I think you really have a
remarkable ability to distill
very complex ideas in machine learning
and communicate them very effectively.
What would you say to somebody
who wants to learn about machine learning,
but is not in a STEM field?
- Hmm, hmm.
- Maybe they're seeking to
serve in a leadership role--
- Yeah.
- and to communicate some of these ideas
in a way that is both
substantial and impactful.
- Yeah.
I think that's a great question.
One of the things that I've
been thinking about is...
During the writing of the book,
a law was passed in
San Francisco, my city,
mandating the use of this
thing called the Arnold Tool,
which is a statistical
risk assessment instrument
in city courts.
I went with the person who's
now the DA in San Francisco
and sat in on arraignment
hearings for a day.
It was maybe a week, or two
after that law went into effect.
It was very interesting
hearing these judges
get this readout of this
statistical system and say,
okay, your trial date
is four weeks from now.
I have to decide whether you stay in jail,
or go home in the next four weeks.
This thing says you're an eight out of 10.
That sounds kind of bad
(John chuckles)
so, I guess you're gonna
go to jail.
I'm exaggerating slightly,
but that was the kind
of thing that one heard.
And I've thought a lot
about how for many people
in non-technical fields, you know,
you could be judge sitting on
the bench for 20, 30 years,
suddenly there's a law
that's passed that says,
now you have to work arm and arm
with this machine learning system.
You don't really know...
You don't have a working knowledge of
any of these bias issues,
or what might make someone
an exception to the rule, or
even really what the risk is
that the risk assessment is predicting.
There are many cases
where the risk assessment
explicitly says, "This
is not for sentencing,"
but people still use it for sentencing.
I think there's a great need for that.
That's part of my motivation
in writing the book, honestly.
I was feeling that there
was a broad community
of non-computer scientists,
whether that's policy people,
doctors,
lawyers, et cetera, that I
could try to serve by a offering
a little bit of a gentle
machine learning 101 curriculum
full of personal narratives
and hopefully readable enough
that it would be useful.
So, that's something
I've thought a lot about.
I've also,
a couple weeks ago, I did an event with
the AI Subcommittee of the UK Parliament
and they asked me this question of like,
all of the parliamentarians
want to know more about AI.
What do we do?
I recommended maybe there should be
some kind of consulting
group, or center of knowledge
within the government that
can then get loaned out
as needed to advise, or work with people
as these sorts of things come up.
I wouldn't mind seeing
something like that in the U.S.
I think it's very much
an open question though
'cause I do feel that fluency
with some of these basic ideas
in machine learning, is becoming just
part of the core curriculum
of being a citizen--
- Yeah.
- being a person.
I mean,
even knowing in a self-driving
car when to be more vigilant
and when to be more relaxed.
One of the interviews
that I did in the book
was with Dean Pomerleau who
did one of the first rides
in a self-driving car in the early '90s
with a computer that was
like 1/10th, as powerful
as a first-generation Apple watch.
He still drove on a highway for two hours.
I asked him what the experience
was like and he said,
well, I knew that if I
went through a tunnel,
I was gonna have to hover
my hands over the wheel
because I knew that this
was out of the distribution
of normal sun-lit road markings.
So, that's the kind of thing where now
every Tesla owner should have
that same kind of spidey-sense
of, oh, there's a weird thing happening
where there's a full moon,
but there's a forest fire,
so it looks like a yellow
light hanging in midair.
I'm gonna just (chuckles)
pay a little bit more attention
now than I normally would.
I do think that's just kind
of part of being a person now.
So, I don't know exactly
what is the best way to address that,
but it's something that's
very much on my mind.
- [John] Yeah, gets at this
issue of trust that we--
- Yes.
- talked about.
- Yeah, absolutely.
- [John] Oh, thank you.
So, let's
open up to questions.
(Brian chuckles)
Lori.
- [Lori] Yeah, so thank you for the talk.
So, I found it very interesting,
but I'm just not fully
understanding the optimistic note
that you ended on.
- Sure.
- [Lori] That's in particular.
So, take the example
you gave where humans...
Maybe this will help with the mask.
Okay.
I'm not even sure it's on.
- It's recording.
- [Lori] Oh, okay, okay.
So, can I pull it down to--
- Yes.
- [Lori] Okay.
So, in that example, what was key was that
the humans that did the better thing,
knew what a backflip was.
It was something they recognized
and so they could make a judgment.
But the real issue for us is
recognizing, or for machines,
is recognizing entirely
new kinds of events,
like a pandemic--
- Hmm.
- [Lori] or a president that
doesn't follow the rule of law,
or something interesting
thing called the internet
and that there's radically
new technological advances.
And when something like that
happens, those rough judgments
of this is better than that,
in other words, those new things,
first, we're terrible at
describing them before they come
and predict them, although
humans are very good
at a kind of one shot learning, so then
they can make judgements
- Uh-hm.
- [Lori] quite quickly.
Machines are not like that.
Moreover, these better-than judgements
that the machine might be
relying on, could, I think,
quite straightforwardly, be invalidated
because everything changes,
or deep things change--
- Yeah.
- [Lori] in all kinds of unexpected ways.
- Yeah.
- [Lori] That just seems to be...
And that's the real problem.
It's not that using machines for things
that we already have control over.
No, it's about trust--
- Yeah.
- [Lori] with entirely
new categories of events.
So, I was just sort of
deeply unclear on...
I mean that seems--
- Sure. Yeah.
- [Lori] like a nice thing, but that's not
for me--
- Yeah.
- [Lori] The real alignment problem.
- I think there's
fundamentally, as you say,
it is the nature of the
world that our world models
are constantly being invalidated
and we're constantly needing to revise
our sense of how things work,
or what the relevant categorical
distinctions even are.
I think your point is well-taken.
The area that, for me, feels relevant
to what you're describing
is the idea of uncertainty,
calibrated uncertainty in a model.
So, for example, the Uber
collision that I referenced,
part of what was going
on was that, as I said,
it didn't recognize what a jaywalker was.
It had this very brittle
category distinction
between pedestrian and cyclist
and so, if you read the
essentially the black box output
of what the system was thinking
in the five seconds before
the crash, it was like,
"Oh, that's a person.
"No, it's a cyclist 'cause
I can see the frame of...
"No, they're definitely walking.
"Their feet are on the...
"No, there's definitely a handle bar,"
and it's flickering between
these two categories.
I think that alone
should have been evidence
that the system didn't really
know what it was dealing with
and its category structure was inadequate.
- [Lori] Sorry, there's a
difference between uncertainty,
where you're not sure if it's
A, B, or C and unknown, okay,
which is a different kind of uncertainty
in probabilistic literature.
Then you haven't got, it's not
oh, is it A, is it B, is it C?
It's some other kind of
thing you can't classify
and that's the problem
I'm trying to target.
- Sure.
- So, I think it's different.
- Yeah.
So, I mean, there's work on this.
Tom Dietrich at University
of Oregon has what he calls,
open category learning.
So, there's this idea of how
do we do object classification
when one of the classifications
is, "I have no idea."
He did a project with the
EPA on, of all things,
identifying flies in a stream.
And it turns out that a lot of the stuff
that you catch in a net
if you put it in a stream,
is not any kind of fly at all.
(audience chuckling)
So, it ended up being that
they needed a very robust,
this isn't any of those things-mechanism.
So, that sort of work, I think,
speaks to what you're talking about.
Obviously I think we need to go further.
- I'm gonna help the MC.
- Question back here.
- Yep, hold on.
I'm coming your way.
Give me one second.
- [John] Just speak up.
- [Undergrad] Yeah, sure.
So, thanks for coming.
I really enjoyed this.
I was interested in studying
the alignment problem.
I'm an undergrad.
I was interested in learning
about it and researching it
and I couldn't do that at Yale.
There was no one doing that at Yale.
So, I had to go work for Stuart Russell
and Jacob Steinhardt.
I had to go to Berkeley, in other words.
I'm sure you love Berkeley.
I love Berkeley,
(all chuckling)
but how do we get more people
to care about these problems?
How do we get more people
at places like Yale
to be actually thinking
about this problem.
When people are building
bridges, no one's like,
oh, there's that one lab
over there thinking about
how to make sure the
bridge doesn't collapse,
but that's not the case in AI.
So, how do we actually
sort of get more people
to be researching this?
- I would say for my part,
I think growing the field is
very much one of the ambitions
that I had with the book.
As I said, one of the ambitions
was to help non-technical people
feel like they can become,
sort of have a working
fluency machine learning.
And the other of my main goals,
was to bring more people into the fold.
So, I'm encouraged that you're motivated.
And I think, partly, students
can exert gentle pressure
on their professors and say,
hey, I would really love
to do some kind of summer
project, or research project
in this direction.
Is there anything cool that you know about
that I could work on?
That's a way that, sort
of from the bottom up,
you can help just sort of grow
the mind-share in this area.
I'm impressed that you took
matters into your own hands.
I also think there might
be some collaborations,
cross-department, that are
not immediately obvious,
but you might find some kindred spirits
in neighboring departments.
We were talking earlier,
one of my closest friends
and closest collaborators, Tom Griffiths,
I met when I was a
Computer Science undergrad.
I was really frustrated
because I didn't feel
like anything that happening
in the CS Department
reflected my desire to learn
more about human cognition.
And I bumped into this guy
from the Cognitive Science
Department that said,
"Well, yeah, I basically use
the tools of computer science
"to think about people," and I was like,
"Oh, there's the guy!
"He was just in the other department."
So, that might also be possible.
- [Undergrad] Sure.
- [Male Student] Thanks again.
Thanks for this talk.
This was super interesting.
I also have been...
Speak up a little more?
I've also been really interested
in this alignment problem.
I actually think I'm gonna
be working on it next year.
For part-time research
doing this kind of thing.
Particularly I wanted to ask,
with respect to what you
were just talking about,
with cognitive science, where do you see,
'cause obviously this seems like kind of
the center of the bulls
eye of this is a CS problem
of some sort.
Where do you think the most
productive ways neuroscientists,
psychologists, cognitive
scientists and general philosophers
can interface with this
problem in a way that...
Where are the areas where
this is the most neglected
and where you think the
most work needs to be done
from that kind of top-down perspective?
- Yeah.
Really cool question.
So, ways that cognitive
science intersects here,
I mentioned temporal difference
learning and dopamine,
that's like one of the seminal things,
but there's still work to be
done there in neuroscience
that continues to this day.
I talked a little bit
about reward shaping.
So, there's this idea from
Andrew Ang and Stuart Russell,
it's called Policy Invariance
Under Reward Transformations.
It's like, what are the
different types of incentives
that you can use that don't result
in a different final policy of your agent?
That's really useful, I think,
not just in computer
science that's useful.
In thinking about
management, or economics.
The term alignment problem is borrowed
from the economics literature.
So, it's interesting.
There are cognitive
scientists like Falk Lieder
at Max Planck in Europe
who is using this idea
of reward shaping and policy invariance
to think about optimal
gamification strategies.
So, how you can break down a task
and assign points to each sub-task,
such that you can incentivize the person
to not procrastinate.
Very cool and that draws
directly on the computer science
of reward shaping.
I mentioned Montezuma's Revenge,
which is a game that's
very stingy with rewards.
It turns out a lot of the progress there
has been done through what's
called intrinsic motivation.
How can we make an agent
that's motivated internally,
rather than by external
points from the environment,
but by its own sense
of exploration and play
and novelty-seeking.
That was a case where computer scientists
like Mart Belmar, DeepMind and Mela,
Deepak Pathak who's at CMU,
they turned to
developmental psychologists,
people who study infant cognition,
people like Alison Gopnik,
Laura Schultz at MIT,
and said, what do you have for
us in terms of formal models
of novelty-seeking
behavior, or self-surprise
that we can plug into our
Montezuma's Revenge thing.
They plugged in this...
They sort of translated it into
reinforcement learning terms
and plugged this novelty
drive into the agent
and then suddenly
(fingers clicking)
it beats the game because
it's playing the game
the reason a human would,
which is to just see
what's on the other side of the door.
That turns out out to be
a more powerful driver
than just looking for external rewards
even if all you're measuring it by,
is its ability to
achieve external rewards.
So, that sort of work
has a lot of connection
to developmental cognition.
Very cool.
And then lastly, this
reward modeling work,
or the field of what's called
inverse reinforcement learning,
you're observing an agent making decisions
and you have to infer the
reward that they're optimizing.
Well, there's kind of a huge
problem here, which is that
we know pretty well that humans are not
perfect reward maximizers.
So, I think there's a lot
of really cool neuroscience,
cognitive science, psychology, economics,
behavioral economics work to be done,
giving a better theoretical framework
to the inverse reinforcement
learning community
about how to work backwards
from how a person,
how you observe a person behaving
and what inferences you're
kind of allowed to make
about what you think their values are.
I think that's a huge open question
and there's so many fields
that really touch that.
- Well, Let me just add
a little bit to that.
So, for many years, decades,
machine learning advanced
by just sort of crude abstractions
for how the mind might work,
the prefrontal neural network.
And things progressed
(John chuckles)
quite remarkably far just given
those types of abstractions.
But now it's progressed to the point
where we can take more
clues from human cognition
to inform artificial systems
as Brian was talking about.
But we're also at the point
where we can turn it
in the other direction
and we can develop models
for learning that can be used
as kind of a computational
laboratory for informing
our understanding for how the
brain works and this something
that's very much of interest
to the Wu Tsai Institute.
So, I think we're at this
kind of turning point
where it becomes mutually profitable
to study artificial intelligence
and human intelligence simultaneously.
- [Ted] Time for a final question.
- [Thalia] Hi, I'm Thalia.
I'm a Physics grad student.
My question is, I wanted to
pick up on the point you raised
about the kind of analogy
to nutritional labeling,
being clear about how an
algorithm was trained,
or what its objective function is.
So, I wonder about what you
see as the future of that?
How it could be built on, or expanded?
In particular, I'm imagining, you mention
what the goal was of Tinder and Facebook,
how they measured success.
And I imagine a user might
have a different experience
if that were labeled really
big at the top of Facebook,
when you're on there.
What's the goal we're trying
(Thalia chuckles)
to get out of you?
- Yeah.
- [Thalia] Now, they might not
wanna do that, but (chuckles)
I wonder what you think
might be the future
for legislation in that regard,
or other means of
incentivizing more transparency
about what is the machine
learning algorithm's goal
that you are interfacing with?
- Yeah.
I love this question.
So, I think this is a huge area
for some kind of regulation
down the line.
I don't know exactly what it
would be, but I'm completely
of a mind with what you're saying,
that we need something like that.
It's interesting to think
about the failures of policy
to provide meaningful consent, right?
If you go to any European website,
there's just a cookie pop-up
that you're like, okay, sure.
So, that's an example of it,
sort of disclosure done wrong
in a way that you just create
sort of consent fatigue
without any actual insight.
It's interesting to me, sites like Reddit,
this is not the most
impressive thing in the world,
but Reddit gives you
four different metrics
by which to sort the replies.
You can sort by new, you can sort by best,
you can sort by controversial.
And I think that is both,
there's both an aspect
of transparency there
and also an aspect of agency.
We're increasingly seeing
Instagram and others relenting
and saying, okay, if you
really want reverse chron,
you can have it.
Twitter does the same thing now.
I think
providing meaningful oversight,
or meaningful transparency in that way,
with something I would really like to see,
is a system where, let say on Twitter,
you could articulate your goals as a user.
You're like, I wanna learn
more about machine learning.
I wanna learn more about public policy.
I wanna look at my friends,
cute pets, or whatever
and you can have some range of goals.
Then for everything that
appears in your timeline,
it's incumbent to Twitter to
tell you which of your goals
it thinks that piece
of content will serve.
And you can maybe have
some feedback mechanism
where you say, no, that's
not an example of a cute pet,
or whatever.
That's not informing me about AI.
But that sort of puts
you in the driver's seat
and then it becomes up
to them to make the case
for how they're achieving your goals.
Something like that, I think,
is quite within the realm
of 2022 machine learning
and will become increasingly possible,
to just verbally articulate
what you wanna do
and have it sort of get the gist.
I wanna say one last thing about
the criminal justice space.
So, we've been talking about
these huge reward models,
language models
that have trillions of
parameters, et cetera.
A lot of what I think is some
of the most exciting work
happening in machine learning,
is not happening with large models,
but it's happening with
exhaustive search over
all possible simple models.
(people chattering)
There's a computer scientist
at Duke named Cynthia Rudin
who has done work on, both
in the medical context
and in the criminal justice context,
on creating optimal simple models
that are competitive
accuracy-wise with neural networks
and come with a sort of
certificate of optimality
that says, of all simple
models we can prove
that this is the best one.
For example, she has a model,
which I can probably tell you
the entire thing off the top of my head.
It's if you are under 20 and male,
predict that you'll be
rearrested, and you're arrested,
predict that you'll be
rearrested within two years,
or if you are under 23 and
you have two priors or more,
or if you have three priors, or more,
predict rearrest within two
years, otherwise predict
you will not be rearrested
within two years.
That is the entire model.
I've just told you the entire thing
and that is competitive for
accuracy against COMPAS,
which is this proprietary
close-source thing
that costs however much money
and no one knows what is really in it
and people write papers,
trying to reverse engineer it
because it's such a big deal.
In some ways I think it is now
inexcusable for a government
to use a proprietary tool
when they could use something
that fits into a single English
sentence because effectively
these models, in my view,
become extensions of the law.
If this is determining, or
at the very least, informing
whether you are released
when you're arrested or not,
pending trial, so you
haven't been tried yet,
that is, in my view, an
extension of the law.
There's the famous Lawrence
Lessig quote, "Code is law."
Well, I think we should demand
that the law be on the books
and legible to the people
that are beholden to the law.
So, to the degree that machine
learning models are the law,
then we should demand
that they're both legible
and publicly available.
I would support legislation for example,
that mandated the use of models like that
in those situations.
- [Ted] So, I think sadly we've
reached the end of our time,
but please join in thanking
Brian and also John
for moderating the speeches.
(audience applauding)
- Thank you.
(audience applauding)
(calming xylophone music)