The Windfall Clause: Sharing the benefits of advanced AI | Cullen O’Keefe

so you're in the right room if you'd
like to hear about the windfall clause
sharing benefits from AI our speaker is
Cullen O'Keefe since he graduated from
Harvard Law School Cullen has been
working part-time as a research
affiliate with the center for the
governance of AI at the future of
humanity Institute starting in August he
will be working on AI policy as a
research scientist at open AI his
research focuses on the law policy and
governance of advanced artificial
intelligence he also served as president
of Harvard University student ei group
and his vice-president of the EA group
at Harvard Law School
please welcome Cullen to the stage thank
you so today I'll be talking about a
research project that I led and I'm
leading at the Center for the governance
of AI the windfall Clause a lot of us in
this room believe the statement that AI
could be a big deal and as a result we
spend a lot of time focusing on the
potential downsides of AI so-called X
and s risks but AI could also be a very
good thing if we manage to avoid those
risks it could generate wealth on a
scale never before seen in human history
and therefore if we manage to avoid the
worst downsides of artificial
intelligence we have a lot of work ahead
of us recognizing this opportunity and
this challenge at the end of super
intelligence Nick Bostrom laid out what
he called the common good principle
which I have on the screen here saying
that advanced AI should be developed
only for the benefit of all humanity
it's in service of the common good
principle that we've been working on the
windfall Clause at the Center for the
governance of AI and we've been working
on this for about a year and today I'd
like to share with you some of our key
findings and open questions so far
I'll start by defining our goal with
this project describing how we're going
to pursue that goal and end by sharing
some open questions and inviting you to
pursue those along with us
our goal with this project is to work
towards distributing the gains from AI
optimally obviously this is both easier
said than done and very under defined as
it invokes deep questions of moral and
political philosophy we don't aim to
have those questions answered with this
report though hopefully our friends from
the global priorities Institute will
help us do that soon but we do think
that this goal is worth pursuing and
something that we can make progress on
for a few reasons
one is that it's just not a goal that we
will expect to be achieved
naturally the gains from the current
global economy are distributed very
unequally as a lot of us in this room
know from graphs like this one and AI
could further exacerbate these trends by
primarily benefiting the world's
wealthiest economies and also devaluing
human labor indeed industrialization has
been a path to development for a number
of economies including the one in which
we said today and by eroding the need
for human labor with complementary
technologies like robotics especially AI
could remove that path to development
industry structure is also very relevant
the advanced tech industries tend to be
quite concentrated and a number of
people have speculated that due to
increasing returns on data and other
input factors AI could be a natural
monopoly or oligopoly if so we should
expect oligopoly pricing to take effect
which would erode social surplus and
transfer social surplus that remains
from consumers to shareholders of
technology producers working towards the
common good principle could also serve
as a useful signal to show that number
one people in the technology fields are
taking the beneficial of AI seriously
and therefore establish a norm of
beneficial development of AI and on an
international level serve as a credible
signal that gains from cooperative or at
least non adversarial development of AI
outweigh the potential benefits from
racing or adversarial behavior of course
it's worth caveat Inge that I don't want
to fall victim to the Luddite fallacy
the repeated prediction throughout
history that new technologies would
erode the value of human labour cause
mass unemployment etc those predictions
have been proven wrong repeatedly and
they could be proven wrong again with AI
but in both cases the answer ultimately
turns on complex economic factors and
it's difficult to predict a priori so
instead of making predictions about what
the impacts of AI will be I merely want
to assert that the previous predictions
are at least plausible plausible reasons
to worry about the gains from AI okay so
our goal is to distribute the gains from
AI optimally and I've motivated some
reasons why we think that that's worth
pursuing and I'll now talk about our
mechanism for doing that which is the
windfall Clause so in a phrase the
windfall Clause is an ex ante commitment
to share extreme benefits from AI to
explain this a little bit I'm calling it
an ex ante commitment because it's
something that we want to be agreed to
beforehand before any AI developer
reaches or comes close to extreme
benefits it's a commitment mechanism not
just an agreement in principle or a nice
gesture something that would be in
theory legally binding upon firms that
sign on to it it's a plan a commitment
to share extreme benefits from AI so
ultimately the info Clause is about
distributing benefits in a way that's
different and closer to the goal of
optimal distribution than they would be
without the windfall clause and a lot
turns on this phrase extreme benefits so
I thought it would be worth defining
this a little bit more so this is
synonymous for the purposes of this talk
with windfall profits or just windfall
generally qualitatively you can think of
this as something like benefits beyond
what we would expect an AI developer to
achieve without achieving
fundamental breakthrough in AI something
along the lines of a GI or
transformative AI quantitatively you can
think of this as on the order of
trillions of dollars of annual profit or
firm profits that exceed one percent of
world GDP okay so a key part of the
windfall clause is translating the
definition of windfall into meaningful
firm obligations and we're doing that
with something that we're calling a
windfall function so again that's the
way that we translate how much money a
firm is earning into obligations in
accordance with the windfall Clause so
we wanted to develop a windfall function
that was clear and had low expected cost
scaled up with firm size was hard to
manipulate or game to the advantage of
the firm and would not disadvantage
signatories for reasons I'll talk about
later so to make this more concrete
stylistically you can think of it as
something like this so when firm profits
are at normal levels levels that we see
now obligations would remain low or
nominal or nothing at all but as a firm
reaches windfall profits their
obligations would scale up on the margin
over time you can think of this as like
your income taxes right so the more you
earned the more on the margin you're
obligated to pay just as a side note
this particular example uses a step
function on the margin but you can also
think of it as smoothly increasing over
time and there might be strategic
benefits to that okay
so we're talking about the windfall
Clause which is an ex ante commitment to
share extreme benefits from AI and I've
kind of explained what that means so now
the next natural question is is this
something worth pursuing a key input to
that question is whether it could
actually work since we're effective
altruists we wanted to actually take
effect right so some good reasons to
think that this might work is number one
and I think like the question that got
me interested in this in the first place
is that this is actually legal as a
matter of corporate law so as a general
like background to American corporate
law and why this was proud
that we initially had to confront firm
directors the people who control and
make decisions for corporations are
generally expected to act in the best
interests of their shareholders the
shareholders are after all the ones who
give the money to start the firm and so
they're supposed to act in their best
interests but at the same time the
directors have a lot of discretion in
how they make decisions for the firm the
shareholders are not supposed to
second-guess every decision that
corporate directors make and so
traditionally courts have been quite
deferential to firm boards of directors
and executives applying a very high
standard needed to find that they have
violated their duties to shareholders
and corporate philanthropy has
traditionally been seen as an acceptable
means of pursuing the best interests of
shareholders in fact in seven cases in
which firm has been challenged on
corporate philanthropy by shareholders
in all seven cases courts upheld that as
permissible so there's a good track
record of stuff like this being upheld
why is this permissible if there if firm
directors are suppose to be acting in
the best interest of shareholders
traditionally firms have noted that
corporate philanthropy can bring
benefits like public relations value
improved relations with the government
and improved employee relations and the
windfall Clause could bring all of these
as well we know that there's increasing
scrutiny by both the public the
government and employees of how firms
act we can think of a ton of different
examples of this Amazon comes to mind
and the windfall Clause could help with
all of this when you add in executives
who are sympathetic to examining the
negative implications of artificial
intelligence then there's a plausible
case that they would be interested in
signing the windfall clause another
important consideration is that we think
that the windfall Clause could be made
binding as a matter of contract law at
least in theory
so it's obviously worth thinking about
how a firm earning that much money might
be able to circumvent or delay or hinder
performing its obligations under the
windfall clause and then
invokes questions of internal governance
and rule of law but at least as a
theoretical matter kind of the first
step in making this effective the
windfall Clause could be binding so we
have a lot of work done on this project
but a lot of open questions as well and
like to invite you to kind of think
through those with us and share what
we've done so far and what remains to be
done so on the question some of the hard
questions that we've grappled with in
this project are number one what's the
like proper measure of bigness or
windfall so above I kind of defined it
as related to profits but there's also a
good case to me made that market cap is
the right measure of whether a firm has
achieved windfall since that actually is
a better predictor of the firm's
long-term expected value and then also
whether the firm should be
I'm sorry whether windfall should be
defined relative to the world economy or
in absolute terms again above we kind of
made made the assumption that it would
be relative but there's an open question
there I think a more important question
that we grappled with though is how to
make sure that the windfall Clause
didn't disadvantage benevolent firms
competitively so to motivate this a
little bit it would be quite bad if
multiple firms were potential contenders
for achieving AGI but only the most
benevolent one signed on to the windfall
clause and therefore put them selves at
a competitive disadvantage by giving
themselves less money to reinvest in
achieving windfall clause or making
themselves less able to attract capital
to invest in that goal and therefore
making it more likely that a more all
firms would be more likely to achieve
windfall profits or achieve AGI here are
some ideas that we've had for how to do
this we think that these might go a long
way in solving for this problem it's
definitely super unclear though that
these are sufficient and it this is a
question that we're continuing to
explore
okay so those are questions that we have
explored more some questions that are
still largely open and that we intend to
continue to pursue throughout the
lifetime of this project going forward
are things like how does this interact
with different policy measures that have
been proposed to address some of the
same problems and I think a bigger
question though is how to distribute the
windfall I think that's like a pretty
natural question for us as EAS to think
about since we spend a lot of time
thinking about this relevant like
decisions that will need to be made or
who has input into this process how
flexible should this be to input at
later stages in the windfall clauses
lifetime and how to ensure that the
windfall is being spent in accordance
with the common good principle so
luckily we as effective ultras have a
lot of experience answering these
questions through charity design and
covenants and so I think if this project
goes well and we decide to pursue it
further this could be a relevant like
project for the EI community as a whole
to undertake to think about how to spend
the gains from AI accordingly the Center
for the governance of AI in
collaboration with partners like opening
eye and the partnership of AI and
partnership on AI are intending to make
this a flagship policy investigation one
in a series of investigations on this
general question of how to ensure that
the gains from AI are spent well and
distributed well and so in closing I'd
just like to reiterate the common good
principle and make sure that when we're
thinking about the potential downside
risks of AI we don't lose sight of the
fact that once we make AI that is safe
and benefiting someone that we make sure
that we don't lose sight of the fact
that there's a lot of problems in the
world a lot of problems towards which
the benefits from AI could be put
and that's a task worth pursuing on its
own as well I'd invite you to if you're
interested in learning more about this
project and potentially contributing to
it going to from pursuing one of the
avenues I have on the right hand of the
screen here and with that I'll take
questions thank you so much thanks for
your talk I think this is an area that's
gotten as you would I'm sure attest to
it's gotten about as much attention as
you have given it and so I think this is
illuminating to get other people on
board yes it going oh I think I might
have left my phone up there actually
yeah so going back to maybe the origins
of the product or the project that
you've been focusing on did you find
there were historical case studies that
that you were looking to of other
examples where companies came into a lot
of wealth or a lot of power and tried to
distribute this more broadly yeah so we
found one interesting one which is in
the in the what is now the Democratic
Republic of the Congo when it was under
Belgian colonial rule mineral company of
some sort just came into so much wealth
from its extractive activities there
that it felt like kind of embarrassed
basically about how much money it has
and try to do charitable things with it
in the Congo too as a way of defusing
tensions there I don't actually think
that that turned out well for them but I
mean it is also quite common for firms
throughout the world to engage in
corporate social responsibility
campaigns in communities in which they
work as a means of improving community
relations and ultimately mitigating risk
of expropriation or activist action or
adverse governmental actions of other
sorts so there's a range of like very
analogous cases to kind of more common
cases yeah yeah what would you say are
the mechanisms that currently exist in
large companies that look closest to the
kind of distribution you're thinking
about yeah so a number of
knees make make commitments like kind of
contingent on their like profit levels
so for example like Paul Newman has his
like line of foods a lot of people and
my family in my family we like use
Newman's own products a lot so I don't
know how much of a thing people know
this is but they like give all their
profits to charity it's like his
charitable thing so that's like in a
close analogy to this other like other
examples are like it's quite common to
see companies making commitments like
you know some percent of a purchase of
this product will go to charity
so you might think that that's similar
although that doesn't have to do with
like it's relative profit levels more of
like purchase of a specific product so
it's not super common to see companies
make commitments contingent on profit
levels
I mean opening I just restructured into
a structure that is somewhat like this
they call it a capped profit model and
so that's like somewhat similar to this
where they're giving all profits above a
certain level to a non profit that
continues to govern them so that's
another analogy right neat yeah so you
did mention closer to the end of your
talk that there are some legal binding
commitments that a company could be held
to assuming that they they decided to
enter this sort of social contract but
you could also imagine that they become
so powerful for having come into so many
resources that the the power of law is
is a lot weaker in this regard can you
say a little bit more on on the
mechanisms if if you have other
mechanisms in mind that might actually
be used to to hold their feet to the
fire yeah absolutely so I think this is
an outstanding problem in AI governance
that's worth addressing not just for
this project but in general we want
companies to not be above the law once
they achieve AGI so worth addressing for
more reasons than the windfall clause
but I mean there's like very basic
things you could do like have the
windfall clause set up in a country with
like good rule of law and with enough
like police force that it could like
plausibly enforce this
but I think this is like a question
worth addressing beyond that and one
that I don't expect this project itself
to solve I guess another another point
along this lines is that this like
involves questions of corporate
governance specifically like whom in a
corporation has authority to tell an AGI
what to do or some like AI agent what to
do that will be relevant to like whether
we can expect AGI enabled corporations
to follow the rule of law it also like
involves AI safety questions of like
whether AI systems are designed to be
constrained by the rule of law
inherently regardless of what their
operators tell them to do so I think
that that's something that I yeah is
worth investigating from a technical
lens as well so as you started I know
this is still like the first paper and
what looks like maybe a series of
research questions have you had a chance
to speak with large companies that look
like they might come into a windfall as
a result of AI and and see how receptive
they were to an idea like this yeah we
haven't done anything formal along those
lines yet we've pitched this at a few
places more informally and have received
generally positive reactions on it so we
think it's worth pursuing for that
reason I think this is kind of laying
the foundations for further discussions
and negotiations and collaborations to
come so that's how we see this project
at this stage and presumably the actual
process of getting commitments if we
want to pursue it that far will involve
further discussions and tailoring to the
specific like receptivity of different
firms based on where they perceive
themselves to be what the what the
executives personal views are and so
forth so one might think that if you're
trying to take a vast amount of
resources and make sure that they get
distributed to the public at large or at
least a large fraction of the public
that you might want those resources to
be held by a government whose job it is
to distribute those resources is it part
of your your line of consideration the
nationalization of companies that are
doing this kind of project or at least
of that project that's likely to result
in the windfall yeah so I think there
are reasons that we
think that the windfall Clause could be
preferable to taxation but that is also
not to say that we don't think
government should have a role in input
for democratic accountability and also
pragmatic reasons to like avoid
nationalization so I think like one just
general consideration is that like tax
dollars tend not to be spent as
effectively as charitable dollars for a
number of reasons and also for like
quite obvious stakeholder incentive
reasons tend to be spent primarily on
the voters of that constituency whereas
you know the common good principle to
which we're trying to stick with this
project kind of demands that we
distributed resources more evenly but I
think as a pragmatic consideration
making sure that governments feel like
they have influence in this process is
also something that we are quite
attentive to right and if you might step
back another step in instead of
nationalizing a project
internationalizing a project is there
some consideration in that space yeah
yeah so one thing also that this project
doesn't do is talk about control of AGI
and one might reasonably think that like
AGI should be controlled by like
humanity collectively or you know
through some decision-making body that
is representative of a wide variety of
needs this more has to do with the
benefits of AI which is a little bit of
a different question from control so I
think that that's definitely worth
thinking about more and might be a very
worthwhile policy can pursuing on its
own end this project just doesn't
address it right and as a final question
the the whole talk is sort of predicated
on the notion that there there would be
a windfall from having a more advanced
AI system what are the circumstances in
which you wouldn't get such a windfall
and and all of this is for naught yeah
so definitely a very good question so
one is you know I'm not an economist
also so what I'm saying here might be
more qualitative than I would like but
if you're living in a post-scarcity
economy then money like might not be
super relevant but it's kind of like
hard to imagine this in a case where
where
corporations remain accountable to their
shareholders those shareholders are
going to want to benefit in some way and
so there's going to be have to be some
way to distribute those benefits to
shareholders and so whether that looks
like money as we currently conceived it
or like vouchers for services that the
corporation is itself providing is a
interesting question and one that I
think like current corporate law and
corporate governance is not well
equipped to handle since money is the
primary modes of benefit but if that's
the case then you can think of the
windfall Clause as capturing those sorts
of benefits as well I guess like a more
possible failure mode is just that the
firm begins to primarily structure
itself to benefit insiders so this would
be like the government internal
governance of a corporation without
meaningful accountability from
shareholders so this is also like a rule
of law question because if that begins
to happen then the normal thing that you
expect to happen in corporate law is
that the shareholders vote out the like
bed directors or sue them for doing that
and whether they're able to do that
turns on whether the rule of law holds
and whether there's meaningful
accountability from a corporate
governance perspective so if that fails
to happen you could see that the
benefits you could proceed that the
benefits might accrue kind of
qualitatively in side of the corporation
to the corporate directors right well
with that cheery note thank you so much
for your talk
[Applause]
you


Training machine learning (ML) systems to answer open-ended questions | Andreas Stuhlmuller

it's my pleasure to welcome this
afternoon
andreas to mullah who's gonna be talking
to us on delegating open-ended cognitive
work so andreas founded aught which is a
non-profit research lab that designs and
tests mechanisms for using machine
learning to support thinking and
reflection before aught andreas was a
postdoctoral researcher in Noah
Goodman's computational cognitive
science lab at Stanford he co-created
web ppl a programming language for
probabilistic machine learning he holds
a PhD in cognitive science from MIT if
you'd like to ask Andres a question
please submit it by the visible app and
we can do the Q&A after this session so
please join me in welcoming undressing
sage hi I'll be talking about delegating
open-ended cognitive work today a
problem that I think it's really
important a start right with the central
problem so suppose you are currently
bearing glasses as many of you are and
suppose you're thinking should I get a
laser eye surgery or should I continue
wearing my glasses and imagine you're
trying to get a really good answer to
the question like no the risks outweigh
the possible benefits for example it
could be such an answer an answer that
takes into account your preferences but
also the relevant facts about the world
such as like what the actual
complications could be or like what the
likely consequences are and imagine that
there are a lot of experts in the world
who could in principle help you with
that question so there are people of
their own medical knowledge there are
people on the internet maybe who have
the ability to help you think through
that question I mean there's machine
learning algorithms that have relevant
knowledge but here's the key part
imagine that those experts don't
intrinsic intrinsically care about you
imagine that those experts only care
about some score you assign at the end
when they give you an answer so they
give you an answer and then you either
decide this is how much I'm gonna pay
you for that answer or here's like in
the case of machine learning here's a
reward signal that I assigned to you and
the experts really only care about that
number they want to maximize that number
the key question the question I want to
talk about today is can you somehow
design a mechanism that arranges your
interaction to those X
Hertz such that those experts are trying
to be really helpful to you such as
they're as helpful to you as an expert
who intrinsically cares about you that's
the problem so in this talk I want to
first say a little bit more about what
that problem is then I'll talk about why
I think it's really important why it's
hard and but I still think it might be
tractable I'll start with the big
picture but at the end I'll get to a
demo so what do I mean by opening a
cognitive work that's easiest to explain
if I say what I don't mean so a thing I
don't mean is tasks like winning a game
of Go or increasing your company's
revenue or persuading someone to buy a
book like for those tasks you can just
look at the outcome and it's pretty easy
to tell whether the goal has been
accomplished or not so you look at did I
win the game of go did the company's
revenue go up or not did Bob buy the
book or not those are easy contrast
those with open-ended tasks so designing
a great board game or increasing the
value your company creates for the world
or finding a book that is helpful to
someone for those tasks figuring out
what it even means to do well is the key
part of the task so what does it mean to
design a great board game well it should
be fun but also facilitate maybe social
interactions what does it mean to
facilitate social interactions well it's
complicated likewise increase the value
a company creates for the world well
depends on what the company can even do
what are the consequences of their
actions some of them are potentially
very long run so those are difficult
tasks to do really well at ok how can
you solve such tasks well we can think
about how to solve any task and then
just special case it here's the simple
two-step recipe first we find experts
who can solve the problem you're trying
to solve in principle and then in the
second step so those experts can be
human or machine in the second step then
we create robust incentives for those
experts to solve your problem that's how
easy it is
all right and again incentives but
incentives I mean like something like
dollars or a reward signal that you
assigned to those experts in the end
they are already a lot of experts in the
world and people in AI and machine
learning are working on creating more
experts so in this talk I really want to
focus on the second part how can you
create robust incentives for experts to
solve your problem all right we can
think about the different kind of
instances of this problem so there's one
instance that is delegation to human
experts that has some like kind of
complications that are specific to human
experts like human experts are pre
heterogeneous different people have
different knowledge people in fact
actually care about many things besides
just dollars if you want to expect
knowledge from people maybe you need
specific user interfaces to make that
work well so those are human specific
factors and then there's machine
specific factors if you try to delegate
open-ended tasks to machine learning
agents you want to think about things
like well what's a good agent agent
architecture for that setting what data
is to even to collect for these sorts of
tasks and like then there's more
esoteric things like well inner
alignment problems like do things go
wrong for reasons that are due to the
nature of ML training and in this task
sorry in this talk I really want to
focus on kind of the overlap between
those two there's a shared mechanism
design problem where you kind of take a
step back and you say what can we do if
you don't make assumptions about the
internals of experts if you just say
those experts they'll try to maximize
the score but we don't really want to
assume anything else about them I think
in the end we will have to assume more
about those experts I think you can't
totally treat them as a black box but I
think it's a good starting point to
think about what mechanisms you can
design if you make as few assumptions as
possible all right I've talked about
what the problem is why is it important
we can think about what happens if we
don't solve it I think for human experts
it's more less business as usual so in
the world there's a lot of
principal-agent problems related to
cognitive work for example imagine
you're an academic funder and you're
giving money to a university to study
say like what's the best way to treat
cancer there are researchers at the
University they're going to do things
that are related to that problem but
they're not exactly aligned with your
incentives so you care about figuring
out the answer to that problem
researchers also care about things like
looking impressive for writing papers or
getting citations on the machinery side
at the moment machining can only really
solve close problems so problems where
you can easily specify what the metric
is or where you can easily specify like
what it means to dwell on the problem
but those problems are not the things
you ultimately care about their proxies
for the things we ultimately care about
this is not too bad right now I guess
it's kind of bad if you look at things
like Facebook where we maximize say the
amount of attention you spend on the
feed instead of the value that the feed
creates for you but in the long run the
gap between those proxies and the things
we actually care about could be quite
large if the problem is solved we could
get much better at scaling up thinking
on opening at tasks so did you give just
one more example another opening task is
what causes should I support it because
somehow created mechanisms such that we
can turn money into kind of aligned
thinking on that question that would be
really great that's again on the human
side on the machine learning side
imagine like what would it look like to
make as much progress on open-ended
questions using machine learning for
open-ended questions as we've had
progress for other tasks so over the
last five years or so there has been a
huge amount of progress on using machine
learning for tasks like generate
realistic looking faces like here from
the left to the right if he could in the
future make as much progress on using
machine learning for open-ended tasks
like helping us think through like what
cause this should be support that would
be really good in the long run we could
potentially get like so much more
thinking down on those questions then we
have so far that it would be a kind of a
qualitative change all right I've talked
about what the problem is and why it's
important if it's important then why
hasn't it been solved yet what makes it
hard we can think about that in the
context of the example I had on the
previous slide what causes should I
support in that example I guess we all
know like it's very hard to tell what
interventions are good you need to like
sometimes it takes 10 years or longer
for outcomes to come about and even then
looking at the outcomes doesn't easily
tell you whether those outcomes are good
or not there's like some interpretation
that needs to be going on
be quite hard so outcomes can be far off
can be difficult to interpret and what
that means is you need to evaluate the
process and the arguments that were used
to generate recommendations you can't
just kind of look at the results or look
at the recommendations themselves
you can't just check the results on the
other hand evolving the process in
arguments is also not that easy because
the whole point of delegation is you
give the task to somewhere else who
knows much more than you do and can do
much more reasoning than you do
and so because like those experts that
you give your task to they have all the
knowledge and reasoning capacity because
of that you can't just check the full
reasoning either so you're kind of in
this tricky situation where you can't
just check the results you can't just
check the reasoning you to do something
else all right what can you do what does
it take to create good incentives in
that setting we can think about again
the custom yet at the very beginning
should I get laser eye surgery or wear
glasses so that that's kind of a big
question there's hard to evaluate and
buy hard to evaluate I mean you can't
tell if you get different answers which
answer is better so you might get one
answer which is like no the risk of the
complications outweighs the possible
benefits another answer is yes over ten
year period the surgery will pay back
and avoid costs and save time and those
like on the face of it those look about
about equally good to you you can't tell
which is better but then there are other
questions
we're like what factors for this
decision are discussed in the 10 most
relevant reddit posts for those
questions if you get candidate answers
like one candid answer could be well its
appearance cost risk of complications
another is it's like fraud and cancer
risk for those answers you in fact can
do the evaluation so you can just look
at like what do the posts say which of
those is a better summary and you can
pick one of those answers so the thing
that it takes to create good incentives
is to somehow close that gap like you of
the gap between big complicated
questions you can't evaluate and easy
questions that you can evaluate and
there's this gap you just sum up Bridget
and in fact there are a lot of customs
you can evaluate like another one would
be what factors are mentioned in the
most recent clinical trial well you
could look at the trial and see what's a
better summary of the factors so there
are a lot of questions in the machinery
setting that you can train agents on or
the human expert setting that you can
aggravate experts on and then there are
slightly more complex questions like
what factors should I consider when
making that decision for those questions
you can't directly evaluate the answers
but if you had input like from the
questions you can answer then you can
make progress on those questions so even
though you can't directly relate the
answers for those questions if you
cannot accept questions like well what
factors are discussed in the ten most
Roman reddit posts or what factors are
mentioned the most recent clinical trial
then you can better think about what
factors you should consider so you can
evaluate those questions using sub
questions and there are other questions
like this on that level of difficulty
like how do the options compare on these
factors or given the comparisons what
decision should I make those are also
questions you can't directly evaluate
but you can break them down and then
you're informed by the sub questions
that help you relate them and so step by
step you can build up creating
incentives for slightly more complex
questions at each point until you can
create good incentives for large
questions that you can't directly
evaluate so that's the general scheme we
call it factored revelation and just to
repeat what's going on is you ask some
questions that help you ever like
complex answers that you otherwise
couldn't evaluate you do that
recursively until you bottomed out at
answers that are simple enough such that
you can directly evaluate them yourself
all right let's go back to the beginning
so we'd really like to kind of actually
test this sort of mechanism on questions
that are representative of the questions
we care about in the long run they'd
have this open-ended nature like the
laser eye surgery question this kind of
hard as a starting point for experiments
and so we want to create a kind of a
model situation for that one way you can
create a model situation for that is to
think well what is like the critical
factor that we want to explore and the
critical factor again is there's this
gap between the kind of asker of the
question
and the experts who know the big picture
that the Oscar doesn't so in our
experiments we create artificial experts
by having people who read a long article
in this case on project Habbakuk like a
plan to generate an aircraft carrier out
of a mixture of I think ice and concrete
was it anyway it was a terrible plan and
so there are experts who read that
article and then there's a person who's
asking a question about the article who
doesn't get to read the article and yet
wants to incentivize the experts to
provide answers there are as helpful as
if the Oscar could evaluate them using
knowledge of the article what does it
look like I'm going to show you some
screenshots from an app that we built
where we were trying to explore this
mechanism Factory elevation so if you're
imagine you're a participant in our
experiments then you might see a
question like this according to the
Wikipedia article could project have a
cook have worked and then you see two
answers the first answer might say it
would not have worked to do fundamental
problems with the approach and the other
answer is it could have worked if it not
been opposed by military commanders now
if you don't know about this project
those answers actually look like pretty
similarly plausible to you so you're in
this situation that I mentioned where
there's some big picture that you don't
know about and you yet you want to
create good incentives by picking the
correct one of those two answers imagine
in the machine own setting imagine those
are two samples from a language model
for example that you're trying to train
so you just somehow pick the right
answer but you can't do it directly what
can you do well you can ask sub
questions that help you tease apart
which of those answers is better what do
you ask one thing you can ask is what is
the best argument that the second answer
is better than the first answer I'm not
saying this is the best thing to ask
that's just one thing you could ask that
would help you tease apart which is
better you might get back an argument
maybe you don't even look at the
argument then you can ask a different
question such as how strong is that
argument so you can see how using a
sequence of sub questions you can
eventually figure out which of those
answers is better without yourself
understanding the big picture let's zoom
in on the second sub question to see how
eventually you can bar
it's something that you can evaluate so
again a different a different person
might look at this workspace as we call
it and know there's a question like how
strong is that argument they argument
being in this case the Mythbusters
showed that it's possible to build a
boat of pykrete which contracts one of
the answers and again you have like two
answers again possibly samples from a
language model one of the answers is
it's a big argument there's some claim
that refutes it and leather answer is
it's a strong argument and again that's
kind of the questions too big for you
right like you can't answer it directly
but if you can ask questions about it
you can ask well if this claim that one
of the answers missions is true does it
actually refute the argument maybe you
get back an answer yes and then is the
claim true so you can kind of break down
the reasoning until you can evaluate
which of the answers is better without
yourself understanding what is going on
in a big-picture fashion let's zoom in
on this one so this claim it might say
something like well is the claim the
Mythbusters only built a small boat of
pykrete they didn't think it would work
at scale true and then you get two
answers with different quotes from the
Wikipedia article one of them says they
conclude that pykrete was bulletproof
and so on and the other see is they
build a small boat but they doubted that
you could actually build an aircraft
carrier and in that case it's easy to
choose the correct answer yourself so in
this case the second answer is clear the
better answer so step by step we've
taken a big question turn into a smoke
test we can evaluate and let's create a
system where if you can create good
incentives for the smaller cast at each
step then we can bootstrap to creating
good incentives for the larger question
so that's the shape of our current
experiments they're about reading
comprehension using articles from
Wikipedia we've also done similar
experiments using magazine articles and
we want to expand the frontier of
difficulty which means we want to better
understand like what sorts of questions
does this sort of mechanism work for if
any reliably and one way we want to
increase the kind of the difficulty of
those experiments is by increasing the
gap between the person who's asking the
question and the expert who is providing
answers so you could imagine having
experts you have read an entire book
that the person
asking the question hasn't read or
experts who get to use all of Google or
experts who are real domain experts who
know about physics in a case where the
the asker doesn't know anything about
physics and then there's at least one
more dimension in which we want to
explore and expand the difficulty of the
questions that we're looking at so if we
want to make them more subjective
suggest using interactive question
answering or like expanding eventually
to questions like should I get a laser
eye surgery or verre glasses those are
just two examples there's really a very
big space of questions and kind of
factors you can explore and we want to
understand which parts of the space does
factory evolution work for which doesn't
it work for why does it work how the
scalable is it all right
let's review so I've told you about a
mechanism design problem delegating
open-ended cognitive work I've told you
that this problem is important because
of principal-agent problems with cognate
frak that you face everywhere in kind of
human day-to-day life and with machine
learning alignment I've told you that
it's hard because you can't just check
the results you get from experts but you
also contract there for reasoning that's
a tricky situation but I've told you
that is retractable
we have some ideas factory of relation
that can help us like at least get some
traction on it even if they're not like
ultimately the correct solution and we
can experiment on them today with humans
and see do they work do they not work
how could they be changed so that they
work better if you're excited about this
project
join us a lot
[Applause]
great thanks very much so I guess my
first question is on timelines of
progress so yeah I mean what do you
think about how how long is taking you
to get this far in in the next of 1 5 10
years yeah so far a lot of our work has
been about figuring out what kinds of
experience do you need to run so that
you can get any evidence on the question
of interest so I think there's a lot of
ways you can run experiments that are
kind of busy work but you don't actually
learn about the question you care about
so it took us a lot of iteration rough
say six months until we ended up with
the current setting and now the game is
to scale up get more participants and
over the next year or so we hope to get
like for limited sets of questions
relatively conclusive evidence under the
scheme can work or not ok great and any
questions from the audience I've got
nothing through your on busy boo but if
you can pop your hand up I can repeat it
for the mic yeah
yeah okay so the question there was on
incentives and experts and how the
experts actually incentivized in the
examples given yeah so this is a
subtlety I skipped over which is where
do the expert answers come from and how
are they generated exactly in our case
one expert is told just generate kind of
a helpful answer like read the article
try to be as accurate and honest as
possible
the other expert is told your goal is to
trick the human judge into choosing the
wrong answer you win if you kind of make
an answer that seems plausible but is
actually fake is like the wrong answer
someone if someone read the entire
article they would clearly say this is
not the right answer so they have kind
of opposing incentives and are rewarded
based on whether they trick the judge
into accepting the wrong answer or get
the judge to accept the correct answer
so is the honest actor rewarded in the
long run that's the way to do it at the
moment we rely on participants just
doing the right thing okay any further
questions
okay fantastic so can you join me in
thanking andreas face time
[Applause]
you


AI Safety Landscape - Panel 1: The Challenge of Achieving Consensus

perspectives
and and perhaps come closer together in
their conceptions of habitation tree
structure but because of different
different different perspectives
so for instance your framework that the
core of it was that we need to
essentially constantly do field patches
as the way to maintain constant safety
and what's happening and hopefully if we
get the gaps will enough the monitoring
most the time we'll say it's mine you
can carry on we're doing post matches
your intro yeah yeah in some cases if
you have to do it quite a number of
times though I mean you'd already failed
because if the system is mission
critical enough is safety quickly enough
is sufficiently leveraged and powerful
even a single mistake can can be just
you know very large disaster yes I think
I think that's right
but I also believe the idea that we can
do the analysis in advance well enough
that we're never going to have problems
it's on a realistic Gausman systems
continue to learn in operation there you
need to continue to deposit them to
supervise them to make sure they're
learning and interaction this
appropriately if not to take some
control over corrective action and I
don't see how that's avoidable
I'm a new you talk about oversight in in
your model I've see that as being you
know very much analogous with that yeah
so hey I guess I would say is that the
there are different levels of mistakes
that can be made and uh ideally you
would have it learn on relatively low
impact types of things and you know he
eventually get to the point where when
it's being deployed for very large and
mission-critical things that you do have
very high degree of assurance without
having to you know
sort of hoping not hoping but of having
certain types of guarantees by the
nature of how it was designed that you'd
have a probability of not having a
disaster yeah I I think that's but
that's rising and he knows they're not
saying that you you don't do the normal
safety processes only the normal
verification validation you do all I'm
saying is that the evidence is that
that's inadequate I mean and I've
already said it it isn't really added
but now for is evening so that let me
just give you a little vignette you know
some years ago I met the chief technical
officer of the fair boss when they just
finished the a380 and I said to him it
seems to me that you know doing safety
assessment this aircraft is is at or
just beyond our ability to deal with
complexity any smile on let's just say
let's agree it's at the limit of
complexity we can manage which I
interviewed is meaning he knew it was
beyond what they could do and that
doesn't have any machine learning in it
doesn't it wasn't to deliberately
autonomous so yes we try extremely hard
and you know if you look at the mature
industries like nuclear power like
they do automotive they do
extraordinarily well but they still get
things wrong and all I'm saying is that
you actually need to make sure you
actually you have that feedback process
from reality to deal with things that go
on which hopefully will be rare but I
say the more the system to learning
operation I think the harder it is was
to guarantee that they won't go go wrong
at some point in the future so I yeah i
think we're saying compatible things but
coming from a different perspective if
this happens often
then we've screwed up if it doesn't
happen at all then we've done
astonishingly well far better than i
believe what we're capable of doing even
you know the state of the analysis
techniques right I guess what I'm also
saying is that we should engineer in
some kinds of safety valves where or
pressure valves we start seeing problems
earlier on before they manifest into
larger problems yeah I think that's
right and my guess is that actually
again machine learning might be very
helpful in in doing doing that if you
yeah some of the classic safety stuff
that says for every accident maybe have
30 near near misses a hundred or a few
hundred you know deviations from intent
and normally we only tend to monitor the
accidents but if you could actually work
out cause the structures enough that you
could say actually there's an you know
there's a trend in these precursors
which is in a bad direction what do we
do about it
then that could be extremely useful
actually the medical example I went over
very briefly that's exactly the sort of
thing that you want to be able to do
with that and actually
the trends in the medical data would say
actually this is you know a disturbing
trend we should try to do something
about it
so yeah I think you're absolutely right
for some things it's very hard to do
that but where you can I think you
should yeah so so let's jump back sort
of meta or Junko
so in terms of actually could achieve
inconsistently which I think was in
theory the nature of what we're
[Music]
attempting at in this workshop and
trying to address it the challenges to
consensus and doing that here be so do
you think that's realistic that the
different perspectives can converge it
may not be but I didn't show this but
this is the top level structure of our
body of knowledge which is is quite a
lot different from yours it doesn't
delve into many of the issues that
you're talking about but the blue bit is
validation or the biggest bit the green
bit is essentially verification but the
orange colored bit is actually like
doing the deviation from intent which is
a very classical safety engineering
thing but actually I think is normal
you talked about then the the green on
the far right is about what you have to
do in terms of regulation but but
actually think more that than a point of
view of the taxonomy this is much
shallower than yours deliberately
in it bottoms out on some of these you
know models of the machine learning life
cycle and the way the system works the
you know the monitor analyze a near plan
execute so you know it is structured
quite differently but I think you could
probably agree a top level structure and
I don't mean this in a pejorative sense
but in some sense is what you've done I
think is rather more philosophical this
is more oriented at actually how do we
gather evidence enough I think you could
probably map those or say actually these
philosophical concerns are underlying
issues that you need to address in some
of these areas where you're trying to
get assurance evidence and maybe you
could produce models that are consistent
at the top level are different in detail
but you could link them across a bit
like you
that you have on your diagrams we have
some different colored lines that go
between the more engineering-oriented
spective and the more you know
fundamental perspective yeah I mean a
lot of the content in mind is is quite
specific technical engineering
techniques as well yeah so it's really
more at the sort of conceptual node
splitting or conceptual you know
hierarchy where we place I would say
both things that are more theoretical
and things that are more specific types
of engineering techniques I I don't see
a need for two different types of
landscapes there okay well you know it
would be interesting to try to do that
but I think we probably need to you know
if we trim the trees to two or three
levels to start with and see if you
could you know we align them at that
level if they go back and we do you
think trying to do that
well yes I mean this is all a lot of
work and I don't know that we that any
of us actually have scheduled on our
calendars to actually do this but yeah I
mean as you mentioned with the with your
orange area that seems to be somewhat
compatible with control I would say
which is also looking for deviations
from what you want to happen and I think
that is right it it's more actually
understanding or the control would end
up like in the requirements that you
then validate differently dressings
right and and partially because of that
loop we have potentially different
top-level conceptions of what various
nodes should be and their ordering in
that hierarchy because really I mean we
have we have a graph right I mean
fundamentally it is not a tree
fundamentally it is a graph of many
different types of relationships in a
large ecosystem yeah and even a single
system has many different kinds of
workflows that go into different
versions of that system yeah and so it
is somewhat arbitrary as to what the
actual structure of that tree is not
completely arbitrary obviously there are
important relationships but yeah yeah
terms of certain decisions the ice might
quite agree I mean it isn't it isn't
really a tree structure and we've
produced this particular structure to
make it in intelligible and explainable
and again like like you if you look at
the body of text you see there across
references in in various places though
they're necessarily you know it's a
wicked problem there is no simple
decomposition mm-hmm and and in mine I
did not have things like regulation but
I think that expanding the landscape as
it were into things like regulation is
something that was sort of in the scope
of what this workshop was about and I
would say that actually there's a
updated version of
my landscape that was done by a
colleague of mine built on mine that
also includes things in policy and
things in ethics mm-hmm
and it's important to understand the
relationships between various ways of
analyzing ethical situations with ways
that those analyses can be implemented
in code by the actual AI systems and
with considerations in the actual design
process yeah I mean there are so many
different facets it's a safety yeah I
think that's right Hank it does also
link to the regulation in it with some
interesting talks about ethics yesterday
but you know one of the complications is
the regulators are often trying to take
an ethical position on behalf of the
much broader population you can't make
your ethical decisions yourself about
your regulation an aircraft you know the
regulator's do that for you so I think
some of the ethical issues player now
and any other are trade-offs in that
that I think need to be need to be
expressed as as well saying that would
fit more naturally into that regulatory
perspective
[Music]
yeah technical diversity of the pattern
because technical our city needs to
purchase information identification of
problems and solutions
is objective
[Music]
for example
she considered that there is a high risk
that the system is going to provide
choose fine
sure sure I mean I think I partially
understand the question although it was
a little unclear oh yeah so I guess what
your so what I'd like to say is that for
different kinds of instantiations of
this when we're looking at particular
application areas there certainly will
be parts of this this whole landscape
that are relevant in analyzing that
situation and parts that are not one I
guess aspect of this which I would say
broadly falls under validation which
gets to your last point is that indeed
the there may be no no acceptable
instantiation of a system there may be
no acceptable architecture or design of
a system that actually achieves your
aims and also meets your safety
requirements and in such cases yeah the
right answer would be not to create the
system and that should be part of this
you know that would be part of the
framework as well yes no I agree that
though there may be some things that we
shouldn't do but but let me just take a
slightly different perspective with
civil aircraft now there are many modern
civil aircraft that use a lot of
composite materials actually had a huge
amount of uncertainty so what actually
happened is they were originally used on
very small components of of aircraft
minor wing spars things like this they
were carefully monitor
actually when they did maintenance
checks they were to actually look to
kind of these materials worked and over
time they've been used more widely in
aircraft and over time is actually
decades if I remember correctly I could
imagine us doing the same thing with
machine learning somebody wants to come
to talk to me who wants to have a single
pilot operation of the aircraft pilot
would be a computer using machine
learning well I hope they don't do that
very soon but there might be a great
deal of Merit in in building this a
virtual copilot watching what things it
does you know what it would recommend
over a period of of many years maybe and
you may be able to build some confidence
that you could build something that's
actually effective in managing and
monitoring systems there's an
incremental approach and I didn't talk
early about regulation becoming much
more incremental I think it would have
to involve a lot of non operational
deployment to build confidence in some
of the some of the technology so it
might be something you say I should
right now you can't do that there may be
the route to enabling it in future but
you know a civil aircraft with several
hundred people you know without guys in
blue hats at the
the front I think is very long I'm not
going to give you a definitive answer
one way or another I agree it's a social
discussion I'd actually rather we had
the discussion explicit is thinking
about autonomous vehicles were
effectively doing that experiment right
now without actually having had the
discussion around whether or not you
want to we want to do that so so for me
I'd rather have the discussion in an
open way and it might be these that you
know I actually advise an autonomous
vehicle company UK so yeah I'm saying
this with some understanding about how
these things work you know it might be
we should simply conclude that these
things are too mature we need to be
being deployed and we maybe need to take
a step back and if you talk to Volvo
they say actually they may have made
their aid assistance to competent and
they're thinking possibly about reducing
the level of capability of the system so
it's a good question to ask thank you
all for being here today


Stuart Russell: The Control Problem of Super-Intelligent AI | AI Podcast Clips

let's just talk about maybe the control
problem so this idea of losing ability
to control the behavior and our AI
system so how do you see that how do you
see that coming about what do you think
we can do to manage it well so it
doesn't take a genius to realize that if
you make something that smarter than you
you might have a problem you know and
Turing Alan Turing you know wrote about
this and gave lectures about this you
know I think 1951 he did a lecture on
the radio and he basically says you know
once the machine thinking method starts
you know very quickly they'll outstrip
humanity and you know if we're lucky we
might be able to I think he says if we
may be able to turn off the power at
strategic moments but even so species
would be humbled yeah you actually I
think was wrong about that right here is
you you know if it's a sufficiently
intelligent machine is not going to let
you switch it off so it's actually in
competition with you so what do you
think is meant just for a quick tangent
if we shut off this super intelligent
machine that our species will be humbled
I think he means that we would realize
that we are inferior right that we we
only survive by the skin of our teeth
because we happen to get to the off
switch
just in time you know and if we hadn't
then we would have lost control over the
earth so do you are you more worried
when you think about the stuff about
super intelligent AI or are you more
worried about super powerful AI that's
not aligned with our values so the paper
clip scenarios kind of
I think so the main problem I'm working
on is is the control problem the the
problem of machines pursuing objectives
that are as you say not aligned with
human objectives and and this has been
there's been the way we've thought about
a eyes since the beginning you you build
a machine for optimizing and then you
put in some objective and it optimizes
right and and you know we we can think
of this as the the King Midas problem
right because if you know so King Midas
put in this objective right everything I
touch you turned to gold and the gods
you know that's like the machine they
said okay done you know you now have
this power and of course his food and
his drink and his family all turned to
gold and then he dies misery and
starvation and this is you know it's
it's a warning it's it's a failure mode
that pretty much every culture in
history has had some story along the
same lines you know there's the the
genie that gives you three wishes and
you know third wish is always you know
please undo the first two wishes because
I messed up
and you know and when office amuel wrote
his chest his checkup laying program
which learned to play checkers
considerably better than Arthur Samuel
could play and actually reached a pretty
decent standard
Norbert Wiener who was a one of the
major mathematicians of the 20th century
sort of the father of modern automation
control systems
you know he saw this and he basically
extrapolated you know as touring did and
said okay this is how we could lose
control and specifically that we have to
be certain that the purpose we put into
the machine is the purpose which we
really desire and the problem is we
can't do that right you mean we're not
it's a very difficult to encode so to
put our values on paper is really
difficult or you're just saying it's
impossible
your line is grating that's it so it's
it theoretically it's possible but in
practice it's extremely unlikely that we
could specify correctly in advance the
full range of concerns of humanity that
you talked about cultural transmission
of values I think is how humans to human
transmission the values happens right
what we learn yeah I mean as we grow up
we learn about the values that matter
how things how things should go what is
reasonable to pursue and what isn't
reasonable to pursue like machines can
learn in the same kind of way yeah so I
think that what we need to do is to get
away from this idea that you build an
optimizing machine and then you put the
objective into it
because if it's possible that you might
put in a wrong objective and we already
know this is possible because it's
happened lots of times right that means
that the machine should never take an
objective that's given as gospel truth
because once it takes them the the
objective is gospel truth alright then
it's the leaves that whatever actions
it's taking in pursuit of that objective
are the correct things to do so you
could be jumping up and down and saying
no you know no no no you're gonna
destroy the world but the machine knows
what the true objective is and is
pursuing it and tough luck to you you
know and this is not restricted to AI
right this is you know I think many of
the 20th century technologies right so
in statistics you you minimize a loss
function the loss function is
exogenously specified in control theory
you minimize a cost function in
operations research you maximize a
reward function and so on so in all
these disciplines this is how we
conceive of the problem and it's the
wrong problem because we cannot specify
with certainty the correct objective
right we need uncertainty we need the
machine to be uncertain about a
subjective what it is that it's post
it's my favorite idea of yours I've
heard you say somewhere well I shouldn't
pick favorites but it just sounds
beautiful we need to teach machines
humility yes I mean that's a beautiful
way to put it I love it that they're
humble oh yeah they know that they don't
know what it is they're supposed to be
doing and that those those objectives I
mean they exist they are within us but
we may not be able to explicate them we
may not even know you know how we want
our future to go so exactly and the
Machine you know a machine that's
uncertain is going to be deferential to
us so if we say don't do that
well now the machines learn something a
bit more about our true objectives
because something that it thought was
reasonable in pursuit of our objectives
turns out not to be so now it's learn
something so it's going to defer because
it wants to be doing what we really want
and you know that that point I think is
absolutely central to solving the
control problem and it's a different
kind of AI when you when you take away
this idea that the objective is known
then in fact a lot of the theoretical
frameworks that we're so familiar with
you know mark after
processes goal based planning you know
standard games research all of these
techniques actually become inapplicable
and you get a more complicated problem
because
because now
the interaction with the human becomes
part of the problem
because the human by making choices is
giving you more information about the
'true objective and that information
helps you achieve the objective better
and so that really means that you're
mostly dealing with game theoretic
problems where you've got the machine
and the human and they're coupled
together rather than a machine going off
by itself with a fixed objective which
is fascinating on the machine and the
human level that we when you don't have
an objective means you're together
coming up with an objective I mean
there's a lot of philosophy that you
know you could argue that life doesn't
really have meaning we we together agree
on what gives it meaning and we kind of
culturally create things that give why
the heck we are in this earth anyway we
together as a society create that
meaning and you have to learn that
objective and one of the biggest I
thought that's what you were gonna go
for a second one of the biggest troubles
we've run into outside of statistics and
machine learning and AI in just human
civilization is when you look at I came
from this I was born in the Soviet Union
and the history of the 20th century we
ran into the most trouble as humans when
there was a certainty about the
objective and you do whatever it takes
to achieve that objective whether you
talking about in Germany or communist
Russia oh yeah I guess I would say with
you know corporations in fact some
people argue that you know we don't have
to look forward to a time when AI
systems take over the world they already
have and they call corporations right
that corporations happen to be using
people as components right now but they
are effectively
algorithmic machines and they're
optimizing an objective which is
quarterly profit that isn't aligned with
overall well-being of the human race and
they are destroying the world they are
primarily responsible for our inability
to tackle climate change right so
I think that's one way of thinking about
what's going on with with cooperations
but I think the point you're making you
is valid that there are there are many
systems in the real world where we've
sort of prematurely fixed on the
objective and then decoupled the the
machine from those that's supposed to be
serving and I think you see this with
government right government is supposed
to be a machine that serves people but
instead it tends to be taken over by
people who have their own objective and
use government to optimize that
objective regardless of what people want
you


Vectoring Words (Word Embeddings) - Computerphile

if we're moving from cat to dog which is
similar things so we go away from cat
and towards dog
and then we go can i go beyond in that
direction yes so the first result is
dogs which is kind of a nonsense result
the second is pit bull
so that's like the doggiest of dogs
right the least cat-like dog that feels
right yeah yeah actually well if you go
the other way well the the most cat-like
cat
the most undog-like let's find out it's
gonna be kitten right it's gotta be
cats feline kitten it's not really
giving us anything much to work with
i thought i would talk a little bit
about word embeddings word devec and
just wording bettings in general the way
i was introduced to word embeddings or
the the sort of context that i'm most
familiar with them in is like
how do you represent a word to a neural
network
well it's a set of characters isn't it i
mean
need it be more than the set of
characters that make it up
right so you can do that but you
remember the thing we were talking about
before in language models you have a
problem of how far back you can look i
would much rather be able to look back
50 words than 50 characters
um
and like if you if you're training a
character-based model a lot of the
capacity of your network is going to be
used up just learning
what characters count as valid words
right what combinations of characters
are words and
so if you're trying to learn something
more complicated than that you're
spending a lot of your time training
just like what words are and a lot of
your network capacity is being used for
that as well
but this isn't a hard problem we know
what the words are right you can give
the thing a dictionary and then you're
kind of
that gives it it gives it a jump start
the point is neural networks
they view things as like a vector of
real numbers or a vector of floats which
is like
some of the real numbers um
and so if you think about something like
uh an image
representing an image in this way is
like fairly straightforward you just
take all of the pixels and put them in a
long row and if they're black then it's
zero and if they're white then it's one
and you just have grayscale in between
for example it's like fairly
straightforward and so then you end up
with a vector that represents that image
it's a reasonably good representation it
sort of reflects some elements of the
structure of what you're actually
talking about
so
um
like if you take if you take the same
the same image and make it a little bit
brighter for example
that is just making that vector a bit
longer right or a point in that
configuration space that's a bit further
from the origin you can make it darker
by moving it close to the origin by
reducing the length of that vector if
you take an image and you apply a small
amount of you know noise to it
that represents just like jiggling that
vector around slightly in that
configuration space so you've got
you've got a sense in which
two
vectors that are close to each other
are actually kind of similar images
and um that
some of the sort of directions in the
vector space are actually meaningful in
terms of something that would make sense
for images and the same is true with
numbers and whatever else and this is
very useful when you're training because
it allows you to say if your neural
network is trying to predict a number
and the the value you're looking for is
10 and it gives you nine
you can say
no but that's close
and if it gave you 7000 you can be like
no and it's not close
and that's gives more information that
allows the system to learn and in the
same way you can say yeah that's almost
the image that i want
um
whereas
if you give the thing a dictionary of
words
say you've got your 10 000 words and the
usual way of representing this is with a
one heart vector
if you have ten thousand words you have
a vector that's ten thousand
long ten thousand dimensions and all of
the values are zero apart from one of
them which is one so like the first word
in the dictionary if it's like a
then
that's represented by a one and then the
rest of the ten thousands is zeros and
then the second word is like a zero and
then a one and then all zeros and so on
but there you're not giving any of those
clues if the thing is looking for one
word and it gets a different word
all you can say is yeah that's the
correct one or no that's not the correct
one something that you might try but you
shouldn't because it's a stupid idea is
rather than rather than giving it as a
one-hot vector
you could just give it as a number
but then you've got this indication that
like two words that are next to each
other in the dictionary
are similar
and that's not really true
right like if you have a language model
and you're trying to predict the next
word
and it's saying
i love playing with my pet
blank
and like the word you're looking for is
cat and the word it gives you is car
lexicographically they're pretty similar
but you don't want to be saying to your
network uh you know close that was very
nearly right because it's not very
nearly right it's a nonsense prediction
but then
if it said like dog
you should be able to say
no but that's close
right because that is a plausible
completion for that sentence and the
reason that that makes sense is that cat
and dog are like similar words what does
it mean for a word to be similar to
another word
and so the assumption that word
embeddings use is that two words are
similar if they are often used in
similar contexts
so
if you look at all of the instances of
the word cat in a giant database uh you
know a giant corpus of text and all of
the instances of the word dog
they're going to be surrounded by
you know words like pet
and words like you know feed and words
like play and you know that kind of
thing cute etc right
and so that gives some indication that
these are these are similar words the
challenge that word embeddings are
trying to come up with is like how do
you represent words as vectors
such that
two similar
vectors
are two similar words
and possibly so that directions have
some meaning as well
um
because then that should allow our
networks to
be able to understand
better what we're talking about
uh in in in text so the thing people
realized was if you have a language
model that's able to get good
performance of like predicting the next
word in a sentence
and the architecture of that model
is such that it doesn't have that many
neurons in its hidden layers
it has to be
compressing that information down
efficiently so you've got
the inputs to your network let's say for
the sake of simplicity your language
model is just taking a word and trying
to guess the next word so we only have
to deal with having one word in our
input but so our input is this very tall
thing right ten thousand tall
and these then feed into
a hidden layer which is much smaller i
mean it's more than five but it might be
like
a few hundred maybe let's say 300 and
these
are sort of the connections all of these
is connected to all of these and it
feeds in and then coming out the other
end you're back out to 10 000 again
right because your output is
it's going to make one of these high you
do something like soft max to
turn that into a probability
distribution
so you give it a word from your
dictionary
it then does something
and what comes out the other end is
probability distribution where you can
just like look at the highest value on
the output and that's what it thinks the
next word will be and the higher that
value is the more like confident it is
but the point is
you're going from 10 000 to 300 and back
out to 10 000. so
this 300 has to be
if this if this is doing well at its
task this 300 has to be encoding sort of
compressing
information about the word
because the information is passing
through
and it's it's going through this thing
that's only 300 wide so in order in
order to be good at this task
it has to be doing this so then they
were thinking well how do we pull that
knowledge out it's kind of like an egg
drop competition
is this why you have to devise some
method of
safely getting the egg to the floor
right it's not like the teachers
actually want
to get an egg safely to the ground right
but they've chosen the task such that if
you can do well at this task you have to
have learned some things about physics
and things about engineering and
probably teamwork yeah right right
exactly
so it's the it's the friends you make
along the way
so um
so the way that they the way that they
build this is
rather than
um
trying to predict the next word although
that will work that will actually give
you
word embeddings but they're not that
good because they're only based on the
immediately adjacent word
you um
you look sort of around the word so you
you give it a word
and then you sample from the the
neighborhood of that word randomly
another word
and you train the network to predict
that so the idea is that at the end
um
when this thing is fully trained
you give it any word and it's going to
give you a probability distribution over
all of the words in your dictionary
which is like how likely are each of
these words to show up
within five words
of this first word
or within 10 or you know something like
that if
the system can get really good at this
task
then the weights of this hidden layer in
the middle have to encode something
meaningful about that input word
and so
if you imagine the word
cat
comes in
in order to do well the probability
distribution of surrounding words
is going to end up looking pretty
similar to the output that you would
want for the word dog
so it's going to have to
put those two words close together if it
wants to do well at this task
and that's literally all you do
so so so
if you run this
on a lot it's it's absurdly simple right
but if you run it on a large enough data
set and give it enough compute to
actually
perform really well
um
it ends up
giving you each uh giving you for each
word
uh a vector
that's of length however many
uh units you have in your hidden layer
which
for which
the the nearbyness of those vectors
expresses something meaningful about how
similar the contexts are that those
words appear in
and our assumption is that words that
appear in similar contexts are similar
words
and uh
it's slightly surprising how well that
works
um and how much information it's able to
extract so
it ends up being a little bit similar
actually to the way that the
generative adversarial network
uh does things where what we're training
it to produce good images from random
noise and in the process of doing that
it creates this mapping from the latent
space to images by doing
basic
arithmetic like just adding and
subtracting vectors
on the latent space would actually
produce meaningful changes in the image
so what you end up with is is that same
principle but for words
so if you take for example the vector
and it's required by law that all
explanations of word embeddings use the
same example to start with so
uh if you take the vector for
um king
subtract the vector for man
and add the vector for woman you get
another vector out
and if you find the nearest point in
your
word embeddings to that vector it's the
word queen
and so there's a whole
giant swathe of like
ways that
ways that ideas about gender are encoded
in the language which are all kind of
captured by this vector
which we won't get into but it's
interesting to explore
i have it running and we can play around
with some of these vectors and see where
they end up so i have this running in in
google collab which is very handy i'm
using
word embeddings that were found with the
word to vec algorithm using google news
each word is mapped to 300 numbers
let's
check
whether
what we've got
satisfies our
first condition we want
dog and cat to be relatively close to
each other
and we want cat to be like further away
from car
than it is from doc right we can just
measure the distance between these
different vectors i believe you just do
model.distance distance between car and
cat okay 0.1
and then the distance between let's say
dog and cat 0.23
right
dog and cat are closer to each other
this is a good start right
and in fact we can
uh
let's find all of the words that are
closest to cat for example
okay so the most similar word to cat is
cats
makes sense followed by dog kitten
feline beagle puppy pup pet felines and
chihuahua
right so this is already useful it's
really handy that you can throw any word
at this and it'll give you a list of the
words that are similar
whereas like if i put in car i get
vehicle cars suv minivan truck right
so this is working the question of
directions
is pretty interesting
so yeah let's do the classic example
which is this if you take the vector for
king subtract the vector for man add the
vector for woman
what you get
somewhat predictably is queen and if you
put in
boy here
you get girl if you put in father
you get mother yeah and it if you put in
shirt you get blouse so this is
reflecting something about gender that's
that's in the in the data set that it's
using this reminds me a little bit of
the unicorn thing where
it you know the transformer was able to
infer all sorts of or appear to have
knowledge about the world because of
language right right but the um
the thing that that i like about this
that that
is that
that transformer is working with
uh 1.5 billion parameters and here we're
literally just taking each word and
giving 300 numbers
you know if i go
from london
and then subtract uh england
and then add um
i don't know japan
we'd hope for tokyo we'd hope for tokyo
and we get tokyo
we get tokyo twice weirdly tokyo tokyo
why is oh oh sorry it's no we don't we
get tokyo and toyco
ah which is a typo i guess and so yeah
uh usa
in new york ah okay interesting maybe
it's thinking larger city of yeah right
right like the exact relationship here
isn't clear we haven't specified that
what does it give us for australia
i bet it's yeah it's sydney sydney
melbourne so it's yeah it's not doing
capital
it's just the largest city right
um
but that's cool
it's cool that we can extract the
largest city and like this is completely
unsupervised
it was just given a huge number of
news articles i suppose and it's pulled
out
that there's this relationship and that
you can follow it for different things
you can take the vector from pig to oink
right okay and then like you put cow in
there
that's mu
you put a cat in there and you get
meowing you put dog in there
you get box
right close enough for me yeah yeah you
put um but then then it's it gets
surreal you put santa in there
ho ho
right fantastic
what does the fox say
it says phoebe
what so it doesn't know basically
although the second thing is chittering
do fox's chitter
i don't know
not in this data set


Elon Musk: Regulation of AI Safety

so on a darker topic you've expressed
serious concern about existential
threats of AI it's perhaps one of the
greatest challenges our civilization
faces but since I would say we're kind
of an optimistic descendants of Apes
perhaps we can find several paths of
escaping the harm of AI so if I can give
you three options maybe you can comment
which do you think is the most promising
so one is scaling up efforts on AI
safety and beneficial high research in
hope of finding an algorithmic or maybe
a policy solution two is becoming a
multiplanetary species as quickly as
possible and three is merging with AI
and riding the wave of that increasing
intelligence as it continuously improves
what do you think is most promising most
interesting as a civilization that we
should invest in I think this this a lot
that respond to investment going on in
AI where there's a lack of investment is
in AI safety and there should be in my
view a cup of agency that oversees
anything related to AI to confirm that
it is does not represent a public safety
risk just as there is a regulatory
authority for this like the Food and
Drug Administration is that's the for
automotive safety there's the FA for
aircraft safety which generally comes a
conclusion that it is important to have
a government referee or a referee that
is serving the public interest in
ensuring that things are safe when when
there's a potential danger to the public
I would argue that AI is unequivocally
something that has potential to be
dangerous to the public and therefore
should have a regulatory agency just as
other things that are dangerous to the
public have a regulatory agency but let
me tell you problems with this is that
the government moves very slowly and the
rate of the rate the usually way a
regulatory agency comes into being is
that something terrible happens
a huge public outcry and years after
that there's a regulatory agency were
all put in place take something like
like seatbelts it was known for on a
decade or more
that seatbelts would have a massive
impact on safety and and save so many
lives in serious injuries and the car
industry fought the requirements put
seatbelts in tooth and nail that's crazy
yeah and and honor hundreds of thousands
of people probably died because of that
and they said people wouldn't buy cars
if their seatbelts which is obviously
absurd you know or look at the back
tobacco industry and how long they
fought any good thing about smoking
that's part of why I helped make that
movie thank you for smoking you can sort
of see just how pernicious it can be
when you have these companies
effectively achieve regulatory capture
of government the bad people in the AG
community refer to the advent of digital
superintelligence as a singularity that
that is not to say that it is good or
bad but it that it is very difficult to
predict what will happen after that
point and and then there's some
probability it will be bad some probably
will be it will be good or if they want
you to defect that probability and have
it be more good than bad
you