AiTech Agora: Prof. Paul Pangaro: Cybernetics, AI, and Ethical Conversations

it's okay yeah go ahead we need an
ai in the background to know when to
start
these simple things we miss simple
things somehow we always want
complicated ones
so the recording is started i take it
good okay so again to try to keep it
informal
uh there's some slides break slides
uh with the word discussion and we can
open it up then and try to come back
the peril and the joy of course is that
we have an extraordinary conversation
and
some slides left on annotated or
unspoken
that's fine with me we can always
continue in other forms
but i uh rely on deborah who is really
remarkable and
wonderful in the time she spent with me
to help fashion this
for this group so i thank her and i also
invite her
to interrupt to say oops
hold it etc so please let's try to keep
it informal
so you should see a screen yes excellent
and now let me move whoops to keynote
so here we are this is a kind of guide
to the passing topics i'd like to flow
through
cybernetics and macy meetings already a
little bit in the preamble from deborah
and then today's ai i'm labeling this
specifically today's ai
it's not all ai but the way we have it
today is
in particular a problem uh wicked
problems are topic
that you know i suspect and i'd like to
use that
as a framing of a discussion of how
cybernetics in conversation
in my view may help
this leads us to pasc and to an idea
that i've been
developing that i call ethical
interfaces
i hope that becomes a waypoint to
ethical intentions
and then coming back to a new set of
macy meetings
so this is uh the idea of what i'd like
to do today
cybernetics and macy meetings many of
you know this history
in the 40s and 50s there was a series of
small conferences
and experts from this extraordinary
range of disciplines you
probably can't name a discipline soft
science or heart science that was not
present and they created this new way of
thinking
and acting in my view and it was a
revolution
of thinking and of framing how we see
the world
and they call this thing cybernetics
many of you know this word all of you
know the word i'm sure
the root is fun comes from the art of
steering
a greek word a maritime society uh using
the idea of steering toward a goal
acting with a purpose now goal is a hot
button here
it doesn't mean i have a fixed goal and
i go to the fixed goal and then i'm done
the word goal can be problematic it
means that we are constantly
considering action and in the framing of
action
considering what the goal might be and
how the goal might change
so i don't want to become a rationalist
here and say we always have a goal etc
or that the goal is always what we're
acting toward we're always acting
that for sure that's the foundation of
cybernetics as argued by andy pickering
and others
but acting with a purpose was the
insight if you will
the the moment of aha which led to the
idea that this could be a field on its
own
and these macy meetings change the world
now we often don't remember that these
three things
cybernetics neural nets nai are
intertwingled
as ted nelson might say these things
the mccullough pits neurons which is the
first idea of over
simplifying a brain neuron in order to
do some calculations
the macy meetings already mentioned and
the book of cybernetics by weiner which
of course was also part of the origin
of cybernetics as a field but i feel
that
the book gets all the credit in the in
the popular zeitgeist and the macy
meetings gets lost
this was uh happening in this era
roughly these days
and neural nets were born out of
mccullen pitts who were at the core
mccullough at least at the of the macy
meetings
and the macy meeting swarms the
zeitgeist i like to say
because the book and the idea of
circular causal
systems influences generations of
thinking
whether or not the word cybernetics
persists or is given the credit
we have another phase the dartmouth ai
conference 56 i believe
and symbolic ai rises owing
largely to smaller cheaper faster
digital machines
perceptrons the book comes along very
very conscious
desire to kill neural nets political
story for another time
and cybernetics languishes and
the dartmouth ai conference was
conceived
against cybernetics they didn't want to
use the word they wanted to
separate themselves from it and with the
rise of the power of some
hey we can do chess hey we can do all
these amazing things we can
do robot arms and move blocks around
fantastic
um this was what was going on in that
era and
as a consequence of this and as a
consequence of other things cybernetics
languished there were some philosophical
issues that
made it problematic i would say for
various political
environments and then hinton
doesn't listen to the perceptron's book
trying to kill neural nets and realizes
that it's an oversimplification in its
criticism
and we have this coming along
and i'm going to tie i don't think i
need to convince you
the consequences of today's ai and big
data
into what uh zubov calls surveillance
capitalism and
the wicked problems that arise there are
many many wicked problems
so let's simplify this chronology for a
moment
don't forget expert systems
but of course after the 80s neural nets
in the 2010s became extraordinarily
powerful
because of big data and because of
massive compute
and today's ai everywhere in our lives
and this is of deep concern to me it's a
deep concern to many people
the recent controversy at google with uh
uh tim mitchell being well being fired
i think is accurate enough and many many
other issues that you know
so what's going on here manipulation of
attention
tristan harris has made this a very
important signal
in today's silicon valley days
we know the manipulation that's going on
all these things we know
this is what i mean by today's ai
it's hardly an exhaustive list but it
feels like a pretty powerful
set to be concerned about
now with colleagues i often use the word
pandemic
and i would claim that ai today is a
pandemic and they say wait a minute it's
not biological
and other comments could be made and i
don't disagree with them
but my feeling is that ai makes the
world we see in the world we live in
and the loss of human purpose in the
morass of all of that
is a concern
and pan demos ben sweeting uh looked it
up in a meeting recently pan
everywhere or all demos same route as
democracy
the people so all the people are
affected
by ai today only two or three billion
are online
but that's close enough for me
so this led me in the process of the
covid arising and um
the shutdown of carnegie mellon and all
of the institutions that you know in the
world that we've
lived in brought me to a couple of
moments that the wicked problems demand
conversations that move toward action
are transdisciplinary if we're going to
do anything about today's ai or today's
wicked problems
certainly we need transdisciplinary but
we also need transglobal
namely geographically inclusive and
ethnically culturally socially inclusive
and also what's
extremely important to me in particular
is trans generational
and maybe we'll come back to this i've
made some efforts in this direction
as part of the american society for
cybernetics as ever mentioned that i
become president of in january
so could we possibly have conversations
of a scope that could address
all of these pandemics um
well that's ambitious audacious
crazy but i note
more and more i'm hearing conversations
in this framing
of the framing of we have global
problems
some are biological pandemics and
there's a lot of other wicked stuff
going on
can we talk about it and i don't mean
that in a
superficial way i mean can't we begin to
talk
in order to move toward action and i'll
come back to that as a theme later
so as a reminder of the macy meetings
we need such a revolution again
to tame today's wicked problems and i
acknowledge those
in the audience now who question the
word tame and that could be a
conversation we might get into
but the idea is is how do we improve the
situation we're in
if we like herb simon who only left
cybernetics as a footnote in his famous
book sciences of the artificial thank
you her
he did admit that it was moving from a
current state to a preferred state
and in that sense i think we can invoke
design
and invoke action so this
might be a stopping point if we wanted
to
ask a question or i can also just keep
going
depending on preference it looks like
you're on a good roll and nothing popped
up in the chat so i would say let's uh
wonderful so why cybernetics what is
this thing it applies across
silo disciplines it's this
anti-disciplinarity thing
focuses on purpose feedback in action i
think you know that it's a methodology
again another term wicked problems
complex adaptive systems
there are many descriptions of this this
particular phrase complex adaptive is
quite
is very much around and i find it fine
it seeks to regulate not to dominate
and it brings an ethical imperative and
we'll come back to this
and i want to acknowledge andy pickering
who coined the phrase
anti-disciplinarity
years ago and has also been
a wonderful influence on these ideas and
on me personally lately so what are the
alternatives
to cybernetics here's the cynical slide
which says not much else is working
so where do we go there are none
apparent alternatives to cybernetics
it's a bit arrogant perhaps
this was an email i got from a research
lab director
in 2014 who had the instinct that second
order cybernetics
times design x crossed with design
crossed with some modern version of the
bauhaus is what we need to
fix science an extraordinary phrase
fix science i love it but this is along
the theme of isn't there something we
can do here if we are from cybernetics
and i love the design component and i
know many of you would as well
since wicked problems cut across so many
different domains we need
deep conversations and this is a recap
of why we need new macy meetings
global and virtual and after i coined
new macy
i went back to andy's work and he talks
about
next macy as a new synthesis in 2014.
so what's missing conversation ronald
glanville some of you know
making a bridge why does it matter
well to tame wicked problems assuming we
can make some improvement to wicked
problems we have to act together we
can't act separately that's obviously
not going to work
to act together we have to reach
agreement
to reach agreement we have to engage
with others and to engage with others we
have to have a shared language to begin
so to cooperate and collaborate requires
conversation
what may come out of it well lots of
things
i would claim to achieve these requires
conversation
what do you get if you have effective
conversation
these things and again i would say
all of these demand conversation so
what's missing
is conversation and now the question is
how does all this fit together what am i
talking about
well conversation in today's ai
let's contrast these things
whereas today's ai maybe all ai is
machinic digital
representational you could argue that
machine language
predictive data animated and i'm
proposing
that cybernetics is a bilingual
sensibility i owe this to karen kornblum
who's on the
call today it's bilingual it goes
to conversation into today's ai
now these things are all intermeshed i
won't talk through how i think they
self-reinforce and make each other but
of course this is what
weiner meant when he talked about animal
and machine
and again uh we might stop here
for a moment i see esther said to fix
the practice of science
fair enough one could dive deep into the
idea of what
what science practice is and how that
gets us to a certain
plateau of understanding but acting in
the world is
is beyond understanding deborah
a pause here shall i keep going i think
if people
uh people can unmute themselves uh so
um feel free i think we're we're in a
good enough flow that if someone uh
comes up uh
we can do that but uh yeah i would say
yeah james has had a question maybe
james would you like to
speak up and ask this question directly
or
well and and we can get to this uh later
paul but i think we'd all be interested
in hearing more about the politics of ai
versus cybernetics historically
especially if it's something that would
be useful to consider
for how things move forward
a long discussion on its own it's a
beautiful point james thanks for
bringing it up
um i'm not sure how to unpick that now
um and i think there are others in the
room who may
be able to expound it better than i can
cybernetics of course grew out of world
war ii
in some ways and grew out of this idea
that
circular causality was everywhere
some criticized cybernetics as being
about control but i think that's a
misunderstanding of the term
um ai in my view grew out of
the desire to use digital to dominate
an environment which could be controlled
by conventional mechanistic means not
embodied organic cybernetic means
that that's a really uh poor job
at the surface of it and i think by
politics you mean something deeper
um maybe we can reserve that for a
little later if there's time or
another session i think that unpicks a
whole world of complications
yeah i think um just one comment along
the way then
uh that the traditional challenge of
incorporating
teleology into the sciences um
seems to sometimes strike people uh
the wrong way about cybernetics and the
other
comment that i'd make is um
because i'm biased but i feel like
cybernetics has a much stronger
theoretical
grounding than artificial intelligence
but one of the things
that really strikes me about artificial
intelligence
is the expectation of the
the artificial so the exclusion of the
human and
uh the the fact that humans are expected
to be part of cybernetic systems
um at least seems much more clear in
cybernetics so i'm i don't know whether
that intersected with the politics at
any point
but that's just a comment to leave along
the way yeah
yeah yeah it's a big box
uh big pandora's box to open
uh i don't want to cherry pick but uh
phil beasley phillip thank you for
coming today it's wonderful to have you
here
um so minsky and pappard wrote a book
called perceptrons which was based on a
paper that was
left on a file server at mit for all to
see
as a book was posed and the purpose of
that paper was to kill neural nets and
it did
the story of von forrester which he told
very often was
for many many years one forester would
go to washington to get money to
support his biological computer lab at
urbana
and one year he went and they said oh no
you have to go see this guy in cambridge
he'll give you the money
because we've decided to centralize the
funding and all of the money now will
come through this guy while heinz went
to see marvin and marvin said
no and that was the end of the bcl
so that's a factual story that isn't is
in the history
phillips raising other extraordinary uh
questions as usual uh which i i'm gonna
duck
that's another interesting conversation
philip maybe we should have one of these
chats uh about the philosophical issues
and the relationship to modernism and
again i would defer to you and others as
perhaps
not perhaps as much more
conversant with that history but thank
you for that
deborah anything else in the chat um no
i think there was a comment there about
uh
ai uh failing or not and maybe we just
scratched the surface this is uh
oh yes volcker yeah yeah yeah
succeeded at many many things but failed
at producing a humanistic technology
in my view a technology that we embrace
and we love and we use every day
i mean who loves
i don't want to make this seem an attack
who loves facebook
you know it's it's uh extraordinary in
many ways and yet
fails in so many ways anyway
yeah i would i would point out that uh
the people who were in the cybernetic
meetings and they see
they didn't think that they succeeded
very well
uh margaret me didn't think that it was
a success and stuff like that so i feel
like that part of it is also
we're living in this moment so to say
success failure is always uh
hard for us to assess black and white
yeah it's much too black and white i
agree with you
so back to norbert weiner and
of course weiner in a cybernetic world
invokes gordon pasque in a cybernetic
world a cybernetic praxis
uh both theory and making of machines
practice action in the world
if you don't know andy pickering's book
the cybernetic brain i can't recommend
it
highly enough it's a particular view of
cybernetics
that others feel doesn't attend enough
to second-order cybernetics or to
um uh
gordon's theories uh but uh
i think it's um it's andy's trying to
make a different point it's it's an
earlier moment
in his arguments for what the history of
the field has meant
and we're also in active conversation
about that and
i think that that thinking and that
research if you will will will continue
so let's go into past this is a picture
of
gordon pasqu's
colloquy of mobiles in 2020 in paris
where my colleague tj mcleish and i were
in february
and installed it at central pompidou it
was part of an exhibition that
unfortunately closed rather quickly
but the original from 1968
was an extraordinary thing these are
physical mobiles the
large uh ones that you see
flesh-colored are females so called the
black ones with the little light in the
middle are so-called
males they do a courtship dance a long
story for another time
but pasc even in the 60s imagined
machines that were autonomous agents
that learned and conversed
and already there was a bilingual
sensibility here of the human and the
social the machinic and the digital
and the interactions were about
resonance and not representation
and the i the fundamental architecture
if you will was interactional and not
stand alone
so if you study this work and other
works of cybernetics it's not a machine
that's smart
or an algorithm that's smart but rather
in the interaction intelligence is
exhibited by way of achieving uh
purpose formulating new goals acting
toward those goals and so on
so this is one way of thinking in a
broad brush of what
colloquial mobiles uh tried to do is
extraordinary work
the goals of conversation theory in my
view are to rigorously understand what
makes conversation work and to make
machines converse in like humans and
like humans not not like machines
we can talk about the limitations of
what that might mean
but also to rigorously understand how
systems learn
because these two things conversation
and learning
are inextricable i think that is
uncontestable and in our daily life we
experience that
but also uh it was very much what past
was interested in and he saw that they
came together he wrote a number of books
i don't know how you write two books of
this length and depth within a short
period of time bernard scott who's on
the call
was part of this era with pasc and was
very important to the psychological
studies and the
of the learning studies that were done
and i value very much the conversation
that bernard and i are continuing
i came across pasqu uh as deborah
mentioned at mit when i was at the
architecture machine which was a lab
started by nicholas negroponte
which was the predecessor to the media
lab although predecessor implies
a continuation of dna that wasn't really
there
the architecture machine as you see
grounded in the upper right in an
awareness
of minsky simon newell papert
everybody these are the ai guys
nonetheless
uh took a balloon ride north to thirty
thousand feet
and thought about interactions of any
kind
in a set of cybernetic
loops in a set of relationships so on
the left you have a
one participant in a conversation on the
right you have b
or beta as it's also labeled and these
are the interchanges some of these are
me
saying something to you some of these
are me taking an action
uh some of these are you observing my
action and saying something and taking a
further action
behind this is actually quite a lot of
detail this is one summary
this is another summary
and all of this is about modeling
conversation
and leading toward for example in the
work i do in interaction design
trying to take these components of
conversation
that we start in a context we start with
a language
we have goals and so on that are
evolving
these lead to some other ideas again
this is just in passing
there are some conversations are better
than others
uh one conversation i like to have is
what i would call an effective
conversation
where something changes
and brings lasting value so not just any
change but a change that brings lasting
value
and these changes may be in information
transaction they may be rationally
emotional and so on
but back to this idea that the goals of
conversation
are these
conversation as this critical aspect
conversation cybernetics
ai and that conversation and that triple
[Music]
diagram theory specifically
is helpful and then we'll get more into
that in a moment
when we talk about ethical interfaces so
again a moment of a brief pause
yeah uh david do you want to say
something about uh
connecting resonance and representation
with your uh
[Music]
yes um so actually i was just
plus wanting james's
comment but yeah
for me it resonates very much from my my
own work and also the
the work that we're trying to uh to get
going um
in a in a large proposal um that you'll
probably hear about more uh
paul uh soon maybe already have
but that's a a lot of work uh
also in my uh domain of human robot
interaction
focuses on modeling humans
you know representing that perhaps to
put into the basis of
machine operations so then we have
human-like
but in isolation it's always a lot of it
is in isolation and the
real interaction which is such an
overused word that uh
that i almost hesitate to use it again
that's being lost and what uh what i
like
very much is that you you call this
resonance which which is a term that's
also being used in many other domains
that
actually already implies that it's more
than interaction
it's actually loops perhaps exciting
each other
and and so i very much like this very
short phrasing
so there was my plus one um thank you
it's very necessary
thank you david appreciate it um
do you wanna you wanna talk through your
comment
yeah sorry i'm so wordy
so um we've been looking at ai as a kind
of continuation of
um let's say operations research applied
at scale to many social contexts
and when we look at the way those
operations work we notice that they
really do not distinguish between
managing things and managing humans so
in a sense they're removing the category
of human
as this distinct category for which we
can have ethics or politics and they're
also centralizing control as rule said
and this comes exactly at a time because
you raised timnit gabriel
where scholars working on race and
anti-blackness
are saying that the category of human
historically and today excludes
especially the black people and a lot of
other disenfranchised
populations so one could argue okay
let's ask for a very
let's insist on a very universal human
and not use the human
as it has been used historically but to
make sure that we center the people that
the category has excluded
or we can be very careful about using
the category human meaning that it has
been almost always historically used to
exclude black people and
other disenfranchised people so i'm just
wondering um
i mean that's a lot of claims like do
you agree that operational control
uh you know with this kind of
centralized ability to set policy on
operating on things in humans not
distinguishing humans and things and
then
what should we do with the category of
human because it keeps on coming out but
we use it maybe a little too easily
i agree generally with what you're
saying and i thank you for the comment
ai of course has become distributed
cybernetics uh was interpreted for
example by stafford beer and fernando
flores
to be a central control of uh economy
in chile but the word control i think is
problematic as i
think i said earlier in cybernetics it
doesn't mean control
it means to attempt to regulate in order
to achieve
action that is effective in the world
and usually that reflects a goal and
purpose that's
behind it all again a wonderful comment
in which there is tremendous richness
maybe a topic of its own
entirely it relates to the politics
topic earlier
i don't think i can eliminate it better
than that right now that's not an
elimination i mean i don't
but i love that point maybe we can
return to that in another form
because i think that's very important i
just wanted to mention in this context
that
david usually remarks that his interest
in cybernetics is to realize that
control is not just bottom-up
and could also emerge in situations and
i also wanted to flag
the connection with resonance uh james
derek lomas
i think paul we talked uh we made that
connection earlier
and derek i don't know if you want to
say anything more about the resonance
right now or
no i'm just loving the conversation and
it's it's going in
a great direction and it's just so much
fun to
get such a great context um
i mean i i find i i know i have a bias
of being attracted to areas of academic
study that seem a little bit off limits
and for some reason
cybernetics has a little bit of that
scent to it and i don't know why
but its association with with with
resonance um
somehow uh confirms itself
somehow we could talk about that also it
makes it make some
points of view uncomfortable and and
again
there's been some discussion of why
cybernetics failed
which in some ways it has again blacking
and whiting something that is really
gray
um and and perhaps a topic for another
time
uh ben i just happen to notice i can't
see all of the stuff going by the chat
it's a magnificent
conversation on its own that i look
forward to saving and savoring
uh ben sweeting mentions about pasc's
model
that yes it does include me with myself
conversing
and having interactions with different
points of view in my own head that was
one of the important things about his
theory in my view
which was it's person to person me to me
as long as i have different points of
view that need resolution
across a boundary across a difference if
you will
and then of course you could even say
schools of thought speaking to schools
of thought
democrats and republicans come to mind
unfortunately
yeah should we move on then wonderful
so again if you base this idea
of ethical interfaces
uh on understanding conversation and
understanding and i'll amplify that in a
moment i believe that the ultimate goal
is to build better machines to build a
better society
but as always the question is how do we
do all this
i like organizing principles i think you
would agree
that there's no such there's no there's
nothing more practical than a great
theory or a great organizing principle
so let me expand this one i i like to
unpack this it'll take a few
steps so i shall act always
so this is me taking responsibility for
my action
and saying that i will always act
so as to increase
the total number of choices now many of
you will
recognize this and i'll give it the
authorship in a moment but for those who
haven't seen it before i want to explain
choices means something very specific
choices doesn't mean
options for example right now in this
moment i could do one of a thousand
different things
and all of those are options to me i
could stand on my head i could turn off
my machine
i could throw my coffee cup again no
they are not choices
a choice is something that i would
possibly
want to do now a viable option
something that would reflect resonance
with me
and who i am and how i see myself and
would be consistent with what i am which
you can phrase
in terms of my goals my purpose
my intention my direction now
i shall act is important going back to
that
and the author of this makes a wonderful
distinction i could try to say
thou shalt i could say you must do these
things
but of course that's me being in this
particular way of distinguishing these
terms
moralistic me standing outside
saying i have the right to tell you what
to do
that's the thou shalt that we recognize
i shall means in a sense the opposite
i'm part of this whole thing
i am indistinguishable sorry i'm
distinguishable but i am
part of the greater flux and i take
responsibility for
what i do in the context of the whole
so this of course comes from heinz
forrester and he called it an
ethical imperative
and some on the call once again are
challenging
this and its limitations i think ben
sweeting in particular
and i look forward to those later
developments
now i want to go here i'm going to
declare this an axiom for an ethical
interface and say as a designer i shall
act always to increase the total number
of
choices not options amazon is about
options
right amazon suggesting things to me
based on what other people have done
are not choices because they're not
about me they're about the big data
they're about the aggregation of
millions and billions of interactions
that in a sense have nothing to do with
me
they have to do with an aggregation but
you know what the hell does this mean
how do we do this right well could i ask
you one question to make sure i
understand
the difference between uh choice and
options
as you see it so uh i read
this book the paradox of choice that you
may or may not
know about and one of the most famous
examples is
supermarkets right where
you would think that if you give people
more and so he calls it choices
if you give people more choices that
that would be a good idea but actually
it also takes away something for people
so
it costs effort and it makes people a
little bit unhappy because they don't
really have the means to make the best
choice or
et cetera et cetera so there's many
downsides which is what he called the
this paradox of choice so do you think
that
you know it's a different use of the
term choice exactly yeah i'm just
speaking
very very specifically what might i do
now to be who i am in taking an action
so heinz used to say paul don't make a
decision
don't make the decision let the decision
make
you which was a way it takes a moment to
reflect on that
which is a way of saying when you think
of yourself as deciding something
you're really being who you are so when
i walk into
a supermarket it can't possibly put in
front of me my choices
because it doesn't know me and our
definition of personalization today
ain't what it should be if it's this
aggregation of the totality of the
billions of ideas
sorry the billions of choices that
people have made and therefore it says
people who bought cigars also bought
smoking jackets
famous example from 20 years ago
so so he's reserving choice to mean
something very specific and i wanted to
mean
that specificity here as well
can i say something but as as uh
ai collects your history and amazon does
that and many other places do it
then they do get to know you uh
sometimes even
better than or some things that then you
know yourself so
i i'm not saying you know uh that it's
uh
it's the kind of knowing and meaning
that we want
but uh yeah go ahead you know we can
unpick that further i think i have a few
slides here about that yeah
to try to unpick it but if not we should
maybe come back to it and
remain so number one actors to increase
choices for a user now part of that
for me is acting
in order to create conditions such that
others may converse
because it's through conversation that
you expose
or learn or have revealed
to you what the options are within which
you can decide which are viable choices
so my claim is that designing
for conversation designing such that
others can converse
is absolutely foundational
and for me that's part of a praxis if
you will of ethical design
so i propose applying models of human
conversation you've seen a
skeleton of that here in the slides i've
shown
strive for interfaces that are
cooperative ethical and humane and i'll
explain those in a moment
and push for new forms of interfaces and
this is really the basis of what i'm
talking
talking about i'm sorry that went by
these are offers
that i think are are worth proposing
to you and to others so if you design an
ethical interface
one idea one intention is to make it
cooperative
so it's cooperative when there are a
sequence of coherent interactions
that enable the participant to evolve
their points of view
such that in understanding and agreement
are ongoing
there's a cooperation that allows a true
conversation
to evolve such that we might have
understanding and agreement
might have we might agree to disagree
one big block cooperative
next block ethical i claim that it's
ethical
when there is reliable transparency of
action and intent
the what in the why such that we might
build trust
now very often we're told that
google has really great search results
and it offers us the best choice for us
as deborah was saying it knows my
history of clicking around
and it uses that to to tell me something
that's coming up next
that's fine uh that's helpful but it
doesn't tell you why it made the choices
and let me take a moment for a brief
parable i call it the parable of luigi's
pizza
it's a little awkward to do in this
context so i'll pretend to be both sides
if i'm in an audience i ask someone to
say
where's their great pizza and i say
right across the street the luigi's
pizza
and then i asked them to ask me why is
it great pizza
and they say paul why is luigi's great
pizza and i say screw you i'm not going
to tell you that
and after they're shocked and they go
back in their seat i then say
i just described google
because google doesn't tell you why it
makes the choices it makes oh yeah there
are 200 signals and it takes into
account recency and reviews
and where you clicked and all of that no
that's a generalized answer
that's like giving me terms and
conditions
that i have to read through in order to
hear the generality of what's going on
but that's not
why it chose luigi's pizza
now if i asked you a friend of mine
colleague of mine where is their good
pizza etc
and i asked you why you thought it was
good and you said screw you paul
i'd never talk to you again but we allow
this
from our machines we allow to be
mistreated by our machines so i'm
claiming that an ethical interface is
one in which i can say why is it great
pizza and it says
well paul your values are you like
sustainable
sourcing you want people to be paid well
you want it to be open late you want
gluten free and in your value terms
this does what you want and therefore
this is a way of
moving ahead of being in the world in a
way that you want to be
it's a valid choice so terms and
conditions
are an answer to the question why
how does google present to you why luigi
is great pizza
but i would claim it's not humane so
this is the third intention
where you can in the conversation create
a direction for the focus and flow
so very dense but these three ideas
cooperative ethical and
humane for me are pragmatic ways of
talking about designing interfaces
and therefore i think could be
a contribution so this might be a place
to start
another conversation
i didn't realize i'm muted i'm also
mindful of the time
uh so let's uh so paul can you make a
decision
we have uh kind of nine minutes left uh
of you know would it be good to uh
close stuff up in a few minutes and then
kind of uh
yeah i can do i can do the rest in just
a couple of minutes shall i do that
yeah yeah i think i think that's good
yeah so
i want to build a better society and
better machines
are part of that this is part of what i
call ethical design
it's a bit highfalutin how do we do that
well we talked about the wicked
challenges this is another paraphrase of
that
to go after those you've heard me claim
we need conversation who do we need in
the conversation well
some people in the conversation could be
this history from cybernetics
both the history and current
practitioners
so the bottom gage and doverly glanville
has passed away carrione and deborah and
gagan are still very close colleagues
but my point about trans generational
these are younger generations coming
along the list on the right are often
students
and they are interested in these ideas
and they are practicing
systems thinking cybernetics etc and
these are the people for me who are
extremely important in these
conversations
and how do we begin we begin with the
new macy meetings
we begin by moving along a path
this happened to be my path today
with considerations that you have heard
and that is what i wanted to say thank
you very much
fantastic um so
i think we could uh take some of the
if you just scroll from the bottom of
the chat poll
um and maybe uh andy
do you want to say something about your
comment
yeah so uh let me
it was essentially the second point that
paul was getting at that building of
trust
um in in a way that is significant to
people and both
uh the building of choice and the
building of trust can happen through
conversation
um but it's a question of which people
prefer today
uh more choices more variety or if
that's what choice means to them
or that building of trust and that
they're making the correct choice the
first time
right
yeah correct choice the first time is a
little tricky yeah of course
certainly you want to let them make a
choice and then recover easily it's that
undo button in the world uh that's
that's tricky
yeah no that's a good observation i
don't have more response from them
do that
what about role
what about what sorry raul made a
comment uh
but i guess he's responding to bernard
so there's a lot yeah go ahead bro
i was responding to some of the
responses on um
issue of race as a valid category
um and i was just pointing to a report
that was written by colleagues
former colleagues of mine at the air now
institute where we look at um
the really urgent uh societal
implications of
of ai systems as they're being deployed
now mostly by
um by larger tech companies um
and how do you how they lead to issues
uh
across categories of gender
race and other important categories
um so that that's a really good report
for those who who think we shouldn't be
using the category
i disagree with that um and i i just
wanted to maybe also respond
quickly to just the former uh discussion
um i think one of the things we do at
delft i think we do pretty well is to
to think about um kind of the normative
aspects ethical aspects
of technology and that's that's part of
this ai tech forum where we think about
meaningful human control
and there's a lot more coming there so i
think when we think about choice
um i i tend to i tend to kind of side
with what
what david said is that oftentimes
choices are also
um about like important trade-offs have
to be made and
and thinking about um the later
implications
of design choices um so it would be
that's what my own personal research is
about it's about hard choices and how to
address the
normative uncertainty and how to create
conversation
the choices that you materialize in the
system so i'm kind of curious
how you think about dealing with the
politics and the normative uncertainty
when
thinking about design of college
cybernetic systems called the i systems
and how it comes back in your your
perspective
yeah it's a beautiful question um
so it's a conversation about the
conversation that you're yeah
yeah well second order cybernetics right
yeah he's easier said than done yeah
i hope go ahead somebody says something
i'd like just to clarify what's why i'm
concerned about the use of the term race
there's a scientific basis for it
and people look across each other mutual
ignorance
some people think they know what the cat
is labeling a category there are better
labels
the ethnic group um
[Music]
culture and so on i have i bet is
i've met people who think that different
races
i've met people who think that different
races are actually different species
they just have they're ignorant of the
basic biology
and as i said before the term race does
not have a
a well-founded scientific basis
thank you bernard and i have had an
exchange about this before the idea is
not to deny difference
but rather not to place it into the into
the label of race but rather into other
labels
which then expand the conversation in my
view
to um the aspects that are really
important which is to discuss
differences and to be inclusive
absolutely thank you thank you
yeah um i think uh jared
you made a comment about uh
conversations or the best part is often
that they develop in unexpected ways do
you want to say something about
that um yeah it's one of those tensions
i i feel as in i have a feeling that you
often treat our ai
as our helpful slaves which is very good
because they do
cool stuff for us and if they're not too
intelligent we don't mind
um so but that means that you often want
to give them very specific
instructions and have them do them as
well as they can and that feels a bit at
odd with the idea of having
conversations and opening up and
having things going in expected ways and
i was wondering um
how would you propose we reconcile that
contrast between
the two um
thank you for that comment um let me put
again in the chat
the link to my page which has in
addition to this pdf that has an
appendix pdf
and in that appendix are additional
slides
which amplify the power of
conversation to create possibility
and that's another way of talking about
conversation as
opening choice and again choice in this
meaningful rich way that lines of course
so if you consider the desire to create
new possibilities
as appropriate and ethical then
conversation
for me is almost the only way i want to
hedge that a little bit from learning to
ride a bicycle i'm not necessarily
talking to myself about it
and you'll see some slides in there
which talk about
what a great conversational partner is
and in particular here's one other
comment to make a complaint i have about
recommendation engines and facebook
feeds and google
ranking and so on is it's based even at
best
on who i was it's making a decision on
my behalf
as if i were in my own past or to put it
a rather more contentious way
as if i were dead because answers are
dead
questions are alive questions are of
them now
so don't give me a search engine that
gives me answers give me an engine that
gives me questions
because questions open up possibilities
and that's another whole research area
i'd like to
develop and there are some slides on
that in the opinions
this is very cool i'm going to be uh
pretty
uh accurate about the the the ending
time
partly because i feel like we we clearly
wet the appetite
of many people here and i want this to
be
an ongoing conversation and we also have
something starting in a couple of
minutes that we want to engage some of
the people here
thank you so much paul and thank you for
such a great audience
it was lovely to have the conversation
ongoing throughout
um and we'll like i said i think we'll
try and make sure that the chat
uh is also copied over because there are
plenty of comments in here that
we just managed and links and stuff like
that
luciano so this is fair to uh to close
now and then
for


183. If I were a Well-intentioned AI 2

to the 180 third session of the AI
safety reading groopman this is the
second part of if I were a
well-intentioned AI last time we covered
what it would be like to be an AI who
understood good hearts law and was one
intention enough to try and overcome it
we examined how you would if you were an
AI go about dealing with it if you were
an image classifier we found that
there's a few techniques and strategies
that you would be able to replicate a
broad concept you might replicate would
be inverse reward design strategy might
be randomizing your defenses against an
adversarial attacker a technique might
be using background Birsa semantics to
find the right features but without more
improved on that much then I rock you
for moment can you please make the slide
full-screen is that better yeah that's
better thank you
now this time we're going to be covering
what happens if an AI can act in the
world and we'll investigate a few of the
types of value functions that give rise
to problematic behaviors so when you're
acting in the world that substantially
complicates the situation but it gives
you more info to work with you'll find
that there are preferences for which
agent ai's can behave in a good heart
like fashion but for the people who are
designing the AI that's not a problem
they want the good heart like behavior
the reason we usually worry about good
heart like behavior is because our
preferences are complex and they may be
very unlikely from the a eyes point of
view we tend to have diminishing returns
negative trade-offs between different
people's values and it may just be that
optimizing for our true value is very
difficult compared to others but if AI
you understood this you would be able to
avoid it a lot of good heart like
behavior so in this article stuart
armstrong gives the example of an AI
trying to find its way through a maze in
have the training the AI encounters on
the red doors so the AI will he put
inside a maze
it will find its way to a red door and
the training session will end it will
get its reward but unfortunately like
last time your training does not prepare
you for the tough decisions of the real
world you encountered a red window and a
blue door you don't know which to go to
based off your training you could
conclude
the humans either meant go to a red
thing a door or go to a red door but
there's no red door so here we can
discount that possibility safely if you
go to the red thing first you'll notice
that the scenario doesn't end so perhaps
you'll update police and say okay I need
to go to the door instead of the red
thing going to the door you realize that
the scenario still doesn't end so you
need to consider that perhaps the humans
want you to just stay at some place
forever
so either stay at a red thing or stay at
a door and you get continually rewarded
now supposing that those are your only
two options then if AI you were to ask
your Creator who either wants you to
stay by a red door by a door forever or
stay by a red thing forever if they
asked 'are kraid whether or not they are
a suppose you know that your Creator
steward either wants you to stay by a
red door red thing or by a door if you
said look I think you're either a door
Maximizer or a red thing Maximizer
should I just maximize whichever utility
I find most likely to divorce the answer
is a resounding yes Stuart says if aiu
is unbiased and finds that it's more
likely I'm a door Maximizer standing by
the door forever is optimal vice-versa
for if I am a red object Maximizer the
point is that
given these utility functions you if you
believe the AI is unbiased as to
whichever one it thinks is most likely
then even if say for example you want
the AI to stand by a door forever and
the AI says I will stand by the door
forever with probability P and I will
stand by the red thing forever with
probability one minus B that will
maximize your expected utility as the
creator why well in this scenario the
utility we get from going from one thing
to another from the red object the door
is zero so essentially we're incurring
the opportunity cost in going between
one object or another so just provided
the Creator doesn't know what
probabilities the AI is assigning for
standing next to a red thing we're
standing next to a door forever the
optimal policy should be off the form
just go and stand in a place forever I
apologize if that was a bit confusing
now Stuart goes on to say that in more
complicated scenarios you're going to
have a great deal more utility functions
that are plausible and you need to be
able to ask your Creator some questions
to cut down the space of functions so he
refers to a paper where they consider
some space of possible reward functions
has some volume fee and the true reward
function that you can't really specify
easily is this red dot here in the left
hand picture
now the gray picture the middle picture
has a gray area which sort of represents
irreducible uncertainty in the reward
function we've encountered this before I
think in one of Rohan Shahs papers where
he mentions that you cannot fix a
utility function from a given policy no
matter what a human behaves like there
is no unique utility function which you
can assign them so there's some
irreducible uncertainty in the space of
reward functions that's possible but the
AI can still ask some questions and
narrow the space down which is what's
going on in the third image so the point
of this image is that you are able to
get with in an arbitrary accuracy of the
ideal reward function up to irreducible
uncertainty in a certain number of steps
the steps is bounded it doesn't increase
too rapidly so asking queries is not in
principle too hard there are some
relatively efficient algorithms existing
but you might ask that why does it still
feel very wrong that aiu happens to be
standing in one place forever
looking at that situation we think hmm
there's something very wrong there so
why did the example feel wrong as Stuart
explains it's because we have some
implicit beliefs about our values we
expect that the true values we hold are
hard to optimize there are penalized and
penalized in probability from the ai's
point of view our values might be
extremely complex and fragile they may
have diminishing returns or it could be
that we have some contradictory values
so there will be negative trade-offs
and of course there is one final element
which is to do with regression in this
case it's called predictable
disappointment and it's a reference a
few times I need believe a common
article that's referred to is called the
optimizers curse essentially it says
that if you construct some prediction
for the true reward given a proxy you
will usually tend to overestimate the
rewards you'll get based off the proxy
this is because if for example you look
at a data set you see that x and y which
are the proxy and the true reward are
correlated but near the most extreme
section for the proxy value we see that
it starts to the regression starts to
break apart and it is no longer a good
predictor of what reward we actually
expect this kind of predictor we use is
usually called maximum likelihood it's
quite common in machine learning it's
basically a type of Bayesian estimation
of what it should occur based of the
maximum likelihood is like Bayesian
regression in a way except it puts no
preference for what kind of parameters
your model should use
this results in models that focus mostly
on the bulk of the data set doing well
but models that discard the edges these
bits here whether the regression comes
apart
they're given less they're not given
much um waiting by the maximum
likelihood method Bayesian regression
can overcome this in certain instances
and it tends to avoid this predictable
disappointment here so you can see the
purple line which is the Bayesian
regression is at a much shallower angle
from this we see that we are much less
often at the extreme values of the proxy
to get a prediction that's too high
so from this point onwards we'll just be
assuming that whatever AI is doing it's
using Bayesian methods with a relatively
informative prior so here is one of the
interesting sections so suppose AI you
nose your nose their Creator your
creator is either door Maximizer or a
red object Maximizer but their rate of
returns for how long you stay next to a
door an extra red object decreases with
time there are diminishing returns this
makes a substantial difference to the
behavior of the AI if the rewards begin
to drop off rapidly enough then
eventually if you stay there too long
then a staying one more time step will
result in a very minuscule increase in
your reward this will get smaller and
smaller and smaller in this graph you
can see that this is plotted as the
derivative of the value as time goes
as the more time you spend standing next
to a door or next to a red object you
get less and less out of it for each
extra turn eventually this will become
so small that your expected utility will
say okay now we should switch over to
the other object you will get a slightly
better rate of returns even though I
might be less likely to be the true
reward function
this requires sufficiently fast enough
this requires the rewards to drop off
sufficient and fast enough if they
decrease for example like 1 over the
logarithm of the number of steps you've
stayed next to the object then behavior
is quite bizarre the AI will determine
that the optimal policy is to switch
between objects only after it stays at
one for infinitely long and then it
switches to another which is rather
useless to its human creators so
diminishing returns can prevent this
good heart like behavior we saw in the
maze and get AI is to swap between one
location and another so that's one way
we can deal with good hearts law another
case is that you may encounter some very
bizarre situations when attempting to
optimize value functions say for example
you're playing a game and you are
driving around a track there's a small
circle where if you repeatedly loop
around you can get infinitely high
rewards this kind of perverse behavior
makes it most rewarding option to pursue
obviously we as obviously the humans
would not wish for this if they tell
this is quite a common problem it's the
usual extremel good heart situation or
it may just be that there's a lot of
low-hanging fruit of different utility
functions that are very easy to optimize
compared to the true reward function
true utility and the AI will just
maximize that collection of utilities
how can you as an AI possibly deal with
that if you know about extra more wood
hurting well one way is by normalizing
if you rescale the rewards so that the
very most any policy will give you no
matter the action you take is just a
reward of size 1 and at the very least
is zero then suddenly extreme situations
like this become useless even though it
there is a very long loop and you can
continually rack up rewards it doesn't
get you that far because you're the
words have been scaled down low enough
now instead of the extreme values of
rewards being what dominates it's just
the probabilities that dominate the most
likely utility functions will be the
most likely to be optimized or the Asian
mixture is governed by probabilities and
if there's a lot of very likely utility
functions then
ruff some of those will be what's
maximized but there's another problem
what happens if the true value function
is incredibly unlikely well for example
this might occur through some fairly
simple situations aiu has a sensible
prior it penalizes complexity and basis
places low probability on very complex
situations but a human comes up and
tells you okay human project value is
fragile and extremely complex then that
means that your prior will naturally
result in a very low probability for the
true reward function
what can a IU possibly do well if we go
back to the example of the maze then
what you might try is maximizing the
minimum value that a you are maximizing
the minimum reward you expect you'll you
will possibly get so for example say you
think that it is overwhelmingly likely
to be the case that your Creator wants
you to stand by doors but you suspect
that if you don't stand by any red
things then the whole situation will
collapse for humans it will become
something undesirable so you anticipate
okay I should try and alternate a little
between the two try and give some
resources over to the very unlikely stay
by a red thing this way both a both
cases are covered and both cases would
get some value
and it may be that humans would be
willing to accept this like a more
plausible example is say you are
existing in a world where there's a
trade-off between freedom and happiness
if you optimize solely for freedom or
solely for happiness then that's
probably going to break things very very
badly
you could either be completely free
perhaps say I don't know a series of
dictators or some hyper capitalist
economy with some very strange dynamics
or you could just be wire headed and
just your brain flooded with endorphins
both of those cases are obviously
breakdowns that a human would say yes
that's just horrible and if you ensure
that rather than maximizing only one
thing you maximize some mixture you
ensure this some base level of resources
dedicated to either then it's less
likely human values will collapse but in
general it's not so easy to come up with
a method that deals with the true value
function being very unlikely this is an
open problem according to Stewart and he
says there is quite an important one all
right
so we will switch over to another
situation now where we'll see that
there's still good hard like behavior
but in some cases it's not so bad in
other cases it's quite terrible so for
example say you have a small cleaning
robot that can move between squares its
rewarded for going to the left or to the
right like the maze the to reward
functions that are possible or mutually
exclusive that is to say if you go to
the left one utility function will give
you some reward but the other will not
however - it adds something to this and
you get the second figure essentially
here he said that if your Creator is a
left Maximizer they'll give you utility
one for every time you are on the left
square utility 0.7 every time you were
on the bottom left and utility 0.5 every
time you're on the bottom right if your
Creator is a right Maximizer the
opposite applies so in this instance
there is a trade off the reward
functions are not usually exclusive if
you go to L R or R L you can ensure that
both utility function your Creator is
sure to get some value out of this
in this case the optimum move for an
agent is to move to LR or RL depending
on whichever utility function and things
isn't more likely this is certainly good
hard like behavior you are stuck doing
one thing forever it's very pushed to
the limits but it's not such a bad
option all utility functions against
something is not really much perverse
incentives etc it's a relatively okay
situation in the second figure however
we add two more two more squares that
can give rewards however there's a
negative trade-off a left Maximizer will
give you 1.5 utility if you go to L
minus R and they will give you utility -
not point 1 if you go to our - L the
opposite is true for a right Maximizer
in this case the optimal Bayesian move
is go to L up minus R and stay there
forever
that's quite good hard like behavior and
we might view this as bad because
there's distinct likelihood for the
right Maximizer to just be suffering for
eternity as the robot is stopped
cleaning that square this shows us that
in general positive while not in general
but we might generalize this to positive
trade-offs partly helping with good art
like behavior it cuts the it reduces the
sting of it negative trade-offs however
make things worse and Stewart
generalizes this little by looking at
the shape of Pareto frontiers so Pareto
frontier is defined as you have to
you have some resources that you can
allocate towards certain actions like
say you have I don't know a number of
paper clips you can produce on the
x-axis and a number of pins you can
produce on the y-axis and there's a
bunch of people with different utility
functions saying oh I want you to make
pins I want you to make paper clips with
such-and-such in such-and-such numbers
or in such-and-such ratio a particular
use of resources is called Pareto
optimal if there is no way to shift
around the resources that could improve
things for one agent and not ruin things
for other agents so every single point
on these lines that you see in these
graphs are different ratio optimal
scenarios they if you try to move to
another point on another line then there
is no where you can move to that would
make everyone happy now there's
obviously a few different graphs here
and the shape of the cross is quite
important the left graph is quite nicely
rounded it's convex and in it any
assignations of resources that's Pareto
optimal is going to have a little bit of
something for every utility function you
can just see by the shape of the curve
that is quite far from the origin at any
point there are a fair bit of resources
being
used for each utility function however
because it's curved what that means is
that say if you decrease the number of
paperclips you have over here say you
know sorry this is bins so say you
decrease the number of pins by some
amount then you will decrease the number
of paperclips by quite a large number
sorry you'll increase the number of
paperclips are quite a large number so
for a lot of agents it's quite likely
that you will be producing a lot of
value because you're only reducing the
number of thing number of pins little
but you're gaining quite a substantial
degree of paperclips
right that's even more than I thought
however as you go further and further
and further eventually you hit a point
where you no longer get much much of a
return by increasing the sorry by
decreasing the number of pins and
turning it into more paperclips so
eventually your most of the utility
functions you encounter are going to be
contributing something in pins and
something in paperclips and there will
be very there is no scenarios where you
will just focus solely on paper clips
and just keep unbounded ly increasing
the number of paper clips or the number
of pins this is different in the case of
a flat for a ratio boundary in this case
you
can trade off the number of paperclips
you produce for some number of pins no
matter where you are and say if you
start out here then any other point is
just as likely so the extreme points
provide satisfice as many utility
functions as basically any other and the
extreme utility functions become
substantially more likely to be
optimized by an AI by AI you if the
frontier is curved inwards or it's
concave then in this case extreme values
tend to be preferred because you can
keep for a small decrease in the number
of paperclips
here you get a larger increase in the
number of pins and you can constantly do
this and just drive things up more and
more and more until eventually you have
a huge number of pins everything is just
turning into pins and there's no paper
clips and that's quite an extreme
scenario which generally you don't like
that's a very classic case of good or
like behavior so we see that we might
generalize the earlier comments about
positive and negative trade-offs in
terms of the shape of Pareto frontiers
in more general scenarios and we can use
that as a criterion to check whether or
not we expect
to encounter substantial good art like
problems and I believe that's it I am
terribly sorry for explaining all of
that so badly but if you have any
questions then you will endeavor to
answer them I think you explained things
very very well


Phi-1: A 'Textbook' Model

the importance of the new Phi 1 model
isn't just that it's small enough to be
on a smartphone set to be open sourced
and capable of interview level python
coding tasks its significance is also in
what the model tells us about the future
of language models and the timelines of
our march to human level intelligence I
spoke in depth with one of the authors
of the paper Ronan eldan to get you more
insights and I'm only going to cover the
best bits so let's start first thing to
notice is how small this model is at 1.3
billion parameters but what does that
number mean well for reference that's
about one percent the size of gpt3 which
was behind the original chat GPT
phenomenon and if recent rumors are to
be believed it's about a thousand times
smaller than the combined parameter
count of gpt4 so we're talking a tiny
model here that could fit on my Samsung
s23 we read that despite this small
scale Phi 1 attains a pass at 1 accuracy
that means past first time of 50 on
human eval testing python coding
challenges and Draco pathy of openai and
Tesla Fame said that we're probably
gonna see a lot more of this creative
scaling down work prioritizing data
quality and diversity over quantity
using synthetic data to create small but
highly capable expert models and the
author I spoke to actually retweeted
that and said for Skeptics the model
will be available on hugging face soon
give it a try back to the paper which
says everyone knows about scaling laws
adding more compute adding more data but
following the footsteps of eldan and Lee
in tiny stories which I'll get to in a
second we explore the Improvement that
can be obtained along a different axis
the quality of the data of course anyone
familiar with my Orca video will know
that data quality is super important but
let's get to this paper they mentioned
and I'm going to give you the 30 second
version of the paper co-authored by
Ronan they created a diverse and
synthetic data set of short stories
using GPT 3.5 and qpt4 and then they
train tiny 28 million parameter models
and smaller actually which as they say
are two orders of magnitude smaller than
gpt2 which was only 1.5 billion
parameters and by curating the synthetic
data carefully look at the difference in
results the ending of this story was so
much better on the tiny model trained on
this synthetic data set especially
compared to gpt2 which is so much bigger
but it says the soup is too old it's a
terrible ending to the story so what did
they do for Phi one well here is the
short version they filtered the stack
and stack Overflow to only get the most
teachable bits of code consisting of
about 6 billion tokens they then created
a synthetic textbook consisting of about
1 billion tokens of GPT 3.5 generated
python textbooks that's not even gpt4
then quite crucially they created a
small synthetic exercises data set
consisting of only 180 million tokens of
exercises and solutions now of course
other people have used the stack before
but as Ronan says I do think that from
the data we do have we are not even
close to extracting everything from it
and look at the results of this tiny 1.3
billion parameter model trained in this
way there have been only two models that
have scored more than 50 on human eval
pass at one that's wizard coder and of
course dpt4 but of course those models
are massively bigger and therefore much
more expensive to train and actually I
find this chart perhaps the most
interesting one of all in the entire
paper you can see so many Trends in one
diagram let me try to pick a few of
these out and remember the scores are
the percentage accuracy on human eval
think moderate level coding challenges
first look at the consistent increase
from when you just train on the filtered
stack versus on the synthetic code
textbook from 11 to 16 12 12 to 20 17-29
this could be the synthetic data Event
Horizon that Sam Altman talked about and
that code textbook was generated using
GPT 3.5 not even gpt4 next compare the
parameter count of the models 350
million on the left and in the center
and 1.3 billion on the right this one
isn't as big a surprise we knew that
increasing the parameters yields better
performance but nevertheless you can see
it vividly in action third and I think
this one is really fascinating look at
the difference between the left and the
center charts the only thing that really
changed was the number of GPU hours and
of course the number of tokens went from
26 billion to 76 billion but wait I
thought the data set size was fixed at 7
billion what gives well of course what's
happening is that they're passing over
the data multiple times this is called
training for more so-called epochs or
passes over the data so these aren't new
tokens they're the same tokens being
trained on more times as Ronan said to
me my personal impression is that many
people in the community thought that we
would never want to do more than like
one or two epochs because we'll start
overfitting and just for 20 seconds I
can't resist bringing in this paper that
they referenced in the textbooks paper
it's essentially talking about how you
can still scale language models even if
you run out of data and take a look at
these two diagrams they say training for
up to four epochs or passes is almost as
good as new data and it's only when you
get to around 40 epochs that repeating
is worthless obviously we don't know
about gpt4 but GT3 seems to be trained
on far less than that but there was one
final trend from this amazing set of
charts that I wanted to point out and
it's probably the most obvious one look
at the huge jump to the dark green bars
that's when they train the model on
those additional synthetic exercises
with Solutions the authors know that one
can only imagine how frustrating and
inefficient it would be for a human
learner to try to acquire wire coding
skills from such data sets like the
unfilled stack as they would have to
deal with a lot of noise ambiguity and
incompleteness in the data we
hypothesize that these issues also
affect the performance of language
models as they reduce the quality and
quantity of the signal that Maps natural
language to code let me quickly give you
a bit more detail about how they
filtered the stack they got about a
hundred thousand samples of the stack
and stack Overflow and then prompted
gpt4 to determine its educational value
or a student whose goal is to learn
basic coding Concepts they then use
those annotations to train a random
Forest classifier that predicts the
quality of a file using its output
embedding essentially a basic searching
mechanism to find out which parts of the
stack are the most educational but at
this point I want to pause and imagine
if they'd used a different prompt
imagine a future paper looking across a
different data set that paper could
prompt gpt4 to annotate the educational
value for student whose goal is to learn
French then you could have an amazing
French speaking model or maybe they
could get it to annotate which examples
would be most educational for learning
to predict the stock market and then
maybe train it on a small synthetic
textbook of successful previous examples
of predicting the stock market I'm just
saying this seems to be a model that
could be applied elsewhere and these
annotations here were the only times
they used Gypsy 4. the rest was GPC 3.5
and as Ronan says gpt4 is not only great
as something we can use directly for
better productivity but it's also a way
to get much better other models and
that's one thing I want openai anthropic
and Google to address the capability of
their models to train smaller models
here by the way is an example of the
kind of exercises and solutions that the
model was then fine-tuned on created of
course by GPT 3.5 and the authors note
that quite remarkably the model after
fine tuning on those fewer than 200
million tokens of exercises and
solutions also exhibits a substantial
Improvement in executing tasks that are
not featured in the fine-tuning data set
for example fine-tuning on codex sizes
unexpectedly improves the model's
ability to use external libraries such
as pygame even though our exercises do
not contain these libraries this
suggests that fine-tuning not only
improves the tasks we targeted but also
makes unrelated tasks easier to distill
it's this unexpectedness that I find
really interesting for example before
training Gypsy 4 did they expect the
emergent ability to do self-repair or
reflection according to this new paper
that ability is not found in GPT 3.5
going back to the Phi 1 paper the
authors admit that there remain a number
of limitations of our model compared to
larger models for code firstly Phi 1 is
specialized in Python coding which
restricts its versatility compared to
multi-language models secondly Phi 1
lacks the domain specific knowledge of
larger models such as programming with
specific apis or using less common
packages it's a bit like the more
classical narrow AI good at only a few
things furthermore due to the structured
nature of the data sets and the lack of
diversity in terms of language and style
it's less robust to stylistic variations
or errors in the prompt it's quite funny
if you make a grammatical mistake in
your prompt it does a lot worse but what
about this we also believe that
significant gains could be achieved by
using gpt4 to generate these synthetic
data instead of GPT 3.5 as we notice
that GPC 3.5 data has a high error rate
I asked Ronan about that speculating
that it's because gpt4 costs more and he
said yeah it costs more also gpt4 is
much slower but another reason is we
wanted to demonstrate something here
that you don't even need a smart model
like gpt4 even GPC 3.5 which isn't that
great at coding is enough so there you
go you could get even better results on
this using gpt4 but at the moment Gypsy
4 is a bit too slow before I get two
timelines some of you might have noticed
the wizard coder results and wondered
how that model did so well despite only
being 16 billion parameters which of
course is 10 times bigger than Phi 1.
well of course I read that paper too as
well as almost every paper referenced in
the textbooks paper The Secret of wizard
coder seems to have been increasing the
difficulty of the training data fine
tune the model with more difficult
examples EG if the original problem can
be solved with only a few logical steps
please add more reasoning steps maybe
complicate the input or deepen the
question or increase the reasoning
involved you can start to see the shared
themes of orca wizard Coda and Phi 1.
this could be what Sarah Constantine was
pointing to in the asterisk magazine
that I read yesterday I'm not sponsored
by them but it was a great issue so do
check it out she said rather than a
refutation of scaling laws or an
acceleration of their slope I think this
is more like a move in a different
direction altogether towards a Cambrian
explosion of little AIS used for
different purposes where getting good
performance on a task depends on the
quality of your task specific data set
like Phi 1 for python that could be
consistent with the state-of-the-art
continuing to progress steadily along
scaling law lines for quite some time
but it could also mean the economic
incentive towards ever bigger models
would diminish and would enter an
entirely New Era where AI progress would
not be driven primarily by semiconductor
scaling or Moore's Law this relates
directly to a tweet from the co-founder
of anthropic Jack Clark he said a world
where we can push a button and stop
larger compute things being built and
all focus on safety for a while is good
that is really interesting to hear from
someone at the top of an AGI lab but I
do have some questions for this policy
if we freeze compute wouldn't that
incentivize every company just to use
algorithmic progress to get more out of
the compute we do have and so on the
safety front I think it's far more
effective active public messaging to
focus on concrete things that everyone
can understand for example in this paper
from Oxford this week llms will in
particular lower barriers to biological
misuse biological design tools will
expand the capabilities of sophisticated
actors concretely bdts may enable the
creation of pandemic pathogens
substantially worse than anything seen
to date and could enable forms of more
predictable and targeted biological
weapons I think this is something that
everyone can get behind and as the paper
says it's been hypothesized that for
evolutionary reasons naturally emerging
pathogens feature a trade-off between
transmissibility that's how much they
spread and virulence that's how deadly
they are AI based bdts might generate
design capabilities that are able to
overcome this trade-off thus for the
first time humanity might face a
security threat from pathogens
substantially worse than anything nature
might create including pathogens capable
of posing an existential threat that to
be honest is my main safety concern
about back to the paper and timelines
here is another snippet of my
conversation with Ronan I said I just
feel like we are much closer to
something really transformative than the
public has quite realized and people
like open AI put out that in 10 years we
will have something as powerful as a
corporation I say three to five years
Ronan replied that depends on how much
resources are actually spent into
training bigger and bigger models I have
no idea what openai and Google are doing
right definitely if this is our main
goal I think it can easily be five years
I said or less Ronan replied or less I
feel like the bottleneck is maybe the
production of gpus and I mean it's not
just to produce the gpus you also have
to build the data centers and connect
them to electricity etc etc I think if
you have all that then yeah I don't see
the barrier with more data higher
quality data synthetic data better and
better algorithms and more and better
gpus and tpus that's what we mean when
we say we don't see a barrier of course
everyone has slightly different
definitions of AGI but almost everyone
agrees that the next five to ten years
are going to be the most critical in
seeing whether more data better data
better algorithms or just more and more
compute will lead to AGI or super
intelligence I loved how Carl Shulman
put it on the dwarkesh Patel podcast if
you generate like close to 10 million
dollars a year
out of the future version h100 and it
costs tens of thousands of dollars with
a huge profit margin now and profit
margin could could be reduced with like
load production that is a big difference
that that chip pays for itself almost
instantly and so you could you could
support
pain 10 times as much to have these Fabs
constructed more rapidly you could have
if AI is starting to be able to
contribute could have ai contributing
more of the skill technical work that
makes it hard for say Nvidia to suddenly
find thousands upon thousands of top
quality engineering hires if AI can
provide that now if AI hasn't reached
that level of performance then this is
how you can have things stall out and
like a world where AI progress stalls
out is one where you go to the 100
billion and then
over succeeding years trillion dollar uh
things software progress
um turns to start turns out to to stall
you lose the gains that you are getting
from moving researchers from other
fields lots of physicists and people
from other areas of computer science
have been going to AI but you sort of
tap out those resources uh because AI
becomes a larger proportion of the
research field and like okay you've put
in all of these inputs but they just
haven't yielded Ajay yet I think that
set of inputs probably would yield the
kind of AI capabilities needed for
intelligence explosion but if it doesn't
after we've exhausted this current scale
up of like increasing the share of our
economy that is trying to make AI if
that's not enough then after that you
have to wait for the slow grind of
things like General economic growth
population growth and such and so think
slow and that results in my credences
and this kind of advanced AI happening
to be relatively concentrated like over
the next 10 years compared to the rest
of the century because we just can't we
can't keep going with this rapid
redirection of resources into AI That's
that's a one-time thing thank you so
much for learning about Phi one with me
and as always thank you so much for
staying all the way to the end do try to
have a wonderful day