AI Safety Investing | Luke Nosek, Gigafund

um from now on the sessions on recording
and with the potential option to publish
that doesn't mean that it will be
published but we'll be reviewing and the
option of that because I think that you
know the topic that we were going to be
tackling today is definitely I think of
interest and the message that we're
hoping to get to be so much further and
a lot of investment types folks who
aren't in those generally so I'm I'm
thinking that it could be useful to
record it alright great so look a few
words about you and I mean there's
really quite a lot to say I think but
you know as a very quick summary you
found it Giga fund and we were just
where you currently were you're
currently doing investments through
after having previously co-founded and
founders fund with Peter Thiel and can
Halloween who you originally co-founded
PayPal with so M you have quite an
interesting history already your main
interests throughout and this time
really has always been in companies that
you really believe in will have the most
significant impact on the world and you
serve on the space export of Directors
ResearchGate
and you also where a director which I
didn't know before actually is the first
seed investor that's crazy okay I had no
idea I also for example didn't know that
Ayana stem or invested in it as well so
they've brought on the board okay well
um I think that's it's quite quite a
strong and quite a strong segue into the
discussion but I think you notice - and
kind of round it up I think you really
really care quite deeply right about the
long-term future for life and I think if
anyone really has a physical perspective
on how we can leverage the private
sector to do that then then it's will
you so thank you so much for for joining
here and I think you know maybe actually
we could start on that that note on on
deep main what what got you interested
and and and kind of
and engaged and defined in the first
place and how was it and to get to get
yon involved well I've been looking at
different AI companies every once in a
while it was an interest area of mine
from from childhood but none of them
were run by people that were any good
CEOs and Dennis was the first one that I
met where there was there was
technological promise and I had heard
about the deep blurring technology maybe
a year before a deep mind but there was
also an incredible CEO like we yeah we
thought Dennis could could could
actually pull it off actually I thought
it was such a strong possibility that he
wouldn't be the one to pull it off that
I heard debating this sometimes we
couldn't go see the investments just see
how something goes we do this thinking
fund as well quick a small amount of
money it's in the hundreds of thousands
and we're not really that confident the
company's going to know but in this case
we invested a million pounds and I also
joined the board as a seed investor
which was Faris was unusual and it was I
believe not recommended by my other
partners to go commit this much the
company but I thought it was important
enough there was a chance it can be very
very important and so I was willing to
do the transatlantic flights and focus
on it and that gave me the Ames wants to
do things like that yeah I think we're
all quite glad and you know given that
deep mind is you know one of the main I
guess one of the main I think hopes for
developing a really strong and
beneficial AI would you kind of like
care to say a few words about how you
got interested in the kind of like AI
safety or and how AI videos long-term
with space in the first place brother
you know my view from when I was a child
was the naive hopeful vision that it
would just be helpful to us you know
it's not gonna be something if there was
a safety she looked me simple he would
be like like as mom's laws were but
they're just rules that don't hurt
people and so then I kind of forgot
about and then I got involved in the
kind of the extra peon and the
singularity movement and these these
groups I should believe there was fun
you know I think that's what marking on
the technological progress would just
enable everything to be great we'd have
crypto everything in AI everything and
humans would be free and prosperous and
happy but it's um I started really as of
housekeys writings that there could be
risks to technologies and to be fair
chiefly expressions first of all I
completely endorse your description and
I would say I'm still in that camp but
it's not that we didn't worry about the
dangers or explore dangers and talk
about dangers we did but overall we were
technological optimists and I still it
yeah it I was also overall technological
optimist I don't know what I am now
maybe we'll get to that later um but
then it wasn't actually until it wasn't
even quite getting involved in marry
that got me really focused on I think it
was when I began to see that there were
companies and a lot of countries like
deep line very reasonable and it was a
few of them so it wasn't just well maybe
this one guy will make it work but
actually oh my god someone's gonna make
this work and it's like soon like in the
time I have an intuition for seeing
things in the timescale of a venture
capital which is ten years and so it
seemed I
if my intuition was right that's telling
me that's right that was wrong
thank goodness but that's what when she
started telling me and then I got pretty
serious all right okay um I think um you
know you you're you were interested I
guess in in AI is like one of the one of
the the things that you know could could
tip us over and I think actually yang
also mentioned yes as eleazar's writings
as a kind of like tipping point that
kind of made him chip into the risk risk
aspect of of this as well but you know
before wanting to kind of like sidetrack
kind of the whole discussion about
covered just yet but what how is your
thinking on risks updated since then has
anything changed since kovat 19 has your
thinking on a I risk changed since that
specifically it has I think it's it's
not directly related but what it shows
us is it shows us it's like a stress
test for our societies or particularly
the law conference see how they do with
a you know society-wide written mildly I
don't call this like become cold of like
existential threats okay and we're not
doing so well like I would view my our
society were going to be this is not so
negative but like you know my father's
like in a really really bad health and
he had get a flu recently and he didn't
do so is hospitalized and respirators
and all of this stuff it was like one
step shy of the rest of the need to put
the ventilator on so he didn't do so
well with the flu I have the same flu I
was like you know sick I'm throwing up
like the sex buddies later fine our
society versus say like other societies
like Singapore and China and how they
dealt with this this is really
concerning and we'll see I I'm hopeful
on this that will do not too bad
but that's that's what causes me some
concern if they're hung first with
planning and then responding to it it's
just all not that impressive so that's
that's one part is just making me a
little more worried about what the state
of things are the private sector and
like did it make you change your views
on how the private sector and can what
kind of role that could be playing in in
cases where you know we suddenly
realized the world is it's kind of like
vulnerable to a furious in certain ways
and we can a little it's a little
disappointing in the private sector I
would have this wasn't my area of focus
but I had assumed that there were other
people who have this as their area of
focus and heal I've been building up
companies which founder on companies
were number decades with a mission to to
carry infectious disease or to prevent
pandemics I just didn't know a lot about
it but you know not that much has that
so a lot of people who were dedicated
emic researchers some people in
government they were very dedicated
people in nonprofits that are but um I
didn't see is much a private sector
preparation for it
I was just thinking today you know what
one of the companies I respected the
most is Amazon and you know just hiring
hundred thousand workers now really now
you mean data in real I have somebody
looking at the are not and the spread
from China before then and then realized
well they're gonna be sitting at home
buying things just like they are in
China hoping I buy it from us maybe we
should have some people to deliver stuff
I'm an Amazon living like two second
weeks okay yeah yeah in the point second
yeah and I think you know this is
something that was also echoed by a few
others and actually also something that
came up in Yan session and because he's
been trying to compile like a document
on how we can use this and kind of like
this risk
this risk is an opportunity to actually
be doing much better and to be growing
like you know a stronger immune system
going forward you know so I think that
they're really right now this vacuum and
to step into into better agency in the
future right about that very soon as
well yeah yes okay I'm hope I'm hopeful
for that still ongoing yes all right
great and I think I think South Korea in
particular I think that one is going to
be I think for the next decade we will
all be looking at what happened in South
Korea and why did it go so well yeah
what did South Korea do right that we
can learn from but I think one thing in
particular is this thing about learning
from other forms of the same kind of
danger is they went through SARS in a
big way this is our SARS experience yeah
okay all right so and we may only get
this one shot okay is it going to be
something that teaches us if this is our
start I've heard it so many people who I
know who are in in front from the East
Asian countries are we gonna have that
same type of lesson or you know hor
meiosis type response to this or
actually or like some older videos like
my father when we come out worse and
this is what I'm looking for yeah and
for AI this is what's gonna tell me oh
man we're really really screwed if we
come out worse from this and and maybe
like we just this is not a discussion
okay we Came 9/11 so but you know I
think as I said in the in the beginning
I do think that Milton Friedman wrote of
you know that only a crisis actually
perceived produces will change when
their crisis occurs the actions that are
taking depend on the ideas that are
lying around I think you know is a
really powerful reminder of like this is
a call to action and not a call I think
to get quite so passive so and with that
as a Segway perhaps you know to talk a
little bit about what you at Google are
doing so you know a dear friend I think
it's really really prominent on on your
website it said you really have this
kind of long-term commitment and this
kind of like commitment to this more
paradigm-shifting long-term success and
you really make a big point of that you
only invest and founders that you really
believe and will be running their
company for decades right and you
actually really insist on meeting them
in person and apparently also sometimes
joining their boards and so you know I
think if you could kind of like give us
a few tips of like and if you were an
investor considering what kind of
company and you should do you invest in
what are you looking for in the founder
that will make you think that a and they
are capable of running a company and it
would be successful and B what are you
looking for in them yeah actually this
sum this example of this situation
health-wise and economic that's
happening is great because one thing we
look for is some founders who who
planned for the worst so we look for a
gig of fund is founder school persevere
for decades but what does that take the
pursuit of decades where you can't die
so you need to have people who plan for
the worst things that can happen in a
very long period of time not just you
know what's going on in a kind of
venture capital upswing which which is
sodium is many many of so when we dig in
with founders will often find that the
scenario planning that they have and
they don't just think of as theoretical
exercise they're really ready to execute
on them on extremely difficult
situations companies being on money or
[Music]
product failures technical failures they
have ways of surviving through all of
them and then as we get to know them
for time we could see what their
responses are you could see like how you
know for instance SpaceX made it through
2008 really increased mine contents Elon
Musk tremendously it's such a difficult
economic period and yet they were able
to you know not just get the rocket
working but also get a huge contract
from NASA and and keep the company
funded during an incredibly incredibly
difficult time so look at the kinds of
companies respond to this some of our
companies are becoming stronger in their
EE remote work they're becoming more
efficient you know they're getting Rajal
in their their financing plants like Oh
some of them this is what we're looking
for actually so it's it's if you're
aware of this as an investor it's a it's
a great time to invest because otherwise
you'd have to invest and then to kind of
wait for them didn't with SpaceX when
they invested in the summer of 2008 and
then as I saw the different difficult
increase my confidence like doubled and
tripled our company alright and I think
you know that's like a hell of a lot of
a good point I think to figure out what
makes companies successful and do you
think do you have any let's say like
ethical you estates or something that
you apply to to seek out funders that
you think are going to be going to be
going to be doing work that is long-term
successful and what do you think about
if those two are ever in conflict with
each other
right because there may be you know
every company that we're building
company that were working on with the
founders is is really that like it's
like a personal extension of them and so
we want to get to know them and why
they're doing it that's a question we
learn to ask a founders fund when we
really begin to invest in this with a
very similar way to keep the fund we ask
them why do and we want to get a very
very deep understanding of why they're
doing it
all right and do you have like if you
post a child's I think you named even
masks as someone who you believe in and
I can like really kind of like sail the
ship during a crisis do you have someone
of like a founder that people can be
could be looking out to as a role model
for like a really strong ethical good
commitment to a positive long term
future
like if you'd had to point pin point to
a role model or have you seen sometimes
you know even the opposite have you
sometimes seen like you know the let's
say what you thought was a safer way of
creating AI or a better way of creating
a I being conflict with business
interests and if so was that handled
well or poorly you know we've only made
two significant AI investments one of
them is deep mind otherwise luminous
computing here at Canyon in both cases
their approach to managing safety and
ethics was different from both cases it
was very very thoughtful and they were
quite quite intent on it
and could you say a few few more words
about Lunas I think people here are
familiar with with deep mind but perhaps
not so with luminous well I just say
what would be a public building that
will be a thousand times oh man okay but
the TPU team left it shocking
okay good make sure I understood that
some of the people from the Google TPU
team are now luminous we have hired a
lot of people from top electronic chip
design companies and top photonics
companies as well I just got an update
today on some progress okay when okay we
can get excited then and thank you and I
think that you know kind of taking it
back to when I pondered your effective
altruism bio for a while and I think one
thing that really kind of perked out of
me there is that you know you stated and
you are interested in a realistic and
critical examination of political
systems and ideology as well and I think
you know that is a pretty comprehensive
I think approach I'm going to also have
as an investor so I think it's quite
interesting because it's more like
ecosystem approach so and what do you
think you know in theory and you know
for AI to go well what role you know do
you see nonprofits for-profits and
academia playing in like an ecosystem
that goes well what problems like each
of those kind of packet is facing and
could they learn from each other you
know I think I think that the role of
the for-profit ecosystem is you need the
primary driver of of action I think that
the nonprofit ecosystem is useful where
there's certain kinds of public goods
like just thinking about
kinds of fundamental research and the
government's important to stop us so
generally I lean pretty libertarian on
it and you know that's the way that I
would wait those three areas and how
much I would put into them kind of
depends on the competence of each of
those areas in that society first if
you've got a really incompetent
government well then you probably have
it done by the pine sector mostly it's
too bad even that depression it's not
good at like stopping itself it like has
a race you know condition problem but
maybe a combination of the private
sector in the nonprofit sector or
something like that just seems to what's
happening here in the u.s. at least but
the ideal thing would be one where all
of those different types institutions
would be very very strong and they could
play their role okay yeah I think you
know that that that you know that is I
think a the perfect way I think how it
would run in theory that being said you
know do you see that there is actually
like a role to play even for companies
for example by oh by by investors and to
enable for example companies like
alternative options to let's say
military contracts or something with the
government by investing in them or do
you somehow see kind of like as an
investor and that there is kind of like
a role to play and kind of like playing
off the different sectors against each
other in a way that is kind of
geopolitically more safely enhancing how
do you mean that question do you mean a
company could choose where the products
end up stuff like that well more in the
sense that you know like um if many
companies are relying you know on
military contracts I think in for for
really a lot of their revenue and I
think as an investor you know you could
almost create an alternatives there and
if you were if you cried out fistic and
say hey listen up you know
we are going to we're going to give you
funding regardless of what what kind of
strings strings you would would have
been tied into with the government I do
and this is a contrarian to you on this
I am not as concerned about the military
contracts as long as they are given to
very pro AI safety companies yeah
that will ensure technologies use what
I've seen happen is at some point the
contracts will go to companies that are
you know just purely profit motivated or
it maybe means worse I you know there is
such a thing but you know and I have and
I have seen companies like SpaceX and
and volunteer I've seen them
successfully navigate that contracting
system and then delivers something that
is a net positive and I would say in the
SpaceX case it's a lot simple it's
simple it's just we're gonna do it at a
lower price and we're gonna kill fewer
astronauts don't you see a coward why do
you see Palantir is a net positive I
looked at Palantir with her and her view
is a net positive because we do have the
ability abroad surveillance right now
and if it's if we don't have the ability
for targeted surveillance there gonna be
two things that will happen one we won't
actually catch the bad guys Phillie of
9/11 love another 9/11 I can't even
imagine what's gonna happen the next
time by the way they actually believe
the reason it hasn't been in 9/11 is
because of the power tier
enough of the people early on that okay
today not all of them we've had some
terrorist incidents but they come on oh
yeah so that is the best present really
to be good for our geopolitical
stability and the lives of these foreign
citizens secondly for civil liberties
you know the broad net you know imagine
you have someone just not even they have
like no software tools whatsoever which
actually is the case suffers so that
well they don't just read through like
every random thing and follow every
random lead and be looking through a lot
of personal info with innocent people
but the more narrow you can make the
search and the faster you can get the
analysts the result that they need the
less private informations can be
trampled on so it's a harm minimization
approach it's I prefer to live in a
world where there isn't the you know the
snow and stuff but that's what we have
yeah so harm minimization and maximum
maximum civil liberties yeah I think one
thing that Mac and I have been toying
around with is how can you do
decentralized surveillance systems that
are based on encryption we're all
monitors also wanna are being monitored
but other monitors but I think you know
we're not quite there and there's a cool
aspect in the Palantir software that's
like this which isn't I don't think has
the proper cryptography in it that mark
would be very impressed with it but I
it has logging built into all of the
inquiries so you can validate if people
spied on you appropriately they did it
and nobody knew because they just
deleted the Excel files afterwards you
know so so unforgeable krypton
cumulative cryptographic hashing and all
those kinds of safeguards I hope so I'm
not close enough to it to get myself in
trouble well
the things that I mean just one of the
things that you can build even retro
actively having accumulated those logs
is you can retro actively build zero
knowledge systems that could prove for
example if the government wanted to
prove we did not spy on person X because
they're being accused of spying on
person X illegally you could build zero
knowledge proof systems that proved that
not that they didn't spy on them but
that the log that resulted in this
cryptographic hash which I they had
already committed to that that log does
not contain record of them spying on
person X you could prove that that
without revealing anything else about
the law that's the what the zero net was
really really cool and it would only
work if Palantir beat out every single
other defense contractor and became the
only source the only source of
information actually to a point of AI a
safety work a lot of people favor the
singleton approach for the monopoly
approach like we're gonna get there
first and you know and will be the will
control the whole thing by being there
first and this is you know this has
[Music]
yeah goodness in the absence of a
competent regulatory approach I think
this might be where we just actually end
up because people will end up competing
and I think this is no argument I see
that it's even better but I would prefer
an approach where there can be multiple
takes out this okay so this is just an
interesting thing that happens with with
a company that does have a very positive
mission and everyone else is doing a
terrible job and and I think the right
thing for them is to go ahead and just
trying to win yeah I think like we
talked about that somewhat in the book
draft that recruiting writing but I
think um you know we're trying to talk
we're talking a lot about compensating
dynamics so how can we how can small er
actors actually create compensating
dynamics by which they kind of
counteract the comparative advantage and
that larger actors accrue as they're
growing and and and as they're having
economies of scale and you know I think
that you could also even make an
argument that companies could do that by
even kind of like by providing some
counter may in some counter weights
against large governments so if you have
a lot of cross cross jurisdictional
collaboration across companies and it
within the private sector that this
actually creates a a better like let's
say anti fragile system because the
power of individual governments will be
somewhat constrained and because you
have this kind of like private sector as
safety net almost build up or give any
thoughts on this do you mean having
companies essentially cooperate in
building somebody that open-source
cooperates to you know build up value
yes yeah open source is the clearest
example one thing that you know I've
seen is that if you have a project
that's purely within a company then the
company
you know then you're always in fear that
the company will suddenly make a
management decision to shut down the
project or to completely change its
goals so one thing I've seen I'm sure
you have as well is sometimes projects
are collaborations between companies or
concerns or whatever and then each of
the participants to each of the
companies has much more confidence that
their company won't shut down their side
of the project because then it's a
visible shutting down of the
collaboration for other companies
yeah yes I think that's it's quite
interesting and um all right so I think
you know I think another thing that you
mentioned is also that I think this was
an interview that was known UEA bio but
as talk to you all over the place so
another thing that you're mentioning is
that you know you are kind of like a
little unsettled I guess that the
Silicon Valley ecosystem has become a
quite expensive beautiful startups and
and then also I'm kind of making is
increasingly showing institutionalized
thinking so I was wondering you know and
what do you mean by and by
institutionalized thinking and and what
what kind of effect do you think does
that have on on creating safe and
beneficiary I yes so here's I mean when
you see companies believe that they have
to make a certain business because it's
the popular business with the VCS or
it's the acceptable way and I have one
of these just recently a few weeks ago
like during I was like wow you cannot be
doing this during the Cova crash reality
because they're all gonna go wait so
it's quite interesting right people are
so stuck there is so in silken denies
institution it's now gone they still
continue to act as if they need to
please these people know you just
preserve cash until they you know a
sensible business and that's what one of
them is that you know building
businesses to please VCS who are making
investments to please their
institutional investors like smart
outcomes and there could be other yeah
there can be it it's harder and harder
as you
this high high social density in Silicon
Valley it's harder and harder to break
out from the pack you very clearly
identified very pretty quickly punished
on Twitter or somewhere else for being
outside the pack when the the density
was low you this was less likely to
happen to you there's more famous to
very weird
companies pursuing extremely different
approaches yeah I kind of agree I'm only
having been there for five years now
it's it's kind of starting to to sink in
the movement okay and I want to give
some time for all the participants here
that have been so patiently waiting I
think if you want to raise your hand at
the raise hand button or just hold it up
in your video and then I'll monitor and
try to try to catch you a question if
you have any I also have a few more in
petal if you prefer me to go along with
that you can't one more thing on this
situations yeah yeah
the other thing that's screwed up is the
pricing the real estate so people that
are innovative of it work in the garage
or like maybe just off with her your own
savings and that's some a lot hard to do
and they think it's fine because huge
venture capital funds have been raised
to pay your rent but you're not as free
with this money as you are with your own
money
you know this is a big loss so all of
these things clove it might be great
thing for Silicon Valley I think I'd
left you know it's I think you know I
hope it it can it can prove itself but
all those lines by the way I want to
applaud zoom at this point they've
probably already saved millions of lives
yep applause applause and I think we can
even do that by our reaction here I'm
giving zoomin applause right now yes
okay
okay well am I missing a question here
and if not then
go ahead and just these interrupt me or
just as oh wait I saw him get some Oh
sano-sama zhamuhe yes okay PCs are very
busy on Twitter that's arguably because
most of them are not that busy elsewhere
so why has it been that aside from
basically conveying the information to
say a broader audience the way Balaji
has there's been actually very little
you know very little forward motion in
terms of Silicon Valley just
unilaterally organizing to do anything
like you probably hear stuff from your
colleagues talking about oh maybe we
should do something maybe we should do
something else and it's like the world
has really shifted a lot of those funds
will just not exist in a year or two
years so why do they seem to be able to
process the information receive it from
the weird corners of Twitter but not
translated into action into new we're
they're useful projects like the
time-sensitive ones like say Matt Palmer
who just you know went out there is is
now manufacturing the n95 masks whether
or not he gets permission to do it it's
great I'm really glad people archives
I've taken that action it's
and I personally experienced this
throughout this site I saw this ahead of
time and then I noticed how difficult it
was to take action early on even seeing
obvious things the the social pressure
is much more stifling than people
realize and and this is also extends to
them that the social behaviors were
supposed to be doing to make the
negative effects of the pandemic like
with a chat earlier that Nicole
organized with some friends to discuss
like Freudian psychology and half
chatting you know like Kovach things one
of them people
it was so why was it so difficult
someone from China there and oh you just
do X Y is easy to do and just do them Y
and Y is so difficult it's the social
pressure you faced from putting a mask
on no one else is doing it recently
slightly just slightly different than
anybody else is way more massive than
anyone wants to admit
it's way more mass and you need kind of
a new like cultural shelling points or
something like that needs to amass a
little you needs to amass a little
before people can start to move and it
this you know this one didn't it moves
so fast there wasn't chance meant to
happen like you can it can only happen
at the maximum rate of cultural
adaptation and so fortunately move is
from private to community knowledge was
the difficult one then and the community
social expectations yes it's a it had
there's a certain speed at which those
those things can kind of happen and it
was slower than the marks well I'm going
to say I only am asking this difficult
question because V sees actually did
shockingly well compared to any other
mainstream class of American society
including Wall Street right like the
market was not pricing any of this stuff
in until you know until basically
everyone knew so in a very real sense
pcs were you know they outperformed New
York let's put it that way not be fair
yeah but collectively I think I'm
disappointed
all right um okay I had David see his
hand up yeah yeah so Luke say Adi
startup approaches you looking for
funding how would you know if it's a nut
positive to found them or not yeah it's
a good question
[Music]
the company is going to work then it is
almost always a net positive because you
want to keep your friends close your
enemies closer
if you can handle it
so so you you would to me that sounds
like you food found or any ages start-up
that approach to see I mean if the show
didn't see any age successful scary you
know
[Music]
alright and as their another question
here the audience monitoring all of you
know okay then I'll go with another one
and so I think you know we've been like
whatever I like I do
those conferences even like I can't help
but think oh my god if we were actually
taking this seriously we will be doing
different things and you know I still
feel like so often you go to a
conference or to a meeting and I'm
hosting home at those meetings where you
know people are presenting something and
it's like nice you talk about it and and
and and it's good but you know it's
still kind of feels like we're circling
the thing and you know there are those
certain kind of like I guess like
packages like conferences or like
companies or kind of like prepackaged
things that we used to am kind of like
do a certain function but sometimes you
know I wonder if that function couldn't
be achieved much better if we were
actually as serious about the thing as
we always say we are and and so I'm
wondering you know if you could do like
a tabula rasa and it like know kind of
like denominators such as conferences
and companies and you know the way that
your social expectation hold you to
engage in those things like you know do
a research review or something and is
you know could you come up with like
better more effective ways of just you
know kind of like coordinating
coordinating ourselves to come up with
kind of like features that get at this
kind of like really insane future that
we can all hold in our hands and but
that you know we seem to be connect and
strapped into a little bit further away
from well it's um one thing that I've
been
yeah if really what inhibits people a
lot of the time is the social pressure
of the community they are in what if we
could get a better understanding of how
that works and what that is to give
people some more flexibility in action
and I'll just give you an example of the
families fun culture for instance which
is that you know the best investing
culture and of course the Gigot fund
that I've ever worked in there there's a
loud a really unique type of independent
thinker to thrive it which could not
thrive in a normal venture capital firm
well you there's a lot of fitting in and
pleasing other people and showing you
have a hot deal and like a lot of
different problems so you could kind of
structure the the group interactions in
a way that in this case it would have
just the outcome of like really really
good thinking and then occasionally some
good action on investments that came up
thinking it was kind of a low bar you
know you didn't need
sustained execution over decades I'm on
that thinking yes didn't get it right
you know episodically so for some of
these other projects that's that's what
I you've been able to solve for in my
life so far
I I don't know how to extend that but I
think there's something to that where
perhaps we need to build a different
type of community in which were maybe
measuring each other fitting in in in
slightly different ways and that would
enable us to take action on the much
ones longer-term impact things all right
yeah thanks I think correct me if I'm
wrong here but I think I read in 0 to 1
that one thing that Peter liked about
you was your at least initial interest
in cryo and like in cryonics despite and
that was the time when and I think when
it was I think much more early and
father sort of in and and he took that
as I kind of like indicator that that
then you not prone to and to group her
thinking that much is it correct I think
I'm this is this is what is really
interesting it's one of the great things
about Peters he was not just create an
environment for those people he was
looking for those people that were you
know will do something that was
obviously even if it was a bad idea
eventually I'm signed up now I signed up
in 1987 so my theory about what like why
cryonics doesn't work yes I that we need
to create an environment for more of
those few people to kill okay and to I
just share their ideas and to be able to
act Oh long term that's gone it's really
able to do the first two parts of the
pseudonym is actually
right here because it was the
pseudonymous Twitter accounts that made
the you know very strong and
organization critical arguments like
right now I think people feel free to be
online who they are but it's really like
the last gasp I think you know I've seen
many people who's who show real and
justified fear even expressing
themselves under pseudonyms right yeah I
mean we actually discussed this in a
previous session where mark was was
having some suggestion could you give a
keyword on that if people want to look
it up the main thing is that we've got
with modern cryptography we've got
pseudonyms but strong unfortunally which
is to say protections against
impersonation so then repeated
electronic interactions repeated
postings whatever under a pseudonym
causes a buildup of reputation a
reputation feedback there's we need to
be careful with synonyms that there's an
unlimited number of synonyms pseudonyms
afraid so the encrypted is known as the
civil problem you can't trust something
just because it claims to be a new
identity you can't invest trust her
identity the trust has to be earned but
once it is earned then that valuable
reputation itself creates an interest by
be the entity that holds it to continue
to earn the positive reputation or he
loses the accumulated capital of other
people's trust but look I think and
somehow we're getting it was more that
there is a disconnect between I think
that pseudonym and then real world
action or like you know coordinating in
person
is that correct Luke um actually no I'm
not um that but that even the that that
as well okay but also that the pseudonym
isn't perfect oh yeah because you will
leak out information
- the pseudonym and yeah when people
nitrite there's also a social aspect to
this you've got to really notice your
psychology you if your intention and it
could be a hidden intention which is by
the way these intentions people are not
aware of is to let people know something
about you and you need that to be
personal
you don't leak it out regardless of the
technology yeah - yeah
Satoshi is an extraordinary example
absolutely yes okay Chris you had your
hand up for a long time yeah no problem
I just wanted to talk to return briefly
to some of the security relevant topics
that we're up I'm you know working on
the pen working at the Pentagon now and
this sort of howdy how do you form the
government side right how do you get
companies from Silicon Valley and
innovators to sort of consider working
with the governments and how do you like
mitigate some of the ethical concerns
that companies may have so I was
wondering Luke if you would talk a
little bit about you mentioned you know
some of the feeling like talents here
was a success case in SpaceX and
wondered if you if you talk a little bit
more structurally about like how do you
think that conversation is going I guess
in the valley and what do you think
needs to what would be desirable in
terms of like creating like governments
or military really and Silicon Valley
sort of relationship that is like
pro-social in the long term especially
with regards
if more entrepreneurs took the
responsibility on themselves and didn't
just you know think like Oh because like
my favorite people in the administration
now now will do that or because I'm
gonna you know I want some program to be
there to help me to make it easy but
really took the long view of like I you
know my business is a very very
important part all Giga fun companies
eventually what if you if you are gonna
be dominating or completely shaping an
entire industry you will be interacting
with the government so we almost always
give me that conversation with people it
might not be for you know 10 plus years
that they're involved in movies 15 years
[Music]
but you need to proactively
independently craft your strategy for
working with with the government and
will be specific to each of these
companies and it shouldn't just be the
acceptance of institutions as they are
its you should take some responsibility
in educating institutions and bending
the institutions to to your need sure
sounds like you think there are things
that funders or like the broader social
organism or whatever can do to like
create bounders that have that mindset
or to like promote that kind of attitude
or do you think it's just sort of we
just gotta go by individuals and just
try to promote people to got the right
you know attitude certainly broadly
socially I we shouldn't be punishing
people working with the government it
seems like the libertarians punished
people for working with the government I
remember a friend friend of mine who's
like I was trying to you know get him in
his space exit like you know it's like
right nor the first the first or second
round us in he's like oh but this is
done this is someone who's like
surviving different government contracts
they're evil
and then there's the the people who
don't like if the wrong parties in the
government now they hate the government
cuz wrong place not they won't work with
them we've got a na I not like place
those companies if they're working and
working you know with with the with the
government's in the wrong party we've
just got to make it okay to work with
the government I provided you're doing
something that is helping America which
you know it's a lot of the companies
that are in the entrenched industries
where they've been doing it for decades
those companies are not doing that
anymore
maybe the intention of the founders it's
not Silicon Valley should have more
confidence in itself and its founders
that the new companies are building will
be trying to do something positive for
the country yeah all right thanks
and I think if you even come to mind but
never examine we have three minutes left
you maybe I want to ask Luka last
question and this has been really great
so far I'm loving I want to comment on
that last one wait I could we have to
examine ask a question because she
hasn't spoken at all yes thanks thank
you um appreciate you having this
discussion so I I was interested in
getting a clearer sense of your thoughts
on the AI safety landscape now I know
that you're looking for founders who are
quite attentive to safety concerns but
it also seems like a lot of people as we
heard in the talk with you on earlier
also shifting their views towards maybe
a slower type of takeoff scenario or you
know less of a prosaic AI then the sort
of you cows can view I mean I'm
interested if this is playing it all
into your thinking on investment
decisions or the kinds of technologies
that you think we need right now so I've
always thought that it's more interested
in interactions of the early ai's I
think it was gonna take longer than
most people thought and it would be the
interactions of earlier' eyes and
society that would kind of get us to
where we are so that's that's why I was
I think the geopolitical landscape and
the founder aspect of it those are all
very very important things and perhaps
if you have this very steep fast takeoff
you just all you have to do is get the
bath right and you you build one and
then you let it loose I think it's gonna
be more iterative so all of these
different pieces end up being lovers
alright thank you so much Suzanne do you
want to chime in again yes I was hoping
to attack on another question but if it
goes too long I tagged it on and once
we're at once with three we're just
gonna stop or tag it on cool yeah I was
just wondering if you think then if it's
more going to be a series of and not
sure this doesn't fly boat you're saying
but like a cascade of effect to the way
that we've seen Cove it has a cascade of
effects on you know social stability are
there are there really different kinds
of startups that you would want to be
investing in in that case it's a great
question and I won't have enough time
the answer yes and I think you should
use your connective about that and and I
think you all so much for joining the
session and I think I really really
cherish the point that you made about
like it's great like community in which
we can hold each other accountable and
which we don't have to signal too much
but in which we can actually have real
discussion I think that's what we are
kind of creating today and I really
thank you all for joining us in this
experiment and I'll meet you in a second
again in the main zoom room for
report-outs
loop do you know where to find it
because you're going to do a
three-minute with product on your
session unless you want me to do it yes
it works yeah yes it works okay and we
meet you in the main room just now in a
sec bye bye


Organizing for Beneficial AGI: Lessons From The Industry | Jingying Yang, Bobi Rakova

I'm going to start the recording and
then start sharing my screen by the way
are you going to put the recording
summer on 1.4 uh yeah oh okay
where if there are you good to be
sharing um I'm not quite sure yet so if
you'll hold that question for a minute I
will I will hide about this let me just
figure out how to print while also on
zoom okay cool um okay great well
welcome everyone to our session
I'm Jing and Bobby will you hi everyone
and so just a couple of housekeeping
things before we get started we'll be
recording our portion of the
presentation mine and Bobby's for public
consumption and then we'll stop the
recording before discussion and Q&A
discussion and Q&A we'll be in Chatham
House Rules so if you have questions
that you would like not to be recorded
please save them for the end that'll be
in you know like probably like 40 ish
minutes and if you want to say something
please use the hand raising function on
zoom to get our attention or chat and
all items in the public zoom chat that
are like links or anything we'll copy
over to the slack channel if you happen
to be using it otherwise use the slack
and then does anyone want to be the
note-taker for this session like no
worries if not bobbi and I will do it
okay cool great then I'm gonna jump in I
would love it if we could all just go
around and introduce ourselves for 30
seconds each maybe name affiliation and
why you're interested in this session
I'll start I have to call people I'll
start with Alexa okay hi first of all
I'm sorry that I didn't volunteer to
take notes I am a terrible note-taker
it's like I'm an anthropologist and I'm
at the University of Cambridge and I
work with Caesar which is essential risk
and I'm also an affiliate with the
leveraging a few to intelligence and
yeah I mean I'm really interested in
ethnography so like that's why you and
why I am curious about this and yeah
excited - great thank you
um can you do you want to go next oh yes
um okay so I my name is Kenya I really
interested in Holly I think he can help
me I can and can help I'll not help the
companies and I have worked in IBM on AI
and they're interested in how it could
actually be be related and it's a
continue I think resulting so what did I
carry me no there we go okay so I I'm
here I was attracted by the organizing
for beneficial AG I definitely an
interest what it looks like to put
together you know present and near-term
projects specifically in creating more
trust in society that's that's where my
interest has been and I thought one
answer
great thank you
sorry my dog is chosen
you
maybe it's a good chance for you to
introduce yourself Bobby oh hi everyone
I've been the research fellow awake Pai
which has been really really great
working on this project and bottom out
project and outside of that I've been
working in industry with a group called
a sponsor go away I then is working very
closely to what we're going to present
and this session at Accenture yeah with
Accenture and also it's been interesting
being involved with some of the
standardization efforts at I Triple E
that I was so very relevant to this
discussion thanks Bobby
um Jessica back to you thank you yes I'm
just because ins Newman I'm a research
fellow at the UC Berkeley Center for
long term cyber security and also work
as an AI policy specialist with the
teacher of Lincoln's to do and yeah I'm
just really interested to learn about
what PA eyes findings have been on those
topics thank you and I'm Jun yang I am
at the partnership on AI I've been
working with Bobby on this research
project and it's been really great
because we've been able to get some
really good insights from AI industry
practitioners and we're really excited
to share with you all how we think that
applies to kind of the long-term
community and what lessons we can really
draw from that so I will just jump into
our presentation so here's the rough
agenda I'll spend about 15 minutes going
over why now is the time to think about
organizational structure for hei teens
Bobby will present some lessons from the
AI industry based on our findings from
this study and then we'll switch to non
recording and have a discussion with all
of you so as an introduction the
partnership on a I along with Accenture
has been conducting research on
organizational enablers and barriers to
kind of fair ml and responsible AI work
that's really talking about kind of
fairness accountability and transparency
in machine learning and which is a
research community that is thinking
about those issues and also brought
broader concerns about responsible AI
and the findings in our researcher will
will be based on 25 as no graphic
interviews that we've done with AI
practitioners in the AI industry there's
more information in the appendix of all
of these slides about our study design
and the goal today is to really frame
how this research on pheromone actually
provides a blueprint for how to create
mission aligned organizations which are
crucial for creating beneficial HEI and
to share findings from the research and
how these lessons can be leveraged to
build stronger HEI oriented teams the
outcomes for today hopefully are that
you all leave here seeing the importance
of human organization and processes for
positive technical outcomes and to maybe
take away one to three action items for
how to transform your own team and
organization into a more mission aligned
place so what do we mean by
organizational structure and culture
organizational structure consists of the
infrastructure and processes that
constrain and guide how people groups of
people within an organization behave
while organizational culture is more of
the norms values and philosophy that
influence how groups of people within an
organization behave and you can see
these really are shaping each other the
infrastructure and processes will shape
what norms develop in an organization
the values of an organization will shape
in turn what processes they put into
place and why are these things so
important for thinking about building
transformational HEI is what I will go
over in the next few slides so
organizational structure and culture
play a really big role in outcomes
concrete example of this is when you
look at the company Zappos compared to
thier nose so as a post
they had a culture of meritocracy where
promotions and bonuses were based on
really demonstrated skill and tests of
capabilities and how employees lived the
10 values that were core to zeppos on
the value alignment piece culture fit is
about 50% of their hiring criteria and
feedback culture is really robust and
honest the outcome was that within 10
years of founding they
required by Amazon for over a billion
dollars on the other hand you have ther
knows where culture of favoritism really
flourished promotions and bonuses were
really based on personal relationships
with the leadership fraud was fairly
rampant and whistleblowers were called
out and punished and actually
intimidated in some cases and the
outcome for their nose was that the
company went out of business and the
company and CEO were sued or
investigated by numerous people so
definitely a stark contrast and the
culture and structures within each of
those organizations very much played
into these outcomes and we really think
that the organizational structure impact
will be actually magnified for Black
Swan type events because Black Swan
events share these characteristics of
being highly uncertain high impact
rapidly changing and also large scale
and all of these characteristics really
exacerbate and amplify the difference
that we were seeing talking about with
Zappos and Sarah knows so taking the
code 19 example as a an example of a
Black Swan event there is a clear kind
of emergent theme that there are
countries with poorly aligned responses
and countries with well aligned
responses so on the poorly aligned part
what we mean is there are broken
feedback cycles kind of fragmented
experimentation in response piecemeal
lockdown efforts driven by a mix of
companies local governments and other
jurisdictions a shortage of personal
protective equipment and testing leads
to unreliable care and unreliable
identification of cases and we see in
those places that are experiencing this
poorly aligned response there's rapidly
increasing infection rate and health
care systems are starting to get
overwhelmed on the other hand we have
places that have actually mounted a
really well aligned response so their
rapid data sharing publicly research
immediate large-scale lockdown
by centralized governments often
redirection of resources to make test
kits and PPE quickly to scale testing
and care and the results and outcomes on
that side are flattening infection curbs
and actually having the capacity to care
for the seriously ill and if you compare
kind of fatality rates across some of
these locations there is an emerging
difference as well so really we believe
that organizational structure has an
even magnified amplitude of impact for
Black Swan type events and eg I could
lead to a transformational Black Swan
event which means that building AGI
requires organizing people towards a
common goal and maintaining mission
alignment through a lot of changes
possibly over a long period of time or
through really dramatic shifts and also
in the face of a lot of uncertainty so
all of that means that especially for
beneficial AGI we really need robust
resilient and effective organizational
culture and structures another insight
that we wanted to share from other
industries is that as technical interest
you scale they tend to actually start
focusing less on technical problems and
more towards human coordination issues
not that they don't also focus on the
technical issues but actually as they
scale the human coordination issues get
more and more complex so we really see
this within the AI industry what the
academic research chooses to prioritize
is often technical issues whereas AI
practitioners really need guidance on
organizational challenges and that's a
gap that the that are particularly
searched is trying to bridge for the AI
industry and whereas in the AGI
community we know that there is a lot of
focus on technical AI safety and there's
also emerging interest and a lot of work
on AI governance and strategy and the
gap that we would like to use these
lessons to prevent is the gap between
all of that great thinking and the
technical tactical implementation of all
of this alignment as this AGI community
scales as well
luckily decades of research exist on how
to effectively bring people together
towards shared goals so we have plotted
here a number of different bodies of
literature and research and we really
believe that taking lessons from all of
these will be crucial for our shared
goal of creating an official hei and the
research that we're gonna share today
specifically draws heavily on industrial
organizational psychology organizational
science and anthropology and we want to
really point out that incorporating
public good has been a challenge across
a lot of previous technical
technological transitions so often there
is a new technology then people identify
concerns for the public good that may or
may not be addressed in immediately with
that technology and often there's an
emergent community that leads change
that actually makes those technologies
better for people in society so as an
example there we had cars emerge in the
last century and then also with cars
emerged the behavior of drunk driving
which is obviously very dangerous for a
lot of people and the organization
called Mothers Against Drunk Driving was
a really strong advocate for policy
change in this space
similarly with furniture there's a real
crisis around there was a crisis around
tip-over and children getting crushed
under furniture when it fell over and
actually parents were instrumental in
advocating for design change at the
product level and also for policy change
on a regulatory level and similarly with
machine learning there's been a lot of
emergent concerns about fairness and
bias within the last maybe five years
and that's really for the public good
because there and this is so important
for justice and all of these things and
there's been an emergent research
community called the fact community that
has been really looking into these
issues and finally mapping it to AGI I
think the concern for public good there
has been really focused a lot on safety
and the AI safety community has emerged
to take charge of this issue and try to
plan out ways that we can head off some
of these safety issues so these latter
two are really emerging communities and
we want to share a great framework about
how transitions can occur between sort
of the existing kind of technological
shift and becoming more aware of these
societal impacts so this is a framework
called the two loops framework from the
BIR Khanna Institute and it really Maps
out how new social movements emerge and
we really think that it's really implore
something like that official AGI because
it's really thinking about how to
integrate this concept of the public
good into every aspect of this industry
so you have at the beginning pioneering
practices that start to emerge within a
dominant system that might be less
interested in these issues then you have
enabling practices within the fund the
dominant system which might include
funding and policies and then you have
hospice password which means work that
is aimed at stopping the practices that
were that were hindering some of these
pioneer practices on the other hand you
start to connect and form networks of
connected efforts of these pioneering
practices you build communities of
practice and grow coherence as a whole
field and that in that way the
transition is navigated towards actually
having these emergent practices of
course there are also outliers outside
the system which informs some of these
emergent practices as well so actually
for today's event in general these
networks of connected efforts and
forming communities of practice is the
thing that we are all doing together so
it's great because this I think
framework really captures how we can
together all help to scale this line of
thinking and this is just an alternative
view of that framework with a bit more
detail of
how this has been used to bring social
innovation to scale and we really think
that this is a great framework to imbue
into this community so I think the AGI
community has a real opportunity to
design institutions to avoid the gap
that exists elsewhere and the gap is
really between having mission alignment
as a nice-to-have
versus having mission alignment as an
indispensable process so what we mean by
that is for example in the nice-to-have
scenario safety work would be ad hoc it
would rely on volunteer time it would be
always pitted against sort of business
interests or other interests for
priority and resources whereas if it
became an indispensable process then
safety work would be seen as actually
crucial to business interests and then
become integrated into all of the
workflows and have dedicated staffing
and internal structure for education and
scaling so our goal and question for
this presentation today is really can we
learn some lessons from other industries
like the AI industry as a whole to
prevent this gap from opening up or to
actually help us design in the first
place in the aspirational State so I
want to make a pitch for why we want to
prioritize organizational structure and
design right now and why we need to
bring in all these lessons from other
industries and right now it's because
many AGI oriented organizations are
actually still small there's definitely
some exceptions to that but it's always
less costly to build a new system than
to retrofit an old one so if we set it
up well now we'll actually be more
efficient than if we set it up a
different way and then later have to
retrofit it so right now is a great time
because the organizations are still
small and we can really set up the
processes however we want based on what
works from other industries and starting
now also leaves more time for informed
experimentation before scaling and
takeoff so I think that's another
benefit from a sequencing perspective
and bringing in lessons from other
industries
I think is really crucial because it's
free to learn from other people's
mistakes and much more expensive to make
our own and then also exposure to more
threat models from other disciplines can
mean more resilient plans for the HEI
community and given how uncertain all of
this is that resilience will be really
key so now I will turn it over to Bobbie
to present some of the findings from our
study ok wonderful and I will share my
screen I think it's ok great we can keep
going ok perfect
I will go on with telling you a little
bit about the motivation behind the
study with that we've done and show you
the results and the discussions around
them and how are they relevant and then
Jeanine well jump in to also make it
more relevant for the HR community ok so
what was the motivation part of it was
the work by computer scientists
Melvin Conway who formulated this
hypothesis called Conway's law which
pretty much demonstrates how
organizational structure directly shapes
all systems are being built so he stated
that organizations which develops
systems are constrained to produce
designs which are copies of the
communication structures of these
organizations and of course this has
been studied in other fields such as
organizational science and so on the
other literature that yunying brought in
but we saw that in computer science as
well there has been some very very
interesting research on that so here are
some of the analysis graphs from
Conway's original paper on this and if
you focus first on the one on the Left
what they demonstrate is that systems
have subsystems as components and they
literally have a pis
in order to have subcomponents connect
and interact between each other so in a
way I all of computer science have been
influenced by object-oriented
programming and that is one of the core
concepts there that those api's are
going to be crucial between how software
functions and similarly in the design
and develop on a vai we cannot escape
from this core concepts in pure software
programming on the right side of this
the diagram you see a comparison between
the structure of these sub components
interacting between each other and
organizational structure so it's
interesting then the linear graph
notation is useful because it provides
an abstraction which has the same form
for the two entities we are considering
on the one hand the design and
organization and on the other hand the
design of the system so actually Conway
in his paper looks at federal government
when he designs and go through this
research so is if it's interesting to
you definitely have a look at his work
here's an example demonstration of
different organizational structures such
that you can imagine how each of them
will inevitably influence the structure
of the outcomes that comes out of those
organizations and you've probably seen
this graphic in media actually Harvard
Business School has framed these
concepts of the same common law as the
mirroring hypothesis and they've drawn
from the literature on organizational
design and organizations as complex
systems as well as on the literature on
product design and products as complex
systems while doing a review of hundred
and forty two empirical studies of
organizations in industry as well as
open collaborative projects like open
source projects in order to investigate
if Conway's law holds in practice even
though their study was conducted in
2016 no other organizations they looked
at are developing a products or services
so considering the supply chain of AI
enabled products and systems it is
interesting to consider even where
there's a mirroring creepily this whole
ultimately we wanted to ask the question
of what organizational structure could
enable mission alignment in the way
machine learning design development and
deployment happens in organizations at
the moment so here is the methodology
that we had for the study and you can
read a lot more of that in the paper
that we are writing at the moment over
the last 7 months we convicted 25
interviews with industry practitioners
involved in work on misuse and
unintended consequences machine learning
key criteria for the participants was
that they are the output of their work
should have impact on the real products
and services beyond research and you can
see the various types of roles people
had in our study so far nigo including
legal policy program managers versus
research scientists software engineers
data science and others there were many
common perspectives that we heard
practitioners expressed repeatedly while
conducting the interviews we saw the
need for thematic analysis which in
corpuses three clusters of data sort of
the dominant and emergent state and
desperation of future state that
practitioners imagined the analysis
within the prevalent or dominant state
state comprises of what we saw is most
common in the data the emerging state
includes what emerged as work practices
which are common across practitioners
but not prevalent and then the spiration
future state consists of ideas and
perspectives practitioners shared when
explicitly asked about what they
envision for the new future state of
their work within their organizational
context and actually when doing the part
about aspirational future state we
iliza future forecasting framework in
order to take participants to a mapping
exercise of what does that future state
look like so here on the Left this
dominant emerging state visual actually
comes from this paper and we thought
it's an interesting a presentation of
emergence that we are seeing in
evolution biology and other many other
fields as well as their organizational
practices that are emerging in the
intersection of AI HDI and these other
fields and we're bringing in in this
discussion okay so here's a summary of
these different dominant emerging and
aspirational states and here you see the
different trends that we identified
during the tamerica analysis we'll go
through them one after the other so
they'll make more sense
but in general all of these trends were
transitions from the current dominant
state towards this future aspirational
state we're mission alignment is
perceived as an indispensable process
inside organizations leading to improved
outcomes while in the current state very
often it is perceived much more as a
nice to have and its only embedded in
the mindset and behavior of individuals
within organizations okay
so in the current state many times
organizations are reactive they the work
that I do is fragmented the most common
incentive for change is catastrophic
medium tension and decreasing linear
tolerance theory machine learning as
accountability and transparency of
machine learning is often perceived as a
taboo topic and the questions come up
such as whose job is this the work is
often not compensated and it's
completely voluntary
it is also perceived as too complicated
it and not easy to understand in there
merging state we saw that some
organizations have set up proactive
evaluation and investigation processes
that provide support and oversight
structures for people within the
organization there will support from
legal and policy teams and there are
also standardized review processes
participants reported proactively using
various internal communication channels
and strategies such as taking
screenshots of problematic algorithmic
outcomes organizing internal discussions
meetups where they would report concerns
and latest research in this field a
growing number of educational
initiatives on onboarding and upskilling
of employees have started as well as
educating customers on how to use
products this was a very major point
where people express more so for the
anticipatory future state then we really
need ways to educate people about
potential misuse and unintended
consequences of the AI that is being
developed inside organization also
people reported the need for clear and
transparent communication internally and
externally incorporating technical and
non-technical tools that orchestrate
algorithmic assessments of operations
mode again internally and externally
customers utilizing their produced a
rhythmic models in different contexts
some practitioners reported their
concerns if they should even engage with
a client without the use of some sort of
assessment framework they express many
ethical and tensions that they've had
personally with certain clients and the
lack of organizational support and
structure to help them navigate that
they reported that traditional
engineering mindset needs to change
towards being more adaptive and than
that dynamic aligned with the dynamic
nature of the issues misuse and then in
the current state most dominant state
among what practitioners shared with us
was the lack of metrics the needs to end
need to communicate lack of metrics to
clients it's hard to have strong KPIs
for many of the issues and their
unintended consequences of lease and
misuse of AI and that makes it even
harder for practitioners who are trying
to have these metrics as part of their
process of design and development many
times practitioners who are not sure how
to measure impact and quantifying what
they do was really difficult because it
was often in this qualitative space
where we often lack metrics where these
metrics exists in other like social
science fields or other disciplines that
making it harder to apply in the pure
engineering work oftentimes people
reported use of inappropriate metrics
and academics metrics being very
difficult to apply in industry contexts
for example user engagement and positive
user experience is often not a metric
easily utilized by researchers but it's
a metric that industry practitioners are
measured many participants reported
being measured on generating revenue and
delivering work in some cases they've
used an abling argument that mitigating
and Italy consequences a misuse kinds of
risks prior to launch is a lot cheaper
and once you launch something and it
goes sideways
however again like without actual
metrics it's much harder for them to
prove that to their organizational
leadership
many of our partitioner stocks to a
balance to us about the misalignment in
the organization in terms of ethics
guidelines best practices what is the
organization of strategy on that then
many times organizations reward efforts
focus on internal organizational
structures in order to sorry internal
organizational education in order to
enable practitioners to navigate that
space and bring awareness to these
potential issues some of the
organizations reported enabling
collaborations where organizational
structure is set up in order to enable
petitioners to more closely collaborate
with marginalized communities or
practitioners in other organizations or
academia they express the need for
easier ways to engage with external
communities as well as understanding
legal considerations and policy
considerations that often make it
difficult in the future state organ
participants reported that the
organization could have a tangible
strategy on implementing assessments and
in relations misuse and any consequences
and it is something that their teams are
measured against so it's part of their
performance evaluation process then
production teams employ qualitative
methods together with existing
evaluation metrics in order to capture
concerns and mechanisms exist to enable
collaborations organizational structure
that could enable practitioners to more
closely collaborate with outside
communities help in defining benchmarks
and making sure everything is performing
well after a deployment and this
continuous monitoring keeps happening
many times in the dominant systems
participants reported lack of
accountability structures inside of the
organization they pretty much didn't
know who's accountable when there is a
potential risk of unintended
consequences
most commonly participants reported that
in their organization has in some ways
very obscure holes and there is a little
uncertainty about role definitions and
responsibilities for fare for this line
of work
several petitioners expressed that they
needed to be a senior person in the
organization in order to make their
concerns hurt and a reputational risk is
was the biggest incentives incentive for
their line of work petitioners would use
it as a leverage point but again it
wasn't as clear in the use case how to
take advantage of that in order to drive
organizational change in the emerging
state many organizations have developed
scaffolding to support the work on
misused and unintended consequences
organizations have the ability to allow
people to craft their own roles in a
more dynamic way informed by internal
and external factors and communities
such as our community accountability is
distributed across organizational
structures and issues get confidently
escalated to management teams are held
accountable to what they've committed to
and as part of their performance
evaluation
they include metrics that correspond to
unintended consequences of misuse and it
happens through internal review boards
publication release Miller norms and
documentation and transparency
guidelines but that is where we really
go into the aspirational future states
where
more and more organizations are having
these clearly defined practices
organizational scaffolding that can
support people to do this line of work
and maybe they have institutionalized
roles where that the work on
investigated and in teleconference and
issues is seen as a main job that the
person does and it's not a sort of
volunteering or plus-one that they do
there are more collaborations and
partnerships based only stakeholder
frameworks and allow people to reassign
a work and lastly the last turn we saw
was going from fragmented to more
aligned kind of work and here in this
dominant state oftentimes those
misalignment between individuals teams
and organizational level incentives and
mission statements individuals doing are
doing a hog work based on personal
values or assessment of importance there
is part of information which relies on
individual relationships within their
organization how they communicate is a
key factor what is their trust
relationship and ability to navigate
multiple levels of year key within the
organization many people talked about
organizational inertia and obscured of
an additional structures weather which
are making it very difficult to do their
work a Whitman impacts are often if you
diffuse or hard to identify there was
lack of data for sensitive groups to
help with evaluation and testing for the
impact and any consequences of their
technology data collected internally at
the company is extremely biased and
petitioners reported having multiple
challenges around building gas
reductions in the organization on the
need for collaboration with external
groups
in the emerging state we saw that overly
rigid organizational incentives can them
demotivate
addressing the tensions that we were
discussing part of the work within
[Music]
investigated unattended consequences is
very much related to conflict resolution
and managing conflicts between different
stakeholders inside organization which
may lead to stress and burn one burnout
for the people that are doing this or
the work which which is what we saw
emerging in the data that people report
it many times we saw that people are
incentivized to produce complex
solutions to problems and complexity is
rewarded and incentivize whether or not
it is needed and again an organizational
inertial competition in competing
priorities make it challenging to
justify it certain research agendas and
especially to make them irrelevant to
industry projects many times industry
specific product related problems may
not have sufficient research married or
more specifically an ability for the
researcher to publish and many
petitioners express that as a challenge
for them to collaborate with their
research teams and more effectively work
on some of the issues that I've seen in
industry lastly in the aspirational
future state ethical tensions are
resolved in accordance with
organizational level mission and values
organizational leadership understand
supports and deeply engages with the
concerns misusing our antenna
consequences there are mainly many ways
in which organizational culture is
aligned with topics of the discussion in
a way that it lets go of the fear being
screen eyes and organizations function
more fluently
every single person understands risk
collectively as teams we understand the
risks and also we have leaders who talk
about it publicly
and admit when mistakes happen and now
jumping back to Jeannie and we don't
hear your Jeanine unmute great okay
great
hear me now so what does putting that
together Bobbi presented a lot of our
findings across these four themes so
really pulling those four together what
does it look like all in one picture for
what we think an aspirational AGI
organization could look like so results
are defined to include societal impact
measured through data-driven efforts and
the results on those societal impact
measures are actually tied to
compensation and financials that's one
piece and other piece would be that the
organization would have deployed
frameworks that incorporate anticipate
anticipating risks in a low-friction way
so that it's not onerous for people to
do so ethical tensions in work are
resolved in accordance with the Ora
global mission and values which keeps
everyone aligned and responsibilities
for safety and societal impact are
integrated throughout business processes
related to product teams and decision
making and that's pretty aspirational
and it's also a little bit vague in
terms of how do we actually get from
what people have today to what that
looks like so we don't have concrete
answers because we haven't been able to
test these solutions yet but I want to
share some potential solutions for
getting to the align future that were
brainstormed by our participants in a
previous workshop and we'd love to hear
your feedback and thoughts about these
so one is effective veto power on
projects so this should be available to
multiple levels there needs to be
whistleblower protections for people who
do bring up whether or not an ml product
product or a particular project should
go forward and a know based on a lack of
mission alignment should always halt
work we need to combine internal and
external pressure neither alone is
sufficient internal oversight can be
more detailed whereas
external pressure can add a lot of
motivation and then we need to build
robust communication channels keeping
people in dialogue and aligned across
all of these different levels and
internally and externally and for this
to work you need structures to make the
debate actually really fruitful that can
even out power dynamics and ensure that
people actually disagree in productive
ways and then leveraging feedback loops
is another potential solution which is
that all of these other solutions really
work in tandem so for example the
whistleblower protections are needed to
let people honestly use the channels of
communication and so key takeaways from
today's presentation are really one now
is a really good time to intentionally
design organizational structures for
teams that are building towards
beneficial AGI because it's usually less
costly and easier to build a new process
than to retrofit an old one to look to
lessons from other industries to avoid
reinventing the wheel and then three
expect to continue investing and
maintenance of processes and
infrastructure this is actually
extremely challenging and we see it
across a lot of other places that people
who do do that have better outcomes and
maintaining alignment as the
organization's
really scale has been historically very
challenging so we should anticipate that
it'll be challenging in this context as
well so with that I'm going to stop
recording and