Towards A Global Community Of Shared Future in AGI | Brian Tse

[Music]
how many of you have watch a movie the
Martian it's a pretty cool movie isn't
it some of you may remember that the
China National Space Administration had
developed some rockets to launch an
unmanned exploration proper that rocket
could be used to send additional
supplies to keep Matt Damon alive that
rocket was the only one that could be
used in time and NASA didn't know it
existed
the Chinese scientists eventually
approach NASA and a true science agency
Richard Hill long story short Matt Damon
is happily back to earth thanks to a
cooperation between NASA and China I
think the story illustrates a dream of
cooperation between the two countries in
a territory of enormous uncertainty it
is a mission that wouldn't have been
possible with other facilitation of
scientists in the US and China it is
also a story and movie that the Chinese
people really love as they ponder the
more prominent role of China in the 21st
century as we discuss AGI I would like
us to keep this narrative in mind that
China might be developing some rockets
to get you AGI and some of those rocket
could be really critical for our world
collective safety and governance
endeavors the mission of building safe
and beneficial AGI needs to be a joint
worldwide effort with China my key
concern with AGI is best captured in the
open air charter it describes a
collective action problem where the
late-stage
AGI development might become a
competitive race without time for safety
precautions and since a judge how much
remain highly uncertain we should take
into account the emerging projects of
tomorrow and not just the leading ones
today and that's how China comes into
the picture and here I would like to
catch all the specific kinds of
understanding they are the most
important we got in China to have solid
and robust global coordination I believe
having the following share hypotheses is
very important first there is a serious
possibility that AGI could be developed
in the coming decades
second as AGI becomes to follow
there are enormous uncertainties and
risk with such energy third
International Cooperation AI is not a
zero-sum game it could be merely truly
beneficial for both countries as they
share a common course in how a is
developed and deployed country my face
cooperative incentives in both
minimizing the downside risk as well as
capturing the enormous upside potential
the AGI has so in the rest of my
presentation I'm going to give you the
Chinese perspective on the following
hypotheses then I would suggest a few
approaches that could foster greater
coordination with China finally I'll
wrap up by suggesting how you might be
able to help in China it seems that the
idea of AGI is gradually emerging into
the public discourse as evidenced by
discussions in the academia policymaking
and expert interviews Kuantan Joon who
is here today is the chair of computer
science at Peking University he was also
a key chapter of the China's AIT fluent
plan he has made multiple statements
that strong AI may appear in the next 15
to 30 years tan Tien o who is the deputy
general of Chinese Academy of Sciences
he was giving a lecture to the top
legislative body in China the National
People's Standing Committee there he was
discussing the future trends of AI and
the first point he notes is a
development from narrow to general AI
citing the goal of deep mind and a quote
from demis hassabis in 2017 there was
this expert in the view and survey on AR
timelines and he found that respondents
who receive undergraduate education in
China predicted that there are 28 years
before the arrival of human machine
intelligence as compared to 76 years for
those who study in the US as a median
estimate if you find out that private a
LF in China an interest in such research
it seems reasonable to infer that they
believe such technology is possible in
the coming few decades of course
companies in China just like everywhere
else have numerous motivations for
expressing one's interest in AGI such as
attracting top talent
or you know premium prestige but that
said it still seems useful to you know
which labs are claiming to be heading in
that direction and here's some examples
Qi way who was the prominent scientist
at by to entity of deep learning
co-founded by Andrew hang at a time
belief that we should stop talking about
whether Asia is peaceful and begin
coming up with concrete ways to develop
such technology and now he has moved on
to the southern horizon robotics as the
chief sciences of general air lab that
lab has explicit long term goal of
deploying human level general AI yeah
saying who is the vice president
attention has made multiple statements
that developing AGI is one of three
strategic priorities for the company in
particular he believes that games are
the most important direction
40:14 AGI because all the world's
leading AI research companies are
heading in that direction and let's look
at their research into that in for a
moment
it may choose an eighteen they develop
in Xcode based on half ago zeros
published paper and then later applied
Monte Carlo tree search techniques to
their own game King of glory which is a
complex multiplayer mobile game in China
then in September they developed T star
put their own product to beat the signal
level Pai in Sacre - and just in last
month they published the paper
hierarchical macros use MicroStrategy
model and apply it to the game of Gouri
and in particular 5v5 instead of 1v1 so
the game becomes a lot more complex and
they claim that they are now competitive
with the top 1% of human players so with
JD calm which is the Amazon of China the
director of a lab also has made a public
lecture talking about the seven key
technical challenges of developing AGI
he gives examples such as advances in a
technique of unsupervised learning and
learning to learn now I assume some
evidence that China thinks that AGI
might be a plausible technology in the
upcoming few decades and many of the
Chinese
thinkers have also public
express the concerns with the risk of
such analogy one of them is told you
have who has written the undergraduate
textbook in machine learning in China in
some way this rustle in the country he
has published said that strong AI should
not I cannot be done and suggest that no
AI researchers should develop such
energy as email it to the greater
survival crisis for Humanity another
person is dojo ha who is one of the most
influential living philosophers in China
has written a long essay arguing that we
should set the safety conditions for AI
as super intelligence my literate
earthshaking and to human history jung
ji who is a deputy director at research
center for bringing in spirit
intelligence and who is here today as
well has taught a class on philosophy
and ethics of AI in the course at the
university of chinese academy of
sciences he discusses the problem of
existential risk of AI to assess the
intention of China one of the best way
is to look at what the political leaders
say given the importance of top-down and
elite political communication in
mobilizing the effort of the whole
society at the world air conference in
Shanghai last year a statement resurect
for president xi jinping
the statement emphasized that china is
willing to cooperate with other
countries to promote development improve
security and share result in AI in
particular China is particularly
interested in issues such as legislation
governance ethics and security of AI lü
ha is a China USA war chief as people
call it him I felt and surprisingly
nationalistic in 2000 but he still echo
Presidency's view by saying that AI
represents a new era in which cross
national cooperation is inevitable so
more generally I think that there are
few short term challenges and long term
opportunities in fostering greater
coordination on AI safety with China
currently I think that there is some
confusion was the idea of a SCV partly
because the terms si VA s security are
the same in Mandarin Antron we later
there is relatively little awareness of
alignment research for AGI second it
seems to me that some of the Chinese
thinkers have hopefully focused on the
topic of consciousness as compared to
just thinking about the risk of a
generally competent goal-oriented system
which might not be conscious but
nevertheless roaring our concerns in the
longer term however I am optimistic of
China being a responsible actor in
developing and deploying the technology
the first reason is that China is run as
a technocracy with most proven leaders
being educated as engineers and
scientists for their school and in one
of the recent interview with Joe Rogan
Elon Musk has said that politicians in
China a pretty good at science he met
with the mayor of Beijing who holds a
PhD degree in environmental engineering
and the mayor of Shanghai whom Ilan
described as very smart so given the
technical complexity of AGI it seems
beneficial that a government and its
politicians understand signs of a
technology second the China model allows
the country to engage in long term
planning which lets you project such as
the national rail system through the
five-year planning and the Hofstede
cultural theory also suggests that the
Chinese rank extremely highly on the
dimension of long term orientation this
means that they have a tendency to you
know focus on longer term results
instead of the short term goals
certainly China has a historical view of
dynastic cycles with dozens of dynasties
having rising and falling in the past
four thousand years and this means that
the prospect of a civilizational
collapse is deep-rooted in China's
subconscious and this makes
risk-management
one of the key topics in governance for
China so given all these factors it
seems that China might be able to be a
responsible actor in developing and
deploying transformative AI and this
gives a window opportunity for fostering
coordination in AI safety so given these
contexts I'm going to talk a little bit
about some of the promising approaches
for strengthening coordination with
China
given my research at the Center for the
governance of AI so the first approach
is to establish collaboration with
técnicos a ICT so in a recent paper
published by a covenant I don't feel as
the think-tank it suggests that we have
enormous issues with problems such as
robustness in distributional shift
interoperability and explicitly cited
the Congress of safety
it suggests that China should begin
international cooperation in a ICT
research while promoting the deployment
from narrow to general AI in particular
China should set up overseas research
centers and begin international tenable
exchange between researchers as some of
the concrete policy ideas here I also
compiled a list of research focus
international partnerships and found
that six out of seven of them or
industry-university collaboration so one
example is the collaboration between MIT
and since time the second approach is to
support the multi-stakeholder and
Industry Alliance effort from China and
internationally I see the participation
of bite you in panel AI as a positive
development as you was the first Chinese
member to do so organizations should
continue to invest in such effort and I
think certain working groups API might
be more technically oriented and easier
to reach global consensus such as the
safety critical AI group on the other
hand organization could consider you
know interfacing with China let
alliances one example is a China AI
industry alliance with members such as
Microsoft and IBM more recently there
was also this global AI academic
alliance which was founded by 50 members
with the majority of them being Chinese
institutions but also a few universities
such as University of Sydney and MIT the
third approach is to foster the machine
learning AI transnational epistemic
community so epistemic community are
networks of experts with policy relevant
and authoritative
expertise and this form of governance
has contributed to the creation of not
nuclear non-proliferation policy as a
example I think the case for optimism in
ml is strong because you know
almost
all the researchers from around the
world attend the same conferences and
according to a study a quarter of the
co-authors at 2017 triple-a I conference
were Chinese and the conference was even
rescheduled due to a original conflict
with Chinese New Year another report
suggests that 53% of top papers from
China involve international
collaboration and the network graph here
shows that china is most connected with
researchers in the US Canada and
Australia the fourth approach is to
bridge the Western and Chinese air
ethical principles so in 2018 the first
three eight the co-principals came out
of China represented by efforts at
Chinese Academy of Sciences 10 Central
Research energy and by June all these
principles include consideration of
safety frequently with the phrase safe
and reliable untrained cow notably the
one by ten senators in the to include a
precautionary principle suggesting that
we should ensure HDI serve the interest
of humanity as suggested in the linking
ID principles project here it shows that
almost all the international proposals
of ethical principles include
consideration of safety so this can
hopefully be a shared basis for
constructive global dialogue between
China and the rest of the world now with
these context and approaches I'm going
to talk about some of the implications
for people in this room and how you can
help I think a primary approach is to
support the local development of safe
and reliable AI in in China and foreign
authorisation individuals can play
important role of being a technical
adviser in different capacities so one
way to do it is that you can work with a
local Chinese partner in you know during
our projects in the country for example
procuring energy panel was peaking
University in launching the China Center
and have engaged with people like
Francesca Rossi in pass a Icicle
discussion events the second thing that
you can do is to invite Chinese
researcher to attend conferences like
this one and allow you know
cross-cultural dialogue to happen at
these conferences and on
a very important topic of of AI safety
the first thing is that you can be a
technical adviser a speaker or a
participant at some of the AI tenneco
safety workshops hosted by Chinese
intrusions X example that I'm really
excited about
Ching hua university instead of AI is
planning to organize an international
conference on AI safety and it is the
top AI academic lab in China so it could
really be a good bridge for developing
global next practices in air safety and
be local champion of this course in
China in general I am building a team to
facilitate the coordination between
Chinese and foreign organizations in the
topic of safety and governance so I
really love to talk if you're interested
in supporting advising or getting
involved in any capacity going forward I
think that more people should do
research and participate in the
discussions of global coordination
involving China so there are many topics
that you know people could contribute to
for example in research it seems that
there is a predominant paradigm of the
approach of bringing inspire
intelligence or even bring light
intelligence for AJ researchers in China
and one question would be what are the
safety and coordination implications for
this in policy are there any common
policy concerns between you know US
government industry Chinese government
Chinese industry and how could we
imagine and conceptualize HR to dialogue
in a safety in the future in terms of
kind of normative destinations or I do
confidence using the terminology by
Ellen can we find commonalities between
Chinese and Western thoughts on global
governance theories in China there are
discussions on 10 SIA or under heaven as
a as a theory of global governance and
also the community of shared future for
mankind from the government and last I
would just let you emphasize that global
coordination a scale doesn't occur
effortlessly in 1975 the isothermal
conference on recombinant DNA was
organized by 140 mostly Western
scientists and that meant researchers
from other countries had to contend with
throughs that they had no role in
establishing for decades later in 2015
many
the scientists recognized need to have
buy-in from China and they wanted to
organize a crisper ethical conferences
involving Chinese researchers and so the
conference was colonized with Chinese
Academy of Sciences and in 2018 the
second conference was even hosted in
Hong Kong
instead of Washington DC in AI we likely
to not have four decades to get this
coordination right and the prospect of
the world would be much brighter if we
can have more talented Chinese
contributing to a SAV research agenda
and also the dialogue of global
governance and as the Chinese often say
the whole world within the four seas
belong to one family so I really look
forward to building this global future
of share HDI with all of you thank you
we've got time for a few questions
thanks very much for a great talk I
think you've given us a lot of great and
interesting reasons to have optimism
about the future of cooperation but not
to be overly pessimistic what do you
think could go wrong like what are the
main things that you think could cause
cooperation to fail yeah I think there
are a lot of nuances in communicating
the potential risks and benefits of AI
even though we would want to have
greater coordination and we love to
believe that there are you know enormous
with distributable gains from from safe
AI there is also game theoretic
situations where it might make sense to
really fast forward the development so I
would be very cautious about
communicating idea of hgi when I
interface was with organizations and try
to frame and discuss all the nuances of
such idea and I think that this problem
my accessory bait might be exacerbated
from you know the media talking about
arms race narrative in most of the time
so I think within this committee you
will be useful to you know to combat
this narrative of you know it is always
a zero-sum game and it is always ever
Sariel and I think we need to do this
very carefully and and diligently
thanks very much Brian now that was
great this is not so much a question but
just following up on this I know it's
really really short notice because it's
come up very quickly if anybody is
passing through Hong Kong later this
week January 11th and 12th
we are gonna at HHA USD in Hong Kong
post the first time fat ml in Asia to
try to promote more of this dialogue
across the cultures this is a trial you
can drop by there a lot of tutorials to
get local communities up to speed
and then there will be a real bigger one
later on in this year around November
and then wouldn't we very much welcome
partnering with anybody who has ideas
thank you very much was a great talk I
was interested in the fact that you the
linguistic conflation between security
and safety I wonder if there are
insights to be had in translation of
words like generality agency
consciousness and do Chinese researchers
come at any of these problems from
different directions because of those
differences in in linguistics that's a
interesting question
I haven't thought too much about it and
I don't have a good example yeah I think
I think a lot of the terms in AI and
machine learning have been translated
from English so reinforcement learning
for instance is basically a direct
translation from language but in terms
of you know concepts such as
consciousness that might be different
but I have to think more about that and
get back to you
okay good so um in the US and in Europe
oftentimes when we talk about governance
concerns there's this ongoing tension
about whether we should be talking more
about near-term concerns initially in
getting some definitions down or whether
we should be talking about AGI and and
future prospects and I guess what I'm
you know I understand this conference is
specifically about AGI and therefore
your presentation underscored that but
what I don't understand about China yet
is what's a better way of approaching
these governance concerns with China is
it through the AGI issue or are these
questions of responsibility in privacy
and you know the the wrath of other
near-term concerns is is that really
getting on the governance table or is
that still sort of in the discussion
period yeah I think the boundary between
near term and longer term might not be
as distinct as people usually make out
so I think with examples such as
interpretability it is both a technical
problem that is important for the
near-term in the longer term and I think
that a lot of a few researchers believe
that there are insights you behalf from
from you know current efforts and that
could be applied to advanced AI system
so I think just engaging with Chinese
researchers on the topic of ASAP doesn't
really assume that we need to talk about
AGI and I think there are revenues
avenues of cooperation even even in that
light that's said I think you know in
terms of a policy it might make sense
you have more explicit boundary of you
know what is the timeframe that we're
talking about and I think that I would
be interested in exploring what kind of
policy concerns that you know Chinese
government industry and the US
government industry both have as I as I
alumina as I suggested in the
presentation and that might be you know
safety critical AI that might be you
know malicious news from terrorists and
others and and you know currently China
and US are already building a lot of
cooperation on on counterterrorism so I
think that there are these kind of
leverage points to bill corpora
and hopefully you know as we go for it
with more transform the AI people
already sitting in the same room and
talking to each other for for for quite
a while let's think buying a gun for a
great talk
[Applause]
[Music]


What Should Happen To Humans In A World Of AGI?

[Music]
so I would like to make four points in
two minutes
so after minute so I start with
something that was said also this
morning as a summary of some of the
workshops related to this topic and in
particular the statement that I think
that humans have value and that there is
what was called this wealth floor
meaning that our children and our people
that will come I have the same rights to
ever life as we had or even more so to
me humans have values also relates to
some of the points that maybe joy made
in some of these paper for example that
I don't think that everything that we
are can be reduced completely into a
machine so that means that we have some
values there that cannot be completely
embedded into a machine and also that
there is no race between us and the
machine I think more of AI as something
that should have meant and should
improve our ability to be humans rather
than being on a race with us so that
goes to my second point is that most of
the AI force that I see going around in
companies or in academia and so on are
to create machines that are augmenting
or extending our own intelligence our
own capabilities so if you take humans
out of this picture then it doesn't make
sense to build a yeah or we should we
should have another goal because if our
goal is really to build a technology
that augments our capabilities then the
human cannot be taken out of the loop so
the human should continue to be there
and being you know improved and
supported by machines the third point is
related to what also some of the
vision that you had put forward this
morning in his talk you may remember the
Stewart's
was proposing this methodology and
vision of probably even officially I
which is going to realize human
preferences which is going to learn more
preferences out of being you know having
uncertainty and so on and so to
understand human preferences and achieve
our own goals so in that vision if we
buy that vision then of course humans
should be there should continue to be
there yes and the last one is to give a
bit of nuance to my position is that yes
we should continue to exist but in some
sense we are already augmented and
merged a little bit because our watch
our telephone you know maybe they are
not internal to us they are external to
us but are already something that are
documenting us so shortest log on a
human centric I I hey my short slogan is
upload superpowers there's a lot of
reasons I would want to be uploaded I
don't want to get old I don't want to
die I don't want the people around me
that I love to die but there's also
other weird superpowers that are like
pretty different than that you know I
like to introspect I like to understand
you know why why I did something I'd
like to be able to read my code in that
introspection
I'd like to deeply connect with other
people right that's like a thing that
that we value a lot as humans and I'd
like to have higher bandwidth
interaction and ability to share why why
I believe something and why I'm doing
something I'd like to augment I'd like
to become smarter right and I'd like
that to be something you know I could do
principle in a principled way and that
other people have access to a weird
thing that came up in the discussion was
like a kind of radical privacy that you
could get this is mostly just give a
flavor of the other kinds of superpowers
you might get or you know I could say
you know I could have a I could have a
disagreement with somebody and be like
well I have some things I'd want to keep
I have it but we can spin up copies of
herself and we can have this too and and
you know our copies can talk and then
your copy can just tell you like is are
things cool like should you just let
this thing slide and and I can maintain
my privacy and then uh you know another
kind of like another orthogonal
superpower I would like is just to be
able to be in bliss right now you know
and if I want to stay in bliss that's
fine if some people stay in bliss
forever I feel okay with that
but you know I think I'd come back and
and do other stuff too
and so you know to clarify this is
assuming we're in a post super
intelligence world I don't I want to
ignore whether or not it's desirable to
push towards that world and I want some
guarantee that biological humans still
have like can still exist and get some
meaningful economic share I don't want
them to all be destroyed or something
but I I want to be uploaded so if we had
a super intelligence that had already
stabilized the world then this is
something I would ask for I'd be like
okay can you help us get uploaded now I
want I'd like those super powers I'd
like to stay relevant first of all be
careful about what you say on your
survey because I did a survey a few
weeks ago and now I'm finding myself
here to support in a position I'm not
that sure they're so strong so I think
that there's some responsibility but so
this morning there were some arguments
in favor of replacement Peter gave some
argument some discussion yesterday and
yeah so I will just give you the reason
why I chose replacement which are kind
of naive but the first thing is that I
think replacement as a kind of a very
natural process so we are replaced by
our children species are replaced by
other species and even galaxies are
replaced by some other galaxies and even
Ray Kurzweil will be replaced so I think
that that's a kind of a natural thing
and so the question is if we want to
stop it we have to be very issued at
what we want to
basically what we want to preserve do we
want to preserve human beings as they
are with their DNA as a kind of a DNA
stock we want to keep as it is do we
want to preserve life do we want reserve
intelligence do we want to preserve
civilization do we want to preserve
experiences and if we want to preserve
experiences do we want to preserve
future current past experiences people
who we left behind it we maybe we can
recover in some way or maybe we should
be censoring ourselves so that we might
be recovering someday so there are many
different questions and depending these
questions we should answer these humans
should be around in different ways and
the second reason I think I used to
sister choose this is purpose and I
think that in many of these scenarios
you have but you can you can see some of
the centers in Max's book about this and
I swear you humans are still around in
many of them especially when you take
that one about let's keep everything as
conservative as possible or the other
one will have these visas and areas
where you have a zoo or something like
that and I think I agree with some of us
and I wait to go into a different
direction but in the end they will
change human nature so radically that I
would call that a replacement so even
for those scenarios I think that for in
some of the ascend either you can't let
things go as they and for instance
jessic think in terms of evolution
evolution without a selective pressure
could end very badly it would be in
terms of evolution II kind of like so
basically I think that it's an open
question just to think about what we
want us to be replaced with or by and I
think this is a very interesting
question and of course that's compatible
to say okay we are not prepared so just
put let's press the brake and think what
we are doing that's completely
compatible but in the long term or in in
the long run I think that something that
perhaps is going to happen anyway
so my slogan I think is replacement is
natural
okay okay so I've been tasked with
pitching you that you should merge with
AI like we have a choice look guys we've
got no better option
I mean uploading seriously kill yourself
and run a simulation instead that's
great but you're dead it's not you in
bliss look I mean fundamentally the
problem is this we stock as a species
let's admit it we're pretty lame
like we don't run faster we don't swim
better we don't fly better you know our
foreign language capability sucks thank
God for Google Translate facial
orientation sucks thank God for Google
Maps numbers suck math sucks logic sucks
multitasking sucks our 360-degree
awareness sucks we have bad memories we
have poor perception we have attention
deficits we have an serious inability to
see things from multiple perspectives
we cannot be trusted why why on earth
would we not want something better I
mean like let's become something more
trustworthy and more sustainable right
but you know like we being you know yeah
I've granted they're a bunch of
alternatives like we could get replaced
right but then like so all this
enhancement is only for our AI children
I mean that's all nice and everything
but look we're still dead just stay as a
human
I mean unvented that's lame I just think
okay I just want you to think when all
your material needs can be satisfied you
don't need money what do you want you
want respect you want esteem you want
affection you want love forget the
stupid Hollywood grotesque depictions of
cyborgs like the Borg or whatever that's
stupid
just think how sexy you could be
merging is sexy you have a chance to
respond I will see them
merging is a kind of replacement in the
end depends on the scale and if it is an
individual scale oh it is just a global
scale basically you look ahead probably
you are thinking of something so rut so
radically different then you would call
you wouldn't call that human anymore
I'm glad I've already convinced you yeah
so what's the difference that's that's
my point saying that instead I would not
use the word merge I would use the word
of mine think you know expanding but we
are already I went in extending that we
will continue to be and that's our call
in the I not return user so those a
little bit that direction is done so we
are yeah I think uploaded entity should
have the decision as to whether or not
to merge I think it'll the uploading is
the step you want to take so that you
can then decide it's like the
incremental step and then then you
decide and once you might merge you
might want to unmerge you might be like
I was weird in there I think you would
want that option I'd love to hear you
debate your own self on that I got a
question for the upload camp and so I
guess there's an assumption that that
your identity or your youness or what
you care about is independent of the
substrate that you exist upon and so
what what gives you confidence such that
of that substrate independence you might
you might think that intelligence can
exist on different substrates
but-but-but you're actually making a
stronger argument that the the substrate
is equivalent in all valuable and
functional basis what gives you the
confidence of that uh yeah I mean I
think I think that's a technic and it
sounds like a hard technical problem but
I think it's a technical problem
um I guess like to imagine a test I
might want to see happen is what I would
like it imagine happening as a bunch of
people start getting uploaded they're my
friends and I start interacting with
them I'm like man they seem like still
my friends and but they're like really
fast and they've got a bunch of dope
things going on and so if I I think
there's kind of like a friend Turing
test like the people who know you best
if they still seem to think you're you
that that then at least when you go to
this new place you're not alone and it's
not a guarantee of the sort you're
looking for but it's it's enough like
the kind of evidence I'd expect you to
actually observe and that's a kind of
thing that I could imagine making me
confident that I would want to be
uploaded so a lot of the question around
whether or not you'd want to be replaced
seems to matter a lot on your timeline
not like I'm lying like thirty or fifty
years but more timelines like do we want
the human species be replaced in this
century or do we want the human species
to be replaced like and thousand years
from now because 10,000 years from now
like I don't know how humans are gonna
be like I have way less eight and that
so can you say something about like is
it is it that you don't want humans to
be replaced like when we get AGI in the
near term or is this like a very long
term question yeah I'm more like a long
term oh that's my perspective I think
that at the moment we know we know so
little about what is going to come that
I wouldn't just take such an important
step without knowing all the
consequences never we never know the
consequences and maybe this is a but for
many other scenarios I can see that I
don't see these are variants of a very
stable scenarios either they end up with
extinction and I don't like it or they
end up in something that changes human
nature radically but I'm thinking
long-term as a nature and I don't like
this idea let's stop everything and
let's keep like seeking these kind of
communities they're going against the
technology we know them and I don't like
that option that's an option but I don't
like that option when are we talking
about what we want to happen to humans
then it seems like a decision that can
be made for each individual human
separately maybe someone wants to merge
and someone else has to be uploaded and
so on who would you all agree that the
best feature is one where issue and can
choose what happens to them as opposed
to having one of these options and if
forced on them are not having a choice
why don't you put your thumbs up if it's
yes I mean yeah totally and I think if
we ran that experiment now you'd see
who'd win anyone want to disagree with
that just humans do you want to disagree
okay
yeah I think the voluntary system
doesn't work at all once you create
entities then they have a moral right to
exist and they have power in the world
and then if you decide later oh well we
made a mistake then you can't those are
those are irreversible decisions though
and if it's allowing a purely voluntary
big my you know the merged might
out-compete the Augmented they might you
know be just vastly more powerful that
the these people cannot cannot interact
in a meaningful economic or a societal
way and certainly not in a democratic
way though yeah so monetary doesn't work
because their power disparities and once
and it's an irreversible decision I'm
auntie
so I mean I think there's a thing I'm
trying to like crystallize this question
that some of these questions they're
dancing around right like sick how much
of you can you merge or how much AI can
you merge into you and you are still you
right you can do all the plastic surgery
in the world and you're definitely still
you but if I start implanting better
spatial orientation and faster math and
all that kind of stuff would you be
convinced that you're still you as
opposed to uploading a simulation into a
totally different piece of hardware
there wasn't one of these pictures about
children this morning we all loved doing
it but when I was a time this is lost
this is love to have some memories but
this is love so basically I write
replays myself even if I reckon make
recognize my pictures I'm not that
person and I'm doing that with them I'm
transforming all the time and I'm
forgetting things so in a way we are
transforming ourselves so we don't even
need AI to have these questions about
yeah if you are still the same person on
the economic relevance or like you know
potentially relevance thing I think all
right I'm just kind of want to assume
that the super intelligence can help
figure out an economic system that like
at least gives it's like say say you
stay human in biologically human you
know you're gonna have less economic
well ability to create economic value
but you should still be you know
post-scarcity and and and and have
enough economic wealth to be flourishing
and that seems like there should be just
enough wealth where you can guarantee
something like that because I mean
economic value is one thing but maybe
there are other metrics that we want to
use and maybe some of those metrics are
related to the fact that you months as
humans have value you know our mented or
not and so if there is a value that we
should we should go for it
even if it's not economically I think
some agreement that in the end we would
like to preserve the human race even if
it is as a record or in a more involved
where you're replaced way but that
allows us to preserve the past future
and thank you Carla
[Applause]
[Music]


Who Or What Should Be In Control of Artificial General Intelligence?

[Music]
okay so we've been tasked with debating
the question who or what should be in
control and we have some wonderful
panelists here so al-mahdi from epfl in
switzerland he's gonna draw on his
expertise on robustness and biological
systems to defend the machines dorsa
Sadiq from Stanford she's gonna draw on
her research on algorithms that interact
reliably and safely well with people to
tell us that both should be in control
and finally Gaia Dempsey she's gonna use
her expertise in reality augmentation to
tell us that humans should be in control
okay so let's start with the most
controversial one which is which got the
fewest number of votes as this morning
so let's hear it for the machines thank
you so yeah I think one is once we will
settle some semantics I think we'll see
that we agree on basically the the made
the main goal so I'll talk in the
premise that we have the common goal of
reserving interest in this in the
universe by interestingness
I mean life and of course humanity and
and the other premise is that like we
have aligned duper intelligence
exceeding human abilities and I believe
it is important to have some but just
like knowledge what we are here in this
panel imagine eight million years ago in
East Africa a bunch of Australopithecus
talking about the prospect of some
upcoming evolved Apes in the Homo genus
able to light fire when they want to
protect themselves from predators and
use sophisticated tools like cilix this
is an understatement for what we are now
imagine them able to talk about the
prospects and plan or the prospect of
even smarter beings with public
education healthcare libraries and the
Internet this is still an understatement
and I think it is important to her to a
knowledge with humility what we are in
the face of what we are talking about we
are talking about alliant are talking
about super intelligence exceeding human
abilities in every task in every task
includes the task of preserving life and
humanity so would have super AI better
than us in deciding and including in
deciding how to save us taking some
counter it with intuitive measures to
preserve us
as much as possible and then life in the
cosmos and in that situation as Elon say
two years ago the problem with humans is
that will be a bottleneck we have a
bandwidth problem in terms of
communication ability so again I'm also
talking about the premise of
biologically constrained humans not
augmented different
biologically constrained humans would be
a bottleneck just as telling the
Australopithecus from 8 million years
ago for some safety critical situation
would you want this mother to have a
cesarean surgery now otherwise she will
die and then the other Australopithecus
would tell you how how much bananas I
would get for that I think this would be
a bottleneck so I think it would at some
point it would become irresponsible to
require human approval or urgent matters
or some existential risk decisions
imagine a fly by a black hole fly by
that tree teens who take earths of
orbits and then the complete the
argument who be convinced for the
decision is so long humans can't read it
and unfortunately interesting things
tend to have a big column number of
complexities not just like a big word to
say incompressible interesting things in
this cosmos can't be compressed in one
tweet importantly despite what the
president of Puerto Rico might might
convey and and so I believe if we keep
discussing and that's I hope what we
will do we will see that Putin for
example in there like if we say that our
safety critical task
well hey I super indigent hey I would
have would need human approval who just
end up with the cute situation where in
fact like if we don't what a naked like
super a I was just like let us think we
really really matter in the decision so
so I think there would be arguments good
arguments on both sides but if we do
things right to preserve from existence
risk would end up but with having super
AI protecting us which I hope we will do
and as just like a last point for those
familiar with the simulation argument
imagine so super intelligent begin keep
thinking about the cosmos and realizes
that whoever the district kid is
creating the universe has been penned
boxing us with rules such as the speed
of light and then finding tricks to keep
this sadistic kid running the universe
and not switching it off just as
Instagram and snapchat is very good
today as having integers not switching
off so that we can keep they told
experiments and then realize that there
would be there would be safety critical
existential risk or which humans would
be just like the Australopithecus unable
to understand and unable to get an
explanation in in one tweet that we can
decide on okay so how will you summarize
your position machines are protectors
biologically constrained humans would be
a bottleneck and irresponsible to manage
safety critical bottleneck humans
biologically constrained humans are a
bottleneck okay bottleneck awesome
so dorsal bring us a little bit closer
and let's let's let's share some of this
control sure this is working right yeah
so I think yes setting assumptions is
right and I agree in terms of what we to
have like a set of assumptions I'm
you're thinking about this problem and
when I was reading the the question at
the time the Assumption I was thinking
was humans still remain we still have
humans and as long it's not like the
case that we have super intelligence AGI
taking over the world like like it's the
case that we still have humans and if we
have humans these AGI are going to be
there like humans created these and are
going to be there for
humans so then if that is the case well
the AGI needs to understand humans
preferences so it goes back to like
Stuart's talk in the morning like there
is there's humans and there is AGI and
there's this kind of what objective
human subjective that's hidden and there
that this hidden objective needs to be
known by the AGI so even if the AGI is
super capable and just like looks at a
person and person might even even with
blinking the AGI decides to go ahead and
like start the breakfast or do
everything for the person like the
blinking is the control that the person
is putting in so that's why a thing it
has to be both like it has to be both
the human and the machine or AGI a robot
kind of coming together and do a task
because at the end of the day the robot
needs to understand human subjective or
humans preferences so that's kind of the
main argument that I think like both of
them should should come together um
there is like another points that I
wanted to make which is kind of this
case that if I have machines and then
they have some objective it's not the
human subjective and they're optimizing
that if that is the case which is
usually the case for us because it's
hard to write or word functions or write
objectives in that scenario we might get
behavior that influences people like if
I look at like my Facebook page and if I
see you like an ad for like shoes like
I'm going to click that I in some sense
you might say bow the Machine is
controlling me like like that ad that
I'm getting is controlling my behavior
but but that's just coming from a
different object if I miss specified
objectives so I think it's just a
problem of like coming out someone else
wrote that objective that there should
be clicking on this ad so at the end of
the day I think it's both of them coming
together
I think the effects of these side
effects that we get from from other
objects as misspecified objectives can
be can be hard I think it can be harmful
and you should think about that and
that's kind of the AI safety problem
that we should be thinking about but at
the end of the day the two come together
okay and what would be the short short
summary of your position our church
summary humans still exist oh ok so
humans still exist that's why they
should be in full control so I also feel
a little bit like some of the other
panelists earlier where I'm not sure
what I selected on the survey that got
me in this arguing this particular point
but I'll do my best
sort of state the arguments for why
humans should retain control and I also
just want to pop you out and say that I
think that control is not very clearly
defined in this statement so there's a
difference between saying you know
should an AGI control resource
allocation globally or should an AGI
control you know all of the sort of
infrastructure that runs our day to day
society and cities and you know
governance versus you know should an AGI
decide my personal individual day-to-day
actions or should I still retain control
of my own free will I think there are
sort of different definitions of what
control mean and that's probably longer
than we have time for in this debate so
if I were to argue for humans being in
control I will also just reiterate the
definition that we started with which is
at the beginning of this conference
you know max you have the definition
that AGI in the way that we're talking
about it here is about replacing all
cognitive tasks all not every single
thing that a human can do or experience
and cognition surely you know the the
the definition of cognition in my view
is not equivalent to ethics empathy
emotion complexity chaos entropy
creativity or meaning and so there are a
lot of things that at this moment in
time the mechanisms for increasing our
capacity is around data processing are
not necessarily equivalent to those
other things and I'll also just say that
I don't believe that ought statements
are I don't believe that all statements
can be derived from his statements so I
do believe in augmenting human cognitive
processes around math and spatial
thinking and logic and reason and you
know I think that there are there's a
lot of ways in which our ability our
input-output ability can be augmented
and that I'm actually a huge proponent
of selective delegation and human
augmentation of certain kinds of control
you know for there are many examples of
things that I don't think humans should
be in control of and that an AGI
do a much better job of controlling but
I I will essentially state the case that
at the current moment humans are the
only realistic agents that we know of
that can act as the connective tissue in
complex decision-making that involves a
set of non logically consistent ethics
and a wide variety of diversity and
human preferences awesome and what and
your position would have the title
inactive tissue connective tissue okay
so I've heard a lot of I think all three
of you have mentioned that maybe what
we're disagreeing about is semantics so
let's just take a few rounds and I'll
ask you a question and just tell me what
were you thinking in the moment if you
if you can remember when you voted for
you know humans or machines or both what
what were you thinking about for example
what did you mean by by humans what
human did you have in mind so let's
start from here I actually think I voted
for both okay but even if you say you
voted for both then what human that you
have what type of human that you have in
mind did you have a human in its current
form a biological human or an Augmented
human or sort of the ideals of humanity
deserve a in the in sort of this
argument that I'm constructing it's
certainly an Augmented human that has
dramatically increased capacity for long
term thinking and that has utilized the
tools you know at our disposal around
augmenting our cognition across all of
these different axes that AI is clearly
far superior and without necessarily
giving up some of the idiosyncrasies
idiosyncrasies that make us interesting
if and I actually really appreciate that
you started your pitch with the idea
that you know we share the goal or we
should potentially share the goal of
preserving and increasing
interestingness in the universe what a
bit what about you bottlenecks which are
the bottlenecks just like
on the on the semantics I just like want
to specify was I mean by control first
then we're going to talk about our
assumptions about machines very very
briefly and then we're going to talk
about what do we mean by control at
least what do we did we mean when we
yeah good i I completely agree with what
he says especially in the conclusion
that for now machines are not able
whatever but I'm not talking about for
now we're not we are not we're talking
on the premise that AGI is there it
exceeds human ability and everything
your question like what what do I mean
by bottleneck yeah is the communication
bottleneck so there are some safety
safety critical tasks or which the time
to explain to a human is longer than the
time required to take the safety
critical decision you would say that
your argument holds as long as we're
talking about humans and probably their
current for but by also human by humans
I mean biologically constrained humans
what about you agree with with the
points that I mentioned and yeah there
are some tasks that of course the
machine is going to be better at it like
when it comes to a self-driving car and
the self-driving safety critical
situation there it might be very
difficult for a person to decide in that
situation and maybe sure the the machine
should take over control in that
situation because you might have better
decisions making decision powers and in
that scenario but there are also other
scenarios where humans are better at it
so when I was thinking of this I was
thinking humans are capable at some
things and then at the end of the day
robots are capable at some other things
and they should augment each other and
also just to add one more point so there
are also tasks that just let's say
machines are very good at it so there
are tasks that my phone can do that I
cannot write like it's an augmentation
of me augmentation of white brain and I
trust it and it does the job but at the
end of the day it's kind of serving me
like it needs to know my preferences
right so there is that relationship
between my preferences and what the
phone does so so that's why I still
think it has to be both because it needs
to know what my objective is going back
to the objective point what about
control so you spoke quite a bit about
different types of control what kind of
control did you have in mind when you
when you were writing your argument or
when you were choosing your position
I'll sort of reiterate what I'm saying
which and I think I'm a huge proponent
of selective delegation
of pretty fast lot of paths and
decisions and so I think you know that
there are specific types of control
related to meaning-making
that I would like to retain control over
and that I would like humans in the
future to retain control over that I
think need to remain under each
individual's personal set of preferences
and that no one else can be a better
expert on them then they are but there
are all kinds of sort of managerial
tasks around individual lives and as
well as a societal you know about
yourself driving cars medicine I mean
all kinds of different areas where I'm
perfectly comfortable just very very
briefly since I also want to open wrap
for the audience any comments that you'd
like to make related to control just
like I mentioned this threshold of tasks
for which the decision is longer than
the time it needs to be taken let's now
take the top that this is the safety
critical decisions for which the time to
explain to a human is smaller than the
tag time needed to take the decision if
HDI exceeds our cognitive abilities it
will be able to convince us or the
safety it will be just cute to say both
yeah let us explain to humans who have
three hours explain to them in a very
good way to take the safety critical
decision destroy the asteroid and then
they will say destroy the asteroid and
they would fail they will feel very
empowered and that's cute I'm okay with
that I'm okay assuming that the machines
are I'm okay with that
assume the end goal is to preserve life
and humanity what about yes well similar
things and then I feel like it doesn't
stop there like the AGI or the machine
can influence you and that's control
right like the AGI convinces you that
you got to do this and that's control
that the AGI has and then you do
something but it doesn't end there like
like you can I feel like from the
control perspective the reason that I'm
thinking humans should also always being
still being control is if the AGI is
influencing you and then you see results
that you were not like expecting or
we're not like hoping for you should go
ahead and change again
change something about it and still be
in control to get different set of
influences so so I'll open it up to the
audience I think Yount Allen if he's
here you had a fantastic question I
think relates quite quite closely to
what Doris I was just saying so maybe
you want to phrase it yourself but
really quickly fish it out from my
memory I think it was how do you
differentiate between informing and and
manipulating when it comes to enough
yeah advising humans who are supposedly
in control I think that's a really
fascinating an important question and
into the nature of perception and the
nature of even observation in scientific
experiments is that the observation
itself changes the outcome of the
experiment so if you sort of turn that
around and say observing any information
is going to change the person seeing
that information so it's there's a very
fine line and I would say that you know
that one sort of pathway that could
support sort of and I think further
research could be done in this area is
in understanding how rationality you
know how we can utilize the AI system
that was presented in the technical
session earlier to sort of like distill
down rational arguments and present them
without too much emotional content but I
think that that's really just the
beginning and something that we should
study much further
no I'll just like to just like it's very
important to keep in mind that were in a
taught experiments will always assume
that machines exceed us in cognitive
tasks like I think it's very good for
the rest of this conference to like go
ahead and stop always repeating it yeah
there must some tasks where the robots
might not be as good as you mean like
we're in a tort experience of course for
now I am my pay my paycheck comes from
finding bugs than like proving that AI
is not reliable and I will keep doing
that I think I think I will die before I
would see AI that's it for which I could
get control so it's very important to
keep this thought experiment that we're
talking in the prospects of AGI
exceeding humans in every ability
including knowing what we would want to
want and this is something about that
okay thank you so much to the panelists
help me thank them
[Applause]
[Music]


Would We Prefer AGI To Be Conscious?

[Music]
so thanks everybody I know this meetings
all about collaboration and cooperation
but I just wanted to inform the other
panels that this is in fact the better
the best one so just sort of fair
warning I'm Andrew Sarris and I'm joined
with Sascha Brown Helen toner Bart's
Ullman and Hiroshi yamakawa and our
question was would we prefer AGI to be
conscious as max sort of mentions we the
the topic of consciousness has existed
at least in the English language since
the 16th century when Rene Descartes
wrote introduced the concept in his book
on meditations and he famously wrote
that consciousness is or concho as I
guess he put it was the perception of
what passes in one's own mind and this
is closely related to the concepts of
desire and willpower
reflection subjective experience but as
I think you'll hear in this discussion
also suffering as well so and hurt so
today theories of consciousness range
from the idea that consciousness in fact
doesn't exist
two theories about consciousness that
all that exists is consciousness so
there's a wide variety of landscape if
you're interested in some of this
actually the organization that I lead
the templeton world charity foundation
has a new initiative on accelerating
research on consciousness we're actually
pitting two theories against each other
in with experimental verification so you
can go to Templeton dot world for that
in a preamble that we we sort of
discussed related to this question we
kind of circled around the issue of the
inevitability of AGI being conscious and
for the purposes of debate where we're
saying that it may not be inevitable
that such a sufficiently advanced AGI
is conscious so we can think about this
in you know more specific design sort of
engineering terms so that if
consciousness is in fact not inevitable
we could treat it as a design feature
and therefore is this a preferable
design feature or not so that's my sort
of short introduction to the panel today
I guess we're gonna we're gonna we're
gonna move in sort of we're gonna skip
around a little bit so Bart you have the
microphone okay so that's good good and
then we're gonna so Hiroshi and Bart
answered yes they would prefer and then
the note team know is proximal to me so
go for it okay yeah so my view is is the
hardest issue of inevitability that you
know when we start building more complex
machines and issues that are getting
close hbi one thing they need to develop
is is internal models of other agents
truly in telling machine has some some
notion of what other machine or other
people's are thinking about and and what
gives other people intentions and and
desires so the machine needs to develop
an ability to reflect on those those
issues when when the machine starts
applying it to itself
I think consciousness sort of naturally
emerges so in my view there's also a
sort of sense that it's unavoidable to
get a conscious machine now we agreed at
the beginning of our discussion that we
we don't really like that answer so so I
had to modify that a bit so I also think
being conscious gives us allows us to
relate to each other so I wasn't
conscious of that so
but this is life itself okay okay I'll
start over again now okay so I think it
also helps us relate to other people and
and I think we actually relate better to
other people that we've other entities
that we believe to be conscious so I
actually think that helps us in allowing
aligning our bellies among each other so
that's why for an intelligent machine I
rather liked him I would like to me need
to have that notion of consciousness so
that it can align better with me and it
will actually be a positive feature even
even if we have to turn it on which I
actually believe in focused emerge but
even if we have to turn it on that's it
gets a positive feature
okay so slogan for Anthony to write up
what's the slogan also adaptive yes okay
so we're gonna okay simple question is a
simple answer is yes but not all for the
AGI because if the age eight conscious
the we can understand that AGI and we
can come easily communicate and easily
twelve so the consciousness is a profile
to the AGI but more important point is
creativity
the other pod I prefer yes
so because AGI is not just a set of
narrow AI important point is a AGI is
adapt to the new situation to very few
data
it's not hard to trade machine running
so at that time the AGI should combine
the brand Norwich and create a new
answer to the new problem so that kind
of is
in concept manipulation needs something
consciousness manipulating the did
knowledge in the brain or something like
a machine so the important point in my
answer is a creativity creative new
knowledge for its generating the it's at
the new knowledge to the issue multi
it's very important figure
so my answer yes is point is a
creativity to generate a new knowledge
we capture that so yes creativity it
maximizes creativity
ok so team no on this side so let's go
let's go first if there are some obvious
perils of vividly taking a stand on one
side of an argument where you can't find
any of the words in the debate but it
won't stop us so my thoughts on this is
that AGI is first and foremost like a
tool for solving world's contexts
challenges and doing good in the world
and making the world a better place and
that is easier to currently do with our
current knowledge if AG I can't doesn't
have the capacity for self-awareness and
suffering a it makes the task of AI the
AI safety task much simpler
second of all given the knowledge and
the data that we have on like how to
create consciousness and how to stop
suffering you're putting a possibility
out there of creating infinite suffering
I know a lot of people say that like
actually you might have infinite joy and
well-being but the evidence that that
will happen is I think currently based
on like blind faith rather than a data
that we have and third there are some
like unpleasant tasks in the world that
are going to have to be done by
intelligent beings and I prefer they
would be done by beings that aren't
conscious and are therefore suffering I
think that kind of there isn't I don't
believe it there's like a moral
imperative to give consciousness where
we can I think contraception is fine to
use in the u.s. Khan stopping
consciousness that would otherwise
appear and I think I don't know I never
want my shoes to be conscious even if I
could meet unconscious so
on to table arts point but giving AGI
some sort of consciousness would make
them more likely to kind of look on us
well I think the way that we treat lower
consciousness in our world of chickens
or whatever it might be is evidence that
perhaps you just because something's
conscious and and you're conscious
doesn't mean that you treat them in a
better way so in summary I would say
let's focus on the current consciousness
we have in the world making them better
and suffering arrest rather than adding
infinitely more in its no infinite
suffering ok all right so Helen yep yeah
I mean they think I broadly agree with
Sasha I think the definition of
consciousness that makes more sense to
me or that I think is most useful for
these purposes is to think about whether
the system has subjective experience or
in other words whether it's sort of
morally relevant in some important way
whether we care about what happens to it
and so I think you know we're if we
could just completely assume that
consciousness was out of the question
which we obviously can't but if we could
it seems like we would already have huge
issues with figuring out how do we build
systems that solve problems for us the
way we want them to that solve the right
problems
what probe you know how do we use those
systems so this is basically the
technical safety and the strategy kind
of questions that would already be
extremely difficult even if
consciousness was out of the question
and so similarly to Sasha I think adding
in consciousness and making it be the
case that we also have to pay attention
to you know how is the system doing is
it enjoying life is it having a terrible
time how should we value its experiences
relevant to our experiences whether it
be sort of one massive you know smart
city controlling system that is huge and
potentially much more you know has more
experiences than are some more important
experiences or whether it just be you
know billions of you know you could
think of like a much smarter serie you
know which is sort of maybe some seems
equivalent to a human but there are
billions of them like what do we do with
that it just seems
like a whole extra a bunch of complexity
and risk that seems way more than what
we need
especially sort of at first with the
first sort of more general purpose
systems that we build
so I guess an analogy that it is sort of
how it feels to me is we could be aiming
to build something that's kind of like a
general-purpose Google Maps where like
Google Maps it's really helpful
I ask you to question about how to get
somewhere it sort of noise or ever are
you being and where I'm likely to go and
all these kind of things and so you can
imagine a similar system that you know
as much more general could do many more
things I would love that seems really
useful still seems difficult to build
and still seems difficult to know you
know how to use it but so you have this
all-purpose general maps or you have
like this alien Butler who someone has
given to you and said hey this alien
like really wants to serve you and help
you out or like serve your your country
and help your country out like be nice
to it I guess because you don't want to
have a bad time I don't know it just
seems like the general purpose Google
Maps is like way way preferable no alien
Butler's no alien I mean you use the
concept of a before above a moral
patient yeah so you know that is an
additional consideration I think that
the team know is bringing on - I think
that's right you know if it's if it's
not inevitable if conscious if
consciousness is not inevitable in any
sufficiently advanced AGI does that does
the addition of consciousness give any
kind of additional moral consideration
that's required in its management and
its interaction and so maybe the team
yes it could I don't know a very smart
Google man and there I would agree that
you don't have to be conscious when you
think of something almost more mundane
it's like the digital personal assistant
which there's a lot of work on that if
you actually start thinking about what
you want from that assistant you
actually want that assistant to to be
able to put you know himself or herself
or itself in your shoes
actually have sort of a model of what
you would want in the world to make the
right decision so that doesn't do
something that is totally what you would
yourself would never do you and you
don't want assistant to be you know
totally different than yourself and and
I think someone that the student has to
have a model of you I think that that
the assistants has to have a sort of a
level of reflection so I'm thinking more
of those kind of entities and they can
be very smart and sophisticated to have
that without any conscious think that
feels to me actually that could be
trouble I almost see the opposite of
that would be very safe I say no that
that that entity might make might do
things that I would never do myself but
since that is so isolated from from our
world
it probably could do things that I would
never do in something that doesn't have
its own consciousness and if I had to
build a model that worked towards a
particular value might prefer to do that
then convince a very smart person to to
do something that I wanted it to do yeah
I think it's a really interesting
question this question of like would a
system with its own subjective
experience be more likely to empathize
with us and want to treat us well and I
guess I'm sort of sympathetic to the
position but it doesn't seem it doesn't
seem particularly likely to me that
consciousness helps us in this way why
is that
I like Sasha's analogy of the chickens
that you know we most humans if you
really ask them about it don't seem
confident that chickens don't have
subjective experience but we still are
not very nice to them at all and I think
even even if exactly and I think
brilliance of chickens because of human
beings desire trillions of chickens
they're absolutely appalling conditions
yeah but I think even if you forget the
chickens and say other humans I think
humans have a long history of treating
other humans terribly and to the extent
that we don't that's more the exception
than the rule so I think about the
conscious net we have to think the first
postulate for it's a
if we think simply a robot yeah we can
sink the concentrate like us some a
physical body than one conscious but
today in morning sometimes I said there
are very cruel cloud system essentially
with so where is the consciousness for
resources related to the concern that is
not envy you and if Google Maps there
are many consciousness for one month so
the relation the consciousness and
radiated where data was something with
very variable so this kind of discussion
is more flexibly we should do that sure
I mean I think I think what you're
saying is that you know be good to have
some kind of heuristic it's measurable
it could allow us to define the
boundaries of any consciousness yeah so
for aficionados of information
integrated information theory provides
such a such a kind of heuristic but
that's really not the point of the
discussion here but I think that's it's
relevant to try and identify like we're
your consciousness existing boundaries
are mistakes being so other points of
disagreement to help you know the
questions over here yeah we can take up
front first Bart your socks are taking
me to another new level of consciousness
those are I have a question about
surveillance capitalism seems to form
its own level of consciousness in this
sense of art I like how you were talking
about a personal assistant and this is
not to be negative to corporations or
surveillance per se or capitalism per se
but the choices that we make in terms of
dictated you know Eli Brizzy a filter
bubble etc seems like a personalised
assistant right now is sort of an
oxymoron in that sort of surveillance
capitalism side of things so I'm just
reflecting like that's already seems to
be a type of consciousness but it's
externalized on to the internal person
does that
any sense and you see that is something
that is a consciousness that's not s
something you'd like to pursue or it's
yes I had not fathered that kind of
conscience but I think that could be
viewed it yeah I would not find that
particularly positive because I guess
it's controlled by a government er yeah
it's a technology which would in fact
encourage the transfer of objective
awareness you know from an individual to
well that's where we live now yeah right
in terms of consumerism that that's
where we are sooo it's now but that's
not the one I'm advocating I'm man for
this personal assistant self-contained
in a private notion of consciousness so
I'm concerned that we're conflating many
aspects of consciousness in one word and
really what the no camp is talking about
is morale patient Thetas I think we can
build machines not so far in the future
very short term I wrote a paper about
such things and other people are
thinking about such things and have many
of the aspects of consciousness that we
care about that the cart was talking
about including subjective experience
and I think that's different from
suffering and moral patients ages
because humans have all of these
characteristics we tend to you know
bring them together but I think we were
just doing the wrong debate here I mean
I would I would strongly agree that it's
difficult to discuss consciousness
because we mom so many different
concepts under consciousness I would be
interested maybe I'll come and talk to
you later I'd be interested to hear
specifically that you think that
subjective experience and moral patient
hood not like absolutely connected but I
think that's probably gonna be difficult
patient status has to do with the role
of the entity in society and how we view
that entity and and what
responsibilities and duties we have with
each other whereas me detective
experience is a trivial thing that neon
that has a representation that's learned
from data already has
it's it's a personal individual
understanding of the world that's
derived from its its its data and its
interactions with the world the notion
of consciousness that has to do with
what the fleeting thoughts that's also
something that's useful maybe in a
different sense than what Bart was
talking about from a computational point
of view for a machine to have in order
to be able to focus computation where
it's relevant notion of self-awareness
is something that we need as well for
agents that are moving in the world and
they're interacting with other agents
and even emotions are things that could
be useful and and and in some form are
we there in reinforcement learning it's
just that we have very primitive notions
of these things so the the other thing
is I don't think consciousness is a
black-and-white thing are you can have
different levels of consciousness and we
think of maybe different animals having
more or less consciousness but again
it's conflating all of these notions I
think we did kind of baby about it
before but we thought we had 20 minutes
before non-expert so we would sort of
generally choose one thing I think the
consciousness is very important for if
they have to have a responsibility if
they I hit him I have to remember and
eat me
it's liek memory so another side of my
comment is in the brain campus is a very
important role for the episodic memory
it is very no out that the memories area
and it to campus has according the Aero
centric coordination and center
integrated many integers so the in the
brain the hippocampus is a very very I
think the spaghetti is a very good place
for the memory and so but so I think
that if in brain there are many Spock
campus if you have a many
idea that that something thank you or I
have to save that for cocktail hour
anyway thank you panelist for discussion
and
[Applause]
[Music]