274. Conjecture Internal Infohazard Policy

hello and welcome to session 274 in the
AI safety.com reading group tonight we
will be discussing the post conjecture
internal info asset policy by Conor
Leahy and others
conjecture is a relatively new AI
alignment organization led by Conor
Leahy and the other authors are Sid
black Chris camell and Andrea miyachi
this is a post on listeron which is just
under a year old by now
so let's first talk about what is an
info Hazard the term info Hazard was
defined by Nick Bostrom in his 2011
paper a typology of information hazards
in which he defines an info Hazard as a
risk that arises from the dissemination
of true information that may cause harm
or enable some agent to cause harm
and because this is Nick Bostrom then he
obviously has like this huge typology
with 33 distinct types but in this
article we are considering a much much
smaller subset
um only the the risks that come from
accelerated AGI timelines and this is
both only like one kind of info Hazard
and it's also restricted to one specific
domain so like if it's information that
accelerates I don't know bio weapon
timelines or something like that then
that's not really relevant to this
yeah and I think this raises the obvious
question what if uh an info Hazard is
uncovered that doesn't uh uh fit into
this um like maybe someone at conjecture
finds a prompt that is really really
good for like creating uh uh
continuations that ends up with like bio
weapons or something like that or worse
than maybe it's a new bezel is
discovered or something like that in
that case we would probably want to do
the opposite of what this is in
particular we do we would not want to
tell Connor if we discover a new uh
Basilisk
uh so the this specific kind of info
Hazard is one that other people are
obviously interested in about and two
weeks before this post was uh written uh
published to less wrong
umkowski had a kind of similar post
about info hazards where he asked for a
name for this uh like a social Hazard
and the one that the one that he
believed was best was an expo Hazard I
must admit I haven't seen other people
use extra Hazard except here so the name
hasn't really caught on and
we'll just continue referring to it as
info hazards
and one of the things that I
um
struck I think probably are
also covered by this info Hazard is like
how does
um conjecture uh
uh deal with confidential information
like for instance pivotal acts if they
get information about those
let's start by talking about the goals
uh
of of writing this document conjecture
lists those goals but I would like to
before that compare the the default
situation where conjecture doesn't write
that post because that has in fact some
advantages like this is Streisand effect
where
writing that you have a
um
and that you potentially have very
dangerous information is something that
draws attention to yourself
um and it is a possibility that just
normal business confidentiality like all
other businesses have would have been
sufficient uh
uh this is of course my thought solution
or something that can that the
conjecture talks about uh explicitly
here
um so one of the goals is to encourage
accountability among the people who may
spread information and one limiting case
that I have uh considered in detail is
Chris Leong uh who um on Petrov Day in
2020 entered the code and took uh took
down less wrong uh mostly because it
didn't take it seriously and he was
maybe also cheated but he also didn't
take it very seriously and this kind of
thing is the kind of thing that could
happen uh with uh with dangerous
information and the question is like how
accountable should we hold him I think
people have mostly forgiven Chris Leung
and I'm actually unsure whether that is
actually correct
a second goal of publishing this is
promoting cross-organization
collaboration
I found two people Tammy who has
something like lock posts that
references and the new organization
called orthogonal that doesn't implement
this policy but just say this is
reasonable and apart from that I haven't
seen any other organizations that have
like made any kind of clear reference to
this that doesn't mean that there is
nothing there right there could
certainly be a number of people who have
collaborated more with conjecture
because they know they have this policy
and perhaps also uh
uh people taking up this kind of policy
without making it explicit but nothing
has been written as fast I can tell see
and finally the goal has been to start a
conversation about uh info Hazard
policies
um this is something we've seen
previously work out in rare cases for
instance uh there was a push towards
having a policy on publishing your AI
alignment strategy for the AI
organizations and that did in fact turn
out to in the end cause anthropic and
deepmind to publish their
um their views on strategy that required
a lot of push like I remember I
personally uh talked with a number of
the of the people at these organizations
and questioned them about why they did
not publish anything like that and
eventually they did in fact publish this
um
and I think uh these these organizations
um I think there is a very wide uh
discrepancy between the organizations in
how uh how they would react to these
kind of uh info answers I think deepmind
and um I don't trust at all I think
these organizations if they learned uh
about secrets that uh from conjecture
and how to train AI faster they would
just you know build it as fast as
possible
um orc and anthropic I think are both
questionable all and are actually
training AIS and Tropic I would be very
hesitant with sharing information with
those the alignment Research Center and
of course conjecture seem like
organizations that
um like have their together uh even
if I don't know enough about the
organizations to really trust them and
Miri is the only organization I would be
confident in actually informing them
about this kind of Expo hazard
what are the considerations uh for this
uh for this policy
um one of them is that we need to be
able to coordinate that's required to
share information and ensure we're not
working at the same thing and like we
are prioritizing towards the the right
areas some kind of in exchange of
information that is potentially in for
hazardous is required and I think the
reason why this is required in
particular is because we are very much
in a race against the capability labs
and that means that
um
that we cannot accept uh I don't think
should accept a too limiting uh info
Hazard policy because that will just
mean they'll be too inefficient
another consideration is it's hard to
tell in advance how dangerous and info
Hazard is and that's of course a problem
both for conjecture and for the people
around them uh and potentially
adversaries
um so um that's an ameliorating
ameliorating factor in my view
there's a consideration that secrets are
hard to keep in particular if you have
to keep them for a long time and if you
have many secrets
I don't actually think that these two
considerations are very strong like the
sure it matters it's harder to keep a
secret for a long time but like the
number of people who know the secret is
way more important and like how what are
these kind of people are they different
people are they
um like the perfect thing you want to do
is something like the CIA that uses
moments very much like this
stereotypically completely non-diverse
and that makes them much more capable of
keeping secrets
they're talking about a trade-off
between Safety and Security at least it
may be a trade-off because in the
upwards sealing of information is in
fact not really a part of a trade
trade-off there isn't as fast I can see
and always trade-off between Safety and
Security
um at least uh there may be something
that I haven't thought about but um and
finally functional decision theory is
described as an important consideration
functional decision theory is that you
should treat your decision as the output
of a fixed mathematical function that
answers the question which output of
this very function would yield the best
outcome
so in order to uh Analyze This there are
two things that could be the output
either a policy or a decision to leak or
not to leak
um and like precisely how you would go
about analyzing this depends on
precisely what they want to
um uh
um
how this analysis goes so I didn't this
may be something we can ask Chris
and finally of all these considerations
I think they're all important but I
think that's one information when one
consideration that really trumps all the
other and that is like we need to get
some kind of a measure of uh what things
are info hazardous and how info
hazardous are they that seems to me as
like the key consideration
okay what is covered and what is not
covered in particular the thing that is
not covered by that is not one of these
info hazards are PR hazards and
reputational Hazards I think that's a
really good clarification because a lot
of companies would certainly keep this
kind of thing uh confidential
um
they try to strive for meter honesty and
that means to be honest about how honest
you are
um and I had some questions about that
but Ilia suit wants that it's dangerous
to ask questions about meter honestly
because you can very easily get to Pro
like object info object little
information like if I ask them about
like
if Miri came to you with a like a
pivotal act uh would what would they do
about that how would they react to this
then it's very easy to choose this kind
of uh information
um search probe for what are they
actually talking do they have this kind
of secrets
so I'm unsure about the best way to to
go about this I'm also unsure about the
best way to go about meter honesty in
general and part of this is because the
incentives are very strong
um like obviously the incentives inside
the organization are extremely strong
like corner for instance I expect you
can probably fire people at will in
Elizabeth talk about this to talk about
people who have like a gun and of course
that's the extreme example but in fact
if Conor wants to fire someone and then
he probably just can so meter honestly
is very hard in this kind of not only
truth-seeking environment
and of course there are other people the
government probably can go to uh an
employee in conjecture and say hey tell
us this or you will go to jail and
you'll get some kind of gag order
against it and that seems like an uh
another problem in general I think me to
honesty is hard enough that if you want
to have like a policy on this you should
reference some existing work on what
this means on it what gives me honestly
actually
and then finally on what is covered uh
they Define it using a limit like the
least info hazardous thing that is still
covered by this is letting it be known
outside of conjecture that they are
interested in using building or
deploying a technique that already
exists in the literature in order to
train networks faster
I think this kind of formulation with
like this limit is here is really great
um I'm not sure I really care very much
about training that works faster I think
that that's like the framing that
conjecture is using and that a lot of
people are using and I think like
smarter more intelligent is really
better in some way
um
uh in in practice of the thing that I
actually care about is whether the
there's a way to train a network to be
more capable among customers six
strategic cognitive tasks uh that's
that's like my horse my idiosyncratic
view on aict that this is in fact the
thing that we should care about
so the rules that conjecture setup uh
have three levels of disclosure secret
private and public uh well with the
secret being the default and they will
think about this after trialing for some
months
um
pen pays in the comments were apply that
this there's a process for changing
information between these level and that
seems really like it's a lot of work to
change something from private to secret
um
uh
this information is that three there are
three levels and three types of
information repositories projects and
ideals and the projects have access
documents with some uh information about
like the secret uh it looks like
repositories don't have access documents
and I think they should have that I also
think that some of the uh rules if
they're written like uh meant to be uh
uh understood by a lawyer or something
like that like these kind of rules end
up becoming like legal rules in some way
um and then there are some formulations
that need to be tied up like knowing
about a secret knowing as secret uh uh
perhaps two different things
uh in this rule Set uh Connolly has a
very special role he's the appointed
info Hazard coordinator he has access to
everything
um
and it's like it's unclear precisely how
he's supposed to interact with it like
he could uh like Donald Trump uh who
claimed he had Declassified uh uh secret
documents in his head and it's not
entirely clear from this whether Conor
actually has the right to just
declassify things in his head and if he
does have that uh then
and then of course the uh the rules uh
binding him are much less strict
Ben pays also in the comments that this
is a potentially very problematic
position for Conor because a CEO may not
be available uh at all times for
employees to talk with uh and I think
this is putting a substantial amount of
responsibility on the CEO who like have
more important things to do
how do they deal with policy violations
well they hope people in conjecture sign
ndas I think getting these ndas actually
enforced seem really really difficult
like if we get something if con if Conor
or conjecture believes that something is
important enough that it is info
hassleless that probably means that it
is something that is strategically
valuable and that and for that reason in
the A's by definition don't cover things
that are in the public interest
um and whether something that is in the
National interest is also in the public
interest is an interesting question it
is not clear that these ndas will hold
up but I'm no lawyer and of course if
they did try to enforce them then the
stressing effect would certainly kick in
and uh for the policy violations
basically Conor has the final uh say he
Stakes his reputation on uh under the
his judgments uh mistakes in the
beginning are of course acceptable and
uh as he says nobody will be fired for
raising a concern in good faith I think
thought that was a kind of a strange
formulation like obviously the thing
that I would expect him to write is that
no one would be fired for admitting and
not very serious mistake in disclosure
during the first couple of months but
that's actually not what he's
technically writing
and this also uh uh this policy is also
in charge in effect for for the uh the
leadership of conjecture and it's
Gabriel who will initiate a process
against Conor
and uh there is a final uh remark that
the policy needs to be practical and
people who are doing experiments in
conjecture which they're doing all the
time they need to look at each specific
case and see okay this is actually not
really dangerous at all and then no
review is necessary and I think this
kind of practicality is probably
required
uh conjecture has a quarterly policy
review uh I won't go through it here but
let's just I'll be really interested in
whether this is a thing that happens and
whether it's something that actually
finds issues uh because I would put at
least a non-zero percentage probability
that this is the kind of thing that
people just forget because everybody's
too busy
finally there's a chapter about the
psychological safety that I think is
important because Secrets do take some
kind of emotional toll and they are
upfront about this and people get
stressed and isolated if they're having
to do with secrets and this kind of
stress and isolation make people more
likely to uh than uh give up on the
secrecy and
one of the things that they're trying to
do is to not have people only working on
secret projects so if someone if their
wife asks them what are you working on
then they can actually say something
instead of zero having emotional support
people
um seems interesting and like how was
like the social structure of secrets in
conjecture
um I think that's all of it is good uh
uh and I'm I'm happy that they are
writing about this I think one of my
final comments uh on like the big
picture of uh this uh info has a policy
is by comparing to government and
Military security clearance systems and
the way these work is that they very
very heavily rely on background checks
that's a very big thing in order to get
access to class five documents and there
are a number of Dimensions you can be
disqualified on for and prevented from
getting a security clearance mental
health issues are problems Financial
issues like if you have unpaid bills or
something like that then you can't get a
clearance if you've ever used drugs or
you have any foreign contexts and all
these kind of things and I expect that
there is a lot of problems
like I expected basically no one in
conjecture would be able to get a
clearance and many of them would fail
for many different reasons and I think
that's not just a thing in conjecture I
think it's in the alignment community in
general
um and I think we are a really bad
Community for dealing with this kind of
sensitive information but that doesn't
absolve us of the obligation to actually
try
that's all I had for today
um I think now instead of ending the uh
the recording because Chris Kamel is
joining us soon I would like to
um
ask people in the reading group for
questions uh what kind of what should we
ask uh Chris Kamel when he joins us in a
moment I've written a couple of things
down but I also like to take your
thoughts does anyone have a question for
Chris
uh yeah I think I brought this up uh in
our last meeting it's just that in my
opinion most people who are interested
in AI safety aren't
um employed in the field I mean I'm
interested in it and I'm not getting
paid for it so like how would someone
like me find out
um about like I I in my mind there's
this pretty big area of information
where you might not want to share with
the general public but everyone who's
working on alignment really should know
about
um
dick and I was wondering how you
disseminate that information like that
um because I think it's really important
that we're all on the same page and
don't have uh duplicated efforts in
terms of alignment research
yeah I think that's a good uh good
question uh I'll write it down
um and and I think we will ask Chris
about how to deal with this kind of info
Hazard like uh
because it is quite unclear people who
are
like uh independent uh alignment
researchers how they actually fit into
this
like one way would be to have like a
community uh info Hazard coordinator or
something like that who like so someone
whose job is like if someone in
alignment have an idea that could be
potentially info hazardous then they
tell that info Hazard coordinator who's
like not employed at conjecture but also
like uh like community-wide whether that
is a position that could make sense
yeah
um I'm not clear just from this paper
and from the
couple of minutes I spent looking at the
conjectures website just what their
product is
is that a sensible question to put it
like that yeah um I think uh there
probably is two things
um
alignment research and some
miscellaneous AI things that they
stumble upon like while they are working
with these then they had some kind of
better way of creating I mean I think it
was like text-to-speech or something
like that
um and then they commercialized a model
that was doing text to speech
or something like that
um so I don't think there is any
obvious
um
yeah I remember
um hearing some of Conor's interviews
from a couple weeks ago and he mentioned
that um they were trying to build some
kind of AI system that used large
language models as a component but not
as the end product
um so I think there's still one of their
end goals is still like just AGI in
general but it's supposed to be safer
than large language models or you know
other systems uh like that
yeah I think that sounds like a
reasonable way to incorporate
okay
um so I think Chris is joining us can
you just excuse me for just a moment
I'm back and
um
hello Chris
hello hello great to have you we are
currently uh recording
um thank you for joining us
um for these questions
um
so we do have a a couple of questions
the first question that I would like to
post is uh the conjectures info has a
policy requires a or talks about a
quarterly review
and you can see here um and my obvious
question is have you done these reviews
and have you learned anything in these
that you can share like how well is the
uh info Hazard policy actually uh is
does it work for you
yeah thanks for the question and just
for this opportunity to talk about the
into hazard policy in general
um so we have done these reviews and the
big thing for us has been kind of
implementing the security protocols on
top of the
um kind of more informal verbal policies
so when we first put this into place we
didn't have a security team and the best
we could do on kind of data separation
was uh in Google workspace like
different shared drives and setting
permissions levels and different slack
channels and setting permission levels
and different GitHub repos and setting
permission levels there and we've now
done a lot more to kind of segment
access to model weights across different
parts of conjecture to split out kind of
product team from engineering team and
who can see what uh and yeah built a lot
more of the kind of technical back end
of the policy
another thing that we've kind of talked
about quarterly is just is it working
um there are projects that I do not know
about at conjecture and that has been a
pretty strong litmus test for us
throughout uh that said there have been
opportunities where people have shared
things internally that have kind of gone
past the secret categorization and
um we've noticed the hopes there and
then there's also been times that things
have been private which is categorized
specifically within conjecture where
someone has spoken to someone who's kind
of not in the private group about it and
part of what was not super well
addressed in the info Hazard policy was
how to deal with those edge cases or
sorry not edge cases just like slip UPS
um
I think the original document said
things like it would be reviewed and
it's a fireable offense if you can do if
you do this kind of thing
um we have not fired anyone from this
it's tended to be that these accidents
happen in really innocuous ways rather
than people kind of explicitly spilling
secrets and I think yeah the policy was
just a little prohibitive and strict as
we first wrote it so we've gotten kind
of lacks on that
um other things that we've changed about
it uh we haven't changed this one yet
but we're currently discussing whether a
fourth level is needed
um there's
some differentiation between the secret
category and the private category that
continues to feel a little restrictive
for how we want to set up team access
there are uh some separate there's
there's some divisions within that about
how we want things to be shared such
that it might make sense to have like uh
secret private semi-private in public so
we're talking about a fourth level but
overall it's worked pretty well I think
conjecture has been tight-lipped about
most of the things that we're doing that
our capability is advancing and that's
the point the input Hazard policy is to
protect your ability to work on
capabilities without letting those
Secrets spill
great
[Music]
lo mein you had a question
uh yeah uh so
the question was that given
how important it is for the alignment
community in general to not have
duplicated effort given the low number
of people working on alignment in
general and also given that a lot of
people working alignment aren't
affiliated with known groups a lot of
people are doing it independently or
part-time
um how do you plan on like disseminating
information to Independent researchers
um
who aren't affiliated with a known group
it's a good question and how yeah and
how important is it in your uh to for
for us to do that in your opinion yeah
um yeah good question and I know there's
been some debate about this unless wrong
recently particularly around
interpretability research
um
so we are on the side of caution and
think that it's very possible for
conjecture to be met negative if
capabilities that we're working on leak
um and so we definitely err on the side
of sharing a lot less at the expense of
not having a super collaborative
discussion about the projects that we're
working on
um we've had slightly different stances
on this in the past like there was one
point where we're trying to do research
Sprints where we were publishing shorter
less comprehensive uh looks into what we
were
investigating
um these are all things that we felt
were not kind of capabilities pushing
but at this point we're pretty much
publishing none of our research not
because everything we're working on is
info hazardous but just because the
trade-off right now between doing things
internally and Publishing externally
um has us you know undervaluating the
the sharing in communities
what I said about conjecture being
negative is roughly we don't think any
of the things that we've done on an
alignment front have significantly moved
the needle in such a way that if we want
to do one tiny thing that sped up
timelines right now
um we'd be net positive so we think it's
very easy for us still at this stage to
to move into the negative side and want
to kind of continue to be cautious and
protective there
is a bigger question here which is that
conjectures research has now
Consolidated around one particular
agenda cognitive emulation and we've
written very little on this publicly I
think it's probably in our interest to
start writing more because the vision is
getting a little bit more crystallized
and it's likely worth it to kind of put
that out in a public way that people
start poking at
so if I can uh expand a bit on La
Maine's uh question let's say that an
independent AI researcher uh let's say
lamine concretely comes up with some
insight related to to this agenda uh
cognitive immolation
um and then he wants to he says this
could be info hazardous
um who should we reach out to is there
someone in conjecture that he should
reach out to oh I see
um yeah I misunderstood that part of the
question thanks to clarifying uh yeah so
this actually has come up in the past
um I've been at an eag where someone
shared with me hey I think I have
something info hazardous
um is it potentially worth someone at
conjecture's time to talk about this
um I think it really depends uh
we don't have a ton of bandwidth
internally they have conversations with
everyone who has a research idea and we
get a lot of people sending messages to
hello at conjecture which is kind of our
generic inbox
suggesting things that they think might
be useful for us
uh it's probably better to try to get
the ear of someone who's kind of trusted
or respected in the alignment Community
who's close to you and kind of get that
view first and then if it seems like
more relevant to the stuff that
conjecture is working on
um then you know bring it over to
discuss with us
um if someone kind of needs a generic
opinion about is this thing info
hazardous or not it seems to me that
it's better for them to put their effort
or to put their time into someone that
they just generally trust really well
um ideally someone who's seen as kind of
an AI safety expert then to reach out to
someone particular at conjecture within
conjecture though the way that info
hazards are shared is uh
as described in the document Conor is
the only person that kind of knows all
info hazards Sid has a view over some of
the technical kind of implementations of
info hazards and then team leads are all
aware of kind of what's on the project
so the idea is kind of share it with the
person closest to you and boil it up
um and yeah if anyone you know knows
people at conjecture it's it's probably
a pretty good crew of people to to
bounce ideas off of but it is hard for
us just to answer to any independent
researcher on them
great
um thank you um Chris you had a question
I don't know uh I tried to answer it
before do you want to go with your
questions or oh
yes well I I just want to um
characterize a sort of
you
conjecture's product as it were
essentially what you are operating um
I gotta I mean some of it is in the
lines of pure research some of it it's
trying to be commercialized
can you just expand on that yeah
um so
in terms of like valuable IP internally
um it's definitely the language models
that we're training and uh we have a
strong team of Engineers who kind of
came from a Luther Ai and built some of
the at the time largest open source
models and we've continued those
training runs internally and so from an
IP perspective it's you know internal
language models and then anything that
you can do with them is kind of the the
direction of what we're thinking about
for product
um there's two paths that conjecture is
currently looking at one is kind of B2B
Enterprise solutions that fall out of
our research agenda or like along the
same build path there an example of this
might be something like uh we need
Strong Quick fine-tuning internally
where maybe we only have a limited data
set and we need to kind of expand that
into a data set of the size that you can
use for fine tuning we want this
pipeline internally as a tool that is
useful to build towards cognitive
emulation it's also something that we
can productize and sell to Enterprise
Partners who maybe have limited data
sets but also want proprietary models to
kind of
work within their system maintain
privacy boundaries between their data
and an external provider
and yeah if it's some sort of custom
Niche use case so that's that's one of
the B2B directions that we're thinking
about
on the b2c front
um the first thing that we tried was a
product called verbalize which was a
speech-to-text model
um it was pretty good as like an API and
it's now served as an API but we tried
to make it into a SAS product and it
just wasn't that great it was a little
buggy we were too slow with it um there
are a whole bunch of other competitors
that came into the space
and so we've mostly scrapped that
um one of the things that we're trying
with right now because eventually we
would like kind of General multimodal
capabilities within conjecture is we're
training a text to speech model and the
idea is then to kind of build a pipeline
from speech to text to language models
to Tech to speech such that you can kind
of speak directly to an llm and have it
speak back to you
this could start out as like a toy
gimmicky thing but eventually it could
be a way of interfacing with language
models that's more interesting to users
right
um
is it
possible it's a possible to speak in
terms of whether you sort of wreak even
currently at the moment or are you
depending well we're hugely yeah yeah
hugely lost making
um we have have very little Revenue uh
still very much in the r d stage
part of the reason to invest in a
generative AI company that is syncing
you know millions of dollars into
training runs but not making anything of
a profit is either because you believe
there is going to be some near-term
Commercial Success from these things or
because you're investing from the
perspective of like AI in the future
will have a lot more control and
accessible accessibility and kind of be
built into larger parts of the economy
we're hedging between both of these
strategies we think within the next you
know three to 12 months will easily be
able to take some of the larger models
we have productize them do some of these
Enterprise Solutions on top of them that
would actually be really meaningful for
businesses
um but there's also kind of a longer
term bet which is
the research agenda that conjecture is
pursuing this cognitive emulation work
is about building safe and Powerful
systems and so the idea is if you can
figure that out well and these systems
are steerable and controllable that
unlocks a ton of value so riskier much
less likely to work much longer term but
really kind of where our main efforts
are focused
it might be worth it just to say a short
bit on cognitive emulation for those who
don't have much context
um
conjecture is Mainline as one of the
more pessimistic
orgs out there on the AI safety problem
is that it is absurd given the level of
today's AI safety to build super
intelligence and we consider this kind
of the mainline path of uh a lot of the
kind of major arcs out there it's it's
not to stop at human level intelligence
it's really to truly build things that
are transformative
um
an example is like anthropics leak pitch
deck said something like uh building
models orders of magnitude larger than
open AI that we expect can automate
significant parts of the economy by 2025
or 2026 and create such a moat that no
one will be able to catch up
um that is not the business that
conjecture is in we're not just building
larger and larger models and hoping to
deploy them for huge economic gain
we are instead trying to build an AI
architecture where language models slot
into cognitive kind of steps that the
the overall architecture has taken such
that when you inspect what it does any
time that you're actually using a black
box it's used in a very limited way
where you can say okay I don't
understand this step but I understand
kind of the overall explainable
reasoning process that the system is
pursuing in general so I might not know
what's going on in the black box but I
do know what the system overall was
doing and so it's kind of a mix of
traditional software some things like
Lang chain Auto GPT and some more kind
of like factored cognition stuff that is
the research direction that we're
pursuing in-house in the language models
are part of this system rather than
being kind of the primary engine that
controls the majority of cognition
that's great thank you
uh I have another question oh sorry um
yeah I have another question unless
someone else wants to go first
right so
um right now I'm feeling a lot of fomo
because like I kind of missed out on the
whole crypto train uh a decade ago and
yeah I mean I kind of caught it but not
as much as I wanted to so uh right now
I'm feeling the same in terms of AI like
I really want to put all of my money
into like
um you know Microsoft and Google but I
also don't want to uh die so is there a
way for me to
um for you know the average person to
invest in groups like conjecture that
have a strong AI safety Focus
um
I'll start by saying I'm the wrong
person to give investment advice and I
feel like this question is maybe best
dealt with not from a money perspective
and more just from a like how does one
get involved in support on AI safety and
maybe one of the ways that people want
to do that is financially but another
way could be you know volunteering time
and another could be you know going
through the upskilling process to try to
become technical uh and you know
proficient in some of the more specific
domains like interpretability
um
I thought about this a lot the question
of you know what can anyone do to
participate because I think there's a
kind of unfortunately very small amount
of
shovel ready work in in AI safety a lot
of the barrier to participation is quite
technical
um so I would say kind of two main
things one is educate yourself there's
tons of reading that can be done online
the conversation is moving very quickly
and simply knowing a lot is I think a
big benefit to
how people can find Opportunities when
they come up maybe there's a job that
opens up that's meaningful to you and
now you know enough to participate in it
maybe there's something that is a
totally unrelated job you're working in
finance or something like that and an AI
conversation comes up where you can
inform people about safety risks versus
not I think a lot of the the change that
will happen around AI safety is from uh
Everyday People realizing that this will
be super you know risky and and making
that clear I think we're starting to see
way more headlines changing around AI
safety and I think you kind of need the
public opinion interchange for expert
policy makers to change their mind to so
more involvement in the conversation is
good
the second thing I would say is like
mental health above finances I I am long
chaos like I have no idea what's coming
but I think it's going to be very
strange one way or another either the
future is going to be really great but
transformed very quickly or it could be
really bad but I think volatility would
be high one way or another
investing in this direction seems risky
and speculative to me on pretty much
every direction I think you can both say
AI is an extreme bubble right now that
could burst or it's you know the thing
that'll transform the future who knows
it you know it could die in the same way
crypto dies
um there's also people who are like I'm
going to put everything in the video
right now also seems like a really risky
thing
um so I would give more advice on get
educated
invest in well-being and kind of
long-term personal stability and
probably stick with a normal investing
portfolio and diversified stuff but
don't listen to me on investment by
financially
thanks for the question
great
um does anyone else have any questions
uh I I think uh in particular uh you you
said you would be uh answering questions
for like 20 minutes half an hour and I
think we are way past that point
um so uh like if you have to leave then
that is perfectly fine not in a rush um
it's 8 49 here now so maybe 10 more
minutes happy to take anything else that
people are curious about
uh yeah
um other than uh I have a question about
other info hazards
um because this we're only talking about
what uh Elias utkovski calls X4 hazards
right and then maybe other like more
very precisely someone may come up with
like a um a really good way to prompt a
language model to make it make really
good bio weapons or someone may more
exotically come up with like a new like
Rocco's basilisk but actually working
um uh and in that case obviously selling
uh uh Conor about this new version of
Rocco's basilisk that actually works is
like really the wrong move right yeah
probably above any of the rules in the
info Hazard policies use good judgment
um so you know something like the
Basilisk would be an example of if it's
obviously and precisely the wrong move
don't do it
um I think the the spirit of the law is
extremely important with info hazards
given that there's so many edge cases
and things that are kind of hard to put
one's finger on
yeah with something like bio weapons
which may be kind of orthogonal rather
than kind of advancing AI capabilities
themselves it's related to another
Hazard and we would definitely consider
that info hazardous I think we generally
put that term around anything that we
think could cause danger and harm to the
world or speed up capabilities in some
kind of way so yeah by risks would
definitely fall into that category
um
yeah oh I I did have a question here
about a functional uh decision Theory
with one of the considerations uh in the
signing the info Hazard policy is
written as just the basically the word
functional decision Theory and like
don't screw it up for everyone
um could you walk us through the
functional decision Theory uh uh
uh like the functional decision Theory
considerations in designing the uh
uh the policy yeah I can try um this is
not really my area of expertise I mostly
understand functional decision Theory as
if I'm going to act from a policy and I
can assume that everyone else is acting
from the same policy uh
how would that kind of change my
decisions here and so as it relates to
info hazards I think a lot of the time
there is a kind of
um
potentially an itch to be the exception
like this is a situation in which I feel
it's safe to tell the person because I
trust them a lot and you know this is
you know the exception to the to the
rule here uh the functionalization
theory is just like no because if you
act by that policy then you assume that
everyone who thinks that they end up in
kind of an edge case scenario will speak
as well
um and so it tends to lean towards
maximum conservative Behavior abiding by
the rules of the policy and ideally you
know ensuring a
game theoretic happy solution of as few
Secrets being shared as possible and
ending up in a world that is the safest
because of that so that's my my rough
hand waving around the subject I imagine
there's much more technical ways to get
into this and describe it
okay uh great I think uh all my
questions have been answered so I am
wondering uh does anyone else have uh
other questions
let me just I would have one question
for the group which is uh just curiosity
what they think of the policy any
critiques any suggestions any kind of
discussion points that came up in your
talk that we're meaningful
um so uh like uh one of them uh that I
can point out here is um like um like
the the rules document seems like it's
not a legal document but it's almost uh
it's something that if there is some
kind of some person who who violates
this then it will be litigated as if it
was illegal
um uh document and there are a number of
like formulations for instance about
mistakes
um uh Connor say uh the document says
that nobody will be fired for raising a
concern in good faith and I think what
he's actually meaning is admitting a uh
no one will be fired for admitting a
non-serious uh mistake in in disclosure
that's at least how I read the document
but it's very much not written as a
legal legal document has a lawyer
actually read this document
we have a NDA built into all of our
contracts that goes through a very kind
of typical
um information protection scheme uh we
threw the word info hazard in there as a
kind of bid to the invisor policy that
we wrote but the policy itself is not
written as a legally enforceable thing
it's written as kind of a socially
enforceable thing I think the idea even
for
um like private uh info hazards ones
that are maybe shared between two
different orgs is if you share this we
tell everyone you shared it and your
reputation is hurt as a participating
organ in this ecosystem
okay so that was one of my comments let
me just see if I've had some other uh
things yeah can I just ask you sorry I
know just um
on less wrong and also on alignment
Forum uh where the document your
document is posted in those two places
and there are not many comments on them
in the year since it's been there so how
do you feel about the response in
general why it disappointed that um
I think it's important for AI safety
labs to have info Hazard policies and my
sense is that people are a lot more
liberal in their Communications at other
firms I think this is bad
I'm not that surprised that there's not
been a lot of comments I think
conjecture is a pretty small org pretty
new doesn't have you know the most
Sterling reputation and so for us to
like do something and expect that a
whole bunch of people follow would be
naive at this stage I would love for a
future where conjecture is
um
in a position to set the precedent on
what good security and safety policies
look like and I think at that point we
can you know re-raise the improviser
policies this is you know a standard
that we think other labs should be able
to meet
we definitely had more private
engagement you know when we wrote it a
lot of people commented on it
um
to my understanding there was one other
small lab that ended up introducing
something quite similar uh with larger
Labs I think it's very hard to
set precedent there but I do know that
it was at least discussed within some of
these larger Labs just from kind of
personal anecil to people sharing that
um so yeah small slash but kind of
within anticipated bounce
so the the thing that
um I think it would obviously be
compared to would be uh like normal
business confidentiality
um like
um the um
um like anthropic doesn't have an info
Hazard policy but they I am sure when
you join anthropy you also sign ndas
let's say probably very close to the
same thing
um and uh like my question is how uh
uh
big uh like there's two things that need
to be compared like
um when you consider whether to have an
explicit info Hazard policy or just say
we have standard business
confidentiality like it draws attention
to the fact that you may have important
Secrets
um but it allows you to make a much more
finely tuned
um uh
do you agree with this trade-off and did
you consider just doing normal business
confidentiality to not draw attention to
the fact that you may have secrets
I don't think we considered that
um
writing the Imposter policy was
important to the founders from the start
uh if you're
in the business of training large
language models and building powerful AI
systems
I don't think you're within normal
bounds of
normal business confidentiality and
assuming that you might have nothing to
hide the number of security attacks on
uh AI companies is increasing and I
heard a stat from an expert that was
something like
I'm recounting what I heard I did not
check this number it seemed really high
to me but it was something like one in
three attacks on AI companies or like or
like targeted attacks are trying to get
at model weights that seemed absurdly
High to me but I think the general
impression was people are aware of the
value of these weights and
um
yeah like number of kind of cyber
attacks are increasing so from a like
technical security perspective uh
extremely important and people know that
the kind of treasure is there and then
from a verbal safety perspective
um I think one of the big things with
the Imposter policy is done besides
making kind of explicit boundaries
internally with silos is bake in this
kind of conversation to conjecture and
give people a vocabulary in which to
talk about it and it really
straightforward way and I think it's
been beneficial
okay
um I don't actually have more questions
um there's one by Urban Pace he posted
on restaurant and argued that uh there's
a risk with this kind of
um policy that
um like it requires that Conor has uh a
lot of time right uh and he that he's
responsive if you have something that
perhaps needs to be a new secret
um then the requirement to discuss it
with Conor if you want to
um
check whether this is an uh an overlap
with another existing secret or
something like that
um that like uh how to do you have any
sense of how well that works is this
actually a problem in real in practice
right now we're small enough that it's
not a problem
um I could imagine in the future maybe I
think we'd have to be pretty big for
this not to be a thing I think the idea
that you can discuss with your lead
um is discussing with the lead is kind
of the first step for things that are uh
in the maybe category it's formally
starting another work stream that
requires Conor's approval
um and in that sense you know Conor's
very involved in sending the research
direction for the company and so I don't
think that would be uh
any concern about time Sensitivity I
think the the larger concern I've heard
raised around the info Hazard policy is
just that it puts a lot of control in
Connor's hands to have total visibility
and to run the company and it's like yep
that's because we trust Conor internally
and think he's doing a great job and so
I think the
a lot of this is very intentional power
concentration and someone who is a
trusted figure and uh for
um
for a lot of reasons while we're small
especially it makes sense for kind of
one person to have full steering
Direction and control
um as we get bigger maybe more checks
and balances makes sense but at this
point it doesn't really feel like a huge
cost we pay
okay so how many people work with
conjecture uh right now it's like 22.
yeah
and um I'm still not really
understanding what the conjecture plan
is
I mean it makes sense in a vacuum but
how does it function in a world where
open Ai and menthropic are very clear
that they're scaling up very quickly
yeah so
in my mind it's like it would be awesome
to live in a world where one of
anthropic open air eye or Deep Mind was
like hey
we're not going to build super
intelligence This Is Us kind of laying
down the swords and
pushing for governance that sets a hard
cap on training runs that talks about
extreme compute governance and need to
regulate this powerful technology
that says this is the limit of power
that we think is appropriate to build to
given the state of alignment research is
so far behind
um
none of the major players seem to be
doing that their governance policies are
largely carving out space for them to be
the ones who are going to continue to
build super intelligence their technical
plans are trained larger and larger
models and build super intelligence and
their safety plans are predicated on
super intelligence like scalable
oversight is a plan where you have one
super intelligence overseeing another
Super intelligence because the
capabilities of these systems surpass
the human level
conjecture is an alternative to that so
the idea is no super intelligence
systems at an explainability that makes
sense to humans and build them from the
bottom up such that they can be
understood by humans push for policy
regulations that prevent people from
building super intelligence and
um yeah mandate the need to kind of push
alignment further before you build more
capabilities
and so in a lot of ways conjecture is
quite deontological in this sense it's
like maybe we won't win but someone out
there should be doing it and our plan is
to try to do it well to scale up to be a
larger voice to be able to hire more
people and work on this agenda
um and to push forward with
an ambitious plan despite the odds
so
um uh if laume wants to be hired by a
conjecture I have no clue if you
actually want that but let's say what
kind of research people should he be
writing and sending to you like on
cognitive immolation is there like a
like a subfield of that that would be
interesting to you or yeah I would push
people more towards hacky engineering
tinkering right now than research
directions
um stuff that's kind of in line with
what we're uh building are like hey if
you've like tried to turn gpd3
internally like just by yourself if
you've looked up papers on model
training if you're familiar with
distributed training if you've like just
gone and hacked open source those are
the type of people we want we want
people who have strong technical skills
that are willing to kind of bootstrap a
project from the bottom up by themselves
um and are really kind of curious about
where AI is going
cognitive emulation is a builder's
research agenda more than an academics
research agenda at this point there are
some hard technical questions like how
do you factor cognition in a way that
scales you know this is a problem that a
lot of people have run into and one that
we are going to run into on this on this
path as well
um but right now we're still in the like
there's just a ton of things that we
need to build and strap together to get
systems where we're even running into
these kind of questions as the limiting
factors
um
so yeah people who have familiarity
playing with language models who have
worked on Lang chain who have worked on
auto GPT who are
generally aware of kind of
state-of-the-art
um research in uh advances like yeah
different training regimes and things
like that those are all interesting
things for us from a research
perspective I would say that we are
doing a lot less interpretability than
we used to be doing we've moved more
towards explainability than
interpretability uh meaning
we are not super optimistic about being
able to kind of penetrate the black box
and instead we want to limit the uses of
black box in a system that is kind of
overall understandable for for how its
logic flows um
so yeah
uh I have another question so none of
the large language models
um thus far that aren't open source uh
let me rephrase that all of The Cutting
Edge large language models are currently
not available in uh either mainland
China or
um
uh Hong Kong Macau places like that is
that due to U.S law or due to Chinese
law or just an internal thing
um
I think there's a lot of complicated
factors right now I think there's also
kind of a big Global conversation around
competition with language models and
where the tech is being built and who
has access to it and
um yeah this is a complicated subject I
think one of the important points that
is uh quite dangerous right now is like
over emphasizing this race or this
competition you know one of the
justifications for building powerful
language models in the west is oh well
what if China gets there first you know
what if one of our competitors builds
the transformative AI system before us
um I think thinking like this is a
rationalization for the building of
super intelligence that's already been
on the agenda for
kind of the major research labs in the
west and at this point it's now a like
sexy policy narrative that can convince
government officials that we should also
continue to build powerful AI I think
this is quite foolish I think there's
many reasons to believe that the race is
not quite as
um tight as people make it out to be I
think there's also many other ways that
we can go about approaching this race by
you know limiting export controls and
things like that
um
and
generally over emphasizing the we need
to win narrative seems super dangerous
to me so that's kind of what I'll I'll
say on the China us China West kind of
llm stuff
what I'm really trying to ask is
um how is there a way for someone who's
interested in AI safety to like
get access to some of the invite-only
models like
um or is it client Cloud yeah
um yeah I think for people who are
working on Research uh it's
um possible to get kind of individual
permissions I don't know how this works
from a uh Regional perspective
um
but the yeah I would I would try
reaching out to some research groups
that have access and trying to see if
there's projects that are meaningful to
work on with them
um there's a number of alignment
upscaling programs for example that are
accessible you know people start with
something like AGI safety fundamentals
and then they consider moving on to
um something like mlab for more
technical upskilling uh Siri Maps where
you get paired with a mentor who's
working in alignment these programs have
access to models and so it might be a
little bit more difficult as an
independent researcher but any type of
training program that connects you into
this space would also give you access
all right thanks
okay does anyone have any final
questions for Chris
then I would like to once again say
thank you very much Chris for joining us
and I've really enjoyed your answers and
good luck at conjecture yeah thanks
Aaron um and thanks for discussing the
subject and inviting me right uh but
before you leave I would just like to
say that
um from the Chinese perspective the AI
race is very very real I mean we're
trying really really hard to not get
left behind uh in the AI race so um it's
not like
it
I I think that the
dismissing the um China us
um AI race is very much a mistake
um yeah thanks for pointing out that
Nuance I think the the point I want to
emphasize is something like
one reason to talk about the race is to
talk about the fact that there are
dangerous capabilities that are kind of
expanding all over the globe and we need
to be careful about building and
um concerned about how these
Technologies can be misused or if they
get deployed and there's misalignment
risks that's good I think pointing at
the risks serious pointing out that
there are competitive actors uh is
important the thing that I'm concerned
about is that one of the ways that the
race narrative gets spun is as a
justification to continue building even
more dangerous Technologies and I think
uh if you don't believe that the race is
neck and neck then it's easier to think
about something like a six-month pause
on AI capabilities in the west as a
um useful and not competitively lethal
strategy for us to take to try to end up
in a safer World
um
yeah a lot of this gets into kind of
technical details about access to
compute and who's building what and
Technical competency and if secrets are
you know developed internally or you
know learned from others uh and yeah I
guess without going down that rabbit
hole I think it's important to kind of
consider both the reality and then how
the narrative gets used socially to
justify different things after them
okay awesome well thank you all again uh
be well have a good night and uh
and good luck
thanks a lot great
okay
so um that was great uh I'm gonna stop
the recording now
and


Existential Risk, AI Safety & Interdisciplinary Research - Anders Sandberg

traditionally academia has been done in
very separate disciplines the deeper for
the philosophy department did not talk
to the economist and of course nobody
would know the mathematicians because we
couldn't understand them but mrs. of
course a problem because in order to
solve the really big problem we actually
need to be extremely disciplined we
actually need to figure out how to solve
stuff that doesn't fit neatly into the
way we have the previous who thinking so
one of the great things about the Oxford
is that we actually do get together we
end up with the mathematicians advising
archaeologists people doing climate
science telling philosophers stuff but
is very useful for the to them later
reformulate the tilt of economists so
the real challenge is of course to
figure out how to think carefully when
you receive is disciplines
interdisciplinary work is quite hard
it's not just about sitting over dinner
and talking to each other it's about
understanding what people are saying and
doing and thinking in areas which you
can't even understand or you can't even
answer why would anybody say in the same
person want to study that so one
particular problem of interest is
existential risk because this is
something that fretless humanity from
all directions in a sense it's both a
question of philosophy but it's also a
question of economics probability and of
course all the natural sciences that
might be related to the problem so we
need to get the economies to talk two
philosophies about discounting rates and
then we can talk to the people trying to
deflect asteroids about how discounted
rates actually affect their ability to
convince the people in politics to do
something about it the people in
politics can actually tells useful
things about what's feasible to
implement the solutions and not and so
on
existential risk is very challenging
because it's always beyond what their
previous with you thinking so that's why
really neways thinking one very
interesting topic is integrating humans
and machines in the past we have created
machines that are directly controlled by
humans our action is directly turned
into an actual Maggie gradually being
given more autonomy we're given motors
we give them a control system we give
them sensor so we can behave
autonomously but the real challenge is
of course to make sure that machines do
what we want and in order to achieve
that either we need to connect our
nervous system to the machine or we need
a machine that is smart enough to figure
out human desires in a machine that is
smart enough to understand the water dia
gram wants to do has to understand you
mustn't a cognitive level it needs to be
fairly intelligent if it's not in talent
enough for if it misses them too much
it's going to be like we're all the
Microsoft paperclip terribly annoying
and people will not want to buy it but
if you have a machine that is so good
that it can figure out what you want
probably just before you actually want
it not only will you want to buy it but
it's also going to be quite a risky
because now we have a very powerful
machine but might actually have a
misunderstand what you want on a longer
skin there are a short-term decides that
I might want an ice cream right now
might be against my long-term desire to
keep my stomach here in a good shape so
the real challenge might be that we need
yet again to add a long-term desires
higher order decides to machines and now
we need to be about the smart as weel or
smarter so in the brain the hemispheres
are connected by the corpus callosum
effect bundle
nerve fibers several million fibers fit
and people often talk about right brain
and left brain people which is an
oversimplification what's actually going
on is of course by the division of labor
between different parts of what if we
could connect a more strongly well it
turns out that in many cases is not a
good idea because we just interfere with
each other it turns out the left side
and right side actually shouldn't know
what we always do however in many
problems we might want to extra brain
power we might want neural interfaces
that allow us to link up fur in their
systems together or it might turn out
that we want to store information
external we might want to bridge from
our bread other brains anything that
allows communication to become better
between heated or between people and
machines is going to be a big deal that
it has the potential to transform the
world


262. Counterarguments to the basic AI Xrisk case 2

hello and welcome to session
263 in the AI safety.com reading group
tonight we'll be discussing the second
part of counter arguments to the basic
AI Exodus Case by catcher Grace
so catcher Grace is still the head
person in AI impacts and today we're
going to focus on counter argument C
ouncil argument C is counter arguments
to the claim that superhuman AI would be
sufficiently superior to humans to
overpower Humanity
before we go into uh into this subject I
would like to add a few comments to a
particular didactic tool that catches
using which is uh among each Gap in the
arguments she presents a section called
what it might look like if this Gap
matters and I've criticized a number of
these and I'll also continue to
criticize a few of them in in this part
but I would like to talk a bit more
about why I think this is a rather bad
choice for getting some kind of
understanding and intuition and the key
problem as I see it is that when you're
setting up these kind of scenarios you
are very it's very easy to add a lot of
things that don't follow from the
arguments themselves
and when someone
reads these arguments and tries to poke
holes in them then
uh adding these kind of uh complications
that don't actually relate to the
arguments in the previous section
um feels uh like a non-secure in in a
sense it doesn't
um it doesn't follow from the argument's
uh below and I guess if you are trying
to make like a fully fleshed out
scenario or something like that you have
to add some some things that are like
that don't necessarily follow in a
strictly logical way
um but I think that they these
complications end up muddying the waters
so much that I don't think it's a good
idea to have these uh these sections
right onto the arguments the first human
success isn't from Individual
intelligence
starts by restating the arguments in
um in her own way again as we saw last
time and in a somewhat uh uh
so optimal way the first is that you
write that the argument claims that
passing human level intelligence is the
relevant bar and this is the relevant
bar for doing things like recursive
self-improvement or doing thing doing a
number of things but not explicitly for
taking over for taking over uh that is
at the super intelligent level not at
the slightly above human level like
um again
um the best uh
description of the actual Arguments for
AIX risk comes in my opinion from the
book super intelligence by Nick Bostrom
and super intelligence is defined as a
being more intelligent than any human
however clever and so we are obviously
not talking at the
um at our level around the average human
this is a confusion that comes up a
number of places in the following text
and I think uh in general
um
uh it would be nice if categories either
use the definitions that uh AI X risk
Advocates are using or made her own more
clear in particular to what extent is
human success caused by individual
intelligence and cultural transmission I
think that's a really really interesting
subject and a lot could be said about
how these two factors
um map together but catcher does not
actually go into how these two things
work together
um and
um and I think it's a shame and I think
it's a shame that it's left somewhat
um
uh somewhat uh underexplored I think it
may be that there are counter agreements
lurking somewhere around there but catch
it does not um go into a discussion
about how individual intelligence and
and culture uh uh interact
there's a few other quotes here the
argument claims that something more
intelligent than humans will inexorably
triumph over humanity and if any of you
have read any uh bustram's book
superintelligence you know that he would
certainly never ever in a million years
make any claim like that the word
inexorably is not used and there's a
good reason for password not to use this
word he would couch this in way more
disclaimers
and a final the argument claims that
humans tramped overall species only
because of individual intelligence and
not culture
this is on the face of it obviously
irrelevant and obviously uh it almost
certainly irrelevant but very clearly
untrue right
um obviously human culture was a
dramatic part of human success now I
don't think anyone would suggest this
and I'm uh struggling to see how uh like
imagine that uh
yutkowski had written this sentence uh
then obviously a lot of people would
attack him and say actually human
culture is a big deal and and they'd be
right of course and it's very clear that
this is the kind of uh counter arguments
that don't work you are not in general
going to find errors in
um
find this kind of totally solely trivial
errors in Nick pastram's work or in
Illinois
Cristiano any of these people because
there has been so incredibly much
scrutiny on the arguments so if you find
a counter argument that is really really
surface level and really really obvious
then almost certainly either someone has
found them before you or uh it's uh or
you are mistaking and Nick Bostrom
obviously he wouldn't say this
now let's go compare a single human
compared to like Humanity if you look at
someone in the human society that is
substantially more intelligent than the
average we could call that a genius then
they basically live in the same way as
humans do
um
give or take I think I would give and
take a bit more depending on precisely
where you how you define a genius
a number of geniuses are in fact living
in substantially different conditions
than average people
um I think for unrecognized geniuses
this may in fact be true
um but once you are recognized as a
genius most likely you're going to live
in a different way than average people
then there's a true but irrelevant claim
that an AI at the human level without
access to information from humanity is
worthless obviously humans without
access to human culture like uh children
that are lost in the jungle and grow up
Among Wolves or something like that
that's
um that that's well known
um
we have a claimed that some information
cannot be obtained by reading or like
YouTube videos or things like that there
are some technical knowledge
um I think it's true and I think Jessica
would probably agree that this perhaps
makes very little difference
um when there's a claim that the pound
individual has in society is most
related by what role Society gives them
and while that is true I think the
relationship is mostly backwards in the
sense that the the person who is most
who's best at Social manipulation will
be given the greatest role in human
society so it's not that you uh Humanity
gives
um gives roles out randomly is Humanity
tries to give roles based on uh
competencies
we may do a poor job but that is in fact
what we're trying to do then there's
this question this um so I'm just going
to read it a person twice as smart as
any other human would research twice as
much as an average human and that's
basically nothing
so this question this I try to read it
several times and I think on balance
this claim doesn't make sense so the
question is precisely twice as smart as
any other human what are we talking
about here
twice as smart as any human that would
be
um like substantially smarter than the
smartest person ever like maybe an IQ of
180 or something like that
um that would be would that person
research twice as much as an average
human what they obviously would research
way way more that would be a a true
Super Genius or something like that so
they would be able to uh in that case
the claim doesn't make sense it's also
possible to read this claim as just a
person that is twice as smart as the
average human person and the average
person is probably not like accounting
for babies and things like that we are
probably talking about a really really
low level and that makes the claim true
but also very irrelevant to the things
we're talking about
catcher has an analogy with the
Manhattan Project when it com which uh
when it comes to like trying to gain to
create technology to obtain precisely
strategic uh advantage
to claims that people often mistake the
actions of a human with the actions of a
human society
I think people do in fact make this
mistake people in general make a lot of
epistemic Errors like
um and that is why in general you should
not engage with uh with arguments
presented by the average person you
should Instead try to find the best
arguments uh presented uh by your
opponents and they argue against those
instead of yeah yeah there's a lot of
people who have seen the Terminator
movie and they are probably really
confused so don't argue with those argue
with Nick Bostrom or yutkowski or
Cristiano or these kind of people
uh that's the note that uh even the
Manhattan Project was not done in the
afternoon even by the smartest person in
the Manhattan Project that's obviously
true I think in in general uh there are
is a sliding skill on to what extent
projects can just can be completed by
some one person having like a Eureka
moment where they jump out of the
bathtub but
a lot of projects involve a lot of hard
work and I think that is in fact a very
commonly understood uh conclusion that a
lot of big projects just require a lot
of grit and hard work and man hours
uh so here we have a counterfactual if
the Manhattan Project was done by it is
just somewhat smarter than a person then
that wouldn't have changed the project
very much I don't really understand the
counter factual here are we again
talking about entities smarter than
humans in general like super
intelligences or are we talking about
that like everybody in the Manhattan
Project gets plus five IQ points or
something like that
um I think it's unclear what catcher is
pointing at but I would point to one
thing that the Manhattan Project in
practice was uh
they when they were building uh the the
first atomic bombs they realized that
enriching uranium was the key bottleneck
and they tried to make centrifuges work
and they couldn't make Center Shooters
work and then they did some other ways
of uh purifying enriching the uranium
that was a huge amount of work and then
after they uh they finished the
Manhattan Project then they just ah of
course we could have um and done
something slightly different and made a
civil type centrifuge and um that of
course that's basically how we've
enriched uranium ever since and that's
way smarter
um and that would have caught the
enrichment process that was super super
artists down dramatically so I could in
fact have seen easily that if some of
the people who were designing this had
been just slightly smarter sure than uh
the Manhattan Project could have
happened dramatically faster and at a
dramatically lower cost
my this is not really my real objection
to this argument my real objection to
this is that the Manhattan Project is a
very very strange projects almost all
projects are very very different from
Manhattan the Manhattan Project so if
you're trying to talk about projects in
general you should probably not
assume that any random project is going
to be like the Manhattan Project
um because in and in this particular
case we do have a project that is very
very likely to be to be relevant so if
you imagine someone is building tpt5
then probably the project to build TV5
is kind of like the project to build
gbt4 which is kind of like the project
to build gbt3 that is kind of like the
project of building gbt2
Etc this is
um I think a way more grounded way of
research of reasoning rather than trying
to find like
the Manhattan Project is in fact known
specifically because it's an outlier
because it's a very very strange project
the next argument is about humans
sharing power with AI
um
human power is generally something that
we obtain like our power over the
environment is not something we obtain
through our own individual intelligence
but by
um buying power from the rest of society
using money or something like that and
that means in general we should expect
AI label to be sold on some kind of
Market in that order
and I think that is in general a true
assumption except in the specific case
where the AI is trying to take over
either because on his own initiative or
because the it's been built by some evil
actor that wants to take over the world
in that specific case we should not
expect AI labor to be sold but hard
then there are arguments against whether
humans will transfer power directly to
AIS
um and I don't think like that's not a
thing that the the original arguments uh
don't really talk about this I'm aware
that there is this movie Terminator
where humans voluntarily give the AI
Control over all the nuclear weapons
um but I I think that's again arguments
from the movie Terminator and not
arguments from Nick Bostrom
um there's the question of a comparison
between agentic Ai and how much power
that would yet compared to a combination
of humans and non-agentic machines
um
first uh categories correctly notes that
human level capability is a moving
Target and something that could
potentially increase dramatically as we
get things like tool Ai and that is in
fact a complication that Nick Bostrom
has already taken into account in the
book super intelligence in chapter 4 on
the Canadians of an intelligence
explosion
um
explicitly talks about this and say okay
we are anchoring the human capability at
the level that it is when a AGI reaches
the human level or something like that
uh I think actually he just anchors it
at
um like 2016 level and then you just
multiply by a factor or something like
that
um that so so that is how Nick pastram
gets around this but I agree if you
don't
um if you don't uh read Nick Bostrom but
you just try to come up with the
arguments yourself it's easy to uh
to overlook some of these complications
and that is indeed a way you can confuse
yourself
now uh how much Superior our agents
going to be compared to uh humans with
two AI
um well there are certainly a number of
disadvantages and uh catch a list some
of them but I think in this case the
canonical reference would be converts
why tool AIS want to be agent AIS this
lays out in substantial details the
arguments for why AI agents May in fact
be radically superior
um can I just say these arguments may or
may not matter much and obviously I
agree in the sense that uh predicting
the future is hot but I would also say
this is not really a counter argument to
State briefly some of your opponents
arguments and just say this may or not
may not matter much is isn't a counter
argument
catcher in fact comes up with an uh
another argument the tool AIS also have
to deal with the interface to humans
which agents do not I uh just re-skimped
burns text and I couldn't find it so I
assume that this is like a new argument
that catcher is presenting for why AI
agents are going to be even more
Superior to combinations of humans and
non-identic machines
um
and then the new argument that catcher
presents says it matters for some tasks
like rapid interaction with humans but
not for like major on off decisions so
maybe more for social impulation and
less for uh strategizing
um
and again that doesn't really seem like
a counter argument in the sense that if
you uh present a new argument for a case
and then show that the new argument
isn't like fully covering that there are
cases where it doesn't apply that's not
really a counter argument
then there's a matter of trust
um because uh this may matter in some
specific cases and the case that catcher
has in mind is the case where AI has a
slight Performance Edge over humans and
already in in this assumption I think it
is a very very narrow set of assumptions
I think in most worlds and across most
tasks either AI is going to be
dramatically Superior to humans or
dramatically inferior to humans
and in particular the the relevant six
cognitive domains that strategic
cognitive domains that Boston have
identified are very unlikely to be a
place where there's like a slight
Performance Edge
um and of course it would be even more
surprising if this is stable if as AI
progresses then humans and true AI also
progresses at precisely the same level
so we get some kind something that's
Stables over centuries that sounds
really really far-fetched
but if we are in this situation where a
genetic AI has a slight uh Performance
Edge over humans with two layer then
this could be balanced
um by the by us not trusting the AI and
there are two reasons why we would not
trust the AI the first is we don't know
whether well values are and we don't
know how they would behave in any given
in any given case
I think even in this case the um the
person who is deciding whether he wants
to employ an AI or a human needs to
consider the uh the competitive Dynamics
where these two claims don't actually
hold for humans either when you choose
to employ a human you don't actually
know what their values are maybe they
will start a labor union maybe they will
I don't know do many many different
things and how will humans behave in any
given case humans sometimes do really
really strange things
um so this also will not hold for for
humans and it might in fact be that
there's a base rate of humans starting
labor unions and there may be a much
lower base rate of AIS starting labor
unions I could totally see that thing
happening
and finally if we don't know that the AI
is aligned then in expectation it's
costed to use AI
again I'm a bit unsure about uh what
catcher is talking about here uh is he
talking about like this lower level
unaligned or is she talking about
takeover scenarios
um if it's just the lower level where
the AI is actually uh you know I don't
know breaking vases uh trying to clean
up or something like that then in that
case we expect more capability to
rapidly
um fix that problem so uh it won't be
costing expectation for very long in the
other case where the AI is trying to
take over
um well in that case obviously right now
uh people
ought to realize that building AIS is
terribly terribly dangerous and they're
doing it anyway so that is a state of
affairs that is likely to going to
persist and people will in fact
generally be unaware of the problems
with AI and just blissfully use it until
in AI takeover
the next argument is about Headroom like
in a given task how much can it be done
better than what we're doing right now a
classic example is tic-tac-toe with
where optimal game
performance can be specified relatively
easily uh like there is no way to do
better than humans in tic-tac-toe that's
just um uh
and a fully solved game to some extent
and to what extent this is General lies
well there are some tasks that probably
have have only little Headroom if we
know what best performance is if we know
that we humans can do it almost
perfectly if we can bound how well the
task can be done or there are
computational limits or things like that
probably there is little Headroom and
probably there's a lot of headrooms in
these kind of things I won't read them
up
um catcher doesn't put much stack stock
in this counter argument probably for
many tasks there's a lot of Headroom
um but it's not obvious and I think um
we should like we should limit our
analysis to Boston's six cognitive
strategic tasks and in these tasks
intelligence amplification strategizing
um social manipulation uh hacking uh
economic productivity and technological
uh research in these it seems clear that
we're not going to get anything like
um best performance out of humans anyway
anytime soon
and all also in this uh catcher has um a
um what what does the world look like if
this Gap matters and she suggests that
if there is little Headroom above the
human level on these tasks that could
lead to a world with a good offense
defense balance like defense is a lot
easier than offense and that could lead
to stability
um that doesn't seem clear to me at all
I don't think that follows at all
intelligence may not be an overwhelming
advantage
and again catch up
represents the arguments as the argument
assumes that any difference in
intelligence will eventually win out
over any difference in other initial
resources and that's obviously trivially
wrong right uh any difference like who
who would say in this kind of claim like
obviously there is a level of difference
where if you're a tiny bit smarter then
you can't overcome that that initial
difference that seems really really
obviously clear
then she has to claim the difference in
performance between top and average
human performance is large relative to
difference between average and random
and
this seems to me to be in fact an
argument for why intelligence could be
an overwhelming Advantage the way I
reach it so but at this stage catcher is
like three meter levels deep in
counter-counter counter arguments uh
that's not meeting levels but three
levels deep in counter counter are
counter arguments and I think it's
possible that either I misunderstood
what she's written or it seems like
she's not in fact presenting a counter
argument here
I also kind of feel that we're at an
enormously high level of abstraction
here and I don't think we can really say
a lot meaningfully at this level
um so let's talk about GPT 4 or gbt3 or
some of these a actual AI projects that
are right in front of us and have a look
at those and we can like measure their
performance we can say a lot of things a
lot more concretely by by talking about
these
instead has a
digression and a link to a discussion
about random chess like what how well do
you play chess if you play random moves
and that is
um like I try to follow some of the
arguments and the links to that and I
becoming clearly more confused than
enlightened by this I don't think you
can say much about like random chess is
really really strange and you have a
really hard time making the other person
Checkmate in random chess and it's just
really really strange
um
catcher also in this section has an
example
um with like three actors at Bob Charles
and David
um I think something has gone wrong in
the editing to this I
I'm quite sure that it's just not just
that I'm stupid that is the reason why I
couldn't pass it I think some sentences
may have fallen out from that example
if another example
um country argument to why a smaller AI
may not be able to take over is that you
can provide some examples of
intelligence not helping elephants are
smarter than Nets but elephants have not
won over gnats
and the reason we uh
this is not actually a counter argument
to uh to the arguments as presented
because elephants cannot do the six
strategic cognitive tasks that we're
interested in and that is precisely the
reason why all intelligences at a level
where they can't do this are irrelevant
and in a strong sense
um elephants and Nets are
um
are equal in that they don't have
general intelligence
so looking at this world it seems to
catch out intuitively that the big
discrepancies in power in the world are
not about competence
I think I would disagree by adding the
complication that there is also the
competence of your ancestors
so what explains the discrepancies in
power that we see in the world right now
it's of course a huge huge question
um and like my intuition is that 40 is
about individual competence 40 is the
competence of your ancestors and 20 is
luck
um so I think in fact the difference
discrepancies in power that we see right
now are mostly about competence
and there's another example
people with an IQ of 130 they earn six
to eighteen thousand dollars more per
year compared to average IQ people
um
it's uh it's obviously an example of
intelligence literally helping and
literally helping substantially it would
be a lot clearer if that was expressed
in percentage rather than absolute
numbers also because some of these are
very old I looked through some of her
references and I don't think I think the
actual number is substantially larger
than this and I think the key problem is
that they are controlling for a lot of
things like who do we marry to a divorce
do you um
um like what kind of education do you
take and if you're controlling for
education then you're taking a lot of
the difference between IQ 130 and IQ 100
people out there because people who have
an IQ of 130 are much more likely to
finish a um an advanced education
there's another reference yes where
politicians appear only slightly more
intelligent than others
I looked at the um uh the paper it is a
substantially smaller scoped uh paper I
don't think you can use it to for this
kind of sweeping
um uh generalizations
um like there's a link to it it's a
draft the the figures haven't been uh
put in uh the conclusions say that
there's a strong positive selection in
politicians in fact being more
intelligent than than others
um it's based on some kind of army
um uh Army measuring system in Sweden
where they see like are people likely to
going to be
um intelligent and uh strong and good
good infantries good soldiers and also
good leaders and I think it was actually
somewhat interesting to me that to
predict whether you will be a a
successful politician and get elected
it's it's better to be intelligent
rather than uh the Army says you could
make a good officer and I think that's
actually very counterintuitive I would
expect the thing that makes you a good
officer to be the same thing that makes
you a um a good politician much or
substantially more than just being smart
but apparently the paper found that it
is slightly more
um important to be smart in order to get
elected rather than to be a good leader
another example of intelligence not
working is Mensa an organization for
people with genius level IQ and
obviously people with um
Mensa I think it's trivial uh truly of
ways to say that Mensa represents a very
very specific subset of the people with
an IQ over 130 and I think there's a
social Dynamic around Mensa that really
really muddies the water and makes this
some horribly horrible evidence
um like the people are not joining
mindset because it's perceived as
socially uh very very uh bad to join
Mensa not uh because it wouldn't help in
any way
intelligence appears to be more useful
for showing off than for strategic
action uh this might be true but that
doesn't bound really how useful it is
for strategic action it may be useful
for both
um
and we have catcher also mentions that
this may just because there's some
Winner Takes all Dynamics in
um in a lot of appearance based Sports
and competitions and things like that
um
catcher also have some anecdotes of
intelligent people who are don't appear
to be helped by their intelligence
um I like you can counter anecdotes with
more anecdotes but I couldn't actually
be bothered to find examples of
successful intelligent people I think we
can all agree that there are a lot of
examples of intelligent successful
people so just pretend that I gave a
list of anecdotes here to counter
catches anecdotes
it is unclear that many goals
realistically incentivize taking over
the universe
I think this may in fact be in the wrong
section remember we are doing counter
arguments to whether superhuman AI would
be sufficiently superior to humans to
overpower humanity and uh in this case
this is whether you actually want to do
that and that is like a different
question to my mind
catcher uses herself as an example she
doesn't try to take over the universe
and I explicitly think this is uh mostly
for lack of ability uh no uh offense to
catch up but she doesn't have a feasible
path towards taking over the universe if
you imagine she had one like she's given
a magical safe that has a take over the
universe button inside then sure I
expect her to go lock picking
um because to try to get that safe open
because taking a pardon that would allow
catcher to take over the universe and
have everything done everyone follow her
commands or something like that would
probably
um really really useful to her
um and uh taking over the universe as a
substance for for your call that seems
really laughable for almost any human
goal I don't in fact think that this is
lovable I think
um it is only because it appears so hard
to take over the world uh and that
hasn't actually stopped people there are
a lot of people who have in fact tried
to take over the world and to these
people it hasn't been lovable it has
been the uh the most likely path to
success to try to take over the world
so the question is how much easier would
it be for an aeon here at this point I
would like to have like or the a
standard reference would be Possum's uh
source of advantage for digital
intelligence uh but it's been written in
many other places there's a lot of
people who have showed scenarios on this
kind of thing
um
catch it doesn't really show gaps in
this arguments as much as just posting
the question and not making any
reference to the answers and just uh
putting a question mark to that
another empirical question is the
quantity of new cognitive labor
um how much will it be well if we assume
it comes out to just a peak human uh
then that is not enough to take over the
world if we say it's like a million
humans then probably can
um I mean I'd actually show that one
million is sufficient one million uh AIS
versus 8 billion humans that's not
trivial even if you have good
coordination it may be possible but it's
not trivial
but we assume that computers get faster
faster and algorithms get um
better and better and we get more and
more larger projects with more
what's it called this process where you
get the learning curve you get better
and better AI cognitive performance over
time and I think that is in fact the key
part of the
the answer to this question that already
now we can ask how much does it cost to
to run this chat gbt3 and have it com
composed like a college essay compared
to having a human composters college
essay and of course we are already
strongly past the point where if you
want to make a million
um college essays cheaply you would
obviously use tpt3 and you would
obviously not have humans do that task
um
and as a claim that if we assume that we
have ai's gradually entering Society so
we have a period of some AI presence but
at a low level then when the AI takeover
happens if it happens then uh the
catcher claims that because we have had
a period of time with AI being gradually
introduced then the eventual takeover
The Coop would be more slow slow moving
I don't think that follows at all I
think it's very possible to have a very
very gradual introduction of AI combined
with a coupe that takes like
milliseconds
the speed of intelligence growth that is
claimed to be ambiguous I don't think
ambiguous is the right word here I think
the word is uncertain I don't think
there is a lot of ambiguity in the
concept I think there is a lot of
uncertainty about how fast a takeoff
would happen
and the fact that it's uncertain how
fast it would happen is not something
that
um the arguments would disagree with
like Bostrom say it's more likely than
not that the Takeover is fast yukowski
has explicitly said that he believes
that everything uh that he that only
really predicts what happened above the
level of atmospheric turbulence
um and I I think I also personally agree
we won't know this in advance
so how could an intelligence explosion
happen one of the key ways is recursive
self-improvement catcher wrote an
article in 2018 with counter arguments
and I made a presentation just like this
with counter counter arguments but I
would say that 2018 is a long time ago
um and like when catcher expressed
uncertainty about in 2018 about how
would technology develop I think it was
prudent and was correct of her to
express uncertainty but it's the 2018
anymore like we literally have
Transformers in our hand right now and
we can see that Transformers can do
things that a lot of people in 2018 were
skeptical would be possible at this
level so I think uh like we have learned
something and I'm sure catcher has also
updated substantially since 2018.
the second reason to suspect an
intelligence explosion is by analogy
with the primate to human evolution that
suggests any movement past human level
will take us to unimaginably uh
superhuman level
again like unimaginably superhuman if
you just go a bit above the human level
it seems like obviously as a silly straw
man right no one would claim this it's
also not very much a part of
um
the intuition behind the intelligence
explosion Nick Bostrom in his argument
does not refer to this at all
um I think in the AI film debate
um yourkoski makes a few small weak I
think he explicitly calls him weak uh
references to this
um
catches as if she hears strong arguments
exist for why we should expect an
intelligence explosion
um if she finds them I would be very
interested in seeing them I have
actually heard the opposite that there
are no strong Arguments for and whether
or not an intelligence explosion would
happen
finally some of the key concepts are
vague in particular control which is a
rather central part in an AI takeover
scenario is not Zero Sum
catch it doesn't go into it more than
this but I think it's an interesting
question and I'd like to go more into it
and of course depending on how you
Define control
if you would Define it as like control
over nature that could increase you
could imagine you have um like an
entropy-based definition of control in
that case you could also see the total
amount of control increase
um
I think the arguments like I'm not going
to say that Nick Bostrom is perfect I
think almost certainly you could make
things more precise you could make
things more mathematically rigorous
um
but on the other hand I also feel that
control is a real thing and my intuition
for why this is a real thing is that the
sense that there are many different kind
of definitions you can come up with and
they all imply each other so if you have
control among one of these definitions
then you have control among all of them
so here are five definitions of what
control means in this specific scenario
so Nick Bostrom has the definition of
strategic decisive strategic advantage
people often use control as like having
economic and political power dominance
or the ability to influence your light
cone the ability to kill others and not
get killed and influencing and directing
others as the
dictionaries says and I think in all
these cases having a totally dominant
and strategic level of control like if
you are perfect at influencing the light
cone you can get a decisive strategic
advantage and if you have perfect
political and economic power then you
can influence and direct others through
any level required
that is all for today thank you and see
you next time