220. June Ku on MetaEthical.AI

hello and welcome to session 220
in the aict.com reading group tonight we
have
june coo with us presenting her work on
mythical
ai she is she described herself as the
um as a computational meter ethicist and
as far as i can google she's the only
person in the world
with that title so june thank you for
coming
uh yeah um i appreciate everyone coming
here and
uh so today i'm going to introduce my uh
research
on uh metatoyi and uh
it's basically a technical proposal for
uh how to compute an ethical goal
function that
would be suitable for like a smarter
than human
artificial intelligence or in slogan
form
how to how to get an ai that does what
we should want it to do
so my approach here is to basically
directly uh
directly tackle some key philosophical
problems
including uh meta ethics and then
problem of intentionality or mental
content
and that's broken into two sections
first semantics and then
first the syntax and then the semantics
so what is math ethics well ethics
is about what to do uh meta ethics is
kind of
a layer more abstract than that it asks
things like what is the meaning
of ethical concepts and
what is the status and metaphysics of
ethical facts
so for instance are ethical statements
the sorts of things that can be true or
false
and if so what in the world would make
it true of ours
um so i think maybe a good intro to meta
ethics is imagine that you're
translating some foreign language
and and you want to know what if
anything you should translate into the
concept
should um one thing you don't want to do
is just
immediately jump to your ethical theory
as an analytic group
so for instance maybe you're a
utilitarian
in your ethical theory i i think you
still shouldn't think that
should is just synonymous with happiness
maximizing
because then you're going to run into
issues if if someone says
some people should suffer in hell then
it seems like you're going to have to
attribute to them this
incoherent thought that suffering in
hell somehow maximizes
happiness when they want to just be
retributivist about it
so if you think of all the different
things that people have held to be
ethical or not
and do so and somewhat coherently we
know what they mean
when even if we disagree with them then
i think that starts suggesting that the
actual content of the ethical theory is
not that
central to the meaning instead i would
look at
the inferential roles or conceptual
intuitions surrounding
the concept of should so for instance
generally if i'm judging that i should
do something that usually comes along
with motivation to do it um if you're if
you're
translating me as saying i should do
something and that never has any tied to
my motivations you might
start questioning what exactly right
translation
so similarly i think it at least
purports to be factual we
assert or deny ethical statements we
argue for or against them
we can wonder and inquire into what
we should do
and when we're saying that someone
should do something
then there certainly is a sense in which
we're trying to
influence their behavior but it's not
just any old
type of influence so we don't
we're not just trying to manipulate or
brainwash to watch them
instead it seems like we're trying to
get them to recognize
reasons that they should do it so so i
think
uh if they should do something then
generally they should be able to
correctly reason from it from something
that they already assessed
um and uh i think
topically talking about what we should
do invites this kind of open-ended
reflection if
if i say you should do x because of y
then then we can ask and turn well
okay but should i do y and it kind of
always makes sense to
ask that question um
and and finally i think there's uh
some so deliberating about what to do
uh seems to not just tend to come along
with
our motivations not just correlation but
i would argue
uh this should actually be at least a
causal tendency
uh so that uh deliberation isn't just
this epic phenomenal thing that has no
causal effect on anything
so i think what this stuff starts
pointing to is that ethics
presupposes a philosophy of action or
some kind of normative psychology
and you might notice that in general
ethics seems to only apply
to agents and usually human
human beings adult human beings and not
to
uh other animals um instead it seems to
be restricted to agents who have some
capacity to
reflect on their desires and when
they're reflecting on their desires
they're assessing them according to some
sort of standard
and that assessment exerts some causal
control
over their desires and then similarly
for any given standard we could assess
them according to some
other standard and
that similarly exerts control over those
standards
and so i model all of this with
positing higher order preferences or
utility functions
so these are things that are going to be
isomorphic to
mathematically isomorphic to normal
utility functions
um but instead of governing
actions they're going to govern other
preferences
through normative judgments um
so this leads to the statement of my
meta ethics
uh which i call norm descriptivism um
which is that ethics reduces to
which values best satisfy these higher
order decision criteria
this is criteria i just kind of use
synonymously with the high order
preferences utility functions and so
my argument for this would be that this
is the best way of systematizing and
explaining the
conceptual intuitions from the previous
slide
and on this view ethical facts just turn
out to be
the correct answers to the questions
that we're asking
in deliberation about what to do um
so i guess to
to go from the matter ethics to the to
the ethics
i guess you would want to figure out
what what are these questions that we're
asking in deliberation
and my approach to that is just to
give a general theory of what meant what
mental representations
whether it's a belief or a goal is also
counting as a mental representation
i'll give a general account of uh
how menstrual reproductions were and um
and then that would fill in the content
of these deliberate questions and
therefore of ethics
so philosophers call this the problem of
intentionality
the problem of intentionality as things
like what are mental representations
and what determines their content in
this first section
um we're going to start with just
determining the logical form
of an agent's representation
and so my answer borrows a lot from
daniel dennett and his intentional
strategy
so here's a quote from him saying what
it is to be a genuine believer
is to be an intentional system a system
whose behavior is reliably
and voluminously predictable via the
intentional strategy
so um as far as i know that then it
doesn't
get into how you might work this out in
technical detail so
so that's what i've been working on um
so i'm going to define a sort of space
of intentional strategies or decision
algorithms
just mathematically what does this space
look like well a lot of it is going to
be pretty familiar from standard
decision theory you're going to have
credences
which are just assigning a subjective
probability from 0 to 1
to various logical causal formulas
that'll include conditional
probabilities as well
[Music]
utility functions or preferences where
you assign some real or rational number
to
some formula being satisfied um
there's going to be the inputs and
outputs
inputs are going to be a subset of the
credences that correspond to
like peripheral sensory brain events so
sense data essentially
um and then and then the outputs would
be the actions governed by
the decision algorithm
so just be like motor output
and all of that is fairly standard so
far
but the main thing that's new is is
these higher order preferences or
utility functions
sometimes i call them accepted norms
and these are again very much like the
utility functions
they're also assigning real rational
numbers to
formulas but in this case these formula
formulas are generally going to be
referring to not the external world but
to
other utility functions or preferences
within the agent
so all of that defines a decision state
and then we're going to have state
transitions that describe the
dynamics of how an agent moves from a
given state to another one
based on new inputs coming in
um so
so that that just sort of tells you all
the all the possible
intentional strategies or decision
algorithms that we might attribute to
our brain
but uh given some some brain we want to
pick the best one
and so so we want some notion of what is
the best intentional explanation
uh it's got a few components um so first
we're looking for one that best
compresses
the brain's behavior and the compression
is kind of a way of favoring
the simplest and best fitting decision
algorithmic explanation
of the brain's transition behavior
next we've got some measures of
rationality so this is going to include
things like probabilistic coherence
uh instrumental rationality and
that's going to include uh the
equivalent of instrument rationality for
the higher order preferences
um and just basically just amounts to
some kind of principle of charity
uh in interpreting what the brain is
doing
if you could attribute to the brain some
rational thing that it's doing and some
crazy thing
then all else being equal attributes
into a more rational thing
and then finally we want these
explanations to be ambitious so it's
trying to account
for as much of the brain data as it can
ideally anything left over in the brain
data is
more just noise than it is a decision
process
um okay so
so so far uh that's
that's really telling us what what is
sort of
uh most useful model of of a brain
but you might have this worry um
about wanting a more realist criteria as
opposed to like the instrumentalist
criteria
and um i i think denit himself is a
little wishy-washy
on how realist or instrumentalist he
wants to be but but
basically i have this worry that
couldn't you just be coming up with this
decision algorithm
as a useful predictive model but but
that's not actually what the brain
itself is doing
and so i add uh a further condition that
i borrow from david comers and um
so thomas has this um uh paper on
how it's
how when a physical system implements a
computation
and so in our case the physical systems
that we're going to be
interested in is going to be the brain
brain states and their causal
relations to further brain states uh so
it's just actually going to be um like a
utah pearl
style causal model of the brain and then
uh and then i've introduced what the
decision states would be and the state
transitions between them
so the implementation function f is
supposed to take
a brain state and tell you what decision
state
is that brain state in so it'll tell you
the credences and
utilities uh preferences things of that
sort
um and so the the trauma criteria is
basically this equation we want to
kind of make sure that whether we start
at a given brain
state and move to the next brain stay
caused by it
and then interpret it with f into the
decision state
that that this route gives you the same
result as if you want
this other rod first you take that brain
state interpret it into the decision
state
and then take the state transition to
reach the final decision state
um and so
so we're going to require this not just
for
the brain states that we've actually
observed but even counterfactually
um in the causal model for for all the
possible brain states
um and there's more details in his paper
does iraq implement every finite state
automata
and and commerce develops this theory as
a way of saying
no it doesn't which is hopefully the
intuitive
result that we want um
okay so so that that covers how we would
take a brain and try to figure out
what formulas the syntax uh that we
should attribute to it
but then given the syntax what if
anything do these
logical expressions refer to or what are
the truth conditions
of the formulas
so there's a few principles that are
grounding the
reference in my theory so first we're
going to have
these self-denoting input expressions so
that subset of credences these are
supposed to be
the sense data and they're going to
refer
to the brain states that implement
credence in them so
they're kind of self-referring that
those brain states are kind of
self-referring in that way
so you might think of their content as
something like this sense statum is
occurring
or if we want to make it even more
simple uh
just a pure demonstrative this
and if we're trying to sort of build up
a theory compositionally
starting from some atoms and building up
molecules then we kind of want to start
with we kind of want to try to find
something that's possible that's very
simple
and primitive and and i think this is a
good candidate
i think everything kind of also carries
information about itself so it's not
surprising that we could
have things stand in for themselves um
and uh also this kind of makes the whole
project
work because these are gonna serve as
anchor points for logical and causal
combinations of these expressions
so if you if you have a bunch of these
sense data referring to their own brain
states
then then we could start talking about a
conjunction of them
or positing a hidden cause that
that causes that conjunction of sense
data and that starts allowing us to
refer to other things
um another thing grounding the reference
is
uh these inferential goals for
connectives
so connected just being things like
conjunction
or disjunction causation
so imagine that you have an agent where
we observe
the following dispositions when they
believe
the proposition p and the proposition q
then they tend to infer this new
proposition p
star q and when they believe p
star q then they tend to infer that p
and infer that q um so you
you might uh um
you might notice that this seems to
match onto the
truth table for conjunction and so this
idea comes from
ned black in his conceptual world
semantics
um and the idea that uh the idea seems
to be that uh
we could figure out that this star
operation seems to
uh be the
conjunction because the inferences that
these are involved in are basically
matching the axioms for conjunction
and so just to generalize to other
connectors being grounded in their
axioms
and so if if we're going through this
process of
uh attributing the syntax um
then when we when we attribute uh
these connectives in a way that deviates
from these axioms these are going to be
uh
punished by the coherence score um
okay and then and then finally just so
those gives you some ways of building up
um some references um
and then there's this uh a more general
idea of how to
if you already have some old terms that
you understand there's a
this ramsey lewis uh sometimes carnet
uh is thrown in method for defining uh
new
new terms originally it was for
theoretical terms like like
for a scientific theory
um here we're gonna just use a simple
example of car theory i think this also
comes from black
um and uh so so imagine this like a
scientific theory and it's introducing
in in this uh lavender color
um it's interested introducing some new
terms
carburetor and ignition chamber
uh using some old terms like fuel and
air
and we're assuming that we already
understand the fuel and error and other
terms
but that this theory is introducing
carburetor and ignition chamber so it
might say the
carburetor mixes fuel and air and sends
the mixture to the ignition chamber
which in turn blah blah blah
now one one one thing you might be
worried about is
if we're maybe defining the carburetor
in terms of its interactions with the
ignition chamber and we're
and we're defining the ignition chamber
in terms of its
interaction with the carburetor is that
going to lead to some kind of
vicious circularity and definitions uh
it turns out that there's this nice
technique
that kind of shows that that's not
really
that big of a concern um what what you
do is called rancification
um and uh you take you take your
uh theory and you replace any of the new
terms with variables
so here carburetor has become x and the
ignition chamber has become y
and then you just do existential
quantification over it so now you're
saying
this is called the ramsay sentence now
you're just saying there exists some x
and there exists a y
such that the x mixes fuel and air and
sends mixture to the y
which in turn blah blah blah
and so this is a nice way of if you
already have the old terms
figuring out the meaning of new terms to
refer to whatever fulfills the
functional or causal goals positive for
them
so in this case we've done it with uh
with objects um but if you use a second
order logic this can generalize
to predicates and relations as well
and so i kind of want to apply this
pretty globally
and holistically um
but uh the usual way
it's it's talked about is as you have
the entire theory
being true so that that we have um
like all of the sentences uh um
are being true but when when you're
moving more towards uh
um filling in all the mental
representations of an agent
then then uh probably they're gonna have
some false beliefs and we'd like to
still
uh apply this method even if some of
them some genome are false
so i'm kind of weakening the condition
here to allow for some error
and and then we'll just kind of try to
find
the the set of uh um beliefs that would
minimize
uh squared or the set of interpretations
uh semantic interpretations of their
beliefs that would uh minimize the
squared error
as we're filling in the functional
causal roles
um and let's see
yeah i i can go into more technical
details later but uh but
uh but just hopefully gets you some
intuitive idea of what principles i'm
relying on
um okay so to put this all together uh
here here's basically how i propose
computing
readiness in five steps
so first step we're going to start with
just assume that we're given
uh in the ai a low-level causal model of
the world
and the adult human brains within it um
[Music]
then we're going to take those brains
and we're
going to attribute the syntax and
dynamics of
mental representations to those brains
part of that syntax is going to include
for these higher order preferences
what they refer to or or at least
at this stage at least their logical
form not yet what they referred to
but uh with with the higher order
decision criteria
we could iteratively apply those
criteria uh
to figure out what rational first order
utility functions
uh these these brains uh should have
and then we create so so far these
rational utility functions are
are are still couched in the agent's
language of thought
so next step we translate using the
semantics
translate their uh utility ratio utility
functions from language of thought
to external world states which would
just be
like the causal model that the ai has
for instance
um and then um
and now uh that helps make it more
uh comparable so now we could aggregate
everyone's rational
utility functions using some social
choice
or social welfare function
and i think that's it that's some credit
for images
and necessary some technical details and
an appendix
and they'll keep it on this slide
uh yeah so um
appreciate any uh questions
okay thank you very much june for your
presentation and so the first question
will be from stuart armstrong okay well
thanks for this uh presentation
there's a lot of interesting ideas in
there
um so the first question is just a
general meta one
where do you see your
project as needing as being the most
incomplete and where do you see it as
being
complete so of these say five steps
uh are some of them uh you think
basically done
or others and others need a lot of work
or what's your feeling on
uh that yeah um
yeah i guessed uh i sort of think of it
as um
yeah i mean so a lot of background is
has been in academic philosophy
but then moving into software
engineering so i'm kind of
taking a a engineering approach to this
i suppose and
just trying to come up with what it
would be like a minimum viable product
for
for um competing friendliness so
um i mean i think that uh
i i think that one thing that could be
filled in more
is um in what areas
am i kind of taking liberties with
biological or psychological plausibility
um i mean so i'm not requiring
uh agents to um reason perfectly by any
means
uh they're it's sort of made into
a great aggregational coherence scale so
that's one way
in which i can accommodate um
some psychological plausibility in there
but there might be further ways or maybe
different types of logics that would be
better able to capture how humans reason
for
or for instance maybe like um
i mean right now all the credences are
sort of in one big
belief box but maybe there's suggestions
from psychology that actually
there's some kind of massive modularity
going on in the mind
maybe that that modularity should be
explicitly
represented in the model um so
so i think there's a category of just
making
it you know maybe there are cases
ways in which uh despite trying to
accommodate more i'm still kind of
assuming that human brains are more
computer-like
than they actually are so there's just
one big
area where i can see a lot of room for
improvement
so would this be about uh in your first
step
for example yeah
well the first step is that's just sort
of the starting point
um um
[Music]
yeah so i'm just assuming that these are
inputs to
the uh uh to the a ai and computing this
um i mean i guess we could relax
some of this stuff as well like
basically if you have
instead of a single causal model that
we're just assuming is an oracle telling
us the truth
of the world um if instead we have
um a probability of distribution over
call the models of the world
then i i think a lot of the stuff should
straightforwardly carry over like
conceptually at least it should be uh
hopefully pretty clear what would be
wrong i don't think that
standard uncertainty is a problem here
that's something we're quite used to
dealing with
you'd like to do the first question uh
sir
sure other people so one of the uh
things we we discussed in the reading
group when we when we read this
was um a path towards a
feasibility basically both in terms of
uh if we are actually implementing this
and also in terms of
uh making a simple end-to-end test
like with the software that you have
already developed would it be possible
to make a
world with two brains that want ice
cream or chocolate
and then and then actually see the
computed utility function from that
uh let's see so right now it it requires
um
infinite computational power um
and then so that's mostly because i'm
using uh the um
como guerral complexity in various
places
which is uncomputable
and you could you could substitute some
finite approximations to those
i think like minimum message length
minimum description length
are are some existing ways of
of having finite approximations to the
chromograph
complexity um and then
um but but even even once you go once
you make that
finite um i there's like
you know virtually uh no
performance optimizations whatsoever in
in most of my code here so so many of
them are simple
brute force algorithms that were more
about
can we even solve this with infinite
computational power
just to sort of make it clear what we're
aiming at um
and and it would it would certainly take
a lot of work to
um to try to pair that pair down
um now i i have uh i i
let's see if i could uh you know right
so i have written some uh
um some i do have some test coverage so
any anytime on my website you see um
a check mark next to the next door
procedure
then then that points to a test of that
procedure
um so i think there was last i checked
maybe like 47
of the procedures have some tests um
so so maybe
yeah it would take a while to get i i i
also would like to build some very
simple toy model
um to uh yeah to try out this
uh this theory there um
yeah it's gonna take some work because
there's just lots of places where
where things are very super duper
exponentially
exponential uh and and some of the
some of the testing procedures i i make
use of some caching and stuff just to
just to be able to have any hope of
testing this stuff
um so there's maybe some engineering
tricks you could do so that it doesn't
actually have to compute
like all all possible um decision
algorithms that might correspond to a
brain
maybe maybe it's enough just to like
sample
from that distribution um so
so yeah certainly there are many places
where
you could start making this more
computationally practical
yeah and aside from a couple of places
where
it was useful to be able to write the
tests
um
there hasn't been much work yet in in
allowing for that kind of end-to-end
test but that certainly would be what a
direction
i am interested in going
okay thank you for your answer um stuart
would you
take it yeah so one of the great
advantages of the way you've laid it out
as a program is that it provides clarity
as to exactly what
is assumed one of the great
disadvantages
is that it doesn't allow us to see how
big an assumption on how much work is
required
on one line of code versus
another some of them may be entirely
true
some of them may need a little bit of
work some of them may be
hiding a whole part of the problem in
now this is the bit that sora knew that
i was going to bring up
there on the occam's razor result
um you defined the best intentional
explanation
as having maximal compression yeah let's
get the slide up
um high rationality
and um ambition
well let me give you an explanation that
is
absolutely fantastic on all these fronts
humans are fully rational in every
single thing that they do
and they always pick the perfect action
this is i've shown in my result this is
gives you the best compression uh it is
obviously fully rational and the
ambition is
perfect it explains everything
um okay so i yeah
are you you're talking about like your
uh no free lunch
here um yeah i uh
i i i've been wanting to dig into that
uh
further i haven't uh i i
just sort of barely skimmed it but i do
wonder whether
um my setup is a little bit different
from
from the one you consider in particular
because
of the trauma criteria that has to be in
place
so so whenever you're attributing some
uh decision algorithm
um it's gonna it's gonna require that it
actually correspond
with um with the brain's uh
transition behavior um that
that does not seem to be a problem in
the
so the humans are fully rational we all
agree as a degenerate example it's wrong
but we need to find out why it's wrong
um and when you do that what happens
is that the utility function expands
to almost the whole of the brain so the
whole of the brain can be seen
as computing the um
the utility function or the reward
function and then
you can zoom in on say a small set of
neurons which are the
input output and they are implementing
the decision procedure which is just
basically
follow what the rest of the brain
has computed so it is it does not
seem to me that it would be that hard
to designate a particular part of things
as the
rational decision makers because that's
a very small
defining a fully rational decision maker
is does not take much
code and you can
basically assign it to just a few
neurons that pass on
the message basically if you want the
rest of the brain
says taking this action is the
right thing to do in terms of utility
the intermediate neurons say thank you
we will therefore take that
action and that's the rational
irrationality module
and then you seem to have a
model that works in your sense yeah i
think this is kind of reminiscent
of um of like putnam's paradox that
uh originally motivated chalmers um
in this paper so um so putnam
it was using this model of uh of like a
finite
finite state algorithms and
and there um
[Music]
any any given finite state uh
any any given state within their um
it was treated as like kind of simple
it didn't it didn't involve any internal
structure
one of the moves that chalmers makes
which i didn't really talk about
um in this slide but it is but it is in
this paper does iraq
implement every finance throughout it is
he moves
uh to what he calls a combinatorial
complementarial state algorithm where
[Music]
instead of allowing the
some simple states to potentially encode
all of the um
some complex state
he wants to more explicitly model the
internal structure within
any given state so within a physical
system is going to be
it's going to be implemented in a bunch
of sub-states
so so i do i do still wonder if that
if that is able to get around um
uh this type of objection um
i have reasons to believe
that the result still applies um
with internals well i i know the result
applies when you know the
internals of the agent um
it just depends on how many
would you would you mind if i shared a
paper
uh here i i know it's your
your talk oh that's not a paper a um
a blog post okay um
should i stop sharing my screen no no
that's fine i'll just
uh put it in the um
the um
now ah okay learning human preferences
black box white box
structured white structured white box
um the
essentially the problem is not just that
you have to identify how the algorithm
happens inside
but what are the correct labels for the
algorithm
for the different parts of the brain
wise
is this a bit beliefs is this motivation
is this rationality of course it's much
more complicated in the human brain
but what are the criteria that you would
use in assigning
these labels to the different parts
of the brain and whether that can be
done
whether that can be automated
is the now it can be automated if you
make
enough assumptions but
the kind of structural assumptions that
you've been talking about
do not seem to be um sufficient
or even close to sufficient okay so the
white
the white box is going to include
knowing its internal structure
yes okay it's um
and the the the thing is that what we
need is what
uh here i call them structural
assumptions in other places i call them
normative assumptions
which are given that you know the
internal structure how you to assign
labels to them now there's a rather
silly example where something is labeled
tuna
but that's basically just to show the
arbitrariness
of the labels and what typically happens
with humans
that resemble sorry when humans
approach these problems that resembles a
bit your
uh your description there is that we
define
these things with very few um
syntactic connections like
the beliefs take input from the senses
the action selector is the bit that
outputs to
the uh the moving about and stuff like
that
and the beliefs go into this and so does
the
preferences and so on but those
few structural assumptions that
generally there's
trillions of ways of taking a complex
algorithm
and matching those so
there seems to need some extra
amount of information or assumptions
what i call structural assumptions
now it's not hopeless
but i don't want i don't i want to talk
about your
approach not about mine uh but
it does seem that this is as it stands i
would say this is a huge gap
in uh your approach
that is just a few lines
on your uh on your meta ethics
page so just as an illustration
of the potential hiding of large
problems
in small small lines of code
um okay well i look forward to uh
i don't think i've seen this this post
before
um
i'm i've been told that i'm not the
clearest communicator
and um in any case so
if you if you want to talk about that uh
please do let me
know yeah
um she would say next question
so jack you had a question
like sort of a clarifying question um
so so
in in your five steps um i think the
third step is um iterating higher
yeah applying higher order decision
criteria to reach rational
utility functions so for that to
occur is there
does there need to be the assumption
that humans or
whatever um whatever
brains you're modeling does there need
to be the assumption that those
have coherent utility functions because
it's not immediately obvious to me
that humans do have coherent utility
functions or that we should expect that
to be true
maybe it is i just don't know yeah
um let's see so
yeah i think i think this might be one
of the places where
um for for the first version
it it i think it's probably going to end
up trying to find
the the utility function that that is
closest
to theirs and that is rational i believe
um
[Music]
yeah i mean that might be something to
to relax
in later stages um
[Music]
yeah i'm just trying to so
um yeah i i guess the way the way that
i've
encoded the utility functions is
um yeah so if i had if i had modeled
if i had modeled the agents
with say like um i guess
ordinal with ordinal utility functions
then there would be room
to model it as uh irrational but because
i
i took a more simple approach of just
modeling them
with cardinal utility functions uh
i think i think that that's going to
make it so that
they they do all have utility function
so
um yeah so i i am i am kind of
interested in can we relax those
assumptions
and still make the project work and my
my intuition says yes like for instance
maybe
maybe maybe uh drop
completeness but we could we could still
do things with uh
the different possible uh ways of making
them complete
um and and still run the algorithms off
of those
uh but that will probably be more for
future work yeah
gotcha okay thanks
okay um my next question
or uh point is instead of pointing out
something that might be unexpectedly
difficult
and to point out something that i think
might be unexpectedly easy
in the the grounding of
uh to use the sort of um the
gophi term the grounding of the symbols
in the brain
which i believe you are um
yeah the tr anyway um translating syntax
to semantics
this may be a lot easier
than uh we think because it seems that
there's an empirical
definition of this which is does
can we use the symbols in the brain to
predict
things about the outside world or can we
what is the relationship between the
symbols in the brain and these symbols
in the outside world
in a large set of environments that the
humans find themselves in
so um yeah yeah i do i do think that
there should be
a lot of fairly easy test cases there
um i think i do have a little bit of a
remaining worry though
especially because the the most crucial
[Music]
symbols to ground would be the ones that
show up in the
higher order preferences so i think
those high roller preferences being a
little bit further removed from
from everyday action i do wonder if
though those are going to be less
amenable to
that sort of treatment and and and there
we want
maybe more of a theory a theory that's
been tested by easy cases but but
theory guiding us um in
in figuring out what those uh
representations are
well you've touched upon a subtle issue
there
which is that our symbols are well
grounded by our experience
in the environments that we know the
symbols that are not
particularly well grounded are
um can are basic are often ones that are
not
well def sorry if you place the humans
or the agents
in a new situation where some of their
symbols don't work the same way that
they're used to
i one of the examples i use is what if
someone
creates a species
human mic a slave species
that want to be slaves definitely want
to be slaves but don't enjoy
being slaves they've recognizably human
they have preferences which is to be
slaves they have
enjoyments uh thing and in this
situation
a lot of say common assumptions that
normally go together
start breaking down or splintering was
the term
uh that i was using and this is the
situation in which you
generally find that these higher order
symbols or these more abstract symbols
that you thought you had a good grasp on
suddenly you aren't so sure anymore or
they can be defined in multiple ways
in a way this is what philosophers are
doing all the time
because they take common concepts
and push them into extreme thought
experiments where they break down
and where and but if a world
with potentially super intelligent ai
is a world where the ai can push
the environment into incredible new
states
where um the philosophical thought
experiments
are there are the actual thing
do you have any way of sort of dealing
with that
when symbols become ambiguous or when
symbols that used to sort of be synonyms
are no longer synonymous
um yeah so i haven't uh i haven't
actually gone into so
so um so in my meta ethics
paper which some of you uh read last
week
and and so far in this presentation i i
actually haven't gone into that much
technical detail it turns out that um
i'm giving sort of a simplified view of
things here
so so maybe i should actually um
get into this appendix slide um so my
first thought and the way i
sort of uh certainly in the meta ethics
paper
had been talking um
i i've been assuming i i don't know if
this is going to directly translate into
into your words they might still be
separate but but just to give you some
idea of
where i actually end up with the
technical specification
um so so in the first pass i kind of
assumed that these higher order
utility functions form some kind of a
neat hierarchy
right so there's maybe there's just one
highest or utility function
and then your job is relatively easy
figure out
figure out what lower order utility
functions that it prescribes
and then keep propagating that down
until you reach the first order utility
function
but i think that's not exactly
psychologically plausible i think more
realistic model would have
usability functions that are maybe able
to mutually
influence each other with no single one
on top
so in this case i want to um
uh it italy choose
sort of allowing them to simultaneously
implement
gutter and ah you're choosing outputs
that that satisfy
each of its decision criteria
and and and you keep simultaneously
updating until you reach some kind of
stationary
equilibrium um i think even that one
comes with an assumption that we not
we might not be able to retain uh then
and
namely it's it assumes that there's a
network topology
of of which which accepted norms
are are governing which other accepted
norms or high order utility functions
and
and it's possible that those actually
are path dependent and depending on what
input you feed it
so so i actually think we have to move
to uh
even more complicated one where now now
we're going to simulate
all continuations of an agent and and
apply a weighted
social welfare function
and uh continuations who better satisfy
those decision criteria and
uh preserve a gentle identity which is
kind of like personal identity
um but basically they better satisfy the
decision criteria and
and there hasn't been other irrelevant
changes to
to their through their values uh those
continuation
are going to have more weight and then
and then you apply a social welfare
function to
to aggregate them um
did you want to these are
this is um these are close to the kind
of thing
that i was have been thinking about and
so
it is um uh it
i don't want to come and say as because
you're thinking similarly to me you must
be right
but um it means at least that you have
been
um thinking about the same issues and
how to
address it um like
the just to check one thing
there is no strict to ordering a
weak higher order utility function can
be overruled by
strong lower order uh preferences
yeah i think so i think that should be
possible like a
yeah uh say a mild religious preference
um towards on sexuality or something
like that
can be overruled by sort of strong
object level
uh ones would you say to
um to pick an example yeah
um let's see yeah i think
so so i under and that case i'm going to
end up saying something very similar
i i think i'm just going to take slight
issue with the way you said it
you described it as the like the first
order
preferences overriding the the metal
level
preference instead what i would say
i i i would probably say
we want to model that as a different
meta-level preference
that says something like um
[Music]
when when you have first order values
that strongly conflict with
some kind of weaker high order
preference then then allow
the the first order to um
to override that one but but that's all
going on within
us this second metal level preference
um so so conceptually i kind of
one one problem in in the language here
is um
higher order preference is ambiguous
between a couple of things
um so so one notion
of higher order preference that's that's
not the one for the most part that i'm
using
is is one that's a first order
preference
it's a preference in the sense that it
is governing actions
um um but it could have some high order
content
uh it could have content that includes
talking about
other preferences so so you could have a
preference to change your preferences
um and and i think those are different
from higher order preferences that are
um
defined not by the content but by its
causal functional goals in changing
your other utility functions because
that first type of
preference that that really is just a
first order preference
governs actions and
and only only affects
uh other preferences through
actions and i think and i think that
when we're talking about
things like being moved by moral
arguments
i think we're not we're not talking
about this pragmatic rap
okay i'll i'll have a follow-up question
on that after
um after the next question
okay so that would be my question uh
one thing i didn't see and maybe that
was because i didn't read it very
carefully
was a an explicit comparison with ikey
um and um so i would like to hear
your thoughts about how this relates to
icn in particular
the thing that i'd say is outside the
universe
and it seems possible to me that uh
your construction also kind of assumes
that the uh
the computer that's implementing this is
outside the universe
or is is that a requirement um
um let's see so
so so all of this is supposed to be
going on like within the mind of the
of the ai um the the ai is supposed to
have
a a complete true causal model of the
world
and the brains in it um
let's see so so i guess
i think i think what is the term like
the cartesian
separation of a few apart from the world
i i guess that could theoretically come
up with the brains or it could come up
with the ai itself see i'm not sure
yeah i mean in a later stage one thing
one thing i do want to eventually get to
is
is things where this stuff might come up
more like
um like i i think that um
meta-semantics and meta ethics is
actually
um uh that actually gives you most of
matter philosophy
so theoretically i think i should be
able to use the resources i built
to do some kind of verification or
validation
of the philosophical process that led to
the creation of the app
then in that case you probably
would run into these
worries with self-reference and it's
is that there the ai would would be
modeling itself
in the world and as being caused by some
causal process
from the brains and and and trying to
check
did they make the right philosophical
decisions um
so so so all of this is
is still very speculative in handwriting
but but that's where i imagine some of
these
some of these issues might come in as it
is now
i don't know if i don't know if they i i
don't think i've had to had
the ai model itself anywhere in here
it's it's just modeling the brains
figuring out what they value and
advocating those values
uh i i don't think i had to have the ai
talk about itself in the world
um so so maybe maybe it's
yeah maybe it's just ambiguous of
um or yeah i guess it's
ambiguous whether whether it does have a
model of itself in the world
okay um
yes so on the um
um sorry i'm
i'm having difficulty uh remembering
exactly
on the when you were constructing the
models there um
and were
ah the different weighting of
the
yes the main point is just quite simply
that most
humans do not are not philosophers
and so we have not constructed higher
order
preferences or meta preferences we've
especially not constructed meta meta
preferences for saying that a weak
first order in the vague sense
a weak object level preference should
over sorry a strong object
level one should overrule a weaker well
certainly they haven't they haven't
regimented their vocabulary to
articulate these sorts of things as much
as philosophers have
i i'm i'm not so sure that they that
they lack these
things though well what i was going to
get to is well i don't know
uh to the extent uh is what about
someone who has not
yet considered the problem
but would would it would have one
solution
if they did consider it and the second
one is what if someone who has not
considered the problem
and could get one of two solutions
depending on
which arguments they read first
both of these are cases that are quite
likely to happen
so would you say that their meta
preferences there are
undefined or yeah well that's
that's exactly the sort of thing that um
that this
this stuff in the appendix was supposed
to
alleviate so so that would be uh
there will be different continuations of
the agent one no time here one argument
another here is a different argument
um and then as as long as as long as
they're both satisfying the decision
criteria equally
then then they might just be on equal
footing when you apply
the the social welfare function between
them to
aggregate their so so are you saying
that ones that are not yet
defined um
can
preferences that are not yet defined but
that could come up
are also included in the set of
preferences to consider
um let's see
i i mean i think i think that's how i
would like it to work i think i did run
into some
some technical issues uh but but
certainly at the very least
um changes to existing preferences
that that might come up depending on
different inputs
those are those are going to be
accommodated in my model
um does did that make sense
um if you're having new preferences with
new symbols
that that that the original agent didn't
use
um that that's actually uh one one
one place where my model doesn't
exactly do well um okay
but as long as it's as long as it's
using the same vocabulary
of the original agent and it's just
changing how much
for instance is valuing things or it
could even be a new a new preference in
the sense that
the original agent was able to represent
these outcomes fine
it was totally indifferent but now you
could have a new preference
among something that the original agent
was at least able to represent
but and then you can have a new
preference on something that they were
previously
indifferent to so all of those types of
changes would be accommodated by this
uh okay well i think this connects with
my first point
about new environments um so we can sort
of bunch them all together
as what happens when there are new
symbols
and both when it's predictable
that the person will always interpret
the new symbols
this way under any realistic
introduction to the subject
and the situations where they can
interpret the new symbols in multiple
different ways
yeah um yeah i'm trying to think
how big of a problem it was
you don't have to solve everything right
now
yeah yeah yeah i'm just trying to think
is it like a big problem a medium
problem
um so so within my model
um i think i think the problem was
because
so think of that last step where i'm
translating
um the translating rational utility
function from language of thought to
external world states
then um
if i was going to try to accommodate
these new symbols
in in this other possible world
are these are these things can be
separate from
the causal model of the world or i guess
they would
it should still be possible from within
the ai's world
um so so maybe as long as as long as you
could still translate them into external
world states
they might have to be like merely
possible external world states or
something
um then maybe it's it'll work but it
gets into like weird
metaphysical issues of how to how to
align this
this stuff um and and then
yeah there was another another part that
i was a little unsure on
uh but that kind of ties into this is
the philosophical purist in me wants to
say
my values are my my values
are grounded in my brain so
so so if you took my my brain and put it
in
different circumstances um
i kind of want to say i have the same
values
so so then when
[Music]
um so so one thing we could do is
is put in a probability distribution
over
over worlds that my brain could be in
but then
but then this probability distribution
is ending up
influencing the continuations and
and so the the philosophical purest in
me didn't want that happening so i
talked
instead about just like all possible
inputs
but but uh but making sure that we get
we kind of get rid of get rid of the
ones that just introduce
non non-cognitive changes like
changes that don't come about from
reasoning about your high order
preferences
are are going to end up getting very low
weight
so i don't know that
that philosophical purism does create a
little bit
extra difficulty um here
and and i'm not entirely sure whether to
give it up
um i personally recommend giving it up
um it's uh because if it's
if it's true you'll find it out anyway
and if it isn't it'll be an obstacle
in the design
i'd recommend hacking together something
that kind of works
and improving on it and if
if sort of purism or more realism or
something like that turns out to work
then it should naturally emerge in that
context
but if if it doesn't work and you try
and impose it then there's
something might break
and that's just my my advice there
yes uh over to you sir yes
um one of the things that came up in the
reading group was
uh this uh methodical is written in
central x which is somewhat of a niche
programming language and if
conflictually this was written in python
um
there is a sense that maybe uh you would
have gotten more engagement
out of it also it seems like the code
seems to be optimized quite a bit
for correctness in that with the tests
and everything
and maybe um optimizing for readability
would have been a better uh choice like
longer variable names and things like
that
[Music]
yeah um i don't know if i would
move to um python but but
i am sympathetic to uh possibly porting
it to a different language
um probably something like pascal would
be
um um
one one that i'd be leaving the most
towards
uh because um
[Music]
i i did i did want to maybe keep it in
a form where um
[Music]
in a programming language that has clear
denotational
semantics um
[Music]
i guess my original idea was something
like
once i have it written up in set theory
then then maybe
it wouldn't uh it wouldn't take too much
to just then just write it in standard
mathematical take notation using uh
latex
for for humans to read right um
or if i wanted to continue with the
machine readable
and executable which which i do like um
and then maybe maybe switching to
something like pascal
where escal is also pretty speech
but but certainly there's many more
haskell programmers than
uh set elec programmers um
and uh yeah so
so if i had infinite times there's
definitely many things
many things i could be doing um yeah i
guess i chose
set alex because it had the clear
denotational semantics and
i can imagine translating it into lattek
later on
um and uh
and it just seemed a little less
overhead than writing in
haskell um
[Music]
so so it yeah i mean if if i had written
it haskell i don't know
would it would it have been worth it to
write it in haskell if it would then
take me
longer to release right uh yeah those
those were the sort of calculations that
i had um
um yeah
um
back to me sure
um so you were talking about
uh diet chronic coherence and other
rationality and coherence
um requirements um i'd suggest that
some of the some of these coherence
requirements
are actually themselves more akin to
meta preferences
than to um than to
requirements um the
the kind of thing that i'm thinking of
is
for instance um
uh temporal coherence
like um whether you um
like at some point people
like uh enjoy eating
they enjoy eating stuff when they're
hungry and they don't enjoy
eating it when they're full um we've
decreed that this is not a um
temporal inconsistency um
for the uh the reason
that yeah there are other things where
our desires go up and down
sometimes we want to watch uh
romantic movies uh sometimes we want to
watch tragedies our desires and
preferences in these areas fluctuate
and we still
think that we have an underlying
coherence despite all that uh
because we sort of but
but other things we decree that we do
have temporal incoherences
um like when
we well yeah sort of we overeat
and we don't and then we purge
or we we contact
an ex that we really shouldn't
and then but to pick a
more narrow example if someone
has a peak of sexual desire
and sleeps with someone at that point
this
is appropriate this is fine this is not
an inconsistency if someone
has a peak of sexual desire and calls
an ex inappropriately at that point this
is a bad decision
so the impact of
sex i should choose a better example but
that's the one that sort of sprang to
mind
um can be seen as both um
positive and negative
so not not positive or negative as time
consistent and not time consistent
uh depending on how we see it and
the way we see it is from our metro
preferences
i apologize but the um
the things are not fully worked out but
a lot of things like consistency
um like there's people who there's the
alley paradox people that
violate various dutch book arguments
and uh lotteries and other things
but they can say that they
that you don't people buy extended
warranties
for things and you can easily check this
is
you lose money there's no overall if you
buy extended warranties for things you
lose money
if you don't you're better off much
better if something breaks
pay for it and you'll that'll be much
cheaper over the long term
but some people value the security
that the extended warranty gives them
now
that feeling of security you could say
it's an irrationality
or you could say that it is a consistent
preference or meta preference
that they are satisfying um
so what i'm suggesting is that a lot of
the coherence
things can be seen as our own meta
preferences
and not meta preferences that everyone
would have in the same way and to the
same extent
um let's see okay so so some of your
examples
um um yeah i think i think it's sort of
like
when when you just describe say uh the
peak of sexual interest
um it could be good or bad depending on
the surrounding context right
um so it was more that
have sex when you are when your sexual
desires the highest
is a perfectly rational and recommended
course of action um
call up your ex that you've had a
difficult relationship with
um when your peak of sexual desire is
that it's
top is incoherent and
probably predictably going to lead to um
uh two mistakes
yeah so so i don't know so for example
but yeah the the basic idea is the same
say fluctuating background thing can in
some case be seen as
part of a single preference function
i have sex when it is most enjoyable
or as a source of irrationality
yeah um it reminds me of um
what one response to the dutch book
might be
to to say it's actually fine and you
build into your preference
um um that i that i
you know if they're leading me into a
circle i'm paying
um five dollars to go from a to b and
from b to c
and and go from c to c to a if you if
you build
enough context in and say you know even
in the move from c to a uh
i i just i just value moving to
me moving to a after having paid five
and ten dollars five dollars to
move the other two times uh
if all of that can be in your preference
you could actually make it like
a coherent utility function a rational
utility function it
it it seems pretty implausible that
anyone is actually
valuing things like that but uh kind of
reminded me
of that but um um
but i know some of these issues seem
seem more about
how how we how we
um just capture things within a single
utility function like
like how context sensitive is our
utility function
[Music]
it's um as i say i'm more just prodding
at the various things yeah
seeing how they work and how um
you would yeah so so
so the actual so are you suddenly
functioning i didn't want to get a
little bit of psychological possibility
because
as you if you had to represent utility
functions as
um um specifying
utilities for for every single possible
state
that that's certainly not we we are not
some giant lookup table with
with with the list of every possible
state right um
so so i i ended up using like an
additive utility function
um so so it it
so given some give it some state you you
figure out
um how many of the formulas that you
place your utility in
um uh make it true and you add up the
associated utilities
um and uh so this would allow for
um you know maybe you maybe you value p
so you put some utility on on the
formula p being true
and and then you have another formula p
and q
um and then you could you could
place a different utility on p and q um
and and then that'll i think that'll
actually add up
because both p and p and q are true then
then
then those will add up to whatever so so
maybe
you you generally like when proposition
p
is true uh but if q is around
then you don't like it right so maybe
you you place five utility and having p
but then negative but then zero uh
negative five uh if you have q
so then if p and q then then you're just
indifferent
so those are some of the things that you
could do with with my current
uh my current model and
certainly there's there's probably
other ways to make these more more
realistic and
and context dependent but but i think
it's a fair amount of that
it is allowed by just the additive
utility functions that i have
and in the limit case that you could you
could just add in
um um you know
that such that it behaves again like
that giant look up table of all possible
states but
but uh if there are places where you
need to get more fine-grained it does
allow
a fair bit of that already but i'm sure
there's
also further improvements tonight um
but but but that did seem a little bit
different from um maybe some of the
measures of um
seeing chronic and diachronic i think
that's executing mostly and
where i work out agentual identity i
didn't know if this is where you wanted
to go with it
but um
this was supposed to be some measure
when you're when
running the continuations of an agent if
this is supposed to measure
um how much you're the same agent or not
and and and it kind of included this um
excellent this is
this is the sort of thing that i've
come to think about recently and you
seem to be ahead of me there
yeah a gentle identity is
the well no i'll sorry i'll
come back uh i'll let soren bring in a
question
okay so um one of the uh
parts of the way that the utility
function is constructed
seems to be almost equivalent to
simulating putting humans in human
brains
in different situations and it's given
that it seems we're trying
basically all states that the universe
can be in it's
it's possible that we are implementing
in fact not just
mind crimes but every possible mind
crime
do you agree um yeah certainly if you
just
naively if you found a a computer that
has infinite computational power
and and you ran this algorithm inside of
it
that then yeah well first it'll probably
crash because there's tons of bugs but
if you could fix all the books and have
hadn't run then yeah there would
probably be tons of mine crying so
so uh so this isn't i'm not suggesting
that anyone actually
just fix fix the bugs and then run it
because uh
that would be a big concern uh the hope
would be
that um um
[Music]
somehow we a lot of what i'm trying to
do here is kind of define
the ground truth or um or
the the value functions and
i'm not imagining that this this would
by any means be the actual algorithmic
procedure that the
uh that the ai uses right um
maybe uh you know maybe maybe it's
but i i do feel like it it can be
important to have this defined somewhere
in the ai so that it kind of knows what
is the ground truth so
so some of these other proposals which
which i'm sympathetic to like
um maybe like trying to get at our
preferences via
debate um and things of that sort
uh or or some of the amplification stuff
uh i
i do feel like um um
those those would be probably closer to
um what we might in the near future do
in in building up some kind of data set
from which we try to infer
what people's uh higher order
preferences are
[Music]
so so presumably you could do things
like that without
um without
doing mind crimes on these people you're
simulating
in in your uh finite human brain
um so so
it might be that um
when you scale this down to run with
actual finite
power that you want to do a lot of that
stuff but
i do kind of think it is important
that there's so that there'll be some
recognition that
basically i don't think we want to
define the ground truth in terms of
um in terms of what what comes out of
those those types of things i think
i i guess i have some worries there so
so so i kind of like
if if i could have these concepts
then then then maybe it can even take
over
uh creating new iterations of what are
the best
methods that
would actually be finite approximable to
this
but it kind of needs some definition of
this to know what is finitely
approximating
and of course as we're doing the
approximations we also
want to make sure that we're not having
it commit mind crimes as well
thank you okay um
so suppose that someone writes a
psychological paper or a philosophical
paper that is relevant to
sorting out people's preferences
how if your
ai has been launched already how do you
imagine it
taking into account this innovation
because one of the biggest difficulties
that i have
uh is to know how much has to be put in
by hand
and how much uh doesn't that's why i've
been looking at
things like dichronic consistency is
this
in the human head so we can delegate it
to
preferences or is it something that we
just want to impose from outside
so similarly so someone comes up with a
psycho
a paper about psychology that
illuminates some aspects of human
preferences and
this thing for argument's sake or or a
philosophical thought experiment
and we'll assume that this is relevant
for this how would
the ai in this
take that into account if this happened
during its run so
um let's see or i i mean
or you can have it published beforehand
if you want
um how would it take this data
that is already out there that is
relevant but not in an ai
parsable form
um let's see so
so um
i i i guess it kind of depends so so is
it like
is it is there a mechanism that
it's positive about how our preferences
work is like the content of that paper
let's say it's the anchoring bias paper
uh it's pointing out a particular bias
that just really realized was there
before
um but now they realize it's there and
they agree yeah this is a bad bias this
is not
a preference um
so this before this anchoring bias
was not known and we'll presume that we
haven't put it enough
for it to identify the anchoring bias
from other principles
but someone writes this paper and
there's a few people who agree
in the world oh yes that's a good idea
so
but how would this enter the algorithm
as a data um
so i mean the current version the
current version doesn't really
work off of data like that right the
current version
is just assuming you have a low-level
causal model of people's brains
and that's where they're that's where
they're getting the the data
um but we but we could talk about maybe
what adjustments
should be made to this algorithm when
you discover a paper like that
and and i do and i do want in the future
to have
the ai maybe have some self-modification
abilities
so so maybe we're talking about like
some future iteration
where where has um
some of those abilities to kind of take
info from a new paper and try to
integrate it with
um other information
um you wouldn't want it
you wouldn't want to hand specify take
this paper or papers on this archive or
thing
you want it to be able to take that kind
of data
understand it and like it might be a
conversation between two top
philosophers
that is relevant for
novel situations or something yeah i
mean
for something like like the rationality
scoring metric right you can make some
adjustments
like so so in general
we apply a principle of charity but if
we know for a fact that humans are very
prone to an
ancient bias maybe we don't penalize
attributing anchoring bias as much as
if it didn't fit some known pattern
right so that would be an example of
some way you could change some of the
scoring mechanisms here yeah but that's
doing it by hand
yeah yeah um yeah so that's what
that's why it's not directly applicable
to this version
but maybe you wanted to talk about some
future version
okay these are
some of the things i'm suggesting are
things to bear in mind
possibly yeah and
okay i think i have
two more questions slash
no three more questions slash comments
um
um yeah so um over to the
the uh the audience well i think uh i
said
is it an hour and a half to june and you
also and we are close to the one and a
half hour mark now
so uh if june if you need to leave or
anything then
please feel free but otherwise i think
uh we should give
stewards uh questions priority towards
the end
yeah okay so
the um the first one
is do you have a population ethics
or a way of solving that issue
um i guess i would i would file that
under um normative ethics
as opposed to meta ethics um so
i mean the reason i ask is because
you're getting the preferences from
brains um this is a certain
a certain population of brains that
could be variable
so how are you going to deal with the
issue of variable numbers
of brains uh
being created or destroyed oh i see
um yeah so i guess um in terms of what
i've coded here
i think i was i was trying to just
simplify it into just take all the adult
human brains at the point in time in
which we're
wondering whether to press the button to
let the let the ai go
um and that
and that presumably if
if uh it's able to get the
get the values that these humans should
have and
and and if these humans should value uh
future generations then
then you should get the result just just
by scanning the existing human sprains
um you should get the result that you
that it should value future humans
okay so this i'm putting in the category
of
uh delegating delegating to current
human
preferences to resolve
um yeah yeah current human preferences
but
idealized right yeah
[Music]
but uh yep that is uh
that's the way i tend to to go for a lot
of these things like especially issues
of identity
but the
other thing is have you thought of a
potential
error detection um
layer or as in
something that
a lot of our beliefs about human
preferences
and human and humans in general are
expressed
in your algorithm but we have some
judgments as this outcome is terrible or
this outcome is stupid
that are hard to capture
in this preference formalism
and could be more captured as
error checking uh i was wondering if
you would if you consider that this
might be
a way to go or some some sort of
catching
uh disastrous errors yeah yeah i mean
certainly for anything this abstract
you you would definitely want as much as
much testing as possible to validate it
um i i mean i don't know how
far i've gotten in actually
working out how you would go about doing
it but but i do think
that there there should be plenty of
behavior
if this was if this was going to be um
uh ever close to production
okay um
i'm i'm looking forward to to that
yeah i have thought a little bit of like
say like a shutdown button going
going back to the idea that that a lot
of metal philosophy
is meta ethics or can be
answered by meta ethics plus
meta-semantics so
so could you could you do something like
um
scan to take take people's brains and
and
and figure out what their concept of
ethics is
um or what what concept of theirs that's
closest to this meta ethics
is does it play the same role in their
deliberations
and then and try to apply it to
the whole algorithm and kind of ask
everybody
with their fate what yeah if we have a
way of
figuring out what the concept i think
various people have
does it actually match up with the meta
ethics that's been programmed into this
um
i was i was more sort of thinking along
the lines
of some form of compressed accurate
summary of where the ai wants to go
oh yeah the checking that humans are not
totally repulsed by that
um this would have been sort of a
separate
thing along the same lines i i've sent
you a somewhat silly example there
where i imagined a problem with cv
um and there the problem is that it
follows the coherent extrapolated
volitions things at every step arguably
but ends up in a terrible uh place
yeah um and
this a lot of what you're
doing seems as if there might be
um you could just think of sort of ultra
buddhist that ends up destroying the
world
um kind of through altruism
but
[Music]
it's hard to tell because of some of it
depends on some of how you
interpret distance between
um various utility functions and
idealization processes but it
seems that it may be vulnerable
to things like that series of rational
seeming steps that end up in a very bad
location
um and some sort of checking or overall
connection with the initial
starting point uh might be something
worth
thinking about yeah um i mean
i guess my theory is supposed to be able
to capture
pretty much any normal reasoning that
humans do
so so if you're able to write up this
example
and use it in an argument about what we
should value
then then theoretically uh
um my model should capture what's going
on
when you're doing that you have some
criteria that you're subjecting
your your uh values to um
and and and then we should be able to uh
tell if that criteria is being applied
correctly or not
and and that would uh this
sort of thing sorry
what do you think yeah so so basically
if you're right that that in that there
is that there is an argument here
um to um i don't know be
aware of like is it fair to say like
these
a chain of transitive uh
normal reasoning um
[Music]
yeah i don't know i don't know how to
summarize this very quickly but
um but but if you're if this is if this
constitutes
uh an argument about how we should
interpret our value
then then within my model
that should be capturable within um some
higher order decision criteria
um that that we could then apply
to get the result that you want this is
something that
could be tested to some extent
empirically because your
um your process is imagining idealized
versions of humans
um and a question
is is this unstable
or is this a stable construction
and if sort of meta
normed arguments like this are included
and have a strong effect i'd expect a
more stable
outcome where shifting around minor
preferences don't make much difference
but if it's unstable
then it could go in many it could
could end up in many areas yeah yeah i
mean
i guess i guess my model right now is
probably going to be agnostic on
exactly how you specify the sort of
initial conditions how you fill in
what the content of people's preferences
are so
so so probably there are some ways of
making a stable in some way that they
can't not say well
and that would be really good to know
what sorts of features in general
make it stable or not there is also um
i haven't been following it as closely
because i've
i'm i'm not really an academic
philosophy
technically anymore uh but but uh
there's a there's been interesting work
in um
experimental philosophy where their
whole idea is
philosopher talk all the time about oh
people have this intuition or that
intuition
uh using those in their arguments they
want to
go out and actually test do people have
those intuitions
how much do they have them is there
diversity in the people who have them
um and i saw people i haven't had a
chance to read the paper but
but uh just skimming it a little bit it
looked like that paper was finding that
there there actually are a lot of like
universalities
in um not necessarily like in the
end answers that they give but but in
the types of
intuitions that that they that they
bring up
uh or you know it it might be that
there's research showing that you could
frame the problem in certain ways
uh to get them to go one direction frame
the problem in a different way to get
them to go the other direction but that
just susceptibility to the framing seems
to be kind of universal
um so i know there's been there's been
some intriguing research that suggests
that
uh that there might be a lot of um
overlap within humans um when it comes
to
ethical decision criteria um and and
that would certainly be uh be better i
mean i think my model can work
even if that's not true even if you
think that there is
much more diversity um but but i do
think that uh
um it's a little bit easier
if if for the most part there's there is
broad overlap
in in what the content of these high
order norms are for
for actual humans
i agree with you and i think there is
quite a lot of overlap between humans
as well and um
i just want to sort of
okay i'll try and keep this brief so
part of what i was thinking
of why there was tests is to distinguish
moral moral systems that are defined by
a stopping point
by stopping conditions from those that
are
constructive from the basis
so if you um
like if you want to have sort of
coherence rational coherence between
your
your preferences and meta preferences
you can either do this sort of building
up
or you can sort of do it until you get
to a point where coherence
has been reached and the
do it until and cv is sort of an example
of do it until
do it until the conditions are reached
and they do it until
things seem to be very dangerous because
this might do a random walk
across all ethics because
all that it really cares about is the
stopping conditions
so that means that there are certain
attractors in ethical spaces and when it
hits one of them it stays there
but we don't know if it's going to hit
any one of them uh any good one
early and it might miss them all and
go out for something bad or and simple
yeah whereas the ones that are when i
was talking about checking
from the original thing to the end i was
sort of saying
ensure that there is not too much of a
distance in a sense
so ensure that it's constrained to build
up
uh rather than just um wandering
from this starting point until something
happened
yeah yeah and and i i do model some of
that uh i
i look at sort of a chain of agents
identity between
each continuation over a whole path and
and kind of we kind of want to ensure
um like not not just that um
the beginning and end uh have have
uh decently high agent
identity scores but but also that maybe
nowhere in the chain was it below a
certain threshold
um but but and i i also
wrote some notes to myself in the in the
code of like
um i think i think i probably had a very
crude stopping
point like like just stop that time
10 million or whatever um but
but obviously i'd like to get a more
principled one like is there some way
where we could
kind of ask these agents are you at a
stopping point
you know it seems like something like
that might might be the best way but
i just didn't want to get into that
complication for this version
i mean ask the agent are we at a
stopping point seems like continue
until this condition is reached
anyway um so i was wondering if we could
talk more about this
uh at some point maybe next week oh yeah
yeah that'd be great
cool we'll sort that out when when other
people
the other thing is that i would
encourage you to try and write it out
a lot of the ideas written not just
in the algorithmic form
mainly because i found it so useful
myself to have to formulate my ideas
uh in written form and try and get other
people to understand
them no matter how imperfect i am at
that
this tends to clarify a lot of things
when you
do the process
[Music]
yeah thanks for thanks for your talk oh
cool yeah i appreciate you being here
i would also like to say thank you very
much june for joining us today
it's been a pleasure and i've certainly
learned a lot and i think
everybody here has really enjoyed both
the discussion and your presentation