#58 Dr. Ben Goertzel - Artificial General Intelligence

[Music]
either people are really excited about
it or they're really terrified of it
those are the sort it seems to be the
two responses i've been working on ai
for three decades and i first started
thinking about ai when i was a little
kid in the
early
late 60s and early 70s when i saw
ais and robots on the original star trek
so i guess
i've had a lot of cycles to
process the
positives and and negatives of it
whereas now like
suddenly most of the world is thinking
through all this for the first for the
first time now we're at the time in
history when
a lot of these ideas
can be can be made real which is amazing
amazing and scary not only are we on the
cusp of creating a form of intelligence
going beyond ourselves but what kind of
intelligence it is
may depend on what the early stage ai's
we're building now
are are actually doing and how they're
organizing themselves and we should make
that as as beneficial as we can
he does it with a bit of a smile right i
should do that too good to go the
following is a conversation with ben
gertel who is one of the most
influential people in the field of
artificial general intelligence and
spearheads many famous projects such as
opencog singularitynet and sofia the
robot
we read the singularitynet paper by ben
and his collaborators and then listened
to ben's appearance on the excellent lex
friedman podcast and we knew we had to
talk to him
by the way if you are interested in
machine learning and ai we cannot
recommend the lex ridman podcast highly
enough that guy is great the field of
artificial intelligence was founded in
the mid 1950s with the aim of
constructing
thinking machines
that is to say computer systems with
human-like general intelligence
think of humanoid robots that not only
look but act and think with intelligence
equal to and ultimately greater than
that of human beings
in the intervening years though the
field has drifted far from its ambitious
old-fashioned roots we should again be
pursuing artificial general intelligence
as a distinct effort from our current
narrow approaches
too many people in the ai community
still succumb to the first step fallacy
which is believing that incrementally
improving current ai systems will lead
towards artificial general intelligence
admittedly we're still waiting on the ai
community to converge on an idea of what
a good agi architecture might look like
and indeed convincing evidence that such
an architecture performs better than a
narrow ai system
dr ben gertzel is an artificial
intelligence researcher ceo and founder
of singularitynet a project combining
artificial intelligence and blockchain
together
suppose my good friend ray kurzweil is
correct that 2029 will get ai's as
smartest people
by 2045 we have ais boundlessly more
intelligent than than people
we're going to find that any ai that has
human level intelligence
is going to have some
species of advanced reflective
consciousness but i mean i'm open to
other possibilities humans are distinct
individuals with our brains trapped in
our heads and communication only by
language which is very low bandwidth
whereas
ai's can swap brain matter whenever they
want to they can join into borg minds
and then subdivide whenever they want to
so it's going to be more like
one big ai mind network
rather than a bunch of ais in in
in competition it seems very plausible
within a few years
of getting to that human level ai
it's going to invent all manner of
amazing new things ben seeks to fulfill
the original ambitions of the field
which is to actually create artificial
general intelligence he graduated with a
phd in mathematics from temple
university in 1990 and ben's approach to
agi over many decades now has been
inspired by many disciplines but in
particular in my opinion from human
cognitive psychology and computer
science
now today
ben's work has been mostly theoretically
driven he thinks that most of the deep
learning approaches to agi today try to
model the brain
there might be a loose analogy to human
neuroscience but they've not really
tried to derive the details of an agi
architecture from an overall conception
of what a mind is
ben thinks that what matters for
creating human level or greater
intelligence is having the right
information processing architecture not
the underlying mechanics by which the
architecture is implemented
now ben thinks there's a certain set of
key cognitive processes and interactions
that an agi system must implement
explicitly such as working in long-term
memory deliberative and reactive
processing perception action
reinforcement learning metacognition and
several others
ben thinks that biological systems tend
to be messy
complex and integrative
searching for a single algorithm of
general intelligence is an inappropriate
attempt to project the aesthetics of
physics or theoretical computer science
onto a qualitatively different domain
now on the introduction to the show i
slightly offended ben by characterizing
him as a go-fi researcher it certainly
wasn't my intention to offend him and i
don't see go fight as a pejorative term
at all i mean
i actually meant that i thought he was a
symbolist which is somewhat synonymous
with go phi in my mind but having
studied more of ben's work since it's
pretty clear to me that he's not a pure
symbolist he's clearly is a little bit
but he's in the rare camp of being
sub-symbolic on a discrete knowledge
graph now sub-symbolic it just means the
abstract concept should be distributed
or sliced over your knowledge substrate
so it's possible to be sub-symbolic even
without using artificial neural networks
so this means by the way that jeff
hawkins hierarchical temporal memory his
htm algorithm is also a sub-symbolic
system
even though the substrate itself is
discrete
now
mlst is really big on diversity of
thought in ai and we focused a lot on
modern variants of gophi or modern day
hybrid symbolism recently as well as not
liking the old-fashioned part let's face
it i mean all ai is old-fashioned right
ben associated the term with the idea
that knowledge should be handcrafted by
humans and his mind went straight to the
disastrous psych project the largest
ever knowledge graph created by humans
and indeed the knowledge acquisition
bottleneck was probably the most
significant failure of expert systems in
the 1980s
actually most symbolists i know today
and even many back then didn't believe
in handcrafted knowledge stores i mean
symbolists say that knowledge should be
acquired through reasoning deduction or
instruction and learning in a few cases
most agree that knowledge bases should
only be seeded with a small amount of
core knowledge
ben agrees with rich sutton that the
knowledge cannot be handcrafted but
sutton also meant in his bitter lesson
essay that the modular cognitive
architecture of an ai system similarly
shouldn't be designed by a human
this is where they diverge
sutton thinks that we should trust in
the blank slate and empiricist models to
discover the optimal structure and
architecture
ben on the other hand thinks we should
design a modular cognitive architecture
using the knowledge that we've gleaned
from cognitive psychology
so i said to ben at the time that i
project all ai researchers onto a kind
of
map or a vector space you know which
represents their research interests and
i had placed ben onto that map on the
basis of the podcast that i'd listened
to in the papers that i have read from
him um in that map he was closest to
marvin minsky and gary marcus two people
i admire greatly by the way and this is
because i believe ben's work is strongly
influenced by cognitive psychology
formal logic and the belief that mind
should be a hybrid of symbolic and
sub-symbolic
ben also thinks that cognitive
architecture should be hand crafted with
designs inspired from our knowledge of
human cognition and also they should be
strongly modular he thinks that the
underlying knowledge representation
should be multi-modal and distributed
composable and captured in a discrete
graph
so ben thinks that we should cede the
knowledge graph to some extent but it
should be able to acquire most of its
knowledge from reasoning imagination
experience and display strongly emerging
characteristics so my map of ai
researchers is probably wrong in many
ways but we all have an internal mental
map that we use and update all the time
i mean in my map i think of empiricists
like yan lacoon or logic philosophy and
formal language folks like john mccarthy
or psychology folks like gary marcus and
even linguistics and cognitive science
folks like chomsky and hofstadter
some folks like john certal and
j mark bishop who we had on the show
don't think that ai could even exist at
all
after the show i asked ben what he
thought of my map and he said he wasn't
too impressed actually he quickly
responded with his own better mental
framework he said that his dimensions
were neuroscience
biology more broadly so things like
evolving ecosystems or maybe even things
like evolutionary algorithms or
open-endedness um the general math
theory of intelligence so things like
aixi
formal logic cognitive science and
psychology
um you know thinking of the human mind
distinct from the human brain embodied
action so the kind of thing that rodney
brooks would talk about in his
subsumption robotics
complex systems and non-linear dynamics
the philosophy of mind general systems
theory and also some of the applied
empirical work so the kind of stuff that
we might be doing in in robotics or
automated theorem proving
he gave deep mind as an example so he
said that they were motivated by the
intersection of neuroscience and
empirical applied work and also the
general mathematical theory of
intelligence
so
ben actually published a paper in 2014
artificial general intelligence concept
state of the art and future prospects
where he boiled down the spectrum of agi
designs into four
even simpler categories right so
symbolic
emergentist
hybrid and universalist
agi theory is a patchwork of overlapping
concepts and frameworks and hypotheses
often synergistic but sometimes
completely contradictory so in this
paper ben made some really insightful
remarks about the nature of general
intelligence that it should be able to
achieve a variety of goals and carry out
a variety of tasks in a variety of
different contexts and environments
that it should be able to handle novel
problems and situations unanticipated by
its creators it should be able to
transfer knowledge from one problem or
context into another and that
arbitrarily general intelligence may not
be possible given realistic resource
constraints
he also pointed out that learning some
tasks might be more efficient than
others and that humans display a higher
level general intelligence than existing
ai programs and finally it seems quite
unlikely that humans happen to manifest
a maximal level of general intelligence
so in the paper ben quickly introduced
his core agi hypothesis which is that
the creation of a general intelligence
won't be on a continuum with narrow
intelligences that it's an entirely
different category although he does
think there's some overlap
ben said in the paper that there were
three fundamental ways of characterizing
general intelligence there's a
mathematical approach an adaptionist
approach and an embodied approach
the mathematical approach was defined by
legend hutter with aixa and later their
so-called universal intelligence
conception of a computer program
executed by an agent achieving the best
average reward calculated over all
possible environments preferentially
weighted to prefer the simplest programs
this kind of measure makes humans look
quite dumb and systems like alphago look
quite smart
legend hutter argued that intelligence
is purely a matter of capability and
behavior and independent of how much
effort is expended the adaptionist
approach was defined by pei wang and
argued that artificial intelligence
should be seen as adaptation to the
environment given limited resources ben
released a paper in 2010 modifying
legend hutter's universal intelligence
paper to account for the factors which
wang raised wang believed that the
essence of general intelligence laid in
the complex compromises needed to
achieve generality using limited
computational resources
the embodiment approach was similar to
the adaptionist approach but held the
contention that intelligence is
something that physical bodies do in
physical environments rodney brooks is
the best known advocate for this
perspective ben cited another paper from
pfeiffer and bernard to summarize this
view they said that the study of
intelligence always seemed to boil down
to compliance and diversity
intelligent agents always comply with
the physical and social rules of their
environment and exploit those rules to
produce diverse behavior
animals and humans have gravity and
friction to contend with advocates of
the embodiment tradition argue that it
makes sense to think about achieving
human-like intelligence the same way
evolution did it which is to say in the
context of controlling a body with
complex senses and actuators in a
complex physical world
now let's get to the punchline do you
remember the four categories of agi
researchers that i promised to tell you
about well symbolic agi advocates argue
that symbolic thought is what most
strongly distinguishes humans from other
animals it's the crux of human general
intelligence what allows us to
generalize so broadly the idea is that
minds manipulate symbols representing
aspects of the world or themselves
generally these architectures have an
explicit knowledge base they have short
and long-term memory banks and they have
the explicit notion of perception
cognition and action
ben points out that symbolic
architectures tend to be quite weak in
learning creativity and procedure
learning and even episodic and
associative memory
ben also points out that the line
between symbolic and connectionism gets
a little bit blurry with some algorithms
because they produce massive and
incomprehensible symbol networks which
are not too dissimilar to connectionist
models in many ways
ben also argues that a purely symbolist
architecture seems to be incapable of
giving rise to the emergent structures
and dynamics required to yield
human-like general intelligence
the emergentist agi school of thought
expects abstract symbol processing to
emerge from low-level sub-symbolic
dynamics perhaps a neural network or
even a real brain
the human brain consists of a large set
of simple elements complexly
self-organizing into dynamical
structures in response to the body's
experience
ben says that no one has yet
compellingly showed how to achieve
high-level functions such as abstract
reasoning or complex language processing
using a purely sub-symbolic emergentist
approach um a pretty strong example of
the emergentist approach is the
cognitive neuroscience folks right so
people like jeff hawkins
the brain is the only example that we
have of a system with a high level of
general intelligence so emulating the
brain seems like the most
straightforward path to achieve agi
so jeff hawkins brain inspired
hierarchical temporal memory or htm
algorithm which we'll cover on next
week's show immediately springs to mind
and there have been many large-scale
brain stimulation projects we actually
interviewed dr simon stringer recently
who's building a brain inspired and
biologically plausible neural network at
oxford university even regular deep
learning folks they view their
architectures as a mergentist hoping
that scaling will lead to these kind of
complex dynamics and emergent symbol
processing
ben says that the brain's cognitive
mechanisms are well tuned to run
efficiently on neural neural wet wear
but are poorly suited for our
computational hardware today
now onto hybrid agi architectures ben
would consider himself a hybrid agi
researcher right so advocates of hybrid
architectures they make the case that
the whole is greater than the sum of its
parts i mean ben captures this idea with
the phrase cognitive synergy which we'll
discuss at great length on the show
today but ben's open cog system is one
such hybrid agi architecture most hybrid
architectures are strongly modular in
some sense but that doesn't mean that we
should think of the individual
components as being entirely
encapsulated black boxes the levels of
interaction depth between the modules
actually vary depending on different
architectural patterns
ben thinks the design goal of a hybrid
system is achieved when different parts
work together synergistically each
module contributing its strengths to
help others overcome their weaknesses on
the flip side though when you try to tie
together qualitatively different modules
you get a brittle system that can't
adapt very well because the different
components can't work together with full
flexibility
now the other week we spoke about
neurosymbolic approaches to ai with
professor gary marcus and professor
lewis lam and when ben talks about
hybrid systems i think he's mostly
referring to their strong modularity and
synergistic potential rather than the
kind of dichotomy of symbolic and
sub-symbolic knowledge representation
finally ben says there are universalist
approaches to agi which are distinct
from the other three it's the idea that
agi algorithms would yield incredibly
powerful general intelligence if only we
supplied them with infinite amounts of
computing power
advocates of this approach think that we
should start at this level of
abstraction and kind of work downwards
rather than the somewhat bottom-up
approaches that we just spoke about now
these approaches can be traced back to
solomonov's theory of induction but
perhaps these days are mostly thought of
in respect of marcus hutter's aixi
system
hutter thought that an agi system would
be controlled by some program and we
could search through the space of
possible programs to find the one that
would have worked best in all the
previous situations that we encountered
and then we could just rinse and repeat
this process with every new situation we
found ourselves in
i mean there's a bit of a no lunch
caveat there of course and that clearly
it will depend on some sense on the
situation space that we've experienced
rather than having kind of general
utility but it would be provably optimal
for the situations that we have been in
it gets a little bit muddy though as it
requires the notion of an arbitrary
reward function
which is linked to the environment
by the way uh schmidt huber's godel
machine from 2006 was conceptually
similar and ben even pointed out in the
paper that achieving general
intelligence with infinite computing
power it's just a mathematical game at
the end of the day i mean it's only
minimally relevant to achieving agi
using realistic amounts of resources so
when we say go phi on mlst we don't
necessarily mean the ai zeitgeist of the
1980s we mean modern day symbolists you
know almost all of which advocate for a
hybrid approach
the ideas are evolving all the time
right the school of thought has recently
been branded
neurosymbolic or the third wave right
most of these folks now embrace using
neural networks as perception modules
and the need to have empirical learning
in addition to logical reasoning their
mindset is in my opinion most
characterized by the belief that ai
systems should be strongly modular that
knowledge should be represented
explicitly and discreetly or at least
mostly discreetly usually in some kind
of graph and they advocate for the
primacy of logic based reasoning for
knowledge acquisition
it's worth also noting that ben has been
pretty harshly critical of neural
networks i mean in particular gbt3 and
its apparent lack of understanding
ben frequently cites marvin minsky's
society of mind as one of his
inspirations how can intelligence emerge
from non-intelligence well marvin minsky
thought that it was possible to build a
mind from many small mindless parts
one possibility i've been working toward
a lot lately is what ai pioneer marvin
minsky had called the
society of minds which is also an
economy of minds the idea that it may
not be one algorithm written by one guy
or one program written by one company
that gives the breakthrough to general
intelligence and then super intelligence
it may be a network of different ais
each doing different things specializing
on certain kinds of problems some of
them specializing in generalization and
abstraction and maybe this network of
ais cooperating together forming sort of
emergent intelligence maybe how we get
the breakthrough you could think of this
as a as a global brain i think it's
almost inevitable by this point
that humanity is going to create
synthetic intelligences
with tremendously greater general
intelligence
and practical capability
than human beings have i mean i think i
know how to do that with the software
i'm working on with my own team
but if we fail you know there's a load
of other teams who i think are a bit
behind us but they're going in the same
direction though right you guys feel
like you're at the tip of the spear with
this stuff i do we use a sort of neural
symbolic knowledge graph use neural nets
to recognize sensory data and you use a
sort of probabilistic logic for
abstract reasoning you use simulations
of evolution processes to come up with
new ideas but you know whether it's a
deep neural net or an open cog system or
one of the other many approaches being
pioneered right now what's cool is
there's
a lot of
research teams working full-time on not
just building
narrow ai applications doing specific
things but trying to break through the
artificial general intelligence open cog
is a project that aims to build an open
source artificial intelligence framework
it defines a set of interacting
components designed to give rise to
human equivalent artificial general
intelligence as an emergent phenomenon
of the whole system
open cog is a software framework which
hosts multi-modal cognitive processes on
top of a common representational
substrate where in plain english a
common knowledge graph which has been
implemented as a graph database
the cognitive processes could in theory
be anything but we're talking about
things like a discrete program search or
evolutionary algorithms or neural
networks or probabilistic logic
programming and so on
you know this is a remarkable feature of
opencog that it provides a unifying
substrate for a whole range of
relational or logical analysis
it opens up the possibility for true
hybrid analysis from the bottom up and
the top down
open cog could in theory represent both
neural and symbolic operations and then
map logical reasoning and inference
between them
so for example
suppose we're training two natural
language models in this framework you
know one classical
computational linguistic style model and
the other one could be a kind of set of
modern transformer type models an open
cog agent could compare the metagraphs
back and forth between the the methods
and search for sub graph fuzzy
isomorphisms and merge edges and weights
between them i mean this has got the
potential to guide the optimization
procedure unblocking the models from
each other and possibly even
sidestepping some of the exponential
blow-ups which would have otherwise
occurred so you know in the most basic
sense ben thinks that we can transfer
learning between different modalities of
analysis right he thinks that we can
effectively white box every type of
analysis and represent it topologically
i think ben's core vision is that
there's some kind of topological
similarities between all knowledge
representations
and interesting things might emerge from
them when we start to bind them together
and form increasingly abstract knowledge
representations so
this unified architecture i mean it has
another advantage in my opinion it
allows for decentralized agents
operating on a common knowledge platform
i mean imagine an agent which was
responsible for encoding i don't know a
mathematical theorem wandering around
the metagraph finding subgraphs that
match the theorem and then adding the
implied links or maybe a machine
learning agent that encodes empirical
statistical relationships between
word phrases and then adding weighted
association links to matching phrases so
overall open cog presents a highly
dynamic almost organic platform for
evolving knowledge and algorithms i mean
it's easy to see
ben's optimism that agi will emerge from
from this approach i mean we're a little
bit more skeptical whether it will in
practice
it all comes down to whether the faith
in such emergence
is just that is it just a faith or could
it actually happen
so
ben is also the creator of
singularitynet a decentralized open api
marketplace for ais built on blockchain
the implications are that it could let
anyone monetize ai yannick made a video
about ben's singularity net a while back
so please watch that i'll put a link in
the description so singularitynet is a
as it says a global a marketplace but it
is also kind of an effort it is a
foundation it has blockchain in it um it
has ai in it it has symbolic computation
it has graphs it has all the things all
the buzzwords you could possibly
want so the high level summary of this
system is that it is a marketplace uh
for apis basically on blockchain where
either humans or apis can call other
apis
and pay them for that service and the
goal is to sort of get a network going
of apis that call apis that call apis
and um
sort of have that built into a global ai
not only marketplace but
like as itself a global ai this is
backed by the singularitynet foundation
and they do a whole bunch of development
of the platform but also research on the
platform i'll put a link in the
description but he did a pretty good job
of walking through their recent white
paper just before we kick off as well
make sure you check out ben's new paper
that he released very recently it's
called the general theory of general
intelligence a pragmatic patternist
perspective
the paternalist philosophy of mind is a
general approach to thinking about
intelligent systems based on the very
simple premise that a mind is made of a
pattern
so a mind is a system for recognizing
patterns in itself and the world
critically including patterns regarding
which procedures are likely to lead to
the achievement of which goals in which
contexts
intelligence can be partially conceived
in this framework he says as the ability
to achieve complex goals in complex
environments where complexity itself may
be defined as the possession of a rich
variety of patterns
a mind is thus a collection of patterns
that is associated with a persistent
dynamical process that achieves highly
patterned goals in highly patterned
environments so i mean i've long been a
an advocate of a philosophy i think of
as as patternism like it's the pattern
of organization
that appears to be the the critical
thing and the
you know the individual
cells and going down further like the
molecules and particles in our body are
are turning over all the time so it's
not the specific
combination of elementary particles
which makes me who i am or makes you who
you are it's a pattern by which they're
organized and the patterns by which they
change
over time so i mean if we can create
digital systems or quantum computers or
femto computers or whatever it is
manifesting the patterns of organization
that constitute intelligence i mean then
then there you are there there's
intelligence right so that that's not to
say that you know consciousness and
experience is just about
patterns of organization there may be
more dimensions to it but when
when you look at what constitutes
intelligence thinking cognition problem
solving you know it's the pattern of
organization not not the specific
material as as
as far as we can tell so he talks about
some of the key drivers to these
dynamics things like evolution and
autopiosis
and association and self-structure and
then he goes on to talk a lot about
cognitive synergy again um anyway it's a
really interesting paper admittedly it's
got a lot of technical jargon in it it's
quite long but i really do recommend you
have a look at that because it's
probably the
most up-to-date thinking from ben anyway
um we had a really really good
conversation with ben and i really hope
you enjoy the conversation as much as we
did and um yeah i hope i didn't piss ben
off too much by calling him a go-fi
researcher i'm sorry ben take it back
uh enjoy the show peace out dr gertzel
welcome to mlst we are so honored to
have you here you're truly an enigma to
me i think you're a rare breed you seem
like one of the last standing gophi
researchers but unlike all of the other
ones you're well funded and to some
extent you've actually realized minsky's
vision now gopher advocates today talk
about the need to represent different
types of knowledge declarative
procedural episodic sensory
intentional and attentional this
knowledge is the dark matter of the
universe of ai but this is what you've
already done in opencog
you've been an inspirational figure in
the ai community for decades now you are
walking talking polymath and oracle
keith remarked that he couldn't believe
how much knowledge could reside in a
single brain
you believe that the creation of
advanced agi systems is mostly an
engineering endeavor which will require
significant input from science and
mathematics and also philosophy
also i think it's fair to say that there
are more complicated sounding words in
your papers than i've seen anywhere else
and you're also an aficionado of
extremely long bulleted lists
so anyway on to my first question on
your open cog page you said that the
secret source of agi is cognitive
synergy you say that it's important that
cognitive processes associated with
different types of memory can appeal to
each other for assistance in overcoming
bottlenecks the human brain appears to
be an assemblage of diverse structures
and dynamics built using common
components and arranged according to a
sensible cognitive architecture you say
that their close interoperation gives
rise to the overall systemic behaviors
that characterize human-like general
intelligence
the components richly and dynamically
support and assist each other and
emergent structures and dynamics quickly
appear now this leads to your core
hypothesis which is that integrating
multiple symbolic and sub-symbolic
learning and memory components in an
appropriate cognitive architecture and
environment
can yield robust human-like intelligence
in some sense it seems like a reasonable
thing to say i mean if you look at a
typical corporation few would deny that
a gestalt emergent intelligence forms on
top of the society of individuals in
that corporation but the key question is
why
i can intuitively understand why a
superintelligence emerges in a
corporation or the human brain for that
matter but i cannot intuit why a
superintelligence would emerge from
singularitynet so what's your take on it
ben
so
first of all i i have to admit your
introduction
got my dander up a little bit because i
totally don't think of myself as a go-fi
researcher i mean the
the reason i didn't do a phd in ai
back in the in the 80s when i did my phd
in mathematics was precisely that
the good old-fashioned ai that was
dominating the ai field at that time
seemed to me to have nothing to do with
with intelligence in any way i was i was
interested in it
i actually i tried to do a phd thesis on
prediction of multiple coupled time
series
using asymmetric hopfield nets basically
which would now be like a plain vanilla
masters degree or undergraduate course
project right and
no one would do it because it was too
like
neural netty and and uh soft and and and
weird right so i
i never felt any affection
for what i took as the primary
characteristics of gofi
which were
you know the idea that the
knowledge you could feed into an ai
system should be fed in by human beings
coding a bunch of of expert rules in
some knowledge base like the psych
project with trump tried to do and then
others before like
that always seemed insane to me so i
mean i
i've
always been a huge fan of experiential
interactive learning of the idea that to
get an agi system has to
recognize patterns in itself and and its
its environment and figure it out as as
it as it goes right and right in in that
sense i felt the transition from
the expert system methodology which was
a key part of gophi back in the actual
good old-fashioned days
the step from that to
supervised learning was
a smaller step
than was commonly
understood because you're going from
hand coding a bunch of rules to
handle labeling or hand categorizing a
bunch of a bunch of
of data which is
in many cases
a step forward but it's still a far cry
from this sort of you know completely
unstructured experiential learning that
that
that one one might seek and then
of course
now
the ml community
is significantly going beyond
traditional supervised learning you have
unsupervised unsupervised
semi-supervised learning and
reinforcement starts to rise to the fore
so you're the mainstream of ai is
getting closer and closer to uh to
experiential and
interactive learning
but uh it's still not really there
any more than than gopher was there back
in in the in the 70s already this brings
us
to
cognitive synergy
in that
the
need for cognitive synergy in an agi
system
really comes out of the need to do
experiential interactive learning which
involves
achieving complex goals in a complex
environment
under
fairly strictly limited
resources so if you
you know if you don't have limitations
on compute resources you can argue the
point in various ways but there's
there's well-known work by marcus huder
in his book universal ai with jurgen
schmidt uber with this paper on girdle
machine i mean
there's at least some interesting
theoretical arguments that with
supermassive compute power
you could make systems that solved all
sorts of incredible complex problems in
complex environments with
really really really really simple
algorithms it could be like a dozen
lines of lisp code or something right
but when you get into the sort of
situation that human brains seem to
evolve for
or into the sort of situation that we
have when building
actual systems now i mean you actual ai
or protoc systems out i mean you're
you're in this situation where
processing power and memory
are not anywhere close to what you would
ideally like them to be given the
complexity of the
of the of the tasks that you have
and that
that logically puts you in in a
situation where you know if the multiple
sorts of things that ai needs to do can
overlap or reinforce each other in some
way
you're going to make better use of those
limited resources right i mean that's
that very very simple line of reasoning
is the crux of cognitive synergy but
then
then you need to get into what are the
types of knowledge and learning
involved in making an a
human-like agi system or
for that matter a transhuman agi system
which one would suppose would include
even more kinds of knowledge and
learning than the human-like agi system
and there you get into like cognitive
science and what are the
what are the kinds of knowledge that
human mind has
what kinds of learning are associated
with those kinds of
of knowledge which leads you onto the
particular ways that
cognitive synergy manifests itself in an
agi design like
open cog which sort of winds you back to
good old-fashioned ai and neural
symbolic ai because if you look at
sensory or perceptual knowledge as one
of the kinds of knowledge
that the human brain evolved to deal
with for the kinds of tasks and
environments that had to deal with
it's clear that
modern deep neural nets are one type of
system that's really good at pattern
recognition and huge amounts of sensory
knowledge
if you look at declarative knowledge
like facts and beliefs including more
abstract ones such as have been explored
in in the logic field
it's pretty clear to me that uncertain
reasoning engines and and various sorts
of logic formalisms provide a good way
to deal with this sort of abstract
knowledge so the cognitive synergy
approach would say
in that particular example take a neural
system or something like that for
dealing with masses of sensory knowledge
take something that's doing something
like a logic system i mean it could be
an appropriately structured neural net
or it could be a it could be a full-on
logic engine to do with the declarative
and then semantic knowledge but then
how do you cover how do you couple them
together so that you have a sort of
perception
cognition feedback that's not just two
black boxes right but but the the the
two components are
sensitive to each other's
intermediate states while they're going
through competition so they can help
each other cut through combinatorial
explosions so that's
that's one example sensory processing
with neural nets and declarative
knowledge processing with logic engines
i don't want to go through all the all
the examples now because that would make
this this answer super long but that
should give you a flip that's that's
fine and i just wanted to um clear up
the bit at the beginning so first of all
when i say a go-fi person i don't mean
that in any way pejoratively we um
we are really interested in gopher in
this channel we have people like gary
marcus on but i i i i grew up thinking
go if i wasn't wasn't in seoul well no
no no
and i project them into a kind of vector
space in my mind you know um pedro
domingo says that there's five tribes in
ai and you know you see evolutionary
people and kind of logic based people
and these kind of empirical people so
the reason why i placed you there in the
vector space is you believe in the
primacy of modularization and the
society of mind the primacy of knowledge
and also the system should be discreet
first i agree with you 100 that uh
knowledge shouldn't be handcrafted by
the way and rich sutton i mean we're
completely aligned on that but but you
can see why on the basis of those things
because my
my view of it would be that
declare that verse semantic knowledge is
very effectively modeled
as
logical propositions and in some logic
which may be an
uncertain consistent intuitionistic
logic and not the kinds of logic that
that people are most commonly dealing
with but
i don't think that say the
the knowledge of what this coffee cup
looks like is effectively well captured
as a set of
of logical propositions it's pretty
clear that
some vec some continuous variable type
model
is is more effective at capturing my
sensory knowledge of what this what this
coffee cup looks like
can i i just append a little bit on that
point because you made a good
good point in that we have these all
these different system different types
of knowledge um some of them may be well
captured by some continuous systems some
may be well captured by discrete systems
they have to interconnect somewhere now
if you look at let's say the current
trend the current type the
i think that most people will go about
this con and say of course the space in
which they connect should be some sort
of a vector space should be some sort of
a distributed thing where we do
you know inner products to search for
similarities and whatnot yet if i
understand you correctly you've made
repeatedly the case that good ways of
interconnecting these systems is
actually through graphs and and through
through sort of graphs on graphs or or
hyper graphs and uh could you
uh maybe go into that a little bit why
do you think that
graphs specifically are good
representations
or or why should graphs be sort of the
the fundamental base
where these different processes
interconnect if i understood you
correctly in that
yeah i've been an advocate of graph
hyper graph and
metagraph
representations for
agi systems and and their ii systems for
a long time before
it became
fashionable with with uh things like uh
neo4j and and gracken and
and so forth i would say that
the distinction of
graphs versus vectors
sort of
points you in the wrong direction in the
sense that i mean
you can represent a graph as a bunch of
vectors as an adjacency matrix or
something and you
you can you can also project a bunch of
vectors into
into into a into a graph if you want to
so but that
the re the real question is what
operations
need to be efficient on this on this set
of of knowledge right and
i mean
most neural net algorithms in in
common commercial use today
they're based on on
matrix math right i mean they're based
on linear operations you're doing inner
products as you mentioned you're doing
you're doing some dimension reductions
and and
then then you're you're doing uh
matrix matrix multiplying and so forth
so i i think that
i i don't believe
that these sorts
of
linear operations on high dimensional
vector spaces
are the right algebra for most of what
what general intelligence needs to do i
i mean i used to teach computer graphics
when i was in a much earlier phase of my
career and which is where the
processing tools now used for ai come
from with
believe it or not young people may not
realize it today but gpus used to be
used for graphics not just for machine
learning right and so i mean
what you see in computer graphics is the
value of vector operations for dealing
with with images and and and and visual
data
and you have something similar in in
audio data so i i think
these linear operations
are more appropriate for this
sensory data
which makes total sense because this is
data coming from
like the 4d
space-time continuum which ha which has
all these linear
symmetries for the same reason that
matrix symmetry groups play such a role
in theoretical physics right and
i think when you get on to
more abstract
cognitive operations or to
procedural knowledge say to do do
complex motor coordination or something
i think these linear algebra operations
uh
are less less critical not not that
they're useless but i think they play a
less central
role and
you know uh
when i
early in my career
declined to do a phd in ai because i
didn't like the sort of hand-coded
expert production system direction that
the ai field was sort of
hypnotized by that at that point in time
what i got deeply into was non-linear
dynamics and chaos theory and and and
self-organizing systems
and i feel like
that's something that's
very much missing in the the modern
approach to
ai with deep neural networks so when i
when i taught neural networks
in the mid 90s when i found co-founded
the cognitive science program at
university of western australia where as
it were as a professor
i mean we had a big focus on
we we we did room of heart and mclellan
stuff with uh like parallel distributed
processing and
multi-layer backdrop which were deep
neural networks right but
but we did
hot field networks and asymmetrical hot
field networks that can organize into
strange attractors recognizing patterns
and data and how to do things and
it seems like this
non-linear dynamical aspect
is not yet been
relearned or regained within the within
the
neural net paradigm as it as it's become
mainstream
but it ties in with why
why graphs are useful because
if you have a complex body of knowledge
and you have a bunch of non-linear
transformations
that you you want to carry out perhaps
in an asynchronous parallel way
on on this large body of knowledge
a graph or better yet a hypergraph or a
metagraph are a
relatively concise and efficient way
to organize
knowledge
that will allow you to efficiently
execute a variety of these
non-linear operations it's quite
different than wanting to multiply a
matrix
a matrix by a vector although you could
convert back and forth between the graph
and vector representations
if you want to i have a question about
that um so
let's for the moment suppose that that's
all correct mathematically i'm sure it
is that a the hyper graph is a very good
implementation substrate for efficient
you know calculations of that of that
sort
is it fair to say that the the brain
because in the brain you know when a
neuron
pulses that pulse will travel to all the
connected you know downstream neurons so
that that's effectively a hyperedge
right and at the same time
axons can sometimes connect back to say
long dendrites which are themselves
a link and so it can kind of mediate
that that link so is it fair to say that
the brain does implement at that level a
hyper graph
in a way dendrodendritic connections are
like hypergraph connections i mean you
have a connection going to a dendrite
which modulates what flows on that
dendrite but
i wouldn't want to
over state that too much because i guess
i i tend to think the brain
is a quite complex continuous
variable quantum chemical system that we
that we barely understand and you i mean
you have
glia doing a lot of stuff in memory
you you you know if you look at eugene
izzakevich who you should interview if
you haven't eugene is kevin she's now
running brain corporation i mean
he had a book
decades ago on the geometry of the
neuron where he looked at the
nonlinear dynamics of flow through
through a neuron and he showed that even
when you have a network of neurons
stimulating hippocampus even if none of
the neurons fire
you know that the slow wave dynamic from
the hodgkin-huxley equation causes like
phase coordination in in the whole
network right so there's
then there's charge diffusion through
the extracellular matrix through just
these water mega molecules floating
around in the brain then
what role does that play in
say solving the binding problem from
cognitive science or in
in unifying parts of the brain into a
what's reported as a state of
consciousness i mean i i feel like uh
yeah definitely there's some hypergraph
structure in the brain and henry marker
and published a paper showing this funky
higher dimensional algebraic topological
structure in the brain so that
but there's probably a lot a lot of
other
totally different funky stuff going on
the brain that we haven't yet
yet penetrated to and that's it
that that's all really cool like i mean
much earlier in my career i thought like
should i try to
figure out how to build a mind just by
semi-ad hoc integrating knowledge from
all different areas of intellectual
pursuit
should i try to make a cosmic
mathematical theory of how intelligence
works
or should i just invent a better brain
imager so we can like we can measure
what the hell is actually happening in
here which may bear no resemblance to
what
current neuroscience thinks because
i remember in the 90s when i was in
cognitive science department arguing
with the neuroscientists
who didn't believe in neuro in
synaptogenesis or neurogenesis in adult
brains you're like no this doesn't
happen and now
you know not now now those things happen
but there's a lot of other stuff that
that
they say just can't happen and we just
don't know so that's
i'm
i'm more bullish on modern neural nets
just as
cool algorithms to do stuff than as
having too much to do with how the brain
works which is
which is fine i mean an airplane doesn't
have to
to flap its wings to be good flying
either sure but but um as you kind of
pointed out earlier we can often get
extremely valuable
inspiration and insight from looking at
the sort of only known generally
intelligent if we want to call it that
you know system that we have and you
mentioned one earlier which is in your
view
you know the kind of deep learning
paradigms right now have lost the
complex non-linear sort of temporal
dynamics that take place in the brain so
i'm just wondering if if you had to
choose a handful of say abstract
properties of the human brain one two or
three things that you think we need to
fold back into
um
the modern paradigms to make
significantly more progress what would
it be i like what's missing right now
would it be those complex nonlinear
temporal dynamics or some other features
or well an awful lot of things are
almost i mean if you're saying what
what's what's missing from modern
neural net architectures almost every
yeah what's missing but the top the top
things that are known in the human brain
i mean i think
so the
deep neural networks that are most
commonly used for sensory data
processing
are
you could view as a crude model of the
feed forward processing a visual or
auditory
cortex and
they capture pretty well what happens
within the first
half second of visual or auditory
perception
then when something takes you longer to
sort of scrutinize and figure out what
it is you're looking at what's happening
there
vision is feeding through the cognition
cognition is speeding back to vision and
you have a nonlinear feedback dynamic
there and i don't think we're really
capturing that in the neural networks
used for sensory data processing and
instead
instead we're going in a different
direction and we're feeding
massively more data than any individual
human has seen into into an algorithm
so that the feed forward processing
can do in these algorithms what the
human brain needs needs feedback
processing to to to achieve right so it
started out as a model of feed forward
processing of visual and secondarily
auditory cortex and and then
then it went in a
it went in a different direction
based on exploiting
what was possible with
with data sets and and hardware
available which which is is is certainly
cool i mean i think if
if i was going to highlight two
or let's say
three abstract properties
without going through a list of all the
hundreds of neural assemblies that are
not modeled in in in current ai
architectures and we'll look at three
abstract properties so we're not
we're not capturing attractor formation
and and non-linear dynamics i go back to
walter freeman's work on on rabbit
brains from the 18 or 90s showing that
rabbit olfactory cortex
recognizes
smells wood by strange attractors or
gary lynch's work
showing that the cognitive parts of the
cortex largely evolved from reptile
olfactory bulb rather than video or
auditory cortex i mean
we're not we're not getting a tractor
formation because that that's just not
what we're trying to do
what gerald edelman explored in his work
on neural darwinism and he was a clock
collaborator later of yugi nizhikovic
whose work i mentioned on the
non-genomics of the neuron
neural darwinism was a theory of the
evolution by a sort of natural selection
of neural neural sub-assemblies i mean
we're
we're not capturing that in modern
neural architectures at all and
simplistic attempts to use cmaes or
other evolutionary algorithms to evolve
neural nets are not are not the same
right i mean
and then
we're not getting what my friend weaver
aka david weinbaum called
open-ended intelligence in his phd
thesis at the free university of
brussels which is
the ability that the brain the the
radical level of plasticity that the
brain
can show like my my mother's partner
lost 30 of her brain in a car crash
literally the first year after that she
thought people were chickens i mean she
didn't no idea what the was going
on
she regained essentially all functions
but lost some traumatic memories from
her youth but there was like
radical rebuilding in in in the brain
and this this ties in with goal systems
also right like
how to what extent are we goal driven i
mean yeah we want we want sex we want
food we want status we want to have fun
but
in the end a lot of our activities are
not
too goal-driven in any comprehensible
way and we're even able to rebuild our
own goals as part of this whole
evolutionary non-linear dynamic that
that our brain our brains are
going through so there's
yeah there's a lot missing at that high
level and then then you then you go in
to the fact that we're not
really modeling most of the cortex or
how the cortex interacts with the
thalamus or the or the basal the basal
ganglia or or
tectum or so many other parts of the
brain which like what demis
hasabus has been after his at that
google d mind and before his cortex
hippocampus interaction and he has very
interesting ideas on that but that's
like one binary interaction
among many dozens of things you could
look at if you're trying to make a
serious model of neuroscience which
which i'm not trying to do although it
would maybe there's some parallel
universe version of me trying to do it
it's certainly cool can i come in on a
couple of things there so you were
talking about the whole um goal driven
aspect of ai we want to get to aix a
little bit later but um on the
modularity of the brain that's
interesting as well
because as you say some people could
suffer from a stroke but they seem to
regain a lot of their brain we spoke to
um jeff hawkins a couple of weeks ago
and his thesis is that the brain is a
general learning algorithm and that's
one of the reasons why um the neocortex
can be repurposed to do different things
in in the in the event of a stroke but
um i want to pick up on something you
said a little bit previous to that which
was about perception being a kind of low
level only process so when we talk about
different schools of thought in
cognitive architectures you said on
lex's podcast that you are surprised
that trivial systems your words not mind
like deep reinforcement learning could
work so well and we should learn from
them and minsky conceived a modular
architecture the society of mind and a
lot of symbolists assert that reasoning
must be separated from perception they
think that the only reason we need a
perception module is to create a
representation so that we can do the
real reasoning downstream on some
abstract knowledge representation but i
think you're a really interesting
character ben because you say we have
sensory knowledge and presumably the
kind of reasoning we perform on sensory
knowledge would be different to other
types of knowledge but having said that
i kind of agree with melanie mitchell
who said that visual reasoning isn't
easily separable from the rest of
intelligence especially general
knowledge and abstraction and language i
think it all boils down to whether you
think reasoning is top down over all
modalities or bottom up happening almost
in situ as jeff hawkins argues that
argues so succinctly
i mean
in in
a complex cognitive system like the
brain or or
a
working agi system i think the
the answer to is it this way or that way
is usually it's both ways right and i
mean i think there's
there's top-down dynamics and there's
and there's bottom-up dynamics going on
in terms of the role of
sensory processing
indeed it it's
it's complex and goes both ways right so
data comes through sensory processing
and then in cases where
sensory processing is too ambiguous
cognitive processing is used for
disambiguation you can have multiple
cycles of feedback
on the other hand
for very many people the process of
creative imagination even about quite
abstract topics
can be sensory right so like if if i
think about a hilbert space or some
relatively
abstract mathematical construction like
i'm not really picturing the infinite
dimensions of a hilbert space in my
mind's eye in a direct way
but there's some at least topological if
not geometric
image in my in my mind there right so
clearly my brain
must be
projecting stuff from cognitive cortex
into visual cortex
and using sensory manipulate using
matrices and rotations and like sensory
knowledge manipulations
as a crutch to help it sort through
some other type of complex cognitive
reasoning it's doing about about the
hilbert space right so i'm in the
because the brain evolution cobbles
together what it can from the resources
available right so as as the human brain
was evolving abstract cognition
all this sensory capability was there so
it made a lot of sense to project
project when it can
abstract math into that
visual domain and or and any any type of
abstract reasoning if you can make it
visual or auditory then then do it right
and that i mean i'm
i'm
fairly highly synesthetic so i i tend to
see music in concrete visual forms and
so i think
you have that sort of leakage and you
have leakage ring cognition and
and vision and all the all these
all these feedbacks are
important
and we don't understand them well like
there's an amazing book from it might
have been the 1950s or 40s by jacques
hardimore the psychology of invention in
the mathematical field
he interviews like a dozen great
mathematicians as to how
how they think creatively
some of them are visual george poglia as
i recall said he thought using grunts
and groans like
which is
is also like vector data right i mean
that's that's sounds with with complex
complex waveforms to them
when i mentioned this to my oldest
daughter she said she thinks in terms of
like typographic words that she can
visualize in in their mind which i'd say
like how the hell do you think at all
that's that's ridiculous
so that we we definitely reuse
sensory knowledge within the cognitive
domain and vice versa and you need a
type you need a type coupling there
which brings us back to cognitive
synergy
you can't say in principle it has to be
that way but that i mean that's how
that's how the human mind appears to do
it
though that brings me also back a little
bit to
to and
it's going to be maybe a bit of a repeat
but
um
all of all of this the brain being this
you know highly dynamical non-linear
whatnot and and clearly the brain itself
is in the domain of let's say continuous
variables much more than than discrete
variables right and so the communication
in the brain also happens in form of
maybe attractors of continuous variables
but nonetheless continuous variables so
is your is your is your advocacy for
graphs i'm i'm just sort of a bit
you know
allow me to be a bit a bit skeptical
here about our graphs the correct thing
at the current time
we're all building ai systems on digital
computers which bottom out on
long arrays of of of zeros and ones
right and i mean they're yeah early in
my career i thought about like can you
can you grow a brain in in a test tube
or get some like complex non-linear
dynamical chemical soup and coax the
emergence of intelligence out of it
very cool the world didn't go that way
right so i mean we're all
we're all looking at series of bits and
then organizing them into higher level
data structures that obey
certain certain algebras right and and i
mean they may be
linear algebras they may be
float or short float in in in here in
your programming language but i mean i
think
you could go a philosophical direction
as say hava siegelman is a researcher
who's going to philosophical direction
saying the human brain
uses all the bits in the real numbers in
classical physics or something right and
then i mean then
then it's an infinite algorithmic
information thing and you can't do it in
a digital computer but if we our people
think
you know quantum mechanics is used in
some profound and fundamental way for
for cognition
if we
set those aside for the moment those are
interesting topics of conversation but
would take the whole podcast if if we
assume
for at least sake of concrete discussion
we could build the human level general
intelligence
in a digital computer which i think i'm
not 100 sure but i think it is the most
likely case right i mean then then
yeah i don't think that
continuous versus discrete
is the most interesting way to look at
it i i more look at it in terms of what
are the
the algebraic operations that that are
are efficient are efficiently
supported and
when we look at
a knowledge metagraph or a knowledge
hypergraph like we're working with in in
the
open cog system
then we have to ask
you know what operations does this need
to efficiently support and this gets
into the a lot of nitty gritty stuff
that i'm thinking about a lot a lot now
in in building hyper on which is the new
version of the opencog system that we're
deep into the detailed design phase for
because
most graph databases out there today
like a neo4j or
they're optimized to support traversal
queries basically where you may you're
looking at path from one thing one thing
to another and
the way we're using a graph in opencog
it's a bit it's more like the way a
graph is used inside a haskell
interpreter
or another functional programming
interpreter although our type system is
much more flexible than
than haskell's because we're doing
gradual dependent dependent typing and
what what that means is you're you're
wanting to support
a wide variety of flexible pattern
matching queries and so that i mean this
gets sort of deep into the
computer science of it but what we've
done
is to formulate
a bunch of different
cognitive algorithms including
probabilistic logic reasoning engine
evolutionary evolutionary
learning engine some algorithms for
attention allocation and refining
motivations
and express these
algorithms in a way that the routine
operations underlying them
boil down to a few common operations
such as
pattern matching
against graphs or hypergraphs
and
this can be fancy pattern matching like
you may need to be able to match a
pattern where a variable is matched by a
whole sub graph or something right so
fancy pattern matching and then
what you call
fold fold operations like catamorphisms
anamorphisms and other fold operations
which are familiar to functional
programmers so
basically if you can fold fairly
flexible pattern matching over
a hypergraph or a metagraph
then
then we think you can get efficient
implementations
of all these different
algorithms we've been looking at which
represent learning approaches
that fit naturally with different
sorts of of of memory like like
declarative knowledge or episodic
knowledge procedural knowledge and for
sensory knowledge
alexei podopov who's uh
one of my lead collaborators within our
our singularity and then hyper on
projects i mean
alexa podapon did a bunch of work
connecting the current version of
opencog pre hyper on
with the
neural models in torch for for vision
and language language processing where
you're sort of a
doing symbolic reasoning on graph nodes
that represent
layers or other portions of of of the
neural net and so i think i'm not
actually trying to do everything inside
the graph
i mean you could you could and but but
on the other hand
torch and tensorflow and these things
are really really efficient so right
what we liked about torch is you have
complete transparency into the compute
graph so you can represent the aspects
of the chart's compute graph inside the
opencog knowledge graph
and when you compose the layers in torch
you're composing the corresponding nodes
and the
in the knowledge graph and so forth so i
i don't think the graph has to do
everything but i mean
it happens we we boiled down
a lot of human-like cognition
into
folded and unfolded pattern matching
operations on
typed metagraphs and that's i mean
that's kind of wonky to to dig into but
on the other hand
you know the construction of a jet
engine is kind of wonky to dig into also
yeah my question was more with regard to
it i agree it like you can implement it
you know however you want ultimately
what you want is the operations on on
the data and when i when i hear graph i
immediately think of
exponential sort of exponential blow up
in connections because if i think of
knowledge it's like okay you know how in
how many ways can this object be related
to that object and those are just two
objects right so how do you sort of
overcome
i have to traverse all these connections
i have to pattern match in it seems like
a good friend of mine
with his own ai project called nara's
non-axiomatic reasoning system
pay wang uh always had one of his many
wise aphorisms about ai was that
forgetting is just just as important as
as as learning right
but the the other aphorism of his i like
even better is
when in circular reasoning when your
circle gets big enough that's called
coherence right so
so he
but i i i i i i think uh
forgetting is is
extremely important you're not your
knowledge graph cannot
cannot contain everything of
of of course but i mean
but i mean if you want neural nets to do
cognition
you're making you're doing you're making
recurrent neural nets anyway i mean i
mean you're having a graph anyway and
then you've got to say like
okay this is not a completely connected
recurrent network at the sparsely
connected neural network and why do we
have only some of these connections and
not others so i guess i
i don't think that cutting through
combinatorial explosion
by
architecting once and for all a
hierarchical breakdown of your knowledge
is is a
is a tractable solution you can do that
in vision
because the stuff coming in through the
retina or the camera
i mean it does have a very natural
hierarchical representation based on
like larger and smaller regions in the
in the input field and even then it
doesn't completely work but it's so it
sort of goes
sort of goes somewhere right so i i mean
you
this just comes down to you need and
you need an intelligent system right i
mean you can't you you you're not making
a completely
connected graph you're keeping those
links
that were found meaningful by the system
in in some
contexts and you're pruning those links
that are predicted not to be useful in
in in the future and
i mean the the brain the brain of course
is also a richly
cross-cross-connected graph and some
some connections are potentiated in some
and some
are not and in the epigenetic phase some
connections grow and and
and some some do not
it's very interesting question whether
an analog computer like a continuous
variable system
would be much better for implementing
human-like general intelligence
it's kind of shitty that the whole field
of analog computing
dwindled many decades ago but i mean
in the end we're all being pragmatists
and working with the
the systems we have at hand well it is
kind of an interesting question it
always makes me think of you know
penrose as uh the emperor's new mind and
is there something in the continuity
or evolution of the brain that makes it
able to do something approaching hyper
computation you know who knows
whoa
yeah i mean
penrose's
argument
in a way
is kind of silly to anyone with a deep
technical knowledge of of
of physics but if you
if you take it as a proxy for a class of
arguments that might include some more
yeah more sensible ones it could be
interesting right
penrose is making two
chief hypotheses
neither of which i think is right but
they might point in interesting
directions i don't know i mean the first
one is that like when he
conceives something that's
mathematically creative
he's carrying out a non-computable
operation of of
like
magical transferring human intuition and
to me it's just crazy that he thinks he
could know that like we don't know
what's going on in our unconscious
processing sure it feels like divine
inspiration but how do you know it's not
just some neural process that's opaque
to your deliberative introspection right
i mean it could be the case but how
could he know that
the other the other hypothesis he's
making is that
this transferring burst of profound
human creativity
is coming from
transferring
computing
enabled by quantum gravity in
microtubules in in in the brain and
what's amusing is his own theory of of
unified physics a twister theory is a
completely discreet theory right so i
mean there's just no evidence in actual
physics of this
of these transferring oracles doing
something so it's on the other hand my
friend james tag claims to have uh a
penrose inspired quantum gravity
supercomputer sitting on his desk in san
diego i haven't i've not been down there
to check it out yet so i'm i'm not i'm
not endorsing it well i guess so i guess
you know i'm
i'm open to the possibility
that there is some
transferring cosmic inspiration aspect
to human human cognition maybe there is
there's a lot of the universe we don't
understand at all
if you're gonna go there i'm also open
to the possibility the digital computer
program running on the digital computer
could access this source of transferring
like
cosmic wizardry right like if
if the gods are implanting creative
inspiration into my brain why can't they
plant it into an open cog right like
some gamma rays hit it in just the right
way why go this far out and another not
even
even further out and then it's
all kind of interesting but if if you're
trying to be a scientist rather than a
philosopher there's just no
there's no there's no evidence there's
no evidence for
for for any for any any of that right
and
that's uh
seems to me that
you know that the idea that penrose has
professed that
girdle's theorem limits
human limits digital computers from
creativity because the the digital
computer blows up to obey the axiom
system it was given and the human brain
somehow bypasses that i think
that makes no sense for a number of
reasons one of which is human humans are
wildly inconsistent so i mean girdle's
second incompleteness theorem is like
you can't you can't be reasonably
powerful consistent and complete but we
for better or worse we bypass that by
being like wildly
logically inconsistent
dynamical systems in in in the in the
first place right but but the other
thing is your digital computer systems
are not sitting in a vat like they're
they're engaged
in practice with the outside world which
is
largely incomprehensible to us and
presumably has massively greater
algorithmic information than than any of
us so you really you have to look at
both humans and digital computers as
somehow modulating the self-organizing
complexity of the world they're
interacting with right it's it's not
like
they're restricted
to their own algorithmic information
forever because they're locked in a box
right so that yeah i think
that's a lot of fun stuff to think about
and uh it is
you know i think that's a difference
between my generation of ai researchers
and then
most of the people in the
coming into the field now is in my
generation
and the generations before even more so
if you found your way to agi
you usually found it from a position of
trying to profoundly think through like
what is a mind how how does the mind
work how the hell should we approach
this whole thing and you had to go down
to ground zero and try to understand
what what is thinking and maybe you got
absorbed in that rabbit hole forever or
maybe you came out with some tentative
hypothesis and decide you want to
implement something and explore
some concrete class of ideas now it
seems like
people come into the field and they want
to do agi because their mom told them it
was a good way to make a high salary or
something right oh yeah and and of
course they assume
google has already figured out how to do
it so it's just a matter of putting in
the
putting in a few a few details to some
of the algorithms so it is a
is it is
remarkable remarkable change in sort of
the orientation of the the average
person going into the field of course
yeah humanity is diverse and there's all
sorts of
of
complex crazy young people too on the
edges yeah um doing a phd now is the
extended interview process for fang but
anyway i wanted to get on to
aixi so you've said that hutter's
universal ai theory aixi the main
article of faith at deepmind by the way
serves as a credible although debatable
theoretic approach to agi and that
several proto-agi systems are emerging
from it already i'm not sure which ones
you meant but we'll get to that in a sec
but your core belief here is that
intelligence is closely tied to the
creation of procedures that achieve
goals and environments in the simplest
possible way you seem to believe that
the best way to think about ai
is in term is in terms of a discrete
decision-making agent now um we think
that aixi is a kind of tautology and the
recent deepmind paper reward is enough
is also full of tautologies i mean all
mathematics well yeah
that may well be the case but for
example um if it were possible to find
an optimal agent you might need to have
an agi already but anyway aixi also
assumes that the environment is
computable and i think there's a huge
gap between some of the very theoretical
notions we talk about in agi and reality
agi researchers cite apparently
incontrovertible and mathematically
rigorous articles of faith whether it's
aixi
universal function approximation or the
so-called touring completeness of neural
networks they're not too incomplete by
the way folks um francois charles
similarly presented a measure of
intelligence formalism which also wasn't
computable now you've said that you
think human intelligence is subsumed by
the aixi agent but i'm interested to
know whether you think that ai xi
is a specialized form of intelligence or
a general form of intelligence after all
it's just a static computer program
right how could it possibly be
generalizable
well the
the main limitation of aixi apart from
it needing
infinite computing power to do anything
the main limitation of aixi
is it's
it's not an open-ended intelligence
in the sense of of david one bomb it's a
system that you give a reward a reward
function right so if if you're willing
to narrow to that context to say that
i'm gonna
consider
an intelligence as something that's
optimizing
maximizing expected reward of a given
reward function i mean then
the theorems of market soothers show
that
aixi
can do that and it
can do that within the constant factor
it can do that as well as any other
system could optimize that reward
function in the in that environment
right so that in that in that
but it reminds me of a
a story every mathematics grad student
in my generation learned is is like a
a physicist was brought in by a farmer
to figure out how many cows could fit in
their pasture the physicist starts out
okay assume a spherical cow and then
goes from there the farmer kicks them
out because cows are not spheres right
then they'll just all roll down the hill
but
but i mean within the physicist paradigm
that that that's where you start and so
i mean xc is definitely assuming a
spherical cow right and you you you run
into limitations that way but
it still tells you something to me to me
it's interesting to know
that if your goal is to optimize
expected reward
in a computable environment and
resources are no issue then agi is
trivial
i mean
axe does it basically by at each step of
operation it searches the space of all
computing programs to figure out which
one would have given it the best answer
in the past it executes that program to
generate the next action
so what what that teaches me i mean it's
it's it's nice math as a mathematician i
i find a
satisfying bit of of theory but
i mean
newton's
you know classical mechanics is also a
satisfying bit of theory and it doesn't
fully describe describe describe the
world as limitations right so
what that tells me though is that even
within the constraint of expected reward
maximization which is a limiting way to
look at general intelligence
it's all about limited resources and
this goes back to pay wang whose
definition of intelligence is
adaptation
to the environment given limited
resources which i think
that in itself doesn't quite capture
everything i care about with general
intelligence but it's uh i think it's
it's pointing in that in a good
direction in that
you know you can optimize your reward
function just by brute force enumeration
of all possible ways to ways to do
things and then bayesian reasoning on
that human brain doesn't do that
why doesn't the human brain do that
because it evolved with limited
energetic resources right well and
this
this tells us
something about how to think about being
practical about building practical
agi systems i mean it tells us we need
we need to think about
you know what are the resource
constraints and how are we dealing with
them no i mean
in in in a sense that's that's obvious
anyway in logic everyone knew
pruning the the search tree is is is is
what really matters but uh i think that
the limitation of maximizing expected
reward though
is also a quite
serious one like there is no loss
function that those of us in this call
are
are systematically maximizing
throughout our lives right i mean that's
not what
what humans are doing if
if you look in weaver's thesis on
open-ended intelligence he's looking at
it a different way he's looking at it
like
an intelligent system is
simultaneously trying to individuate
itself meaning to keep keep its own its
own boundaries there and keep on
existing
and it's trying to
transcend itself in what he calls a
transductive chain like to build
build something new and better out of
itself and that this is like being and
becoming back to hegel right and i mean
there's
there's a contradiction between these
two which lead you in a para-consistent
logic direction but if you look at our
goal as to maintain ourselves
while transcending and improving and
developing ourselves
it's not clear
this sort of
open-ended intelligence
motivational system is well captured by
writing down
a loss function or an expected reward
function and saying you're
you're maximizing that like i'm sure my
the reward functions that i approximate
have changed over the course of my life
so far and i'm and i'm i'm only 54 and
i'm still in in in human form right so
there's yeah i think that that's a
fascinating body of theory but of course
there's
there's many limitations could i just
say because it'd be your
anthropomorphizing a little bit there
but just looking at the mechanics of the
because i'm i'm really trying to
understand this
this reward function it has to be a
function of the environment but where
does it come from why isn't it deceptive
why is it a smart idea to pl to pluck a
reward function out of thin air and
optimize on it and then say well it
probably isn't but marcus certainly
didn't didn't make that up right
i mean he he's he's just putting
statistical statistical
decision theory
together with algorithmic algorithmic
and for information theory i mean you if
you want to be cynical you could look at
axe as the sort of a reductio ad
absurdum of of of reinforcement learning
great i mean i mean he's he's just
extrapolating to the the logical
consequence of it like th th this th
this will do it and then you're done
i'm going so
the the steep mind paper i don't know if
you have have read it but this um paper
about reward is enough it makes a
slightly different claim
that's a profoundly silly paper although
although i mean i love rich sutton and i
mean he's he's both a sweet good-hearted
wonderful human being and he's he's
contributed
fantastic
things to the field but i think that
there's a long pathology in the ai field
which is
assuming that if if some
algorithm or approach
in theory is enough to lead to human
level agi
like given enough resources
that that matters a whole lot and i mean
i think
you know schmidt uber's girdle machine
paper shows that logical theorem proving
is enough but it's pretty easy to see
that genetic programming must be enough
i mean if if you if you have a big
enough population you're
evolving programs and so i mean
in the end
i would say
almost
any learning approach will be enough
if it has the property that
no operating program for the ai system
has probability zero like as long as
long as you will explore the search
space of things to do
without leaving out any options in in
the in the end you in the end you'll get
it even monkeys are enough it's not that
rl is enough
is wrong if you can
if you construe it broadly enough it's
just not
it's not interesting because because i
don't think it's enough in real life
given realistic given realistic resource
constraints just as i don't think pure
logical theorem proving is enough in
real life given given real resource
concerns so i just want to push back on
that a tiny bit so um there was a chap
called france while charley came along
and he wrote this beautiful paper on the
measure of intelligence and um yeah i
know i know
we love francois i think he's giving a
keynote at our agi conference in in
we've had him on the show and we're his
number one fans actually but no anyway
he came along and and he said because
obviously another conception of
intelligence is about flexibility and he
says that intelligence is about the
ability to adapt to novelty using
knowledge you already have as
efficiently as possible and essentially
it's about being able to create a skill
program but in his formalism of
intelligence i mean that's basically the
same as as pei wang's uh adapting to the
environment no not not quite
the difference is and i and i and
expected
i i hope you'll be um
kind of um you'll be into this idea that
our measure of intelligence should take
into account experience and priors and
knowledge because we're knowledge people
right and i i that doesn't seem to be
happening with some of these conceptions
of intelligence because otherwise surely
you could just brute force you could
just do every possible thing in the
united
states using relative algorithmic
information relative to some assumed
knowledge base so that that doesn't
change any literally changes almost none
of the math i mean that's so that that
wasn't
marcus's focus the thing is with all his
theorems they're all kind of in the
limit anyway so i mean if if you assume
some finite body of knowledge it just
changes changes the multiplier constant
by some fixed amount and then and and
doesn't matter but yeah the i mean the
contextual
aspect of into of intelligence and the
fact that it depends on what knowledge
you're bringing in obviously is
is incredibly important in
in actual reality and i mean what
what we care about is
human-like general intelligence in
human-like everyday
context which is what we i mean that's
what we care about we care about
initially so i i do think that's
important and
of course part of these contexts
is that you need to be able to get out
of the way of the car before it runs you
over right i mean i mean you you need
there's a
there's
temporal and spatial and energetic
constraints on on the system which are
coupled in particular ways with the tat
with the task set right so that there's
a lot there's a lot of uh
there's there's a lot of of
suddenly there i mean i think
francois chole's conception
of general intelligence
to me is
very similar to pey wangs which is
weaker than the than
than weavers because it
it
it's sort of
to me those
that conception of adapting to the
environment
uh
based on limited resources even if the
environment is unpredictable and
extrapolating from the past to adapt to
the environment
that captures the individuation part of
weaver's notion of open-ended
intelligence it doesn't capture the
self-transcendence part where an
intelligence is trying to develop into
something fundamentally beyond what it
was and i think in psychology there's a
difference between learning and
development right and
development kind of fundamentally
refactors what the sys what the system
is it's not just going to do stuff
within the architecture of the system
and i mean i have
at home here i have
a three-year-old kid and a
five-month-old baby who are my fourth
and fifth human offspring right and you
at these ages
it's not just learning right i mean
they're new
new portions of their brain are coming
online and our new new networks or
assemblies in their brain are coming
online anyway and the way they come
online is conditioned
by their environment and and and by what
they're doing and
i i would want to embrace
fundamental development of that nature
as a key aspect of
of general intelligence i mean you might
you might say we're just not ready to
deal with that yet and we want to master
learning within a given stage of
development
first but it's a little subtle if you
look at the human analogy right because
humans as they grow up they're radically
developing
in conjunction with the
the the learning that that that they're
doing and the the fact that our one
reference system is doing development
and learning concurrently in a coupled
way would suggest that that's
that may be something we should
we should take into account and that and
that that if you go computer sciencey
and look at logic systems and and
program learning
that brings you toward like
meta reflection and and uh you know can
can the
if you have a a programming language
with a type system
can can the system create new types and
insert them into its interpreter or
something right i mean the the form that
the form that fundamental development
and the system refactoring itself takes
in the human brain versus in a more
computer science based system like
opencog hyperon
may be different but still i think
focusing on the development aspect
pushes you in a different direction
which
i don't i don't see
even even the more uh radical voices in
the current deep learning world like
french squash chole i don't see them
going there yet but maybe in a couple
years things are evolving fast could i
ask you a quick follow-up on the notion
of pure intelligence so um i know you
subscribe to ray kurzweil's view of the
singularity happening within the next
few decades and personally i subscribe
to melanie mitchell and francois chalet
and douglas hofstadter i know you said
that he was a huge inspiration for you
their school of thought is that there's
no such thing as pure intelligence it's
better thought of as a process their
embodiment etc our intelligence is
surprisingly specialized even though we
can write software to solve problems
which we can't so how universal is
intelligence and and in particular how
anthropocentric is intelligence now you
said on lex's show that there are many
ways to build a flying craft right
perhaps the same is true of general
intelligence but it's also fascinating
how many emergent features of
intelligence may be decoupled in some
sense to the intelligence which produced
it for example different intelligences
might discover the same mathematical
axioms it's almost as if all roads might
lead to rome so how do you think
different intelligences might be
comparable
well i think all the roads that we're
talking about leading to rome
are
existing
mostly within the same physical universe
and even on the surface of the same
planet right so there there's a lot of
common
context there and that i would say
if you want to accept known physics as
an axiom
it would follow from any conception of
intelligence i know of
it would follow that no
finite physical system
is ever going to be maximally and wholly
general intelligent i mean there's a
there's like beacon stain bound for the
amount of information can be stored in a
certain amount of mass energy like you
you're going to have finite algorithmic
information for any phony finite
physical system you you can't build you
can't build the
axee or like a full-on
girdle machine in in the physical
universe
and yeah that means you're not creating
a maximally general intelligence and as
as i've often said
humans are not a maximally general
intelligence i can run a maze in
two dimensions better than 750
dimensions and and
and so forth right and i mean humans
you know i was a math professor for a
while i saw how
even fairly intelligent humans struggle
with very basic mathematics like people
people
i mean we elected donald trump president
we can't be that generally intelligent
right
people have profound
profound limitations in their general
intelligence but yet
that doesn't mean everything should be
considered equal like i still think it's
meaningful to say
like i'm more generally intelligent than
than this glass or than that than an
earthworm or
or a dog or something and
so i i guess maximal general
intelligence is
i can't say it's impossible to exist i
can say
if we assume the laws of physics as a
context it doesn't exist in that domain
like there's an alternate
phenomenological view
where the laws of physics are built up
by the perceiving mind it's itself if
you want to go the the buddhist
psychology direction maybe everything
begins from a ground of of infinite
general intelligence but but once we get
trapped in this samsaric realm of the
physical universe and we're down to fun
and algorithmic information right and so
that's i mean that's
that's fine there's there's still a lot
of room to go way way smarter than the
human brain
without hitting the beaconstein bound
and the limitations of general
intelligence posed by the laws of
physics like
just like we are not the highest jumpers
or the fastest runners we're almost
surely not the
the smartest thinkers right so they're
you you can get through a singularity
and the whole probably a whole bunch of
multilarities beyond human level general
intelligence
before you hit the limitations opposed
by the laws of physics
and what the laws of physics will look
like
to the beings we created that are 100
times smarter than us
who knows we seem to have revised the
laws of physics radically every
every 50 years or so or so anyway right
so again doug hoffs douglas offstander i
mean he had his book gordon lester bach
which i read in the mid 70s when i was a
kid i mean that
that was the first time i
encountered sort of the whole academic
ai community just sort of
in passing during gorda lescherbach
because before that i knew ai only from
science fiction so it was it was just
very inspirational to me to see how
these like
actual professors and serious thinkers
were
thinking about how to really build ai
systems but uh i've diverged from
douglas in various ways since then i
mean he thinks the singularity is kind
of a bunch of bunk but he
he also
thinks current deep neural systems
are completely unrelated to general
intelligence and there's almost nothing
to be learned from them except what not
to do and they're i mean i i i don't
want to
misquote douglas who has his own nuance
point of view but that's that's the
impression i get from conversations with
him on this anyway where is my
my view is more that
these
currently highly successful deep neural
models and other ml approaches
are
quite informative about how to emulate
parts and aspects of human level general
intelligence and they're great for
building some kinds of commercial
systems i think they're missing a lot of
aspects of of human-like general
intelligence and that that incrementally
improving on them probably is not going
to be the most effective path to get the
human level general intelligence
but i don't think they're
totally off in a useless direction
there's all sorts of cool things to
to to learn to learn from there i i
and
as you know unlike douglas hofstra
i am an optimist that we're gonna get to
in general intelligence
beyond the human level
during during for example my career
assuming nothing very unfortunate
unfortunate uh
befalls me and yeah this
this will bring about some form of
technological
singularity which is a whole other
whole other big topic but yeah well
while we're on the
the phrase singularity we should at
least briefly mention singularitynet
which is the project i'm leading at this
time also yes we were we were also
pretty interested in in singularitynet
you you said
uh if i recall that uh
correctly that open cog and and your
rewrite of it they're sort of um about
connecting knowledge in a quite
integrated sense narrowly maybe even in
in ram on a single computer and then
singularitynet is about connecting
knowledge in a loosely coupled way
across
essentially the entire internet
so
in essence singularitynet is
if i understand correctly a marketplace
for apis right there's not much to do
with ai per se like i can i can put up
an api that you know gives me a database
access and and
like why do i have to pay for it and and
all that kind of stuff could you
elaborate a bit sure
why is this a good substrate for
from
an ai theory view
i guess
you could view there as being
multiple ways to get to a practical
agi
and
one way would be
you build
some group builds a really smart
algorithm
that is just running running on their
computers the way the way they want it
to and it's thinking and learning and
abstracting i mean it may get out of
control eventually but uh but and anyway
it's initially behaving by one group's
design
another approach which is interesting in
concept anyway
is you have a more decentralized network
where different parties have
have
put different ais doing different things
one guy may have put a perception engine
online for visual data another guy may
have put an abductive reasoning engine
online that another guy put a
computational creativity engine online
these different components
have a language for describing what they
what they can do to each other for
describing their their properties to
each other
and and and then
general intelligence
emerges or crystallizes from this this
network of of agents and
no no one no one human party designed it
right now
some folks have hypothesized this would
just happen on the internet right so
this the internet is is a playground
where
multiple parties are putting ai systems
out there and they're all communicating
and then combining and then connecting
with each other
if you think about it more deeply it
seems like the way the internet is
constructed
it's not very well oriented to this sort
of
autonomous
assembly of algorithms written by
different people into into uh
into autonomously acting super
assemblies and so forth but but you
could build a system that was right so
you could what we did in singularitynet
we built a system that
could serve as a sort of primordial soup
of agents where any anyone
anyone can put an ai agent into into the
singularity net framework and
the ais in the system can
describe to each other what they do what
data they take what what outputs they
give
how much processing power they take to
do a certain thing how much they're
gonna they're gonna charge so there's a
language for the ai's to describe what
they're doing what they're doing to each
other
and then
the way blockchain is used there is just
in the plumbing to
to ensure that there's no need for a
central hub in this network like anyone
anyone can put an ai online to this
network it announces it's their
peer-to-peer to the other ais in the
network and then it's
it's it's it's part of the club and
some folks
within our singularity net group
they're really pushing to have
benevolent human level agi
emerge from this sort of
society of minds where different members
of this society are
coded by different people with different
with different
aims in mind the way i i mean if that
happened i would be i would be the very
proud father of the uh emergent network
in the singularity of the emergent mind
in the singularity net network it would
be awesome but i mean the
the way i'm looking at it
is more like
i'm building opencog hyperon which is a
pretty specific cognitive architecture
part of the reason we're moving from the
legacy open cog system to the hyperon
system
is
i want something that better makes use
of
large-scale massively distributed
processing power the other main reason
is i want something that interfaces more
efficiently with deep neural net
libraries
but part of the way hyperon
will do massively distributed processing
is through integration with singularity
in that platform so the way i'm looking
at it is
we can roll out opencog hyperon on the
singularity net network as a bunch of
interacting agents and
this can be
really really loosely i mean this can be
like the cognitive cortex of uh
of of an agi that lives in
singularitynet so
maybe somebody else
writes the
visual perception cortex
and and we've written the cognitive
cortex with hyper on and then
someone else writes
you know a cerebellum that controls
factory machinery someone else writes a
cerebellum that controls humanoid robots
right and
the the really interesting thing would
be
if the visual cortex so to speak that
one guy wrote and the hyperon-based
cognitive cortex that we wrote
if they could interact in a way that
enabled
like substantive
feedback between perception and and and
cognition like can they share
intermediate state
along a channel that they created and
so together with
iohk the the company that that
works on the on the cardango blockchain
we're working on we're working on the
what we're calling an ai dsl which is a
very rich sort of dependently typed
functional language
for communication between different ai
processes in in this in this network so
i mean that's
that's the vision
and aspiration with singularitynet and i
would say
within our organization we have some
people who think
hyperon is just one
ai system among many that will run in
the decentralizing like
prebiotic soup of ai that will
self-organize into intelligence
we have others who think
hyperon is going to be the agi
and then the fact that it's deployed in
singularity net
it's nice that it will be there instead
of aws it may stop governments from
taking it over once it becomes a super
intelligence
but they think it may not be critical to
the actual ai part of it and i
i'll be happy if it's either way or so
or some combination to those i have a
question though about the
i mean i assume at some point the
publishing of eight of apis the
discovery of apis that has to be
automated you know the machines have to
be able to do that themselves and i mean
yeah yeah but so far nobody's been able
to successfully do that in any api
context even outside of
apis it's like a notoriously difficult
problem right so
how do we get away from the sort of
still human bottleneck the need for
humans to sit there and write down apis
and write glue layers and understand the
semantics et cetera
yeah i mean that that's uh
that's a quite hard problem and i mean i
think i think
that to me is a problem where
automated reasoning
meets
functional programming
and and uh and the answer may be found
there and we we've seen this
and this is
the solution to this is totally not
completed and deployed in the current
singularitynet platform like right now
humans are writing the api so we're
totally not there yet but
if the ai agent
is written
in say haskell or another higher level
functional programming language i mean
then
you have a lot more affordances there
for the same reason that you can do
automatic verification of programs in
these languages much more easily than
say say a c plus program and we're
we deal with questions there like
suppose someone wrote
you know an algorithm doing decision
tree learning or something simple like
suppose you want that algorithm to
operate in a secure multi-party
computing context with security for
sharing data in multiple places
or you suppose you want that algorithm
to run distributed across a bunch of
machines can you do automatic program
transformation
of the ai algorithm to allow to do that
without the human having having to
rewrite it and write what you
clearly
the route i see to making that possible
is where the algorithm is written
in haskell or o'connell or some higher
order function some functional language
and
and you have a semantic description of
what the algorithm does
which which is is linked into the to the
actual actual code right and i mean
until you have that
you're you're you're kind of
kind of flailing around and i i mean i
think
if you have those things right now
we could do that sort of thing in an
academic playground sort of sense like
we could do it with
with
sorting algorithms and simple list
processing algorithms and stuff but it's
still not
it's not tractable to do it with with
really really complex algorithms but i
think there's
an interesting path there and this is
one of the reasons i'm so psyched about
the partnership of singularitynet with
iohk and cardano because they've got a
they've got an army of brilliant
functional programming academics within
within their organizat their
organization and
you know here we're going away from
modeling the human brain and i mean
we're going into okay we're
we're taking cognitive architectures
that are loosely inspired by the human
mind
but then we're sort of uh
we're we're
doubling down on current computing
infrastructure right and we're saying
okay yeah
let's deal with programming languages
let's make a codic modality where the ai
can
consent programming languages let's
let's make a cerebellum where the ai can
manipulate
the execution graphs underlying a
haskell program and just
embrace that and that
that may be the right path to like a
meta reflective agi system but
there's interesting balance there right
because the further you go in that
direction
the less you can draw on
on human cognitive science let alone
neuroscience to
to uh motivate what you're doing but
it's a
yeah very
very fun domain to be playing with and
then you're ready just um a quick
question because it sounds like you're
in a way describing the ultimate micro
services architecture but
sometimes it's more efficient to move
the compute to where the data is and to
do things monolithically
um
end to end right because it you just
have so many i o bottlenecks and these
networks yeah
it can be and with singularitynet
it's not like all the processing has to
be on everyone's
phone or in their like uh
pacemaker or something right i mean i
mean you you can you can have
you can have a server farm running a
whole bunch of of
nodes the
the question is really
if you want to move those things off
that server farm
like
what what are you doing is this just a
bunch of nodes in a decentralized
architecture
who happen to have gathered together on
that server farm because it's more
efficient for them to have a
conversation that way
or does moving them off that server farm
involve
you know doing a lot of
of human work to port from the apis of
that server farm to the apis of
something else because right
right now we run a lot of singularity
agents on aws
but the singularitynet daemon wraps up
all the all the aws specific calls so
that the agents can be moved off aws
without changing any of any of their
code
but yeah the data that's in buckets on
aws will be accessed much more slowly by
them once you move them somewhere else
but they'll still be accessed by the
same api calls awesome um dr ben gertzor
it's been an absolute honor to have you
on the show thank you so much we really
appreciate you coming on
absolutely thank you thanks for having
me interesting uh all
fascinating stuff and uh
similar to with uh lex friedman it's
good to be able to dig a little deeper
than in the average podcast absolutely
well yeah we pride ourselves on being a
you know a technical podcast so yeah we
we really it's a good show yeah yeah
amazing amazing thank you so much so
that was dr ben gertzel how did you guys
feel about that conversation
i i learned a lot and i found a lot of
what he had to say super fascinating and
gave some great references to follow up
on like
i mean uh so i've got some fun reading
to do to
when i can make the time
what was your comment keith he said that
the aixi was the reductio absurdum
of intelligence or something no of of
reinforcement learning right
there are a bunch of great quotes in
there
and then i think it was i think you said
pay wong or i have some note here on who
said this but i like the as the circle
of your circular logic becomes large
enough it becomes a coherent
coherent system something like that
doctor what did you what did you make of
the conversation
i say this almost every time but it i
think it highlights the importance of of
just talking to people and and
and uh because
you know we were reading all of his
papers watching his talks and so on um
ultimately when you interface with
someone one to one and and really give
them a chance to directly respond to
questions and so on
people always seem to be like a lot more
agreeing and and you you think a lot
more like oh yes that's you know that's
actually
you know true people seem a lot more
not not that he was he was ever
unreasonable but people seem
in your mind a lot more reasonable than
when you just read their papers and you
go like well
yeah
i guess that's a reason for building
sofia right to give that uh to get that
human interface
yeah well i have to say he he
for for being
kind of out of the mainstream of ai he's
very good at
marketing in general right
you know generally the things
he does be it this with with
singularitynet sophia and so on even
or you know even opencog in in a sense
um they seem to be well received and
people seem to pay attention and if it
is only because sophia has a like a
human face which everyone in the
technical field knows like it like it
doesn't matter if an ai has a human face
right and there's a bit of like this
emotion thing go like they have in that
project but
now sophia every time a journalist wants
a picture of an ai that they they put
sophia there right so so
yeah it would have been also interesting
to ask him maybe a bit about a few few
tips because that is
a thing he's definitely successful at
right um
being being sort of businessy while
doing his own thing while not being in
the mainstream right not following the
hype which i find really cool like yeah
he he is an enigma to me
um
he's such a unique character he's a rare
breed but reading his papers
at first i was thinking my god there's
so much technical jargon in here and i
don't think it was gratuitous we had
this conversation at first you almost
don't want to believe it you think oh my
god i can't understand what the hell is
going on here but you go and look up all
of these definitions whether it's
distinction metagraphs or hetaraki or
you know goloice connections or whatever
else and um it's it's all completely
legit you know
the only answer is that this guy is a
polymath he is incredible because i went
online i i saw is there any criticism of
this guy and there isn't there isn't so
but at the same time he is a businessman
and he is making the case and he is
getting lots of funding successfully
he's he really is spinning all of these
plates at the same time and and that's
pretty impressive
yeah i agree with you i think it's i
mean it seems clear to me to just the
magnitude of his knowledge
and also by the way pragmatic experience
i mean he's though he has this huge
background of theoretical knowledge and
study he also has been
involved in building systems in my
opinion that's a huge
benefit to anyone's learning is actually
going out there and trying to practice
trying to build something trying to do
something in reality he's done that and
so he's just got this huge
mountain of knowledge here and when you
ask him a question and we're trying to
get to point z and we're starting at a
there's a long path there and i'm sure
all of us have been in this situation
when say like
a a younger kid ask you something about
you know what string theory and you're
like
okay um we got quite a distance to go
from like what you know to string theory
and so it's a long path there and i
think that's what what you run into a
lot with
with ben when he's answering questions
um yeah
he's almost got too much knowledge
it's it's kind of overwhelming because
because he's all like you know like um
we watch some videos because today we we
framed the conversation and i thought
the conversation with lex was really
good again because lex framed it because
some of the conversations you hear with
ben online you know one one minute it's
nature the next minute is parapsychology
the next minute it's the meaning of the
universe the next you know what i mean
and it it's all over the place but the
reason it's all over the place is
because he can go all over the place
quite quite confidently
but there were a couple of um things
that i noticed though because i kind of
got the impression with ben at the
beginning that a little bit like awry
who we had on the podcast he almost
thinks that ai should just be this
confection of everything but the kitchen
sink you know let's throw neural
networks in there let's throw um you
know logic systems in there let's let's
do evolutionary programming episodic
memory no problem that's a simulation
you know let's just kind of conflate
everything together and and
and believe strongly in the emergence of
some kind of ai on top of that and and
that's where i think we're a little bit
more skeptical well i have a i have some
feedback on that which is he also has a
massive amount of knowledge of cognitive
i mean human cognitive processing right
and so i think that in my opinion i'm
guessing that that leads him to just
have a tremendous amount of
understanding
of the value in the
in what's already been built by
evolution and kind of the human mind
that we need to
utilize that if we want to make more
rapid progress
in ai rather than just rediscovering all
this stuff you know 500 years from now
on singularitynet yannick made a video
about singularity now i assume that we
all agree that something like
singularity in there is a core substrate
for any future artificial general
intelligence with all of the caveats
that we spoke about you know is is it
really decentralized i'm not sure it is
i mean yeah some things are on the
blockchain but presumably if i want to
plug my ai in there i'm probably going
to be a private company and it's
probably going to be surprisingly
centralized and also is it efficient to
have such a strong decoupling between
all of the services and having to you
know the io of having to send the data
between the services and as you said
keith the real problem is it's an
intractable problem to describe what
what a what a function does you know
let's say we plug in google's face
recognition api and microsoft's one
it's it's almost impossible to describe
the comparative behavior of those two
apis
well especially to have a machine be
able to do it because it has to be
automated i mean when you and when you
say to me oh have you seen
yannick's new face recognition facial
recognition program
i pretty much already know what what
it's doing right i know the semantics of
that i mean and
and maybe i just go read i can just skim
through his fact and kind of get the
rough idea of like the little nuances or
the cool stuff that he's done to make it
make it better right that's a you know
machines can't do that like they can't
do that yet rather i should say right
and people have been working on this api
you know reflection automated discovery
standardization et cetera for
forever and it's an unsolved problem
that's the real like
even if they could do it it would be
brittle
right because
a facial recognition api you're talking
about a neural network with billions of
parameters
it's not possible to meaningfully
compare those two programs
well yeah i mean it's
i think yeah it's not possible it's more
about
what keith i think said is that
maybe it's not that we can describe all
the programs we have today in terms of
an api right but tomorrow we're going to
come up with a variant even of a deep
learning system even of a of a face
recognizer but that does something
different right the input is still going
to be give me an you know an rgb
array and i'll give you like a list of x
y coordinates um
however the meaning of that
is it's going to be so hard to describe
this in any form of of a way that
other machines could then use
as a subroutine what other machines
could then be like
without humans telling them to go face
recognized machines could be like oh
wait this thing could be useful for my
thing right and yeah
like
i mean the whole point of this system is
to have a customer interface where you
abstract the actual physical api from
the customer so the customer just says
i'm getting facial recognition but i
don't know what the api is but it's even
harder with ai models because you have
so many failure modes right so
if if i did um switch over to google's
api tomorrow maybe because the system
decided it was cheaper the failure modes
are different it is this is such a um
it's just typically
the the singular dna paper describes
that you could have some sort of agents
on in the system some agents would be
there sort of to purely um evaluate
other agents so technically that problem
could be solved in that you could say i
want facial recognition uh with these
and these and these sort of quality
specifications and then other agents
would sort of continuously evaluate all
the apis that are offered and could tell
you well this api is good in this kind
of data and this api is good in this
kind of right once we solve the
description problem
these kinds of things become possible
but without it not so much but see that
but that's the key there is like because
yannick you know yannick said could
right there's that little word in there
and i've noticed this would i notice
this without ben which is that again
that because he's
such a vast knowledge base you know he's
easily flowing from stuff that's already
been done and proven to things that are
visionary and occasionally there'll be a
little if
that's in there or could right and if
you don't pay really careful attention
to that you'll miss the fact that we're
talking about something
hypothetical that hasn't been done yet
so i'm really glad that he was very
clear about
you know in some of those answers and
singularity net saying okay this isn't
there today right this is an active area
of work we don't have a solution to this
yet
um so that we know like it's we're not
there yet um
and this is going to be the really
really really difficult part i think
it's been an intractable problem for a
long time
exactly and i think another interesting
dichotomy is he's he's kind of like an
engineer he says he can engineer agi
i'm an engineer as well and engineering
to me software engineering is all about
immutability reproducibility testability
etcetera so i want to write some
software i want to write some tests
against it i want to make sure the test
passed and i want to put it into
production do i want to put together put
something into production which is
dynamically changing and
and morphing god no i mean that's that
sounds like a nightmare to me
sounds like fun to me it's a nightmare
of possibility
but even when we design ml devops
systems or anything like that we we
build the model and then we come up with
a load of model validation tests and so
on behavioral tests we decide that this
thing's good to go and then when it's in
production it's immutable we have
versioning and all of the customers are
calling that version of the model we do
not change it i mean for god's sake
could you imagine if i just changed the
model when the customers still thought
they were calling version 1. i mean
we'll have to we'll have to change maybe
a bit of what we expect out of like
we're not gonna we're not gonna use
a singularity net in the future for you
know doing our excel spreadsheets like
all of the things we do today will still
be handled largely by
you know classic software i think we're
going to have new modes of getting
serviced by the internet with some of
the stuff we want to do where it is just
going to be an acceptable mode of you
know this
this is kind of what the network it's
it's more like if i go to 4chan and and
ask around to get help with a problem
right
i'm not um sometimes i'm going to get a
good unexpected answer better than i
could have come up with myself sometimes
i'm gonna get some trolling sometimes
you know like
you're gonna be different this is really
interesting though because i think that
what you just said there yanny that is
the mindset of a lot of people in the
deep learning community right now that
is the mindset of people who think that
we can use gpth3 in a business setting
and and i think we can't
right because i want to be able to test
things and i want to understand its
behavior and understand its inner
workings whereas another mindset is kind
of like well you know let's just sample
this thing and you know it might work it
might not work do you know what i mean
sure but if i mean if you don't if you
don't want to use a product based on on
gpt3 then don't like then don't use it
right i mean that that's
uh the question is would anyone want to
use it would anyone want to uh pay for
you know a service that comes from
singularitynet where you can't exactly
be sure what you're getting any
particular day i don't know but
well that's interesting because if you
think about it um
there are some use cases where it's
acceptable so we all agree creative
think fiction and art that's fine
actually information retrieval was fine
as well and that's because you have a
human operator on the end and the human
operator is kind of interactively
interpreting and querying this thing but
anything non-interactive or anything
which is safety critical or anything
which we we need to understand it's in
the workings then i think we couldn't
use it interesting sure yeah no no i
mean
yeah of course but that's the same today
right if you if you program a software
and that software even though it's
programmed in a classic method can't
fulfill the safety criterion
we don't use it
i do i do think it's just it's going to
be a new mode of operation where people
will understand like as long as people
understand what you're getting
uh it's not that it's not like classical
software and they're willing to accept
that then why not well what about the
gpt3 suicide example
well if people if people think of this
gpt3 therapist not like a real human
therapist or a piece of infallible
software but like what if i went to
4chan or to reddit
and asked what should i do there are
going to be some
few
responses saying please go you know
please go away
you know please leave the earth
um
same thing
well yeah i mean maybe the problem is
just that we anthropomorphize ai too
much and we don't have that that way of
thinking about it
or too little right if i actually
anthropomorphize gpt-3 as a collection
of the whole internet
um
then i'm going to be clear on hey the
whole internet actually contains people
that are not very helpful as well
plus i think part of the idea is too
that
you know if we have
if we have this better platform you know
singularity neck plus let's say open cog
and whatnot we'll end up with better ais
like ais that that won't suggest you go
kill yourself or
that that sort of flawed ai will be part
of a system and then downstream there's
kind of like a sanity checking ai that
understands human emotion or something
and goes like
i i don't think you can you can say that
to this patient you know like i mean we
even have that today right which is like
there's plenty of um
let's say popular figures on twitter
that people wish had you know just just
an admin that would kind of double check
their messages before they go out right
so i think the idea is we can build
better systems and then of course
there's going to be extensive testing
and that's all part of like
okay this little part of the graph over
here it's really been thoroughly tested
for self-driving now for something like
the equivalent of
you know one million driving hours or
something we've done in the last week uh
okay it's ready to
to go to the first ring of users and
then and then so on so all the same
engineering all the same engineering
paradigms
can be applied to this it's just that
the only the the kind of upfront design
work uh
won't be the same yeah and and i think
it's really interesting what he's done
as i said as a as a software engineer
i'm really really intrigued by what he's
done and we've been speaking to quite a
few symbolists recently and it kind of
feels that he's actually done what
they've been selling us on
it's open cog
you know he's come up with all of these
different
um categories of knowledge and ways to
represent them ways to take many
different diverse ai algorithms and get
them to um to talk with each other using
common representations that's exactly
what they've been talking about doing
yeah but you got to be pay careful
attention to those little qualifying
words in there like if could
you know et cetera some of it's done
some of it's not done
i think the limiting factor is not
whether it can be done or not it's just
whether it will work and there's
i think he said that the existing system
was written some time ago so there's a
computational
problem with scalability
as we were asking him i think
scalability is always going to be a
problem when you're traversing these
graphs you have a an exponential blow up
but the bigger question though is to
what extent
could
this kind of emergent interesting
dynamics happen
i can see how it happens in the brain
and you know in a corporation but if you
have a society of minds where the minds
are discriminative ai algorithms how are
those minds going to
create a kind of chain reaction an
interesting dynamic so much
well
yeah
that's that that's more of a
belief i think i mean
emergence to me it was always kind of a
magical term where you just say well
things come together and emerge and
my question is i've no doubt that um
emergence happens and and and if you put
stuff into systems especially open-ended
systems then maybe some complex you know
behavior will emerge
though why
like
the transition from something will
emerge to ai will emerge that is my my
kind of main contention with the whole
with the people that just kind of say
okay things
and then say emergence right like give
me a tangible reason why the thing that
emerges should be
intelligent because you know in like you
look at ecology or so
and lots of systems emerge right and
sure you can you can call like an ant
colony intelligent
maybe right
you know you can maybe you can make the
argument but even
absent and colonies
most ecological systems aren't by any of
these definitions intelligent yet
they're incredibly complex right they
emerge out of the behavior of so why
why emergence to me
it doesn't mean
intelligent intelligence will emerge
like something will emerge doesn't mean
intelligence will emerge so i need an
additional reason there i i think that's
a that's a philosophical cul-de-sac
right you i i think that um
our planet is an intelligent agent you
know james lovelock gaia theory he kind
of says that you can think of all of the
ecosystems on the planet as being a kind
of intelligence so whether it's an
intelligence or not i think we could we
could debate that but um it is a complex
dynamic system and maybe you could think
quite generally as kenneth stanley does
that intelligence as a kind of
information accumulation
right but i think the problem is
brittleness because when the primitives
are because he has this kind of type
theory
system of how the agents talk to each
other and it's so specific the reason
why you get emergence in the brain is
because there's no brittleness it's very
very easy for the neurons to communicate
with each other and it's very very
composable but when your modules
expect a very specific type of contract
with a very specific type it just seems
to make it less likely that you'll get
complex behavior emerging
you know over actually really through
the process of the show you know like
say talking to people like
fristen the free energy principal and
others since then
an answer to kind of yannick's question
um i think that
intelligence
almost necessarily from any environment
in which
uncertain processes threaten to
eliminate your code
so so in other words in order for for an
intelligent thing to survive and
replicate itself it has to have some
measure of ability to predict the future
and to predict the future in a flexible
in a flexible way otherwise all this
stuff that keeps bombarding it you know
for us it's
physical stuff okay in the case of
singularity.net it's a marketplace of
demand so varying demands are
bombarding the market right and there's
this cryptocurrency economy in there and
so if you don't accumulate enough wealth
to do your processing and survive then
you'll get fewer ratings and eventually
you'll disappear or you'll be in
a dead node a dead zone with the uh
of the marketplace right and so i think
as soon as you set up an environment
where
there's
uncertain changes to the environment
incoming you know threats if you will
and you've got entities that that can
replicate and survive
i think you're going to wind up with
things that emerge necessarily that have
some ability to predict those uncertain
you know changing environments and to be
able to adapt and then that meets the
criteria of some definitions of
intelligence
so i mean the
i'm not sure that
you could think of the
like the earth
without
predicting
like without agents who agents that
predict like like like bacteria are
enormously efficient
at adapting to the various challenges
that that wanna that are uncertain and
want to kill them
without the ability to predict like i
can see the argument that you'd say well
you know on earth after all humans
emerged or or even like some
like uh let's say
big cats or so can also predict the
future to a certain degree they they
emerged but i can imagine a world where
evolution just doesn't pick that strand
of future predicting things in the same
pressures so i think where i where i
kind of would challenge that is that i
don't think prediction is binary zero or
one i think there's kind of a continuum
of prediction and i would personally
argue that
that bacteria as that as a bacterial
system are predicting you know they they
they receive some inputs and they have
some coding in there that tells them i
need to swim this way
right it's not it's not they're not
consciously thinking so i'm not saying
they're they're consciously
reasoning but they are a predictive
system in the sense that like when
something touches their flagellum or
whatever
they've
learned or been you know programmed by
evolution to swim in the other direction
and that's because in the environment
when something touches your flagellum
you know maybe it's trying to eat you or
whatever so i think they they are you
know just we have like
very tiny you know mic like a microchip
that has you know 20 circuits that would
be considered 20 components that we
would consider like predictive right
like various control circuits and
and whatever so it's just a
it's a very kind of
simple and minor form
of prediction and then at the other end
of the scale are people no i don't think
the earth is predictive like i mean i
don't think the earth is a whole the
giant ball like it's not
an intelligence because it has no
ability to control or react or predict
its environment at all it's just
following the simple physical laws you
know if an asteroid big enough hits it
it's going to melt and and
and end up with just a you know new
volcanic surface or whatever or
you know
frozen lava surface but so my point is
just it's a spectrum and i think that
you know bacteria are a minor minor
minor form of you know predictive
machine
let's just quickly finish off with aix i
this is something that um
rightly or wrongly i don't know why um
i'm skeptical about this but aixa
depends on having a computable
environment we've heard so many folks
talk about intelligence from a different
perspective whether it's hawkins
thinking about it as
a general learning algorithm or people
like
mark bishop
denying computationalism
or people like melanie mitchell talking
about embodiment for example
i personally really don't like the idea
that you could describe an intelligent
agent
as
literally finding the best computer
program that gives you the best reward
whatever the hell the reward is on all
of the previous situations you've been
on i mean
because the result is just a computer
program that i mean yeah it has to be a
simple computer program but in the worst
case it could just be almost a
memorization saying like a hash table
saying in that situation you should have
done this and in that situation you
should have done that at least francois
chile is talking about generating a new
skill program in that situation in
response to novelty but it just seems so
um reductionist just thinking of
intelligence as being a static program
in that particular situation it's it's
debasing my notion of intelligence
i i don't i mean maybe that's a personal
problem i don't know i'm just kidding
with you like a
i don't know
i guess because uh maybe because
you know over the years i became very
interested in kind of like different
theories of of time you know like
diacronic temporal identity or that sort
of thing i'm not so worried about things
being like
static per se i totally could buy the
idea that a black box could land on the
earth and be
just a bunch of circuits
that will never actually change and yet
i would deem that that box to be
intelligent because it's
you know just uh built by some hyper
advanced um civilization and it's just
that good for uh the majority of
problems so i think that's kind of the
point of aixi is that this this black
box
even though it's static it's so extreme
in its uh
resources you know i.e you can just
infinitely evaluate a bunch of infinite
things and ensure once you get to
infinities a lot of weird stuff happens
because at that point
i mean aixi is so extreme that
you know
its own evaluation of its own
probability of existence would be zero
because it's actually not a computable
like aixi is is implementing an
algorithm that's uh
incomputable because of this sort of
infinities right and so and yet it
assumes that the environment and the
reward function is a computable
computable function so i think that's
why it seems a bit weird it's just
because we're dealing with the
absurdities of
of infinities and i think if you
you know if you if you actually have to
do something with limited resources
you can't do it with that that type of
very static
you know thing you've got to have
something that that adapts and forms new
connections and
and whatnot right but but then it gets
to the point of
it's it's just um it's a mathematical
curiosity
it's the same thing as touring
completeness
right who cares
no no because the
so these things
um
all these kind of like limiting cases
they're very very useful for the thought
process and and design like in the case
of in this in the case of say
turing completeness or
aixi or whatever um
well in the case of aixi what it tells
you is that the idea of reinforcement
learning that's why i liked i liked so
much his thing that it's the reductio
absurdum of reinforcement learning you
know it would tell you that okay if we
did have infinite resources this would
be an optimal
system that solves every reward problem
as well as any other
you know reward problem could so then we
might know okay then it should do pretty
well even if we're only doing a limited
resource version
you know we might be kind of close to
optimal a lot a lot of ways like
how is it
it's like saying oh if i could speak to
an oracle and he could tell me exactly
what to do in every situation that
that's not intelligence
no no
i'm not saying
that this is a good definition of
intelligence what i'm saying is that
these type of limiting considering the
limiting cases
are very useful
pragmatically even though they're
theoretical constructs i don't think
they are so similarly there's a first
step fallacy in ai which is that you
know assuming that things that we're
doing now if you keep doing them they'll
get you closer to ai but i think the
first step fallacy also works in reverse
so if you go from this ridiculous um
notion mathematically of aixi and
thinking that we can step backwards from
that and it would give us any reasonable
intuition about how to build ai i think
that's flawed
i mean you might be right but let's take
a simple example from one of our prior
shows that we know is not really flawed
you know the the multi-armed bandit
were uh we were talking about if you
have these results these asymptotic
results that have optimality okay so as
n goes to infinity this this procedure
is optimal what use is that in practice
well if you can show that those limits
are
tight enough right if if they start to
become optimal with very few samples
like a hundred or a thousand or
something that even though that
algorithm we don't have a proof that
that algorithm is optimal for small
cases
knowing that it's optimal asymptotically
and that it has this kind of tight bound
is actually useful even in the practical
you know practical kind of scenarios
any thoughts doctor culture
i liked his take on it in that yes we
can learn stuff from it even even
and he was
quite clear because when you read his
papers it's like okay let's view the
most general case which is this system
right that can just root give me any
program that works in all my prior
situations
and the papers read as if
his plan is to take that and then sort
of
bring that down uh sort of say okay how
can we make this but practical
right and and and there i was like um
and also when when we confronted him
with this reward is enough paper which
in essence seems like
you know in my mind it seems quite
similar to this aixi paper right it's
sort of in the similar spirit but and he
was immediately like oh that's silly
it's
i feel like
what he said today seemed completely
reasonable it's like look here are these
um
systems in theory right we you know
they're interesting thought experiment
we might be able to learn something from
them right mathematically however in the
real world we have constraints we have
you know we're evolved with limited
memory limited time and so on and the
systems
that intelligent systems with the
constrained might be of a completely
different nature than the intelligence
systems without the constraint and
therefore
that would be my criticism we cannot
just reduce these all powerful systems
to practical systems because i'm pretty
sure
like the actual intelligent systems
they're they're nothing like program
searchers he said in his um uh
patternist paper 2021 so the big paper
that he's just released
he said um at the one extreme there's
been the approach of starting with the
general theory of ai and then deriving
practical systems from this theory and
implementing them
and marcus hutter and his students have
been the best example of this approach
with hutter's universal ai theory
serving as a credible although debatable
general theory
theoretical agi approach and a number of
relatively practical proto-agi systems
emerging from it he did say just below
that in his paper that dr arthur franz
is the person who is
furthest down the road in
using hutter's ideas
um towards a general ai awesome well
this has been great guys thanks so much
again thank you
another episode in the bag and we shall
see you all next time
for
it's actually it's it's a very special
episode it's um
it's such a special edition i'm not even
gonna say what it is
are you gonna wear a funny hat no no
this one this one is so special we've
been filming on location
with some very very interesting people
and we've already accumulated a lot of
footage
okay yeah this one is going to be really
good
um although we're actually going to
publish jeff hawkins first so you'll see
that one but obviously this is after
jeff hawkins
right see you later folks thank you bye
thank you