Human-Robot Coproduction: non verbal sharing of mental models with AR/VR (Doris Aschenbrenner)

reactions and i look forward to the
discussion
um i hope everything works on the
technology side
you should see now uh my presentation
and
uh yeah maybe i just like to share a
couple of like insights of you from my
background
originally i'm computer scientist and
then i did my phd
uh within the area of human uh like
let's say
tailor maintenance a remote maintenance
for industrial robots
together with companies and a tno like
extra university institution and i came
to tudor for my post talk and
kind of stayed here and
my interest and that's maybe i dive a
bit in the domain context because that's
really relevant for my research
and i'm not sure how many of you are
already familiar with that so i try to
bridge the area of manufacturing with
human-centered
designer molars and may some of you i
already know i read
the names you are familiar with one but
maybe not with the others so
please i would try to kind of give a bit
an introduction to
more or less the problem so um
so you should see my video now and you
know that their
development of human industry can be
developed into four industrial
revolutions so mechanical
um revolutions um steam machine and
these kind of things
electrical engineering and reclical
introduction
then computers and automation and then
finally this is what we call now the
fourth industrial revolution where you
have automation
coming to a new level with artificial
intelligence and new types of robots and
these kind of things
and the interesting thing is um
yeah within that area also the
work of people change a lot so we're not
facing only a technological change which
is called in this fourth industrial
revolution for industry 4.0 but it is
also happening in other parts of the
world within different names for example
made in china 2025
um but we're also encompassing um well a
social change so
there is an aging society and we are
also
having uh some migration streams um
and here we have all these questions
about how
is this future work within manufacturing
looking like
this is getting much more um
this question is getting much more
interest at the moment so for example
the world economics forum forum or there
was also a very cool
uh conference from stanford
uh on on ai and the future of work i'm
not sure whether you
were aware of that otherwise i should
just share um
the link maybe and um i see
this kind of research within this
context of course not solving all of
this question
and the interesting thing for me is that
there are basically
four different types of future scenarios
so which you can only read in literature
and there's a very nice
unfortunately german paper who kind of
summarizes a bit of that research about
future work in the production industry
and they come basically up with like
four different streams the first one is
that the robot will take over
that's what you mainly also hear in mass
media and i think everybody if you have
been tortured with this kind of all the
robots will take over the world stuff
and there is also a contra scenario
which is more or less on the
okay within that new technology we also
can use this
new technology in order to come to a
more human
centered new type of organization and
these are the homogenic
um either the one when or the other
scenarios
and there are also other scenarios that
are discussed in literature
one is definitely that there will be
winners let's say in the higher
in the higher up um uh quali in a higher
qualified
region um for example uh yeah if you
regard
like our jobs in the end or i love this
quote
which says well there will be two types
of people in the world those who tell
computers what to do and those are told
by computers what to do
i think this this polarization scenario
goes in this direction
and and then there is also another
scenario which is also interesting to
have in mind
that stuff is dissolving and dissolving
so you don't have any boundaries anymore
with respect to space and also hierarchy
because of the strong modularization
so these are the two more or less
diversification scenarios
and my faculty has more or less
the um aim to design for our future and
if we want to go
in the envisions future that we also say
it's the preferable
future then we choose choose to design
for the scenario of these four
which also for our side is the most
preferable one and this is the the
second one where the humans are helped
by technology
which i call it among others as the
operator for the theory scenario
and what does this operated 4.0 mean
well you have this force industrial
revolution stuff is getting much more
complex
less transparent but we still have in
high demands of safety
and of course efficiency and the humans
and the robotics colleagues needs better
ways to communicate
with each other in order to make that
happen so apart from the factory 4.0 we
also need the operator for
zero which we envision here a bit in the
superhuman style
and how does it look like exactly
the the basic paradigm is that we have
this cyber physical production system
which is more or less the manufacturing
environment
and we have the human in the center in
the interaction with that system
and we have more or less technology
helping this human to be better
in his or her work and enhance the
physical capabilities
so this could be for example using an
exoskeleton
and then we have the enhancement
possibilities of descending capabilities
so that's where i talk a lot here
in this talk um about using augmented
virtual reality in order to improve
um on one hand ascending capabilities
but also on the other hand
cognitive capabilities but you can also
envision much more
uh yeah different functions than ar we
are in these kind of two realms
and uh one thing that is very important
to understand is that there are we have
like
technical challenges which are mainly
discussed so complexity
uh dynamics so that stuff is not
non-linear
and then we don't have a not transparent
situation of the manufacturing
environment
and but we also and these these
challenges with indian manufacturing
industry or the robotics domain are
very much discussed a lot
but people tend to only talk about the
technology
and if we regard on the theory behind of
a
socio-technical work system then this
looks like this so you have some kind of
work cell and you have some input coming
in you have some output
going out and you have of course the
task and the physical system involved
with the task
and this is what we call the technical
subsystem and a lot of stuff
is only like what you leading literature
at the moment is only focusing this
using ai for uh predictive maintenance
or something like that then it's kind of
like centered only on that
that part of the system but the system
is larger
we have the people with this cognitive
and social abilities
and we have the structure of the entire
factory or manufacturing environment
which is of course interacting a lot
with the technical system
and we of course need to focus also on
the inner dependencies in order to
really make the entire thing work
and that is something well i think the
designers among you
are kind of people that have something
to do with you in fact to say yeah well
that's logical that's what we always do
um but it's not entirely logical
especially in the manufacturing domain
there was a lot of stuff that was only
focusing on the technical development
and there are a lot of comp
opportunities if you want to use human
centered or human computer interaction
within these industrial environments
you have less training you might have a
higher safety
a quicker problem solving and an
increasement of the well-being
um and this comes more or less to our
guiding
questions which are a bit stolen by from
the dutch
research agenda for artificial
intelligence
so we try to design an augmentation
layer
so that humans and robots can
productively interact and understand
each other
and and we want the human to trust
autonomous system
and we want to enable task sharing so
mutual understanding
uh between both partners yeah in order
to come to such a nice
um yeah well handshake situation uh
where
it's not only the human doing the work
but it's also not only the robot doing
the work
so and what would we understand by this
human robot co-production which is
the framing that we had um if you regard
manufacturing environment this stuff
looks like normally like this so you
have a lot of
like sometimes dirty machinery big
machinery
and some robots that are encaged so you
can see the bottom here there's a robot
who has a safety cage around it and
humans are basically only able
to kind of interact with these big
robots from a distance
um and this is currently a bit changing
because there are these collaborative
robots which you also i think
already know and they are designed so
that the human can
clo work in close interaction with them
and we don't require any fences anymore
we can have direct interaction
t readily quicker programming and
the market is increasing a lot in this
area because these kind of small robots
can take
away a couple of like small
manufacturing tasks and they're much
cheaper
and yeah they're quite promising but
um we still have some stuff to dissolve
there
maybe as a kind of overview why this is
interesting or why the market is kind of
increasing at the moment
um if you regard um high
large enterprises for example uh
automotive is not a perfect example but
let's use automotive um you have a high
production volume
and you have different parts that are
coming from that have low
um uh that have low high production
variation so for example
i need a whatever car and i need a
specific kind of seat i need this
so the car itself comes with a lot of
volume
but the different components come with
low volume
so and this is making the um high
um uh the the large enterprises
um being enabled to automate part of the
production already quite quickly within
the
third industrial revolution let's put it
like that um
and they can do highly automated stuff
they can do high volume low variation
stuff quite well
and they have optimized the factories
for that um
but if you regard small and medium-sized
enterprises or
um also other people that do let's say
batch size one or small batch size
production
they are less automated less low volume
and higher variation
and this means oh we need a better human
robot collaboration on this low volume
area so how does it look like
with the human on one hand robot on the
other hand we have some kind of
interface in there
and i still stick very much to some
quite
old theory um from sheridan where you
have
where you say that you have a different
task from human
women need to plan past teach a robot
monitor
robot is doing the right thing intervene
eventually teach again
and then learn and that still is more or
less
the basic things that still are there
maybe they're a bit quicker than they
were before
and this kind of human supervisory
control
is um yeah using a lot of different
mental models
so i don't want to give you too much
in-depth discussion but you kind of have
a
mental concept of stuff how stuff works
and what is quite interesting that if
you have this kind of control chain
there are a lot of different mental
models that are coming to pass for
example
if you here see the different components
the human has
a mental model of how the robot will
operate
to this place will show a specific
representation of the robot
which is always only a picture and
depicts also
the mental model that the programmers of
the display or the interaction
software has then of course we have an
internal
mental model of the computer which might
be a bit different to what the human
actually sees and can understand
and everything which has been designed
as being a control panel
also has an embedded mental model in
there
how it's designed and how stuff would
work and the interesting thing
within manufacturing industry this is a
bit of a dancing bear problem
a dancing bear problem is well known in
human-centered
interaction theory so you're so glad so
if you look at a bear that is dancing of
course it's animal cruelty and we know
about that
but if you look at that bear and you
possibly you like it and you say well
cool the bear is dancing
and you're saying oh well that's cool
because you never saw a bear that was
dancing
um but if you regard yet human dancers
and you give a
or b values for that the bear doesn't
fit at all
this kind of classification but you're
still happy that the bear is dancing
because it's the only bear that you know
and this is more or less the same which
happens with human interaction
especially in specialized industry
you're so happy that something is
solving your problem that it might be
over complicatedly solving it but you're
still happy
hashtag sap or something like that um
and there are this is just the thing
that we are covering
and there are a couple of worker needs
within that area for example
of course human want to stay healthy and
the work
should be sufficiently demanding but not
too demanding the human wants to
understand
what's going on and how to control the
system
and of course on an even higher level
you won't want to trust the system and
don't fear that it kind of is overtaking
him or her
and feeling valued by the entire context
so a lot of stuff to couple and this is
only
more or less the basic layer is physical
ergonomics and we have
cognitive ergonomics and then we have
emotional aspects or what we call user
experience
um which is a bit more than that um
and here of course there should be
design methods for kind of making that
clear there are design methods from
other areas but they're not
that well established within the
manufacturing field
so coming to the overall research topic
that my
group and i'm trying to couple is to how
to design a hybrid human robot system
that is able to optimize
both workable being and the overall
system performance
to really come to some kind of handshake
working together situation
i quickly go through some related
research i think a couple of people will
know some of this um first of all i like
very much the trading and sharing
control
theory so that if you have a human
worker then you have a specific load
that that human worker is able to carry
and if i have a computer system i can
use that computer system in order to
extend
the capabilities of the human so it's
not
only the human load it's an increased
load by
having a computer taking part of that
job but you also can use the system
in order to relieve the system so the
load is the same but the human has
kind of some relief in there you also
can use it as a backup to the human
and but then also there are some fields
where you
say okay but not that many to be honest
where the system or the automatic system
is replacing the human
but with a less load because human is
much more capable still
than an autonomous system and i also
like very much the levels of automation
also this
is quite old but nevertheless especially
kind of refined for the field of
manufacturing
so in more or less it gives a great uh
yeah kind of um yeah difference
between the total manual um case
and the totally automatic case and it
kind of defines
some some more or less discrete areas in
the mean
uh while where you can say okay there is
kind of a
um especially we are interested in this
supervision
and intervene case and not not too much
in the uh
closed loop case and of course
there is a lot of classification on how
humans
and robots can interact here the
so-called levels of interaction
on the left side about the constellation
of the group so between humans and
robots multiple humans multiple robots
and on the right side more or less the
quality of the interactions so is
both are both players active in the task
is one only supportive maybe some
inactive but somehow present
or is there some kind of intuitive hand
over thing
of course that's where we're all aiming
for but it's really hard to design
and then you also have this level of
collaboration which is a bit more on the
physical side here if you can regard the
robot
and the human either they totally
separated that's the normal case for
nearly
all of the industrial cases that we are
currently also inquiring
and these kind of co-existing or
synchronized or even cooperation or even
collaboration cases so coming more and
more to this kind of
shared thing that's still quite very
unique because
um it's also with a lot of effort
involved within real industrial cases
there was a very nice phd thesis from uh
our uh yeah associated with the stuff
that we are now doing
which unfortunately he doesn't work with
us anymore but if you're interested he
also had a very nice
uh work on using this kind of
operator-centered production design
and you can can look it up if you want
and the other thing that we are very
interested in
in order to make this kind of
interdependent teamwork situation
possible we need to have legibility
so predictive ability between what a
robot is aiming to do
and this is has been proven to increase
safety
comfort surprise or lessen surprise to a
certain extent
increase efficiency and also the
perceived value
of the human worker
um and how do we do that and how do we
increase
what on the human side is happening on
the robot side legibility is more or
less incorporated but on human side we
want to do
situation awareness you want to kind of
get
the human to a point that he or she is
understanding what is going on
and situational awareness is basically
more as a measure for understanding
what's going on and it kind of is
defined in different levels okay i know
that there's a lot of
like discussion on that whether this is
a valid
concept but i like it very much because
it's really applicable also for my
domain
and on one hand you say perception i
would like to know
what is there i would like to kind of be
able uh
to identify all the critical elements
the second thing i want to comprehend
what is the meaning and the significance
of the situation
and then i also in order to plan and
to interact with each other i need to be
able
to project how the future state will
develop
and this is involved also with concept
of sense making i don't go into detail
here
and then later on also into sharing the
mental models between the human
and the robots also these kind of if i
know what a human or what the robot is
kind of aiming at
then it also will increase my situation
awareness
our specific focus is then to say okay
we want to design this kind of
augmentation layer for this human robot
co-production within the era of
manufacturing
and here i come back to the uh social
technical system stuff that i have
introduced earlier
so we still have this human and robot
cell with some input coming in some
output going out and we have these
like combination of the social system
and the technical system
and our augmentation layer is enhancing
the physical sensing and cognitive
capabilities
mainly the last two in order to come
from this kind of
normal human worker to our worker 4.0
and we have these two factors worker
well-being and work performance that we
want to optimize for
and the specific focus that i would like
to kind of
enhance here because other people within
my group are more on a cognitive side
for example or more on the physical side
i'm using augmented virtual reality as a
tool
in order to kind of yeah improve this
overall system
and to come back to these uh research
questions
and then breaking down these research
questions a bit so that you can have
some comprehension on
on what you're actually doing um so we
want to design a human
augment augmentation layer so that the
humans and the robots can productively
interact and understand each other's
behavior in context
and here let's break that down with
respect to literature and also kind of
the stuff that we actually can measure
we want to help with situational
awareness we want to help with sense
making
we want to help with decision making and
we want to help with the sharing of
mental models
so let's have a bit of a dive in
for example if you want to improve the
situation awareness uh what you could
do is like we also um are of course
interested in level one and level two
situation awareness
but mainly we are also very much
interested in having a level one and two
and then
the level three which is the projection
i want to know what the stuff is going
to do
and a very basic example maybe but quite
comprehensible of what is feasible
is increasing the safety by
projecting the future trajectory of a
driving robot
so here is the example study um
we have a person walking a specific
way and we have a robot where we know
the robot will have a specific
trajectory
and we have two conditions in one
condition we don't have a projection and
the other one we have a projection on
the floor
and uh here you can see it's based on a
video study
and you can see this is the video
material that participants were look
watching as
and here the normally the system would
uh
the the participant would be
interrupted to watch the video and ask
what he thinks or what she thinks that
the robot will do next
and you can see here we're doing these
experiments within the era of
semi-cell and yeah death was quite
yeah predictable so we had different
scenarios
of different interaction scenarios and
you can quite see if there
are specific type of scenarios it's
quite
really helpful to have some kind of
projection in there with other scenarios
and this was on scenario four for
example this was that the human is
actually
doing a task and then the robot comes in
we don't have any significant difference
um but and that was really really nice
to just see okay what can we do
in real world on one hand but also and
within respect to these
sitting away situational awareness and
the other example
is not with driving robots but with
moving robots collaboration robots and
here you can see
that we have made the task a bit up
because yeah in order to have it more
controllable
there is a person packing stuff into um
for packaging and part of it should be
done by the person
part of it should be done by the robot
and that's more or less the same
like more or less similar setup than the
first study
and um what we're doing here
we're using the same
situation in virtual reality so here in
virtual reality you can more or less
also say
let's switch on a perceived future
trajectory of the robot for example here
you can see that small
um a moving uh trajectory
so that there is some kind of projecting
possibility of the future and of course
you can have a lot of different
visualizations for that
um and this helps you to understand okay
what will the robot will be doing next
and the nice thing is that we're not
only able to do that in virtual reality
but we also can use
augmented reality for this and here you
can see someone putting on the microsoft
hololens
and we have developed some nice
framework where you have
um you can see the robot moving and also
on the left side also the
virtual robot moving uh we have more or
less a framework where you can have
all the stuff that you were seeing in
virtual reality it's developed in unity
and with that kind of feedback framework
to the robot operation system
you can have the same visualization
stuff also happening
in augmented reality and the question is
here and it is unfortunately ongoing
sorry for that
um which kind of visualization would
help and in which scenario does it help
does it help in real life
um situation does it help in the uh
only virtual reality environment um
when yeah where are the benefits where
did you get the biggest benefits for
this kind of situation awareness with
respect
to understanding what the robot is going
to do next
hey and where do we apply that then this
is more or less the
um the the last case that i want to show
you
and this is an application where we have
we work together with the robot with the
bicycle manufacturer
and the idea is to share tasks within
bicycle manufacturing between human and
robot because there are some tasks that
are really
not easy automatable and how are we
going to kind of do this kind of task
sharing
and if we do discuss sharing how are we
going to communicate
the task sharing and the stuff between
the human and the robot
yeah this is much more to it so we have
done
quite a stuff a couple of stuff so far
that
we have developed a digital twin of the
samix l
environment within unity so that you can
use that for experimentation
we have designed some kind of control
room features within unity
for some xl which we are hopefully
implementing somewhere in the future
also there in real life we did a couple
of studies on automation
capabilities for this bicycle case uh we
did a couple of papers on using
augmented virtual reality for helping
within the field of manufacturing
and also for planning manufacturing
tasks if you want to read more
i'm totally happy to share also some
more examples later
in the discussion but i just wanted to
conclude it
here um yeah this is everything
is only possible because we have such a
great team
um and uh all of that work is no
not the work of someone alone it it
always is the combination of people
and i have a wonderful team i'm so happy
uh that we work together
um i have a quick video i'm not sure i
need to look at a time
which i could share with you of like
more or less all of the projects that we
have going on right or now
but i'm not sure if we have the time or
we will not start the discussion first
thank you very much for the attention
and um yeah if you want to look to
watch the video we can also totally do
that thanks
thanks doris um is there any immediate
like
question that people have for doris or
else i'm actually quite happy to see
more examples because i think that
they're great
so uh that's actually quite exciting
so yeah doors can you maybe quickly
yeah i guess yeah okay okay just like
try to do it because i try to have it
running with sound which is
um hopefully working
uh no that doesn't work because if i try
to
upload it that is not working
okay um yeah let's see if it works if
you get
if you don't have sound please let me
know
sound no we don't hear anything so you
have to narrate
yeah i'm sorry and that's also something
i could do but they have such a nice
sound
um okay let's see if it works now
do you have thought now hey hello
welcome to this virtual tour of our
research projects at some excel
the research we do at some excel is
focused on the future of manufacturing
and sustainable human robot interaction
this is our team and we all welcome you
we're excited to tell you about our
research and show the great facilities
at some excel
hi i think most of you already know me
i'm your friend
i work at the applied labs but also work
here at some excel
and here at some iselle i helped to
develop all of the research facilities
for our projects and it also helped
bridge
the research we do here at sun excel
with the research we do at the applied
labs
so let's have a look inside
so this is the main hall of some excel
so the raw ones can be found here
it's 2 000 square meters robots
and very cool projects and in the combat
area
we have the robofish project in the
robofits project
we're helping a bike manufacturer to do
production of bikes with cobalts
let's have a look at some more projects
we do here in the combat area
hello there i'm jonas i'm an xr
developer
my primary work concerns the topic of
digital
twinning this does not only include the
visualization of
cyber physical systems like robots
or agvs but also the development chain
behind it
hi this is elvis so over the previous
year
i've assembled and have been developing
the ros
composite pick and place workbench and
together with others i've been working
on tooling
so that we can visualize soft robotic
actuators in real time in ar
apart from using cobots we also do
projects with
mobile robots these power robots can
drive
autonomously around factories let's have
a look at that
hi i'm denis this year i'm happy to be
a member of two projects first one is
collaborating and coupled agv swarms
where we use mobile robots to improve
intro logistics and second one is
profits where we use robot arms to
improve bicycle assembly line
hello my name is martijn and i've been
researching the possibilities of
applying spatial augmented reality
in the smart factory context an example
of this
is to use projected arrows to improve
the communication and safety of
autonomous vehicles
hi my name is better caseman the koch
project my colleagues and i have been
working a fleet management system
called rooster brewster's goal is to
simulate schedule
and plan tasks for robotic fleet in a
warehouse situation
hi my name is nil naga i'm a controls
engineer on the team
and for the past year i've been working
on setting up navigation software for
multi-robot systems
so that robots like this one can be used
to carry stuff around
factory shop floors and warehouses on
another front quite recently i'm
involved
in extending the bobrov project which is
a robotic arm program
to paint
so let's have a look in the rest of the
sub example
here at some excel there's also a really
really big robot
it's called a gentle robot and it's
situated in this corner
let's have a look
this robot is huge it measures
12 meters in length 10 meters wide and
5 meters high different types of tools
can be attached to this giant robot
aerospace engineer will use it for
drilling riveting halls in giant
airplane wings
but imagine our faculty attaching a
giant 3d print
head to this robot then we will be able
to 3d print giant structures
prototypes of car bodies or even large
outside furniture pieces
all of these robots here at some excel
produce a really amount
of data it's hard to comprehend for a
human being
my name is samuel kernan i developed an
assistant that can
automatically generate a report based on
a conversation with a technician
this saves time reduced the perceived
workload and resulted in
reports of higher quality for my phd
we'll be developing an assistance that
can provide cognitive support to factory
workers
while they use analytical tools
my name is sudei and i'm gonna join the
department and the koala project as a
postdoctoral researcher soon in december
i've been working mainly on recommender
systems
since my master's thesis in stockholm
and then over my phd
and then my postbook in ap
to dealt see you all soon
hello my name is santiago i'm a product
design engineer and i am
participating in the diamond project as
a postdoc where we are developing a
digital intelligent assistant for
supporting the maintenance activities
of the maintenance staff at
manufacturing companies
all the data the robots create we also
have developed
the virtual world of some itself
let's have a look inside this virtual
some excel world
my name is danielle ponto and my work is
mainly focused on extended reality or xr
i work for the mirrorless project where
we create a digital twin
where robots can be viewed and
controlled remotely for this project we
create tutorials where we teach how to
use this digital twin framework
hi my name is irina and i am responsible
for a virtual playground community
it's a community that connects
researchers students and startups
interested in vr and ar technology we
have
regular online meetups with experts from
all over the world
and will be happy to see new members
hello my name is jasper and my calling
is teaching
which is why i'm here to make all of
these exciting new technologies
accessible to steve
okay so i think that's it that's only
the teaching program which i
kind of miss now yeah i hope you like it
and we still have another
i have more videos because augmented
virtual reality is always with a lot of
videos
but i hope you like to kind of give a
bit of an feeling what we're doing
this is great thanks uh doris that's
really really exciting stuff
it was originally presented for our
faculty because we didn't have the
possibility to show them in real life so
um that's the reason why it's a bit on
the ide side of telling stuff i just
wanted to kind of share that with you
because it gives much more tangible
feeling to it
yeah now great so with that
i was wondering uh if anyone had some
questions for doris like more like
about the projects that you showed
oh yeah i see that people would love to
uh if you have a link to this video
doors
apparently people are very keen to uh
okay yeah we have don't have it online
we have showed it
within this kind of uh these natalis
thing
so that's something i can definitely
show you um
but i think uh we should make a real
version for youtube because that was
only for internal purposes
uh we will do that um i i still hope
that it will come soon and then i will
share you at the link
cool very good so i actually have a
question to just to kick it off if
that's okay
um that was fine and uh so so actually
two questions because one thing that i
think is really cool that you're working
with an actual bike's bicycle company
right on this cool production
so i was wondering uh are you also
applying your um
external reality for those people so the
actual line workers
and and do you know how they like it
whether they like working with the road
whether
adding this uh layer uh makes it like
their work
there they enjoy it more i was just
wondering
yeah so so the real application cases
within the augmented virtual reality
domain
are mainly for other purposes like
especially for
maintenance tasks i did some old studies
for that
i can link them if you want so that way
we
actually did also a comparison on using
students as
participants for these kind of
applications and real workers
and the interesting thing which is maybe
not that maybe already obvious
is if you test these stuff with students
it's nice you will get some kind of
results but
in the end you really need to test it
with the real end user
and they will see the stuff entirely
differently
um so everything would be doing we also
try to really involve the end users
within the bicycle project we are not
actually there what we did within a
bicycle project if you want we can also
share another video sorry for that
and we built an envision scenario
for the kind of co-production within vr
because the problem if you want to talk
with workers on what they like and what
they
would prefer they actually don't have a
clue
what to long for and what robots are
capable of and how could this envision
situation look like
and we uh basically were using the
approach of
using the virtual reality environment
for
setting them into that future scenario
and then having a discussion with them
we did this together with robo house and
i think we officially release
it uh the video or something like that
soon
um and this is where we applied
human like human robot co-production for
the for the bicycle industry with
virtual reality
the main stuff i'm doing on uh augmented
reality assisted
things at the moment i was doing a lot
on maintenance and repair
tasks and i might come back to assembly
but at the moment we do composite layup
so what we do is
uh as you saw this kind of pick and play
stuff with the with the robot and using
the composites
here is where the industry in within the
aerospace domain
has a lot of interest
within the manufacturing industry my
main focus would not be the direct
assistance for single worker cases
i did a lot of cases for multi-workers
so
if you have someone on the phone and
collaborating with someone local
a couple of situations that we have a
lot at the moment with due to corona
and this is where i'm it's called
computer supported collaborative work
there is where i'm uh yeah have worked a
bit more
because like at the moment everybody is
doing repair and instruction
maintenance a kind of suggestion stuff
and there's already a lot out there in
the industry
and so it's it's not that interesting
anymore because it's like a lot of stuff
has been already discussed there so only
the more complicated cases
multiple humans or humans and robot
systems
or what we will do in the koala project
uh this cognitive advisor
the um the ai system giving you
cognitive advice so that is going to be
a bit more interesting than the
uh like normal okay i know how the
instruction works and i give you some
some tasks if you want to read something
on that i have a very large comparison
study
um with 150 participants um
but i'm not sure if i want to do that
again and uh
on kind of how ar and vr can ar
visualizations can help on uh this kind
of
instruction based uh stuff cool yeah
no definitely so if you can share it
afterwards uh yeah i'll uh
i'll be happy to um luciano
hi ladies thank you very much this was
really fascinating uh presentation
and i was uh i really like also the
example
you show the project you show about the
robot projecting
the expected trajectory on the ground
and
yeah i can imagine that really helps for
the operates for the people to
understand have a little bit more mental
model of what the robot is intending to
do
uh and i'm just thinking
a little bit about this interplay so as
soon as you give this notion so maybe
they're gonna feel
the operator might feel feel a bit more
comfortable and get like closer to the
robot and that that could be like
this kind of emerging interaction
patterns on this and that's
maybe the the so the menthol
that helps the humans to form the mental
model but how does it also
helps the robot from the mental model
and the adaptation
on that so you have any thoughts on that
direction
yeah so so the interesting thing is with
the autonomous ground vehicles within
factory interlude
logistics this is a field which is a
rapidly evolving at the moment within
the manufacturing industry
so a lot of questions that we are
discussing currently within the
autonomous driving community
is kind of entering the factory like on
through the back door so a lot of
questions that we have on the normal
street interaction stuff is kind of
entering now the manufacturing world
so the question i think what is
important to know is that autonomous
ground vehicles are not entirely new
to manufacturing they are quite common
actually
um but they are not self-steered and
kind of like swarm-like behavior
they have their dedicated routes they
have the very very strict safety
routines on
stoppings is there is any obstacle they
need to stop and these kind of things
and they are interacting at the moment
quite predictable
so um because they have this kind of
lines more for example on the floor
which they are following
and these kind of different passages and
factories
are designed and like compared to
streets factories are designed in such a
way that humans behave as part of the
machine so there are
very strict rules on how humans are able
like
allowed to behave and on these rules is
quite easy to develop all the rules for
the robot so it's kind of a very
rule-based and of course safety critical
environment
um so the real interaction thing is we
would normally imagine it with like
autonomous cars or something like that
at the moment don't really arise because
currently the systems are not really
self-controlled
if they are getting and that's a very
nice vision that we have developed
together with magna magna is
a car manufacturer in austria who
are using these autonomous ground
vehicles and want to use them in a
self-organized fleet
and here you start to have this
controllability of the
ai system because this is like the
system is
independently going to decide what it
wants to do and kind of self-organizing
what it's the next steps will be and
here and that's the point where
interaction gets more important that's
why we came into that project
we're not that much the like typical
robotics engineers we are much more on
the
yeah still rule-based interaction and
uh two things that we have inquired here
is like we have
a couple of different scenarios that we
wanted to look into
and one is definitely the close
interaction on the shop floor and here
is more or less the main question what
you have with human steered forklifters
like the interaction with humans walking
humans human steered forklifters
and atvs and here the main thing is i
want to know what the thing is going to
do next so that's why we came up with
this projected trajectory thing
um i think it gets more complicated if
you have mobile manipulators
because then you don't only have the
robot driving
but you also have an autonomous part
which is able to manipulate stuff
and this is going to make the stuff even
more interesting but we're not there yet
let's put it like that so it i hope that
answers your question
and i can share a couple of links if you
want with the case that magna envision
it
and they build also nice videos sorry
this is just a bit of the industry
domain they always make videos
um and i will share the smart factory
version of magna and this is quite
interesting actually
uh where they also see a couple of uh
and here without and that's maybe one of
the topics that i want to kind of really
raise
is um that compared to the traditional
manufacturing
uh to the traditional manufacturing from
the employees where
nobody really needed to take care that
much of the human
outside of safety um constraints
um now if the stuff is getting more and
more intelligent we really need to take
care about the interaction and this is
quite new for that field
i hope that answers your question that
does there's a lot probably
so thank you very much thanks thanks so
i saw that akari racism too and i'm
david
so yeah i i just wanted to say it's it's
amazing
yeah i like especially this example with
the
ground kind of wheeled robot yeah
because i'm myself doing uh
human av interaction with the thomas
vehicles so uh we also have this kind of
really nice uh analogy there i think but
uh yeah
my question was basically the same one
that luciano's but i want to elaborate
on that
uh you mentioned that you're interfacing
already now with the
autonomous vehicles uh industry and uh
well the way i understood it is that
they are trying to bring some of the
approaches
uh that they're using into the workflow
right but
i was uh also interested if uh some kind
of uh
ar vr based interfaces are already being
used uh
in autonomous vehicles uh interactions
interacting with humans
and uh yeah if you have any plans of
going there at all or maybe you just
know of any relevant work
yeah i'd be happy to look at the
references i have a literature survey
for that if you want i have a graduation
project
and you also can have our code and we
also have the fleet management code that
we published as students
you can also see if that helps you to a
certain extent
um and yeah let's just start to get in
contact because we're currently
proposing to
put the stuff into an age horizon europe
um
attempt together with magna because yes
i see that the market is increasing a
lot within the fts or far loser
transport statement that's the german
world
or the autonomous ground vehicles or
autonomous guided vehicles within the
factories i am not quite sure why they
took so long um because there were a lot
of systems already on the market
but there had been really a recent push
uh for that and new
also new standardization and these kind
of things if you want to elaborate
further on that we also have an
international working group together
with other universities
on the topic and if you want to
participate i'm always happy to have
a new people joining us there
okay sounds perfect let's let's get in
touch
cool all right thanks zakari
david
where's that button yes
top right no i hate to change between
different
uh systems and i'm always searching for
this two functions again
it's not working no maybe uh try the
button
no also not
oh okay you're gonna type the question
perfect
okay give him a keyboard you want to
talk to us
perfect interface
okay maybe also restarting sure
okay yeah how do we evaluate that that's
very they're very good
very very very very good question um so
um situational awareness is more or less
horrible to measure
and there are also some psychologists
that disagree with the concept overall
um it has been proven to be very well
helpful for the aviation industry and
also for
military contexts and it has been
applied a lot in the manufacturing world
i am mainly using um the
situation awareness rating technique
which is a very
brief questionnaire um but to put it a
bit on the context i like i don't do a
real sagat approach which is the the
original enslave approach
um but you saw that we for example use
this interruption technique that we are
interrupting on a specific point in the
interaction which people are seeing on
the internet in in the video
and on that point we are asking the
questions on
legibility related questions where we
don't have standardized questions so
like what do you think what robot will
do next
if you want we also can share the entire
study with you i haven't published it
yet but
yeah i should um but there is a
graduation
which has a couple of these kind of
tasks in there
and then we of course also use um
experience usability so there are also a
couple of standardized questions that
you can
questionnaire that you can use from the
usability side
and if you want to evaluate more into
depth for presence
related things if you really are in
an augmented reality or virtual reality
setting then you need to
um also evaluate on presence
and sometimes also really important to
measure the immersive tendencies
beforehand so i have a kind of set of
of questionnaires that i tend to use
including then also
task load you can use nasa tlx you can
also use other methods
and if you want i can if anybody
interested i can just share the
methodology that we are
kind of now using most of the time
and i want to add that i'm only using it
as a tool
and i'm not doing research on these
methods directly i'm just like using
them how they are recommended to be used
within the human factors domain
and i find it much more valuable to use
standardized questionnaires as much as
possible then you can really compare
also to other
applications if there are really
but for the manufacturing industry the
most of the questionnaires are not
really validated for that application
case so they are validated for
specific parts of the application cases
um
for example but the most of the ux
research is definitely
on let's say screen based interaction
and we
do much more than that right i found my
my microphone button again so um
so that that seems very uh let's say uh
really usability kind of questionnaires
and usability kind of uh
of methodologies i'm familiar with um
with all of them but uh so i was
wondering if you
look into well-being then those kind of
questionnaires they don't
typically address that and also if you
look into
the question that that concerns us at ai
tech which is uh
you know do people actually feel
responsible
when they work with these kind of uh
robotic systems you know that
uh then then also we need other things
than the the standardized questionnaire
so i was wondering if you could reflect
on those two elements there's a
well-being
and and responsibility over what
actually happens in the production
process
yeah so we found when we respect to the
well-being um
my chair is a professor peter fink who
is
uh his uh yeah more this special area is
comfort
and uh well-being within the context of
uh
flights but he originally comes from the
work economics domain
so within the work ergonomics there are
a lot of like measurements that you can
take
especially for example now for the
bicycle case we did a ruler analysis for
the current status
so this is also standardized method that
you can use for physical ergonomics
um and for in order to assess the
quality of the physical interaction
like before and after treatment let's
put it like that we use these kind of
standardized methods there are existing
comfort questionnaires for example i did
a study on
uh comfort on on the sense glove
stuff for example i think you know it
with with with then
and these people um and here we use
there are also some standardized
questionnaires that we're using there
and within cognitive ergonomics we work
together
with chalmers university and they have
done in tremendous work
on cognitive ergonomics within the field
of assembly
and they also have very nice methodology
which we applied for example also again
a cxi that's also a complexity index
which they derive from the
automation from the assembly tasks and
here we
look into more or less also
pre-treatment after treatment
so first for the analysis of a task for
the different levels of
automation and how the automatability
and then after we have finished but we
haven't done
a really finished task yet and
then the uh plan is to look into the
finished
the new version more or less and
compared it to complexity index and
perceived cognitive ergonomic factors
so i'm mainly relying on existing
methodology here
because i have the feeling there are a
lot of people doing great work there
which i can use then as a tool and and
then
rather kind of focus on making the stuff
work and seeing
if our design is really kind of
resulting in some improvement instead of
kind of doing research on the
methodology itself
if that answers your question uh partly
i mean i get your choice but um
but my question also was uh
basically in terms of responsibility
what do you what do you think
we could use should use what we need to
develop because it's not there
i understand you don't do it but but
yeah just reflect
on that so what we have is what we
definitely have is
um we use the virtual reality setting
for the bicycle in order to kind of
use this as a tool within a responsible
research and innovation approach with
claudia
from tbm to use more or less the vr
envision setting within a methodology
setting
of research responsible research and
innovation
and they have kind of a bunch of tools
on
making sure that the workers values are
well captured
and then embedded later on in the system
and that's something i find very
interesting and very relevant
another study that we do uh yeah sorry
nick i see it um
and the another study that i was doing
was asking
robot developers out in the industry
if they are considering human factors
and the end user at all
and because that's something where i
wanted to start in order to just
justify the need at first because there
are a lot of people within the robotics
domain which you might know and
better than i do but also in the
manufacturing domain who don't really
see the necessity yet
so my study what i was doing and i can
share the methodology with you if you
want
is to go out and ask robot manufacturers
in or robot builders within project
context
if they consider these kind of typical
user-centric approaches that we
use as a methodology and the answer is
basically no they don't have any clue
and they don't think about the end user
and i think that's the point where i
tackle this kind of responsible approach
if the developers don't care about it
then we cannot fix it like afterwards
that easily
good thank you great
thanks uh thanks everyone thanks doris
also for uh
for this really really inspiring talk
it's great and um
so yeah like i said in the chat i'll
talk i'll talk with you to see how we
can share these references that you
mentioned
the best with the people that are
interested and thank you very much for
uh for the for being here
everyone else for for uh for you for
your
attention and uh well we'll see you next
week
thanks so much and please send me an
invite innovation in invitation next
week i'm so curious to see
the presentation of your other guys
thanks definitely see you bye