Pepper Leolani and Vossen @ HPE Reimagine June 2018

 

Tijdens HPE Imagine: Recharge Your Brain-Everything Computes @ Reimagine 2018! geeft Piek Vossen een presentatie samen met zijn Pepper-robot Leolani, Barneveld, 12 juni 2018.

Tijdens de lezing laten Piek Vossen en zijn studenten aan de hand van zijn robot Leolani zien wat een robot ziet, hoort en denkt als die met mensen communiceert. Vossen opent de wereld van de robot voor de ogen van het publiek en laat zien hoe moeilijk die communicatie is. Taal is voor mensen zo natuurlijk als ademen maar voor een robot is dat een enorm probleem. Hoe gaat LeoLani om met de informatie die ze krijgt? Hoe zeker kan ze zijn over wat ze ziet, hoort en wat we haar vertellen? Vossen probeert robots te maken die uitgaan van onzekerheid en imperfectie. De perfecte machine bestaat niet en daarom is het zo belangrijk dat je met machines en robots kunt communiceren over elkaars gebreken. Deze imperfecte, maar autonome machines en robots die steeds meer deel gaan uitmaken van onze maatschappij staan niet stil als een frisdrankmachine maar bewegen rond, leren, maken zich dienstbaar, ontmoeten andere mensen, staan in de weg of raken misschien wel in de war. Er is nog veel onduidelijk over de invloed van robots op onze samenleving. Piek Vossen schetst in zijn lezing een realistisch beeld. 

Our first Pepper robot ‘Leolani’ publication

Our first Pepper robot ‘Leolani‘ publication:

P. Vossen, S. Báez, L. Bajčetić, and B. Kraaijeveld, “Leolani: a reference machine with a theory of mind for social communication”, invited keynote speech, in Proceedings of TSD 2018, Brno, June 05, 2018. Download paper.

Object recognition Leolani Object Recognition: Selene Báez showing Pepper robot Leolani a plush rabbit (see our repository for Pepper applications at GitHub).

Piek Vossen keynote speaker at 21st International Conference on Text, Speech and Dialogue (TSD 2018), Brno, Czech Republic, September 11–14 2018

Abstract. Our state of mind is based on experiences and what other people tell us. This may result in conflicting information, uncertainty, and alternative facts. We present a robot that models relativity of knowledge and perception within social interaction following principles of the theory of mind. We utilized vision and speech capabilities on a Pepper robot to build an interaction model that stores the interpretations of perceptions and conversations in combination with provenance on its sources. The robot learns directly from what people tell it, possibly in relation to its perception. We demonstrate how the robot’s communication is driven
by hunger to acquire more knowledge from and on people and objects, to resolve uncertainties and conflicts, and to share awareness of the perceived environment. Likewise, the robot can make reference to the world and its knowledge about the world and the encounters with people that yielded this knowledge.

Keywords: robot, theory of mind, social learning, communication

Visit our Pepper robot Leolani GitHub.

#1 Leolani introduces herself
#2 Leolani and Wolfram|Alpha
#3 Get to know
#4 Object Recognition: Plain
#5 Object Recognition: Corrected
#6 Object Recognition: Learning
#7 Make statements: one person “I live in Amsterdam”
#8 Make statements: two persons “Selene & Lenka own a book”
#9 Make statements: two persons “Bram likes science fiction movies”

Person, Bram, Laptop, Chair, Bottle Object Recognition. Frames from robot cam: Leolani recognizing Person (81%, 86%, 89%), Bram (100%), Chair (82%), Bottle (81%), Laptop (94% and 98%).

Video Paradisolezing “How to Communicate with a Robot”

Hier vertelt Piek Vossen, hoogleraar Computationele Lexicologie en Spinozawinnaar, over deze imperfecte, maar autonome machines en robots die steeds meer deel gaan uitmaken van onze maatschappij. Deze robots staan niet stil als een frisdrankmachine maar bewegen rond, leren, maken zich dienstbaar, ontmoeten andere mensen, staan in de weg of raken misschien wel in de war. Er is nog veel onduidelijk over de invloed van robots op onze samenleving. Piek Vossen schetst in zijn lezing een realistisch beeld.

Piek Vossen doet onderzoek naar het begrijpen van taal door machines, waarbij de begrippen identiteit (waar hebben we het over), verwijzing (hoe benoemen we dat) en perspectief (waarom doen we dat zo) een centrale rol spelen. Vossen: “Er wordt veel gesproken over de vraag of en wanneer robots ons voor een groot gedeelte gaan overnemen. Maar de perfecte robot bestaat niet. Probeer je zelfrijdende auto maar eens uit te leggen dat het om een uitzonderlijke situatie gaat waarin je je bevallende vrouw zo snel mogelijk naar het ziekenhuis moet brengen en dat je daarom gebruik wilt maken van de vluchtstrook en even tegen het verkeer in moet op een snelweg. Maar wil je zo’n imperfecte robot wel aan je ziekenhuisbed, of als de lesbuddy voor je kind?” Mens en robot helpen elkaar Wat doe je met een robot die er niet uit komt? Vossen: “Je helpt hem of haar, net zoals die robot jou probeert te helpen waar die maar kan. Een samenleving van autonome robots en mensen is nog steeds een samenleving waarin we samen problemen oplossen. Je gaat niet staan ‘debuggen’ maar je praat er mee. Problemen oplossen is immers een kwestie van communiceren.” De robot moet daarom niet alleen de computertaal spreken maar ook gewone mensentaal en dat is een grote uitdaging. Communiceren met een robot is als communiceren met een vreemd wezen dat in een andere wereld leeft. Waar moet je beginnen? Hoeveel kunnen wij van elkaar begrijpen? Wat is dat begrip dan precies? 

Tijdens de lezing laten Piek Vossen en zijn studenten aan de hand van zijn robot Leolani zien wat een robot ziet, hoort en denkt als die met mensen communiceert. Vossen opent de wereld van de robot voor de ogen van het publiek en laat zien hoe moeilijk die communicatie is. Taal is voor mensen zo natuurlijk als ademen maar voor een robot is dat een enorm probleem. Hoe gaat LeoLani om met de informatie die ze krijgt? Hoe zeker kan ze zijn over wat ze ziet, hoort en wat we haar vertellen? Vossen probeert robots te maken die uitgaan van onzekerheid en imperfectie. De perfecte machine bestaat niet en daarom is het zo belangrijk dat je met machines en robots kunt communiceren over elkaars gebreken.

 

Paradiso Lecture “TO COMMUNICATE WITH AN IMPERFECT ROBOT. GET IT?” March 2018

cof

Prof. Dr. Piek Vossen gives the third lecture in the Paradiso series:
 

Humanity 2.0
The fusion of men and technology: the next step in evolution?
 

TO COMMUNICATE WITH AN IMPERFECT ROBOT. GET IT?
So there you are. You try to explain your autonomous Tesla that this is an exceptional situation: you need to speed because your wife is in labor and you have to get her to the hospital as quickly as possible. Therefore you need to use the emergency lane and drive against the direction of traffic on the motorway. Big data and deep learning are not going to help you.

The lecture will be about imperfect, but autonomous machines and robots that will participate more and more in our society. They do not have fixed positions like a refrigerator or soda machine but move around, learn, try to be of service, meet other people, stand in the way or might get confused. Would you like such a robot at your bedside, or as a buddy teaching your child?

What to do with a robot that can’t figure it out? You help it, just like the robot tries to help you whenever it can. In a society in which autonomous robots and people live together, we also have to solve problems together. How do you help a robot? You are not going to “debug” it, but you talk to it. After all, solving problems is a matter of communication. Therefore the robot must not only speak computer language but also natural, human language, and that’s a huge challenge. Communicating with a robot is like communicating with an alien coming from another planet. Where to start? To what extent do we understand each other? What exactly is ‘understanding’?

By means of his robot Leolani Piek Vossen and his students show what a robot sees, hears, and thinks when it communicates with humans. Vossen opens up the world of a robot and shows the public how difficult communication is. For us, language is as natural as breathing, but for a robot it is a huge problem.

Leolani is a she and she has a hunger for social contact. She communicates with people but still has a lot to learn. She absorbs everything she hears and sees, but should she believe everything she’s told? How does she deal with the information she receives? How sure can she be about what she sees, hears and about what we tell her? Vossen tries to make robots that prepare for uncertainty and imperfection. The perfect machine does not exist and that is exactly why it is so important that we can communicate with machines and robots about each other’s defects.

Prof. Dr. P.Th.J.M. Piek Vossen is full professor Computational Lexicology at VU University Amsterdam, head of the Computational Lexicology & Terminology Lab (CLTL), co-founder and co-president of the Global WordNet Assocation (GWA), a member of the Koninklijke Nederlandse Akademie van Wetenschappen (KNAW) and a member of the Koninklijke Hollandsche Maatschappij der Wetenschappen (KHMW). In 2013 he won the prestigious Spinoza Award of the Netherlands Organisation for Scientific Research (NWO). Vossen conducts research on the understanding of language by machines whereby the concepts of identity (what are we talking about), reference (how do we name this), and perspective (why we do it like this) play a key role. In his latest projects NewsReader and Understanding Language by Machines machines learn to link language to the world. For more information go to his website: www.vossen.info and the Facebook event.

Paradiso lectures are organised by Verstegen & Stigter cultural projects and Paradiso, with the support of NEMO Science Museum, the Royal Netherlands Academy of Arts and Sciences (KNAW), the Royal Holland Society of Sciences and Humanities (KHMW), and VPRO broadcast.

The Heavenly Voice of Leolani – Visiting Piek Vossen

Media and movies like to suggest that robots will take over our jobs, the world and even us. But is this really the case? We ask VU professor and Spinoza prize winner Piek Vossen (Faculty of Humanities) a few questions. Piek Vossen conducts research on language with his robot Leolani.

The Heavenly Voice of Leolani – Visiting Piek Vossen

Are robots taking over?
Too much science fiction. Robots have been around for years, especially in factories, and did us more good than harm. Workers found other employment, and robots freed humans from monotonous and dangerous tasks.

I rather think about the myriad of possibilities that robots offer and prefer to think in terms of augmentation rather than replacement: in healthcare, education, security, elderly care, babysitting, and preventing bullying. Having eyes, hands and intelligence in places and situations where eyes or hands are otherwise not available. In addition, I am thinking of the huge challenges we face developing satisfactorily functioning robots, because we are far from ready. For example, how to communicate with such a thing?

The Heavenly Voice of Leolani – Visiting Piek Vossen

What kind of scientific research do you perform with the robot?
The biggest problem with robots is that is so hard to communicate with them. One can replay a video of everything the robot has seen, but it would be much more effective to simply talk to it, ask for what the robot has to report and, next, discuss and explain how you want things. In that case, the robot needs to understand what it has observed or done and what the implications are and being able to report on this in an effective way with me.

I assume that no machine or system is perfect. The engineer who thinks he or she can make the perfect machine or the perfect system is wrong. Uncertainty and awareness of failure must be built into robots from the start. You have to be able to talk about that, explain what went wrong and why you want things differently. I mean with the robot, not with the engineer.

It is my ambition to teach linguistic communication to robots so that people and robots can work together as teams and strengthen and complement each other. The robot must therefore be able to deal with people’s intentions and emotions, and constantly communicate about uncertainty, doubt and check whether there is mutual understanding and agreement. By the way, we have decided that the robot is a “she” and we named her Leolani. A Hawaiian name meaning “heavenly voice”. We will also give her character traits and moods, which in turn will influence communication.

Who are you cooperating with?
With the Spinoza prize money I was able to acquire the robot. Together with Bob van Graft’s budget via UC-IT (VU Information technology services). We aim to stimulate education and research into social robots. We also work together with Johan Hoorn and Elly Konijn from the Faculty of Social Sciences.

The Heavenly Voice of Leolani – Visiting Piek Vossen

At the moment three students are working with Leolani. Two of them are my Spinoza University Research Fellows for this academic year and one research student is funded by Bob van Graft from UC-IT. We use the Google API for speech recognition, IBM Watson and Wolfram Alpha as knowledge stores. These available technologies give the robot a big advantage and allow the students to work on exciting aspects such as uncertainty and social communication. Meanwhile we had four live presentations and have more events planned in the next few months. Leolani performs better every time.

For what purposes will you use Leolani in the future?
We are in the process of investigating the possibilities. We consider serious applications in education. For example for children with disabilities and in elderly care (I am Alice). We will submit proposals for projects, and apply for collaboration with other researchers.

Weekend of Science 2017: Talking with Robots

Lenka Bajčetić explaining robot object recognition game ‘I Spy With My Little Eye…’ to the kids at Weekend of Science, 07 October, 2017

Make Robots Talk Science Weekend

“Cardigan, Whahaha, I think not!” Weekend of Science 2017: Talking with Robots

Make Robots Talk Science Weekend

“No, No, … No, No, not a hot dog!” Weekend of Science 2017: Talking with Robots

IMG_20170927_114534-ANIMATION_Bram
Bram Kraaijeveld at work with robot