In my humble opinion…

The last step in the project, after we were able to overcome all the technical difficulties like Kivy language, was the building of the a suitable screen for our purposes, this is a poke-able rear screen. Doing this we avoid the problem of calibrating the Kinect device each time and for each user, and foremost we could do the setup just once.

The first attempt of a rear screen

Our first attempt was building a very big frame and using a table cover or bed sheet as the screen. But we found several serious problems:

1. The frame was too big for moving.
2. The frame wasn’t rigif enough and with the interaction of the users it got deformation.
3. The screen, after an user interaction, never got back to normal and keept deformed forever.

Among all the problems (and others just not commented here), the last one was totally frustrating, because all the platform depends on the stability of the screen. If the screen is not just plain, Kinect will detect that, and it will translate points to the Kivy application when there is no one actually. Everything was wrong.

Choosing the most beautiful tissue

The alternative was to use a stretch fabric with the capacity to recover the initial shape after any, virtually, number of interactions. Even with hard punchs. But we didn’t know where to buy that or if it was even cheap enough for our students’ pockets. Fortunatelly, Prof. William Turkel recommend us Fabric Land, with three locations in the city and a lot of options for fabrics. I must say that the place was a bit weird for me, with a bunch of middle age ladies looking for good materials. I felt like Mrs. Doubtfire there. Finally one girl, very gentile, young and nice, helped us to find what we wanted, and she sold us at the price of $5 per meter!

Colours pins! That's the most decorative we can be

And with all the raw material ready, we got down to work and we built, after several tries, the proper rear and stretch screen. Just in the first test we discovered that accuracy was amazing. And the most interesting thing: somehow, the fact that the screen is elastic, demands interaction from the users and keeps the people playing. So, we can say mission accomplished!

The other day, during the weekly class, I just realized there is no document explaining the architecture of our project. Roberto and I have been talking a lot about it, but never wrote anything detailed from a technical point of view. In past entries I talked about the idea behind the project, called Gamex, and how it is going to work. Because we are really excited with the project, sometimes we get lost our selves in the universe of coding (and fighting with Kivy) and we forget to document technical issues. So I hope to fix that with the current blog post.

General Architecture of Gamex

The architecture of the project, as depicted in the image above, has different parts that I’m going to describe illustrating a complete application cycle:

1. In a previous phase, and in order to save some time and increase performance, we pre-process a set of baroque paintings applying different techniques to recognize faces in pictures. All the data is stored using JSON files for medatadata and JPEG for images format.
2. Then, the user, I mean, the audience of the exhibition, walk in front of the screen.
3. The screen, thanks to a projector, shows a slideshow of baroque paintings and calls for the action, it demands interactivity. Maybe some fancy text with blinking effect or similar. Every phrase it correspond to a different game and different process to collect data. These ones are the current games we are developing (it is very easy to extend to collect data to, for example, just the virgins, or the saints, or the children, etc):
• “Hey! Punch these people in the face,” to collect information from the position of the heads in the painting.
• “Dude, better if you get this people’s eyes out,” to collect the pair of points related to the position of the eyes.
4. When the user punch or touch the screen, a change in the deepness of the screen is produced. This alteration is observed by a Microsoft Kinect device.
5. The signal is encoded as a point or set of points using STK and sent to a TUIO server.
6. Tha main application written in Kivy is listening to that server. When a point is received into the TUIO server, the Kivy application translate it into a mouse click in the application frontend.
7. In that moment, the information about the point is stored in a MongoDBNoSQL database, actually a key-value store. We create something similar to a table for every image with two different lists:
1. The first one store a list of points and a timestamp for the face punch game.
2. The second one is intended for the get the eyes out game and store a couple of points.
8. If the point provided by the user is inside an area previously calculated in our metadata, we assign high points to provide some feedback to the user. If the point is new, we mark that information and show different score.
9. When the number of points the user gives is similar to the number of faces we get in the preprocessing stage, the slideshow browse to another image and it starts again.

The second game is a way to tune the information we are collecting. For that game, Gamex is going to show paintings with information about where the faces are supposed to be. With the information about the position of the faces in the painting and the position of the eyes, we can calculate the size of the head and, even, create a probability heat map of where the faces are. Finally we will be able to enhance the algorithms used to detect faces.

Of course, this project is senseless if we are not able to get a lot of users and if we require only a minimum effort in the users side to interact. Let’s see what we are able to do.

As a part of our project, we have to provide a way by which the user can point out the faces in a serie of Baroque paintings. Theses faces are previously calculated using face recognition algorithms and SimpleCV, a simplified API for OpenCV thar, of course, doesn’t work on my 64bits machine. In fact, there are differente methods, with advantages and drawbacks, used at the same time. But algorithms are not perfect, and in general they miss some “obvious” faces as long as they wrongly tag things likes dresses as faces.

Intro screen

Above these lines you can see the initial screen before starting to play. In the future we are thinking about including more than one game. The first one is focused on identifying faces through the input of the users. The second one could be, for example, a game in order to filter that previous selection, something like, once we hace faces selected and extracted, a Invaders-like game in which users has to shot just to faces of certain kind. Very useful to filter and provide even more information for our algortihms.

However, we need to ask for the user the minimal interaction ever, so we expect from the user just one single touch on the screen. And this raises a problem: how to, with a single point in the screen, get the face the user is trying to point. To resolve this we are exploring two approaches.

The first one is to wish the users are going to make a circle around the heads or faces of the characters on the painting. But this is against our main goal to request as minimal interactions as possible. Our idea is the users can punch the screen where they think faces are, not draw a circles around. We need to think in another approach.

How the algorithms work so far.

So the next step is to provide a way to make even more usefull the input from the users. And we decided the creation of a second game. In one hand, we previously did a detection of some faces, so we can calculate how near are the single touchs to the already recognized faces. But we don’t know this information for new faces that our algorithms missed out. If this happens, all we have is a point in the painting and no way to figure out if it is relative to a face. In this scenario, we mark these points for second stage in our process. Thus a second game in which we show to the users these images properly cropped and we pretend they just poke eyes out. Then, we have information about almost the center of the face (the previous single points), the current position of the eyes, and information about the medium size of the rest of the heads our algorithms did detect before: we can now triangulate the shape of the face/head. So in this way we filter a lot of information just making the users play a different game. The common feature in both games? The violence: Punch and poke eyes out!

Slowly but surely. That’s our mantra in this project since it implies a lot of coding and to handle very different techonolgies, some of them –I really mean all of them– in the very stages of development. Anyway, we finally decided to implement our multi-touchable surface using a projector, a Kinect device and the libraries SKT and Kivy. As the next video shows, we are getting some advances, but we develop slower than we expect.

And finally, our first proof of concept of everything working together. It’s about a very basic project that is able to handle several interactions at the same time. At the moment, it only allows to draw lines and identify how many different contact points are there using one color for each one.

Behind scenes, what it happens is described below:

1. The Simple Kinect Touching allow us to define the boundaries of the projected screen.
2. We adjust a bunch of parameters related with deepness in order to focus only in the coordinates that mean something is touching the screen.
3. Then, that information is transformed and sent according the TUIO protocol to a local server. Now we have service streaming of data relative to the touchs and movements on the screen.
4. In this point, we run our Kivy client application, that we call gamex, by setting the input interface as a TUIO server instead of the mouse.
5. Black magic.
6. Profits!

And that’s it. We really hope to have time to focus in the development of the machine learning application once the most difficult technical issues seem almost solved. However, to do the projection on a bed sheet or table cover is going to be kind of traumatic for setting the paramereters in the Kinect recognition layer. We will need a very rigid wood frame or something for making the surface as smooth as we can.

This week has been, again, very funny. I was participating in the building of a MakerBot 3D printer. I have to say that is easier than I thought and all you need is inside the box. I couldn’t be all the days needed to finish the printer because I had classes to attend. But anyway, the days I did do I enjoyed a lot.

Finishing the MakerBot 3D Printer

Moving on to something else, Roberto and I were talking to Prof. William Turkel about some ideas for our final project. And after listening a lot of very good ideas and proposals, and given the fact that our time is finite, we’ve almost defined our project. It has three different parts that I am going to detail:

1. We are going to build a multi-touchable big screen using a table cover. Yes, using a table cover. To do that we’ll use a Microsoft Kinect device. The Kinect, as much of you already know, is able to get the deepness of the scene what the device is focusing to. So, we also have a projector that it will be connected to one of our computers. Thanks to the Simple Kinect Touch library that we found, we can define the area of the projected screen as a delimited plain surface. Done this, any change in the surface is properly detected by the deepness camera of the Kinect. There are many ways to transform that information into a single point to emulate a mouse click. Maybe we’ll calculate the middle point according the distorted surface by pressing the screen. But what screen? The table cover on which the projector will be used.
2. In the step above we explained briefly a way to have an arbitrary surface as a touchable screen. After this, we are going to show images in the screen. But not random images. In a parallel process, and using maybe OpenCV and some improvements to the face recognition algorithms, we extract the position of the bounding box for the faces from certain baroque paintings. However, this process is not always as good as we expect. And that’s the reason of the step three.
3. In the final step, we use the bounding boxes extracted for each image. With that information, we project a image in the big screen and now,  using Kivy or Processing, we ask for users to touch the devil, angel, saints, virgins or Jesus faces. One of every type at the same time. So we can use that information as features to enhance a machine learning algorithm with scikit-learn. We can even propose different methods to touch the faces: pray for angels, punch for devils, etc. In this way, we give feedback (formally training) to recognise different kinds of faces, because our first algorithm is only able to detect faces and it doesn’t very well for faces in pictures.

And that is. The idea is to have this in some museum or public place, pretending that people play with the screen selecting the type of “touch” to give. And, in the background, providing very useful information for face recognition patterns in artworks.

But, due to the difficulty of having the tings working in 64 bits Linux OS, we are moving to 32 bits. It’s really better to focus on the task to do instead of the installation of requirements.

We also have already some materials to build the screen, I hope during this week to have a very first version of the code, that will be hosted ad GitHub. The name, in a lack of a better one, is Gamex: Game Exhibis for Machine Learning.

This is the story of a very big frustration. I’ve been an evangelist of Open Source from almost 8 years now. One of the first thing I always defend is the the freedom of a free operative system. Of course, with great power comes great responsibility, but that didn’t really worry me because actually I’ve always been enjoying of dealing with the system. There is something special in the satisfaction of getting things working in Linux. Even, and secretly, it is one on the reaseons because we like Linux, because when something gets wrong, we have the power to fix it. And we love this.

But this mantra could have been a mess. Actually I thought I wasn’t able to get the things done. In this case, I was trying to install the open library for computer vision, called OpenCV, in order to use it from Processing. But my laptop has a 64 bits processor, and there started the problem. The most of the libraries are compiled and have binaries for 32 bits systems, so it’s supposed to could be enough if I compiled the libraries by myself. But that’s not alwasy as easy as it looks like. After try, nor one, two, three, four, but six different methods, I got OpenCV libraries working on my machine with Python bindings, and also the OpenNI and NITE PrimerSensor enabled, what it’s supposed to be good in you want to connect a Kinect (in the future I would like to use Point Cloud Library, aka PCL). And not the same luck for CUDA. Once the OpenCV was working, of course, I discovered the fantastic OpenCV PPA with 64 bits libraries packaged for Ubuntu 11.10.

After OpenCV, I needed to bind the library with Java first, and with Processing then. But this step hasn’t been likely at all. I tried the common OpenCV for Processing, I also tried the recommendations made by my classmate Roberto (whose laptop is 32 bits processor based) and generating again the libOpenCV.so following several instrucionts. Even with another different library called JavacvPro, but nothing, I always got the the error:

wrong ELF class: ELFCLASS64


So, if I wanted to build a proof of concept of our idea of a green LED blinking when the camera detects a fece, and a red one when it doesn’t, I had to use the almighty Python for what the OpenCV was already working pretty well. Then I took the typical facedect.py example and added a couple of lines for connecting to Arduino, so the new code looks like:

import numpy as np
import cv2
import cv2.cv as cv
import serial
from video import create_capture
from common import clock, draw_str

arduino = serial.Serial('/dev/ttyACM0', 9600, timeout=1)

help_message = '''
USAGE: facedetect.py [--cascade <cascade_fn>] [--nested-cascade <cascade_fn>] [<video_source>]
'''

def detect(img, cascade):
    rects = cascade.detectMultiScale(img, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30), flags = cv.CV_HAAR_SCALE_IMAGE)
    if len(rects) == 0:
        arduino.write("N")
        return []
    rects[:,2:] += rects[:,:2]
    arduino.write("F")
    return rects

# The rest of the code is the same

And having the next Arduino program loaded

#define GREEN 8
#define RED 7

int val = 0;
int face = 70; // "F"
int none = 78; // "N"

void setup() {
  pinMode(GREEN, OUTPUT);
  pinMode(RED, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  if (Serial.available()) {
    val = Serial.read();
    if (val == face) {
        digitalWrite(GREEN, HIGH);
        digitalWrite(RED, LOW);
    } else if (val == none) {
        digitalWrite(GREEN, LOW);
        digitalWrite(RED, HIGH);
    }
    Serial.println(val);
  }
}

And here we can watch the amazing result of the work

The second week in the fascinating world of Arduino left us very interesting things. The class was about connecting new kinds of periphals, like buzzers or LCD displays. So, the idea of my classmate Roberto and me was to show the last tweet from twitter in a LCD display. After that, and due to the time limitations of the class, we changed our minds and decided analyze the last tweet in a sense of positive or negative message, and then brigth a blue LED for positive or red LED for negative. But sadly, this also was too much for one and a half hour.

So, what we finally did was to communicate the Arduino device with twitter through serial port connecting to a Python program. The workflow is the next. A Python program in a infinite loop gets the last tweet, then analyzes the positive and negative sentiment (using ViralHeat API) of hte text, after that, it sends a signal to the Arduino device, that it is listening in a specific port. Regardless to the singal received, plus the message, Arduino does the right LED to brigth.

The message is not trivial, and it has to be defined. In our case, we first send the signal for blue, red or both (whether the sentiment was unable to get), followed by a special token as separator, all the characters of the text for a future implementation of the LCD display. The separator token should be a not printable character or some you are sure it is nor possible to use in a twitter message.

So here you can find the Python code fofr the program. As you can see, we used the twython library for connecting with Twitter. Also, if you are using Linux, in order to know the right port to connect, go to the Arduino window → Tool → Serial monitor, and see what it is written in the title of the window. It used to be something like

/dev/ttyACM0

.

import serial
import time
import urllib
import requests

from json import loads
from twython import Twython

# Token, we choose not printable values
SEPARATOR_TOKEN = 666
# Sentimen API
url = "http://www.viralheat.com/api/sentiment/review.json?text=%s&api_key=<KEY>"
# Connect to arduino via serial port
arduino = serial.Serial('/dev/ttyACM0', 9600, timeout=1)

# Write to Arduino
def writeToArduino():
    twitter = Twython()
    public_timeline = twitter.getPublicTimeline()
    last_tweet = public_timeline[-1]
    text = last_tweet["text"]
    # We get the numbers of the each character
    codes = [ord(c) for c in text]]
    # Get the sentiment
    response = requests.get(url % urllib.quote(text))
    sentiment_response = loads(response.content)
    if sentiment_response["mood"].lower() == "negative":
        sentiment = 0
    elif sentiment_response["mood"].lower() == "positive":
        sentiment = 1
    else:
        sentimen = 2
    # Add characters codes and sentiment value
    arduino.write(codes + [SEPARATOR_TOKEN, sentiment])

while True:
    writeToArduino()

And finally the Arduino code.

#define GREEN 8
#define RED 12

const int sepToken = 666;
int val = 0;
int lastVal = 0;

void setup() {
  pinMode(GREEN, OUTPUT);
  pinMode(RED, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  if (Serial.available()) {
    val = Serial.read();
    if (lastVal == sepToken) {
      if (val == 0) {
        digitalWrite(GREEN, HIGH);
        digitalWrite(RED, LOW);
      }
      if (val == 0) {
        digitalWrite(GREEN, HIGH);
        digitalWrite(RED, LOW);
      }
      if (val == 0) {
        digitalWrite(GREEN, HIGH);
        digitalWrite(RED, LOW);
      }
    } else {
      // Print "val" to the LCD Display
    }
    Serial.println(val);
    lastVal = val;
  }
}