Monthly Archives: April 2012

Final Post: Gamex and Faces in Baroque Paintings

Face recognition algorithms (used in digital cameras) allowed us to detect faces in paintings. This has gave us the possibility of having a collection of faces of a particular epoch (in this case, the baroque). However, the results of the algorithms are not perfect when applied in paintings instead of pictures. Gamex gives the chance to clean this collection. This is very important since these paintings are the only historical visual inheritance we have from the period. A period that started after the meet of two worlds.

1. Description

Gamex was born from the merging of different ideas we had at the very beginning of the Interactive Exhibit Design course. It basically combines motion detection, face recognition and games to produce an interactive exhibit of Baroque paintings. The user is going to interact with the game by touching, or more properly poking, faces, eyes, ears, noses, mouths and throats of the characters of the painting. We will be scoring him if there is or there is not a face already recognized on those points. Previously, the database has a repository with all the information the faces recognition algorithms have detected. With this idea, we will be able to clean mistakes that the automatic face recognition has introduced.

The Gamex Set

The Gamex Set

2. The Architecture

A Tentative Architecture for Gamex explains the general architecture in more detail. Basically we have four physical components:

  • A screen. Built with a wood frame and elastic-stretch fabric where the images are going to be projected from the back and where the user is going to interact poking them.
  • The projector. Just to project the image from the back to the screen (rear screen projetion).
  • Microsoft Kinect. It is going to capture the deformations on the fabric and send them to the computer.
  • Computer. Captures the deformations send by the Kinect device and translates them to touch events (similar to mouse clicks). These events are used in a game to mark on different parts of the face of people from baroque paintings. All the information is stored in a database and we are going to use it to refine a previously calculated set of faces obtained through face recognition algorithms.

3. The Technology

There were several important pieces of technology that were involved in this project.

Face Recognition

Recent technologies offers us the possibility of recognizing objects in digital images. In this case, we were interested in recognizing faces. To achieve that, we used the libraries OpenCV and SimpleCV. The second one just allowed us to use OpenCV with Python, the glue of our project. There are several posts in which we explain a bit more the details of this technology and how we used.

Multi Touch Screen

One of the biggest part of our work involved working with multi-touch screens. Probably because it is still a very new technology where things haven’t set down that much we have several problems but fortunately we managed to solved them all. The idea is to have a rear screen projection using the Microsoft Kinect. Initially though for video-game system Microsoft Xbox 360, there is a lot of people creating hacks (such as Simple Kinect Touch) to take advantage of the abilities of this artifact to capture deepness. Using two infrared lights and arithmetic, this device is able to capture the distance from the Kinect to the objects in front of it. It basically returns an image, in which each pixel is the deepness of the object to the Kinect. All sorts of magic tricks could be performed, from recognizing gestures of faces to deformations in a piece of sheet. This last idea is the hearth of our project. Again, some of the posts explaining how and how do not use this technology.

Calibrating the multi-touch screen

Calibrating the multi-touch screen


Last but not least, Kivy. Kivy is an open source framework for the development of applications that make use of innovative user interfaces, such as multi-touch applications. So, it fits to our purposes. As programmers, we have developed interfaces in many different types of platforms, such as Java, Microsoft Visual, Python, C++ and HTML. We discovered Kivy being very different from anything we knew before. After struggling for two or three weeks we came with our interface. The real thing about Kivy is that they use a very different approach which, apart from having their own language, the developers claim to be very efficient. At the very end, we started to liked and to be fair it has just one year out there so it will probably improve a lot. Finally, it has the advantage that it is straightforward to have a version for Android and iOS devices.

4. Learning

There has been a lot of personal learning in this project. We never used before the three main technologies used for this project. Also we included a relatively new NoSQL database system called MongoDB. So that makes four different technologies. However, Javier and me agree that one of the most difficult part was building up the frame. We tried several approaches: from using my loft bed as a frame to a monster big frame (with massive pieces of wood carried from downtown to the university in my bike) that the psyco duck would bring down with the movement of the wings.

It is also interesting how ideas changes over the time, some of them we probably forgot. Others, we tried and didn’t work as expected. Most of them changed a little bit but the spirit of our initial concept is in our project. I guess creative process is a long way between a driven idea and the hacks to get to it.

5. The Exhibition

Technology fails on the big day and the day of the presentation we couldn’t get our video but there is the ThatCamp coming soon. A new opportunity to see users in action. So the video of the final result, although not puclib yet, is attached here. It will come more soon!

6. Future Work

This has been a long post but there is still a few more things to say. And probably much more in the future. We liked the idea so much that we are continuing working on this and we liked to mention some ideas that need to be polished and some pending work:

  • Score of the game. We want to build a better system for scores. Our main problem is that the data that we have to score is incomplete and imperfect (who has always the right answers anyway). We want to give a fair solution to this. Our idea is to work with fuzzy logic to lessen the damage in case the computer is not right.
  • Graphics. We need to improve our icons. We consider some of them very cheesy and needs to be refined. Also, we would like to adapt the size of the icon to the size of the face the computer already recognized, so the image would be adjusted almost perfectly.
  • Sounds.  A nice improvement but also a lot of work to have a good collection of midi or MP3 files if we don’t find any publicly available.
  • Mobile versions. Since Kivy offers this possibility, it would be silly not to take advantage of this. At the end, we know addictive games are the key to entertain people on buses. This will convert the application in a real crowd sourcing project. Even if this implies to build a better system for storing the information fllowing the REST principles with OAuth and API keys.
  • Cleaning the collection. Finally, after having enough data it would be the right time to collect the faces and have the first repository of “The Baroque Face”. This will give us an spectrum of how does the people of the XVI to XVIII looked like. Exciting, ¿isn’t it?
  • Visualizations. Also we will be able to do some interesting visualizations, like heat maps where the people did touch for being a mouth, or an ear, or a head.

6. Conclusions

In conclusion we can say that the experience has been awesome. Even better than that was to see the really high level of our classmates’ projects. In the honour of the truth, we must say that we have a background in Computer Science and we played somehow with a little bit more of adventage. Anyway, it was an amazing experience the presentation of all the projects. We really liked the course and we recommend to future students. Let’s see what future has prepared for Gamex!

Some of the very interesting projects

Some of the projects

This post was written and edited together to my classmate Roberto. So you can also find the post on his blog.


Filed under Analysis, Tasks

Building the proper screen

The last step in the project, after we were able to overcome all the technical difficulties like Kivy language, was the building of the a suitable screen for our purposes, this is a poke-able rear screen. Doing this we avoid the problem of calibrating the Kinect device each time and for each user, and foremost we could do the setup just once.

The first attempt of a rear screen

The first attempt of a rear screen

Our first attempt was building a very big frame and using a table cover or bed sheet as the screen. But we found several serious problems:

  1. The frame was too big for moving.
  2. The frame wasn’t rigif enough and with the interaction of the users it got deformation.
  3. The screen, after an user interaction, never got back to normal and keept deformed forever.

Among all the problems (and others just not commented here), the last one was totally frustrating, because all the platform depends on the stability of the screen. If the screen is not just plain, Kinect will detect that, and it will translate points to the Kivy application when there is no one actually. Everything was wrong.

Choosing the most beautiful tissue

Choosing the most beautiful tissue

The alternative was to use a stretch fabric with the capacity to recover the initial shape after any, virtually, number of interactions. Even with hard punchs. But we didn’t know where to buy that or if it was even cheap enough for our students’ pockets. Fortunatelly, Prof. William Turkel recommend us Fabric Land, with three locations in the city and a lot of options for fabrics. I must say that the place was a bit weird for me, with a bunch of middle age ladies looking for good materials. I felt like Mrs. Doubtfire there. Finally one girl, very gentile, young and nice, helped us to find what we wanted, and she sold us at the price of $5 per meter!

Colours pins! That's the most decorative we can be

Colours pins! That's the most decorative we can be

And with all the raw material ready, we got down to work and we built, after several tries, the proper rear and stretch screen. Just in the first test we discovered that accuracy was amazing. And the most interesting thing: somehow, the fact that the screen is elastic, demands interaction from the users and keeps the people playing. So, we can say mission accomplished!


Filed under Tasks

A Tentative Architecture for Gamex

The other day, during the weekly class, I just realized there is no document explaining the architecture of our project. Roberto and I have been talking a lot about it, but never wrote anything detailed from a technical point of view. In past entries I talked about the idea behind the project, called Gamex, and how it is going to work. Because we are really excited with the project, sometimes we get lost our selves in the universe of coding (and fighting with Kivy) and we forget to document technical issues. So I hope to fix that with the current blog post.

General Architecture of Gamex

General Architecture of Gamex

The architecture of the project, as depicted in the image above, has different parts that I’m going to describe illustrating a complete application cycle:

  1. In a previous phase, and in order to save some time and increase performance, we pre-process a set of baroque paintings applying different techniques to recognize faces in pictures. All the data is stored using JSON files for medatadata and JPEG for images format.
  2. Then, the user, I mean, the audience of the exhibition, walk in front of the screen.
  3. The screen, thanks to a projector, shows a slideshow of baroque paintings and calls for the action, it demands interactivity. Maybe some fancy text with blinking effect or similar. Every phrase it correspond to a different game and different process to collect data. These ones are the current games we are developing (it is very easy to extend to collect data to, for example, just the virgins, or the saints, or the children, etc):
    • “Hey! Punch these people in the face,” to collect information from the position of the heads in the painting.
    • “Dude, better if you get this people’s eyes out,” to collect the pair of points related to the position of the eyes.
  4. When the user punch or touch the screen, a change in the deepness of the screen is produced. This alteration is observed by a Microsoft Kinect device.
  5. The signal is encoded as a point or set of points using STK and sent to a TUIO server.
  6. Tha main application written in Kivy is listening to that server. When a point is received into the TUIO server, the Kivy application translate it into a mouse click in the application frontend.
  7. In that moment, the information about the point is stored in a MongoDBNoSQL database, actually a key-value store. We create something similar to a table for every image with two different lists:
    1. The first one store a list of points and a timestamp for the face punch game.
    2. The second one is intended for the get the eyes out game and store a couple of points.
  8. If the point provided by the user is inside an area previously calculated in our metadata, we assign high points to provide some feedback to the user. If the point is new, we mark that information and show different score.
  9. When the number of points the user gives is similar to the number of faces we get in the preprocessing stage, the slideshow browse to another image and it starts again.

The second game is a way to tune the information we are collecting. For that game, Gamex is going to show paintings with information about where the faces are supposed to be. With the information about the position of the faces in the painting and the position of the eyes, we can calculate the size of the head and, even, create a probability heat map of where the faces are. Finally we will be able to enhance the algorithms used to detect faces.

Of course, this project is senseless if we are not able to get a lot of users and if we require only a minimum effort in the users side to interact. Let’s see what we are able to do.


Filed under Tasks