Theodore is an interactive installation capable to write sentimental letters based only on image content. The main idea underlying this project is to find out if machines are able to understand feelings, context, and generate intimate and confidential letters that express human-like sentiments using only pictures as input. For this purpose, the machine learning frameworks for the web ml5.js and face-api.js are used in the field of image recognition, in particular facial expression detection, landscape and landmarks recognition, as well as text generation.
The interaction between the user and the installation happens through a mobile web application, the system invites the user to upload a picture with the subject of the letter. In a second step, the user inserts his name, as well as the name of their loved ones and the type of relationship. After this process, the system starts to analyze the picture, extracting the emotional traits of the subject, along with contextual attributes from the location. In conjunction with the basic information inserted by the user, these attributes are passed to the text generation algorithm. Once the text has been generated, the system sends the result to the physical installation. The generated letter is now ready to be printed by an analog typewriter automated by an Arduino MEGA and 33 solenoid type motors powered by four 8-channel relay modules and a one channel relay.
The installation consists of a steel structure measuring 110 cm high, 30 cm wide and 35 cm in depth. At the top of the structure is positioned a typewriter Olivetti Lettera 32, german edition. 32 12 volt solenoids are mounted on the front and back wall of the structure. These motors are connected to the typewriter by a metal wire rope that controls 32 keys on the typewriter, an extra solenoid is used to manage the carriage return level of the typewriter. In order to get the solenoids to work, four 8-channel relays module and one channel relay module are used. The relay modules are controlled by the Arduino MEGA, which is connected to a laptop via serial communication.
Theodore was exhibited at JSNation in Amsterdam (2019, https://medium.com/@amsterdamjs/these-were-the-artists-showcased-at-jsnation-2019-52fbfa4d1743) and at the media art festival LAB30 in Augsburg (2019, http://www.lab30.de/programm-2019/theodore). It also had to be exhibited at InTake in The Hague and Meta Marathon (https://www.metamarathon.net/) in Düsseldorf this year, both events were canceled due to coronavirus.
Love the concept!