Interactive displays

Let’s look at how to use Mosami to build an interactive display triggered by the viewer’s actions – a video output of a presenter+slide with two different layouts. When the viewer is far away, it will emphasize the presenter over the slides, but swap focus to the slides for better readability when the viewer leans in for a closer look.

What’s under the hood? Let’s take a look.

Code for this example is from our examples library at https://launchpad.net/mosami/+download in the file nearfar.py

The main thing is that we want our application to respond to the viewer’s position. To do that, we use the DetectFace module, which outputs a stream of messages with the bounding box (x,y,w,h) of the detected face(s), and attaching a callback function to handle those messages. Whenever a new DetectFace message arrives, it checks to see whether the (single) face is big or small compared to a threshold. To make it less jumpy, we accumulate those results, and add a slow drift to “far” if no face is detected.

if msg['num'] == 1:
     if msg['width'] > self.defaults['face_width_threshold']:
          self.state['tracker'] += 1.0
     else:
          self.state['tracker'] -= 1.0
else:
     self.state['tracker'] -= 0.05

After that, the callback thresholds the recent activity and chooses a “near” or “far” layout accordingly

if self.state['tracker'] > 20:
     self.state['layout'] = 'near'
     self.state['tracker'] = 10
if self.state['tracker'] < -20:
     self.state['layout'] = 'far'
     self.state['tracker'] = -10
self.mix.msg('switch_layout', layout=self.state['layout'])

Finally, we want that ‘switch_layout’ message to change the display. First, we need the basic display elements. For the presenter, we remove the background with CutoutPerson. For the slides, we use ViewPDF to convert the slides into a video stream. Then, we composite them together using LayoutMix, which allows one to preconfigure multiple layouts.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s