ajfisher's doing space

From the Blog

This is the distillation of several projects I’ve worked on over the last year. I’m only showing one of these as I actually captured my build process on the most recent; being an interactive light display for a demo during my presentation at the Sketching in Hardware conference in Portland, Oregon.

This approach isn’t just limited to lights. Any time you want to make an interactive display that numerous people can interact with at once then this is a good approach – and it’s web based for interaction so you can do all sorts of interesting interfaces.

I’m not going to go into the serious detail of under what conditions you would want to do this and things around messaging architecture etc. If you want to find out more, please see the talk I gave at Linux Conf and this one at Sketching (where this project was demoed).

Overview

This project consists of three parts:

  • * A networked ardiuno that takes messages from the server and controls the RGB light display.
  • * A web interface where users can touch or click circles representing the lights and change them different colours.
  • * A web server which processes all of the messages between the web interface and the arduino.

When a user visits the site on their mobile or desktop device they will be presented with an interface that allows them illuminate a light by clicking or touching a circle on screen. Users can change their “personal” colour actively or using a randomiser and where two users illuminate the same light their colours will mix. The lights will take the colour passed in the message from the server and then gradually fade to nothing over time.

Stuff you need

There are a few things you’ll need to get hold of:

  • * A networked arduino – I prefer the Freetronics EtherTen but and arduino with Ethernet Shield will work just as well.
  • * Enough light modules to make your display. Mine are Freetronics RGB Modules (which this code is written for) but you could use BlinkMs or other light modules. The key is they need to be individually addressable and only use a few pins on your arduino. I used 19 for my display which was arranged like a filled hexagon.
  • * A computer that can run as a server. This was designed for Linux, I’ve tested it on my Mac as well but for Windows, there’s no guarantees.
  • * A stack of jumper wires or other to join your light modules together.
  • * Headers to attach to your light modules.
  • * Something to mount your lights onto (cardboard, perspex etc)
  • * Something to diffuse the light (I used ping pong balls which work really well)

Software to install

Server

Let’s get the server up and running first. I’m assuming both Linux and Mac here, I’m also assuming you have python installed and you are running VirtualEnv and you’ve created a new virtual environment to play in.

My environment was called sketching so let’s work on that:

$ workon sketching

Now we’ll get the code and install the dependencies

$ git clone git://github.com/ajfisher/sketching-conf-demo.git
$ cd sketching-conf-demo
$ pip install -R requirements.txt

Pip will then install everything you need. You may run into a couple of lib event issues if you’ve never installed it before. See the notes here under installation which may apply (particularly if you’re running OSX): https://github.com/stephenmcd/django-socketio

Now you have everything installed without errors test it with:

$ cd sketching
$ python manage.py runserver_socketio 127.0.0.1:8000

You should get notice that the server is running. At that point, direct your web browser to http://127.0.0.1:8000 and you should get a screen with a series of circles on it.

If this is working then the django server is pretty much good to go, the big thing you’ll need to change is to make the server accessible beyond just your computer. As such run the server like this:

$ python manage.py runserver_socketio HOST_IP:PORT

Where HOST_IP is your IP address of your computer on the network and the port is the port number you want to run on. I generally run on about 8000 or so as low port numbers require running as root and for the moment you can continue to use high numbers for development.

Make sure your server is up and running before connecting your arduino up.

Arduino setup

The first thing you need to do is include the ArduinoWebsocketIOClient in your libraries. The easy way to do this is simply copy the library across to your sketchbook/libraries folder and dump it in there and restart your Arduino IDE. I prefer to do this a little more cleanly and symlink the foilder from my sketchbook so if I make any changes it’s still part of my git repo and I can manage them.

Once you’ve done that and restarted the IDE so it’s available to the environment open up the “Connect to Server” sketch to test you can actually do just that.

You will need to change the vales of the server variable on the line that looks like:

char server[] = "10.0.1.15";

and the port number on the line that looks like:

int port = 8000;

To whatever the address was you set your server to run on.

Compile your sketch then upload it to your arduino. Make sure it’s connected to the network and then power it on and connect to it using the serial monitor as well.

The aim of this sketch is to check that you can connect to the server, subscribe to the “pong” socketIO channel and detect messages coming through.

If the connections and subscriptions don’t work ensure your arduino can see the network, can see your server (eg firewalls have the right rules) and watch the log where your server is running as it will spit out a wealth of info about whether the device connected or not.

Assuming this has all worked, then you can load up the interface in your browser by going to http://ip:port/ and if you click on the circles you should see messages appear on the serial monitor with the RGB values and the position of the circle you hit.

Once you’ve confirmed all of this is working you can load the real sketch onto the arduino called “colour controlled with delay”.

Build the display

Lights

You can arrange this however you want if you want to change the HTML as well but my version set the lights in a filled hexagon shape that was offset with 3 lights then 4, then 5 then 4 then 3. Using an offset grid makes it easy to stack circles together as you can pack them densely – they also look good. To make this arrangement we need 19 RGB modules.

I got the modules and soldered 0.1″ male header to each one. Testing each one in a jig I made to ensure they worked as I was finished. This simply cycles through RGB values and you drop the new module in alongside one know to work and you can test that it accepts input and passes values along the chain.

As an aside these modules use a “ripple” effect to pass the values along the chain. It happens really fast so you can’t see it but you can effectively chain them as long as you like and you just pass the messages in a list and it flows down the chain. If you want to change just one light in the middle though, you have to pass the entire list again with just that one module changed. It only takes about 20ms to do this though so it’s pretty fast.

Now we have our modules, we need to wire them together. I use pre-crimped wire that was about 8cm long from Poloulu as I hate crimping wires and the prospect of doing 160 ends would have sent me mad. If you want to do it though, go right ahead. Once done you just connect them all up together into a great big long chain, output to input along the length.

Once all of the modules are wired up, you connect the 5V and GND wires to the arduino and then connect the CKI pin on the module to arduino pin 2 and the SDI pin of the module to arduino pin 3.

Test that your lights are all working correctly by loading the “colour sequence test” sketch onto the arduino. You should see each of the lights cycle through a set of colours in turn almost like a christmas light display. This will help you debug any wiring issues with your modules.

Once you have all of the lights working then you can mount them.

Mounting the lights

My display was designed to be pulled apart for transportation but is surprising resilient as well. Holes are made in the mounting media between the lights and the wires are folded in half and poked through the holes. Ping pong balls mounted on top of the lights with a small hole cut out apply pressure outwards on the wire which creates enough tension to hold them in place. This is especially the case when you offset rows like bricks.

I tested the design on thick cardboard first just by poking holes through it to get the measurements correct. This all depends on what you use to diffuse the light. My ping pong balls were about 4cm in diameter. I actually discovered there’s a fair amount of variation here so you’ll have to adjust for your own components.

Once I had the measurements correct, I transferred it to 10mm acrylic which had 5mm holes drilled into it. Using wet and dry I sanded the hole edges to make them smoother after stripping a couple of wires pushing them through. The fit will be tight but will hold very securely.

Light fittings

Ping pong balls make surprisingly good diffusers for LEDs. I cut 5mm square holes in them and then run a line of “white tac” (like blutac but white) along the perimeter of the square and then press them down onto the RGB modules. This holds really well and can be removed. If I was making this permanent, something like sugru would be appropriate and will bond tightly.

I work from the centre outwards in a spiral as it makes it easier once you have many balls in place as the pressure of them will make working difficult in the middle. One thing you’ll notice as you’re nearly finished is that the top of bottom rows are almost floppy but the centre ones are nice and tight. We’ll fix that in the next step.

Using either tiny spots of superglue or Sellotape “super small dots” which are amazingly sticky, stick each of the balls to its neighbour where the surfaces touch. As I mentioned above the centre ones won’t really need it because they are packed so tightly but you can use this to transfer some of that rigidity to the top and bottom rows which will then pack in nicely with the rest.

Once you’re finished you’ll end up with 19 ping pong balls that are tightly packed together and very rigid. I also pinned a connector wire to the acrylic to make it easy to connect the arduino to it without the wire flapping around.

Running

Connect up your arduino as before, 5V, GND, pin 2 to CKI and pin 3 to SDI. You can run your colour test again if you want to make sure it’s all good or now load up the “colour controlled with delay” sketch, changing the IP and port numbers to that of your server.

Now once you hit the web page with the interface on it, each time you select a circle you’ll get the corresponding ping pong ball light up on the display.

Multiple people can all play with this at once so once you’ve got your display get a bunch of devices together and play around with it. You can get some interesting interaction effects happening when you have more than one person hit the same light at once or just after each other as it’s fading.

Where to now?

There’s a lot you can do with this system. I’d love to see someone do this with a building or other types of interactive display. If you do, drop me a note / post a comment and send me a link. Video and a wider gallery of photos is available on the project page.

Acknowledgements

Thanks to Steve McDonald who put together the Django SocketIO library that underpins a lot of the server side of things. Also thanks for Kevin Rohling who wrote the original arduino web sockets library that I adapted to work under socketIO.

As a web oriented person I tend to think web-first whenever I think of mobile. I posted a little while back about how to get a mobile phone to pull it’s orientation data out using some javascript and pass that data to a web server. This has evolved somewhat now and I have a packaged Git Hub project that shows you how to do it properly.

One of the demo projects is controlling an RGB light using the web browser’s gyro in order to get it’s orientation then mapping that to the three colour channels.

There are three core parts to the application:

Arduino / light set up:

In this scenario I’m just using a serial connection for simplicity. All the arduino is doing is looking for data that’s packaged up as a sequence of bytes that looks like this:

Byte 0 (int) 255 header byte
Byte 1 (int) 255 header byte
Byte 2 (int)  0-180 val for x
Byte 3 (int)  high byte for y
Byte 4 (int)  low byte for y
Byte 5 (int)  high byte for z
Byte 6 (int)  low byte for z

The code simply looks at the incoming data stream, looks for the double 255 bytes then reads the next 5 bytes into a buffer and converts them into variables. Check out the code on github.

Once it has the data it then maps to the RGB LED appropriately.

Web Browser

Modern mobile browsers (Firefox on Android, Mobile safari on iOS) give you access to the device API. This give us device motion and device orientation events that will fire every time the device moves. Using the orientation event we can trap the data about the way the device is facing then send that back via web sockets to the server.

var socket;
var room = "light";
 
var last_sent = (new Date()).getTime();
var threshold = 100; // msec between sends
 
window.addEventListener("deviceorientation", update_gyro, true);
 
function update_gyro(e) {
	// gets the gyro position
    var x, y, z = 0;
    var o = deviceOrientation(e);
 
	update_text(o.gamma, o.beta, o.alpha);
    if ((new Date()).getTime() - last_sent > threshold) {
		socket.send({room: room, action: 'movement', x: o.gamma, y: o.beta, z: o.alpha, method: 'orientation'});
    }
}
 
function update_text(x, y, z) {
    // updates the text on the screen
    $("#xval").text(x);
    $("#yval").text(y);
    $("#zval").text(z);
 
}
 
$(function() {
 
	var started = false;
	// socket tester.
	$("#test").click (function () {
		socket.send({room: room, action: 'test'});
		z++;
	});
 
	// this is the browser test version
	$(window).bind("mousemove", function(e) {
		var x = e.pageX;
		var y = e.pageY;
		var z = 255;
		socket.send({room: room, action: 'movement', x: x, y: y, z: z, method: 'mouse'});
		update_text(x, y, z);
	});
 
	// now we do the mobile version.
 
	socket = new io.Socket();
	socket.connect();
	socket.on('connect', function() {
		socket.subscribe(room);
	});
 
    socket.on('message', function(data) {
        switch (data.action) {
            case 'bcast':
                console.log(data.message)
                //console.log(
                break;
        }
    });
 
});

All this code is doing is simply hooking up to the socketio server (discussed below) and subscribing to a room. From there we register the event handler for the orientation events and then take the data, normalise it to make it consistent across devices and then pass it back via a websockets message to the socketio server.

Django SocketIO

The Django web sockets server is pretty much the glue in the middle. We create a view which takes web sockets messages. It does the pre-processing on the data from the phone to split it up and write it to the serial port using the correct message protocol we defined earlier.

@events.on_message(channel="^light")
def message(request, socket, message):
    #import pdb; pdb.set_trace()
    message = message[0]
    if message["action"] == "movement":
        socket.send({"action": "ack"})
 
        # pick up the values from the socket message
        x = int(message["x"])
        y = int(message["y"])
        z = int(message["z"])
 
        # now normalise the values as needed
        if message["method"] == "orientation":
            # put the vals back into +ive integer range as needed
            x += 90 #normalise 0-180
            y += 360 # normalise 0-360
 
        if x > 180:
            x = 180
        if y > 360:
            y = 360
        if z >= 360:
            z = 0
 
        # work out the y bytes
        # leaving this like this even though it's normalised back to 180 deg.
        # just in case there's any more changes to firefox.
        yh = y >> 8
        if y > 255:
            yl = y - 256
        else:
            yl = y
        # work out the z bytes
        zh = z >> 8
        if z > 255 :
            zl = z - 256
        else:
            zl = z
 
        print "x: %s y: %s z: %s" % (x, y, z)
        try:
            ser.write("%s%s%s%s%s%s%s" % (chr(255), chr(255), chr(x), chr(yh), chr(yl), chr(zh), chr(zl)))
        except:
            #do  nothing - this is a good test anyway.
            print "Doing nothing as no serial connection: %s" % SERIAL_INTERFACE
 
    elif message["action"] == "test":
        # this is a test of the socket
        print "Test of the socket"
        socket.send({"action": "bcast", "message": "got a test"})

 

Nothing too difficult with this, we register the function as a message handler and then check the message as it comes in, do some work on the data and then write it out to the serial connection with a test if something’s wrong.

This is a really basic proof of concept that can be done with a handful of components but shows the fundamentals of how this interaction can work. From here we could control servos or just about any sort of actuator you’d like.

Smart phones are becoming more sophisticated devices by the month it seems – you can even make phone calls with them I’m told. I’ve played around quite a bit with using bluetooth as a connection method between arduinos and mobile phones and whilst it works fairly well there’s a lot of overhead and your battery life is terrible for both phone and device.

One of the areas I’m looking into at the moment though is how mobile web browsers, particularly using the device APIs and web sockets can create interesting interaction points with nothing more than a browser on one side and then some interpretation of messages on the other.

Presently to achieve this I’m using Django Socket IO – a great little web sockets server and because it’s written in Python allows you full access to a serial connection to then talk to the arduino.

On the browser side it’s simply a case of hooking the relevant Device Event – devicemotion or deviceorientation then providing a call back to do something with it – in this case sending the data to a web socket connection.

// assumes some socketio code here to register to a room.
 
window.addEventListener("deviceorientation", update_gyro, true);
 
function update_gyro(e) {
	// gets the gyro position
    var x, y, z = 0;
    var o = deviceOrientation(e);
 
	update_text(o.gamma, o.beta, o.alpha);
    if ((new Date()).getTime() - last_sent > threshold) {
		socket.send({room: room, action: 'movement', x: o.gamma, y: o.beta, z: o.alpha, method: 'orientation'});
    }
}

After this we write a handler to take the data and do something with it. In this case a Django view that is a web socket message handler.

import serial
 
from django.shortcuts import get_object_or_404, render, redirect
from django_socketio import events
 
SERIAL_INTERFACE = "/dev/ttyUSB0"
SERIAL_BAUD = 115200
 
try:
    ser = serial.Serial(SERIAL_INTERFACE, SERIAL_BAUD, timeout=60)
except:
    print "Can't get a serial connection"
 
@events.on_message(channel="^light")
def message(request, socket, message):
    message = message[0]
    if message["action"] == "movement":
        socket.send({"action": "ack"})
 
        # pick up the values from the socket message
        x = int(message["x"])
        y = int(message["y"])
        z = int(message["z"])
 
        # now normalise the values as needed
        if message["method"] == "orientation":
            # put the vals back into +ive integer range as needed
            x += 90 #normalise 0-180
            y += 360 # normalise 0-360
 
        if x > 180:
            x = 180
        if y > 360:
            y = 360
        if z >= 360:
            z = 0
 
        # work out the y bytes
        # leaving this like this even though it's normalised back to 180 deg.
        # just in case there's any more changes to firefox.
        yh = y >> 8
        if y > 255:
            yl = y - 256
        else:
            yl = y
        # work out the z bytes
        zh = z >> 8
        if z > 255 :
            zl = z - 256
        else:
            zl = z
 
        #print "x: %s y: %s z: %s" % (x, y, z)
        try:
            ser.write("%s%s%s%s%s%s%s" % (chr(255), chr(255), chr(x), chr(yh), chr(yl), chr(zh), chr(zl)))
        except:
            #do  nothing - this is a good test anyway.
            print "Doing nothing as no serial"
 
    elif message["action"] == "test":
        # this is a test of the socket
        print "Test of the socket"
        socket.send({"action": "bcast", "message": "got a test"})

 

So this is really really early stage at this point and I’ve got a couple of demos that I’m building at the moment so will post them once they’ve solidified.