ajfisher's doing space

From the Blog

This year was the first year I had Christmas “at home” for a long time – probably over 10 years – without having to go to one set of parents or the other for the holidays. As a result, this was the first year we have had a proper tree, decorations and friends around for Christmas day since I got back into hardware.

Clearly this combination meant I was going to have to do something that used arduino and interacted with something online. Inspired by the Tweetmas Tree made by Alpha One labs a couple of years ago I wanted to do something similar – especially given twitter is my social network of choice, had hundreds of LEDs laying around as well as some spare arduinos and a Raspberry Pi.

If you want to use my code then you can get the whole package in this Gist: https://gist.github.com/4335388 If you want a gallery with lots of images and some video then check out the project page.

Concept

As with many things that are going to be situated in the family room there needs to be some discussion about “acceptable technology use”. To that end everything had to be tidy, no power supplies, no bare wires (due to little fingers on Christmas Day), the decorations had to be “tasteful” and once emplaced I had to be able to remotely administer it if needed (as other things would be around the tree – like presents).

I opted for some internally lit baubles that would have graphical icons on them representing different things (presents, Santa, family, love etc) which would activated via twitter keywords. An arduino or two would control the individual baubles by PWMing LEDs (to create a fade in, fade out effect rather than a harsh blink) and they in turn could be controlled via a Raspberry Pi over USB Serial to pass messages.

All designs have constraints leading to compromises – I had plenty of power to the tree (for the normal lights) but not much network connectivity due to where it was in the room. As such rather than running networked Arduinos (would have to run CAT5 = unsightly) or creating a mesh network or something with XBees (more fragile, more “moving” code parts) I opted for a more simple solution – use a Raspberry Pi for essentially networking and processing tweets and then use serially connected Arduinos for the light control. Each piece plays to its respective strengths and given how tiny they all are, they could be packaged under the tree very tidily with the RPi using WiFi for network connectivity.

In the design I wanted to get a sense of tweet volume so each bauble has a decay period. Thus one tweet wouldn’t individually light a bauble for very long but they would add up cumulatively so as you get closer to Christmas Day the baubles would stay lit for longer and longer periods as the tweet volume increased and then gradually die off again as we left Christmas towards New Year.

Baubles

Getting hold of plain old baubles was a royal pain; I couldn’t source them anywhere in Australia so had to turn to Ali Express to see what could be had. Thankfully a very helpful supplier in China came to the rescue once she found out what I was doing – which meant I didn’t need to buy a thousand baubles!

Once I had them, the problem with the baubles was that they were totally clear acrylic – they are so well made in fact that they look just like hard soap bubbles – which meant that any electronics inside would be totally visible and you would just get spotlights from the LED. Time for some experimentation.

After trying different ways of sanding, spraying and etching to diffuse the light, we arrived at the following method to create a nice looking bauble (big thanks to my ever suffering wife for her contribution to this):

Each design was done in Inkscape as SVG. Some I drew, some were public domain and a couple were from the Noun Project (attribution at the end).

Using an eggbot, the design was drawn on the bauble. This had its own challenge as the hook on the bauble meant you had to test the design to not include a path where the bauble would rotate and hit the pen on the hook during travel. This was as simple as just running the eggbot with no pen and running it at a higher drawing speed to test.

Bauble being drawn by eggbot

Once the design was drawn on the surface I then used an engraving tool to etch the design on the internal, concave face of the bauble. This allowed the light to catch an edge and diffract, highlighting the design in the diffused bauble more strongly. Doing this by hand gave me a good excuse to get an engraving head for my EggBot for next Christmas.

Etched Santa

Next up we sanded the outside face of the bauble with varying grades of sandpaper / wet and dry. Starting from about 200 through to 800 grit. The rough starting point removed the original design, leaving the etch and really scratched up the surface of the bauble. The finer grades smoothed out the scratch marks and left a surface that nicely diffused and made the whole bauble illuminate when hit by light.

20121219_142558

A tiny notch (3mm) was put in the bauble edges near the eyelet in order to allow the LEDs wires to pass in. This was purposefully kept a very tight fit as it helped with keeping the LEDs in place. A knot in the wire on the internal side and then passing the wire through the eyelet (before attaching a ribbon for hanging) provided some strain relief as well as holding the LEDs in a nice spot for light.

Finally, the internally etched design was inked using a Sharpie to make the design stand out when lit internally.

LED control

I had a large number of super bright white LEDs left over from another project so I decided to use those. Warm white LEDs would have looked a little nicer as plain white LEDs are quite cold but they do look icy and contrast nicely to the warm white lights we had on the rest of the tree.

The LEDs were run off a pair of 9V supplies that were lying around – one for each of the arduinos. The arduinos would be run from the 5V USB supply so we only needed to worry about powering the LEDs.

Two LEDs were more than enough to light the baubles however being water clear they were too focussed. Out with the sandpaper again and after a minute or so each you have nicely diffused LEDs.

Hand diffused LED

I had some concerns that at full tilt the LEDs would be drawing too much current through the arduino as there were 4 baubles attached to each one, and I wanted to use 9V in order to be able to power two LEDs in series (5V will only power one due to the voltage drop). Using a simple NPN transistor switching circuit was a simple solution for this. Now there were a few too many bits per bauble to simply wire together, I’d need to make a shield.

Bauble Shield

The shield is dead simple. A screw terminal for each bauble for easy connection and removal, through a transistor and connected to the 9V power supply which was set up as an alternative input and rail on the shield. Each bauble’s transistor is switched off an arduino PWM IO pin.

Bauble Shield

Arduino Code

With the shield done and tested the code to control the lights was done next. The arduino would receive a message over the serial connection using a simple protocol:

<Channel Number> <msecs>\n

To turn a light on, all that was needed was to send a channel number and the time to turn it on for. If the light is off it will fade it up and then stay lit for more or less the number of milliseconds sent through. If the light is already on then the time will simply be added to the number currently left for that light to display – this is how multiple tweets in a short period can “stack” and keep a light illuminated.

The code is a bit long for here but you can see it in the gist.

The code was designed to run in parallel using a state machine so the state of each light was maintained and managed in the loop so no instruction would block the others if it was fading or checking the serial connection for incoming messages. This is a technique I use a lot and is similar to coding the animation of multiple sprites with different forces being applied to them so I can use the same concepts when designing a physical interaction.

An additional concern was that I wanted each of the Arduinos to not have any config code as I didn’t want to have to reflash them at any point. As such each arduino was given a “LIGHTNODE” value (0 or 1 in this case as there were only 2) and some code to identify itself (sending “n\n” over the serial connection). This meant that if an arduino reset for any reason and came up on a different serial device, the Raspberry Pi would still understand which one it was talking to. With a bit more time I should have probably put that in the EEPROM of the arduino itself (and having come across this multiple times before, I’m convinced arduino makers should burn UUIDs into each device to solve this).

At this stage we’ve now got baubles that can be lit via serial commands from a PC based on any input we decide to throw at it. Time to give it one.

Twitter stream processing

Twitter processing in Python is trivially simple to do using Tweepy. The hardest part is really getting all your OAuth creds set up properly. The easiest way to do this is to go to dev.twitter.com and set up developer access. From there create an application and on the application details page you’ll have the crucial pieces – Consumer Secret, Consumer Key, Access Token and Access Token Key.

Again, for space I won’t repeat the code but the script can be found in the gist.

Once authenticated the code is really just watching the twitter streming API for a bunch of keywords defined in the config.py file. When one is found it gets pushed into the callback listener class for processing which determines which keyword fired and which bauble should be lit. This results in a serial command to the relevant arduino and channel to do it.

As part of the stream class you have to implement handlers for error and timeout. In this case we don’t really care about them, we’ll just log them to StdErr so they can be viewed if needed but I don’t want the application dying as a result.

The only exception to this is a rate limited message from twitter – which essentially tells you you’re hitting the API too hard. At this point I just sleep the script for 5 seconds to slow things up a bit. If you get to this point the baubles will probably all be lit anyway so you’re not missing much.

All of this is run from a virtualenv on the Raspberry Pi which is the usual way most people work with python in order to manage varying dependencies (which in this case is Tweepy and PySerial). The script is then run in a screened terminal instance so it can be detached and still run. I could have used supervisord to have it fully daemonized but it wasn’t really worth it given this was all that was running on the RPi and it was trivial to restart via a quick SSH command if it was necessary (it wasn’t).

Gotchas, lessons and changes

RPi Power

The biggest issue that came out of this project was the instability of the Raspberry Pi seemingly due to power supply issues. I had the arduinos running off a laptop doing the processing for about a week without any restarts or issues at all. The Raspberry Pi had a string of issues however. This may be my Pi in particular but reading reports of general power supply issues makes me think it is not. Even running everything through a powered USB hub for the arduinos and WiFi adapter didn’t help particularly.

In the end I removed the WiFi adapter and the stability improved markedly. To get network connectivity I resorted to using PowerLine Ethernet which worked extremely well.

There were still stability issues when there was a lot of network activity and messages going to the serial connections at the same time which never seemed to resolve fully with different power adapters.

I understand why the RPi team chose to use 5V power adapters over MicroUSB given the ubiquity of mobile phone chargers, however the variability in output voltage stability across adapters is immense. I used everything from iPod adapters (apparently capable of 2A) to Galaxy phone charger (only capable of 0.7A but extremely stable voltage under load) to sony and Asus chargers (0.8A and 1.2A respectively but terrible stability under load with wild swings). Even different USB cables could provide a difference by up to 0.2V on the same charger. In the end I settled for the Samsung charger at 0.7A with a powered USB hub to smooth out the drops but it still resulted in an occasional crash.

Phone chargers, whilst ubiquitous, are designed to trickle charge a battery, not to power a device (if you’ve ever tried to USE your phone when it’s nearly empty and it’s on charge you’ll have seen this first hand) so the vagaries you wouldn’t notice on a charger are very noticeable on a Pi doing some processing.

Given this and the crashing I experienced as a result, were I doing this again I’d probably use a more “serious” 5V power supply designed for truly powering a device and then run it in via the Vin pins or make up a micro USB connector.

As an aside, overclocking did seem to help with this – I’m not sure why but it almost doubled the time between failures.

Twitter rate limiting

Initially I designed the twitter code to run in threads, one for each bauble, tracking keywords independently. It was a nice bit of code that worked well until I scaled it out to the eight baubles and started getting HTTP 420s back from twitter. This was because I was running too many authenticated streams at once. This was annoying as it meant I ran all my keywords in one stream instead (one connection but 8x the work) and had to do processing on the RPi side to pull the tweets apart and route to the appropriate bauble. Unfortunately there’s nothing you can do with twitter’s rate limiting policy but a couple of dummy accounts could have probably solved that.

Finished result

I am quite pleased with the finished result. The effect was “tasteful” enough to pass the family and friend test, I got to actually build something that was in use the entire Christmas period and whilst I probably overstressed the problems I had with power for the RPi it didn’t affect things short of a quick power cycle once every several hours (and with more time I could have resolved properly).

The code, the shield design and the baubles are all reusable so I can use this again next year with little effort (or extend it) and it’s all available in GitHub (if anyone wants the shield design let me know and I’ll post one) free for anyone to use.

Many in the arduino / maker community are exposed to this sort of thing a lot and connecting lights to twitter is pretty easy so the effort to reward ratio of a project like this is very high (and allows you to concentrate on the design of the outward expression of the visualisation). Other than my wife and child, none of the other twenty friends around on Christmas Day get exposed to this type of display so the novelty value was really good and there was heaps of interest as a result.

In the words of a friend of ours, “It makes me feel as though there are people all over the world celebrating with their family and friends just like we are, and you’ve brought them all into the room with us” – and if that’s not what doing this sort of technology is all about then I don’t know what is.

Attribution

The icons I used for Family and the Snowflake were from the Noun Project. The family icon was designed by Marco de Silva and the Snowflake was by Danielle Garbouchian.

This is the distillation of several projects I’ve worked on over the last year. I’m only showing one of these as I actually captured my build process on the most recent; being an interactive light display for a demo during my presentation at the Sketching in Hardware conference in Portland, Oregon.

This approach isn’t just limited to lights. Any time you want to make an interactive display that numerous people can interact with at once then this is a good approach – and it’s web based for interaction so you can do all sorts of interesting interfaces.

I’m not going to go into the serious detail of under what conditions you would want to do this and things around messaging architecture etc. If you want to find out more, please see the talk I gave at Linux Conf and this one at Sketching (where this project was demoed).

Overview

This project consists of three parts:

  • * A networked ardiuno that takes messages from the server and controls the RGB light display.
  • * A web interface where users can touch or click circles representing the lights and change them different colours.
  • * A web server which processes all of the messages between the web interface and the arduino.

When a user visits the site on their mobile or desktop device they will be presented with an interface that allows them illuminate a light by clicking or touching a circle on screen. Users can change their “personal” colour actively or using a randomiser and where two users illuminate the same light their colours will mix. The lights will take the colour passed in the message from the server and then gradually fade to nothing over time.

Stuff you need

There are a few things you’ll need to get hold of:

  • * A networked arduino – I prefer the Freetronics EtherTen but and arduino with Ethernet Shield will work just as well.
  • * Enough light modules to make your display. Mine are Freetronics RGB Modules (which this code is written for) but you could use BlinkMs or other light modules. The key is they need to be individually addressable and only use a few pins on your arduino. I used 19 for my display which was arranged like a filled hexagon.
  • * A computer that can run as a server. This was designed for Linux, I’ve tested it on my Mac as well but for Windows, there’s no guarantees.
  • * A stack of jumper wires or other to join your light modules together.
  • * Headers to attach to your light modules.
  • * Something to mount your lights onto (cardboard, perspex etc)
  • * Something to diffuse the light (I used ping pong balls which work really well)

Software to install

Server

Let’s get the server up and running first. I’m assuming both Linux and Mac here, I’m also assuming you have python installed and you are running VirtualEnv and you’ve created a new virtual environment to play in.

My environment was called sketching so let’s work on that:

$ workon sketching

Now we’ll get the code and install the dependencies

$ git clone git://github.com/ajfisher/sketching-conf-demo.git
$ cd sketching-conf-demo
$ pip install -R requirements.txt

Pip will then install everything you need. You may run into a couple of lib event issues if you’ve never installed it before. See the notes here under installation which may apply (particularly if you’re running OSX): https://github.com/stephenmcd/django-socketio

Now you have everything installed without errors test it with:

$ cd sketching
$ python manage.py runserver_socketio 127.0.0.1:8000

You should get notice that the server is running. At that point, direct your web browser to http://127.0.0.1:8000 and you should get a screen with a series of circles on it.

If this is working then the django server is pretty much good to go, the big thing you’ll need to change is to make the server accessible beyond just your computer. As such run the server like this:

$ python manage.py runserver_socketio HOST_IP:PORT

Where HOST_IP is your IP address of your computer on the network and the port is the port number you want to run on. I generally run on about 8000 or so as low port numbers require running as root and for the moment you can continue to use high numbers for development.

Make sure your server is up and running before connecting your arduino up.

Arduino setup

The first thing you need to do is include the ArduinoWebsocketIOClient in your libraries. The easy way to do this is simply copy the library across to your sketchbook/libraries folder and dump it in there and restart your Arduino IDE. I prefer to do this a little more cleanly and symlink the foilder from my sketchbook so if I make any changes it’s still part of my git repo and I can manage them.

Once you’ve done that and restarted the IDE so it’s available to the environment open up the “Connect to Server” sketch to test you can actually do just that.

You will need to change the vales of the server variable on the line that looks like:

char server[] = "10.0.1.15";

and the port number on the line that looks like:

int port = 8000;

To whatever the address was you set your server to run on.

Compile your sketch then upload it to your arduino. Make sure it’s connected to the network and then power it on and connect to it using the serial monitor as well.

The aim of this sketch is to check that you can connect to the server, subscribe to the “pong” socketIO channel and detect messages coming through.

If the connections and subscriptions don’t work ensure your arduino can see the network, can see your server (eg firewalls have the right rules) and watch the log where your server is running as it will spit out a wealth of info about whether the device connected or not.

Assuming this has all worked, then you can load up the interface in your browser by going to http://ip:port/ and if you click on the circles you should see messages appear on the serial monitor with the RGB values and the position of the circle you hit.

Once you’ve confirmed all of this is working you can load the real sketch onto the arduino called “colour controlled with delay”.

Build the display

Lights

You can arrange this however you want if you want to change the HTML as well but my version set the lights in a filled hexagon shape that was offset with 3 lights then 4, then 5 then 4 then 3. Using an offset grid makes it easy to stack circles together as you can pack them densely – they also look good. To make this arrangement we need 19 RGB modules.

I got the modules and soldered 0.1″ male header to each one. Testing each one in a jig I made to ensure they worked as I was finished. This simply cycles through RGB values and you drop the new module in alongside one know to work and you can test that it accepts input and passes values along the chain.

As an aside these modules use a “ripple” effect to pass the values along the chain. It happens really fast so you can’t see it but you can effectively chain them as long as you like and you just pass the messages in a list and it flows down the chain. If you want to change just one light in the middle though, you have to pass the entire list again with just that one module changed. It only takes about 20ms to do this though so it’s pretty fast.

Now we have our modules, we need to wire them together. I use pre-crimped wire that was about 8cm long from Poloulu as I hate crimping wires and the prospect of doing 160 ends would have sent me mad. If you want to do it though, go right ahead. Once done you just connect them all up together into a great big long chain, output to input along the length.

Once all of the modules are wired up, you connect the 5V and GND wires to the arduino and then connect the CKI pin on the module to arduino pin 2 and the SDI pin of the module to arduino pin 3.

Test that your lights are all working correctly by loading the “colour sequence test” sketch onto the arduino. You should see each of the lights cycle through a set of colours in turn almost like a christmas light display. This will help you debug any wiring issues with your modules.

Once you have all of the lights working then you can mount them.

Mounting the lights

My display was designed to be pulled apart for transportation but is surprising resilient as well. Holes are made in the mounting media between the lights and the wires are folded in half and poked through the holes. Ping pong balls mounted on top of the lights with a small hole cut out apply pressure outwards on the wire which creates enough tension to hold them in place. This is especially the case when you offset rows like bricks.

I tested the design on thick cardboard first just by poking holes through it to get the measurements correct. This all depends on what you use to diffuse the light. My ping pong balls were about 4cm in diameter. I actually discovered there’s a fair amount of variation here so you’ll have to adjust for your own components.

Once I had the measurements correct, I transferred it to 10mm acrylic which had 5mm holes drilled into it. Using wet and dry I sanded the hole edges to make them smoother after stripping a couple of wires pushing them through. The fit will be tight but will hold very securely.

Light fittings

Ping pong balls make surprisingly good diffusers for LEDs. I cut 5mm square holes in them and then run a line of “white tac” (like blutac but white) along the perimeter of the square and then press them down onto the RGB modules. This holds really well and can be removed. If I was making this permanent, something like sugru would be appropriate and will bond tightly.

I work from the centre outwards in a spiral as it makes it easier once you have many balls in place as the pressure of them will make working difficult in the middle. One thing you’ll notice as you’re nearly finished is that the top of bottom rows are almost floppy but the centre ones are nice and tight. We’ll fix that in the next step.

Using either tiny spots of superglue or Sellotape “super small dots” which are amazingly sticky, stick each of the balls to its neighbour where the surfaces touch. As I mentioned above the centre ones won’t really need it because they are packed so tightly but you can use this to transfer some of that rigidity to the top and bottom rows which will then pack in nicely with the rest.

Once you’re finished you’ll end up with 19 ping pong balls that are tightly packed together and very rigid. I also pinned a connector wire to the acrylic to make it easy to connect the arduino to it without the wire flapping around.

Running

Connect up your arduino as before, 5V, GND, pin 2 to CKI and pin 3 to SDI. You can run your colour test again if you want to make sure it’s all good or now load up the “colour controlled with delay” sketch, changing the IP and port numbers to that of your server.

Now once you hit the web page with the interface on it, each time you select a circle you’ll get the corresponding ping pong ball light up on the display.

Multiple people can all play with this at once so once you’ve got your display get a bunch of devices together and play around with it. You can get some interesting interaction effects happening when you have more than one person hit the same light at once or just after each other as it’s fading.

Where to now?

There’s a lot you can do with this system. I’d love to see someone do this with a building or other types of interactive display. If you do, drop me a note / post a comment and send me a link. Video and a wider gallery of photos is available on the project page.

Acknowledgements

Thanks to Steve McDonald who put together the Django SocketIO library that underpins a lot of the server side of things. Also thanks for Kevin Rohling who wrote the original arduino web sockets library that I adapted to work under socketIO.

I’m a big fan of networked arduinos – especially these EtherTens from Freetronics. When an arduino is networked it opens up a degree of interactivity and awareness that is hard to achieve with serial – not least because your processing can occur somewhere physically distant from your display.

This project (pictures and video in action on the project page) was a proof of concept on exactly that set up, having remote processing that could be anywhere – in this case processing the twitter streaming API for keywords, and sending messages over the network to an arduino which would then manage the display. There are a lot of different ways you could do this including using things like web sockets and doing pubsub messaging etc but this was designed for simplicity and to prove the model more than anything.

To build this project you need an arduino that has networking capability – as mentioned above I use an EtherTen but you could use a standard Ethernet shield as well. Similarly you could do with a WiFi shield too with only a few mods to the code.

The arduino is simply connected to an RGB LED (or light disc in my case) with each of the colour pins coming through to a PWM pin on the arduino. In the code below there are references to “channels” – this code makes very little assumptions about the set up so it simply has the notion that there are PWM channels and you want to do something with them.

The full arduino code is here, however here’s some highlights:

  channel[0] = 3; // etc this will point channel 0 to Digital PWM pin 3
 
  // set up each of your channels as an output.
  for (int i=0; i<MAX_CHANNELS; i++) {
    if (channels[i] > 0) pinMode(channels[i], OUTPUT);
  }

The above code is in the setup() function and maps each of the channels to the relevant PWM output (technically you could actually use any output but you won’t get an analog range on it later).

This next block of code actually manages the timing of the various channels and sets them independently of each other. The code is run much like a state machine with each channel operating independently of the others so there’s a countdown counter that determines how much longer to pulse the channel and that counts down over the loop, eventually switching off the channel when it expires.

  // check the situation with the channels and fade out if the timer has exhausted
  boolean active_channels = false;
  for (int i=0; i<MAX_CHANNELS; i++) {
    if (channels[i] != 0 && channel_timers[i]> 0) {
      DEBUG_PRINT("Timer CD");
      DEBUG_PRINT(i);
      DEBUG_PRINT(": ");
      DEBUG_PRINTLN(channel_timers[i]);
      channel_timers[i] -= (millis() - last_time);
      last_time = millis();
      if (channel_timers[i]<= 0) {
        channel_timers[i] = 0;
        fadeout(i);
      } else {
        active_channels = true;
      }
    }
  }
  if (!active_channels) last_time = 0; // reset the master timer to zero if nothing's using it.
 
  delay(5);

And along with some adaptations of the standard web serving code this pretty much runs just like any other arduino web server.

Assuming you’ve got this up and running you should be able to hit http://yourip-address/1/1000 from a browser and it will pulse channel 1 for 1 second. You can do the same thing on 0, 2 & 3 as well if you have all 4 channels operating. To change channels, change the first part of the URL and to change the time it pulses for change the second parameter thus http://yourip-address/[CHANNEL]/[TIME]

Once that’s running you can then script just about anything to hit those URLs and cause the light to pulse for a defined duration. Here’s how I did it with twitter. This is a python script that runs off the command line. Again, I won’t show the full code but you can get it here.

The main bit of code that does the calling is this:

stream = FilterStream(args.user, args.pwd, track=words)
 
for tweet in stream:
    print "+++++"
    print tweet["user"]["screen_name"]
    print tweet["text"]
 
    try:
        response = urllib2.urlopen(args.url).read()
    except Exception, e:
        print "Unexpected error:", sys.exc_info()[0]

Here, the stream is created (you have to supply twitter credentials to access the streaming API. Words is simply a list of keywords you want to watch. Then every time the stream gets a new tweet it will go round the loop, spitting out the tweet and then calling the URL that was defined (eg http://someip-address/CHANNEL/TIME).

This script is designed to run from the command line so you can run multiple of these as background processes, each of them calling your arduino whenever a tweet comes through that matches. I’ve run this for days at a time without any drama so it’s pretty stable.

Whilst this has been designed to work with twitter, the process is general and I’ve used this with all manner of different sources such as getting an email or an order occurring on a website. Anything you can trap as an event you can then trigger the arduino to do something.

As a web oriented person I tend to think web-first whenever I think of mobile. I posted a little while back about how to get a mobile phone to pull it’s orientation data out using some javascript and pass that data to a web server. This has evolved somewhat now and I have a packaged Git Hub project that shows you how to do it properly.

One of the demo projects is controlling an RGB light using the web browser’s gyro in order to get it’s orientation then mapping that to the three colour channels.

There are three core parts to the application:

Arduino / light set up:

In this scenario I’m just using a serial connection for simplicity. All the arduino is doing is looking for data that’s packaged up as a sequence of bytes that looks like this:

Byte 0 (int) 255 header byte
Byte 1 (int) 255 header byte
Byte 2 (int)  0-180 val for x
Byte 3 (int)  high byte for y
Byte 4 (int)  low byte for y
Byte 5 (int)  high byte for z
Byte 6 (int)  low byte for z

The code simply looks at the incoming data stream, looks for the double 255 bytes then reads the next 5 bytes into a buffer and converts them into variables. Check out the code on github.

Once it has the data it then maps to the RGB LED appropriately.

Web Browser

Modern mobile browsers (Firefox on Android, Mobile safari on iOS) give you access to the device API. This give us device motion and device orientation events that will fire every time the device moves. Using the orientation event we can trap the data about the way the device is facing then send that back via web sockets to the server.

var socket;
var room = "light";
 
var last_sent = (new Date()).getTime();
var threshold = 100; // msec between sends
 
window.addEventListener("deviceorientation", update_gyro, true);
 
function update_gyro(e) {
	// gets the gyro position
    var x, y, z = 0;
    var o = deviceOrientation(e);
 
	update_text(o.gamma, o.beta, o.alpha);
    if ((new Date()).getTime() - last_sent &gt; threshold) {
		socket.send({room: room, action: 'movement', x: o.gamma, y: o.beta, z: o.alpha, method: 'orientation'});
    }
}
 
function update_text(x, y, z) {
    // updates the text on the screen
    $("#xval").text(x);
    $("#yval").text(y);
    $("#zval").text(z);
 
}
 
$(function() {
 
	var started = false;
	// socket tester.
	$("#test").click (function () {
		socket.send({room: room, action: 'test'});
		z++;
	});
 
	// this is the browser test version
	$(window).bind("mousemove", function(e) {
		var x = e.pageX;
		var y = e.pageY;
		var z = 255;
		socket.send({room: room, action: 'movement', x: x, y: y, z: z, method: 'mouse'});
		update_text(x, y, z);
	});
 
	// now we do the mobile version.
 
	socket = new io.Socket();
	socket.connect();
	socket.on('connect', function() {
		socket.subscribe(room);
	});
 
    socket.on('message', function(data) {
        switch (data.action) {
            case 'bcast':
                console.log(data.message)
                //console.log(
                break;
        }
    });
 
});

All this code is doing is simply hooking up to the socketio server (discussed below) and subscribing to a room. From there we register the event handler for the orientation events and then take the data, normalise it to make it consistent across devices and then pass it back via a websockets message to the socketio server.

Django SocketIO

The Django web sockets server is pretty much the glue in the middle. We create a view which takes web sockets messages. It does the pre-processing on the data from the phone to split it up and write it to the serial port using the correct message protocol we defined earlier.

@events.on_message(channel="^light")
def message(request, socket, message):
    #import pdb; pdb.set_trace()
    message = message[0]
    if message["action"] == "movement":
        socket.send({"action": "ack"})
 
        # pick up the values from the socket message
        x = int(message["x"])
        y = int(message["y"])
        z = int(message["z"])
 
        # now normalise the values as needed
        if message["method"] == "orientation":
            # put the vals back into +ive integer range as needed
            x += 90 #normalise 0-180
            y += 360 # normalise 0-360
 
        if x &gt; 180:
            x = 180
        if y &gt; 360:
            y = 360
        if z &gt;= 360:
            z = 0
 
        # work out the y bytes
        # leaving this like this even though it's normalised back to 180 deg.
        # just in case there's any more changes to firefox.
        yh = y &gt;&gt; 8
        if y &gt; 255:
            yl = y - 256
        else:
            yl = y
        # work out the z bytes
        zh = z &gt;&gt; 8
        if z &gt; 255 :
            zl = z - 256
        else:
            zl = z
 
        print "x: %s y: %s z: %s" % (x, y, z)
        try:
            ser.write("%s%s%s%s%s%s%s" % (chr(255), chr(255), chr(x), chr(yh), chr(yl), chr(zh), chr(zl)))
        except:
            #do  nothing - this is a good test anyway.
            print "Doing nothing as no serial connection: %s" % SERIAL_INTERFACE
 
    elif message["action"] == "test":
        # this is a test of the socket
        print "Test of the socket"
        socket.send({"action": "bcast", "message": "got a test"})

 

Nothing too difficult with this, we register the function as a message handler and then check the message as it comes in, do some work on the data and then write it out to the serial connection with a test if something’s wrong.

This is a really basic proof of concept that can be done with a handful of components but shows the fundamentals of how this interaction can work. From here we could control servos or just about any sort of actuator you’d like.

Smart phones are becoming more sophisticated devices by the month it seems – you can even make phone calls with them I’m told. I’ve played around quite a bit with using bluetooth as a connection method between arduinos and mobile phones and whilst it works fairly well there’s a lot of overhead and your battery life is terrible for both phone and device.

One of the areas I’m looking into at the moment though is how mobile web browsers, particularly using the device APIs and web sockets can create interesting interaction points with nothing more than a browser on one side and then some interpretation of messages on the other.

Presently to achieve this I’m using Django Socket IO – a great little web sockets server and because it’s written in Python allows you full access to a serial connection to then talk to the arduino.

On the browser side it’s simply a case of hooking the relevant Device Event – devicemotion or deviceorientation then providing a call back to do something with it – in this case sending the data to a web socket connection.

// assumes some socketio code here to register to a room.
 
window.addEventListener("deviceorientation", update_gyro, true);
 
function update_gyro(e) {
	// gets the gyro position
    var x, y, z = 0;
    var o = deviceOrientation(e);
 
	update_text(o.gamma, o.beta, o.alpha);
    if ((new Date()).getTime() - last_sent &gt; threshold) {
		socket.send({room: room, action: 'movement', x: o.gamma, y: o.beta, z: o.alpha, method: 'orientation'});
    }
}

After this we write a handler to take the data and do something with it. In this case a Django view that is a web socket message handler.

import serial
 
from django.shortcuts import get_object_or_404, render, redirect
from django_socketio import events
 
SERIAL_INTERFACE = "/dev/ttyUSB0"
SERIAL_BAUD = 115200
 
try:
    ser = serial.Serial(SERIAL_INTERFACE, SERIAL_BAUD, timeout=60)
except:
    print "Can't get a serial connection"
 
@events.on_message(channel="^light")
def message(request, socket, message):
    message = message[0]
    if message["action"] == "movement":
        socket.send({"action": "ack"})
 
        # pick up the values from the socket message
        x = int(message["x"])
        y = int(message["y"])
        z = int(message["z"])
 
        # now normalise the values as needed
        if message["method"] == "orientation":
            # put the vals back into +ive integer range as needed
            x += 90 #normalise 0-180
            y += 360 # normalise 0-360
 
        if x &gt; 180:
            x = 180
        if y &gt; 360:
            y = 360
        if z &gt;= 360:
            z = 0
 
        # work out the y bytes
        # leaving this like this even though it's normalised back to 180 deg.
        # just in case there's any more changes to firefox.
        yh = y &gt;&gt; 8
        if y &gt; 255:
            yl = y - 256
        else:
            yl = y
        # work out the z bytes
        zh = z &gt;&gt; 8
        if z &gt; 255 :
            zl = z - 256
        else:
            zl = z
 
        #print "x: %s y: %s z: %s" % (x, y, z)
        try:
            ser.write("%s%s%s%s%s%s%s" % (chr(255), chr(255), chr(x), chr(yh), chr(yl), chr(zh), chr(zl)))
        except:
            #do  nothing - this is a good test anyway.
            print "Doing nothing as no serial"
 
    elif message["action"] == "test":
        # this is a test of the socket
        print "Test of the socket"
        socket.send({"action": "bcast", "message": "got a test"})

 

So this is really really early stage at this point and I’ve got a couple of demos that I’m building at the moment so will post them once they’ve solidified.