Project Image

Tweeting Raticator

Reporting Rat Doom via Twine

Killing Rats

When I'm not at the Media Lab, I'm at home in Somerville's own Union Square, newly minted hipster capital of Massachusetts. But, you know, we lived here before it was cool.

One down-side to the density of chic restaurants and weekly farmer's markets is a healthy population of rats. These dirty bastards aren't just narcing us out to the 411, they're also digging in the garden and shed, and scaring my neighbors with delicate sensibilities away from summer evening BBQs. We've tried the snap traps, and they are less than ideal. Without relaying the grizzly details, there is collateral damage to the local wildlife, and worse, some real American History X moments in dealing with the mangled survivors.

In 2012 (video below) we hunted the rodent hordes with a far infrared camera. But this is a job for machines.

So we turn to technology for a solution. I suppose Emerson would be proud. I present The Raticator. Look at this machine. It is a little tunnel with electrodes on the floor. When the rat goes in, it gets zapped. It's the most humane thing I've seen. And It's been pretty effective so far, except that the dainty little thing can't stand getting wet, so one has to keep an eye on the weather. The biggest problem is, the rat's favoritest place is behind the shed, and that's where the Raticator is most successful. But I might go a day without seeing that it's caught something. This is where the Twine comes in.

Making it Tweet

So the other night I invited my landlord Adam, and another friend, Scott, over to hook a Twine up to the Raticator.

The Twine is a little blue box that interfaces your Lonely Objects™ to the social web. In short, you plug in some sensors, and it can give your devices a voice on Twitter. Now, I am a curmudgeon, and primarily use Twitter to check on my laundry. This seems to be taking the world in a good direction.

The goal here is simple. Connect the Raticator to the twine, make the Raticator tweet when a rat is killed, and then make a nice web interface so I don't have to see the most recent nonsense from Dan Schultz when I check on the trap.

Ingredients

This shows the parts you'll need to make the Raticator tweet. Clockwise from the top: Raticator, 2.5mm Headphone Plug, Twine breakout board, Twine, Twine External Cable (included with breakout board), AAA Batteries. Not pictured: small-gauge wire.

Ingredients: Raticator, Twine, Batteries, Beakout Board, 2.5mm Headphone Plug, Cable.

This project is, electrically, quite simple, since the Raticator was already designed to toggle an external indicator using a 2.5mm headphone jack connector. This is visible in the back ¼ of the Raticator.

When the Raticator catches a rat, it flashes it's LED, and closes a switch (implemented with a transistor) connected to the 2.5mm headphone plug. The Twine breakout board is designed to respond to an open or closed switch. So all we need to do is connect the IN and GND pins of the Twine Breakout Board to the output jack of the Raticator. The guys over at Supermechanical have done a nice write-up here, explaining how to connect a switch to the breakout board (See their ).

You can buy a 2.5mm headphone plug from Digikey, or tear apart an old-school cellphone headset to get one. A stereo plug will work if you have one, but this only requires a mono plug (two conductors). Once you have a plug, connect one wire to the tip and one wire to the outer shell. To connect the plug to the Twine breakout board, connect the wire from the outer shell to GND and the wire from the tip to IN.

Here's a picture of me soldering some wires to the headphone connector using my mobile soldering and electronics station, the Electrocart.

Pizza Break

It's never a good idea to do too much hard work without pizza. Be sure to take a pizza break at this point. Try not to mix your solder and your pizza though.

Matt, Scott, and Adam

Finishing the Hardware

At this point, you should have a 2.5mm headphone plug with wires connected to the shell and the tip. All you have to do is plug the tip into the IN and the shell into the GND of the twine breakout board. Make sure both the Raticator and the Twine have batteries. The Raticator comes with instructions to use lithium AA batteries, and I found that I only started catching things successfully once I switched to the recommended type.

We used a rubber band (technical term for those stretchy rubber loops you sometimes get with broccoli or newspapers) and some double stick tape to keep everything in place on top of the Raticator. Here's what it looks like:

Finished Hardware

Then, you must, of course, connect the breakout board to the Twine using the included cable.

Setting up the Twine

I won't go too much into the details here, since it's a) pretty self explanatory and b) well covered over at the Supermechanical site. They have a nice support forum as well.

The basic idea is to setup a Twine rule to post a message to your twitter account whenever the breakout board switch is closed. For this application, we decided to give the Raticator it's own Twitter account. When you go to your Twine's dashboard, create a new rule for the breakout board, and select Twitter as the action. If you're already logged in to your twitter account, you'll have an opportunity to give the Twine permission to post on the account.

Pick a message with some character. You can also embed other information in the tweet to do a little data mining. Of course the tweet will be time-stamped, but we also threw in the current temperature. After bagging a few rats this way, you'll end up with a nice web-accessible record of your trophies. You should be able to see our Raticator's feed below.

You could stop here. After all, it's totally functional. But for a little extra flair, we decided to create our own custom website: http://wwwwwwww.media.mit.edu/ratdoom, that will pull from the Twitter feed to automatically update a rat kill counter.

Creating a Custom Website

This would have been a simpler task a few years ago. Just grab a javascript library to pull from Twitter client side. However, Twitter has phased out all but their latest v1.1 REST API, which focuses only on authenticated, server side, interactions. So your site will have to have a little server side magic to make this happen. Twitter uses OAuth for authentication. It's a very complex standard with lots of parts. Luckily, other people have written very nice libraries to make this easier. For example, the python library python-twitter v1.0 and newer implements OAuth and is compatible with Twitters v1.1 REST API.

The Server

To keep this super simple, I'm going to use the Apache web server with a python-based CGI script. The model will look like this:

Simple block diagram.

In this model, a client browser makes a web request to your web server for a CGI script. Unlike a flat HTML document, which is served directly to the client browser, the CGI script is run. Its output is treated as a web page and sent to the client browser. In our case, the CGI script authenticates with Twitter, queries Twitter for the number of posts on the Raticator's Twitter account, and then generates a web page from the data it has retrieved.

Interfacing with Twitter

We're interested in having our server side, CGI, script access the Twitter API for our own account only, which makes things a bit simpler than setting up an entirely general Twitter app that allows other users to log in. The first step is to generate an access token and access secret using dev.twitter.com. Twitter has created some helpful instructions. Follow them. In this step you're creating a new “application” that represents your custom web page.

Once you have created a new application, you need to get the consumer key and consumer secret from Twitter. To do this, go dev.twitter.com/apps, select the application you created in the previous step, and check out the OAuth settings section. Make note of these hexadecimal strings, since you'll need them when writing the CGI script.

CGI with Python

So a disclaimer is warranted. This is definitely not the efficient™, scalable™, fashionable™ way to get this job done. But Apache comes preconfigured in most Linux distros, as does Python. All the packages I'm going to use are available in Fedora 19, for example. I'm going for minimum effort and required knowledge here.

Most distros already have a cgi-bin directory configured for Apache. In Fedora it's located in

/var/www/cgi-bin

Files ending in .cgi placed in this directory will be executed by the server to generate a dynamic web page. If lots of users are requesting pages this way, the server will have to do a lot of work, since each instance of a script requires launching an entirely new process to handle the request. But with just a few users it should be fine.

If your Apache server isn't setup to have a cgi-bin directory, a Google search will find many helpful tutorials on this subject for your specific OS.

In order to make the Python code a little easier to read, I'm using the Python templating library Kajiki. Our python CGI script will read from an HTML template to generate the specific web page served to the user.

So without further ado, here's a simple python CGI script that you could place in your cgi-bin directory to power your custom Raticator page. You'll need to install python-twitter, and python-kajiki on your server, as well as, of course, Python 2.6. You'll also need to fill in the keys from the section above for your specific Twitter account and application.

#!/bin/python

import cgitb
import twitter
from kajiki import XMLTemplate
import math

f = open("ratdoom-twitter-template.html","r")
template = XMLTemplate(f.read())
f.close()

cgitb.enable()

consumer_key = "your key here"
consumer_secret = "your secret here"
account_token = "your token here"
account_token_secret = "your token secret here"

api = twitter.Api(consumer_key = ck, consumer_secret = cs, access_token_key = account_token, access_token_secret = account_token_secret)

statuses = api.GetUserTimeline("e_raticator")

dead_rats = len(statuses)

print "Content-Type: text/html"     # HTML is following
print
print "<!doctype html>"

rats_per_row=5
rat_rows=int(math.ceil(dead_rats/float(rats_per_row)))

print template(dict(dead_rats=dead_rats, rats_per_row=rats_per_row, rat_rows=rat_rows)).render() 

You could copy/paste this to a file called ratdoom-twitter.py.cgi in your cgi-bin directory. In order for the server to run it, you have to make it accessible and executable by the server. This is usually as simple as:

chmod ug+x ratdoom-twitter.py.cgi chgrp apache ratdoom-twitter.py.cgi

This assumes that your webserver runs as the user apache. Check your config file (On Fedora, /etc/httpd/conf/httpd.conf) to find out if yours differs.

An astute reader will note that I haven't given the Kajiki template file. I'll reproduce that below. The rest of the resources can be grabbed from my site, or recreated to your taste. Save the below to ratdoom-twitter-template.html in your cgi-bin directory.

<html>
<head>
    <title>Twine Twitter Rat Kill Getter</title>
    <base href="https://wwwwwwww.media.mit.edu/ratdoom/"/>
    <link href="ratdoom-twitter.css" rel="stylesheet"/>
</head>
<body>
    <div id="container">
        <h1>Twine Twitter Rat Kill Getter</h1>
        <br />
        <div id="ratbox_l">
            <img id="titleimg" src="raticator.png"/>
            <div id="total">Death Toll: ${dead_rats}</div>
            <br />
            <table>
                <tr py:for="tablerow in range(rat_rows)">
                    <td py:for="ratimg in range(min(rats_per_row,dead_rats-(tablerow*rats_per_row)))">
                        <img src="rat-small.png" alt="dead rat"/>
                    </td>
                </tr>
            </table>
        </div>
        <div id="ratbox_r">
            <a class="twitter-timeline"  href="https://twitter.com/e_raticator"  data-widget-id="373277551126274048">Tweets by @e_raticator</a>
            <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+"://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script>
        </div>
    </div>
</body>
</html> 

You will want to replace the Twitter widget above so that your page displays a feed for your rat trap and not mine! You can get a new widget by logging in to the Twitter account for which you wish to create the widget, and then going to twitter.com/settings/widgets. Click New Widget, and then User timeline.

Also note that if you just see a link instead of the feed you may be using a browser plugin like DoNoTrackMe, which will block the javascript from Twitter. You can usually turn it off on a per-page basis.

And here's my simple CSS stylesheet:

body {
    background-color: rgb(0,0,0);
    background-image: url(cloth.jpg);
    background-position: top left;
    background-repeat: repeat;
    background-attachment: fixed;
    font-family: "HelveticaNeue-UltraLight","HelveticaNeue-Light","Helvetica Neue Light","Helvetica Neue","DejaVu Sans",sans-serif;
    font-size: 32px;
    font-weight: 100;
    line-height: 36px;
    color: #CECECE;
}
p {
    color: inherit;
}
h1 {
    color: inherit;
    font-size: 36px;
    font-weight: inherit;
    text-align: center;
}
#ratbox_l {
    text-align: center;
    float: left;
    width: 50%;
}
#ratbox_r {
    float: right;
    width: 40%;
}
#container {
    max-width: 1000px;
    margin: auto auto auto auto;
}
#titleimg {
    margin: 0 0 20 0;
}
#total {
    margin: 0 0 0 0;
    background-color: #222222;
} 

A Prettier URL

So now your site should be totally functional. But who wants to type in something like http://wwwwwwww.media.mit.edu/cgi-bin/ratdoom-twitter.py.cgi? To make a prettier URL we can use the Apache mod_rewrite module.

To do this, often the best way is to create a .conf file in the Apache config.d directory, if your server is configured to use it. Otherwise you can just tack this into your regular Apache config file. I created a file called:

/etc/httpd/conf.d/ratdoom.conf

with these simple contents:
<Directory "/var/www/html/ratdoom">
    RewriteEngine On
    RewriteRule ^index\.html$ http://%{HTTP_HOST}/cgi-bin/ratdoom-twitter.py.cgi
    RewriteRule ^$ http://%{HTTP_HOST}/cgi-bin/ratdoom-twitter.py.cgi
</Directory>

Now you can go to the simple URL http://wwwwwwww.media.mit.edu/ratdoom. If your Apache server isn't loading the mod_rewrite module by default, a quick Google search should clear that up for you.

And here it is! This is how the site looks for me!

Ratdoom website

A Caching Server

Cache flow diagram

When John told me he might post this on the SUPERMECHANICAL.BLOG, it occurred to me that my flimsy CGI design probably wouldn't stand up to much traffic. The above stuff is fine for personal use, though. The key point to consider, I think, is Twitter's own rate limiting. In the design above, each time a user requests the page, a new process is created on the server, which runs the CGI script, connects to Twitter, and downloads the feed for the rat trap Twitter account.

To avoid being rate limited, I would like the server to cache responses. Luckily python-twitter already has a caching mechanism built in. Unfortunately, the CGI model doesn't allow us to use it in a straightforward way. The python-twitter API instance must stay alive between Twitter API calls, but in the CGI model the process exits after the request is handled.

To solve this, I've tweaked the above CGI script a bit. It now follows a modified pattern. When created, the CGI script checks to see if its custom daemon is running. If not (i.e. this is the first call) the CGI script forks a daemon. Either way, the process launched by Apache then queries the daemon, which performs the Twitter API request, and sends the result back to the CGI process.

In this way, the process containing the connection to Twitter's API remains alive, and able to cache data retrieved from Twitter. Only the CGI script has changed from above, so I'll reproduce it below. It has a couple additional requirements now: python-daemon and python-lockfile.

#!/bin/python

import cgitb
import os
import sys
import signal
import struct

cgitb.enable()

srvoutf = "/tmp/raticatorout"
srvinf  = "/tmp/raticatorin"
pidf = "/tmp/raticator.pid"

twittername = "your_twitter_account_name"

def cleanup():
	os.remove(srvinf)
	os.remove(srvoutf)

def setuptwitter():
	consumer_key = "your consumer key here"
	consumer_secret = "your consumer secret here"
	account_token = "your token here"
	account_token_secret = "your token secret here"
	api = twitter.Api(consumer_key = consumer_key, consumer_secret = consumer_secret, access_token_key = account_token, access_token_secret = account_token_secret)
	api.SetCacheTimeout(15)
	return api

def querytwitter(api):
	statuses = api.GetUserTimeline(twittername)
	return len(statuses)

def server():
	srvin = -1
	srvout = -1
	msg = ""
	api = setuptwitter()

	while msg != "R":
		srvin =  os.open(srvinf,os.O_RDONLY)
		msg = os.read(srvin,1)

		if len(msg)>0:
			dead_rats = querytwitter(api)
			if srvout < 0:
				srvout = os.open(srvoutf,os.O_WRONLY)
			os.write(srvout,struct.pack("<H",dead_rats))
		os.read(srvin,1) #remove EOF
		os.close(srvin) #close input (cgi proc will exit)
		srvin = -1
	os.close(srvout)
	cleanup()

def setup_fifos():
	if not os.path.exists(srvoutf):
		os.mkfifo(srvoutf)
	if not os.path.exists(srvinf):
		os.mkfifo(srvinf)

setup_fifos()

child = -1
if os.path.exists(pidf):
	try:
		f = open(pidf,"r")
		child = int(f.read())
		f.close()
		# check if the daemon is still running
		if not os.path.exists("/proc/" + str(child)):
			#if not, set child back to -1
			child = -1
			os.remove(pidf) #remove the stale pid file
	except:
		child = -1

if child == -1:
	child = os.fork()
	if child == 0:
		import daemon
		import daemon.pidfile
		import twitter
		context = daemon.DaemonContext(
			working_directory="/var/www/cgi-bin",
			umask=0o002,
			pidfile=daemon.pidfile.PIDLockFile(pidf)
		)
		context.signal_map = {
			signal.SIGTERM: cleanup,
			signal.SIGHUP: "terminate"
		}
		with context:
			server()

from kajiki import XMLTemplate
import math

clientout = os.open(srvinf,os.O_WRONLY)
os.write(clientout,"Q")

clientin = os.open(srvoutf,os.O_RDONLY)
dead_rats = os.read(clientin,2)
dead_rats = struct.unpack("<H",dead_rats)[0]

f = open("ratdoom-twitter-template.html","r")
template = XMLTemplate(f.read())
f.close()

print "Content-Type: text/html"     # HTML is following
print
print "<!doctype html>"

rats_per_row=5
rat_rows=int(math.ceil(dead_rats/float(rats_per_row)))

print template(dict(dead_rats=dead_rats, rats_per_row=rats_per_row, rat_rows=rat_rows)).render()

os.close(clientin)
os.close(clientout)  

This is the version of the code you'll see running if you go to my site. If you use this version of the code, make sure that you update the rewrite configuration above accordingly. I called the above file ratdoom-twitter-cache.py.cgi.

Details

email:
Matt

Contact

The best way to get in touch with me is email.

Bootstrap

This website uses twitter's Boostrap 3 CSS framework.

Untame

The theme for this site is based heavily on the How to Integrate Simple Parallax with Twitter Bootstrap tutorial offered by untame.net. Otherwise, they had nothing to do with this, so don't blame my lack of style on them! Thanks untame.net for being awesome and sharing!

Related Stuff

Supermechanical just launched a new product, which was successfully funded on Kickstarter.

I previously put my washer and dryer on Twitter, long before Twine existed.

Updates

This site was last updated on September 1, 2013.

Copyright © 2008-2013 Matt Hirsch