gr_header - interleave other data into files along with samples for later retreival

Questions, comments: j c o o l e y (at) m e d i a (dot) m i t (dot) e d u
<-- back

For my Master's Thesis, I am capturing and analyzing rf spectrum data and need to be able to be able to later reconstruct the sample rate, decimation, as well as geographical location from gps on where the data was recorded.

There is certainly a variety of ways to do this, but I chose to create my own file format which allows a separate thread to modify and retrieve a small amount of data stored at the start of each block of samples. The gnuradio processing pipeline operates on an arbitrary block of int, short, or complex samples at a time.

Normally, the binary file format looks like this:

[block of samples of short, int, or complex]
[block of samples of short, int, or complex]
...

I have modified the format to store extra information ahead of each block of samples. This information is always stored as integers (regardless of short, int, or complex sample size) as follows:

[int size of header data, always present, always >= 1 to include size of sample block]
[int arbitrary data, optional]
[int arbitrary data, optional]
[...]
[int size of sample data block, always present, always >= 0]
[block of samples of short, int, or complex]
[int size of header data, always present, always >= 1 to include size of sample block]
[int arbitrary data, optional]
[int arbitrary data, optional]
[...]
[int size of sample data block, always present, always >= 0]
[block of samples of short, int, or complex]
...

Here is the c++ code that implements this. It relies on posix threads in order to guarantee exclusive access to the header data storage. Usage is the same as for the standard file source and sink blocks from gnuradio, except that the repeat option is not allowed in the file source case, and the file sink takes an initial tuple for the header:

You have to compile these files into your gnuradio libraries (see gnuradio's How to Write a Block for details on how to do this).

So, how is this all used? You need to create a separate thread to operate on the sink and source's set and query data methods. The set method can take a python tuple of any size of your choosing. The size of the tuple can change at any time, too. Here are some test programs. Note that I have placed my libraries in their own block called "vircomm", you'll have to change this accordingly:

This does work for a usrp source too, but I haven't included my example here because it has some other customizations I don't want to get into here.

There are, of course, better ways of doing this that setting/querying based on time. It's probably good enough for my application, though.