The PIC-Cam is a five-frame video digitizer with on-board memory. It is designed to be a self-contained video digitization solution which can connect to either a host computer or a wireless transmitter via a standard RS-232 serial port. While the digitization operations occur in real time, it is a "low-bandwidth" device, requiring only a relatively slow connection to the host machine.
The primary application of the PIC-Cam is the creation of super-resolution image mosaics, or "PICs" (Pencigraphic Image Composites). Using algorithms developed at the MIT Media Lab and elsewhere, several pictures of the same scene which have a great deal of overlap can be "stitched" together to provide a composite image which has higher resolution and a larger panoramic view than the original component images. The constraint of a large percentage of overlap leads naturally to the use of a video camera to provide the component images; thus, a real-time digitizer such as the PIC-Cam is desired to capture "sweeps" of video for later processing on a host computer.
A block diagram of the PIC-Cam is given in Figure 1. The digitizer sends data out a 16-bit data bus, upon which lie several bidirectional bus transceivers (74AS646) which act as interface chips for the 16 MB (4Mx32) DRAM SIMM, JPEG compression engine (not pictured), and UART.
The control logic is implemented on four Altera FPGAs: three 7128E and one 7160E series. The code is written in AHDL, the "Altera Hardware Description Language", which is a derivative of the industry standard VHDL and supports Altera-specific macrofunctions and optimizations. The "MAIN" FPGA controls most of the bus transcievers, the address counters, DRAM refresh, and the reset bootstrapping for the system. "7110Init" contains register initialization code for the Philips SAA7110 digitizer. "CTRCHIP" contains the address counters for the DRAM, including one for digitization, one for transmission, and one for extracting data in the order required for JPEG compression. "JPUACTL" (JPEG and UART Control) contains the initialization sequences for the JPEG engine and UART, as well as the logic to begin a compression sequence and to transmit a byte from the data bus out the RS-232 port.
The layout of the board is illustrated in Figure 2. The SAA7110 receives a 26.8 MHz clock signal from a quartz crystal, and uses an on-chip PLL (Phase Locked Loop) to generate two system clocks, one twice the frequency of the other, which are synchronized to the incoming video signal. For PAL input signals these clocks run at 29.5 MHz and 14.75 MHz; for NTSC, the clocks are 24.5454 MHz and 12.2727 MHz. These signals are buffered through two 74AS1034 drivers and distributed to the rest of the kit. The analog power supply for the SAA7110 is generated using a Motorola 7805 voltage regulator from an external +12V power supply to generate a relatively stable +5V power source.
The 7110Init FPGA has an independent oscillator which acts as its clock signal. Because the Philips chip can digitize both NTSC and PAL standards, it is possible that it will change the clock frequency of its output clocks during the initialization process (in fact, this only happens if it is locked into NTSC mode, as PAL is the default). This can wreak havoc on the initialization chip, so it must be independently clocked from the rest of the system.
The ten 74AS646 bus transceivers are used to interface the DRAM, JPEG input and output busses, and UART to the system's 16-bit data bus. A 646 also acts as a register; when clocked, it stores the eight bits of data presented at the appropriate side into internal D flip flops. This stored data can then be transmitted at a later time to the other bus. This technique is used during the digitization sequence to allow all 32 bits of the DRAM to be used to store the 16-bit-per-pixel image data. Design notes for the DRAM and ZR36050 busses are illustrated in Figure 3 and Figure 4 , respectively.
The CTRCHIP FPGA contains two 24-bit counters used as DRAM address pointers during the digitization, compression, and transmission operations. It also contains a specialized 23-bit counter which accesses the image data in 8x8 block order, as is required for JPEG compression. During a digitization/transmission operation with no compression (as is currently implemented), one 24-bit counter is used to store the current location where digitized data is being stored, while the other counts from the beginning of the memory up to the location of the first counter during a transmission sequence. When compression is enabled, one of the 24-bit counters will store the current location of the (variable-length) compressed image data; this information will then be used to determine the end of transmission. Figure 5 shows the general design constraints for the DRAM interface. Figure 6 gives a timing diagram for DRAM reads during JPEG compression. Figure 7 shows the DRAM and bus transceivers' operation during a digitization cycle. Figure 8 illustrates the address mapping for accessing the pixels of the digitized images in 8x8 block order.
The UART subsystem is designed to be able to asynchronously transmit a byte presented to it on the system bus. For this reason, the UART Controller is the only FPGA which controls its own bus transceivers. Upon receiving a transmission request, it clocks the appropriate byte from the system bus into either its high or low 646, and stores the information about which byte to send (high or low) into an internal D flip flop. Once it raises its "busy" signal, the main controller is free to perform other tasks until the busy signal goes low. Serial communication is performed using hardware (CTS/RTS) flow control at a rate of up to 57,600 bits per second. Bidirectional RS-232/TTL voltage conversion is performed using two Maxim MX233s, which are +5V-powered RS-232 drivers and receivers.
The JPEG compression chip, the Zoran ZR36050, requires a fairly large number of internal registers to be initialized upon startup. Unfortunately, blocks of these registers are at non-contiguous addresses, and the number of registers necessitates a 10-bit address. Furthermore, the address and data must be presented to the ZR36050 simultaneously. Therefore, three Intel 28F256A EEPROMs were used to store the initialization code for the JPEG chip; two contain the ten address bits, and one contains the 8-bit register data. The JPUACTL FPGA controls the transmission of this data to the ZR36050, as well as the generation of the "start of compression" signal.
Figure 9 illustrates the pin mappings for the PLCC sockets used to house the Altera FPGAs, as well as the PQFP socket for the ZR36050. Figure 10 contains diagrams of the bus termination and voltage regulator circuits.
The design of the PIC-Cam prototype began with research into what parts were necessary to construct the device, and ordering of these parts. Spec sheets were obtained, in many cases in PDF format using the World Wide Web, and initial timing charts were generated. The top-level block diagrams were written and refined, and the data bus interfaces for the various subsystems were designed. As the last parts arrived towards the end of November, approximately two and a half weeks had been spent on this process at an average of about five hours a day, for a total of about 65 hours.
At the same time the final parts were arriving, the implementation of the control logic using the Altera FPGAs was begun. The Altera development environment, Max-Plus II, contains an excellent simulator with which it is possible to simulate a chip's behavior given certain input signals with a high degree of accuracy. This simulator was used extensively during development to find several bugs before actual hardware was built. The primary coding of the control logic was done over a period of about five days at an average of fourteen hours a day, for a total of about 70 hours.
The prototype of the PIC-Cam was implemented on a wire-wrap board with power and ground planes. The wrapping of the initial hardware took seven days at an average of fourteen hours a day, for a total of about 98 hours.
Once the hardware was constructed and successfully turned on, the debugging process began. The first problem encountered was that the Philips SAA7110 was not being initialized properly. The SAA7110 contains a serial interface called the I^2C (Inter-IC) Bus as its sole means of communication with other chips. It is typically recommended that a Philips microcontroller, such as the PCF8584, be used to perform a parallel-to-serial conversion to communicate with such ICs. However, after repeated attempts, the 8584 failed to clock out any data over the serial bus to the 7110. Therefore, the serial data clocking mechanism was implemented in an Altera FPGA, and the wired-AND I^2C bus was implemented using discrete logic (a 7409, which is an open-collector AND gate).
Once the 7110 was successfully initialized, an NTSC signal was attached to the system. This caused the 7110 to cease outputting a stable system clock and the horizontal and vertical blanking synchronization pulses. The initialization code was modified to open the PLL and fix the chip in NTSC mode, in an attempt to prevent it from trying to adapt its clock frequency depending on the standard of the incoming video signal. Unfortunately, the chip failed to operate properly in NTSC mode. At this point a Philips application engineer recommended that the 26.8 MHz crystal oscillator which was driving the 7110's clock be replaced with a Philips clock crystal, to allow a feedback path from the chip to the incoming clock. Once this was implemented, the chip outputted NTSC-rate clocks and timing pulses, but continued to fail once an NTSC input was attached. It is very likely that the culprit is the stability of either the clock or the incoming analog power supply; however, at this time, the PIC-Cam only supports PAL video signals.
The implementation of the serial port was relatively straightforward. One problem encountered was the failure of the UART to transmit data. This was discovered to have two causes; first, the input clock was too fast (29.5 MHz, while the UART's top clock speed is 24 MHz), so the half-speed system clock was used instead. Second, the serial cable which was constructed was discovered to have incorrect pinouts. The serial port pin list for a PC was obtained from the World Wide Web, and the system successfully communicated with a host computer.
At this time it was discovered that the AV321 video card in DEC AlphaStations could be programmed to generate a PAL video signal. A test pattern was therefore generated and used as input to the system, at which time it was discovered that the DRAM interface was not functioning properly. Several modifications were made to the control logic to ensure that setup and hold times were met and exceeded by as wide a margin as possible. However, the major problem was ultimately discovered to be that the DRAM socket was not making proper contact with the wire-wrap board. It was firmly re-glued, and the device produced recognizable digitized video via the serial interface. The color space conversion software, implemented as part of the serial port transfer utility for the host computer, was modified to perform the YUV to RGB conversion properly.
At this point the PIC-Cam, aside from JPEG compression, was implemented. The debugging process took about seven days at an average of fourteen hours a day for a total of about 98 hours. In total, the design, implementation and debugging of the initial PIC-Cam prototype took about 330 hours.
These are scanned versions (LZW-compressed B/W TIFFs, between about 30-150k) of the original handwritten design documents for the system. Some changes were made between these designs and the final implementation; the code, below, is the final authority. In particular, the two-phase clock is not used for the generation of /RAS and /CAS.
These are copies of spec sheets for chips used in the PIC-Cam which were obtained elsewhere on the World Wide Web. The original locations are indicated next to each spec sheet.
This code was developed on a Hewlett-Packard 700 series machine running HP-UX 9. It may be general enough to compile on other platforms.
Note that at the time of this writing the JPEG subsystem had not been debugged. Therefore, this code may contain more errors than usual. Caveat Emptor!
When obtaining parts from large companies in industry, it is useful to have an idea of the relationships among the various parties involved. Such a picture often comes only from experience in dealing with such companies.
Companies such as Philips do not deal directly with developers using their hardware. Instead, other companies, known as representatives, contain the sales force for the products which the larger company produces. The rep often has knowledge of the inventories of the distributors which actually ship the parts. Both reps and distributors are often regional. Larger companies may have local field offices as well.
As of November, 1996, the rep for Philips (http://www.philips.com) is the John E. Boeing Company (Jebco); a Massachusetts office is at (508) 256-5800. One distributor for Philips' chips is Arrow Electronics (also known as Capstone), reachable at (800) 833-3557. The SAA7110 and PCF8584 for this project were obtained from Arrow Electronics.
NEC (http://www.nec.com) manufactures DRAM and many other types of semiconductors. The DRAM specifications were obtained from NEC in the form of the "Dynamic RAM Modules Data Book", document number M10584EJ2V0DBU1. The "How To Use DRAM" section provides a good tutorial on the timing constraints and refresh requirements of DRAM. However, NEC's distributors did not have the necessary DRAM SIMM (4 MB x 32 bits, part number MC-424000A32) in stock at the time. Instead, a "standard" SIMM was purchased from Kingston Memory (http://www.kingston.com), through one of their distributors, Spectrum Trading (800) 990-0902. The equivalent part number was KTM4X32L-60G (60 ns, gold contacts).
Zoran (http://www.zoran.com) manufactures audio and video compression integrated circuits. Its rep in Massachusetts is the Nashoba Group in Chelmsford, at (508) 256-9900. Spec sheets for the ZR36050 JPEG Compression Engine were obtained from this company. The distributor for this company is Edge Electronics, from which the chip was purchased.
Emulation Technologies (http://www.emulation.com), at (408) 982-0660, manufactures many types of adapter sockets for converting very fine pin pitch chips into a pin layout for prototyping. In particular, the socket for the ZR36050 was purchased from this company; the part number is AB-100-QF06Z-P8-M-1. This adapter contains a zero-insertion-force socket coupled with a circuit board which provides an array of machine pins which can be plugged into a wire-wrap board. The socket for the PCF8584 (which was at the time available only in a SOIC package), part number AB-20-SOL1-A-M, was obtained from ET as well.
Digi-Key (http://www.digikey.com) is a distributor famous for the breadth of its product line. Various capacitors and inductors, as well as the 16550 UART, were purchased from this company. Digi-Key is available at (800) 344-4539.
The original 26.8 MHz clock oscillators were purchased from General Electronic Devices in San Marcos, CA, at (619) 591-4170. This company provides a wide variety of speeds of crystals and oscillators.
An excellent discussion of serial communication, including proper pinouts for a PC's (or HP's) serial port, is available at http://dragon.herts.ac.uk/data/datasheets/serial.html.
Bill Butera saved the day by providing the initial test PAL signal from the Alpha. Without that signal the system would never have been debugged.
Hanoz Gandhi answered many late-night questions and provided much advice on how to use the available tools, both hardware and software. He also assisted greatly during the rewriting of the 7110Init chip, by suggesting the upgrade path to the 7160E.
This project was funded by Steve Mann as part of the MIT Media Lab's Wearable Computing Project.