article3.htm

基于FPGA的嵌入式机器人视觉识别系统模块源代码

<table align="right"><tr><td><center>
<h3>figure 8.1: Additional Verilog Structure</h3>
<p>
<img src="floor2_plan.gif"></table>
Given all of the structure, video capture and blob detection implemented in the Verilog code above, it takes surprisingly little additional code, presented left, to implement laser range finding.  This code is instantiated from the file video_capture.v.  This module and the dual-ported block RAM module it instantiates are highlighted red in fig.8.1, right.  One piece that really shines through here is the encapsulation and layering functionality offered by a modeling language like Verilog.
<p>
The premise of operation is simple.  The code instantiates a 512x8 block RAM (dual ported) using one of the dedicated hardware resources of the FPGA.  The module monitors the incoming address during valid video captures.  As long as the video <code><b>line_counter</b></code> variable is zero, each successive pixel location scanned is stored as a zero effectively initializing the function's data space.  After line zero is done, pixels are only stored if they pass the window comparator values set through registers from the processor bus.  This is just another way of saying a minimum and maximum setting for R, G, and B are compared against the color space value of each pixel.  If a match is found, the line number (<code><b>line_counter</b></code>) is stored in the linear array indexed by the variable <code><b>pixel_counter</b></code>.  
<p>
The fact that video presents, time-wise, top of screen to bottom of screen, in this case, is very handy.  The laser range finding function only keeps a 2-dimensional record of data points in front of the robot.  The concern is always with objects that are closer, which will over write objects (false readings, etc) that recorded previously (farther up the video image).  The important correlation here is that objects closer to the robot occur later in time in the video stream.  Note: The previous assertion only holds true for the physical / mechanical configuration of laser and camera depicted in fig.8.4, below.   <i>For a discussion of the mechanics of laser range finding functionality, refer to the authors previous article on the subject that covers this and other topics in detail.</i>  Later, to read the data back out of this linear array, the processor bus latches an address into the second side of the dual port, and the data is fed directly out.
<p>
The two images below come from a piece of 2D graphing terminal software.  They show laser returns, recorded with this color video system, looking at both a square item (fig.8.2) and a cylindrical item (fig.8.3).  In this application, two methods where used.  First, for a small robot that performs in a controlled environment like the Trinity Fire Fighting competition, it is sufficient to simply set the optical filter values within the FPGA.  These filter values are memory-mapped registers written to through the u-Processor bus interface.  The filter values are configured to "look for" / pass the red color of the laser light and the function in the FPGA can pull the data out of the video image automatically.  For implementation in other noisy real world scenarios (contrasted against the engineered environment of the Trinity contest), the reader should consider the use of optical filters like those suggested in the authors other articles on laser range finding (link below).
<p>
<br clear="right"><p>
<center>
<h3>Figure 8.2: Laser Range Image Square</h3><p>
<img src="range_data.gif"width="562"height="177">
<p>
<h3>Figure 8.3: Laser Range Image Spherical</h3><p>
<img src="range_data2.gif"width="534"height="149">
</center>
<p>
<table align="right"><tr><td><center><h3>Figure 8.4: Physical Setup for Laser Ranging</h3><p><img src="para_4.gif"width="583"height="438"></table>
Referring to the author's earlier articles, the images above (8.2 & 8.3) are returns from a physical setup modeled as shown in fig.8.4 right.  The difference between that setup and the one used with the color FPGA implementation is that the camera is <u>not</u> rotated 90deg onto its side.  Additionally, the signal acquisition is performed by extraction of color / signal level within the FPGA as opposed to the use of comparators referenced against the analog black and white signal of the pervious version.  To read more on that implementation, follow this link to a previous SRS-Encoder article: <a href="http://www.seattlerobotics.org/encoder/200110/vision.htm">A Real Time Laser Range Finding Vision System</a>.
<p>
This graphing terminal program is invaluable for developing distance based robotics sensors.  This program, G-Term, was developed by a friend of the author, <u>Alan Erickson</u> and is available for download by amateur robotics enthusiasts.  This program was used to generate fig.8.2 and fig.8.3, above.
<table><tr><td><center><a href="http://www.seattlerobotics.org/encoder/200110/G_Term.zip"><img src="lg5.gif"><p><b>G-Term</b></a></table>
<br clear="right">

<p>

<p>
<table width="340" align="left"><tr><td><center>
<h3>Figure 8.5:</h3>
<p>
<img src="new_set_010.gif"width="320"height="240"alt="laser range data overlaid on image through FPGA / processor interaction.">
</table>
The last image in this section (figure 8.5) is a direct digital captures from the vision system.  The image shows an overlay of tracking of the structured light output extracted by the FPGA real time and provided to the processor system at the same time as the captured image.  This image was captured through the use of a second piece of software, I2Term also written by Alan Erickson.  It is presented at the bottom of the next section.
<p>
<br clear="left">
<hr>
<p>

<h2>Application 3: FPGA <=> Processor Image Transfer</h2>
<p>
Application sections one and two, above, focus on blob detection and structured light algorithms with the goal of off loading as much of the processing task from the on board embedded processor to external operators.  Even with that goal realized, there are however sequentially based processing algorithms that are well suited to offline application with-in an embedded processor.  In this section the project review will focus in on the mechanics of retrieving image data from the high speed SRAM attached to the FPGA and moving that data into the processors memory system.
<p>
<a name="cap_code"></a>
<table align="left"background="grid.gif"width="80"cellpadding="8">
<tr><td><center>
<a href="cap_code1.htm"><img src="code.gif"><p>Updated Verilog<p>dj_top.v:</a>
<tr><td><center>
<a href="cap_code2.htm"><img src="code.gif"><p>Updated Verilog<p>ram_scheduler.v:</a>
<tr><td><center>
<a href="retrieve.htm"><img src="code.gif"><p>'C' Function<p>Retrieve_Image:</a>
</table>
Any number of processes are available for this work, including not moving the data at all, and, rather, applying sequential algorithmic processing from the embedded processor system to the data through an FPGA "tunnel".  Of course there are trade off's to be considered like processor bus speed and path delays due to the multitude of other accesses that must continue to the same SRAM real-time.  (Continued image capture of the next frame, data extraction for display purposes, 2x additional vision systems running concurrently and their associated processing members.)  
<p>
Given the system level constraints, data path constraints and timing issues a decision point drove to the choice of sequential data transfer of the captured image through the processors data bus interface for storage in external memory locally mapped.  Two replacement files, updated versions of <code>dj_top.v</code> and <code>ram_scheduler.v</code>, are provided, left.  These files implement sequential image retrieval from the FPGA to the Processor over the standard processor 8-bit data bus.  The primary changes / additions from the two previously presented versions of these files are highlighted in <b><font color="blue">bold / blue</font></b>.
<p>
From an architecture stand point, when the processor is writing to the image data SRAM for image overlay, the interface is that of a wide parallel memory mapped space allowing instantaneous random access to any pixel in the array.  This functionality is made possible due to the wide fifo's that can store both the address and data to be written for later writing to the SRAM when alternate accesses are being made.  This is taken care of by the ram_scheduler.v module. 
<p>
<table align="right"><tr><td><center><h3>Figure 8.9</h3><p><img src="floor4_plan.gif">
</table>
In the case of reading data in a random manner, there is no way to know the address for pre-fetch operations unless data is accessed sequentially and run, in a similar manner to storage, through a fetching fifo.  This allows data to be present whenever the processor requests it and still allows the data accesses to the shared SRAM resource when time slotting by ram_scheduler.v allows.  Further, this allows the same sequential bus interface methodologies already implemented in other section and not available on lower end processors such as PICs, AVRs or 8051 thus making it more accessible to a larger body of armature roboticists.
<p>
Layered into the data flow diagram, image 8.9, right, the two new blocks required to implement FPGA to processor image data transfer are highlighted in red.  The code changes to implement the functionality is confined to two modules (dj_top.v, video_capture.v).  The listing below outlines the theory of operation:
<p>
<ul>
<li>Variable <code>retrieve_addr[17:0]</code> stores the address of the next data point to retrieve.
<li>The processor bus provides a means to reset this address corresponding to a write to address <code>PROC_RESET_READ_PROCESS</code>.
<li>Retrieve fifo full flag is used to trigger an additional request to the ram scheduler for more data.
<li>ram scheduler retrieves data based on the address provided by the retrieve engine, off line, and drives it into the fifo.
<li>With-in the ram scheduler module, a new case has been added to the dispatch state and two new states have been added to the state machine to handle the case of data requests to the external SRAM to retrieve raw video samples.
<li>The second state added to the ram scheduler triggers a write into the fifo of the retrieve engine through assertion of the bit wide variable <code>retrieve_data_ready_strb</code>.
<li>Retrieve engine output is made available to the existing processor bus read mux in two halves.
<li>Two addresses are provided due to the 15 bit nature of the data and the 8-bit processor bus.
<li>Successful reads from the processor bus to the second address will advance the fifo's output index, changing the state of it's full flag and restarting the process.<b>***</b>
</ul>
<b>*** Note:</b> <i>some buffering is added to detect the de-assertion of the processors /CS (read = raising edge), to hold of the incrementing of the fifo such that the data driven onto the bus does not change until the cycle is over.</i>
<p>
<br clear="left">
<br clear="right">
<p>
<p>
<table align="left"><tr><td><center>
<h3>Figure 8.10:</h3>
<p>
<img src="mage_000.gif"width="320"height="240"alt="captured scene">
</table>
The image, left (8.10) is yet another example of direct digital capture through the system and transfer directly to the PC.  In this image the back wall is some 30' back and the vertical separation between the camera and laser field generator is less than 2 inches.
<p>

<table align="right"width="150"><tr><td><center><a href=""><img src="lg5.gif"><p><b>I2Term.zip</b></a></table>
The work to this point centers around FPGA firmware and minimal support by the embedded processor system.  At some point in the development it would be helpful to move digital images directly into a PC for further research / work.  To the rescue, <u>Alan Erickson</u> with this gem of a piece of software he developed to aid in exactly this situation.  <a href=""><b>I2Term</b></a> is that program.  This software combines a terminal program and digital image capture / display functionality with auto image save features.  This software was used to capture the images in figure 7.3 ~ 7.6 and 8.5, above.  The software will begin capturing data after receipt of the character 0x07 in the serial stream.  Data to be captured is in R,G,B color tripples, 8 bits each.  That is all there is to it, just send enough data to fill the matrix defined by the image size in the image settings dialog box.  The software is available for use by armature roboticists by downloading the zip file to the right.
<br clear="left">
<p><hr>
<p>
<a name="i2c"></a>
<h2>Other Verilog Functions:</h2>
<p>
<table align="right"background="grid.gif"width="80"cellpadding="8">
<tr><td><center>
<a name="djm1"></a>
<a href="djm1.htm"><img src="code.gif"><p>FPGA_I2C Instantiation:</a>
<tr><td><center>
<a href="djm2.htm"><img src="code.gif"><p>'C' I2C Calling Functions:</a>
<tr><td><center>
<a href="i2c.pdf"><img src="adobe.gif"><p>I2C Verilog Source:</a>
</table>

In this implementation the I2C bus (multiple 3x) connection runs between the VIP and the secondary FPGA (off board).  This is where some of the IP (intellectual property) available through <a href="http://www.opencores.org">www.opencores.org</a> can be leveraged.  Since the logic description in Verilog is abstracted as a high level language, it can easily be re-targeted to Xilinx (or any other) fabric and used as a building block within any given HDL design.
<p>
<ul>
<li>The first of three files, right, shows how to instantiate (set up and call) the I2C IP function.
<li>The second file, right, is the u-Processor side 'C' language calling functions to interface to the I2C engine.
<li>The final file is the actual I2C code from the opencores web-site with license statements intact.  It was originally 4 separate source files that have been pasted together into a single file.  It was written by: <u>Richard Herveille</u> and is redistributed here with his permission.
</ul>
<p>
<table align="left">
<tr><td><center><a href="i2c_debug.htm"><img src="zi2c_debug.jpg"width="150"height="111"><p>Click to Enlarge</a>
</table></a>
The photo, left, is one of those shot during the debug process of the I2C IP engine.  Depicted in this photo is a screen shot of one of the author's digital logic analyzers monitoring the SDA and SCL lines.  The analyzer used to capture these signals is an HP1662A by Hewlett Packard.  This piece of digital trouble-shooting equipment plays an invaluable role in debugging the system without the need for extensive simulation.  The tradeoff of in logic analyzer vs simulation time does come with the penalty of having to sacrifice some IO pins.
<p>
An interesting undocumented note with respect to the SAA7111A Video Input Processor data sheet is that the ACK at the end of a read operation is actually a NACK. Debugging with the logic analyzer really helps to find little missing bits like this. Note the NACK is the wide high pulse at the end of the top trace in the picture.  The I2C specification clearly defines this mode of operation for standard bus transactions