UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Adventurer
Adventurer
6,848 Views
Registered: ‎10-24-2016

Image sensor fpga configuration

Jump to solution

Good morning to all. 

 

I am new to this forum and with Xilinx devices, and i want to create a system to test image sensors. 

 

I have been looking for information and asking for this to Xilinx distributors, but I have to confirm what is the best way to carry out the design.

 

I need to create a device which has to be able to test any image sensor ( or most of them in the market), so the design must be prepared to work with 8bit, 10bit, 12,14 and 16 bit sensors, and ready to work with up to 64 LVDS pairs. Main features that the design must have are:

 

· Ready to work with up to 64 LVDS pairs.

· Ready to work with 8,10,12,14,16-bit sensors.

· 1 SPI bus to communicate with the sensors and modify its internal registers

· 1 UDP link to send raw data from the sensor to a PC ( i dont need to process image inside fpga, only get all pixel information     from the sensor and send it in raw format to the PC. Data is processed in the PC)

· External memory to load data before send them to the PC through UDP link ( i suppose...)

 

At this moment i am quite lost about the xilinx resources i have to use ( there are hundreds of documents with hundreds of sheets. Impossible to read all of them). I have seen information about the usage of Microblaze throgh EDK tool to create a soft processor and all the peripherals connected to it ( SPI, UDP link, DMA,...) but i have no clear which resources to use, becuase it exists AXI4 lite, AXI DMA, quad spi,... and too much resources like these ones ( as i told you this is the first time I work with XIlinx plattform).

 

So my question is what are the steps you think i have to follow? Is to make a design without using Microblaze possible? 

 

As far as I know, to work with LVDS it is required to use Artix7 because it has HR ports, is not? Also, i have read that i have to use ISERDES and OSERDES for this task and IODELAY to adjust bit skew.

 

Summing up, it could be good if you pass me, at least , a block diagram with all the elements i need to get my design working. Once i have this information, i can look for information of all the blocks involved in the design to make it work.

 

Thanks by advance for all your support. Any help is really appreciated. 

 

Regards.

0 Kudos
1 Solution

Accepted Solutions
Highlighted
Adventurer
Adventurer
12,764 Views
Registered: ‎10-24-2016

Re: Image sensor fpga configuration

Jump to solution

Again thanks for your answer. 

 

So you say that one solution could be to work with DDR flip flops. Flip-Flop inputs connected to outputs from the sensor and flip flop outputs connected to the DMA core to pass information to the DDR memory, and then one VHDL module could recover this information and create an UDP message (not at the same time of readout) to send it to the PC, is this correct? (as i told you I dont need to process images, only to send raw data to one PC). In this way Microblaze could not be required. 

 

The problem I see is that time required to capture and send data could not be enough to transmit frames per second required.  How many pixels did your sensor have?? How many fps were you able to capture with this system?? Do you think is possible to capture data and send directly to UDP core without working with external memory?? 

 

Regarding to UDP link to link communication, what data rate is possible with the system you describe?? Anyway, Is there specific UDP Core in ISE/Vivado??

 

Regarding to development kit, i am afraid about signal integrity. Because I dont know if these boards are prepared to work with LVDS lines at high speed ( 100 Ohms resistors to LVDS bus, controlled impedance on signal traces, lenght of PCB traces, ....). In addition, image sensor will be 50cms far from the FPGA PCB, so this can also degrade signal quality. Anyway, Could you recommend me one of them?? 

 

And regarding calibration. I have seen (not in deep) that bit deskew is possible in Artix7 family. But, is byte and word deskew possible? How do you do dinamic clock phase calibration??

 

Thanks for your answers.

 

Regards.

 

 

0 Kudos
10 Replies
Instructor
Instructor
6,841 Views
Registered: ‎08-14-2007

Re: Image sensor fpga configuration

Jump to solution

If I were designing this, I would not use MicroBlaze, however you might find it easier at least for the UDP link portion of the design if you have one.  I designed a camera that used UDP for data output without a Microblaze, but it used a static IP address and expected a point-to-point link with no "discovery" process.  That camera used a sensor with 84 LVDS pairs and a 64-bit non-ECC DDR3 SODIMM for image storage.  It did not store images and read out at the same time.  The FPGA was the largest Artix-7, XC7A200T-2FFG1156C.

 

You don't need HR banks for LVDS.  HP banks support LVDS as well.  However it may be that some of your lower speed sensors need LVCMOS at greater than 1.8V, and you would need HR banks for that.  Artix-7 is not the only series with HR banks.  Kintex-7 and some Virtex-7 parts have HR banks as well as HP banks.

 

It is important to design the external memory with enough bandwidth to handle the highest-speed sensor you will test.  Most 7-series parts can fairly easily do DDR3-800 (400 MHz clock, 800 Mbps per bit of DQS to the memory).  If you don't need to write and read the memory at the same time, i.e. the memory acts like a DVR that either records video or plays it back but not both at once, then the bandwidth of the memory only needs to be slightly more than the maximum sensor data bandwidth to handle memory refresh and inefficiencies during memory row switching.  If you need to read the memory while it's being written, the bandwidth needs to be a good deal more than twice the sensor bandwidth to handle the total data rate as well as inefficiencies that come with switching the direction between read and write.

 

Whether or not you need to use the IOSERDES depends on the bit rate of the sensor.  Typically sensors that output LVDS do require deserialization in the IOSERDES or at least the use of DDR input flip-flops.  The camera I mentioned used DDR inputs and it also required dynamic clock phase calibration to sample data at up to 500 Mbps.  For calibration it helps if the sensor is able to output a test pattern.

 

How were you planning to connect the FPGA to the myriad of sensors?  Will there be some sort of adapter card?  You might think of prototyping this with a standard development kit that has an FMC socket.

 

 

-- Gabor
0 Kudos
Adventurer
Adventurer
6,834 Views
Registered: ‎10-24-2016

Re: Image sensor fpga configuration

Jump to solution

Thanks for your answer gszakacs.

 

I have to go out of the office. I will answer you tomorrow. 

 

Thanks again for your support. 

0 Kudos
Highlighted
Adventurer
Adventurer
12,765 Views
Registered: ‎10-24-2016

Re: Image sensor fpga configuration

Jump to solution

Again thanks for your answer. 

 

So you say that one solution could be to work with DDR flip flops. Flip-Flop inputs connected to outputs from the sensor and flip flop outputs connected to the DMA core to pass information to the DDR memory, and then one VHDL module could recover this information and create an UDP message (not at the same time of readout) to send it to the PC, is this correct? (as i told you I dont need to process images, only to send raw data to one PC). In this way Microblaze could not be required. 

 

The problem I see is that time required to capture and send data could not be enough to transmit frames per second required.  How many pixels did your sensor have?? How many fps were you able to capture with this system?? Do you think is possible to capture data and send directly to UDP core without working with external memory?? 

 

Regarding to UDP link to link communication, what data rate is possible with the system you describe?? Anyway, Is there specific UDP Core in ISE/Vivado??

 

Regarding to development kit, i am afraid about signal integrity. Because I dont know if these boards are prepared to work with LVDS lines at high speed ( 100 Ohms resistors to LVDS bus, controlled impedance on signal traces, lenght of PCB traces, ....). In addition, image sensor will be 50cms far from the FPGA PCB, so this can also degrade signal quality. Anyway, Could you recommend me one of them?? 

 

And regarding calibration. I have seen (not in deep) that bit deskew is possible in Artix7 family. But, is byte and word deskew possible? How do you do dinamic clock phase calibration??

 

Thanks for your answers.

 

Regards.

 

 

0 Kudos
Adventurer
Adventurer
6,788 Views
Registered: ‎10-24-2016

Re: Image sensor fpga configuration

Jump to solution

And how many DDR flip flops are available in Artix7 family?? Is there any example of how to use them??

0 Kudos
Instructor
Instructor
6,767 Views
Registered: ‎08-14-2007

Re: Image sensor fpga configuration

Jump to solution

So many questions... :-)

 

The camera I described had 17 megapixels of 14 bits each.  It could run 120 frames per second at full resolution.  This results in something over 3.5 gigabytes per second.  There was no way to run this directly to UDP, at least not with a single channel.  So the camera operated by storing frames to DDR3 memory and then reading it out to the PC much more slowly than the images came in.  My UDP ran over a single 1G Ethernet link.  I'm not sure if there is a Xilinx core for UDP - I wrote my own for this camera.  I believe there is an opencores.org UDP core, but as I recall it was more complex than I needed for this simple point-to-point static IP case.  If you go with MicroBlaze you can use lwip (lightweight IP) and then the UDP part should be simple enough.  However this would probably mean additional external memory for the MicroBlaze in order to hold the program required.

 

You didn't mention the data rate of your worst-case imager, but it's likely that any imager with 24 pairs of LVDS coming out will have more data bandwidth than any single Ethernet connection.  That means it's unlikely you'll be able to get away without external memory.  If you decided to use a PCI Express plug-in form factor board, you might have the necessary bandwidth to avoid external memory, but remember that even then the maximum data rate of PCI Express is not guaranteed to be always available, so you'd still need some elastic buffering in the system to ride through short "outages" when the bus or the targeted PC host memory is busy.

 

Development kits that have FMC connectors are typically wired to support high speed LDVS signalling.  And having the imager a long way from the electrically noisy FPGA is not really a bad thing.  You might have more issues supporting the smaller CMOS imagers that use higher interface voltages.  They may require level shifters to bring the interface down to 2.5V or 1.8V depending on the supporting card.

 

Artix parts have DDR flops for every user I/O pin.  They also have IOSERDES for each pin.  That gives you a lot of flexibility.

 

One other thought is that you could use a much smaller FPGA if it only needs to support one imager type at a time (using a separate configuration file).  If the imager and configuration memory were both on the same adapter module, you could swap imagers and FPGA configurations together.  Also you may need to change the Vcco voltage for banks connecting to the imager, and that could be jumpered through the adapter as well.

-- Gabor
0 Kudos
Instructor
Instructor
6,765 Views
Registered: ‎08-14-2007

Re: Image sensor fpga configuration

Jump to solution

I missed the question about clock calibration...

 

7-series parts have multimode clock managers (MMCM) which have the ability to do fine phase shifting dynamically.  There is a simple interface that allows you to increment or decrement the clock phase by one step at a time.  Each step is 1/56 of the VCO clock period, which is usually divided down to generate the output clock so it would give you a multiple of 56 steps per period of your output clock.  Continuing to increment results in wrapping back to phase 0 so you don't really need to decrement (I didn't in my calibration module).  The sensor I used had the ability to put out alternating all 1's and all 0's and I used that to calibrate the clock phase.  Since I didn't use the IOSERDES, I didn't need to align data words.  However the IOSEDES has sonething called bitslip, which allows you to align bits within words.

-- Gabor
0 Kudos
Adventurer
Adventurer
6,738 Views
Registered: ‎10-24-2016

Re: Image sensor fpga configuration

Jump to solution

Wow!!! Great answer. 

 

Thank you very much Gabor.

 

The worst case, up to this moment, has been an image sensor with 12 Mpixels, working with 64 LVDS pairs at 300fps. So at least, i would like to have a board ready to accomplish a similar case to this. 

 

I think I have enough information ( at least to start). I will look for information about UDP link to link transmission to try to do it in the way you say (it seems to be faster).

 

Let me know if I can ask for help to you in case of doubts during design, because it seems that you know a lot about this topic.

 

Again Thanks!!

 

 

 

 

0 Kudos
Adventurer
Adventurer
6,734 Views
Registered: ‎10-24-2016

Re: Image sensor fpga configuration

Jump to solution

Sorry... the last question  :)

 

What is the difference between DMA and VDMA ?? Is the last one the one recommended for video data?? Which of them do you think I have to use??

 

Regards.

 

 

0 Kudos
Instructor
Instructor
6,712 Views
Registered: ‎08-14-2007

Re: Image sensor fpga configuration

Jump to solution

Looking at your worst case, I come up with a frame average data rate of 3.6G pixels per second.  For an 8-bit pixel (you didn't say) that would be 3.6 GB/s, which means you want to have probably 4GB/s or more of available peak bandwidth to your external memory.  If you use board-mounted parts you'd want at least 40-bit wide memory at 800 Mbps.  An SODIMM is typically 64 bits wide and would give you comfortable headroom for any inefficiency in the DMA interface.

 

I'm not too familiar with the Xilinx DMA or VDMA cores.  I typically roll my own when working with high-speed video.  You might want to start a new thread for that question in the "DSP and Video" forum.  My main issue is that I've been doing this sort of thing for longer than the video cores have been around, so it's just easier for me to reuse my own designs.

-- Gabor
0 Kudos
Adventurer
Adventurer
4,256 Views
Registered: ‎10-24-2016

Re: Image sensor fpga configuration

Jump to solution

Gabor.

 

Thanks for all your answers. I hope to keep alive after design it  :)

 

Kind regards.

0 Kudos