cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Visitor
Visitor
445 Views
Registered: ‎01-19-2020

Help with Video Processing Pipeline

Hi, I am looking for help in developing a simple video processing pipeline in Vivado 2019.2. I am a student and my only experience with Vivado has been from a high-level synthesis class, where I used Vivado to integrate custom HLS IPs. This was not very deep, we mostly relied on Vivado's automation. So it is okay to assume I am missing basics, and please feel free to link to other resources where I can learn more. 

My Custom HLS IP in this case is currently a simple blurring operation via 2D convolution. But this is meant as a placeholder, the blurring operation isn't really necessary for this application.

My final application is going to be something like: 

CMOS image sensor with MIPI interface -> Sensor Demosaic -> Custom HLS IP -> object memory

and I am targeting the XZCU15EG

However, our board has not finished fabrication yet, and so I would like to have the following ready in the meantime:

TPG generating Bayer data -> Sensor Demosaic -> Custom HLS IP -> object memory

and I have a Pynq Z-2 with a Zynq 7020 I can test with right now.

The Custom HLS IP currently implements AXI stream input to AXI stream output; I was able to follow the example HLS codes available on GitHub to create something that passes simulation and cosimulation.

But I have several questions regarding the Vivado block diagram, and I'll split them into two parts. 

Test pipeline (for z7020):

1. How do I generate Bayer data using the Test Pattern Generator Version 8? It seems from documentation that this feature was removed starting from the Test Pattern Generator Version 7.
2. How do I draw the block diagram in Vivado for this system? Specifically, how do I handle the AXI streaming between blocks? For example, if the Zynq processing system handles interrupts, is it enough to Concat the interrupt signals from Sensor Demosaic and my Custom HLS IP? What other things do I have to worry about? Will I need to implement the AXI Interconnect IP? 
3. How do I write to object memory? Will I have to use the Video Frame Buffer Read or Write blocks? If there is an alternative, which is easier, and which is more modular?
4. Is it okay for the Demosaic and Custom HLS IP blocks to share clock and reset signals? They're currently set to operate on images of the same size.

Application pipeline (for XZCU15EG):

1. How do I create an AXI stream from a MIPI interface?

 

Note: I have looked at the Xilinx Video Series (https://forums.xilinx.com/t5/Video-and-Audio/Xilinx-Video-Series/td-p/849583) but there's nothing about MIPI there, and the TPG tutorial doesn't mention anything about generating Bayer data. Again, if there are other posts/resources that seem like they are relevant (and especially if they are meant for beginners), please share them. 

Best, 
Roumen

Tags (2)
0 Kudos
5 Replies
Highlighted
Moderator
Moderator
314 Views
Registered: ‎11-09-2015

Hi @roumen.guha 


Test pipeline (for z7020):

1. How do I generate Bayer data using the Test Pattern Generator Version 8? It seems from documentation that this feature was removed starting from the Test Pattern Generator Version 7.

Florent - You cannot. You can potentially write a HLS IP which would create a bayer layer from the output of the TPG. It should not be too complicated.
Another solution which is what I have done in my Video Series 11 is to use a bayer image as input to the simulation.


2. How do I draw the block diagram in Vivado for this system? Specifically, how do I handle the AXI streaming between blocks? For example, if the Zynq processing system handles interrupts, is it enough to Concat the interrupt signals from Sensor Demosaic and my Custom HLS IP? What other things do I have to worry about? Will I need to implement the AXI Interconnect IP? 

Florent - This is basic vivado design. If you really have been through my video series you should understand. I am not touching about the interrupts this is the only thing missing. But yes, you can just concat the interrupts and feed them to the processor

3. How do I write to object memory? Will I have to use the Video Frame Buffer Read or Write blocks? If there is an alternative, which is easier, and which is more modular?

Florent - What do you mean by object memory?

Yes if you need to move from AXI4-Stream to Memory, then yes you need ot use a DMA. For video data the easiest is to use the AXI VDMA or Video Frame buffer IPs

4. Is it okay for the Demosaic and Custom HLS IP blocks to share clock and reset signals? They're currently set to operate on images of the same size.

Florent - Yes this is ok as long as you ensure the clock is fast enough.

Application pipeline (for XZCU15EG):

1. How do I create an AXI stream from a MIPI interface?

 

Florent - Using the MIPI CSI2-RX Subsystem IP 

Note: I have looked at the Xilinx Video Series (https://forums.xilinx.com/t5/Video-and-Audio/Xilinx-Video-Series/td-p/849583) but there's nothing about MIPI there, and the TPG tutorial doesn't mention anything about generating Bayer data. Again, if there are other posts/resources that seem like they are relevant (and especially if they are meant for beginners), please share them. 


Yes I wanted to write some articles around MIPI but never had time. For the Bayer data, as mentioned you need to look at the Video Series 11.


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Highlighted
Visitor
Visitor
299 Views
Registered: ‎01-19-2020

Hi @florentw, thanks for your reply! Glad to have caught you. 

Please give me 2 days to reply. I'm working on this between classes, and I'm about to go to bed for today.

Regarding my earlier question "How do I draw the block diagram in Vivado for this system?" I worded this poorly. I was really asking for general tips, possible things I should watch out for, possible design trade-off decisions that I need to make. Things generally not covered in a tutorial series, and things an absolute beginner may miss. Since asking that question, I've been going through @bluetiger9's posts and materials.

0 Kudos
Highlighted
Moderator
Moderator
232 Views
Registered: ‎11-21-2018

Hi @roumen.guha 

Do you have an update on this? Is your issue now resolved? If so, could you please mark the solution as an 'accepted solution'? If not, could you please update the community on your current status? 

Aoife
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Highlighted
Visitor
Visitor
168 Views
Registered: ‎01-19-2020

I'm currently still facing some issues, though they're not exactly the same ones as before. Let me know if I need to make a new post for this.

I generated an example MIPI-to-HDMI design and added my custom HLS block to the psng0 subblock, between the Gamma LUT and the VPSS (used for colorspace conversion to YUV for HDMI output). I then used Run Block Automation and attempted to validate the design, but got the following error: 

[BD 41-1811] The interconnect </peripherals_ss/microblaze_0_axi_periph> is missing a valid slave interface connection

And adding a single slave interface to the AXI Interconnect block in peropherals_ss only leads to further errors when I use Run Block Automation:

[BD 41-2168] Errors found in procedure apply_rule:

but doesn't specify what the errors are, and what I need to do to fix it. Any pointers?

0 Kudos
Highlighted
Moderator
Moderator
105 Views
Registered: ‎11-21-2018

Hi @roumen.guha 

Can you please create a new post for this?I think it will help you get answers from the community and help users find your post if they are having similar problems. 

Aoife
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos