UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor mattenii
Visitor
1,933 Views
Registered: ‎10-14-2017

PYNQ video stream processing

Jump to solution

I want to do video processing using the PYNQ-Z1 board. The default base overlay contains HDMI in- and output IP by Digilent, which converts the TMDS signals into an 'RGB stream'. According to the official PYNQ documentation it is possible to insert a video processing block in this video stream, but gives no further information on how to actually do this. This is however what I want to accomplish somehow.

 

I have been unable to find any exact specification of this RGB stream, nor have my own experiments resulted in anything useful. Even the simple passthrough as implemented below gives nothing but a black screen.

 

void stream_proc (hls::stream<uint32_t> &src, hls::stream<uint32_t> &dst)
{
#pragma HLS interface ap_ctrl_none port=return
#pragma HLS interface axis port=&src
#pragma HLS interface axis port=&dst

    uint32_t p;

    src >> p;
    dst << p;
}

The resulting src port is connected to the RGB input stream and the dst port to the VDMA input stream. I'm not sure though where exactly to connect the clk and reset.

 

Summarizing:

- Is the passthrough itself correct?

- If so, how do I connect it?

- Do any other signals have to be modified?

- If processing takes multiply clock cycles, does anything else have to be delayed as well?

- As a bonus: what would be the best way of introducing control signals; e.g AXI or GPIO?

 

Any ideas and suggestions are appreciated. I have some decent experience with VHDL and FPGAs in general, but am a newbie with both HLS and Zynq/PYNQ.

Tags (2)
0 Kudos
1 Solution

Accepted Solutions
Scholar u4223374
Scholar
2,579 Views
Registered: ‎04-26-2015

Re: PYNQ video stream processing

Jump to solution

(1) I suspect not. Normally the video stream will have at least a TUSER and TLAST line, in addition to the TDATA one that you've already got. To use this you need:

 

 

#include <ap_axi_sdata.h>
typedef ap_axiu<16,1,1,1> pixel_data;
typedef hls::stream<pixel_data> pixel_stream;

void stream_proc (pixel_stream &src, pixel_stream &dst)
{
#pragma HLS interface ap_ctrl_none port=return
#pragma HLS interface axis port=src
#pragma HLS interface axis port=dst
#pragma HLS PIPELINE II=1
    pixel_data p;

    src >> p;
    dst << p;
}

(I am guessing that it's a 16-bit TDATA stream, based on what I've seen on other boards. You can check the true width in the block design).

 

 

(2) Can you post a screenshot of the block design? Normally you'll have a HDMI input block connected to an S2MM DMA (writes data to RAM) and a HDMI output block connected to an MM2S VDMA (reads data from RAM). The use of a RAM-based framebuffer means that the design can handle a mismatch between the input framerate and output framerate. Your block can go between the HDMI input and S2MM VDMA, or between the HDMI output and MM2S VDMA, or it can be an entirely separate block that modifies the framebuffer directly. Your HLS block should share the same clock and reset signals as the blocks on either side of it.

 

(3) I don't think so, unless you're doing something like video scaling where obviously the output timing information will change.

 

(4) This can be tricky, because the input and output may be relying on 1 pixel per clock. In this case, having a block that modifies the framebuffers in RAM would be ideal.

 

(5) AXI, assuming those control signals come from the Zynq CPU.

4 Replies
Scholar u4223374
Scholar
2,580 Views
Registered: ‎04-26-2015

Re: PYNQ video stream processing

Jump to solution

(1) I suspect not. Normally the video stream will have at least a TUSER and TLAST line, in addition to the TDATA one that you've already got. To use this you need:

 

 

#include <ap_axi_sdata.h>
typedef ap_axiu<16,1,1,1> pixel_data;
typedef hls::stream<pixel_data> pixel_stream;

void stream_proc (pixel_stream &src, pixel_stream &dst)
{
#pragma HLS interface ap_ctrl_none port=return
#pragma HLS interface axis port=src
#pragma HLS interface axis port=dst
#pragma HLS PIPELINE II=1
    pixel_data p;

    src >> p;
    dst << p;
}

(I am guessing that it's a 16-bit TDATA stream, based on what I've seen on other boards. You can check the true width in the block design).

 

 

(2) Can you post a screenshot of the block design? Normally you'll have a HDMI input block connected to an S2MM DMA (writes data to RAM) and a HDMI output block connected to an MM2S VDMA (reads data from RAM). The use of a RAM-based framebuffer means that the design can handle a mismatch between the input framerate and output framerate. Your block can go between the HDMI input and S2MM VDMA, or between the HDMI output and MM2S VDMA, or it can be an entirely separate block that modifies the framebuffer directly. Your HLS block should share the same clock and reset signals as the blocks on either side of it.

 

(3) I don't think so, unless you're doing something like video scaling where obviously the output timing information will change.

 

(4) This can be tricky, because the input and output may be relying on 1 pixel per clock. In this case, having a block that modifies the framebuffers in RAM would be ideal.

 

(5) AXI, assuming those control signals come from the Zynq CPU.

Visitor mattenii
Visitor
1,839 Views
Registered: ‎10-14-2017

Re: PYNQ video stream processing

Jump to solution

Thanks for the answer u4223374! Using your example code I got the stream processing working. In my case the TDATA stream is 32 bits wide though, with each pixel in RGBA format. I also attached the clock and reset to the corresponding pins on other blocks and added maskable invert processing.

 

Regarding your answer (4), I meant multiply cycles in terms of latency, not interval. My current block has no latency yet, but I can imagine some frame clock has to align with the correct pixels. I will try some other processing and update when I got some results.

 

For future reference I have attached a screenshot of the block design and the code I used.

 

#include <stdint.h>
#include <hls_stream.h>
#include <ap_axi_sdata.h>

typedef ap_axiu<32,1,1,1> pixel_data;
typedef hls::stream<pixel_data> pixel_stream;

void stream(pixel_stream &src, pixel_stream &dst, uint32_t mask)
{
#pragma HLS INTERFACE ap_ctrl_none port=return
#pragma HLS INTERFACE axis port=&src
#pragma HLS INTERFACE axis port=&dst
#pragma HLS INTERFACE s_axilite port=mask
#pragma HLS PIPELINE II=1

	pixel_data p;
	src >> p;
	p.data = p.data^mask;
	dst << p;
}

 

PYNQ_streambd.png
0 Kudos
Scholar u4223374
Scholar
1,834 Views
Registered: ‎04-26-2015

Re: PYNQ video stream processing

Jump to solution

@mattenii Multiple cycles of latency should not be an issue. The other blocks all have some inherent latency anyway, and if it's all going in to RAM (as it appears to be) then latency really won't matter much. You just need to make sure that the TLAST and TUSER signals in the AXI Stream line up correctly (TUSER set on the first pixel of each frame, TLAST set on the last pixel of each line) - that's what the VDMA uses to correctly align the frame in RAM.

 

You might want to drop an AXI Stream FIFO between the HDMI input and the HLS block. HLS often doesn't process data particularly smoothly (eg. it'll do 1 pixel/clock for one line, then wait a few cycles before starting the next line) and this might cause issues for the HDMI block. A Stream FIFO with a length of at least one line should neatly compensate, and it'll cost very little (probably one block RAM).

0 Kudos
Visitor mattenii
Visitor
1,817 Views
Registered: ‎10-14-2017

Re: PYNQ video stream processing

Jump to solution

@u4223374 Thank you for the additional information. As you said latency is no problem as long as you keep track of the TLAST and TUSER signals. I haven't yet needed the FIFO, but I'll keep it in mind just in case. Thanks again!

0 Kudos