HLS OpenCV: Certain operator combinations don't work

[signato]

Hello,

I have the problem that certain combinations of OpenCV operators (functions) don't work. These operators do work when used in other contexts.

"Don't work" means I get a colored-noise output. Debugging (ILA) shows that the HLS core processes some input data (as deep as the debugging memory goes), however, no output.

The streams are VGA (640x480) RGB image sequences at 60 Hz, the design is similar to that in XAPP1167.

A reduced example (all functions in hls namespace):

void hls_reflex (AXI_STREAM& video_in, AXI_STREAM& video_out, int shift, unsigned reflex_thresh)
{
 #pragma HLS INTERFACE axis port=video_in bundle=INPUT_STREAM
 #pragma HLS INTERFACE axis port=video_out bundle=OUTPUT_STREAM

 #pragma HLS INTERFACE s_axilite port=shift
 #pragma HLS INTERFACE s_axilite port=reflex_thresh

 #pragma HLS INTERFACE ap_stable port=shift
 #pragma HLS INTERFACE ap_stable port=reflex_thresh

    IM_8UC3 im3_in   (IROWS, ICOLS);
    IM_8UC3 im3_out  (IROWS, ICOLS);
    IM_8UC1 im_blue0 (IROWS, ICOLS);
    IM_8UC1 im_blue1 (IROWS, ICOLS);
    IM_8UC1 im_blue  (IROWS, ICOLS);
    IM_8UC1 im_green (IROWS, ICOLS);
    IM_8UC1 im_red   (IROWS, ICOLS);
    IM_8UC1 dst_spots (IROWS, ICOLS);
    IM_16SC1 grad_x (IROWS, ICOLS);
    IM_16SC1 grad_y (IROWS, ICOLS);
    IM_32SC1 grad_xy (IROWS, ICOLS);
    IM_8UC1 grad_thresh (IROWS, ICOLS);

 #pragma HLS dataflow
    AXIvideo2Mat (video_in, im3_in);
    Split (im3_in, im_blue, im_green, im_red);
        Duplicate (im_blue, im_blue0, im_blue1);

    Threshold (im_blue0, dst_spots, reflex_thresh, 255, HLS_THRESH_BINARY);

    Sobel<1, 0, 3> (dst_spots, grad_x);
    Sobel<0, 1, 3> (im_blue1, grad_y);

    fp_t fshift = shift * fp_t (1E-3);
    AddWeighted (grad_x, fshift, grad_y, fshift, fp_t (0), grad_xy);
    Scale (grad_xy, grad_thresh, fshift);

    // works
    Mat2AXIvideo (grad_thresh, video_out);

    // doesn't work
//    Merge (grad_thresh, im_green, im_red, im3_out);
//    Mat2AXIvideo (im3_out, video_out);
}

This one works, however, when I add the Merge operator (last-but-one line), I get the described noisy output.

Any help appreciated

Signato

[signato]

Hi all,

meanwhile I reduced the problem to the essence (I hope):

A three-channel (RGB) stream is split into single channels, one of them is processed somehow, then the result and the other two are merged again.

void hls_reflex (AXI_STREAM& video_in, AXI_STREAM& video_out, int shift, unsigned reflex_thresh)
{
 #pragma HLS INTERFACE axis port=video_in bundle=INPUT_STREAM
 #pragma HLS INTERFACE axis port=video_out bundle=OUTPUT_STREAM

 #pragma HLS INTERFACE s_axilite port=shift
 #pragma HLS INTERFACE s_axilite port=reflex_thresh

	IM_8UC3 im3_in   (IROWS, ICOLS);
	IM_8UC3 im3_out  (IROWS, ICOLS);
	IM_8UC1 im_blue  (IROWS, ICOLS);
	IM_8UC1 im_green (IROWS, ICOLS);
	IM_8UC1 im_red   (IROWS, ICOLS);
	IM_8UC1 grad_thresh (IROWS, ICOLS);

 #pragma HLS dataflow
	AXIvideo2Mat (video_in, im3_in);
	Split (im3_in, im_blue, im_green, im_red);

	Dilate (im_blue, grad_thresh);

	// works
//	Mat2AXIvideo (grad_thresh, video_out); /*

	// doesn't work
	Merge (grad_thresh, im_green, im_red, im3_out);
	Mat2AXIvideo (im3_out, video_out); // */
}

So, there seems to be some problem with buffering of the other two (here im_green, im_red), or synchronization with the modified stream, respectively.

Or am I thinking wrongly?

Any help appreciated

Signato

[naz_rb]

@signato Not an answer. Just thinking, it does not make sense for me how it works at all:

Mat2AXIvideo (grad_thresh, video_out);

Your video_out is supposed to be 3 channel, and you provide grad_thresh, which is 1 channel. Does Mat2Axivideo() expands it to all three channels?

[u4223374 · Accepted solution]

I think the problem is that your im_blue/grad_thresh is delayed compared to the other streams. If you use the STREAM pragma and specify a FIFO length for the other two (maybe 10 lines to start with, reduce it once the system works) then that should solve the issue.

[signato]

@naz_rb: You are right - I chose this code only for demonstration purposes. However, Mat2Axivideo () does work with a one-channel input - it produces RGB where R and G are '1' (255) always, and the input channel is mapped to Blue.

[signato]

@u4223374: Yes, you are right (meanwhile I found out myself): Streams that are undelayed during operations at other streams need to be buffered:

#pragma HLS stream depth=1300 variable=im_red.data_stream
#pragma HLS stream depth=1300 variable=im_green.data_stream

The depth in this example is a bit more than two line lengths of the image (I use 640x480).

This requirement is mentioned shortly in xapp1167, as I found out after long search.

Maybe this could be a candidate for automatic implementation in the synthesis? The buffer length should be calculable easily.

Thanks for all answers, especially to u4223374!

[u4223374]

@signato Good to hear that it's working.

That definitely does seem like something that HLS could do automatically - at least for most situations. Obviously it's possible to write a function for which HLS won't be able to calculate the latency (and therefore can't determine the correct buffer length) but this situation tends to be fairly rare. I've provided feedback along these lines before (right back in HLS 2014.4, if I remember correctly) but it's not been done yet.

[topshang]

Hello，

   my problem is similar with you, I want to process three channels ,but the program only process the first channel. like follow code, last merge pictures just display blue channel. That's to say, Program can execute the first channel process, but  when execute the next clause appear too many warnnings ,like following:

   hls::Split(img1,im_blue,im_green,im_red);

   hls::Remap<1080, 480, 640, 0, 0, HLS_16SC2, HLS_16SC2>(im_blue,im_blue_new,mx1,my1);

   hls::Remap<1080, 480, 640, 0, 0, HLS_16SC2, HLS_16SC2>(im_green,im_green_new,mx1,my1); 

   hls::Remap<1080, 480, 640, 0, 0, HLS_16SC2, HLS_16SC2>(im_red,im_red_new,mx1,my1);

   hls::Merge(img1r_b,im_blue_new,im_green_new,im_red_new);

WARNING: Hls::stream 'hls::stream<short>.1' is read while empty, which may result in RTL simulation hanging.

WARNING: Hls::stream 'hls::stream<short>.2' is read while empty, which may result in RTL simulation hanging.

...and so many this tpye warning

can you tell me the reason and how to solve this question ? thank you ~

[signato]

Hi,

I think your problem is different, but I might have a quick solution for you:

hls::Merge () - as most HLS library functions that I'm aware of - takes source parameters first, and destination parameter(s) last. In your case, img1r_b (I guess, that's your result image) should be the last argument.

That's the reason why img1r_b is empty when Merge is trying to read from it.

@Xilinx: It would be very helpful if in the warnings and error messages the real names of streams and other variables would be given, instead of 'hls::stream<short>.1' like in this case.

Moreover, the order of function arguments is a bit unfortunate since many C libraries use the order some_function (dst, src).

Hope this helps,

Signato

[u4223374]

@signato The names of streams disappear during compilation (ie they're not available to the Xilinx tools). However, Xilinx was able to implement a nifty workaround for the hls::Stream types by allowing the user to give them an explicit name:

int main(void) {
	hls::stream<char> b("hello world");

	b.read();
}

WARNING: Hls::stream 'hello world' is read while empty, which may result in RTL simulation hanging.

Hopefully they'll do the same for the hls::Mat types at some point.

@topshang You need to replicate the map matrices as well as the image. These matrices are absorbed during the call to Remap, so you can only use each one once.

[topshang]

Yes，you are right. Must replicate the map matrices to every remap function, and can we guess if every variable only use once?

[topshang]

yes， thank your answer,  remap function  process three channels simultaneously, so map matrices mx1 and my1 need to do Duplicate for every remap function. otherwise, These matrices are absorbed during the call to Remap. just like @u4223374 says.

[topshang]

Thank you very much..