UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Observer signato
Observer
5,590 Views
Registered: ‎12-07-2010

HLS OpenCV: Certain operator combinations don't work

Jump to solution

Hello,

I have the problem that certain combinations of OpenCV operators (functions) don't work. These operators do work when used in other contexts.

"Don't work" means I get a colored-noise output. Debugging (ILA) shows that the HLS core processes some input data (as deep as the debugging memory goes), however, no output.

The streams are VGA (640x480) RGB image sequences at 60 Hz, the design is similar to that in XAPP1167.

 

A reduced example (all functions in hls namespace):

void hls_reflex (AXI_STREAM& video_in, AXI_STREAM& video_out, int shift, unsigned reflex_thresh)
{
 #pragma HLS INTERFACE axis port=video_in bundle=INPUT_STREAM
 #pragma HLS INTERFACE axis port=video_out bundle=OUTPUT_STREAM

 #pragma HLS INTERFACE s_axilite port=shift
 #pragma HLS INTERFACE s_axilite port=reflex_thresh

 #pragma HLS INTERFACE ap_stable port=shift
 #pragma HLS INTERFACE ap_stable port=reflex_thresh

    IM_8UC3 im3_in   (IROWS, ICOLS);
    IM_8UC3 im3_out  (IROWS, ICOLS);
    IM_8UC1 im_blue0 (IROWS, ICOLS);
    IM_8UC1 im_blue1 (IROWS, ICOLS);
    IM_8UC1 im_blue  (IROWS, ICOLS);
    IM_8UC1 im_green (IROWS, ICOLS);
    IM_8UC1 im_red   (IROWS, ICOLS);
    IM_8UC1 dst_spots (IROWS, ICOLS);
    IM_16SC1 grad_x (IROWS, ICOLS);
    IM_16SC1 grad_y (IROWS, ICOLS);
    IM_32SC1 grad_xy (IROWS, ICOLS);
    IM_8UC1 grad_thresh (IROWS, ICOLS);

 #pragma HLS dataflow
    AXIvideo2Mat (video_in, im3_in);
    Split (im3_in, im_blue, im_green, im_red);
        Duplicate (im_blue, im_blue0, im_blue1);

    Threshold (im_blue0, dst_spots, reflex_thresh, 255, HLS_THRESH_BINARY);

    Sobel<1, 0, 3> (dst_spots, grad_x);
    Sobel<0, 1, 3> (im_blue1, grad_y);

    fp_t fshift = shift * fp_t (1E-3);
    AddWeighted (grad_x, fshift, grad_y, fshift, fp_t (0), grad_xy);
    Scale (grad_xy, grad_thresh, fshift);

    // works
    Mat2AXIvideo (grad_thresh, video_out);

    // doesn't work
//    Merge (grad_thresh, im_green, im_red, im3_out);
//    Mat2AXIvideo (im3_out, video_out);
}

This one works, however, when I add the Merge operator (last-but-one line), I get the described noisy output.

 

Any help appreciated

Signato

Tags (2)
0 Kudos
1 Solution

Accepted Solutions
Scholar u4223374
Scholar
9,389 Views
Registered: ‎04-26-2015

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

I think the problem is that your im_blue/grad_thresh is delayed compared to the other streams. If you use the STREAM pragma and specify a FIFO length for the other two (maybe 10 lines to start with, reduce it once the system works) then that should solve the issue.

View solution in original post

13 Replies
Observer signato
Observer
5,307 Views
Registered: ‎12-07-2010

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

Hi all,

meanwhile I reduced the problem to the essence (I hope):

A three-channel (RGB) stream is split into single channels, one of them is processed somehow, then the result and the other two are merged again.

void hls_reflex (AXI_STREAM& video_in, AXI_STREAM& video_out, int shift, unsigned reflex_thresh)
{
 #pragma HLS INTERFACE axis port=video_in bundle=INPUT_STREAM
 #pragma HLS INTERFACE axis port=video_out bundle=OUTPUT_STREAM

 #pragma HLS INTERFACE s_axilite port=shift
 #pragma HLS INTERFACE s_axilite port=reflex_thresh

	IM_8UC3 im3_in   (IROWS, ICOLS);
	IM_8UC3 im3_out  (IROWS, ICOLS);
	IM_8UC1 im_blue  (IROWS, ICOLS);
	IM_8UC1 im_green (IROWS, ICOLS);
	IM_8UC1 im_red   (IROWS, ICOLS);
	IM_8UC1 grad_thresh (IROWS, ICOLS);

 #pragma HLS dataflow
	AXIvideo2Mat (video_in, im3_in);
	Split (im3_in, im_blue, im_green, im_red);

	Dilate (im_blue, grad_thresh);

	// works
//	Mat2AXIvideo (grad_thresh, video_out); /*

	// doesn't work
	Merge (grad_thresh, im_green, im_red, im3_out);
	Mat2AXIvideo (im3_out, video_out); // */
}

So, there seems to be some problem with buffering of the other two (here im_green, im_red), or synchronization with the modified stream, respectively.

 

Or am I thinking wrongly?

 

Any help appreciated

Signato

 

0 Kudos
Explorer
Explorer
5,289 Views
Registered: ‎04-23-2014

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

@signato Not an answer. Just thinking, it does not make sense for me how it works at all:

 

Mat2AXIvideo (grad_thresh, video_out);

 

Your video_out is supposed to be 3 channel, and you provide grad_thresh, which is 1 channel. Does Mat2Axivideo() expands it to all three channels?

0 Kudos
Scholar u4223374
Scholar
9,390 Views
Registered: ‎04-26-2015

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

I think the problem is that your im_blue/grad_thresh is delayed compared to the other streams. If you use the STREAM pragma and specify a FIFO length for the other two (maybe 10 lines to start with, reduce it once the system works) then that should solve the issue.

View solution in original post

Observer signato
Observer
5,184 Views
Registered: ‎12-07-2010

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution
@naz_rb: You are right - I chose this code only for demonstration purposes. However, Mat2Axivideo () does work with a one-channel input - it produces RGB where R and G are '1' (255) always, and the input channel is mapped to Blue.
0 Kudos
Observer signato
Observer
5,178 Views
Registered: ‎12-07-2010

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

#pragma HLS stream depth=1300 variable=im_red.data_stream
#pragma HLS stream depth=1300 variable=im_green.data_stream

 

 

0 Kudos
Scholar u4223374
Scholar
5,169 Views
Registered: ‎04-26-2015

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

@signato Good to hear that it's working.

 

That definitely does seem like something that HLS could do automatically - at least for most situations. Obviously it's possible to write a function for which HLS won't be able to calculate the latency (and therefore can't determine the correct buffer length) but this situation tends to be fairly rare. I've provided feedback along these lines before (right back in HLS 2014.4, if I remember correctly) but it's not been done yet.

0 Kudos
Visitor topshang
Visitor
2,854 Views
Registered: ‎10-29-2017

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

Hello,

   my problem is similar with you, I want to process three channels ,but the program only process the first channel. like follow code, last merge pictures just display blue channel. That's to say, Program can execute the first channel process, but  when execute the next clause appear too many warnnings ,like following:

 

   hls::Split(img1,im_blue,im_green,im_red);

 

   hls::Remap<1080, 480, 640, 0, 0, HLS_16SC2, HLS_16SC2>(im_blue,im_blue_new,mx1,my1);

   hls::Remap<1080, 480, 640, 0, 0, HLS_16SC2, HLS_16SC2>(im_green,im_green_new,mx1,my1); 

   hls::Remap<1080, 480, 640, 0, 0, HLS_16SC2, HLS_16SC2>(im_red,im_red_new,mx1,my1);

 

   hls::Merge(img1r_b,im_blue_new,im_green_new,im_red_new);

 

WARNING: Hls::stream 'hls::stream<short>.1' is read while empty, which may result in RTL simulation hanging.

WARNING: Hls::stream 'hls::stream<short>.2' is read while empty, which may result in RTL simulation hanging.

...and so many this tpye warning

 

can you tell me the reason and how to solve this question ? thank you ~

0 Kudos
Visitor topshang
Visitor
2,829 Views
Registered: ‎10-29-2017

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution
 
result_480p.bmp
0 Kudos
Observer signato
Observer
2,794 Views
Registered: ‎12-07-2010

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

Hi,

I think your problem is different, but I might have a quick solution for you:

hls::Merge () - as most HLS library functions that I'm aware of - takes source parameters first, and destination parameter(s) last. In your case, img1r_b (I guess, that's your result image) should be the last argument.

That's the reason why img1r_b is empty when Merge is trying to read from it.

 

@Xilinx: It would be very helpful if in the warnings and error messages the real names of streams and other variables would be given, instead of 'hls::stream<short>.1' like in this case.

 

Moreover, the order of function arguments is a bit unfortunate since many C libraries use the order some_function (dst, src).

 

Hope this helps,

Signato

0 Kudos
Scholar u4223374
Scholar
2,277 Views
Registered: ‎04-26-2015

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

@signato The names of streams disappear during compilation (ie they're not available to the Xilinx tools). However, Xilinx was able to implement a nifty workaround for the hls::Stream types by allowing the user to give them an explicit name:

 

int main(void) {
	hls::stream<char> b("hello world");

	b.read();
}
WARNING: Hls::stream 'hello world' is read while empty, which may result in RTL simulation hanging.

Hopefully they'll do the same for the hls::Mat types at some point.

 

 

@topshang You need to replicate the map matrices as well as the image. These matrices are absorbed during the call to Remap, so you can only use each one once.

 

Visitor topshang
Visitor
2,261 Views
Registered: ‎10-29-2017

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

Yes,you are right. Must replicate the map matrices to every remap function, and can we guess if every variable only use once?

0 Kudos
Visitor topshang
Visitor
2,255 Views
Registered: ‎10-29-2017

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution

yes, thank your answer,  remap function  process three channels simultaneously, so map matrices mx1 and my1 need to do Duplicate for every remap function. otherwise, These matrices are absorbed during the call to Remap. just like @u4223374 says.

0 Kudos
Visitor topshang
Visitor
2,252 Views
Registered: ‎10-29-2017

Re: HLS OpenCV: Certain operator combinations don't work

Jump to solution
Thank you very much..
0 Kudos