cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Voyager
Voyager
2,358 Views
Registered: ‎07-06-2016

Gstreamer and sending data from PS to PL

Jump to solution

Hi,

I've currently got working a project using gstreamer for video encoding using the VCU. It takes video frames from a USB camera, encode them and then save it to a file.  

The camera interface is done at the PS side and the frames are pushed to the gstreamer pipeline.

Now I need to do some frame processing in the PL side before pushing the frames into the pipeline, is there a gstreamer element that allows this? 

What would be the best and more efficient way to do this?

Any ideas/help would be appreciated.

Thanks in advance

Tags (1)
0 Kudos
1 Solution

Accepted Solutions
Highlighted
Voyager
Voyager
2,056 Views
Registered: ‎07-06-2016

I'm already got it working, using the first option, with the mem2mem on passthrough mode and I configure the sensor demosaic IP registers using devmem.

View solution in original post

30 Replies
Highlighted
Moderator
Moderator
2,243 Views
Registered: ‎11-09-2015

Hi @joseer 

It depends on what you are trying to do. If you processing is only scaling using the Video Processing subsystem, then yes, you will be able to use Gstreamer because the VPSS as a V4L2 driver which can be called using Gstreamer.

But if you are using your custom IP, it will work only if you have the proper drivers.

Regards


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Highlighted
Voyager
Voyager
2,232 Views
Registered: ‎07-06-2016

Hi @florentw ,

Thanks for your answer,

What I'm trying to do is to convert raw bayer frames to RGB using the Sensor Demosaic IP, then convert them to NV12 and finally feed them to the VCU for encoding.

The data (frames) are received though USB and buffered to memory in the PS domain, I'm able to push the frames to a gstreamer pipeline using the appsrc element, encode and save them in a file, but obviously the resulting video color format is wrong, so I need to do a bayer interpolation and conversion to get the correct NV12 format supported by the VCU.

I could use the PS and video convert element to process and prepare the frames but that is quite processing intensive for the CPU so I need to use the PL in order to try to keep as much as possible the frame rate.

So in summary what I'm trying to do is (In red is the step I need to implement) :

USB frames --> memory (PS)---> push to Gstreamer pipeline (PS)  --> Bayer demosaic&NV12 conversion (PL) -> back to Gstreamer pipeline (PS) 

Or:

USB frames --> memory (PS)--->Bayer demosaic&NV12 conversion (PL)  -->memory (PS) --> push to Gstreamer pipeline (PS)   

I saw the mem2mem framework as a potential solution but unfortunatelly it doesn't like that supports Y8/GRAY8 or similar format needed to send the raw data to the PL....

Do you know what would be the best way or ways to do this?

0 Kudos
Highlighted
Moderator
Moderator
2,158 Views
Registered: ‎11-09-2015

HI @joseer 

I am not sure if that would work but what if you read the data as YUYV8. You will get the data as if it was 2 pixel per clock (just assume the chroma sample is actually a Y) and then you use an AXI4-Stream data width converter to move back to one pixel per clock.

This is just a idea, I have no idea if this will integrate well in the framework.

Let me know if you find a solution (or if you try and it work), I would be interested

Regards


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Highlighted
Voyager
Voyager
2,150 Views
Registered: ‎07-06-2016

Hi @florentw ,

Thanks for the reply and suggestion.

In summary what I'm seeing are two potential solutions/options, please fell free to add more alternatives (if there're or anyone know), or correct this ones:

- Frame conversion within the gstreamer pipeline: setup the mem2mem driver framework for a passthrough data but insert a axi4-Stream data width converter IP with the sensor demosaic IP, so the PL will look like:

frame-read-IP(YUYV8)-->axi4-Stream converter -> Sensor demosaic IP(RGB) ->frame-write-IP

- Frame conversion before to push to the gstreamer pipeline: setup a video pipeline (will be the same as above) and use the v4l2 driver to send the raw frames memory 2 memory (PS->PL->PS).

The first one seems the easiest but not sure if it would work, I'll try to post the results...

Highlighted
Voyager
Voyager
2,057 Views
Registered: ‎07-06-2016

I'm already got it working, using the first option, with the mem2mem on passthrough mode and I configure the sensor demosaic IP registers using devmem.

View solution in original post

Highlighted
1,502 Views
Registered: ‎11-07-2019

Josser,

I know it's been a while since you got your video processing pipeline working.  I am trying to do the same thing using the 2019.1 Xilinx development environment (Vivado and PetaLinux) with a USB camera storing Bayer BGGR video frames in DDR.  I need to retrieve the data from DDR and send it to a video processing pipeline in the fabric to Demosaic, perform gamma correction, and colorspace convert to NV12 or NV16 prior to returning to DDR and  placing it in a gstreamer pipeline allowing the VCU to encode to H.264.

I have been successful at creating a video processing pipeline in the fabric with the following IP:

FrmBufRead => SensorDemosaic => GammaLUT => VPSS CSC => VPSS Scaler => FrmBufWrite

The xlnx,video vcap driver successfully loads the V4L2 pipeline based on my updated device tree.  This results in the proper /dev/video0, /dev/video1, and /dev/v4l-subdevn Linux devices and my userspace application can query and initialize the V4L2 framework as expected.  I also understand the FrmBufRead does not support the BA81 pixel format and could use some advise on sequencing the Demosaic CR when starting the stream on the front end of the pipeline.  I am able to request the necessary buffers, queue the buffers and start streaming on the front end of the pipeline (the FrameBufferRead end) but cannot queue the buffers and start streaming on the back end of the pipeline (the FrameBufferWrite end) using the xlnx,video driver.

I would like to try using the mem2mem driver with my pipeline in the fabric since this would allow the use of the v4l2video0convert pluging within gstreamer and eliminate the need for the cumbersome userspace application to deal with the xlnx,video driver.  I can get the mem2mem driver to load correctly with just the FrameBufferRead and FrameBufferWrite IP in the fabric but cannot get it to load correctly with any other IP (i.e. Demosaic, Gamma LUT, VidieoProcessingSubsystem CSC only, VideoProcessingSubsystem Scaler only) between the FramBufferRead and FrameBufferWrite.

When updating the device tree within the system-user.dtsi file, what updates to the xlnx,video vcap section are required when the xlnx,mem2mem is being used?  Does the xlnx,video vcap section need to be completely removed in order to use the xlnx,mem2mem driver?  Do both xlnx,video and xlnx,mem2mem sections need to be included?  Looking at the Xilinx wiki at https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/80576536/Mem+2+Mem+Composite+Video+Framework and underlying pages, it is unclear how to configure the device with the mem2mem when the xlnx,video vcap driver is defined as well.  That is, when the device tree generator creates the xlnx,video vcap entries, what needs to change if the mem2mem is to be used?  Do the xlnx,video and xlnx,mem2mem drivers co-exist within the device tree configuration or can you only use one at a time?

Whenever I try using just the xlnx,mem2mem driver in the device tree, my kernel takes an OOPS and panics with a NULL pointer when loading the mem2mem driver.  When trying to combine the xlnx,video and xlnx,mem2mem drivers in the device tree, the mem2mem driver does not initialize properly with "...DMA initialization failed" and "...DMA tx channel not found" errors.

I've included a portion of my system-user.dtsi that produces a working xlnx,video driver but the mem2mem fails.  It does not include the Gamma LUT nor VPSS Scaler since it represents an attempt with a less-complex pipeline in the fabric.

I would really appreciate any help you (or anyone) can provide to get me past this issue.

Thanks!

0 Kudos
Highlighted
Voyager
Voyager
1,462 Views
Registered: ‎07-06-2016

Hi roger.a.luty@nasa.gov ,

Sorry for the late response, I don't know if you've already solve it but I'll try to share my experience on this.

The only way I had this working fine was using the mem2mem driver in pass-through mode  (you'll need to specify this in the gstreamer 'v4l2videoXconvert'  element with "disable_passthrough"  = 1), this setup also allowed me to easily add other custom IPs.

First of all I wouldn't recommend to use both drivers at the same time since essentially both are doing the same thing, defining a pipeline  (I always had conflict errors). Looking at your DT (I guess it is in your system-user.dtsi file)  I think that is the main issue.

For instance let's say you want to implement in the PL the next pipe: frmRead-> demosaic IP-> frmwrite ,  the system-user.dtsi should look like similar to this:

 

/include/ "system-conf.dtsi"
/ {

 chosen {
	bootargs = "earlycon console=ttyPS0,115200 clk_ignore_unused uio_pdrv_genirq.of_id=generic-uio";
 };

 video_m2m_pipe0 {
         compatible = "xlnx,mem2mem";
         dmas = <&v_frmbuf_rd_0 0>, <&v_frmbuf_wr_0 0>;
         dma-names = "tx", "rx";
 };


};


&v_demosaic_0{
	clock-names = "ap_clk";
	clocks = <&clk 71>;
	compatible = "generic-uio";
	reg = <0x0 0x80120000 0x0 0x10000>;

};

(......)

 

This device tree should generate:

- A v4l2video0convert element which can be used from a user space gstreamer application.

- And a video device (video0 in this case) which can be used with the v4l2 to configure the pipeline.

Things to consider:

- Any IPs within the pipeline should be configured before launching the gstreamer pipeline, for simplicity I opted to use the UIO driver (which it will also allow easily to setup custom IPs in future).

- The  'v4l2video0convert'  element is configured in user space like this: "disable_passthrough"  set to 1 and "capture-io-mode" and "output-io-mode" equal to the correct bit format (e.g: 4 would be for NV16 format).   

- Make sure the frame write and read IPs have enabled the correct bit stream format and also the video device is configured with same format, e.g. to use the NV16 make sure is enabled in the frame buffer IPs and to setup the device you can run the next command in petalinux:

v4l2-ctl --device /dev/video0 --set-fmt-video=width=1280,height=1024,pixelformat=NV16

Hope it helps.

 

 

 

 

0 Kudos
Highlighted
1,445 Views
Registered: ‎11-07-2019

Thanks Joseer for the reponse!

To be clear, when defining the mem2mem node in the device tree no endpoints are necessary, correct?  This results in only a /dev/videox file and no /dev/mediax or /dev/v4l-subdevx files.  Controlling the  IP within the FPGA (i.e. frmbufs, demosaic and vpss) is done using UIO or devmem prior to starting the gstreamer pipeline.  The v4l2video0convert needs to be configured as you mentioned within the gstreamer pipeline.  And yes, it makes sense to use only xlnx,video or xlnx,mem2mem since they both provide similar but definitely different ways to represent the Xilinx pipeline to userspace.

Thanks again.  I'll let you know how it goes...

0 Kudos
Highlighted
Voyager
Voyager
1,435 Views
Registered: ‎07-06-2016

Hi,

No problem,

Yes that's right, with that configuration I managed to get a working pipeline where I can send the video stream through the fabric for processing and get it back again to the user space.

Only pointing out that the frame buffers are not configure through UIO (or devmem), you can set them up in Vivado with the formats and later select the video device format and resolution you need and the driver should do the rest for you.

Let me know if it works.

Br

 

0 Kudos
Highlighted
1,335 Views
Registered: ‎11-07-2019

Hey Joseer,

Again, I appreciate your response.

I've had some success but still have an issue and questions.  I have the following simple video processing pipeline defined in the PL:

frmbuf_rd => sensor_demosaic => vpss_csc_only => frmbuf_wr

The only updates I've made to the device tree defines the xlnx,mem2mem driver as follows:

/ {
    video_m2m {
        compatible = "xlnx,mem2mem";
        dmas = <&v_frmbuf_rd_0 0>, <&v_frmbuf_wr_0 0>;
        dma-names = "tx", "rx";
    };
};

I also have enabled the sensor_demosaic and vpss_csc IP using devmem.  I also wrote a script to monitor the state of all four IP and the output from that script after starting gstreamer is included below:

FRMBUF BASE:0xa0060000
Control      0xa0060000 0x00000004 Start:0 Done:0 Idle:1 Ready:0 FlushPend:0 FlushDone:0 AutoRestart:0
GblIntEn     0xa0060004 0x00000001 Done:1 Ready:0
IPIntEn      0xa0060008 0x00000002 Done:0 Ready:1
IPIntSt      0xa006000c 0x00000000
Width        0xa0060010       1280
Height       0xa0060018        720
Stride       0xa0060020       1280
VideoFmt     0xa0060028 0x00000018
Plane1Buf    0xa0060030 0x74700000
Plane2Buf    0xa006003c 0x00000000
FieldID      0xa0060048 0x00000000

FRMBUF BASE:0xa0070000
Control      0xa0070000 0x00000001 Start:1 Done:0 Idle:0 Ready:0 FlushPend:0 FlushDone:0 AutoRestart:0
GblIntEn     0xa0070004 0x00000001 Done:1 Ready:0
IPIntEn      0xa0070008 0x00000002 Done:0 Ready:1
IPIntSt      0xa007000c 0x00000000
Width        0xa0070010       1280
Height       0xa0070018        720
Stride       0xa0070020       2560
VideoFmt     0xa0070028 0x0000001c
Plane1Buf    0xa0070030 0x73f00000
Plane2Buf    0xa007003c 0x00000000
FieldID      0xa0070048 0x00000000

DEMOSAIC BASE:0xa0040000
Control      0xa0040000 0x00000001 Start:1 Done:0 Idle:0 Ready:0 AutoRestart:0
GblIntEn     0xa0040004 0x00000001 Done:1 Ready:0
IPIntEn      0xa0040008 0x00000002 Done:0 Ready:1
IPIntSt      0xa004000c 0x00000000 Done:0 Ready:0
Width        0xa0040010       1280
Height       0xa0040018        720
BayerPhase   0xa0040028          3

CSC BASE:0xa0000000
Control      0xa0000000 0x00000001 Start:1 Done:0 Idle:0 Ready:0 AutoRestart:0
GblIntEn     0xa0000004 0x00000001 Done:1 Ready:0
IPIntEn      0xa0000008 0x00000002 Done:0 Ready:1
IPIntSt      0xa000000c 0x00000000 Done:0 Ready:0
FmtIn        0xa0000010          3
FmtOut       0xa0000018          3
Width        0xa0000020       1280
Height       0xa0000028        720

Interrupts:
 42:          7          0          0          0     GICv2 121 Level     xilinx_framebuffer
 43:          0          0          0          0     GICv2 122 Level     xilinx_framebuffer

Also, I created a bayer video file and stored it in /dev/shm/file1.  Therefore, the gstreamer command I'm using to play the file is included below:

gst-launch-1.0 filesrc location=/dev/shm/file1 blocksize=921600 ! video/x-raw,format=GRAY8,width=1280,height=720,framerate=30/1,interlace-mode=progressive,colorimetry=bt709 ! v4l2video0convert disable_passthrough=1 ! video/x-raw,format=UYVY,width=1280,height=720,framerate=30/1 ! xvimagesink

Note the interrupts for the topmost xilinx_framebuffer (frmbuf_rd) has a value of 7 since I've tried running this gstreamer command 7 times.  This means the frmbuf_rd is generating an interrupt but the frmbuf_wr is not generating an interrupt.  The xlnx,mem2mem driver is configuring the frmbuf_rd and frmbuf_wr IP and I'm manually configuring the sensor_demosaic and vpss_csc.

The main issue is the pipeline produces no output.  If I set the disable_passthrough=0 and set the video/x-raw,format=GRAY8 within the caps following v4l2video0convert, I get the unprocessed data out.

The main question I have is do the device tree nodes for the sensor_demosaic and vpss_csc require endpoints that connect the sensor_demosaic output to the vpss_css input?  If the answer is YES, what do the sensor_demosaic input and vpss_csc output endpoints need to be set to?   Normally I would update the system-user.dtsi to connect the sensor_demosaic intput and vpss_csc output endpoints to the xlnx,mem2mem node but when I do this, the system produces a fatal OOPS at startup.  Also, the device tree generator does not automatically produce any of the endpoints mentioned.

I can't help but think these missing endpoints may be impacting the sensor_demosaic and vpss_csc Linux device drivers somehow preventing the data flow but I have no evidence of this.  Another thought is I've included an AXI4-Stream Subset Converter between the frmbuf_rd and sensor_demosaic and could be preventing the data flow somehow.  Can you explain what may be broken with my design and/or logic trying to analyze this issue?

Thanks in advance!

0 Kudos
Highlighted
Voyager
Voyager
1,310 Views
Registered: ‎07-06-2016

Hi,

I see you've done quite a lot of progress!

When you set disable_passthrough=0  it will 'passthrough' the stream, (i.e it won't get through the PL),  passthrough=1 will force it to get it through the PL, where I think the issue is the pixel format, sorry I forgot to mention a few things about this:

- First you have to use same pixel format to write and read, you can convert to other (like RGB) within the pipeline but make sure the frame buffers write  and read work in same formats.

- The only format I managed to get the mem2mem driver working was NV16, with the version 2018.3, not sure if in newer versions would work with other pixel formats. (Note that NV16 will be how it is stored in memory on the PS side, the PL stream will be yuv422).

- To get this to work properly you will have to make sure you store correctly the raw pixel stream in memory before to push the data into the pipeline with gstreamer i.e make sure the raw data byte will be on the LSB of the yuv422 16 bit word (I used appsrc element to push the data). Then in Vivado if you connect the frame buffer read (with pixel data width 16) to the demosaic IP (which it has 8 bit input) will get only the LSB so you'll have the correct data stream format in the PL, just make sure the frame buffer write IP receives the yuv422 format in this case.

Hope it helps

0 Kudos
Highlighted
1,289 Views
Registered: ‎11-07-2019

Thanks Joseer for the response.

I managed to get the pipeline streaming.  The main problem was I became complacent and stopped checking whether all IP was being probed properly when the Linux kernel boots.  It turns out that during the process of switching between xlnx,video and xlnx,mem2mem, I didn't realize the port definitions of the vpss are still required even when the endpoints for each port are not defined.  Since the xlnx,mem2mem does not require the port definitions and the device tree generator was not creating them for the vpss, I assumed this was OK.  WRONG!  Once I added the port definitions to the vpss node within the system-user.dtsi without endpoints, the vpss Linux driver was happy and it probed successfully.  The pipeline now works!  Sort of...

I'm now trying to get bayer image data through the frmbuf_rd to the sensor_demosaic since technically frame buffers don't support bayer pixel formats.  Even though the 8-bit bayer pixel format and GRAY8 pixel format are the same size (1 byte/pixel), there seems to be an issue passing the bayer data from my camera stored in DDR through the frmbuf_rd and to the sensor_demosaic.  I can see the image after processing through my pipeline but the pixel format on the output is not correct.  No matter what I set the vpss_csc FormatIn and FormatOut to, the output image is the wrong pixel format and looks mostly green.  My pipeline is currently setup to handle all 8-bit formats based on the frmbuf customization withing Vidado.  Therefore, I can set the input to GRAY8, the output to NV12, NV16, UYVY, YUY2 and get data.

Anyway, thanks for your help.  If you have any ideas about correcting the pixel format issue, I would appreciate hearing from you again.  Otherwise, good luck to you and stay safe...

0 Kudos
Highlighted
Adventurer
Adventurer
1,176 Views
Registered: ‎03-31-2020

Hi roger.a.luty@nasa.gov 

I am facing a similar issue.

I got an error with gstreamer

0:00:01.015889170  2962   0x558b08a180 WARN          v4l2bufferpool gstv4l2bufferpool.c:807:gst_v4l2_buffer_pool_start:<v4l2video2convert0:pool:src> Uncertain or not enough buffers, enabling copy threshold

Did you define the input and output ports like below?

video_m2m {
				compatible = "xlnx,mem2mem";
ports {
				
						 port@0 {
							 
							    
						};
					 
						 port@1 {
							    
					 
						};
				};

}


ctrlvpss: vpss@0xa2200000 {
		
		compatible = "xlnx,v-vpss-scaler-2.2";
		
		ports {
					 #address-cells = <1>;
					 #size-cells = <0>;
					 port@0 {
			
						 
					};
					
					 port@1 {
			
						 
					};

					
		};


	};

 

 

0 Kudos
Highlighted
1,081 Views
Registered: ‎11-07-2019

I have since switched back to using the xlnx,video driver rather than the xlnx,mem2mem.  However, when I was using the xlnx,mem2mem, I had no ports defined within the m2m node.  Whenever I tried to include ports for the pipeline using m2m, the driver would not initialize and would not be usable.  Once I removed the ports from the m2m driver, it would initialize but no /dev/v4l-subdevX files would be created and the V4L2 utilities (media-ctl, yavta, and v4l2-ctl) could not be used.

The resulting xlnx,mem2mem node was defined as shown in the code segment.

video_m2m {
    compatible = "xlnx,mem2mem";
    dmas = <&v_frmbuf_rd_0 0>, <&v_frmbuf_wr_0 0>;
    dma-names = "tx", "rx";
};

Notice no port definitions are defined.

As I mentioned, since the pipeline defined using the m2m driver without ports didn't allow for the use of the V4L2 utilities, I moved back to the xlnx,video driver that DOES initialize properly when ports are included.  I also updated the hardware design to replace the v_frmbuf_rd with a vdma having just the read channel enabled at the front of the pipeline.  This allowed for the proper Bayer pixel format from the Sony imager being used but caused the pipeline to free-run since the vdma had no external synchronization.  This led me to modify the xilinx_dma.c driver to force the frame count interrupt to be enabled.  This allows the vdma to run one frame and halt to keep the pipeline from spinning between video frames.  Since Gstreamer seems to enable the vdma every frame, this works out well.  I suggest you look at lines 1103-1106 of the xilinx_dma.c source file.  A few strategically placed comments within the IF statement can ensure the frame count interrupt is enabled.

There is likely a more elegant solution to the free-run problem with the vdma.  However, this is work for me right now and I'm moving forward with the remaining userspace software implementation.

Good luck!  Working with the Xilinx tools has been very challenging and I hope more complete working examples of a hardware design and associated userspace software are presented in the future to help software engineers migrate from their comfy userspace realm to the Xilinx environment. 

Highlighted
Adventurer
Adventurer
1,071 Views
Registered: ‎03-31-2020

Hi roger.a.luty@nasa.gov,

Thank you so much for your feedback, which would save lots of my time with endless trials and errors.

 

0 Kudos
Highlighted
Voyager
Voyager
1,061 Views
Registered: ‎07-06-2016

I'd like to clarify a few points about this, first is that I'm using mem2mem driver in passthrough mode with a couple of xilinx IPs in the middle of the pipeline and it is working fine and the driver creates an associated video device to the pipeline, and I'm able to use the v4l2 tools for instance in order to query the pipeline capabilities.

But since I didn't define any ports in the DT, I'm not able to use the IPs v4l2 drivers for IP configuration (subdevices are not created) but instead you can use devmem or (my preference) the uio driver to configure the IPs as you would have to do anyway for a custom IP regardless the approach you're using, so you don't necessary need any /dev/v4l-subdevX to work with the mem2mem pipeline.

The only thing is that you have to make sure to get the IPs configured before starting any pipeline transaction.

 

 

 

Highlighted
Adventurer
Adventurer
1,056 Views
Registered: ‎03-31-2020

Hi @joseer,

I got exactly the same situation as yours. It seems that XIlinx's instructions on this is not enough for users to achieve the expected outcome. In my case, I wanted to use VPSS (Colorspace conversion, and Scaler), but it seems very difficult with devmem. I had to think about running the standalone driver on a separate PS core.

0 Kudos
Highlighted
Voyager
Voyager
1,050 Views
Registered: ‎07-06-2016

Hi @peakpeak , agreed Xilinx tools can be a bit challenging and time consuming sometimes....

I'm not sure what you're trying to achieve (colour conversion, scaling...) despite the VPSS it's quite complex IP you should still be able to configure it without having to use the v4l2 driver, the IP has axi-lite interface and accessible register space I would try uio driver instead devmem...

Highlighted
1,002 Views
Registered: ‎11-07-2019

All,

The complexity of the VPSS is exactly the reason I moved back to the xlnx,video driver rather than using xlnx,mem2mem.  Sure, you can spend hours writing code to use UIO or devmem to configure a CSC-only and/or Scaler-only VPSS prior to starting the pipeline, or you can move to xlnx,video and use yavta and or media-ctrl and do it very quickly.

I finally moved back to the xlnx,video driver since the Sony imager produces Bayer BGGR format and the v_frmbuf_rd does not support any Bayer format.  I could shove the Bayer BGGR format into the  v_vrmbuf_rd as GRAY8 but the rest of the Gstreamer pipeline would not process the video properly.

Anyway, happy coding.  And thank you again Joseer for all your help along the way.

Highlighted
Adventurer
Adventurer
943 Views
Registered: ‎03-31-2020

Hi @joseer ,

Sorry to bother you again in this topic. Have you succeeded in using the VPSS as generic uio device? I have been going through the VPSS driver and see that it relies heavily on memory configuration like

XV_csc_WriteReg(InstancePtr->Config.BaseAddress, XV_CSC_CTRL_ADDR_HWREG_WIDTH_DATA, Data);  (on linux), which is equivalent to Xil_Out32
this basically calls
*(volatile u32*)((BaseAddress) + (RegOffset)) = (u32)(Data)  in my case, it causes segmentation fault.

I could see the uio device created, but my mmap() always return as 0xffffffff. I am not sure if this approach is even feasible....

0 Kudos
Highlighted
924 Views
Registered: ‎11-07-2019

I know you're asking joseer this question, however, there must be something wrong with your mmap call parameters for it to return -1.  Check the error code coming back after your mmap call and that should shed some light.

Highlighted
Voyager
Voyager
908 Views
Registered: ‎07-06-2016

Hi,

As roger.a.luty@nasa.gov  is suggesting I would have a look to the mmap call first and maybe to the open file call too, for what you're saying it looks like the segmentation fault it's coming from a wrong address pointer value  returned by the mmap function. 

What I'd do:

- mmap shouldn't return -1 (0xffffffff) so I would capture the returned value and make sure to stop the program execution if it is -1 and show the error. That will avoid    entering into a segmentation fault

- Make sure when opening the uio file (/dev/uioX) you capture any error as well. Like that you can make sure you're trying to access to the right uio.

- I don't know if it would help but the mmap call should look similar to this: 

    ptr =mmap(NULL, 0x10000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
    if (ptr == MAP_FAILED) {
        printf("Mmap IP call failure.\n");
        return -1;
    }
Highlighted
Adventurer
Adventurer
891 Views
Registered: ‎03-31-2020

Hiroger.a.luty@nasa.gov ,

I tag joseer as he is the starter of this topic. Your answer is always welcome :). Thank you guys very much! It is invaluable to me

0 Kudos
Highlighted
Adventurer
Adventurer
839 Views
Registered: ‎03-31-2020

Hi @joseer and roger.a.luty@nasa.gov,

Could you please help me again:

I could devmem all other IPs in my design except for the VPSS, which is configured as

csc_0: vpss@0xa2200000{
		compatible = "generic-uio";
		reg = <0x0 0xa2200000 0x0 0x10000>;
                reset-gpios = <&gpio 80 1>;
		clocks = <&misc_clk_0>;
	};
The
devmem 0xa2200000 32 0x0
results in Linux hangs.

Did I miss something? My vivado design is confirmed as fine, as it works with xlnxvideoscale driver.

I have also done the following:

Set 
bootargs="console=ttyPS0,115200 root=/dev/mmcblk0p2 rw rootwait earlyprintk consoleblank=0 cma=1100M cpuidle.off=1 clk_ignore_unused loglevel=8 uio_pdrv_genirq.of_id=generic-uio";
Select
 • <*> Userspace I/O platform driver with generic IRQ handling
 • <*> Userspace platform driver with generic irq and dynamic memory
(The VPSS does not have interrupt pin though) • <*> Xilinx AI Engine driver

 And I could see the uio0 appeared as the vpss at the corresponding address.

0 Kudos
Highlighted
Voyager
Voyager
830 Views
Registered: ‎07-06-2016

Did you check that the map uio address you're trying to access is correct?

You can check it here: 

ls /sys/class/uio/uioX/maps/
and check for instance map0 addr:
cat /sys/class/uio/uioX/maps/map0/addr

 

 

0 Kudos
Highlighted
Adventurer
Adventurer
826 Views
Registered: ‎03-31-2020

Hi @joseer ,

Yes, I checked that. I also tried the 64-bit-version commands but not succeeded.

root@xilinx-zcu104-2019_1:/sys/class/uio/uio0/maps# cat map0/addr
0x00000000a2200000

root@xilinx-zcu104-2019_1:/sys/class/uio/uio0/maps/map0# cat name
vpss@0xa2200000
root@xilinx-zcu104-2019_1:/sys/class/uio/uio0/maps/map0# cat offset
0x0
root@xilinx-zcu104-2019_1:/sys/class/uio/uio0/maps/map0# cat size
0x0000000000010000



 

0 Kudos
Highlighted
Voyager
Voyager
813 Views
Registered: ‎07-06-2016

Sorry I've never used the VPSS core, I though that you could access to the registers as you would do with any IPs, but reading the documentation it looks like only the subsystem driver it's supported see at the bottom of the 12 page .

So on my view you've got two options, using the video driver instead mem2mem to generate the subsystem device or create your own scaler or color IP using for instance HLS....

Highlighted
Adventurer
Adventurer
796 Views
Registered: ‎03-31-2020

Thanks very much @joseer for your help. I hope roger.a.luty@nasa.gov could help on manual configuration of the csc as it seems he already did it.

By the way, you and roger mentioned using video driver, does that mean something like:

video_cap {
		compatible = "xlnx,video";
		dmas = <&fbrd 0>, <&fbwr 1>;
		dma-names = "port0", "port1";

		ports {
			#address-cells = <1>;
			#size-cells = <0>;

			port@0 {
					remote-endpoint = <>;
				};
			};
			port@1 {
				
				vcap0_in1: endpoint {
					remote-endpoint = <>;
				};
			};
		};
	};

So, I still need to configure the ports in VPSS, which is something that I want to avoid? Or there's another way to work around this?

I also wanted to avoid HLS as the NV12 video stream format is not well documented and I think I would get trouble with converting AXIStream to Mat.

0 Kudos
Highlighted
737 Views
Registered: ‎11-07-2019

Yes, the VPSS is a beast.  However, running as a CSC is easier than as a Scaler since then you have a vertical scaler and horizontal scaler to deal with.  The resulting system.dts nodes for my Demosaic, Gamma LUT, CSC, Scaler and xlnx,video are shown in the code segment below.

		v_demosaic@a0040000 {
			clock-names = "ap_clk";
			clocks = <0x25>;
			compatible = "xlnx,v-demosaic-1.0", "xlnx,v-demosaic";
			interrupt-names = "interrupt";
			interrupt-parent = <0x4>;
			interrupts = <0x0 0x5b 0x4>;
			reg = <0x0 0xa0040000 0x0 0x10000>;
			reset-gpios = <0x14 0x79 0x1>;
			xlnx,max-height = <0x870>;
			xlnx,max-width = <0xf00>;
			xlnx,s-axi-ctrl-addr-width = <0x6>;
			xlnx,s-axi-ctrl-data-width = <0x20>;
			ports {
				#address-cells = <0x1>;
				#size-cells = <0x0>;
				port@1 {
					reg = <0x1>;
					xlnx,cfa-pattern = "bggr";
					xlnx,video-width = <0x8>;
					endpoint {
						remote-endpoint = <0x28>;
						phandle = <0x2a>;
					};
				};
				port@0 {
					reg = <0x0>;
					xlnx,cfa-pattern = "bggr";
					xlnx,video-width = <0x8>;
					endpoint {
						remote-endpoint = <0x29>;
						phandle = <0x31>;
					};
				};
			};
		};

		v_gamma_lut@a0010000 {
			clock-names = "ap_clk";
			clocks = <0x25>;
			compatible = "xlnx,v-gamma-lut-1.0", "xlnx,v-gamma-lut";
			interrupt-names = "interrupt";
			interrupt-parent = <0x4>;
			interrupts = <0x0 0x5c 0x4>;
			reg = <0x0 0xa0010000 0x0 0x10000>;
			reset-gpios = <0x14 0x7a 0x1>;
			xlnx,max-height = <0x870>;
			xlnx,max-width = <0xf00>;
			xlnx,s-axi-ctrl-addr-width = <0xd>;
			xlnx,s-axi-ctrl-data-width = <0x20>;
			ports {
				#address-cells = <0x1>;
				#size-cells = <0x0>;
				port@0 {
					reg = <0x0>;
					xlnx,video-width = <0x8>;
					endpoint {
						remote-endpoint = <0x2a>;
						phandle = <0x28>;
					};
				};
				port@1 {
					reg = <0x1>;
					xlnx,video-width = <0x8>;
					endpoint {
						remote-endpoint = <0x2b>;
						phandle = <0x26>;
					};
				};
			};
		};

		v_proc_ss@a0000000 {
			clock-names = "aclk";
			clocks = <0x25>;
			compatible = "xlnx,v-vpss-csc";
			reg = <0x0 0xa0000000 0x0 0x10000>;
			reset-gpios = <0x14 0x7b 0x1>;
			xlnx,colorspace-support = <0x0>;
			xlnx,csc-enable-window = "false";
			xlnx,max-height = <0x870>;
			xlnx,max-width = <0xf00>;
			xlnx,num-video-components = <0x3>;
			xlnx,samples-per-clk = <0x2>;
			xlnx,topology = <0x3>;
			xlnx,use-uram = <0x0>;
			xlnx,video-width = <0x8>;
			ports {
				#address-cells = <0x1>;
				#size-cells = <0x0>;
				port@0 {
					reg = <0x0>;
					xlnx,video-format = <0xc>;
					xlnx,video-width = <0x8>;
					endpoint {
						remote-endpoint = <0x26>;
						phandle = <0x2b>;
					};
				};
				port@1 {
					reg = <0x1>;
					xlnx,video-format = <0x3>;
					xlnx,video-width = <0x8>;
					endpoint {
						remote-endpoint = <0x27>;
						phandle = <0x2c>;
					};
				};
			};
		};

		v_proc_ss@a0080000 {
			clock-names = "aclk_axis", "aclk_ctrl";
			clocks = <0x25 0x25>;
			compatible = "xlnx,v-vpss-scaler-2.1", "xlnx,v-vpss-scaler-1.0";
			reg = <0x0 0xa0080000 0x0 0x40000>;
			reset-gpios = <0x14 0x7c 0x1>;
			xlnx,colorspace-support = <0x0>;
			xlnx,csc-enable-window = "true";
			xlnx,enable-csc = "false";
			xlnx,h-scaler-phases = <0x40>;
			xlnx,h-scaler-taps = <0x6>;
			xlnx,max-height = <0x870>;
			xlnx,max-num-phases = <0x40>;
			xlnx,max-width = <0xf00>;
			xlnx,num-hori-taps = <0x6>;
			xlnx,num-vert-taps = <0x6>;
			xlnx,pix-per-clk = <0x2>;
			xlnx,samples-per-clk = <0x2>;
			xlnx,scaler-algorithm = <0x2>;
			xlnx,topology = <0x0>;
			xlnx,use-uram = <0x0>;
			xlnx,v-scaler-phases = <0x40>;
			xlnx,v-scaler-taps = <0x6>;
			xlnx,video-width = <0x8>;
			ports {
				#address-cells = <0x1>;
				#size-cells = <0x0>;
				port@0 {
					reg = <0x0>;
					xlnx,video-format = <0x3>;
					xlnx,video-width = <0x8>;
					endpoint {
						remote-endpoint = <0x2c>;
						phandle = <0x27>;
					};
				};
				port@1 {
					reg = <0x1>;
					xlnx,video-format = <0x3>;
					xlnx,video-width = <0x8>;
					endpoint {
						remote-endpoint = <0x2d>;
						phandle = <0x30>;
					};
				};
			};
		};

		vcap_csi {
			compatible = "xlnx,video";
			dma-names = "port0", "port1";
			dmas = <0x2e 0x0 0x2f 0x0>;
			ports {
				#address-cells = <0x1>;
				#size-cells = <0x0>;
				port@0 {
					direction = "input";
					reg = <0x0>;
					endpoint {
						remote-endpoint = <0x30>;
						phandle = <0x2d>;
					};
				};
				port@1 {
					reg = <0x1>;
					direction = "output";
					endpoint {
						remote-endpoint = <0x31>;
						phandle = <0x29>;
					};
				};
			};
		};

As mentioned previously, the front end of this pipeline is a Video DMA Read channel-only IP and the back end is a Video Frame Buffer Write IP.  I also had to modify the xilinx_dma.c to force the frame count interrupt to enabled as I also mentioned in a previous post.

Using the xlnx,video driver allows for using the V4L2 and media libraries to easily configure the IP rather than dealing with it manually.  I've used the following URL extensively to figure out how to use the V4L2 and media libraries in my userspace application.

https://www.kernel.org/doc/html/v4.9/media/uapi/v4l/v4l2.html

I hope this helps.  Good luck...