cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
kumakich
Contributor
Contributor
1,145 Views
Registered: ‎02-17-2021

How to modify "VCU TRD Multi Stream Video Capture and Display" to make MIPI CSI to work in xilinx low latency mode?

Hi,

I am working on a project trying to modify the TRD

"VCU TRD Multi Stream Video Capture and Display"

to include the SyncIP as to make the gstreamer pipeline to either

1.accept MIPI CSI input an\d

2.capture the frame in xilinx low latency mode.

 

Hardware[SyncIP] has been created by adding the syncIP in vivado, based on "VCU TRD Multi Stream Video Capture and Display"

and after importing the created xsa file and petalinx-build I got the firmware image.

 

The problem iI am having is that although I can get the frame from MIPI device 

like:

 

gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=4 ! video/x-raw,format=NV12,width=3840,height=2160,framerate=60/1 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id=a0070000.v_mix async=false sync=true"

 

or

 

yavta -n 3 -c10 -f NV12 -s 3840x2160 --skip 7 -F /dev/video0

 

but when trying to use with the xilinx low latency mode I could get only green screen.

 

gst-launch-1.0 -v v4l2src io-mode=4 device=/dev/video0 ! video/x-raw\(memory:XLNXLL\), width=3840, height=2160, format=NV12, framerate=60/1 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

 

I tried to compare the following 2 files provided in: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010368517/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2+-+Run+and+Build+Flow

1.vcu_llp2_psddr_hdmi.dtsi (working in hdmi rx/tx with lowltency mode, but no mipi input)

2.vcu_trd.dtsi(no latency mode enabled, but mipi supported)[this design is modified to accept the syncIP]

and don't know which part in the file should I modify to make it work.

Please kindly let me know if you know what I should do.

Your kind assistance would be greatly appreciated.

Thank you in advance.

 

Best,

 

 

0 Kudos
25 Replies
kumakich
Contributor
Contributor
1,102 Views
Registered: ‎02-17-2021

Some additional information:

According to the link below, 

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010368517/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2+-+Run+and+Build+Flow

To switch? between different design, one needs only to 

1.design the hardware and export the xsa file.

2.input the xsa to petalinux project

3.swich the system-user.dtsi link

 

I am thinking that, 

if I add the Sync IP to the MIPI TDR hardware, then

merge the mipi csi-2 part of Design Project dtsi VCU TRD Multi Stream Video Capture and Display[vcu_trd.dtsi]

to

Xilinx Low Latency PS DDR NV12 HDMI Audio Video Capture and Display[vcu_llp2_psddr_hdmi.dtsi]

will it work?

0 Kudos
kumakich
Contributor
Contributor
1,059 Views
Registered: ‎02-17-2021

Again, some additional information.

I also tried to compare the pl.dtsi files in

components/plnx_workspace/device-tree/include/pl.dtsi

from the one I am working on[vcu_trd + sync_IP]

		sync_ip_0: sync_ip@a02f0000 {
			clock-names = "s_axi_ctrl_aclk", "s_axi_mm_aclk", "s_axi_mm_p_aclk";
			clocks = <&zynqmp_clk 72>, <&misc_clk_0>, <&zynqmp_clk 72>;
			compatible = "xlnx,sync-ip-1.0";
			interrupt-names = "host_irq";
			interrupt-parent = <&gic>;
			interrupts = <0 90 4>;
			reg = <0x0 0xa02f0000 0x0 0x10000>;
			xlnx,encode ;
			xlnx,num-chan = <1>;
		};

 to the one created based on zcu106-llp_audio_nv12_wrapper.xsa

		llp2_sync_ip_0: sync_ip@a00c0000 {
			clock-names = "s_axi_ctrl_aclk", "s_axi_mm_aclk", "s_axi_mm_p_aclk";
			clocks = <&zynqmp_clk 72>, <&misc_clk_4>, <&zynqmp_clk 72>;
			compatible = "xlnx,sync-ip-1.0";
			interrupt-names = "host_irq";
			interrupt-parent = <&gic>;
			interrupts = <0 106 4>;
			reg = <0x0 0xa00c0000 0x0 0x10000>;
			xlnx,encode ;
			xlnx,num-chan = <4>;
		};

And didn't get much luck.

I alse refer to the provided 

https://www.xilinx.com/support/answers/75894.html

and don't really understand where I need to add as to make the trd_vcu.xsa working with sync_ip.

Please kindly help anyone knows where I did wrong.

Thank you in advance.

0 Kudos
kumakich
Contributor
Contributor
1,021 Views
Registered: ‎02-17-2021

Stilll working on it, but I kind of found the document:

https://www.xilinx.com/support/documentation/boards_and_kits/zcu106/2020_2/ug1250-zcu106-vcu-trd.pdf

I noticed there is this thing:

Table 5-7: Interrupt ID Map for Full-fledged Design

HDMI Frame Buffer Write_0:90

and

MIPI Frame Buffer Write:105

I am guessing I just need to change the number of interrupt from

interrupts = <0 90 4>;
to
interrupts = <0 105 4>;

in components/plnx_workspace/device-tree/include/pl.dtsi

to make MIPI camera work with SYNC_IP?

 

Is there a simple way to do this before petalinux-build-ing this project? from hardware setting or something?

as the components/plnx_workspace/device-tree/include/pl.dtsi won't show until petalinux-build is executed.

 

0 Kudos
kumakich
Contributor
Contributor
941 Views
Registered: ‎02-17-2021

Hello again, still working on this problem.

I confirmed with the hw designer in my team about the interrupt number, and 

the previous 90 was correct. so the not-working-problem has nothing to do with it.

 

interrupts = <0 90 4>;

 

and I double checked the patches required in vcu_trd and hdmi_llp2 design. all included.

 

SRC_URI += "file://user.cfg"

SRC_URI_append = " \
	file://0001-media-xilinx-TPG-Add-IOCTL-to-set-PPC.patch \
	file://0002-drm-xlnx_mixer-Dont-enable-primary-plane-by-default.patch \
	file://0003-drm_atomic_helper-Supress-vblank-timeout-warning-mes.patch \
	file://0004-drivers-misc-add-support-for-interrupt-based-PCIe-en.patch \
	file://0005-xilinx_pci_endpoint-Add-resolution-use-case-and-64-b.patch \
	file://0006-Added-ioctl-to-support-different-formats.patch \
	file://0007-v4l-xilinx-Driver-support-for-Xilinx-AXI4-Stream-Bro.patch \
	file://0008-arm-zynq-Add-MAX20087-driver.patch \
	file://0009-arm-zynq-Add-avt_multi_sensor_fmc-driver.patch \
	file://0010-max9286-serdes-Fix-source-pad-media-format.patch \
	file://0011-ar0231-Fix-the-media-bus-format-to-GRBG.patch \
	file://0012-avt_multi_sensor_fmc-Add-dependency-on-REGULATOR.patch \
	file://0013-max20087-Remove-unused-members.patch \
	file://0014-v4l-xilinx-dma-Removed-stream-count-and-number-of-dm.patch \
	file://0015-dma-mapping-avoid-calling-memset-unless-gfp-flag-set.patch \
"

 

 

Here is the current situation:

 

Working:

 

yavta -n 3 -c10 -f NV12 -s 3840x2160 --skip 7 -F /dev/video0

Encode and  Decoded by cpu->OK but really lag in latency

 

gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=60/1 ! omxh264enc ! video/x-h264 ! queue max-size-buffers=0 ! omxh264dec ! video/x-raw ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=false -v

 

 

Not Working:

Green screen only: once the xilinx memory is used:

 

gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw\(memory:XLNXLL\) ,width=3840, height=2160, framerate=30/1 ! kmssink bus-id=a0070000.v_mix

gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw\(memory:XLNXLL\) ,width=3840, height=2160, framerate=30/1 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 show-preroll=false sync=true' sync=true -v

 

 

Once the following command run, all the working examples above will also end in [MCU trace: Channel has hang]

#Mcu trace: Channel has hang
gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 ! video/x-raw\(memory:XLNXLL\), width=3840, height=2160, format=NV12, framerate=60/1 ! omxh265enc num-slices=8 control-rate=low-latency gop-mode=low-delay-p target-bitrate=25000 cpb-size=500 gdr-mode=horizontal initial-delay=250 periodicity-idr=240 filler-data=0 prefetch-buffer=true ! video/x-h265, alignment=nal ! queue max-size-buffers=0 ! omxh265dec low-latency=1 internal-entropy-buffers=5 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 show-preroll-frame=false sync=true' sync=true -v

 

I also checked the IRQ in linux as below, the number of a02f0000.sync_ip never increases while in the example of HDMI llp2 nv12, the sync_ip does increase.

root@zcu106_vcu_trd:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  3:      30420      20985      19447      27144     GICv2  30 Level     arch_timer
  6:          0          0          0          0     GICv2  67 Level     zynqmp_ipi
 12:          0          0          0          0     GICv2 155 Level     axi-pmon, axi-pmon
 13:      19632          0          0          0     GICv2 156 Level     zynqmp-dma
 14:          0          0          0          0     GICv2 157 Level     zynqmp-dma
 15:          0          0          0          0     GICv2 158 Level     zynqmp-dma
 16:          0          0          0          0     GICv2 159 Level     zynqmp-dma
 17:          0          0          0          0     GICv2 160 Level     zynqmp-dma
 18:          0          0          0          0     GICv2 161 Level     zynqmp-dma
 19:          0          0          0          0     GICv2 162 Level     zynqmp-dma
 20:          0          0          0          0     GICv2 163 Level     zynqmp-dma
 21:          0          0          0          0     GICv2 164 Level     Mali_GP_MMU, Mali_GP, Mali_PP0_MMU, Mali_PP0, Mali_PP1_MMU, Mali_PP1
 22:          0          0          0          0     GICv2 109 Level     zynqmp-dma
 23:          0          0          0          0     GICv2 110 Level     zynqmp-dma
 24:          0          0          0          0     GICv2 111 Level     zynqmp-dma
 25:          0          0          0          0     GICv2 112 Level     zynqmp-dma
 26:          0          0          0          0     GICv2 113 Level     zynqmp-dma
 27:          0          0          0          0     GICv2 114 Level     zynqmp-dma
 28:          0          0          0          0     GICv2 115 Level     zynqmp-dma
 29:          0          0          0          0     GICv2 116 Level     zynqmp-dma
 31:          0          0          0          0     GICv2  95 Level     eth0, eth0
 33:        121          0          0          0     GICv2  49 Level     cdns-i2c
 34:        129          0          0          0     GICv2  50 Level     cdns-i2c
 35:          0          0          0          0     GICv2  42 Level     ff960000.memory-controller
 36:          0          0          0          0     GICv2  57 Level     axi-pmon, axi-pmon
 37:         43          0          0          0     GICv2  47 Level     ff0f0000.spi
 38:          0          0          0          0     GICv2  58 Level     ffa60000.rtc
 39:          0          0          0          0     GICv2  59 Level     ffa60000.rtc
 40:          0          0          0          0     GICv2 165 Level     ahci-ceva[fd0c0000.ahci]
 41:        213          0          0          0     GICv2  81 Level     mmc0
 42:       1861          0          0          0     GICv2  53 Level     xuartps
 44:          0          0          0          0     GICv2  84 Edge      ff150000.watchdog
 45:          0          0          0          0     GICv2  88 Level     ams-irq
 46:         12          0          0          0     GICv2 154 Level     fd4c0000.dma
 47:          0          0          0          0     GICv2 151 Level     fd4a0000.zynqmp-display
 49:          0          0          0          0     GICv2 121 Level     xilinx_framebuffer
 50:      22830          0          0          0     GICv2 125 Level     xilinx-hdmitxss
 51:      22865          0          0          0     GICv2 127 Level     xlnx-mixer
 52:       6291          0          0          0     GICv2 137 Level     xilinx-csi2rxss
 53:       6285          0          0          0     GICv2 138 Level     xilinx_framebuffer
 54:         81          0          0          0     GICv2 126 Level     a0050000.i2c
 55:        363          0          0          0     GICv2 139 Level     a0051000.i2c
 56:          0          0          0          0     GICv2 122 Edge      a02f0000.sync_ip
 57:          0          0          0          0     GICv2 141 Level     a02e0000.v_scenechange
 58:      12351          0          0          0     GICv2 128 Level     a0120000.al5d, a0100000.al5e
 59:         23          0          0          0     GICv2 124 Level     xilinx-vphy
 60:          0          0          0          0     GICv2  97 Level     xhci-hcd:usb1
IPI0:     14347      21691      25375      22072       Rescheduling interrupts
IPI1:        19         49         16         51       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:         1          0          0          0       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts

 

Do I also need to change other things in kernel or driver adding the sync_IP to VCU_TRD?

As far as I see, the difference between VCU_TRD and HDMI_LLP2_nv12 is

1.diffenret xsa[hardware]

2,device tree link

As mention before I modified the device tree and add the mipi part into it.

Did I miss anything?

 

Does xilinx have any working example with MIPI CSI RX with SYNC_IP?

As I see you provide the full-fledged design in the document.

Is it possible you also share that?

 

Thank you for your help in advance.

Tags (1)
0 Kudos
kumakich
Contributor
Contributor
890 Views
Registered: ‎02-17-2021

add some info.

when trying to stream, 

GST_DEBUG=1,v4l2src:9 gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 num-buffers=1 ! video/x-raw\(memory:XLNXLL\), width=3840, height=2160, framerate=60/1 ! omxh264enc num-slices=1 control-rate=low-latency gop-mode=low-delay-p target-bitrate=25000 cpb-size=500 initial-delay=250 periodicity-idr=240 filler-data=false prefetch-buffer=true ! video/x-h264,profile=main, alignment=nal ! queue max-size-buffers=0 ! omxh264dec low-latency=1 internal-entropy-buffers=5 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 sync=true' sync=true -v

I got these log, I guess I need to look into the gstv4l2src.c file? Is this the proper one? How can I find it in the local/build project folder?

https://github.com/dylany-xlnx/xvfbsync/blob/master/xvfbsync.c

 

0:00:01.546284740  1274 0xaaaab2a4cb70 DEBUG                v4l2src gstv4l2src.c:1600:gst_v4l2src_event:<v4l2src0> XLNX-LowLatency consumer ready, start DMA
0:00:01.852970393  1274 0xaaaab2a4cb70 INFO                 v4l2src gstv4l2src.c:1586:start_xilinx_dma:<v4l2src0> XLNXLL: DMA started:
0:03:59.521919940  1274 0xaaaab2b2ab90 ERROR               xvfbsync xvfbsync.c:343:xvfbsync_syncip_add_buffer: SyncIp: Couldn't add buffer
[  180.693419] xilinx-csi2rxss a00f0000.mipi_csi2_rx_subsystem: Stream Line Buffer Full!

 

 

0 Kudos
kvasantr
Moderator
Moderator
834 Views
Registered: ‎04-12-2017

Hello @kumakich 

after going through your thread it looks like you are successfully able to capture the raw frames using yavta and using gst-launch you are able send raw data to display path successfully.

I think the problem lies underneath somewhere in the software stack of VCU because of which VCU is unable to access the raw video frames from memory. for your information Xilinx has not tried Xilinx low-latency use case with MIPI camera interfacing. 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
kumakich
Contributor
Contributor
821 Views
Registered: ‎02-17-2021

hi @kvasantr 

Thank you for your reply.

>I think the problem lies underneath somewhere in the software stack of VCU because of which VCU is unable to access the raw video frames from memory. for your information Xilinx has not tried Xilinx low-latency use case with MIPI camera interfacing. 

 

Previously while building with VCU_TRD and VCU_TRD_LLP2_PSDDR_NV12_HDMI,

the only difference I noticed was the 

1.0014./0015. patches

2.system-user.dtsi file.

if the problem is in "software stack of VCU",  that means the current software stack of VCU is able to handle either  VCU_TRD and VCU_TRD_LLP2_PSDDR_NV12_HDMI.

but not able to handle the one I am working on "VCU_TRD based design, SYNCIP added" right?

 

how can I debug this kind of problem?

# do I need to confirm the gstreamer? or kernel source code?

 

Please kindly help, thank you so much.

0 Kudos
kumakich
Contributor
Contributor
790 Views
Registered: ‎02-17-2021

Hi @kvasantr 

Please correct me if my understanding is wrong:

Question:

I will need to modify the GStreamer and  maybe OpenMAX to support MIPI when using MIPI + SYNC_IP?

 

Details:

-------------------------

Fact A

[Programming Sequence of Synchronization IP] in

https://www.xilinx.com/support/documentation/ip_documentation/vcu/v1_2/pg252-vcu.pdf

has the defination of "software stack of VCU" as The Video Codec Unit (VCU) software stack has a layered architecture programmable at several levels of abstraction available to software developers, as shown in the following figure. The application interfaces from high level to low level are:
• GStreamer
• OpenMAX Integration Layer
• VCU Control Software (to my understanding this has nothing to do the the gstreamer pipeline)

 

Fact B

Xilinx VCU_TRD is provided, the same software or firmware with different hardware designs can handle designs either with/without SYNC_IP

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010303239/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2+-+VCU+TRD+Multi+Stream+Video+Capture+and+Display  ←This does't include SYNC_IP but has MIPI as RX input

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010303367/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2+-+Xilinx+Low+Latency+PS+DDR+NV12+HDMI+Audio+Video+Capture+and+Display←This do include SYNC_IP but does't support MIPI RX

 

Fact C

Gstreamer pipelines work well in either 1 or 2 case, the only difference is using  video/x-raw(memory:XLNXLL) or video/x-raw

 

And according to you:

I think the problem lies underneath somewhere in the software stack of VCU because of which VCU is unable to access the raw video frames from memory.

This might means 

• GStreamer
• OpenMAX Integration Layer the two layers can support only:

①:HDMI + MIPI + SDI without SYNC IP

②When SYNC_IP enabled, the input source should only be HDMI 

 

In another word, I will need to modify the GStreamer to support MIPI

when using MIPI + SYNC_IP?

 

 

Thank you in advance for you kindly help.

0 Kudos
kumakich
Contributor
Contributor
767 Views
Registered: ‎02-17-2021

Some additional information:

We found that strangely when the mipi rx system is configured as FHD@30FPS the Gstreamer pipeline works like magic.

 

gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 ! video/x-raw\(memory:XLNXLL\), width=1920, height=1080, framerate=30/1 ! omxh264enc num-slices=8 control-rate=low-latency gop-mode=low-delay-p target-bitrate=25000 cpb-size=500 initial-delay=250 periodicity-idr=240 filler-data=false prefetch-buffer=true ! video/x-h264,profile=main, alignment=nal ! queue max-size-buffers=0 ! omxh264dec low-latency=1 internal-entropy-buffers=5 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 sync=true' sync=true -v

 

 

and the latency of Capture/Encoder/Decoder is actually working as Xilinx Low-latency mode as the latency is 1[ms] Encoder:18[ms]] Decoder:17[ms] as shown below:

 

root@zcu106_vcu_trd:~# grep -inr "report latency" /run/latency.txt | grep v4l2;

3596:0:00:00.706194911  5195 0xffff84004ca0 DEBUG                v4l2src gstv4l2src.c:791:gst_v4l2src_query:<v4l2src0> report latency min 0:00:00.001000000 max 0:00:00.166666665

816:0:00:00.433125617  5195 0xaaab0001a770 DEBUG            omxvideoenc gstomxvideoenc.c:3524:gst_omx_video_enc_set_format:<omxh264enc-omxh264enc0> Input is using XLNX-LowLatency

gstomxvideoenc.c:2659:gst_omx_video_enc_set_latency:<omxh264enc-omxh264enc0> retrieved latency of 18 ms

1328:0:00:00.507035202  5195 0xaaab000cf8a0 DEBUG            omxvideodec gstomxvideodec.c:2544:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> Enable XLNX-LowLatency
1337:0:00:00.507222741  5195 0xaaab000cf8a0 DEBUG            omxvideodec gstomxvideodec.c:2481:gst_omx_video_dec_set_latency:<omxh264dec-omxh264dec0> retrieved latency of 17 ms
 

 

Still the CSI pipeline full log show as before:

 

Setting pipeline to READY ...
[10704.911875] xilinx-csi2rxss a00f0000.mipi_csi2_rx_subsystem: Stream Line Buffer Full!

 

 

 

However, even a slight modication of refresh rate to 60, the pipeline stalls again.

Please kindly shed some light.

Thank you in advance again.

kvasantr
Moderator
Moderator
686 Views
Registered: ‎04-12-2017

Hi @kumakich 

from PG232 of MIPI CSI RX SS it tells me that 

"Ensure line buffer full condition is not set in the MIPI CSI-2 RX Controller Interrupt Status register. Core setting this bit implies that the input data rate is higher than the output data rate. Consider either decreasing input data rate (DPHY Line rate) or increase output data rate (Select appropriate output pixel per Clock: Single, Dual, Quad)"

can you confirm from your IP settings what it has been set correctly?

are you able to capture the 60FPS content using yavta in the memory?

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842522/Xilinx+V4L2+MIPI+CSI+driver

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
kumakich
Contributor
Contributor
670 Views
Registered: ‎02-17-2021

Hi @kvasantr

Thank for your reply! 

Will confirm it.

>are you able to capture the 60FPS content using yavta in the memory?

Yes , the log shows the capture is around 60 fps.

 

root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# yavta -n 3 -c10 -f NV12 -s 3840x2160 --skip 7 -F /dev/video0
Device /dev/video0 opened.
Device `vcap_mipi_csi2_rx_v_proc_ss_sca' on `platform:vcap_mipi_csi2_rx_v_pr' is a video output (without mplanes) device.
Video format set: NV12 (3231564e) 3840x2160 field none, 1 planes:
 * Stride 3840, buffer size 12441600
Video format: NV12 (3231564e) 3840x2160 field none, 1 planes:
 * Stride 3840, buffer size 12441600
3 buffers requested.
length: 1 offset: 3240556896 timestamp type/source: mono/EoF
Buffer 0/0 mapped at address 0xffff9a749000.
length: 1 offset: 3240556896 timestamp type/source: mono/EoF
Buffer 1/0 mapped at address 0xffff99b6b000.
length: 1 offset: 3240556896 timestamp type/source: mono/EoF
Buffer 2/0 mapped at address 0xffff98f8d000.
0 (0) [-] none 0 0 B 772.696291 772.696303 19.207 fps ts mono/EoF
1 (1) [-] none 1 0 B 772.712954 772.712964 60.013 fps ts mono/EoF
2 (2) [-] none 2 0 B 772.729617 772.729626 60.013 fps ts mono/EoF
3 (0) [-] none 3 0 B 772.746281 772.746290 60.010 fps ts mono/EoF
4 (1) [-] none 4 0 B 772.762946 772.762955 60.006 fps ts mono/EoF
5 (2) [-] none 5 0 B 772.779609 772.779618 60.013 fps ts mono/EoF
6 (0) [-] none 6 0 B 772.796274 772.796283 60.006 fps ts mono/EoF
7 (1) [-] none 7 0 B 772.812938 772.812947 60.010 fps ts mono/EoF
8 (2) [-] none 8 0 B 772.829601 772.959351 60.013 fps ts mono/EoF
9 (0) [-] none 9 0 B 772.846267 773.104969 60.002 fps ts mono/EoF
0 Kudos
karnanl
Xilinx Employee
Xilinx Employee
663 Views
Registered: ‎03-30-2016

Hello @kumakich 
# Just some comments.

When you see mipi_csi2_rx_subsystem reports "Stream Line Buffer Full" :
1. Could you please share MIPI CSI-2 RX Subsystem configuration ?
    ( Please share GUI screenshot or XCI file )
2. What is video_aclk clock frequency ?
    ( Are you following clock guidance mentioned in PG232 Chapter3 "Clocking" section ? )

Kind regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
kumakich
Contributor
Contributor
614 Views
Registered: ‎02-17-2021

Hi @karnanl 

Thank you  for your reply.

The HW design was modified from the 2020.2 TRD multiple streaming one, by adding  only the SYNCIP to MIPI CSI input and VCU encoder.

I believe that nothing has been modified in the part of MIPI.

# Sorry the HW part is done by another memeber of my team, will confirm asap, and show you the GUI configuration.

 

the video_aclk in the pl.dtsi is using 

zynqmp_clk 72

and I cannot find the defination in the whole project.

Where can I find it? please kindly advice.

 

Best,

0 Kudos
kumakich
Contributor
Contributor
589 Views
Registered: ‎02-17-2021

Hi @karnanl 

Sorry, still been requesting hardware desgin from other member.

Just add some additional informaiton:

When running the pipeline, we got the following status of register, according the the PG232 pdf.

0x00060000 18,17Bit LineBufferFull+Stop
0x00020000 17Bit Stop
 
 
 
#when pipeline hangs:
[ 6275.248114] Mcu trace: Channel has hang
root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# devmem 0xA00F0024 32
0x00060000
 
 
#when the pipeline is runnning in background with "&"
root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# devmem 0xA00F0024 32
0x00020000
 
 
And we are looking into the PG232 Chapter3 "Clocking" setting.
Trying to find the difference between TRD-PSDDR_NV12_LLP2 (HDMI input with SYNCIP)
and the design we are working on.
 
The "Pixels per clock" of own current design is 2, and we are modifying it to 4 to try some luck,
 

 

karnanl
Xilinx Employee
Xilinx Employee
567 Views
Registered: ‎03-30-2016

Hello @kumakich 

>When running the pipeline, we got the following status of register, according the the PG232 pdf.
>0x00060000 18,17Bit LineBufferFull+Stop
>0x00020000 17Bit Stop

Bit[17] == "Stop-state" is not an issue. MIPI D-PHY will
Bit[18] == "Stream line buffer full" . You need to find the root-cause why this bit is asserted.


1. Could you please share MIPI CSI-2 RX Subsystem configuration ? ( Please share GUI screenshot or XCI file )
2. What is video_aclk clock frequency ?
   ( I can double check if your video_aclk clock frequency is fine )
3. Could you please read "Packet Count" register a few times to check if the number is increased ?
   ( Register address 0x10 bit[31:16] )


# From (1)(2), If we can confirm video_aclk clock frequency is fine,
   and from(3), if "Packet Count" status is increased
   I believe back-end modules is not ready to receive data , so MIPI CSI-2 RX input TREADY is low.


Kind regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
kumakich
Contributor
Contributor
548 Views
Registered: ‎02-17-2021

Hi @karnanl 

Thank you for your reply!

Here is the screenshot of the MIPI RX subsystem, and the xci file is uploaded.

kumakich_0-1619131577032.png

>3. Could you please read "Packet Count" register a few times to check if the number is increased ?
   ( Register address 0x10 bit[31:16] )

Will get back to you asap.

kumakich
Contributor
Contributor
536 Views
Registered: ‎02-17-2021

Hi @karnanl 

Now we have the 4K30FPS worked.

1.FHD30P can run continously like [run pipeline]->[ctrl+c]->[run pipeline]->[ctrl+c]... without much trouble, #sometimes the pipeline fails, but basically it works by run the pipeline cmd again.

2.4K30P, strangely, the works only if the FHD30P pipeline is successfully  executed once. and the 4K30P pipeline doesnt start the second time. 

It has to be something like FHD30p_Camera_Setting->FHD30p_pipeline_exec->4K30p_Camera_Setting->4K30p_pipeline_exec and loop

3.60fps doesn't show in either FHD or 4K mode.

4.if the framerate of 4K is set to around 10 or 5 FPS, the 4K pipeline can be successfully executed contineously: start->stop->start->stop

5.All the log of pipelines above show

xilinx-csi2rxss a00f0000.mipi_csi2_rx_subsystem: Stream Line Buffer Full

 

 

>3. Could you please read "Packet Count" register a few times to check if the number is increased ?
   ( Register address 0x10 bit[31:16] )

And about the 0x10 register,

When 4K30P pipeline run in background: with &

we check the register from 0x00 to 0x40, the 0x10 is increasing.

./re/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 64, dropped: 4, current: 30.00, average: 24.25
g.sh
Core configuration options (0x00): /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 79, dropped: 4, current: 29.95, average: 25.16
0x00000001
Protocol configuration options (0x04): 0x0000001B
Internal status of the core(0x10): 0x842C0000
Global interrupt enable registers(0x20): 0x00000001
Interrupt status register(0x24): 0x00020000
Interrupt enable register(0x28): 0xD03DFFFF
Short packet data(0x30): 0x00000000
Clock lane status information(0x3C): 0x00000002
Clock lane status information(0x40 44 48 4C): 0x00000020
0x00000020
0x00000020
0x00000000
root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 95, dropped: 4, current: 30.05, average: 25.87
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 110, dropped: 4, current: 29.99, average: 26.36
./re/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 126, dropped: 4, current: 30.01, average: 26.77
g.sh
Core configuration options (0x00): 0x00000001
Protocol configuration options (0x04): 0x0000001B
Internal status of the core(0x10): 0x79360000
Global interrupt enable registers(0x20): 0x00000001
Interrupt status register(0x24): 0x00020000
Interrupt enable register(0x28): 0xD03DFFFF
Short packet data(0x30): 0x00000000
Clock lane status information(0x3C): 0x00000002
Clock lane status information(0x40 44 48 4C): 0x00000020
0x00000020
0x00000000
0x00000000
root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 142, dropped: 4, current: 30.01, average: 27.10
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 158, dropped: 4, current: 30.00, average: 27.37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 174, dropped: 4, current: 30.00, average: 27.59
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 190, dropped: 4, current: 30.00, average: 27.78
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 206, dropped: 4, current: 30.00, average: 27.94
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 222, dropped: 4, current: 30.00, average: 28.08
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 238, dropped: 4, current: 30.00, average: 28.20
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 254, dropped: 4, current: 30.00, average: 28.31
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 269, dropped: 4, current: 29.99, average: 28.40
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 285, dropped: 4, current: 30.00, average: 28.48
./reg.sh /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 301, dropped: 4, current: 30.01, average: 28.56

Core configuration options (0x00): 0x00000001
Protocol configuration options (0x04): 0x0000001B
Internal status of the core(0x10): 0x47580000
Global interrupt enable registers(0x20): 0x00000001
Interrupt status register(0x24): 0x00020000
Interrupt enable register(0x28): 0xD03DFFFF
Short packet data(0x30): 0x00000000
Clock lane status information(0x3C): 0x00000002
Clock lane status information(0x40 44 48 4C): 0x00000020
0x00000020
0x00000020
0x00000020

 

FHD30fps register log:

GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 858, dropped: 7, current: 30.00, average: 29.91
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 874, dropped: 7, current: 30.00, average: 29.91
./GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 890, dropped: 7, current: 30.00, average: 29.91
/reg.sh /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 906, dropped: 7, current: 30.00, average: 29.92

Core configuration options (0x00): 0x00000001
Protocol configuration options (0x04): 0x0000001B
Internal status of the core(0x10): 0x84D00000
Global interrupt enable registers(0x20): 0x00000001
Interrupt status register(0x24): 0x00020000
Interrupt enable register(0x28): 0xD03DFFFF
Short packet data(0x30): 0x00000000
Clock lane status information(0x3C): 0x00000002
Clock lane status information(0x40 44 48 4C): 0x00000020
0x00000020
0x00000020
0x00000020
root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 922, dropped: 7, current: 30.01, average: 29.92
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 937, dropped: 7, current: 30.00, average: 29.92
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 953, dropped: 7, current: 30.00, average: 29.92
./re/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 968, dropped: 7, current: 30.00, average: 29.92
g.sh
Core configuration options (0x00): 0x00000001
Protocol configuration options (0x04): 0x0000001B
Internal status of the core(0x10): 0xBD3D0000
Global interrupt enable registers(0x20): 0x00000001
Interrupt status register(0x24): 0x00020000
Interrupt enable register(0x28): 0xD03DFFFF
Short packet data(0x30): 0x00000000
Clock lane status information(0x3C): 0x00000002
Clock lane status information(0x40 44 48 4C): 0x00000020
0x00000020
0x00000020
0x00000020
root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 983, dropped: 7, current: 29.95, average: 29.92
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 999, dropped: 7, current: 29.98, average: 29.92
./reg.sh /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 1015, dropped: 7, current: 30.05, average: 29.93

Core configuration options (0x00): 0x00000001
Protocol configuration options (0x04): 0x0000001B
Internal status of the core(0x10): 0x33720000
Global interrupt enable registers(0x20): 0x00000001
Interrupt status register(0x24): 0x00020000
Interrupt enable register(0x28): 0xD03DFFFF
Short packet data(0x30): 0x00000000
Clock lane status information(0x3C): 0x00000002
Clock lane status information(0x40 44 48 4C): 0x00000020
0x00000020
0x00000000
0x00000020
root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 1030, dropped: 7, current: 29.97, average: 29.93
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 1046, dropped: 7, current: 29.99, average: 29.93
./re/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 1062, dropped: 7, current: 30.02, average: 29.93
g.sh
Core configuration options (0x00): 0x00000001
Protocol configuration options (0x04): 0x0000001B
Internal status of the core(0x10): 0xE13F0000
Global interrupt enable registers(0x20): 0x00000001
Interrupt status register(0x24): 0x00020000
Interrupt enable register(0x28): 0xD03DFFFF
Short packet data(0x30): 0x00000000
Clock lane status information(0x3C): 0x00000002
Clock lane status information(0x40 44 48 4C): 0x00000020
0x00000020
0x00000020
0x00000020
root@zcu106_vcu_trd:/mnt/sd-mmcblk0p1/shell# /GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 1078, dropped: 7, current: 30.00, average: 29.93
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 1094, dropped: 7, current: 30.00, average: 29.93
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 1110, dropped: 7, current: 30.01, average: 29.93
/GstPipeline:pipeline0/GstFPSDisplaySink:fpssink: last-message = rendered: 1126, dropped: 7, current: 30.00, average: 29.93

 

FYI

the pipeline cmd:

FHD 30P

GST_DEBUG="*v4l2*:6,*omx*:6,*base*:6" GST_DEBUG_FILE=/run/latency.txt \
gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 ! video/x-raw\(memory:XLNXLL\), width=1920, height=1080, framerate=30/1 ! omxh264enc num-slices=8 control-rate=low-latency gop-mode=low-delay-p target-bitrate=25000 cpb-size=500 initial-delay=250 periodicity-idr=240 filler-data=false prefetch-buffer=true ! video/x-h264,profile=main, alignment=nal ! queue max-size-buffers=0 ! omxh264dec low-latency=1 internal-entropy-buffers=5 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 sync=true' sync=true -

 

4K 30P

GST_DEBUG="*v4l2*:6,*omx*:6,*base*:6" GST_DEBUG_FILE=/run/latency.txt \
gst-launch-1.0 -v v4l2src io-mode=dmabuf device=/dev/video0 ! video/x-raw\(memory:XLNXLL\), width=3840, height=2160, framerate=30/1 ! omxh264enc num-slices=8 control-rate=low-latency gop-mode=low-delay-p target-bitrate=25000 cpb-size=500 initial-delay=250 periodicity-idr=240 filler-data=false prefetch-buffer=true ! video/x-h264,profile=main, alignment=nal ! queue max-size-buffers=0 ! omxh264dec low-latency=1 internal-entropy-buffers=5 ! video/x-raw\(memory:XLNXLL\) ! queue max-size-bytes=0 ! fpsdisplaysink name=fpssink text-overlay=false 'video-sink=kmssink bus-id=a0070000.v_mix hold-extra-sample=1 sync=true' sync=true -v

 

Camera/VPSS configuration

FHD30p

media-ctl -v -V '"IMX274 3-001a":0 [fmt:SRGGB8/1920x1080@1/30 field:none colorspace:srgb]' -d /dev/media0;
media-ctl -d /dev/media0 -V "\"a00f0000.mipi_csi2_rx_subsystem\":0 [fmt:SRGGB8_1X8/1920x1080 field:none]";
media-ctl -d /dev/media0 -V "\"a00f0000.mipi_csi2_rx_subsystem\":1 [fmt:SRGGB8_1X8/1920x1080 field:none]";
#Demosaic IP
media-ctl -d /dev/media0 -V "\"a0250000.v_demosaic\":0 [fmt:SRGGB8_1X8/1920x1080 field:none]";
media-ctl -d /dev/media0 -V "\"a0250000.v_demosaic\":1 [fmt:RBG888_1X24/1920x1080 field:none]";
#Gamma LUT IP
media-ctl -d /dev/media0 -V "\"a0270000.v_gamma_lut\":0 [fmt:RBG888_1X24/1920x1080 field:none]";
media-ctl -d /dev/media0 -V "\"a0270000.v_gamma_lut\":1 [fmt:RBG888_1X24/1920x1080 field:none]";
#VPSS: Color Space Conversion (CSC) Only
media-ctl -d /dev/media0 -V "\"a0240000.v_proc_ss\":0 [fmt:RBG888_1X24/1920x1080 field:none]";
media-ctl -d /dev/media0 -V "\"a0240000.v_proc_ss\":1 [fmt:RBG888_1X24/1920x1080 field:none]";
#VPSS: Scaler Only with CSC
media-ctl -d /dev/media0 -V "\"a0200000.v_proc_ss\":0 [fmt:RBG888_1X24/1920x1080 field:none]";
media-ctl -d /dev/media0 -V "\"a0200000.v_proc_ss\":1 [fmt:VYYUYY8_1X24/1920x1080 field:none]";

modetest -D a0070000.v_mix -s 45:3840x2160-30.00@BG24;
modetest -D a0070000.v_mix -s 45:3840x2160-60.00@BG24;

4K30p

yavta -w '0x0098c981 4' /dev/v4l-subdev1;
yavta -w '0x0098c981 4' /dev/v4l-subdev2;

media-ctl -v -V '"IMX274 3-001a":0 [fmt:SRGGB8/3840x2160@1/30 field:none colorspace:srgb]' -d /dev/media0;
media-ctl -d /dev/media0 -V "\"a00f0000.mipi_csi2_rx_subsystem\":0 [fmt:SRGGB8_1X8/3840x2160 field:none]";
media-ctl -d /dev/media0 -V "\"a00f0000.mipi_csi2_rx_subsystem\":1 [fmt:SRGGB8_1X8/3840x2160 field:none]";

#Demosaic IP
media-ctl -d /dev/media0 -V "\"a0250000.v_demosaic\":0 [fmt:SRGGB8_1X8/3840x2160 field:none]";
media-ctl -d /dev/media0 -V "\"a0250000.v_demosaic\":1 [fmt:RBG888_1X24/3840x2160 field:none]";

#Gamma LUT IP
media-ctl -d /dev/media0 -V "\"a0270000.v_gamma_lut\":0 [fmt:RBG888_1X24/3840x2160 field:none]";
media-ctl -d /dev/media0 -V "\"a0270000.v_gamma_lut\":1 [fmt:RBG888_1X24/3840x2160 field:none]";

#VPSS: Color Space Conversion (CSC) Only
media-ctl -d /dev/media0 -V "\"a0240000.v_proc_ss\":0 [fmt:RBG888_1X24/3840x2160 field:none]";
media-ctl -d /dev/media0 -V "\"a0240000.v_proc_ss\":1 [fmt:RBG888_1X24/3840x2160 field:none]";

#VPSS: Scaler Only with CSC
media-ctl -d /dev/media0 -V "\"a0200000.v_proc_ss\":0 [fmt:RBG888_1X24/3840x2160 field:none]";
media-ctl -d /dev/media0 -V "\"a0200000.v_proc_ss\":1 [fmt:VYYUYY8_1X24/3840x2160 field:none]";

modetest -D a0070000.v_mix -s 45:3840x2160-30.00@BG24;
modetest -D a0070000.v_mix -s 45:3840x2160-60.00@BG24;

 

0 Kudos
karnanl
Xilinx Employee
Xilinx Employee
519 Views
Registered: ‎03-30-2016

Hello @kumakich 

I can see that ISR (0x24) does not show any error. (other that Buffer line full)
Core Status Register (0x10) also shows that Packet Count is increasing (0xBD3D0000->0x33720000->0xE13F0000),
So I don't see any issue here.

Thanks for sharing. Your MIPI CSI-2 RX SS IP configuration.
    Line-rate = 1440 Mbps
    Lane = 4
    Pixel Format = RAW10
    Pixel-Per-Clock = 2
Minimum clock frequency of video_aclk = 1440 x 4 / (RAW10*2)=288MHz is required for your IP configuration.

Could you please confirm if your video_aclk clock frequency is 288MHz or higher ?

Kind regards
Leo

PG232_video_aclk_calc.png


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
0 Kudos
kumakich
Contributor
Contributor
512 Views
Registered: ‎02-17-2021

Hi @karnanl 
Thank you for prompt reply.

Yes we can confirm the frequency is

299.970032MHz
So it should be fine right?
Should I look into anything elese?
 
Thank you for your help in advance.
0 Kudos
karnanl
Xilinx Employee
Xilinx Employee
432 Views
Registered: ‎03-30-2016

Hello @kumakich 

Yes, I do not see any issue related with MIPI CSI-2 RX IP.
video_aclk==299.970032MHz  should be fine.
MIPI CSI-2 RX buffer full probably caused by back pressure from other module in the pipeline.
( I can see that in your HW is working at FHD@30FPS  if low latency mode is not enabled )


Let me share this result with internal teams. Please wait for feedback.
BTW, would you able to narrow down this issue more,
Probably by implementing Encoder or Decoder separately ?


Kind regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
kumakich
Contributor
Contributor
417 Views
Registered: ‎02-17-2021

Hi@karnanl 

Thank you for the reply.

>Let me share this result with internal teams. Please wait for feedback.

Thank you for all your help

 

>Probably by implementing Encoder or Decoder separately ?

Surely will confirm this.

I believe before, we have tried source->encode->fakesink pipeline, so may be the problem in around the input/encode part.

Anyway, I will also check the decode part.

Best,

 

0 Kudos
kumakich
Contributor
Contributor
302 Views
Registered: ‎02-17-2021

@karnanl 

Just a little bit update:

We managed to force the MIPI IP Core pipeline not to stop by using the method below and make the mipi working with SYNCIP in llp2 now.

in this link:https://forums.xilinx.com/t5/Video-and-Audio/Re-Four-MIPI-CSI-2-Rx-IPs-error-in-implementation/m-p/977744 

Amazing, the solution is actually from you again!

 

In my understanding, the mipi somehow got line buffer full and the pipeline is stopped by

1.stop the response to IRQ

2.stop the MIPI IP core

and there suppose to be a restart of the IPcore I guess, but some how it can't restart.

Maybe it's due to the unchanging state of line buffer full?

 

And the root cause is the one you descibed:

 

It will happen because your MIPI instances are enabled, but since the back-end is disabled (cannot accept any video data).
video_out_tready is set to low, so buffer inside MIPI CSI-2 RX IP will be full. 

 

 

Anyway, we are happy that we can now get the latency number as the document PG252.pdf

Xilinx Low-Latency
Capture:1 ms 
Encode:10 ms
Decode:9 ms
karnanl
Xilinx Employee
Xilinx Employee
274 Views
Registered: ‎03-30-2016

Hello @kumakich 

Thank you for updating this thread.

First, I am not sure why MIPI CSI-2 RX report line buffer full at the first place.
But if downstream modules cannot accept any video data, while MIPI IP is enabled, MIPI CSI-2 RX IP will report buffer full.
If MIPI CSI-2 RX IP report buffer full, MIPI core will be disabled and we do recommend to do hard IP reset.

BTW, I am glad your design is working now.
So, I believe you only comment out xcsi2rxss_stop_stream() in xcsi2rxss_s_stream() in the xilinx-csi2rxss.c driver ?!
If yes, then I should give our driver team a feedback on this.

Kind regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
kumakich
Contributor
Contributor
263 Views
Registered: ‎02-17-2021

Hi @karnanl 

Thank you for your advice!

>But if downstream modules cannot accept any video data, while MIPI IP is enabled, MIPI CSI-2 RX IP will report buffer full.

Understood, will research on the downtream module, the decoder on the full pipeline or the ddr on the encoder->fakesink pipeline.

 

>So, I believe you only comment out xcsi2rxss_stop_stream() in xcsi2rxss_s_stream() in the xilinx-csi2rxss.c driver ?!

Actually as we have been modifying the TRD-multiple stream design, the source code branch is 2020.2 release?

the following xcsi2rxss_enable(&xcsi2rxss->core, false); was commented out.

 

static void xcsi2rxss_stop_stream(struct xcsi2rxss_state *xcsi2rxss)
{
	xcsi2rxss_interrupts_enable(&xcsi2rxss->core, false);
	//xcsi2rxss_enable(&xcsi2rxss->core, false);
}
#from 
https://github.com/Xilinx/linux-xlnx/blob/release-2020.2.2_k26/drivers/media/platform/xilinx/xilinx-csi2rxss.c

 

And I saw the master branch has been modified into the code as blow:

Since we didn't use the master branch, not so sure about the difference.

 

static void xcsi2rxss_stop_stream(struct xcsi2rxss_state *state)
{
	v4l2_subdev_call(state->rsubdev, video, s_stream, 0);

	/* disable interrupts */
	xcsi2rxss_clr(state, XCSI_IER_OFFSET, XCSI_IER_INTR_MASK);
	xcsi2rxss_clr(state, XCSI_GIER_OFFSET, XCSI_GIER_GIE);

	/* disable core */
	xcsi2rxss_clr(state, XCSI_CCR_OFFSET, XCSI_CCR_ENABLE);
	state->streaming = false;
}
#from
https://github.com/Xilinx/linux-xlnx/blob/master/drivers/media/platform/xilinx/xilinx-csi2rxss.c

 

 

Best,

karnanl
Xilinx Employee
Xilinx Employee
195 Views
Registered: ‎03-30-2016

Hello @kumakich 

>Actually as we have been modifying the TRD-multiple stream design, the source code branch is 2020.2 release?
>the following xcsi2rxss_enable(&xcsi2rxss->core, false); was commented out.

I've discussed this with our driver experts today.
This will be addressed in 2021.1 driver.
In the new driver, we are leaving it to application to call stream off which will reset the core unlike in previous driver where the core was reset in IRQ handler.

# Please note that in 2021.1 we are having new upstream driver (means that it is from Linux mainline kernel).

Kind regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------