Using RFdc driver in Linux - Failed to open device

[rgebauer]

I tried using the RFdc driver in a Yocto tool flow (and Vivado/SDK 2018.3) with a Linux userspace application looking like:

#include "xrfdc.h"

#define RFDC_DEVICE_ID 	0

void rfdc_test()
{
    struct metal_init_params init_param = METAL_INIT_DEFAULTS;
    XRFdc_Config *ConfigPtr;

    if (metal_init(&init_param)) {
        printf("XRFdc metal_init");
    }
    else{
        printf("XRFdc metal_init not initialized.");
        return;
    }

    ConfigPtr = XRFdc_LookupConfig(RFDC_DEVICE_ID);
    if (ConfigPtr==NULL){
        printf("ConfigPtr is NULL!");
        return;
    }
    else{
        printf("ConfigPtr is initialized.");
    }
}

using the selftest as example where the commands originate from:

https://github.com/Xilinx/embeddedsw/blob/release-2018.3/XilinxProcessorIPLib/drivers/rfdc/examples/xrfdc_selftest_example.c

Unfortunately, this results in the following error message:

XRFdc metal_init
metal: error:
 Failed to open device a0000000.usp_rf_data_convertermetal: error:
Invalid device id 0ConfigPtr is NULL!

Any ideas how to resolve this issue? It says invalid device ID but as far as I understood the `0` is intended if not run bare metal?

In `/usr/lib` files for `librfdc` and `libmetal` exist and the rfdc Yocto recipe is also a dependency of the userspace driver (builds fine):

It also seems like the device tree from the hardware project is transferred correctly, as there is a character device located in `/sys/devices/platform/amba_pl@0/a0000000.usp_rf_data_converter` with `modalias`:

of:Nusp_rf_data_converterT<NULL>Cxlnx,usp-rf-data-converter-2.1

Not sure if this <NULL> is intended there?

If you need more information, please let me know. Any help is highly appreciated!

[rgebauer]

There was a little mistake in the check for the metal_init() swapping error and normal case... so metal_init() already fails with a ENODEV (status -19).

I guess this happens here:

https://github.com/OpenAMP/libmetal/blob/v2018.10/lib/system/linux/device.c#L629

when trying to register the bus - which seems to fail. (There is a debug log output for the page size but none for any registered bus.)

Any idea why this could be failing?

[rgebauer]

So I am a litte bit further right now, but it is still failing

1) The VFIO/UIO kernel modules have not been builtin the kernel

Adding them to the kernel config helped (append them in Yocto):

Spoiler

#
# Device Drivers > DMABUF options
#
CONFIG_SYNC_FILE=y
# CONFIG_SW_SYNC is not set
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=y
# CONFIG_UIO_CIF is not set
CONFIG_UIO_PDRV_GENIRQ=y
CONFIG_UIO_DMEM_GENIRQ=y
# CONFIG_UIO_AEC is not set
# CONFIG_UIO_SERCOS3 is not set
# CONFIG_UIO_PCI_GENERIC is not set
# CONFIG_UIO_NETX is not set
# CONFIG_UIO_PRUSS is not set
# CONFIG_UIO_MF624 is not set
CONFIG_UIO_XILINX_APM=y
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO_VIRQFD=y
CONFIG_VFIO=y
# CONFIG_VFIO_NOIOMMU is not set
# CONFIG_VFIO_PCI is not set
CONFIG_VFIO_PLATFORM=y
CONFIG_VFIO_AMBA=y
# CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET is not set
# CONFIG_VFIO_PLATFORM_AMDXGBE_RESET is not set
# CONFIG_VFIO_MDEV is not set
# CONFIG_VIRT_DRIVERS is not set
CONFIG_VIRTIO=y

2) Then the following error occured when trying to lookup the config:

metal_irq_unregister: irq -1 is larger than the max supported 1024.

So I thought maybe I need to connect the interrupt signal of the RFdc block to the PS... so I did.

3) Now there additionally is another error - somehow the RFdc ist not detected as UIO compatible?

metal: debug:     opened sysfs device platform:a0000000.usp_rf_data_converter
metal: error:     failed to scan class path /sys/devices/platform/amba_pl@0/a0000000.usp_rf_data_converter/uio
metal: error:     metal_irq_unregister: irq -1 is larger than the max supported 1024.
metal: error:     
 Failed to open device a0000000.usp_rf_data_convertermetal: error:     
 Invalid device id 0

I also tried adding

&rfdc {
        compatible="generic-uio";
};

to the device tree, and while it is taken into account, it doesn't change anything unfortunately.

Any idea what I am missing here?

[patocarr]

Hi @rgebauer 

  How's your usp_rf_data_converter node in device tree looking like? In my case:

		usp_rf_data_converter@a0080000 {
			clock-names = "s_axi_aclk\0m0_axis_aclk\0m1_axis_aclk\0m2_axis_aclk\0m3_axis_aclk\0s0_axis_aclk\0s1_axis_aclk\0s2_axis_aclk\0s3_axis_aclk";
			clocks = <0x18 0x1a 0x00 0x1a 0x00 0x1a 0x00 0x1a 0x00 0x1b 0x00 0x1b 0x00 0x1b 0x00 0x1b 0x00>;
			compatible = "xlnx,usp-rf-data-converter-2.3";
			interrupt-names = "irq";
			interrupt-parent = <0x04>;
			interrupts = <0x00 0x5a 0x04>;
			num-insts = <0x01>;
			param-list = <0x00 0x8a0 0x00 0x00 0x00 0x00 0x1000000 0x1000000 0x1000000 0x1000000 0x1000000 0x1000000 0x9a999999 0x99990940 0x00 0x7940 0x00 0x6940 0x18000000 0x3000000 0x1000000 0x00 0x9eefa7c6 0x4b371a40 0x4000000 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x1000000 0x1000000 0x9a999999 0x99990940 0x00 0x7940 0x00 0x6940 0x18000000 0x3000000 0x1000000 0x00 0x9eefa7c6 0x4b371a40 0x4000000 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x1000000 0x1000000 0x9a999999 0x99990940 0x00 0x7940 0x00 0x6940 0x18000000 0x3000000 0x1000000 0x00 0x9eefa7c6 0x4b371a40 0x4000000 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x1000000 0x1000000 0x9a999999 0x99990940 0x00 0x7940 0x00 0x6940 0x18000000 0x3000000 0x1000000 0x00 0x9eefa7c6 0x4b371a40 0x4000000 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x1000000 0x00 0x2000000 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x00 0x10000000 0x1000000 0x00 0x00 0x00 0x1000000 0x1000000 0x9a999999 0x9999f93f 0x00 0x7940 0x00 0x6940 0x18000000 0x6000000 0x1000000 0x00 0xc3f5285c 0x8fc20140 0x4000000 0x1000000 0x2000000 0x1000000 0x2000000 0x1000000 0x2000000 0x1000000 0x2000000 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x1000000 0x1000000 0x9a999999 0x9999f93f 0x00 0x7940 0x00 0x6940 0x18000000 0x6000000 0x1000000 0x00 0xc3f5285c 0x8fc20140 0x4000000 0x1000000 0x2000000 0x1000000 0x2000000 0x1000000 0x2000000 0x1000000 0x2000000 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x1000000 0x1000000 0x9a999999 0x9999f93f 0x00 0x7940 0x00 0x6940 0x18000000 0x6000000 0x1000000 0x00 0xc3f5285c 0x8fc20140 0x4000000 0x1000000 0x2000000 0x1000000 0x2000000 0x1000000 0x2000000 0x1000000 0x2000000 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x1000000 0x1000000 0x9a999999 0x9999f93f 0x00 0x7940 0x00 0x6940 0x18000000 0x6000000 0x1000000 0x00 0xc3f5285c 0x8fc20140 0x4000000 0x1000000 0x2000000 0x1000000 0x2000000 0x1000000 0x2000000 0x1000000 0x2000000 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00 0x00 0x8000000 0x1000000 0x00 0x00>;
			reg = <0x00 0xa0080000 0x00 0x40000>;
			phandle = <0x84>;
		};

Thanks,

-Pat

[rgebauer]

Hi @patocarr 

Thank you very much for your response!

The node in the device tree as taken from

yocto_dir/build/tmp/work/zcu111_zynqmp-xilinx-linux/device-tree/xilinx+gitAUTOINC+b7466bbeee-r0/build/system-top.dts

looks as follows:

                usp_rf_data_converter@a0000000 {
                        clock-names = "s_axi_aclk", "m2_axis_aclk", "m3_axis_aclk", "s0_axis_aclk";
                        clocks = <0x3 0x47 0x49 0x49 0x49>;
                        compatible = "xlnx,usp-rf-data-converter-2.1";
                        interrupt-names = "irq";
                        interrupt-parent = <0x4>;
                        interrupts = <0x0 0x59 0x4>;
                        num-insts = <0x1>;
                        param-list = <0x0 0xa0 0x0 0x1000000 0xe4000000 0xe4000000 0x1000000 0x1000000 0x1000000 0x0 0x0 0x1040 0x0 0x40af40 0x0 0x406f40 0xa000000 0x2000000 0x1000000 0x0 0x1000000 0x0 0x2000000 $
                        reg = <0x0 0xa0000000 0x0 0x40000>;
                };

It looks quite similar to yours... just phandle is missing, the param-list differs a bit and the second number in interrupts is slightly different.

(This is now the version without the additional compatible="generic-uio" device tree overlay I described above)

Do you see any major difference here that could explain the behavior?

[patocarr]

@rgebauer 

  The differences seem understandable given the different configurations. The interrupt difference may simply indicate a different interrupt net going into the GIC.

  I usually generate the .dts from the end .dtb output with the following command below.

$ cd images/linux
$ dtc -I dtb -O dts -o system.dts system.dtb

   Not familiar with the path from Yocto you've attached. I assume the phandler would be added at a later stage of the build, otherwise that'd be a problem.

Thanks,

-Pat

[rgebauer]

Thanks @patocarr !

Decompiling the dtb results in exactly the same node as I posted above.

So it seems the phandler is somehow missing - do you know why this could be the case?

What is the meaning of it?

[patocarr]

@rgebauer 

  It may be nothing. Looking at 2019.2 dtb, some nodes don't have the "phandle" property, but on 2020.1 most nodes do.

Thanks,

-Pat

[rgebauer]

Ok @patocarr 

Any other idea where I can look at or what I can try in order to find out where the error originates from?

It would be awesome to find a solution for this.. I already did some trial & error testing and tried a bunch of different things the last few days but nothing helped so far...

Thanks again for your time, Pat!

EDIT: I am using the 2018.3 version

[rgebauer]

@klumsde @jovitac You helped with similar issues before - maybe you have an idea what could be the problem here?

Thank you in advance!

If you need more information, please let me know.

[stephenm]

Since the rfdc uses libmetal. So, the same code should work on baremetal and Linux. In baremetal, the IP is configured usig the info in the xparameters.h file. However, for linux this is passed via the param-list in the DT node

This is covered here:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/115966024/How+configuration+data+gets+passed+to+RFDC+driver+in+Baremetal+and+Linux+in+2018.3

Can you share your XSA file?

[rgebauer]

Hi @stephenm 

Thanks for your reply!

I cannot share the hardware file right away, but I can try to create a MWE for this.

I also tested some further and found out that the kernel config line

CONFIG_UIO_PDRV_GENIRQ=y

is apparently not taken into account during kernel built (or reset somehow to =m but I cannot load it manually it seems). Can this be an issue?

I also followed this udev debug guide : https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/96960682/Linux+Loadable+Kernel+Modules#LinuxLoadableKernelModules-UdevDebug

to simulate a kernel boot with debug output. The following two messages related to the RFdc appeared:

KERNEL[1224.177244] add      /devices/platform/amba_pl@0/a0000000.usp_rf_data_converter (platform)
UDEV  [1224.232879] add      /devices/platform/amba_pl@0/a0000000.usp_rf_data_converter (platform)

For other UIO devices there were also paths to ./uio subdirectories, e.g.:

UDEV  [1224.225885] add      /devices/platform/amba/ffa00000.perf-monitor (platform)
UDEV  [1224.226302] add      /devices/platform/amba/ffa00000.perf-monitor/uio/uio0 (uio)

 As the error says it cannot open the "/uio" path of the rfdc module, maybe this is also part of the problem?

[stephenm]

Can you share the rfdc DT node?

[rgebauer]

Hi @stephenm 

I shared it above, but sure, here it is again:

                usp_rf_data_converter@a0000000 {
                        clock-names = "s_axi_aclk", "m2_axis_aclk", "m3_axis_aclk", "s0_axis_aclk";
                        clocks = <0x3 0x47 0x49 0x49 0x49>;
                        compatible = "xlnx,usp-rf-data-converter-2.1";
                        interrupt-names = "irq";
                        interrupt-parent = <0x4>;
                        interrupts = <0x0 0x59 0x4>;
                        num-insts = <0x1>;
                        param-list = <0x0 0xa0 0x0 0x1000000 0xe4000000 0xe4000000 0x1000000 0x1000000 0x1000000 0x0 0x0 0x1040 0x0 0x40af40 0x0 0x406f40 0xa000000 0x2000000 0x1000000 0x0 0x1000000 0x0 0x2000000 $
                        reg = <0x0 0xa0000000 0x0 0x40000>;
                };

 Or did you mean something different?

Thank you!

[stephenm]

I just seen you where using a uio driver. If you do this, then the rfdc driver wont be used. How did you generate the DT node, did you use Petalinux, or Yocto.

This wil use the devicetree generator to generate the rfdc node based on the config in the XSA file.

In your error:

metal: debug:     opened sysfs device platform:a0000000.usp_rf_data_converter
metal: error:     failed to scan class path /sys/devices/platform/amba_pl@0/a0000000.usp_rf_data_converter/uio
metal: error:     metal_irq_unregister: irq -1 is larger than the max supported 1024.
metal: error:    
Failed to open device a0000000.usp_rf_data_convertermetal: error:    
Invalid device id 0

The failing line is in the driver is, everything after that doesn't really matter

https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/rfdc/src/xrfdc_sinit.c#L134

On the Linux console, can you do

• cat /sys/bus/platform/devices/

Do you see the device here?

[rgebauer]

Hi @stephenm 

---

I just seen you where using a uio driver. If you do this, then the rfdc driver wont be used.

---

I just use it as it told me UIO/VFIO needs to be present... should I use a different one? If so, how to do this?

---

How did you generate the DT node, did you use Petalinux, or Yocto.

---

I use Yocto to build the image. The device tree should result out of the HDF file from Vivado (I am still using v2018.3).

---

The failing line is in the driver is, everything after that doesn't really matter

https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/rfdc/src/xrfdc_sinit.c#L134

---

Inside the method, this line fails because it cannot open the uio node in the device within libmetal (doesn't exist):

https://github.com/OpenAMP/libmetal/blob/master/lib/system/linux/device.c#L179

---

On the Linux console, can you do

• cat /sys/bus/platform/devices/

Do you see the device here?

---

Yes, it is there:

root@zcu111-zynqmp:/# ls /sys/bus/platform/devices/ | grep usp_rf
a0000000.usp_rf_data_converter

and the content looks like this:

root@zcu111-zynqmp:/sys/devices/platform/amba_pl@0/a0000000.usp_rf_data_converter# ls
driver           modalias         power            uevent
driver_override  of_node          subsystem

There is no "uio" directory to open and therefore, the libmetal command complains and fails...

Do you know how to fix this?

[rgebauer]

---

@stephenm wrote:

Can you share your XSA file?

---

I reduced my hardware design to a minimal working example (which basically is just the RFdc block):

The HDF file (I am using Vivado 2018.3) can be downloaded until October 12 using this link:

https://gigamove.rz.rwth-aachen.de/d/id/smbFKVKapYGPuF

It still results in the same error as before.

Does this help narrowing down where the problem originates from?

[rgebauer · Accepted solution]

So I finally figured it out - here the summary so it might be helpful for someone facing similar issues:

0.) Make sure that the code you're testing with is correct. It is also helpful to turn on the metal debug log mode to get further insights:

struct metal_init_params init_param = {.log_handler = metal_default_log_handler, .log_level = METAL_LOG_DEBUG};
[...]
metal_init(&init_param);

1.) In order to use use the RFdc driver with libmetal, you need to make the UIO and VFIO kernel modules built-in.

I included them using

bitbake -c menuconfig linux-xlnx

and then copied the relevant part of the generated ".config" file in a Yocto recipe so it is included in future builds as well.

My code snippet of the config looks like this:

#
# DMABUF options
#
CONFIG_SYNC_FILE=y
# CONFIG_SW_SYNC is not set
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=y
# CONFIG_UIO_CIF is not set
CONFIG_UIO_PDRV_GENIRQ=y
CONFIG_UIO_DMEM_GENIRQ=y
# CONFIG_UIO_AEC is not set
# CONFIG_UIO_SERCOS3 is not set
CONFIG_UIO_PCI_GENERIC=y
# CONFIG_UIO_NETX is not set
# CONFIG_UIO_PRUSS is not set
# CONFIG_UIO_MF624 is not set
CONFIG_UIO_XILINX_APM=y
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO_VIRQFD=y
CONFIG_VFIO=y
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PLATFORM=y
CONFIG_VFIO_AMBA=y
# CONFIG_VFIO_PLATFORM_CALXEDAXGMAC_RESET is not set
# CONFIG_VFIO_PLATFORM_AMDXGBE_RESET is not set
# CONFIG_VFIO_MDEV is not set
CONFIG_IRQ_BYPASS_MANAGER=y
# CONFIG_VIRT_DRIVERS is not set
CONFIG_VIRTIO=y

2.) Connect the interrupt signal from the RFdc block in your block design to the ZYNQ (not sure if this is really necessary, but it atleast changed the error message and might be helpful anyway depending which functionality you want to have access to...)

(I also added a concat between zynq/pl_ps_irq0 and rfdc/irq as a colleague recommended this - also not sure if this is really necessary, but feel free to try if it helps...)

3.) Make sure that the kernel config you set in your Yocto recipes also end up in the final config.

Here, I already noted that it seems that one line of the config is not taken into account:

CONFIG_UIO_PDRV_GENIRQ=y

I finally found out that this is due to a different Yocto layer (meta-openamp) that sets this config entry to "m" and which has a higher layer priority than my custom layer... As a quick fix, I just changed the file in meta-openamp and now it works...

Thanks to @patocarr and @stephenm for the help and suggestions.