I2C device causing kernel hang on v2020.1

[bhig]

Hello,

We have a project that has been working well for awhile on v2019.1, however we are now attempting to migrate it to the 2020.1 suite and are running into problems with the kernel hanging at startup.  We've traced the issue to the two built-in I2C devices.  For some reason, when they are enabled in the device tree, the kernel hangs at startup, if we set them both to "disabled" then the kernel will boot.

What is really strange to us, is that the v2020.1 kernel and root file system will boot with the I2C devices present if we use our 2019.1 generated BOOT.bin.

We've tried the following combinations:

FSBL, ATF, PMUFW, U-Boot	FPGA bitstream	Kernel, RootFS	Result
2019.1	2020.1	2020.1	Boots
2020.1	2020.1	2020.1	Hangs during I2C probing at boot
2020.1	2020.1	2020.1 modified w/ I2C devices set to "disabled" in DT	Boots with errors in our applications due to lack of I2C devices

We also have 1 instance of an AXI IIC controller in the PL, which has no problem, although we did try disabling that as well to see if there was some conflict but it made no difference.  It doesn't matter if we enable i2c0 or i2c1, either one causes the system to hang.

One interesting difference is the output of the PM framework is different between the 2019 and 2020 kernels.

on 2019:
# echo pm_get_node_status 37 > /sys/kernel/debug/zynqmp-firmware/pm
# cat /sys/kernel/debug/zynqmp-firmware/pm
GET_NODE_STATUS:
        NodeId: 37
        Status: 1
        Requirements: 1
        Usage: 1
# echo pm_get_node_status 38 > /sys/kernel/debug/zynqmp-firmware/pm
# cat /sys/kernel/debug/zynqmp-firmware/pm
GET_NODE_STATUS:
        NodeId: 38
        Status: 1
        Requirements: 1
        Usage: 1

But on 2020:

# echo pm_get_node_status 37 > /sys/kernel/debug/zynqmp-firmware/pm
# cat /sys/kernel/debug/zynqmp-firmware/pm
GET_NODE_STATUS:
        NodeId: 37
        Status: 1
        Requirements: 3
        Usage: 3
# echo pm_get_node_status 38 > /sys/kernel/debug/zynqmp-firmware/pm
# cat /sys/kernel/debug/zynqmp-firmware/pm
GET_NODE_STATUS:
        NodeId: 38
        Status: 1
        Requirements: 3
        Usage: 3

Also, we know the physical I2C is working fine, since we write out some configurations to devices on both busses from U-Boot which succeeds in all versions.

Attached are boot logs from each of the cases described in the table.

Any help or places to look next would be greatly appreciated. We're tempted to just continue using the 2019.1 boot files with our 2020.1 FPGA and kernel image but mixing versions seems like the wrong approach and we don't want to run into latent problems down the road.

Thanks!

[maps-mpls]

You might want to take a look at your device-tree.bbappend, system-user.dtsi, and if called out in device-tree.bbappend, any other files such as pl-custom.dtsi.  These are all in:

project-spec/meta-user/recipes-bsp/device/tree/
project-spec/meta-user/recipes-bsp/device/tree/files/

Compare against

components/plnx_workspace/device-tree/device-tree/pl.dtsi
components/plnx_workspace/device-tree/device-tree/zynqmp.dtsi

Also if you're using a devCard, look at the dtsi for its bsp.

This is what I do, but I am doing custom hardware on a Xilinx devCard.  Everything works for 2020.1 for me.  There is more than one path through the jungle, though, so there may be other better ways.

[bhig]

Thanks @maps-mpls, we're actually using the OSD flow and building our petalinux kernel and rootfs through build root instead of yocto/petalinux tools.  However we compared all of our device tree files against the zynqmp-zcu106-revA.dts file present as part of the kernel build.  We're actually using the Avnet UltraZed-EV SOM with a custom carrier card and have also compared our DTS to the premade device tree file from the Avnet board.  

The strange part, is we tested removing ALL I2C devices and just leaving an empty device node such as the below and it still causes the boot to hang.  We're just including the standard zynqmp.dtsi and have a custom pl.dtsi to match our custom FPGA image.  If we switch both of the below nodes to "status = "disabled"" the system will boot.  And our U-Boot device tree has both of them enabled in order to write config changes over I2C and there is no issue.

    &i2c0 {
        status = "okay";
        clock-frequency = <400000>;
    };
    &i2c1 {
        status = "okay";
        clock-frequency = <400000>;
    };

[bhig]

I should also note, the exact same device tree file works perfectly fine with our v2019.1 kernel image.

[maps-mpls]

I haven't had any problems, but you might want to take a close look.  For the ZCU104, for example, the following two .dtsi files are different for different versions of Petalinux/Vivado despite both being named zcu104-revc.dtsi

https://github.com/Xilinx/device-tree-xlnx/blob/master/device_tree/data/kernel_dtsi/2020.2/BOARD/zcu104-revc.dtsi

https://github.com/Xilinx/device-tree-xlnx/blob/master/device_tree/data/kernel_dtsi/2020.1/BOARD/zcu104-revc.dtsi 

Though both are named zcu104-revc.dtsi.  So check the official .dtsi files for each revision...it is possible there is a clue there.  I'm not sure if for example some of the label changes need to match up with some of the #defines or other definitions in the build.  It could be a red herring, but it might bear fruit.

[bhig]

So by subbing out each boot component with their 2019.1 version that works, I've narrowed it down to an FSBL issue.

With 2019.1 FSBL, and the rest as all 2020.1 components, the system boots with I2C devices enabled in the kernel DTS.

With 2020.1 FSBL, and and a known working 2019.1 kernel, the system hangs on I2C access. 

So something that the 2020.1 FSBL is doing is causing the memory at 0xFF030000 not to be initialized once the kernel runs, but is working in U-Boot.  One difference is when U-Boot runs and accesses the I2C device, the bitstream is not programmed yet so I'll be looking to see if that's causing an issue and what other changes are present between the 2019 and 2020 FSBL images.

I had added some debug code to the driver and observed that it hangs when trying to access the controller memory:

[    6.827931] cdns-i2c ff030000.i2c: Entered I2C Probe.
[    6.832976] cdns-i2c ff030000.i2c: After Set drvdata
[    6.837930] cdns-i2c ff030000.i2c: After of match node
[    6.843062] cdns-i2c ff030000.i2c: After pinctl
[    6.847582] cdns-i2c ff030000.i2c: resource r_mem-start 0xff030000 and end 0xff030fff
[    6.855413] cdns-i2c ff030000.i2c: io remapped to address 0x11205000
[    6.861755] cdns-i2c ff030000.i2c: After ioremap
[    6.866372] cdns-i2c ff030000.i2c: After irq retrieve
[    6.871418] cdns-i2c ff030000.i2c: After init completion
[    6.876746] cdns-i2c ff030000.i2c: before pm framework calls
[    6.882399] cdns-i2c ff030000.i2c: after pm framework calls
[    6.886748] ata1: SATA link down (SStatus 0 SControl 320)
[    6.887973] cdns-i2c ff030000.i2c: before of clock read
[    6.898561] cdns-i2c ff030000.i2c: before set clk
[    6.903255] cdns-i2c ff030000.i2c: before req irq
[    6.907972] cdns-i2c ff030000.i2c: before cdns i2c init
[    6.913196] cdns-i2c ff030000.i2c: writing 0xb0e to register 0x11205000
<< hang >>

[maps-mpls]

Oh, are you using the I2C in the PS or the PL (some devCards have both options open).  You might be on to something there.

[bhig]

In our system we use both I2C peripherals in the PS and also instantiate one AXI I2C in the PL. The PL I2C has no problems, just the PS side devices have an issue.

It's really weird, its almost as if something is disabling the I2C controllers during the Linux boot, such that if I do even a simple "devmem 0xff030000" to read out the control register, the entire system hangs.

With my 2019 image, I can replicate this by setting the DTS setting to "disabled" and then after booting trying to access devmem.  However, in my 2020 image it's being set to "okay" so whatever is supposed to be enabling that peripheral is clearly not when run with the 2020.1 FSBL. 

[maps-mpls]

Are you loading the bitstream during the FSBL?  Did you change any sequencing there?  Maybe set the PL I2C to disabled in your dtsi.

I've heard about people having trouble with the PS I2C controller, but I have not had any.