cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
bhig
Contributor
Contributor
588 Views
Registered: ‎09-01-2017

I2C device causing kernel hang on v2020.1

Hello,

We have a project that has been working well for awhile on v2019.1, however we are now attempting to migrate it to the 2020.1 suite and are running into problems with the kernel hanging at startup.  We've traced the issue to the two built-in I2C devices.  For some reason, when they are enabled in the device tree, the kernel hangs at startup, if we set them both to "disabled" then the kernel will boot.

What is really strange to us, is that the v2020.1 kernel and root file system will boot with the I2C devices present if we use our 2019.1 generated BOOT.bin.

We've tried the following combinations:

FSBL, ATF, PMUFW, U-BootFPGA bitstreamKernel, RootFSResult
2019.12020.12020.1Boots
2020.12020.12020.1Hangs during I2C probing at boot
2020.12020.12020.1 modified w/ I2C devices set to "disabled" in DTBoots with errors in our applications due to lack of I2C devices

 

We also have 1 instance of an AXI IIC controller in the PL, which has no problem, although we did try disabling that as well to see if there was some conflict but it made no difference.  It doesn't matter if we enable i2c0 or i2c1, either one causes the system to hang.

One interesting difference is the output of the PM framework is different between the 2019 and 2020 kernels.

on 2019:
# echo pm_get_node_status 37 > /sys/kernel/debug/zynqmp-firmware/pm
# cat /sys/kernel/debug/zynqmp-firmware/pm
GET_NODE_STATUS:
        NodeId: 37
        Status: 1
        Requirements: 1
        Usage: 1
# echo pm_get_node_status 38 > /sys/kernel/debug/zynqmp-firmware/pm
# cat /sys/kernel/debug/zynqmp-firmware/pm
GET_NODE_STATUS:
        NodeId: 38
        Status: 1
        Requirements: 1
        Usage: 1

But on 2020:

# echo pm_get_node_status 37 > /sys/kernel/debug/zynqmp-firmware/pm
# cat /sys/kernel/debug/zynqmp-firmware/pm
GET_NODE_STATUS:
        NodeId: 37
        Status: 1
        Requirements: 3
        Usage: 3
# echo pm_get_node_status 38 > /sys/kernel/debug/zynqmp-firmware/pm
# cat /sys/kernel/debug/zynqmp-firmware/pm
GET_NODE_STATUS:
        NodeId: 38
        Status: 1
        Requirements: 3
        Usage: 3

Also, we know the physical I2C is working fine, since we write out some configurations to devices on both busses from U-Boot which succeeds in all versions.

Attached are boot logs from each of the cases described in the table.

Any help or places to look next would be greatly appreciated. We're tempted to just continue using the 2019.1 boot files with our 2020.1 FPGA and kernel image but mixing versions seems like the wrong approach and we don't want to run into latent problems down the road.

Thanks!

Tags (4)
0 Kudos
8 Replies
maps-mpls
Mentor
Mentor
471 Views
Registered: ‎06-20-2017

You might want to take a look at your device-tree.bbappend, system-user.dtsi, and if called out in device-tree.bbappend, any other files such as pl-custom.dtsi.  These are all in:

 

project-spec/meta-user/recipes-bsp/device/tree/
project-spec/meta-user/recipes-bsp/device/tree/files/

 

 

Compare against

 

 

components/plnx_workspace/device-tree/device-tree/pl.dtsi
components/plnx_workspace/device-tree/device-tree/zynqmp.dtsi

 

 

Also if you're using a devCard, look at the dtsi for its bsp.

This is what I do, but I am doing custom hardware on a Xilinx devCard.  Everything works for 2020.1 for me.  There is more than one path through the jungle, though, so there may be other better ways.

 

*** Destination: Rapid design and development cycles ***
0 Kudos
bhig
Contributor
Contributor
456 Views
Registered: ‎09-01-2017

Thanks @maps-mpls, we're actually using the OSD flow and building our petalinux kernel and rootfs through build root instead of yocto/petalinux tools.  However we compared all of our device tree files against the zynqmp-zcu106-revA.dts file present as part of the kernel build.  We're actually using the Avnet UltraZed-EV SOM with a custom carrier card and have also compared our DTS to the premade device tree file from the Avnet board.  

The strange part, is we tested removing ALL I2C devices and just leaving an empty device node such as the below and it still causes the boot to hang.  We're just including the standard zynqmp.dtsi and have a custom pl.dtsi to match our custom FPGA image.  If we switch both of the below nodes to "status = "disabled"" the system will boot.  And our U-Boot device tree has both of them enabled in order to write config changes over I2C and there is no issue.

    &i2c0 {
        status = "okay";
        clock-frequency = <400000>;
    };
    &i2c1 {
        status = "okay";
        clock-frequency = <400000>;
    };

 

 

0 Kudos
bhig
Contributor
Contributor
454 Views
Registered: ‎09-01-2017

I should also note, the exact same device tree file works perfectly fine with our v2019.1 kernel image.

0 Kudos
maps-mpls
Mentor
Mentor
445 Views
Registered: ‎06-20-2017

I haven't had any problems, but you might want to take a close look.  For the ZCU104, for example, the following two .dtsi files are different for different versions of Petalinux/Vivado despite both being named zcu104-revc.dtsi

 

https://github.com/Xilinx/device-tree-xlnx/blob/master/device_tree/data/kernel_dtsi/2020.2/BOARD/zcu104-revc.dtsi

https://github.com/Xilinx/device-tree-xlnx/blob/master/device_tree/data/kernel_dtsi/2020.1/BOARD/zcu104-revc.dtsi 

Though both are named zcu104-revc.dtsi.  So check the official .dtsi files for each revision...it is possible there is a clue there.  I'm not sure if for example some of the label changes need to match up with some of the #defines or other definitions in the build.  It could be a red herring, but it might bear fruit.

 

*** Destination: Rapid design and development cycles ***
0 Kudos
bhig
Contributor
Contributor
418 Views
Registered: ‎09-01-2017

So by subbing out each boot component with their 2019.1 version that works, I've narrowed it down to an FSBL issue.

With 2019.1 FSBL, and the rest as all 2020.1 components, the system boots with I2C devices enabled in the kernel DTS.

With 2020.1 FSBL, and and a known working 2019.1 kernel, the system hangs on I2C access. 

So something that the 2020.1 FSBL is doing is causing the memory at 0xFF030000 not to be initialized once the kernel runs, but is working in U-Boot.  One difference is when U-Boot runs and accesses the I2C device, the bitstream is not programmed yet so I'll be looking to see if that's causing an issue and what other changes are present between the 2019 and 2020 FSBL images.

I had added some debug code to the driver and observed that it hangs when trying to access the controller memory:

[    6.827931] cdns-i2c ff030000.i2c: Entered I2C Probe.
[    6.832976] cdns-i2c ff030000.i2c: After Set drvdata
[    6.837930] cdns-i2c ff030000.i2c: After of match node
[    6.843062] cdns-i2c ff030000.i2c: After pinctl
[    6.847582] cdns-i2c ff030000.i2c: resource r_mem-start 0xff030000 and end 0xff030fff
[    6.855413] cdns-i2c ff030000.i2c: io remapped to address 0x11205000
[    6.861755] cdns-i2c ff030000.i2c: After ioremap
[    6.866372] cdns-i2c ff030000.i2c: After irq retrieve
[    6.871418] cdns-i2c ff030000.i2c: After init completion
[    6.876746] cdns-i2c ff030000.i2c: before pm framework calls
[    6.882399] cdns-i2c ff030000.i2c: after pm framework calls
[    6.886748] ata1: SATA link down (SStatus 0 SControl 320)
[    6.887973] cdns-i2c ff030000.i2c: before of clock read
[    6.898561] cdns-i2c ff030000.i2c: before set clk
[    6.903255] cdns-i2c ff030000.i2c: before req irq
[    6.907972] cdns-i2c ff030000.i2c: before cdns i2c init
[    6.913196] cdns-i2c ff030000.i2c: writing 0xb0e to register 0x11205000
<< hang >>
0 Kudos
maps-mpls
Mentor
Mentor
409 Views
Registered: ‎06-20-2017

Oh, are you using the I2C in the PS or the PL (some devCards have both options open).  You might be on to something there.

*** Destination: Rapid design and development cycles ***
0 Kudos
bhig
Contributor
Contributor
365 Views
Registered: ‎09-01-2017

In our system we use both I2C peripherals in the PS and also instantiate one AXI I2C in the PL. The PL I2C has no problems, just the PS side devices have an issue.

It's really weird, its almost as if something is disabling the I2C controllers during the Linux boot, such that if I do even a simple "devmem 0xff030000" to read out the control register, the entire system hangs.

With my 2019 image, I can replicate this by setting the DTS setting to "disabled" and then after booting trying to access devmem.  However, in my 2020 image it's being set to "okay" so whatever is supposed to be enabling that peripheral is clearly not when run with the 2020.1 FSBL. 

0 Kudos
maps-mpls
Mentor
Mentor
328 Views
Registered: ‎06-20-2017

Are you loading the bitstream during the FSBL?  Did you change any sequencing there?  Maybe set the PL I2C to disabled in your dtsi.

I've heard about people having trouble with the PS I2C controller, but I have not had any.

*** Destination: Rapid design and development cycles ***
0 Kudos