Verifying Memory Mapings on Zynq SOC / Debugging System Hang on Reading PL AXI Peripheral via /dev/mem

[nib]

Hi all,

I have just successfully generated my first Zynq bitstream with a custom AXI slave (using ISE 14.7). It has been assigned an address range via XPS (0x40000000-0x40000FFF). It has been hooked up to M_AXI_GP0 via an axi_ext_slave_connector (and the interconnect which automatically gets chucked inbetween).

I am now making my first tentative attemts at accessing my brand new peripheral via /dev/mem from within linaro linux. My slave is meant to expose a set of registers (some of which trigger stuff on writing). The problem is that any attempt to simply read an address in the assigned range seems to hang the entire system (at least my ssh connection dies and I have to power cycle the board to regain access). The same seems to happen with most other addresses beyond the RAM's high address, although I have successfully read from a GPIO peripheral which is part of the bitstream (it was already there before I started working on the project).

Judging by this post, it is possible for this to happen when the "translation table has been configured with your memory range as unaccessible".

Also, this thread mentions that an invalid AXI implementation may cause the system to hang, when interacting with it, but, unlike this thread suggests, I do not seem to get a "bus error" when I try to access an unoccupied address, so the above explanation of the memory range being marked as inaccessible still needs to be ruled out first.

Other potentially relevant things I came accross are the kernel disabling clocks to peripherals it thinks aren't in use and the enabling of level shifters between PS and PL. Neither of these problems seem likely to be the cause for me, though, since there is already a working PL peripheral present in the project. However, the existing stuff is hooked up via M_AXI_GP1 (rather than port zero), so please let me know if this should invalidate my assumption.

No other searches and documentation have helped me find the answer to my crashing SoC problem, so, if I am not obviously on the wrong track, I'm hoping someone can help me debug my problems by answering the questions below.

Questions:

1. How does the SOC know that the address of my peripheral should be mapped to M_AXI_GP0? I.e. when the kernel/PS tries to read that physical address, how does the request end up on the correct AXI port?
2. Does all the relevant SOC configuration happen via the bitstream, or do I need to be doing anything with the FSBL? Uboot? The kernel?
3. Can I check the SOC-level address range mapping somehow from within linux, after loading the relevant bitstream?

[linnj]

Hi,

You didn't say how you were trying to access the memory when you get the issue. Have you tried using devmem to read/write the memory as that the easiest and does not rely on any s/w you write?

It seems like you have done your homework and understand some of the issues as you're right you can access the PL GPIO so everything (clocks and reset) should be good.

1. I believe the SOC knows how to do this thru the internal switches. It's all address based and should just work.
2. Since the PL GPIO is working the logic should be there and enabled in the PL. Yes you need a driver in Linux to handle the device or a user space driver with UIO. You also need a device tree that is updated to show your device in it and this can be done manually thru the SDK or with Petalinux.
3. No, not until you have a driver that reads the information from the device tree and then maps the device memory into the kernel page tables. Until then nothing in Linux cares about your new device.

Hope that helps,
John

[linnj]

Just to be clear, devmem is a command line utility in the Xilinx root filesystem.

And I forgot to mention, Petalinux can help create a driver template such that a platform driver is minimally functional using the petalinux-create -t modules command. This is a great start, but the reality is you're going to need to learn all the details whether you use Petalinux or not.

[nib]

Ah, thanks John, I am very grateful for your response.

Sorry, I mentioned /dev/mem in the headline, but then I completely forgot to elaborate. I was trying to access the peripheral via the /dev/mem device (as root), using two different tools:

1. pydevmem
2. Sven Anderson's C tool, which I found via Jan Gray's very useful blog entry.

I believe both of these tools basically use mmap() on /dev/mem and do the relevant error checking, so I don't think that should be a problem.

Thank you for pointing out the "devmem" tool. I wasn't aware that it existed. When I read "devmem" previously, I assumed people were just abbreviating /dev/mem a little. I will try to find the tool (however, my platform is the Parallella and I am trying to work on the basis of their linaro linux image, so I am not sure if that will cause me trouble).

Just to clarify your responses to my questions a little:

 1./2. So basically, if it is configured in XPS, then the resulting bitstream alone should contain all the necessary configuration and once that bitstream is loaded onto the chip, all internal switches on the SoC should be set correctly for mapping those addresses to the correct hardware? As far as you know, is there no other way to configure the hardware switches? I.e. the FSBL, uBoot and kernel/device tree should be irrelevant to the internal switch configuration of the SoC?

 3. So there is no way to check that the internal switches are set correctly? I just have to trust that my XPS settings end up on the device and I have no way of ruling out human error, or a problem with the tools (except if the peripheral happens to work)?

Also: Do you know if a system hang is the "expected outcome" for reading an address which isn't mapped to any peripherals in XPS (but is valid for being mapped to) ? I'm guessing that addresses adjacent to the range I configured in XPS might still get mapped to the same AXI port by the hardware anyway, which would explain why reading the peripheral, or an adjacent, unassigned address has the same effect. Still, it would be great to be able to read the hardware's mapping somehow, to verify that.

Assuming all of the above is as it sounds to me, then I guess the error must lie in my peripheral, so I will try attaching a "dumber" peripheral, after seeing if I can try the "devmem" tool and I get my board working again, in general...

Unfortunately, it seems that my latest bitstream is causing my board to fail to boot entirely, for unknown reasons. I panicked for a while, thinking I had bricked my board, because I could have sworn that that bitstream booted fine, at least for the first two times. At any rate, especially since building a fresh bitstream takes about 15 minutes and I further have to physically plug the SD card into a different machine to write the bitstream, testing is going very slow. So, as a side note: I'm not sure if it is common, on Zynq boards, that the boot partition is not visible to the kernel after boot (and thus can't be re-written remotely), but if you happen to know a way around that, I would be very grateful to hear it.

Thanks again for your help!

Nib

[nib]

---

@nib wrote:

I'm not sure if it is common, on Zynq boards, that the boot partition is not visible to the kernel after boot (and thus can't be re-written remotely), but if you happen to know a way around that, I would be very grateful to hear it.

---

Ah, found the answer to this one:

sudo mknod -m 660 /dev/mmcblk0 b 179 0
sudo mknod -m 660 /dev/mmcblk0p1 b 179 1

As per the beginning of this wiki entry. At least on the Parallella, these parameters to mknod will allow you to mount the boot patition normally.

[nib]

After managing to sort out my issues with booting on my self-built bitstream (it turned out I was using a devicetree file which was meant for a different bitstream configuration (HDMI vs. headless)), I was finally able to poke around at random addresses some more. In the course of that, I found that the behaviour where all reads from invalid addresses hung the system was no longer present.

Some specific address ranges still hang and so does the *exact* range which is assigned to my peripheral, so now I am confident that (at least presently) my AXI implementation is the problem and causing all the crashes I care about. Such a relief :D (even though still I wish I had a clue what was going on earlier).

For anyone's interest, the range:

0x40000000-

0x7FFFFFFF

appears to *not* hang the system, when there is no peripheral assigned. (At least on the Parallella, with a headless-based bitstream and a stock headless device tree.) A few pokes at the 0x8******* range still produced hangs, although I am not aware of anything being assigned to that range, but that doesn't bother me so much, for now.

Anyway, I will plod on with trying to fix my peripheral and put the prior behaviour down to probable (albeit unknown) human error (hopefully not glitchyness :/ ) .

Thanks for your help again, John, and hopefully the experiences I have documented will be of use to someone else too. I will report back here, if more funniness occurs.

[nib]

For anyone facing similar issues: Norman Wong had some further helpful details in his post on another thread.