Zynq Linux Freeze on Memory Mapped Peripheral Write

[krbreton]

Hi All-

I have a  Zynq ZC702 Board that I have loaded with a simple bitstream in the PL.  In addition to some basic peripherals, it maps a few of the LEDs to a memory space over the AXI bus.

With a baremetal application, I am able to change the pattern on the LEDs easily. (See Below)

However in linux, with both a user application (mmap based), and a kernel driver (ioremap, iowrite32), as soon as I write to the memory address, the processor freezes completely - I even seem to lose the JTAG connection to the device.  

(See Below).  The value is written out the AXI bus - I can see the correct new pattern on the LEDs but all communication with the Zynq is gone.

Is there something I'm missing?  I have to admit, I'm feeling kind of foolish, not being able to talk to a simple memory mapped peripheral.

Thanks, 

-Kyle

---------------BAREMETAL CODE (Works)---------------------------

char *pLEDS = (char *)0x6F400000;
while(1)
{
for(ii = 0; ii < 1000; ++ii)
print("Hello World\n\r");

*pLEDS = ~*pLEDS;
}

--------------User Space Code (Freezes After Write)--------------------

/* offset for mmap() must be page aligned */
page_offset = target & ~(sysconf(_SC_PAGE_SIZE) - 1);

if((fd = open("/dev/mem", O_RDWR | O_SYNC)) == -1) {
printf("/dev/mem could not be opened.\n");
return -1;
} else {
printf("/dev/mem opened.\n");
}
fflush(stdout);

/* Map one page */

map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, page_offset);
if(map_base == (void *) -1) {
printf("Memory map failed.\n");
} else {
printf("Memory mapped at address %p.\n", map_base);
}
fflush(stdout);

virt_addr = map_base + (target - page_offset);

uiValue = *(volatile unsigned int *)virt_addr;
*(volatile unsigned int *)virt_addr = 0x00000055;

// COMPLETE FREEZE RIGHT HERE

if(munmap(map_base, MAP_SIZE) == -1) {
printf("Memory unmap failed.\n");
}

--------------Kernel Space Code (Freezes After Write)--------------------

u32 uNewVal = chan_cfg->chan;
unsigned long *pLEDPointerBase;

printk("<1> Remapping memory space");
pLEDPointerBase = ioremap(0x6F400000,0xFFFF);

printk("<1> Reading from memory space");
chan_cfg->chan = ioread32(pLEDPointerBase);

printk("<1> Writing to memory space");
iowrite32(uNewVal, pLEDPointerBase);

// COMPLETE FREEZE RIGHT HERE

[norman_wong]

I've that sort of thing in a couple of cases. One case is where the pins of a PS peripheral is not assigned to MIOs. Accessing the peripheral's memory range causes a hang. Probably does not apply here. Other case is that translation table has been configured with your memory range as unaccessible. Usually this is both for read and write. Odd that a read is okay but write is not. An additional test would be to try the access from U-boot. Assuming 32 bit access and the bitstream has been loaded by the FSBL:

md.l 0x6F400000 1

mw.l 0x6F400000 0x55555555

I believe U-boot uses a completely flat memory space with no MMU or translation. Don't know about Linux. The MMU code is fairly obtuse. The ioremap() should take care of the MMU.

[krbreton]

Norman-

Thanks for the response.  U-Boot display and write commands work just fine.  

After hooking up chipscope, the AXI write appears correct (and as I mentioned, the pattern on the LEDs works).  

I have been able to get the "mmap" version working by not running it through the gdb server (download to the Zynq and run locally seems to work).  However, I am still getting the same freeze on the kernel driver side, and I've tried moving the peripheral (and the corresponding memory accesses to 0x40300000, with identical results.

Still very stumped...

-Kyle

[norman_wong]

Quite a puzzle. I've done much the same thing and maybe by luck that it worked. My address range was in the 0x40000000. For sure, ranges such as 0xC0000000 don't work. Documented as reserved in a one very small note it the manual. I broke the rules a bit by accessing memory directly rather than using iowrite32().

Perhaps, ioremap() failed. Check the returned pointer for null. Maybe try mapping a smaller size...like 4 bytes. You could try using the pointer directly although that's apparently discouraged.

If you do figure it out, please post back. Sounds like it would be a handy thing to know.

[linnj]

It sounds like the mmap version is working, but kernel version is not if I'm clear on it.

pLEDPointerBase = ioremap(0x6F400000,0xFFFF)

The 2nd argument should be a size, an even multiples of pages (4K). I don't know why that would make it fail, but you should really check the return value of 0 as a failure. But the reads work so I'm not sure this is the problem. We're using __raw_read and write in our drivers because the memory from ioremap is device memory which doesnt need barriers in general (based on what I remember :)).

[pbutov]

Hi krbreton,

I have the same problem. Can you tell me how do you solved it ?

Best regards, Paul

[kpiter]

There is no issue... check ioremap value (virtual address) and correct address range.

In kernel mode this works:

void *cfgPtr;
cfgPtr = (void *)ioremap(AXI_DEVICE_BASE, 0x1000);
iowrite32( REG_VALUE, cfgPtr + AXI_ADDRESS_OFFSET);
iounmap(cfgPtr);

[nib]

---

@norman_wong wrote:

[...]

Accessing the peripheral's memory range causes a hang. [...] Other case is that translation table has been configured with your memory range as unaccessible. Usually this is both for read and write. [...]

---

Hi Norman, old post, I know, but I am having the problem you are referring to, where both read and write cause a hang. (I have a thread on it here.) But I would love to know if the "translation table" you are referring to is purely a hardware feature/the SoC's internal configuration (configured via the bitstream), or if it is something configured by software, at runtime (or if it is maybe just the "regular" virtual memory mappings, managed by linux). I would really love to be able to verify that my requests to my PL peripheral are actually being mapped to the correct AXI port, as both reading the assigned memory range and reading an unassigned address seem to have exactly the same outcome (the aforementioned hang).

[norman_wong]

Sounds like you found you solution in the other thread. Some comments:

The translation table should only be an issue outside linux, ie. Bare Metal, FSBL or u-boot. The Linux MMU code should modify the translation table appropriately. A translation table hang is not really a hang but an exception that runs an handler than infinitely loops. The translation table itself is buried in the FSBL. Earlier FSBL releases has table entries that disabled large chunks for PL memory space. Later FSBL releases seem to enable more.

The other type of hang seems entirely PL related. To me it looks like a bus hang. The PL appears to stop the PS in its tracks. In some cases because a peripheral's pins don't route to anywhere. Other cases, I think something in the PL logic has decided the PL should not proceed.  I'm am firmware guy so I don't know the specific details. Some ranges just don't work. My FPGA guy would move an interface to a slightly different range and it would work. Other places no work. No reason. I am suspicious that the tool was generating bad logic. Check the TRM, there are certain memory ranges that are strictly reserved.

[nib]

Ahh, thanks Norman, that was very helpful.

So, from what I know now, I am going to guess that the overall mapping of addresses works as follows:

• There is a hard address map (with minor reconfigurability for the OCM), which is responsible for making my requests end up on the correct AXI master port. (This is detailed in page 112 of the current TRM.)
• Presumably, the bitstream contains some configuration for the relevant hard AXI interconnect, which allows it to filter requests to addresses which have not been assigned to a peripheral. (Based on the fact that accessing unassigned addresses in the range of the same AXI master port as my crashing peripheral does not produce a hang (any more) and all that I am changing on the FPGA is the bitstream.) Alternatively, this feature might be implemented in the soft interconnect, which XPS placed between the hard master port and the "external slave connector".
• Above that, there is a "translation table", which is configured by the FSBL. (Presumably, this is effectively just an initial configuration of the hardware MMU?)
• Finally, the Linux kernel will modify the translation table via its MMU code, as it sees fit. (This part is why I presume, in particular, that the "translation table" effectively represents runtime configuration of the hardware MMU.)

Thanks again for all the details - I feel like I have a much clearer picture of what is going on on the chip now and how the various hard- and software bits interact.

I checked the TRM, as you suggested, but, judging by that, I don't think my peripheral was ever mapped into an invalid range. At least the address range was always within the 0x4000_0000-0x7FFF_FFFF range, which is meant to be mapped to M_AXI_GP0 and prodding any addresses through out that range has not caused a hang, since the initial trouble I was having.

I have already had a good dose of buggy behaviour from XPS and other ISE tools, so I would not at all be surprised if the tool was generating bad logic, as you suggest, but I am also still very new to FPGA and embedded development, so human error is not at all out of the question. (Honestly, I would rather it was my fault: Learning this stuff is already hard enough, without unreliable tools ;) . )

[jgorgen]

I had a similar issue on a ZC706 using the latest ( commit 6530a9b3...) checkout of linux-xlnx 2014.2

My issue turned out to programmer error (mis-typed address)

For me, the PS was hanging on an iowrite32() command that was not word-aligned.

That is,

void * register_base_address = ioremap_nocache(0x43c00000,0x10000);
iowrite32(0x01, register_base_address ); //succeeds
iowrite32(0x01, register_base_address  + 0x02); //hangs after write

For debugging purposes, I used printk() statements for each iowrite32 address to ensure they were word aligned, which is how I found this  my bug.

[norman_wong]

I am surprised you were able to do address arithmetic on a void pointer. The compiler should not have allowed it.

[zkulis17]

I had a similar 'hanging' issue when upgrading a design from Vivado 2013.3 to 2015.4 and running the latest Linux kernel (4.0.0). It turns out that the Linux kernel introduced a new way of handling FPGA clock gating in 2013: https://lkml.org/lkml/2013/10/8/418. When I inspected the device tree file (dts) created from the old (2013.3) system.xml hardware description, I noticed that fclk-enable was set to 0:

&clkc {
	fclk-enable = <0x0>;
	ps-clk-frequency = <33333333>;
};

Since the clock enable bit flags were set to 0, the FPGA clocks were essentially gated when running in the latest Linux kernel (4.0.0). This caused all ioread32() or iowrite32() calls to address space 0x40000000-0x7fffffff to hang. Once I set fclk-enable to <0x1>, the hanging problem went away.

[yunq]

zkulis17 's  solution is right. Most likely there are some misconfigration about the clocks... We have seen the same problem, but after modifying the device tree, it works...

[milosz.malczak@gmail.com]

I had problem with freezing Linux if I didn't include processor_system in Vivado design. I don't know if You do it, but processing_system IP must be included

[kieucua1503]

This is exactly my solution.