UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Contributor
Contributor
7,189 Views
Registered: ‎02-24-2016

Corrupted memory when probing remoteproc (OpenAmp)

Jump to solution

Hi all, I hope you may help me with this...

 

My problem looks simple, the memory seems to be corrupted when I use ZYNQ in AMP mode (OpenAmp).

 

The problem:

  • I have one application running on linux. If I run this app "alone", I get all time the expected output.  
  • When enabling AMP (modprobe remoteproc...), and running again same app, the output changes. I do not need to load any particular firmware in the second core, I've tested some with similar results (e.g. just running an infinite while on core1...). 
  • IF (a bit of a mistery to me...I run the app from sdk debugger (using TCF), I always get the expected output (independently of using AMP or SMP).

A bit more of context:

  • I'm using ZC706 dev. board, vivado, petalinux, etc.,  2015.4.
  • I successfully followed ug1186. I can run any given example without errors (echo test, matrix mul & proxy examples). 
  • I generated a custom kernel with petalinux (basically add some missing libraries and enable ssh and so on).
  • The linux application that I'm talking about just handles few VDMAs and HLS IPs blocks, and performs a few software calculations with one ARM (find max and a IFFT). However, it does use a higher part of the ddr to store some intermediate results (i.e. 0x2000 0000 - 0x4000 0000). I did reduce linux memory range in the DT (reg = <0x0   0x20000000>;).

What I have tried...:

  • Follow ug1186 again (...and again). 
  • Play with the device tree: limit memory ranges, create reserved memory bank for app intermediate results (from 0x2....0) 
  • Having a look at dmesg...nothing special.
  • ...and of course asking sir google many times...and other not relevant stuff.

 

Assuming that the problem is in the DDR memory...

why does it happen only when I probe remoteproc? Is linux corrupting the higher part of my ddr? How can I control this?

 

I hope you could give me a tip or a hint on what I am doing wrong. I start running out of ideas and I really don't know how to tackle this issue.

 

I look forward to hearing from you,

Regards,
Garbí

 

0 Kudos
1 Solution

Accepted Solutions
Contributor
Contributor
12,986 Views
Registered: ‎02-24-2016

Re: Corrupted memory when probing remoteproc (OpenAmp)

Jump to solution

Hi all again,

 

I found the problem, it was not related to cache, though.

 

The way I configure the DT to define the memory region is valid, and linux does not cache the memory out of this region. The problem was related to how I was handling VDMAs.

 

Main issue was that when enabling AMP, and running my code on one core instead of two, the code started crashing and reading corrupted data. The VDMAs were getting several errors like "early end of frame error". This error was quite unlikely in my design so I started to think that the problem was during the configuration of the proper VDMAs: somehow, the VDMA seems not to have enough "time" to configure itself before I start using it.

 

My workaround consisted on adding extra clock cycles between when I configure and when I start using VDMAs. My application is back to live (as it was before trying AMP ;-) ).

 

In case you have similar issues, check if the configuration is being applied on time and whether you get VDMA errors or not, and be sure that you clear the interrupts.

 

Let me know if you run into similar problems, ;-).

Best regards,

Garbí

0 Kudos
3 Replies
Xilinx Employee
Xilinx Employee
7,165 Views
Registered: ‎02-12-2013

Re: Corrupted memory when probing remoteproc (OpenAmp)

Jump to solution

from what you have described, you have a Linux applcation to work with your IPs and it will access 0x20000000 to 0x40000000.

 

If you run the applicaiton in SMP mode it works, if you use the XSDK debugger to access 0x20000000 to 0x40000000, the result is expect in both OpenAMP and SMP case.

 

However, when you are running your Linux app, when remoteproc loads the 2nd CPU, the Linux applicaiton gets corrupted result, but if you use XSDK debugger to access the result, it is correct.

 

In this case, i would suspected it is related to CACHE. Could you confirm it is the case that even when you get the wrong output from the Linux applicaiton, you can still correct output if you directly access the memory, e.g. from XSDK, and if you stop your applicaiton, and run it again, it will work properly after the remoteproc loads the 2nd cpu

0 Kudos
Highlighted
Contributor
Contributor
7,125 Views
Registered: ‎02-24-2016

Re: Corrupted memory when probing remoteproc (OpenAmp)

Jump to solution

Hi Jliang, thanks for your reply,

 

Yes, you may be right, it's probably related to cache, but...

  • Why does it happens only when probe remote proc? 

and if I limit the memory region in the DT...

  • Why does I need to mark as uncached a memory region that is not used by the kernel?

That's how I define this region...simple...and my PL vdma uses from 0x2...0 onwards.

 

	memory {
		device_type = "memory";
		reg = <0x00000000 0x20000000>; 
	};

 

 

Anyway, I'll try to do what you mentioned, I believe it worth checking it.

Any suggestions on how to mark the region as non-cached? it is such a big memory region xD. You think I have to use dma_alloc_coherent()???

 

Any tips are really welcome!

 

Thanks for your help, 

Garbí

0 Kudos
Contributor
Contributor
12,987 Views
Registered: ‎02-24-2016

Re: Corrupted memory when probing remoteproc (OpenAmp)

Jump to solution

Hi all again,

 

I found the problem, it was not related to cache, though.

 

The way I configure the DT to define the memory region is valid, and linux does not cache the memory out of this region. The problem was related to how I was handling VDMAs.

 

Main issue was that when enabling AMP, and running my code on one core instead of two, the code started crashing and reading corrupted data. The VDMAs were getting several errors like "early end of frame error". This error was quite unlikely in my design so I started to think that the problem was during the configuration of the proper VDMAs: somehow, the VDMA seems not to have enough "time" to configure itself before I start using it.

 

My workaround consisted on adding extra clock cycles between when I configure and when I start using VDMAs. My application is back to live (as it was before trying AMP ;-) ).

 

In case you have similar issues, check if the configuration is being applied on time and whether you get VDMA errors or not, and be sure that you clear the interrupts.

 

Let me know if you run into similar problems, ;-).

Best regards,

Garbí

0 Kudos