UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Explorer
Explorer
5,685 Views
Registered: ‎08-28-2008

Microblaze interrupt issues

hi

  1. We use Dual Microblaze V8.50.a system in a K7 325T device, the problem happens always on the slave Microblaze(the 2nd one)
  2. The microblaze should get in to a interrupt, but it does not.
  3. We using a chipscope to capture the PC of the corresponding Microblaze, and we map the captured PC back to ELF and dissemble the code as point A in the picture shows.
  4. Note sample 32, after instruction msrset, Microblaze should go to the interrupt the PC should go to 0x8, but is does not(in-> PC in the table)
  5. Xmd dumps the INTC register: ISR-> 0000000C; IPR-> 0000000C; IER->0000001F; MER-> 00000003   for sure, there are interrupts
  6. For MSR: we stop the Microblaze in the XMD, and dump MSR -> msr: 00000086, for sure, interrupt is open
  7. A chipscope point B shows the MSR tracing when the interrupt happens
  8. In xmd, while I input stop-> con to Microblaze, the Microblaze goes interrupt immediately the chipscope show the wave changing when Microblaze goes interrupt by stop->
    con to Microblaze. Point C  in the picture
  9. The issue happens seems on the special instruction sequence, when we insert some debug print in the code, it does not happen.

thanks a lot for the supports

(the picture is not very clear in the web Page, To save the picture in local, & open it is would be more clear)

mb_interrupt.png
0 Kudos
5 Replies
Xilinx Employee
Xilinx Employee
5,663 Views
Registered: ‎08-06-2007

Re: Microblaze interrupt issues

Hi,

 

So if I understand you correctly, this is happening.

When MicroBlaze enables interrupts with the "msrset" instruction it will not take an interrupt.

How long time to do wait before stopping MicroBlaze?

 

Later (after how long time?) after you have stopped MicroBlaze and do a "con" command in XMD then MicroBlaze will take the interrupt immediately.

 

Waveform B seems odd. It looks like Trace_MSR_Reg(13) is toggling.

That bit is the Interrupt Enable (IE) bit in MSR. Could you show some more signals when this is happening?

What is MicroBlaze executing which could cause this behaviour?

 

Göran

0 Kudos
Explorer
Explorer
5,654 Views
Registered: ‎08-28-2008

Re: Microblaze interrupt issues

hi Göran

 

thanks you for your reply, the situation is complex, let me answer your question first,

How long time to do wait before stopping MicroBlaze? -> we trigger the interrupt, the CPU could not get into interrupt, we stop it

Later (after how long time?) after you have stopped MicroBlaze and do a "con" command in XMD then MicroBlaze will take the interrupt immediately.-> some time few seconds, sometime minutes

That bit is the Interrupt Enable (IE) bit in MSR. Could you show some more signals when this is happening? -> yes, the software close/open interrupt some time.

 

Here we have some update information,

This issue may due to AXI interconnect or DDR controller.

The system configuration is show as the picture: and an attachment MHS,

a.       AXI4: DDR controller is only one AXI4 slave; and 4 AXI4 Master ports, 2 of the Microblaze AXI DC ports, 1 external write only port, 1 external Read/Write Port

b.       AXI_lite: 2 of Microblaze AXI DP Master ports; 8 peripheral slave

 mhs.jpg

1. The problem happens, as it is described before Microblaze 1 cannot get into interrupt, but it should

2. When the problem happens, Microblaze 0 cannot be stopped in the XMD, in the same time, targeting to Microblaze 1 and stopping Microblaze 1, Microblaze 0 also stopps.

3. When the problem happens, in the XMD, after stop->con on Microblaze 1,  Microblaze 1 goes in interrupt correctly

4. 2 AXI monitors are added on 2 Microblaze DC ports, to monitor the status when the problem happens

        a. on mormal startup Most of the transaction are write, due to the write through cache type, few reads may due to the Microblaze cache hit.

        b. There is no data transaction on Microblaze 0 AXI DC BUS, the ready signals of Write Ports from AXI interconnect are invalid, the valid from Microblaze 0 are high, AXI interconnect does not acknowledge the request, -> it may explain why Microblaze 0 could not be stopped in XMD, -> go to Point2; when stop only Microblaze 1, data transaction starts on Microblaze 0 AXI DC BUS

cs1.jpg

 

      c. A lot of data transaction on Microblaze 1 AXI DC BUS, it seems normal.

cs2.jpg

 

5.In my MHS, note PARAMETER C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 32, of Microblaze, we change the value to 1, the issues disappeared

6.In my chipscope you can see there are lot of single write transaction from Microblaze 1.

7.A hypothesis, is that, in the situation of siggle transaction to DDR controller, the bandwidth usage gets externally low, as C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 32, Microblaze could issues at most 32 transaction to interconnect, it cost all the DDR bandwidth, the other master could get the change.

8.When C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 1, the Microblaze could only issue 1  transaction and wait the respond then next can be sent, it leave the change to the other port.

9.Another situation we observe may also support it: we watch the AXI write transition on one of our External Master port. When  C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 32, the respond get a big latency some time >2000 cycles, when the issues happens, the Master never get any respond, and dead; however When  C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 1 we never observe such big latency,and dead lock

10.So, here is the question, we get 64bit DDR Running @ 800MHz interface, and internal 256bit AXI DDR controller bus, we have 6.4Gbytes line rate, if the DDR efficacy is 50%, we still have 3.2Gbyte/s for write + read, why the Microblaze can cost the all the bandwidth if my hypothesis is correct?

 

0 Kudos
Xilinx Employee
Xilinx Employee
5,648 Views
Registered: ‎08-06-2007

Re: Microblaze interrupt issues

Hi,

 

If MicroBlaze is stalled on a memory access so can't XMD stop it in a normal way.

XMD should report that MicroBlaze is stalled on a memory access.

 

Could you create a trace on MicroBlaze_1 directly after it has enabled the IE bit in the MSR?

I would like to see Trace_PC, Trace_Instruction, Trace_Valid_Instr, Trace_MSR_Reg, Trace_Excep_Taken and Trace_Excep_Kind. That might give a hint why MicroBlaze is not taking the interrupt.

A few hundred instructions after the IE bit is set should be enough to see what is going on.

You can set the trigger in Chipscope to the PC of the msrset instruction together with Trace_Valid_Instr = '1'.

 

In fact, it might taken the interrupt but the interrupt handler is accesses the DDR memory which is stalled from MicroBlaze_0 and thus the interrupt handler is also stalled.

 

Göran

0 Kudos
Explorer
Explorer
5,621 Views
Registered: ‎08-28-2008

Re: Microblaze interrupt issues

Hi

 

Now, it is clear and it is the problem of the AXI & DDR efficacy. In some current AXI configuration the Microblaze costs all the DDR bandwidth, before the ending of the external port transaction, there is no interrupt set. When we stop MB1, the transaction of the external port continues, the interrupt is set.

 

After we studied the AXI interface  and the DDR controller, we may need some performance tune now, could you help:

  1. We have AXI DDR3 controller, Running @ 400MHz, with 200MHz 256bit AXI interface, 2 Microblaze running @ 200Mhz, we change the cache type to write back to increase the write performance.
  2. We have 1 read/write 128bits external AXI ports, 1 write only 128bits ports, both of the port has MAX about 1.2GByts transaction, most of the transactions are full 4k byte burst
  3. We tune the setting of the AXI and DDR controller. The performance just has little changes.
  4. We test the DDR bandwidth utilization is around 50%, is it normal?
  5. We have Microblaze read latency, MIN 60 cycles, and max >1000cycles latency, is it normal?
  6. How can we increase the DDR bandwidth utilization, and reduce Microblaze read latency in this architecture?

 

Thanks

BR

Yilei

0 Kudos
Xilinx Employee
Xilinx Employee
5,611 Views
Registered: ‎08-06-2007

Re: Microblaze interrupt issues

Hi,

 

Hmm, I'm not an expert on tuning MIG and AXI.

 

DDR3 at 400 MHZ and 64-bit data have a maximum (100% utilization) of 400*2*(64/8) = 6.4 GByte/s

I think 70% utilization is more normal and thus you have around 4.5 GByte/s of bandwidth.

 

The 256-bit, 200 MHz AXI interface can provide 6.4 GByte/s read and 6.4 Gbyte/s write when 100% utilized.

 

Your two 128-bit AXI ports each consume 1.2 GByte/s so it will consume roughly 50% of the available DDR3 bandwidth.

 

A 4kbyte burst is (when 100% utilizing the AXI) 128 AXI clock cycles, can be much longer if buffers gets full etc.

 

Since AXI allows many outstanding access requests so can multiple 4k burst be in transfer at the same time and thus up to 1000 clock cycles latency can be expected for MicroBlaze read requests.

How many outstanding AXI requests do your AXI Masters allow?

 

Can you chop up the bursts into smaller parts so you don't have 4k burst sizes?

That would most likely improve MicroBlaze read latency but might lower the AXI interconnect utilization.

 

Since MIG only has one AXI ports, so must there be an AXI interconnect that handles your two AXI masters and MicroBlaze two AXI cache interfaces.

 

How is this AXI interconnect configured?

 

Increasing the caches on MicroBlaze will lower the cache misses and MicroBlaze AXI traffic.

Also increasing the cache line length to 8 might improve but it's a little dependent on the memory access pattern.

 

Using writeback caches, will reduce the AXI traffic but I wonder how you handle data coherency between the two MicroBlaze since one MicroBlaze might have dirty data that the other MicroBlaze is requesting.

 

Another completely different approach is to use our latest System Cache IP core.

It will provide a L2 cache between the AXI interconnect and the MIG.

If your memory access pattern has some locality so will some of the memory accesses only hit the L2 cache and thus reduce the DDR3 traffic.

In the latest version we also support data coherency between two MicroBlaze and a generic AXI port (if it doesn't have caches). 

 

Göran

0 Kudos