05-09-2013 10:43 PM - edited 05-09-2013 10:56 PM
thanks a lot for the supports
(the picture is not very clear in the web Page, To save the picture in local, & open it is would be more clear)
05-12-2013 11:36 PM
So if I understand you correctly, this is happening.
When MicroBlaze enables interrupts with the "msrset" instruction it will not take an interrupt.
How long time to do wait before stopping MicroBlaze?
Later (after how long time?) after you have stopped MicroBlaze and do a "con" command in XMD then MicroBlaze will take the interrupt immediately.
Waveform B seems odd. It looks like Trace_MSR_Reg(13) is toggling.
That bit is the Interrupt Enable (IE) bit in MSR. Could you show some more signals when this is happening?
What is MicroBlaze executing which could cause this behaviour?
05-13-2013 05:35 AM - edited 05-13-2013 05:43 AM
thanks you for your reply, the situation is complex, let me answer your question first,
How long time to do wait before stopping MicroBlaze? -> we trigger the interrupt, the CPU could not get into interrupt, we stop it
Later (after how long time?) after you have stopped MicroBlaze and do a "con" command in XMD then MicroBlaze will take the interrupt immediately.-> some time few seconds, sometime minutes
That bit is the Interrupt Enable (IE) bit in MSR. Could you show some more signals when this is happening? -> yes, the software close/open interrupt some time.
Here we have some update information,
This issue may due to AXI interconnect or DDR controller.
The system configuration is show as the picture: and an attachment MHS,
a. AXI4: DDR controller is only one AXI4 slave; and 4 AXI4 Master ports, 2 of the Microblaze AXI DC ports, 1 external write only port, 1 external Read/Write Port
b. AXI_lite: 2 of Microblaze AXI DP Master ports; 8 peripheral slave
1. The problem happens, as it is described before Microblaze 1 cannot get into interrupt, but it should
2. When the problem happens, Microblaze 0 cannot be stopped in the XMD, in the same time, targeting to Microblaze 1 and stopping Microblaze 1, Microblaze 0 also stopps.
3. When the problem happens, in the XMD, after stop->con on Microblaze 1, Microblaze 1 goes in interrupt correctly
4. 2 AXI monitors are added on 2 Microblaze DC ports, to monitor the status when the problem happens
a. on mormal startup Most of the transaction are write, due to the write through cache type, few reads may due to the Microblaze cache hit.
b. There is no data transaction on Microblaze 0 AXI DC BUS, the ready signals of Write Ports from AXI interconnect are invalid, the valid from Microblaze 0 are high, AXI interconnect does not acknowledge the request, -> it may explain why Microblaze 0 could not be stopped in XMD, -> go to Point2; when stop only Microblaze 1, data transaction starts on Microblaze 0 AXI DC BUS
c. A lot of data transaction on Microblaze 1 AXI DC BUS, it seems normal.
5.In my MHS, note PARAMETER C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 32, of Microblaze, we change the value to 1, the issues disappeared
6.In my chipscope you can see there are lot of single write transaction from Microblaze 1.
7.A hypothesis, is that, in the situation of siggle transaction to DDR controller, the bandwidth usage gets externally low, as C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 32, Microblaze could issues at most 32 transaction to interconnect, it cost all the DDR bandwidth, the other master could get the change.
8.When C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 1, the Microblaze could only issue 1 transaction and wait the respond then next can be sent, it leave the change to the other port.
9.Another situation we observe may also support it: we watch the AXI write transition on one of our External Master port. When C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 32, the respond get a big latency some time >2000 cycles, when the issues happens, the Master never get any respond, and dead; however When C_INTERCONNECT_M_AXI_DC_WRITE_ISSUING = 1 we never observe such big latency,and dead lock
10.So, here is the question, we get 64bit DDR Running @ 800MHz interface, and internal 256bit AXI DDR controller bus, we have 6.4Gbytes line rate, if the DDR efficacy is 50%, we still have 3.2Gbyte/s for write + read, why the Microblaze can cost the all the bandwidth if my hypothesis is correct?
05-13-2013 05:54 AM
If MicroBlaze is stalled on a memory access so can't XMD stop it in a normal way.
XMD should report that MicroBlaze is stalled on a memory access.
Could you create a trace on MicroBlaze_1 directly after it has enabled the IE bit in the MSR?
I would like to see Trace_PC, Trace_Instruction, Trace_Valid_Instr, Trace_MSR_Reg, Trace_Excep_Taken and Trace_Excep_Kind. That might give a hint why MicroBlaze is not taking the interrupt.
A few hundred instructions after the IE bit is set should be enough to see what is going on.
You can set the trigger in Chipscope to the PC of the msrset instruction together with Trace_Valid_Instr = '1'.
In fact, it might taken the interrupt but the interrupt handler is accesses the DDR memory which is stalled from MicroBlaze_0 and thus the interrupt handler is also stalled.
05-16-2013 07:38 AM
Now, it is clear and it is the problem of the AXI & DDR efficacy. In some current AXI configuration the Microblaze costs all the DDR bandwidth, before the ending of the external port transaction, there is no interrupt set. When we stop MB1, the transaction of the external port continues, the interrupt is set.
After we studied the AXI interface and the DDR controller, we may need some performance tune now, could you help:
05-17-2013 12:35 AM
Hmm, I'm not an expert on tuning MIG and AXI.
DDR3 at 400 MHZ and 64-bit data have a maximum (100% utilization) of 400*2*(64/8) = 6.4 GByte/s
I think 70% utilization is more normal and thus you have around 4.5 GByte/s of bandwidth.
The 256-bit, 200 MHz AXI interface can provide 6.4 GByte/s read and 6.4 Gbyte/s write when 100% utilized.
Your two 128-bit AXI ports each consume 1.2 GByte/s so it will consume roughly 50% of the available DDR3 bandwidth.
A 4kbyte burst is (when 100% utilizing the AXI) 128 AXI clock cycles, can be much longer if buffers gets full etc.
Since AXI allows many outstanding access requests so can multiple 4k burst be in transfer at the same time and thus up to 1000 clock cycles latency can be expected for MicroBlaze read requests.
How many outstanding AXI requests do your AXI Masters allow?
Can you chop up the bursts into smaller parts so you don't have 4k burst sizes?
That would most likely improve MicroBlaze read latency but might lower the AXI interconnect utilization.
Since MIG only has one AXI ports, so must there be an AXI interconnect that handles your two AXI masters and MicroBlaze two AXI cache interfaces.
How is this AXI interconnect configured?
Increasing the caches on MicroBlaze will lower the cache misses and MicroBlaze AXI traffic.
Also increasing the cache line length to 8 might improve but it's a little dependent on the memory access pattern.
Using writeback caches, will reduce the AXI traffic but I wonder how you handle data coherency between the two MicroBlaze since one MicroBlaze might have dirty data that the other MicroBlaze is requesting.
Another completely different approach is to use our latest System Cache IP core.
It will provide a L2 cache between the AXI interconnect and the MIG.
If your memory access pattern has some locality so will some of the memory accesses only hit the L2 cache and thus reduce the DDR3 traffic.
In the latest version we also support data coherency between two MicroBlaze and a generic AXI port (if it doesn't have caches).