cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Visitor
Visitor
823 Views
Registered: ‎12-13-2019

XDMA Configuration registers reset after cold boot

I have some troubles getting a stable design using the XMDA 4.1 core on a kintex 7 410t and the xilinx drivers under win10.

The design is very simple: XMDA->interconnect->BRAM

When performing a cold boot, most of the times, the configuration registers of all engines (DMA/IRQ, SGDMA etc..)  are reset to default.
Weird part is: this happens after the driver is loaded. And only after a cold boot and not always.
I build in some checks in the driver to check what happens on boot time and all registers are properly set. (checked, by reading back the registers)
However, when windows is fully operational and I run XDMA_INFO, all registers are reset to default. as if the XDMA core received a soft reset. (soft reset not enabled!)
I checked hardware. Power: all good and stable. Normal PERST behaviour. user_link_up is normal etc.. 
IRQ behavious is normal. No spurious interrupts.

I tried on different host PC's. still the same. I tried on another board (Hitech Global and Trenz). all the same.

Could it be that the root complex or host pc (windows) is sending a hot reset? Is that even supported?
What else can cause all register to go default?

thanks in advance for the suggestions.

Tags (4)
0 Kudos
9 Replies
Highlighted
Visitor
Visitor
734 Views
Registered: ‎12-13-2019

Okay. I got it working now.
I'm not 100% sure what is going on, but it seems that the XDMA config registers are reset after a successful initialization of the driver at boot time and before Windows is fully started.
I think there is some kind of hot reset command or a power state change command that causes the core to reset it's registers.
I have modified the driver to take a copy of all register settings at the end of the driver hardware initialization function and apply these settings again at the first time the user tries to access the card via the driver. 
This makes the card work smoothly.
Additionally MSI had to be enabled and the user interrupt input had to be hard wired to 0 to make sure no false interrupts were generated.

 

If anyone can comment on the PCIe power state command or hot reset command: much appreciated.

Highlighted
Adventurer
Adventurer
668 Views
Registered: ‎11-13-2018

Hello,

We have a very similar issue. We are using the AC701 evaluation board with the XDMA IP on Vivado 2019.2. We have created a custom driver based on the example driver given by Xilinx. We noticed that on cold reset that our dma utility would hang, but register reads/writes using the other BARs (DMA bypass, AXI lite) work fine. After a warm reset, everything seems to function as expected. 

At first I wondered if this was a hardware configuration problem, but that doesn't seem to be the case since everything else functions properly (device is recognized by host, user_link_up is normal, register operations are functional). Is there some strange reset that could be occurring inside the XDMA IP (or from the host like OP mentions)? 

Looking into the driver we saw that the registers are not being set up properly. @jaccohouter mentions the same issue, so we will try the solution they used. 

I'll post again if we find anything else worthy of reporting. It would be nice to hear what a Xilinx employee thinks about this.

Thanks in advance for any suggestions or support!

0 Kudos
Adventurer
Adventurer
615 Views
Registered: ‎11-13-2018

Hi @jaccohouter,

We tried your fix and are still having some trouble restoring the register settings. Where exactly did you apply the settings in the driver?

Thanks in advance!

0 Kudos
Highlighted
Visitor
Visitor
584 Views
Registered: ‎12-13-2019

Good morning @bhall0107 

First: check, using XDMA_INFO, whether the 'DESCR ADDR LO' and 'POLL WB ADDR LO'  are all set to 0 after cold boot. If so, highly likely the registers are reset. 

In the driver I made the following changes:

In file dma_engine.h, struct XDMA_ENGINE_T, added:

//backups
UINT32 control_bck;
UINT32 pollModeWbLo_bck;
UINT32 pollModeWbHi_bck;
UINT32 intEnableMask_bck;
UINT32 firstDescLo_bck;
UINT32 firstDescHi_bck;

In file device.h, struct XDMA_DEVICE_T, added:

// backups
UINT32 userIntEnable_bck;
UINT32 channelIntEnable_bck;
UINT32 userVector_bck[4];
UINT32 channelVector_bck[2];

In file driver.c, function EvtDevicePrepareHardware, added at the end, just before return status:

/**************************************************************************/
/*Windows somehow resets the XDMA register */
/*Here we make a copy and in EvtDeviceFileCreate we restore the registers */
/* */ 
/**************************************************************************/
TraceVerbose(DBG_INIT, "Creating copy of control registers.");
for (UINT dir = H2C; dir < 2; dir++) { // 0=H2C, 1=C2H
for (ULONG ch = 0; ch < XDMA_MAX_NUM_CHANNELS; ch++) {
XDMA_ENGINE* engine = &(xdma->engines[ch][dir]);
if (engine->enabled == TRUE) {
TraceVerbose(DBG_INIT, "Control register: 0x%08X", engine->regs->control);
engine->control_bck = engine->regs->control;
TraceVerbose(DBG_INIT, "Backup Control register: 0x%08X", engine->control_bck);
engine->pollModeWbLo_bck = engine->regs->pollModeWbLo;
engine->pollModeWbHi_bck = engine->regs->pollModeWbHi;
engine->intEnableMask_bck = engine->regs->intEnableMask;

engine->firstDescLo_bck = engine->sgdma->firstDescLo;
engine->firstDescHi_bck = engine->sgdma->firstDescHi;
}
}
}

xdma->channelIntEnable_bck = xdma->interruptRegs->channelIntEnable;
xdma->userIntEnable_bck = xdma->interruptRegs->userIntEnable;
for (UINT vec = 0; vec < 4; vec++) {
xdma->userVector_bck[vec]= xdma->interruptRegs->userVector[vec];
}
for (UINT vec = 0; vec < 2; vec++) {
xdma->channelVector_bck[vec] = xdma->interruptRegs->channelVector[vec];
}
/**************************************************************************/

 

And finally in file file_io.c function EvtDeviceFileCreate at the beginning:

/**************************************************************************/
/*Windows somehow resets the XDMA register */
/*Here we make a restore the registers, when needed. */
/* */
/**************************************************************************/
XDMA_ENGINE* enginex = &(xdma->engines[0][0]);
if (enginex->regs->pollModeWbLo == 0) {
TraceVerbose(DBG_IO, "Restoring engine channels");
for (UINT dir = H2C; dir < 2; dir++) { // 0=H2C, 1=C2H
for (ULONG ch = 0; ch < XDMA_MAX_NUM_CHANNELS; ch++) {
XDMA_ENGINE* engine = &(xdma->engines[ch][dir]);
if (engine->enabled == TRUE) {
engine->regs->control = engine->control_bck ;
engine->regs->pollModeWbLo = engine->pollModeWbLo_bck;
engine->regs->pollModeWbHi = engine->pollModeWbHi_bck;
engine->regs->intEnableMask = engine->intEnableMask_bck;
engine->sgdma->firstDescLo = engine->firstDescLo_bck;
engine->sgdma->firstDescHi = engine->firstDescHi_bck;
}
}
}
xdma->interruptRegs->channelIntEnable = xdma->userIntEnable_bck;
xdma->interruptRegs->userIntEnable = xdma->channelIntEnable_bck;
for (UINT vec = 0; vec < 4; vec++) {
xdma->interruptRegs->userVector[vec] = xdma->userVector_bck[vec];
}
for (UINT vec = 0; vec < 2; vec++) {
xdma->interruptRegs->channelVector[vec] = xdma->channelVector_bck[vec];
}
}
/**************************************************************************/

 

Please note: I'm not using the bypass channel and I have only 1 C2H and 1 H2C channel enabled.

Kind regards

Jacco

 

 

Highlighted
Adventurer
Adventurer
561 Views
Registered: ‎11-13-2018

@jaccohouter Thanks so much for your response! We are going to try this out. I also just wanted to clarify one thing. When you mention, "Additionally MSI had to be enabled and the user interrupt input had to be hard wired to 0 to make sure no false interrupts were generated," are you referring to enabling the MSI Capability Structure and connecting a constant 0 to the usr_irq_req port?  For some reason we are getting a lot of spurious interrupts.  

Thanks again!

0 Kudos
Highlighted
Visitor
Visitor
548 Views
Registered: ‎12-13-2019

Dear @bhall0107 

Yes correct. I had the same issues.

After enabling the MSI-X capabilites and tieing the user_irq to 0, the interrupts where gone.

msi.JPG

irq.JPG

 

Good luck!

Highlighted
Adventurer
Adventurer
430 Views
Registered: ‎11-13-2018

@jaccohouter Thanks for your response, it was very helpful. We were able to load the registers back using the method you described. One thing we had to change from what you suggested was enabling MSI interrupts but disabling MSI-X and legacy. For some reason our results were worse with MSI-X enabled. We still are getting the spurious interrupts, but our DMA utility works properly now.

Is there anything you changed in the driver to support MSI-X interrupts? It seems like we are missing something to get that to work. I'm not sure that we will be using close to all of the interrupts that MSI-X provides, but maybe using it would fix our problem with spurious interrupts. 

Again, thanks for your support. It is much appreciated!

0 Kudos
Highlighted
Visitor
Visitor
384 Views
Registered: ‎12-13-2019

Goodmorning @bhall0107 

I'm afraid I can't help you further. We are not using any interrupts and are using the driver in pollmode.

Regards

Jacco

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
255 Views
Registered: ‎08-02-2007

the XDMA does not support ASPM or other lower power status, please  try to check cfg_power_* signals and see if there is any thing  abnormal 

 

------------------------------------------------------------------------------
Don't forget to reply, give kudo and accept as solution
------------------------------------------------------------------------------