UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Contributor
Contributor
228 Views
Registered: ‎10-01-2018

QDMA : User space does not recieve data although interrupt gets triggered

Hello XILINX,

Please note that I have enabled a DEBUG while building the qdma driver.

Initially during adding of the queue we can see the below errors in kernel logs.

[Mon Apr 15 13:34:06 2019] qdma:qdma_err_intr_setup: qdma82000-p0000:82:00.0, write reg 0xb04, val 0x1000000.
[Mon Apr 15 13:34:06 2019] qdma:qdma_err_intr_setup: Error interrupt setup: val = 0x01000000, readback = 0x01000000 err_intr_index = 0 func_id = 0
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: qdma82000-p0000:82:00.0, write reg 0x24c, val 0xfff.
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: glbl_err interrupts enabled: reg -> 0x0000024c, value = 0x00000fff
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: qdma82000-p0000:82:00.0, write reg 0x258, val 0x1f9023f.
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: dsc_err interrupts enabled: reg -> 0x00000258, value = 0x01f9023f
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: qdma82000-p0000:82:00.0, write reg 0x268, val 0xf.
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: trq_err interrupts enabled: reg -> 0x00000268, value = 0x0000000f
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: qdma82000-p0000:82:00.0, write reg 0x24c, val 0xfff.
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: glbl_err interrupts enabled: reg -> 0x0000024c, value = 0x00000fff
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: qdma82000-p0000:82:00.0, write reg 0x24c, val 0xfff.
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: glbl_err interrupts enabled: reg -> 0x0000024c, value = 0x00000fff
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: qdma82000-p0000:82:00.0, write reg 0x24c, val 0xfff.
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: glbl_err interrupts enabled: reg -> 0x0000024c, value = 0x00000fff
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: qdma82000-p0000:82:00.0, write reg 0xf0, val 0xffffff11.
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: ecc_sb_err interrupts enabled: reg -> 0x000000f0, value = 0xffffff11
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: qdma82000-p0000:82:00.0, write reg 0xf8, val 0xffffff11.
[Mon Apr 15 13:34:06 2019] qdma:qdma_enable_hw_err: ecc_sb_err interrupts enabled: reg -> 0x000000f8, value = 0xffffff11

Not sure if the above will affect adevrsely. Anyways when the DMA Engine in hardware is transfering the data to the qdma driver we can see 

pr_debug("Data IRQ fired on PF#%d: index=%d, vector=%d\n",
xdev->func_id, vector_index, irq); of libqdma/qdma_intr.c  in kernel logs.

i.e the interrupt getting fired. But dma_from_device does not see the data?

Please advice,

Kind Regards,

Vikas L

sankalp@96  FYIP..

 

0 Kudos
6 Replies
Contributor
Contributor
199 Views
Registered: ‎10-01-2018

Re: QDMA : User space does not recieve data although interrupt gets triggered

[Mon Apr 15 17:22:22 2019] qdma:data_intr_handler: Data IRQ fired on PF#0: index=2, vector=211
Below additional debugs we have added.
[Mon Apr 15 17:22:22 2019] qdma:data_intr_direct: Inside data_intr_direct
[Mon Apr 15 17:22:22 2019] qdma:data_intr_direct: schedule_work_on CPU:47

Some relevant logs from before:

Mon Apr 15 17:01:02 2019] qdma:intr_vector_setup: qdma82000-p0000:82:00.0 requesting IRQ vector #0: vec 209, type 0, qdma82000-p0000:82:00.0-error.
[Mon Apr 15 17:01:02 2019] qdma:intr_vector_setup: qdma82000-p0000:82:00.0 requesting IRQ vector #1: vec 210, type 1, qdma82000-p0000:82:00.0-user.
[Mon Apr 15 17:01:02 2019] qdma:intr_vector_setup: qdma82000-p0000:82:00.0 requesting IRQ vector #2: vec 211, type 2, qdma82000-p0000:82:00.0-data.
[Mon Apr 15 17:01:02 2019] qdma:intr_vector_setup: qdma82000-p0000:82:00.0 requesting IRQ vector #3: vec 212, type 2, qdma82000-p0000:82:00.0-data.
[Mon Apr 15 17:01:02 2019] qdma:intr_vector_setup: qdma82000-p0000:82:00.0 requesting IRQ vector #4: vec 213, type 2, qdma82000-p0000:82:00.0-data.
[Mon Apr 15 17:01:02 2019] qdma:intr_vector_setup: qdma82000-p0000:82:00.0 requesting IRQ vector #5: vec 214, type 2, qdma82000-p0000:82:00.0-data.
[Mon Apr 15 17:01:02 2019] qdma:intr_vector_setup: qdma82000-p0000:82:00.0 requesting IRQ vector #6: vec 215, type 2, qdma82000-p0000:82:00.0-data.
[Mon Apr 15 17:01:02 2019] qdma:intr_vector_setup: qdma82000-p0000:82:00.0 requesting IRQ vector #7: vec 216, type 2, qdma82000-p0000:82:00.0-data.

 

[root@fgpa fpga_temp]# dmactl qdma82000 stat
qdma82000:statistics
Total MM H2C packets processed = 0
Total MM C2H packets processed = 0
Total ST H2C packets processed = 0
Total ST C2H packets processed = 1
[root@fgpa fpga_temp]#

 

I believe the below code in dma_xfer_utils.c ::read_to_buffer should have returned the data.

/* read data from file into memory buffer */
rc = read(fd, buf, bytes);
if (rc < 0) {
printf(
"%s, read off 0x%lx + 0x%lx failed %zd.\n",
 fname, offset, bytes, rc);
 perror("read file");
 return -EIO;

}

instead it goes into rc < 0 and we do not receive the data.

@bethe  @coryb @kvasantr  FYIP..

Kindly advise.

 

 

 

0 Kudos
Contributor
Contributor
178 Views
Registered: ‎10-01-2018

Re: QDMA : User space does not recieve data although interrupt gets triggered

Seems like the data interrupt gets invoked however the user interrupt does not get invoked.

Kind Regards,

Vikas L

0 Kudos
Contributor
Contributor
152 Views
Registered: ‎10-01-2018

Re: QDMA : User space does not recieve data although interrupt gets triggered

Dear XILINX Tech Support,

I am now debugging inside the QDMA Kernel driver using ftrace and trace-cmd. Unfortunately have to get it to work the hard way. I hope someone takes a look at the issue from your side as well.

Kind Regards,
Vikas L

0 Kudos
Contributor
Contributor
138 Views
Registered: ‎10-01-2018

Re: QDMA : User space does not recieve data although interrupt gets triggered

Some more debugging info. May be best to look at the below information,I'll take the effort. Hoping a XILINX Moderator  also takes an effort to look at it or atleast directs it to the correct person.

Software/Host Side:
1. Kernel built using:
    make pf -- just one pf for now.
    make tools
    make install
2. module inserted using modprobe qdma mode=2
3. Enabled kernel tracing and using trace-cmd figured out the flow inside the driver 1. when a message arrives from PCI Express and 2. when dma_from_device.c using dma_xfer_utils.c pulls out data calling read on the fd . Probably I understand the qdma driver code well now.
4. The below are facts of whats going on.
a. When the driver is loaded , queue is added and started I can see the SOFT IRQs getting created in logs and also verified then in /proc/ file system see below.
[root@fgpa 210]# ls
affinity_hint qdma82000-p0000:82:00.0-user smp_affinity_list
node smp_affinity spurious
[root@fgpa 210]# pwd
/proc/irq/210
[root@fgpa 210]#

b. [root@fgpa tmp]# dmactl qdma82000 stat
qdma82000:statistics
Total MM H2C packets processed = 0
Total MM C2H packets processed = 0
Total ST H2C packets processed = 0
Total ST C2H packets processed = 3
[root@fgpa tmp]# ls -latrh

The packets are increasing when dma_from_device is called.
d. "dma_from_device -d /dev/qdma82000-ST-0 -s 8"  is the actual command.
Please note the data is sent into the QDMA IP when this command is waiting on the read.

e. Further debugging using trace-cmd etc shows me  that the 
 qdma_request_submit_st_c2h() in libqdma/libqdma_export.c returns an -EIO which is propogated to the same via qdma_request_wait_for_cmpl().

On the hardware side I mapped the Completion signals and the other main data signal and dont see anything wrong in it ?
Can some one in XILINX suggest  as really I've done a lot to get us up and running on QDMA both the hardware and software side.Happy to provide any more information needed. But really need the engineer who wrote QDMA to look at it. This issue has been around for quite some while and is a SHOW STOPPER for progress.

cmpltion.png

 

data.png

 

Kind Regards,

Vikas L

0 Kudos
Contributor
Contributor
120 Views
Registered: ‎10-01-2018

Re: QDMA : User space does not recieve data although interrupt gets triggered

@bethe  @deepeshm @kvasantr @coryb @borisq  Please kindly note this issue is a showstopper for us in moving ahead on QDMA-ST and more so I have to get it to work . I believe we are nearly there , we have tried on our side as best as we can but in the end I feel need some quick help.Please request someone to pick it up and help me in closing it out.Sincere apologies but I really feel QDMA is fatnastic , just need to get it on board.

Kind Regards,

Vikas L

0 Kudos
Contributor
Contributor
72 Views
Registered: ‎10-01-2018

Re: QDMA : User space does not recieve data although interrupt gets triggered

A bit more of debugging info:

When we call read on the queue file descriptor in user space eventually the function trace gets to 

qdma_st_c2h.c::desq_st_c2h_read. Iv'e added debugs in here as below:

int descq_st_c2h_read(struct qdma_descq *descq, struct qdma_request *req,
bool update_pidx, bool refill)
{
struct qdma_sgt_req_cb *cb = qdma_req_cb_get(req);
struct qdma_flq *flq = (struct qdma_flq *)descq->flq;
unsigned int pidx = flq->pidx_pend;
struct qdma_sw_sg *fsg = flq->sdesc + pidx;
struct qdma_sw_sg *tsg = req->sgl;
unsigned int fsgcnt = ring_idx_delta(descq->pidx, pidx, flq->size);
unsigned int tsgoff = cb->sg_offset;
unsigned int foff = 0;
int i = 0, j = 0;
unsigned int copied = 0;
pr_debug("VIK:0 descq->pidx: %d, pidx : %d , flq->size: %d ",descq->pidx,pidx,flq->size);
pr_debug("VIK:1 cb->sg_idx : %d",cb->sg_idx);

if (!fsgcnt)
return 0;

if (cb->sg_idx) {
for ( ; tsg && j < cb->sg_idx; j++)
tsg = tsg->next;

if (!tsg) {
pr_err("tsg error, index %u/%u.\n",
cb->sg_idx, req->sgcnt);
return -EINVAL;
}
}
pr_debug("VIK:2 fsgcnt: %d ",fsgcnt);
if(tsg){
pr_debug("VIK:3 TSG valid");
}
while ((i < fsgcnt) && tsg) {
unsigned int flen = fsg->len;
unsigned char *faddr = page_address(fsg->pg) + fsg->offset;

foff = 0;
pr_debug("VIK:2.1 flen: %d",flen);

while (flen && tsg) {
unsigned int toff = tsg->offset + tsgoff;
unsigned int copy = min_t(unsigned int, flen,
tsg->len - tsgoff);

if (!req->no_memcpy)
memcpy(page_address(tsg->pg) + toff,
faddr, copy);

faddr += copy;
flen -= copy;
foff += copy;
tsgoff += copy;
copied += copy;
pr_debug("VIK:2.2 copied: %d",copied);

if (tsgoff == tsg->len) {
tsg = tsg->next;
tsgoff = 0;
j++;
}
}

if (foff == fsg->len) {
struct qdma_sdesc_info *sinfo = flq->sdesc_info + pidx;

pr_debug("VIK:2.3 foff == fsg->len : %d",fsg->len);

if (sinfo->f.eop)
descq->cidx_cmpt_pend = sinfo->cidx;

pidx = ring_idx_incr(pidx, 1, descq->conf.rngsz);

i++;
foff = 0;
fsg = fsg->next;
}
}

pr_debug("VIK:3");
incr_cmpl_desc_cnt(descq, i);

if (refill && i){
pr_debug("VIK: Refilling");
qdma_flq_refill(descq, flq->pidx_pend, i, 1, GFP_ATOMIC);

}

flq->pidx_pend = ring_idx_incr(flq->pidx_pend, i, flq->size);
if (foff) {
fsg->offset += foff;
fsg->len -= foff;
}
pr_debug("VIK:4");
if (i && update_pidx) {
i = ring_idx_decr(flq->pidx_pend, 1, flq->size);
descq_c2h_pidx_update(descq, i);
}

cb->sg_idx = j;
cb->sg_offset = tsgoff;
cb->left -= copied;

flq->pkt_dlen -= copied;
pr_debug("VIK:copied is %d",copied);
return copied;
}

 

below are the dmesg logs:

[Wed Apr 24 19:33:38 2019] qdma:qdma_c2h_packets_proc_dflt: VIK: Calling descq_st_c2h_read
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:0 descq->pidx: 1, pidx : 0 , flq->size: 2048
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:1 cb->sg_idx : 0
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:2 fsgcnt: 1
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:3 TSG valid
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:2.1 flen: 0
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:2.3 foff == fsg->len : 0
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:3
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:4
[Wed Apr 24 19:33:38 2019] qdma:descq_st_c2h_read: VIK:copied is 0

Logically debugging I would expect flen should be equal to be greater than 0. However it is 0.

Please may I request a look into the same.

Kind Regards,
Vikas L

 

 

0 Kudos