cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
syamcr
Visitor
Visitor
814 Views
Registered: ‎10-24-2018

PCIe DMA driver (xdma) kernel module crashes

I have successfully built the xdma kernel module from the latest sources (git). But when I insert the module, it crashes.

Kernel versions: 4.15.0-76-generic, 5.0.0-23-generic (former is a physical PC with the VC707 board, latter is just a VM)

Crash apparently happens when xdma_threads_create() calls xdma_kthread_start() 8 times in a loop. The first two iterations go fine, but the third iteration results in a crash.

dmesg output:

======================================

[ 360.387621] xdma:xdma_threads_create: xdma_threads_create
[ 360.387623] xdma:xdma_threads_create: kzalloc completed (**print added by me**)
[ 360.387838] BUG: unable to handle kernel paging request at ffffffffbee9ec20
[ 360.387870] IP: xdma_kthread_start+0x9a/0x120 [xdma]
[ 360.387878] PGD b5320e067 P4D b5320e067 PUD b5320f063 PMD fe05d5063 PTE 80003ff4acb61062
[ 360.387893] Oops: 0000 [#1] SMP PTI
[ 360.387900] Modules linked in: xdma(OE+) xt_multiport snd_hrtimer binfmt_misc nls_iso8859_1 snd_hda_codec_hdmi intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 ip6t_REJECT nf_reject_ipv6 snd_hda_codec_realtek snd_hda_codec_generic nf_log_ipv6 input_leds crypto_simd snd_hda_intel xt_hl nouveau snd_hda_codec hp_wmi wmi_bmof mxm_wmi sparse_keymap glue_helper snd_hda_core snd_hwdep cryptd ip6t_rt snd_pcm video ttm drm_kms_helper intel_cstate snd_seq_midi snd_seq_midi_event nf_conntrack_ipv6 nf_defrag_ipv6 snd_rawmidi intel_rapl_perf ipt_REJECT nf_reject_ipv4 snd_seq snd_seq_device snd_timer drm i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt nf_log_ipv4 serio_raw nf_log_common
[ 360.388005] lpc_ich xt_LOG snd shpchp xt_limit xt_tcpudp soundcore xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 mac_hid xt_conntrack mei_me mei ip6table_filter wmi tpm_infineon sch_fq_codel ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack libcrc32c parport_pc ppdev iptable_filter lp parport ip_tables x_tables autofs4 hid_generic usbhid hid psmouse e1000e ahci ptp pps_core libahci
[ 360.388070] CPU: 0 PID: 3893 Comm: insmod Tainted: G OE 4.15.0-76-generic #86-Ubuntu
[ 360.388079] Hardware name: Hewlett-Packard HP Z440 Workstation/212B, BIOS M60 v01.65 11/19/2015
[ 360.388096] RIP: 0010:xdma_kthread_start+0x9a/0x120 [xdma]
[ 360.388103] RSP: 0018:ffffa5690b0b7bf8 EFLAGS: 00010206
[ 360.388112] RAX: 000000000001ec20 RBX: ffff938fa097da20 RCX: ffffffffc0aa4e91
[ 360.388120] RDX: ffffffffbee80000 RSI: ffff938fa097da20 RDI: ffffffffc0aa20c0
[ 360.388128] RBP: ffffa5690b0b7c10 R08: ffff938fa097da24 R09: ffff938fa097da24
[ 360.388137] R10: 0000000000000277 R11: ffff938fa097da33 R12: ffff938fa097da24
[ 360.388145] R13: 0000000000000004 R14: ffffffffc0aa2070 R15: ffffffffc0aa2300
[ 360.388154] FS: 00007f6082299540(0000) GS:ffff93902fc00000(0000) knlGS:0000000000000000
[ 360.388163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 360.388171] CR2: ffffffffbee9ec20 CR3: 0000000fe052e003 CR4: 00000000001606f0
[ 360.388179] Call Trace:
[ 360.388193] xdma_threads_create+0xee/0x160 [xdma]

===========================================

The second VM does not have any FPGA hardware. For that 5.0.0 kernel, the driver source had to be modified as per: https://forums.xilinx.com/t5/PCIe-and-CPM/PCIe-DMA-driver-compilation-issues-in-Linux-Ubuntu-19-04/td-p/1022239 to get it to compile.

How can I fix this problem? And why 8?

Thanks in advance.

0 Kudos
Reply
4 Replies
yosana
Observer
Observer
732 Views
Registered: ‎05-31-2018

I have same problem on Ubuntu 18 (4.15.0-76-generic #86-Ubuntu), but crash happens in fist xdma_kthread_create() in my case.

dmesg output:

[ 63.536210] xdma: loading out-of-tree module taints kernel.
[ 63.536270] xdma: module verification failed: signature and/or required key missing - tainting kernel
[ 63.536795] xdma:xdma_mod_init: Xilinx XDMA Reference Driver xdma v2019.2.51
[ 63.536796] xdma:xdma_mod_init: desc_blen_max: 0xfffffff/268435455, sgdma_timeout: 10 sec.
[ 63.536806] xdma:xdma_threads_create: xdma_threads_create
[ 63.537365] BUG: unable to handle kernel paging request at ffffffffb9c9ec20
[ 63.537427] IP: xdma_kthread_start+0x9a/0x120 [xdma]
[ 63.537461] PGD 61a0e067 P4D 61a0e067 PUD 61a0f063 PMD 45bb59063 PTE 80003fff9e361062
[ 63.537520] Oops: 0000 [#1] SMP PTI
[ 63.537544] Modules linked in: xdma(OE+) nfsv3 nfs_acl nfs lockd grace fscache snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic joydev input_leds snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl x86_pkg_temp_thermal snd_seq_midi intel_powerclamp hid_generic mxm_wmi coretemp snd_seq_midi_event kvm_intel kvm snd_rawmidi irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc i915 snd_seq aesni_intel snd_seq_device aes_x86_64 snd_timer crypto_simd glue_helper cryptd intel_cstate drm_kms_helper intel_rapl_perf snd usbhid hid drm soundcore i2c_algo_bit fb_sys_fops syscopyarea mei_me sysfillrect mei shpchp intel_pch_thermal sysimgblt wmi video acpi_pad mac_hid sch_fq_codel parport_pc ppdev sunrpc lp parport ip_tables x_tables autofs4 e1000e ptp r8169 pps_core
[ 63.537881] ahci mii libahci
[ 63.537894] CPU: 0 PID: 1103 Comm: insmod Tainted: G OE 4.15.0-76-generic #86-Ubuntu
[ 63.537955] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z170M-ITX/ac, BIOS P7.00 10/26/2016
[ 63.537989] RIP: 0010:xdma_kthread_start+0x9a/0x120 [xdma]
[ 63.538007] RSP: 0018:ffffaa9781f27bf8 EFLAGS: 00010206
[ 63.538024] RAX: 000000000001ec20 RBX: ffff89299af5da20 RCX: ffffffffc080ae91
[ 63.538046] RDX: ffffffffb9c80000 RSI: ffff89299af5da20 RDI: ffffffffc08080c0
[ 63.538068] RBP: ffffaa9781f27c10 R08: ffff89299af5da24 R09: ffff89299af5da24
[ 63.538091] R10: 00000000000003cb R11: ffff89299af5da33 R12: ffff89299af5da24
[ 63.538113] R13: 0000000000000004 R14: ffffffffc0808070 R15: ffffffffc0808300
[ 63.538135] FS: 00007fc79691c540(0000) GS:ffff8929b5c00000(0000) knlGS:0000000000000000
[ 63.538160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 63.538178] CR2: ffffffffb9c9ec20 CR3: 0000000460f70003 CR4: 00000000003606f0
[ 63.538209] Call Trace:
[ 63.538233] xdma_threads_create+0xd7/0x140 [xdma]
[ 63.538264] ? 0xffffffffc040e000
[ 63.538278] xdma_cdev_init+0x6c/0x90 [xdma]
[ 63.538295] xdma_mod_init+0x60/0x1000 [xdma]
[ 63.538312] do_one_initcall+0x52/0x19f
[ 63.538326] ? _cond_resched+0x19/0x40
[ 63.538340] ? kmem_cache_alloc_trace+0xa6/0x1b0
[ 63.538357] ? do_init_module+0x27/0x213
[ 63.538371] do_init_module+0x5f/0x213
[ 63.538399] load_module+0x16bc/0x1f10
[ 63.538415] ? ima_post_read_file+0x96/0xa0
[ 63.538432] SYSC_finit_module+0xfc/0x120
[ 63.538448] ? SYSC_finit_module+0xfc/0x120
[ 63.538465] SyS_finit_module+0xe/0x10
[ 63.538480] do_syscall_64+0x73/0x130
[ 63.538495] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 63.538513] RIP: 0033:0x7fc796435839
[ 63.538527] RSP: 002b:00007ffdaddca148 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 63.538553] RAX: ffffffffffffffda RBX: 000055827d24d780 RCX: 00007fc796435839
[ 63.538577] RDX: 0000000000000000 RSI: 000055827b36cd2e RDI: 0000000000000003
[ 63.538601] RBP: 000055827b36cd2e R08: 0000000000000000 R09: 00007fc796708000
[ 63.538625] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000
[ 63.538650] R13: 000055827d24fe90 R14: 0000000000000000 R15: 0000000000000000
[ 63.538674] Code: dd e3 ec f7 0f b7 53 14 48 c7 c0 20 ec 01 00 4d 89 e0 48 c7 c1 91 ae 80 c0 48 89 de 48 c7 c7 c0 80 80 c0 48 8b 14 d5 e0 b6 7a b9 <8b> 14 02 e8 1e 99 ea f7 48 3d 00 f0 ff ff 48 89 43 48 77 1e 0f
[ 63.538757] RIP: xdma_kthread_start+0x9a/0x120 [xdma] RSP: ffffaa9781f27bf8
[ 63.538781] CR2: ffffffffb9c9ec20
[ 63.538795] ---[ end trace 623182c6a77702c3 ]---

0 Kudos
Reply
yosana
Observer
Observer
709 Views
Registered: ‎05-31-2018

FIY, XDMA driver HEAD commit 8b8c70b697f049649d5fa99be9c6bc4302d89ac9 is working for me now.

syamcr
Visitor
Visitor
693 Views
Registered: ‎10-24-2018

@yosanaThanks a ton for that info.

I too can confirm that 8b8c70b697f049649d5fa99be9c6bc4302d89ac9 is compiling fine and does not crash when the module is inserted. I have not tried actual data transfer yet though. @yosana Have you been able to successfully transfer data with this version?

0 Kudos
Reply
yosana
Observer
Observer
675 Views
Registered: ‎05-31-2018

Yes, I've tested with Gen3 x8 XDMA core generated by Vivado 2019.2 on my KCU1500 board (XCKU115 FPGA). But my test was just 4KB send/recv.