UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor rocco0815
Visitor
3,134 Views
Registered: ‎11-21-2017

How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

Hi there,

 

We are using Xilinx Vivado & SDK 2017.2 with trenzboard and a Xilinx Zynq7020.

We decided to use openAMP to get Petalinux on the first core (PS0) and a baremetal on the second core (PS1).

So by compiling Petalinux and creatig a UBOOT image, we can boot up Petalinux without problems and also our Baremetal ist working. At the moment, our BAremetal is equal to the Xilinx "echo_test". So after Petalinux has booted up, like decsribed in "ug1186" we can load the baremetal on the second core with following command:

Source "ug1186" (Oct 2017) page 37:

 

Running the Echo Test

1.   Load the Echo test firmware and RPMsg module:

 

 

echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
modprobe rpmsg_user_dev_driver

2. Run test:

 

 

echo_test

So can successfully test the echo programm (send, receive = OK).

 

 

So the problem now::

I can succesfully run the compiled program on SP1 (tested it --> echo_test is working), BUT how do i debug the baremetal program in SDK correctly?

 

Currently I am doing this:

1. Booting up Petalinux

2. Execute the following commands (as described)

 

echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
modprobe rpmsg_user_dev_driver

3. Run Debugging at SDK

 

4. Suddenly the baremtal seems to freeze at "waiting for eventes"

 

TCF Debug Virtual Terminal output:

 

Starting application....
Try to init remoteproc resource
Init remoteproc reource done
Waiting for events...

 

 

So compared to the Petalinux Bootup, after executeing Step (1)  and (2) without SDK (so the compiled baremetal on SD card is used), the absolutely exact same baremetal suddenly doesn't hang on "waiting for events", it got working "eproc remoteproc@0 is now up"

-->

 

Starting application....
Try to init remoteproc resource
Init remoteproc reource done
Waiting for events...
eproc remoteproc@0 is now up

 

Anyone got an idea, why the baremetal isn't working when debugging?

 

 

 

Btw here is my config (as described in "ug1186"):

Debug conifg:

debugConfig1.png

debugConfig2.png

debugConfig3.png

 

Also here is my BSP Setup:

bspConfig1.png

bspConfig2.png

0 Kudos
8 Replies
Moderator
Moderator
3,100 Views
Registered: ‎05-10-2017

Re: How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

Take a look at this forum thread which describes the debug setup

https://forums.xilinx.com/t5/OpenAMP/Development-JTAG-debug-approaches-for-PS1/td-p/698051

 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor rocco0815
Visitor
2,984 Views
Registered: ‎11-21-2017

Re: How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

Hello jovitac,

 

thanks for the link, I already had seen this.

The thing is, I can debug the Baremetal application, the only thing is:

 

When I run my Application booted up from SD Card, everything is working, including the RPMSG communication between PetaLinux and the Baremetal.

 

But, when I start the SDK and try to debug my Baremetal, my Baremetal suddenly "freezes" meaning:

 

The baremetal executes "hil_poll" function (code from the echo example):

LPRINTF("Waiting for events...\n");
do { hil_poll(proc->proc, 0); } while (!evt_chnl_deleted);

I've followed the "hil_poll" and got into the "static int _poll(struct hil_proc *proc, int nonblock)" at "zynq_remoteproc_a9.c".

 

So my baremetal stays forever in this loop and won't leave it:

 

code of function:

"static int _poll(struct hil_proc *proc, int nonblock)"

while(1) {
		vring = &proc->vdev.vring_info[i];
		intr_info = &(vring->intr_info);
		flags = metal_irq_save_disable();
		if (!(atomic_flag_test_and_set(
			(atomic_uint *)&(intr_info->data)))) {
			metal_irq_restore_enable(flags);
			virtqueue_notification(vring->vq);
			kicked = 1;
			if (i)
				return 0;
			i++;
		} else if (!i) {
			metal_irq_restore_enable(flags);
			i++;
		} else {
			if (kicked) {
				metal_irq_restore_enable(flags);
				return 0;
			} else if (nonblock) {
				metal_irq_restore_enable(flags);
				return -EAGAIN;
			} else {
				_rproc_wait();
				metal_irq_restore_enable(flags);
				i--;
				continue;
			}
		}
	}

 

It seems, that my baremetal is waiting for an event from Petalinux, but Petalinux isn't responding?!

BTW: My Core0 (with Petalinux) is "running" when debugging

0 Kudos
Explorer
Explorer
2,974 Views
Registered: ‎02-22-2012

Re: How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

Check also this forum and also this forum messages. It looks like your debugging BM has "missed" LX CPU0 notification kick SGI sent to BM CPU1. If this is the case, than you have to re-synchronize LX CPU0 and BM debugging CPU1 RPMSG communication channel. I did it with "JTAG CPU rendezvous" and then sending another kicking SGI from CPU0 to CPU1.

0 Kudos
Observer fbruder
Observer
2,898 Views
Registered: ‎09-11-2017

Re: How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

hey,

thank you for your answer!

My workmate and (@rocco0815) and I had a look at the two forum links but did not find out how to accomplish it with PetaLinux and a bare metal application.

Do you have experience how it can be achieved with PetaLinux?

Can you please expain it in more detail?

 

Sorry, we are very new to the howl topic.

 

Manny thanks!

 

 

0 Kudos
Explorer
Explorer
2,864 Views
Registered: ‎02-22-2012

Re: How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

Hi,

I don't use PetaLinux so I don't know details on how remoteproc is handled and supported there. I use LX from xlnx-linux tree. The problem you described above is similar to one I had (endless wait in BM FW for rpmsg to start, described in forum links I gave).

Note also that I was doing this 1+ year ago with XSDK 2015.4 and xlnx-linux-2015.4.1 (kversion 4.0). I checked today the diff of remoteproc with current xlnx-linux-2017.3 (kversion 4.9). There are changes.

But any way I will try to describe better the problem and what I did. At least you will have track to check with your problem.

The problem I had was that BM FW (CPU1) startup was too late (slow) to arrive in rpmsg initialization and start listening, so it missed the kicking SGI sent from LX (CPU0). This kicking SGI is sent by zynq_remoteproc (zynq_rproc_kick()) driver only once. My BM FW (OpenAMP echo_test example) was too late, because I JTAG debug it with "attaching to running process" method. In order to debug startup of the BM FW code I put "JTAG CPU rendezvous" while loop in the beginning of main. Once I JTAG attach to BM FW and break in, I change the value of global variable g_DebugStop, so after continue run the while will exit. Doing this, my BM FW was always to late to get to rpmsg initialization and start listening. To solve this problem I add "irq" kernel attribute to zynq_remoteproc driver that allows me to fire additional kicking SGI when I want to. This was my case.

The other case that I know about and the reason was also that BM FW (CPU1) was too late is describedin this forum message here. The problem of there was hit because the simple increment of heap size in FreeRTOS OpenAMP example, resulted in longer BM FW startup (initialization of bigger heap, longer memset initialization). Because of this additional delay in BM FW startup, it was too late to arrive in rpmsg initialization and start listening, the kicking SGI from LX was already gone missed.

Now, if you are doing something similar that would delay your BM FW startup (e.g. JTAG break in entering main or other stuff that would prolong BM FW initialization), than there are good chances that you have the same problem I had.

 

I solved the problem by adding "irq" kernel attribute (one of standard LKM user to kernel space communication) in zynq_remoteproc driver code:

 

//----------------------------------------------------------------------------//
ssize_t irq_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
{
    struct platform_device *pdev = to_platform_device(dev);
	u8 irqnum = buf[0] - '0';
	if (irqnum >= 16) {
		dev_err(&pdev->dev, "Invalid soft IRQ num=%u\n", irqnum);
	}
	else {
	    dev_info(&pdev->dev, "call gic_raise_softirq num=%u\n", irqnum);
		gic_raise_softirq(cpumask_of(1), irqnum);
	}
	return count;
}
static DEVICE_ATTR_WO(irq);
//----------------------------------------------------------------------------//

Knowing also that I need to fire SGI irqnum=15 (defined as "#define VRING0_IPI_VECT 15" in platform.h file of BM OpenAMP echo test example), I did it with with:

echo '?' > /sys/devices/soc0/amba/0.remoteproc/remoteproc0/irq

ASCII('?')-ASCII('0') is 15 and "irq" kernel attribute of zynq_remoteproc driver is exposed as "/sys/devices/soc0/amba/0.remoteproc/remoteproc0/irq".

 

Now as I said I don't know details of remoteproc in PetaLinux and also the remoteproc has changed in later kernel versions. But it is worth that you look at it and try. First find VRING0_IPI_VECT SGI interrupt vector (if this did not change) in OpenAMP BM FW application you are using. Then try with fire this SGI from LX (CPU0) to BM (CPU1).

 

Hope it will help you.

 

0 Kudos
Observer fbruder
Observer
2,761 Views
Registered: ‎09-11-2017

Re: How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

Hi,

first MANNY THANKS for your explenation! Now the links you posted have made a lot more sense to me.

 

I foud "VRING0_IPI_VECT" in my BM application in the file "platform_info.h". I think they renamed it to:

 

#define VRING0_IPI_INTR_VECT              15

I copied the "/opt/Xilinx/PetaLinux/components/yocto/source/arm/layers/meta-openamp/recipes-kernel/rpmsg-user-module" directory from the PetaLinux installation to "<PetaLinux project>/project-spec/meta-user/recipes-kernel/" so that the module is from the installation is overwritten. I made the following changes to the "rpmsg_user_dev_driver.c" file in the recipe:

 

Added this somewhere at the top:

//----------------------------------------------------------------------------//
ssize_t irq_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count)
{
    struct platform_device *pdev = to_platform_device(dev);
    u8 irqnum = buf[0] - '0';
    printk(KERN_INFO "gdg - irq - %d\n", irqnum);
    if (irqnum >= 16) {
        dev_err(&pdev->dev, "Invalid soft IRQ num=%u\n", irqnum);
    } else {
        printk(KERN_INFO "gdg - raise irq - %d\n", irqnum);
        dev_info(&pdev->dev, "call gic_raise_softirq num=%u\n", irqnum);
        gic_raise_softirq(cpumask_of(1), irqnum);
    }   
    return count;
}
static DEVICE_ATTR_WO(irq);
//----------------------------------------------------------------------------/

And in the function near the end:

static int rpmsg_user_dev_rpmsg_drv_probe(struct rpmsg_device *rpdev)
{
    struct _rpmsg_eptdev *local;
    struct device *dev;
    int ret;

    dev_info(&rpdev->dev, "%s\n", __func__);

    local = devm_kzalloc(&rpdev->dev, sizeof(struct _rpmsg_eptdev),
            GFP_KERNEL);
    if (!local)
        return -ENOMEM;

    /* Initialize locks */
    spin_lock_init(&local->queue_lock);

    /* Initialize sk_buff queue */
    skb_queue_head_init(&local->queue);
    init_waitqueue_head(&local->readq);

    local->rpdev = rpdev;
    local->ept = rpdev->ept;

    dev = &local->dev;
    device_initialize(dev);                                                                                                                                                                                        
    dev->parent = &rpdev->dev;
    dev->class = rpmsg_class;

    /* Initialize character device */
    cdev_init(&local->cdev, &rpmsg_dev_fops);
    local->cdev.owner = THIS_MODULE;

    /* Get the rpmsg char device minor id */
    ret = ida_simple_get(&rpmsg_minor_ida, 0, RPMSG_USER_DEV_MAX_MINORS,
            GFP_KERNEL);
    if (ret < 0) {
        dev_err(&rpdev->dev, "Not able to get minor id for rpmsg device.\n");
        goto error1;
    }
    dev->id = ret;
    dev->devt = MKDEV(MAJOR(rpmsg_dev_major), ret);
    dev_set_name(&local->dev, "rpmsg%d", ret);

    ret = cdev_add(&local->cdev, dev->devt, 1);
    if (ret) {
        dev_err(&rpdev->dev, "chardev registration failed.\n");
        goto error2;
    }

    /* Set up the release function for cleanup */
    dev->release = rpmsg_user_dev_release_device;

    ret = device_add(dev);
    if (ret) {
        dev_err(&rpdev->dev, "device reister failed: %d\n", ret);
        put_device(dev);
        return ret;
    }

// IRQ stuff
//----------------------------------------------------------------------------//
    /* Add irq file for rsync when debugging */
    ret = device_create_file(&local->rpdev->dev, &dev_attr_irq);
    if (ret) {
        dev_err(&rpdev->dev, "device_create_file %s %d\n", dev_attr_irq.attr.name, ret);
        goto attr_up_err;
    }
//----------------------------------------------------------------------------//

    dev_set_drvdata(&rpdev->dev, local);

    dev_info(&rpdev->dev, "new channel: 0x%x -> 0x%x!\n", rpdev->src, rpdev->dst);

    return 0;

attr_up_err:
    device_remove_file(&local->rpdev->dev, &dev_attr_irq);

error2:
    ida_simple_remove(&rpmsg_minor_ida, dev->id);
    put_device(dev);

error1:
    return ret;
}

After I build the project I got the irq file under a different directory as you described (/sys/devices/soc0/remoteproc@0/remoteproc/remoteproc0/virtio0/virtio0:rpmsg-openamp-demo-channel/irq).

 

If I now add the interrupt handler and interrupt setup method to my echo_test BM application like this:

Again somewhere at the top:

 

 

static void on_SGI(void *CallBackRef) {
	LPRINTF("Interupt\r\n");

	//reading interrupt status acknowledges pending interrupt
	#define ICCIAR (XPAR_PS7_SCUGIC_0_BASEADDR | 0x10C)
	u32 status = Xil_In32(ICCIAR);

//	status = XScuGic_SoftwareIntr(&Intc, 0, XSCUGIC_SPI_CPU0_MASK);
}

//Mostly copied from BSP auto-generated xgpiops_int_example
#define INTC_DEVICE_ID XPAR_PS7_SCUGIC_0_DEVICE_ID
static int SetupInterruptSystem() {
	int Status;
	XScuGic *Intc;
	XScuGic_Config *IntcConfig; //GIC config

	Xil_ExceptionInit();

	IntcConfig = XScuGic_LookupConfig(INTC_DEVICE_ID);
	XScuGic_CfgInitialize(&Intc, IntcConfig, IntcConfig->CpuBaseAddress);

	//connect to the HW
	Xil_ExceptionRegisterHandler(XIL_EXCEPTION_ID_INT,//== XIL_EXCEPTION_IRQ
	(Xil_ExceptionHandler)XScuGic_InterruptHandler, &Intc);
	#define SGI_NUM 15
	Status = XScuGic_Connect(&Intc, SGI_NUM, (Xil_ExceptionHandler)on_SGI,(void *)&Intc);
	if (Status != XST_SUCCESS) {
		return XST_FAILURE;
	}
	XScuGic_Enable(&Intc, SGI_NUM);

	// Enable interrupts in the Processor.
	Xil_ExceptionEnableMask(XIL_EXCEPTION_IRQ);
	return XST_SUCCESS;
}

And call "SetupInterruptSystem" from main.

 

I can raise the interrupt linke that:

 

echo '?' > /sys/devices/soc0/remoteproc@0/remoteproc/remoteproc0/virtio0/virtio0:rpmsg-openamp-demo-channel/irq

And catch it in my BM application.

 

But how do I send the SIG kick you described? Or do I have to set some status in the interrupt handler, because my BM application is stuck in a loop (measurementcontrol_bsp/ps7_cortexa9_1/libsrc/openamp_v1_3/src/open-amp/lib/rpmsg/remote_device.c, in the function "rpmsg_rdev_remote_ready") and waits for some status.

I'm quite lost. I do not know how I can accomplish that.

 

Xilinx XSDK and PetaLinux are version 2017.2.

 

Manny thanks and thank you for your patience!

0 Kudos
Explorer
Explorer
2,724 Views
Registered: ‎02-22-2012

Re: How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

The

#define VRING0_IPI_INTR_VECT              15

is the "SIG kick" interrupt.

You can send it with:

echo '?' > /sys/devices/soc0/remoteproc@0/remoteproc/remoteproc0/virtio0/virtio0:rpmsg-openamp-demo-channel/irq

You should not change your echo_test BM interrupt handler. You must leave this interrupt handling to BM rpmsg infrastructure as it was. The only change you need to do is to add "irq" kernel attribute in LX, so that you can fire "SIG kick" whenever you want.

What you did in echo_test BM (changing interrupt handler) is good only for you to see if SIG is send by LX and then cached in your BM. Test purpose only, for you to see that sending SIG works. You should revert interrupt handling changes in echo_test BM.

 

From (@rocco0815) first post I assumed that the problem is only(!) if try to debug BM, else it works OK.

 Said that I would underline this:

- Your problem can be different from what I had and solved.

- The RPMSG code in LX and in BM has changed from version where I used it (XSDK 2015.4 and LX from Xilinx linux github tree tagged with xilinx-2015.4.1).

 

Here is briefly what I did:

I modifed LX kernel xilinx-2015.4.1 configuration so that drivers: virtio, virtio_ring, remoteproc, virtio_rpmsg_bus, rpmsg_driver and zynq_remoteproc were build as loadable modules, xxx.ko files ("m" config option).

Those drivers provide LX kernel RPMSG infrastructure and OpenAMP support.

I modified zynq_remoteproc driver to add "irq" kernel attribute (you add it in your rpmsg_driver).

As BM I used XSDK openamp_test example and in LX user space I used echo_test (it opens /dev/rpmsg0 and sends data to remote core with integrity validation).

I modified BM openamp_test and add "JTAG CPU rendezvous" while loop. As I already wrote, I need that to attach with JTAG to BM application and debug it.

To recap, the only modifications I did was adding "irq" attribute in zynq_remoteproc driver and while loop in openamp_test BM.

 

Steps I executed on LX after boot where:

 

#Config RemoteProc
mkdir /lib/firmware
cp ./openamp_test.elf /lib/firmware/image_echo_openamp
insmod ./virtio.ko
insmod ./rootvirtio_ring.ko
insmod ./remoteproc.ko
#Start RemoteProc
insmod ./virtio_rpmsg_bus.ko
insmod ./rpmsg_driver.ko
insmod ./zynq_remoteproc.ko firmware=image_echo_openamp
#In this point I attached with JTAG to BM
#Note that because of JTAG CPU randezvous while loop the BM has missed kicking SGI,
# there was no /dev/rpmsg0 device created, so I sent kicking SGI with: echo '?' > /sys/devices/soc0/amba/0.remoteproc/remoteproc0/irq #Here the /dev/rpmsg0 came up (use dmesg) and available to e.g. ./echo_test rpmsg test application. ....
#In this point I detached JTAG from BM, leaving BM running. #Stop RemoteProc rmmod rpmsg_driver rmmod virtio_rpmsg_bus rmmod zynq_remoteproc #At this point I can repeat and loop steps from "Start RemoteProc" again without reboot.
#E.g. Rebuilding openamp_test.elf or using different BM application in next iteration.

 

 

 

 

0 Kudos
Observer fbruder
Observer
1,626 Views
Registered: ‎09-11-2017

Re: How to debug PS1 with SDK? Config: PS0 = Petalinux, PS1 = Baremetal

hey,

 

it is working if you issue these commands from Linux:

echo stop > /sys/class/remoteproc/remoteproc0/state
echo start > /sys/class/remoteproc/remoteproc0/state

 

So first start the baremetal application from Linux, then start debugging from the XSDK and then run the two commands above.

 

See also this thread:

https://forums.xilinx.com/t5/OpenAMP/OpenAMP-baremetal-application-debugging-problem/td-p/866530

 

Cheers!

0 Kudos