GIC Setup In Baremetal AMP Architecture

[tombsj]

I have a baremetal AMP architecture (Zynq UltraScale+) that does not use OpenAMP. Here is a description of my processors:

- A53_0 (Core 0) to get interrupts from TTC for Ethernet

- A53_1 (Core 1) to get interrupts from Watchdog Timer, and a few other custom PL interrupts

- A53_2 (Core 2) to get interrupts from custom PL IP

For startup, Core 1 comes up first, sets up shared memory space, and then brings up Core 0 and Core 2.

My question is, how do I setup my GIC with this architecture? I have interrupts working for just Ethernet (Core 0), but I need to incorporate Cores 1 and 2. Currently, I only setup the GIC on Core 0, but obviously need to expand that. My current code for setting up interrupts on just one core is:

- XScuGic_LookupConfig

- XScuGic_CfgInitialize

- Xil_ExceptionRegisterHandler

- XScuGic_SetPriorityTriggerType

- XScuGic_InterruptMaptoCpu

- Xil_ExceptionEnable

- XScuGic_RegisterHandler

- XScuGic_EnableIntr

- XTtcPs_EnableInterrupts

Does each core need to have their own instance of the GIC? Do they each go through this entire process? I have been hearing that I cannot call CfgInitialize from each of the cores, because it will overwrite the configuration for the cores that have already setup the GIC.

I think that I can use my architecture to setup the GIC on Core 1 (first core up) and then just setup interrupts on Cores 0 & 2 - but there is not much out there for setting up baremetal AMP interrupts.

[tombsj · Accepted solution]

Update: Posting the solution that I came up with, since there is close to nothing available online for 2 (or more) cores all running baremetal.

The root of the problem is, to follow the programming sequence provided in documentation, you would think each core wants to setup the GIC and get a instance pointer using XScuGic_CfgInitialize. However, depending on the timing of the CPUs each calling this function, they will overwrite the settings in the distributor interface that were set by other CPUs (race conditions). Reading the Architecture Specification for the Arm Generic Interrupt Controller (v2 for Zynq Ultrascale+ MPSoC APU) each CPU has its own private "CPU Interface" registers, and shares the public distributor registers. So for the most part, the CPUs can each setup the GIC as if they were standalone (meaning not AMP architecture) except for the distributor interface.

The basic steps: 

- One CPU (pick whichever one is up first, and you are considering "master core") runs the built in function XScuGic_CfgInitialize to setup it's CPU interface with the GIC, and the Distributor interface.

- Other CPU(s) run a modified version of XScuGic_CfgInitialize, where they setup the CPU interface, but do not touch the Distributor Interface

- Other CPU(s) setup the interrupts priority, trigger type, set the function handler (ISR callback function) and then enable the interrupt.

- The CPU that setup the Distributor Interface ("master CPU") calls XScuGic_InterruptMaptoCpu to setup the distributor registers such that the interrupt is sent to the correct CPU. There is alot of talk on the forums about how the XScuGic_InterruptMaptoCpu function doesn't work, but long story short that was fixed in one of the 2017 versions, I am using 2018.2 and it works properly.

The timing will look like this for 2 cores:

Core 1 (Master)                                                        Core 0 (Slave)

XScuGic_LookupConfig

XScuGic_CfgInitialize

Xil_ExceptionRegisterHandler

Xil_ExceptionEnable

                                                                                XScuGic_LookupConfig

                                                                                (Modified CfgInitialize) (More below)

                                                                                Xil_ExceptionRegisterHandler

                                                                                Xil_ExceptionEnable
                                                                                XScuGic_SetPriorityTriggerType
                                                                                XScuGic_RegisterHandler
                                                                                XScuGic_EnableIntr
                                                                                XTtcPs_EnableInterrupts
                                                                                XScuGic_InterruptMaptoCpu

Obviously the setup multiple interrupts, some of these functions may need to be called multiple times, but the above is for setting up one interrupt vector. 

For the modified CfgInitialize function, all you need to do is skip the DistributorInit() call. Within the XScuGic_CfgInitialize function, there is a preprocessor directive that will skip DistributorInit() if "USE_AMP"  is defined in the BSP compiler flags. One word of caution, this flag affects the way L2 cache is used/managed on the cores that define USE_AMP, and it is possible that other built in libraries will be affected by the definition. I did not go this way, because I was unsure of what else USE_AMP might affect.

Instead, I wrote my own CfgInitialize function, in which I copied the meat from the XScuGic_CfgInitialize function. Function I used below:

/*

    @brief Initializes the Gic Interface for a slave core

    @details With an AMP architecture, shared peripherals 

    must be managed properly. The GIC (Generic Interrupt Controller)

    Distributor interface is to be setup by the Master Core (CPU 1).

    Unfortunately the only built in way to control this is to define the USE_AMP

    flag, but that has implications on other parts of the design. So this

    function was created and to do all of the things that XScuGic_CfgInitialize

    would do, except skip the DistributorInit step for the slave cores.

    @param pInstancePtr Pointer to the XScuGic instance

    @param pConfigPtr Pointer to a config table for the particular

    device this driver is associated with

    @param EffectiveAddr is the device base address in the virtual memory

    address space. The caller is responsible for keeping the address mapping

    from EffectiveAddr to the device physical base address unchanged once this

    function is invoked. Unexpected errors may occur if the address mapping

    changes after this function is called. If address translation is not used,

    use Config->BaseAddress for this parameters, passing the physical address

    instead

    @return XST_SUCCESS if initialization was successful

*/

static signed int ModifiedGic_CfgInitializeSlave(XScuGic *pInstancePtr,

                XScuGic_Config *pConfigPtr, unsigned int EffectiveAddr) {

    unsigned int Int_Id;

    (void) EffectiveAddr;

    Xil_AssertNonvoid(pInstancePtr != NULL);

    Xil_AssertNonvoid(pConfigPtr != NULL);

    if(pInstancePtr->IsReady != XIL_COMPONENT_IS_READY) {

        pInstancePtr->IsReady = 0U;

        pInstancePtr->Config = pConfigPtr;

        for(Int_Id = 0U; Int_Id<XSCUGIC_MAX_NUM_INTR_INPUTS; Int_Id++) {

            /*

            * Initalize the handler to point to a stub to handle an

            * interrupt which has not been connected to a handler. Only

            * initialize it if the handler is 0 which means it was not

            * initialized statically by the tools/user. Set the callback

            * reference to this instance so that unhandled interrupts

            * can be tracked.

            */

            if  ((pInstancePtr->Config->HandlerTable[Int_Id].Handler == NULL)) {

                pInstancePtr->Config->HandlerTable[Int_Id].Handler =

                                    StubHandler;

            }

            pInstancePtr->Config->HandlerTable[Int_Id].CallBackRef =

                                pInstancePtr;

        }

        XScuGic_Stop(pInstancePtr);

        //Note: Main difference between this function and XScuGic_CfgInitialize

        // is that DistributorInit is missing

        //The below two non-comment statements replace XScuGic fn CpuInitialize

        //Program the priority mask of the CPU using the Priority mask register

        XScuGic_CPUWriteReg(pInstancePtr, XSCUGIC_CPU_PRIOR_OFFSET, 0xF0U);

        /*

        * If the CPU operates in both security domains, set parameters in the

        * control_s register.

        * 1. Set FIQen=1 to use FIQ for secure interrupts,

        * 2. Program the AckCtl bit

        * 3. Program the SBPR bit to select the binary pointer behavior

        * 4. Set EnableS = 1 to enable secure interrupts

        * 5. Set EnbleNS = 1 to enable non secure interrupts

        */

        /*

        * If the CPU operates only in the secure domain, setup the

        * control_s register.

        * 1. Set FIQen=1,

        * 2. Set EnableS=1, to enable the CPU interface to signal secure interrupts.

        * Only enable the IRQ output unless secure interrupts are needed.

        */

        XScuGic_CPUWriteReg(pInstancePtr, XSCUGIC_CONTROL_OFFSET, 0x07U);

        pInstancePtr->IsReady = XIL_COMPONENT_IS_READY;

    }

    return XST_SUCCESS;

}

Hope this helps someone else, I know that my team was unsuccessful in finding any documentation for setting up interrupts in an AMP environment with all of the cores running standalone OS (no OpenAMP support).

[InetK1248]

I found a bug in the function: s32 XScuGic_CfgInitialize in xscugic.c 
The calculation of the Cpu_Id was wrong for CPUs > 1. So I changed it to the right calculation:

//u32 Cpu_Id = CpuId + (u32)1; // Here is the BUG!
u32 Cpu_Id = ((u32)0x1 << CpuId);

As the DistributorInit(InstancePtr, Cpu_Id); used this Cpu_Id the Distributor registers were configured for the wrong CPUs.

In newer versions (from v4.0 on) of xscugic.c the code is already changed a lot. 

[Rmccarty]

thank you so much for this. the xilinx documentation is vague and the online examples are old and/or useless.

turns out that commenting out the init dist call in Cpu1 and then adding a maptocpu for cpu1 in the cpu0 isr initialization did the trick.

very simple, but not at all clear from the Xilinx driver code or the trm.

thanks again.

ps using zynq 7020.