cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Visitor
Visitor
821 Views
Registered: ‎10-21-2020

Target locked error when programming multiple FPGAs parallely

We have a setup where two Xilinx FPGA xc7v2000t devices are individually connected to a computer through dedicated cables. We use Vivado Tcl Console (v2020.1 ) to program the FPGAs in our automation environment. Before programming, both the devices are power-cycled. I am able to successfully program the two devices in sequence one after the other. However, I am trying to reduce the programming time by launching two vivado instances in parallel and program each device simultaneously. While doing so, I get this error randomly – Targets(s) ", jsn-JTAG-SMT2-XXXXXXXXjsn-JTAG-HS3-XXXXXXXX" may be locked by another hw_server.

I am looking for help to be able to program the devices parallelly 100% of the time without any issues.

Here is my python script that launches hw_server and cs_server on different ports and programs FPGAs parallelly. Hw_server.exe, cs_server.exe and vivado.exe are killed (if running) before launching this python script.

 

def program(hw_server_cmd, cs_server_cmd, cmd):

    cs_server_process = subprocess.Popen(cs_server_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, encoding='utf8')

    cs_server_process.wait()

    print("cs_server started")

    hw_server_process = subprocess.Popen(hw_server_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, encoding='utf8')

    hw_server_process.wait()

    print("hw_server started")

    p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, encoding='utf8')       

    p.wait()

    stdout, stderr = p.communicate()

    p.wait()

    hw_server_process.kill()

    cs_server_process.kill()

    print('Command [{}] return value: {}'.format(cmd, p.returncode))

    print('stdout for cmd [{}]: {}'.format(cmd, stdout))

    print('stderr for cmd [{}]: {}'.format(cmd, stderr))

 

if __name__ == '__main__':

    cs_server_cmd_1 = r"C:\Xilinx\Vivado\2020.1\bin\cs_server.bat -d -stcp::3041"

    hw_server_cmd_1 = r"C:\Xilinx\Vivado\2020.1\bin\unwrapped\win64.o\hw_server.exe -d -stcp::3121  -e \"set jtag-port-filter XXXXXXXXXX\""

    cmd_1 = r'C:\Xilinx\Vivado\2020.1\bin\vivado.bat -mode tcl -source ProgFPGA_1.tcl'

 

    cs_server_cmd_2 = r"C:\Xilinx\Vivado\2020.1\bin\cs_server.bat -d -stcp::3042"

    hw_server_cmd_2 = r"C:\Xilinx\Vivado\2020.1\bin\unwrapped\win64.o\hw_server.exe -d -stcp::3122  -e \"set jtag-port-filter YYYYYYYYYYYY\""

    cmd_2 = r'C:\Xilinx\Vivado\2020.1\bin\vivado.bat -mode tcl -source ProgFPGA_2.tcl'

 

    fpga_prog_threads = {}   

    fpga_prog_threads[0] = threading.Thread(name=f'Programming FPGA on Platform 1', target=program, args=[hw_server_cmd_1, cs_server_cmd_1,  cmd_1])

    fpga_prog_threads[1] = threading.Thread(name=f'Programming FPGA on Platform 2', target=program, args=[hw_server_cmd_2, cs_server_cmd_2, cmd_2])

    for t in fpga_prog_threads.values():

        t.start()

    join_all_threads(list(fpga_prog_threads.values()), 3600)

 

Here is one of the tcl scripts I used. The other file also contains same set of commands but different port numbers and Serial number.

    open_hw_manager

    connect_hw_server -url 127.0.0.1:3121 -cs_url 127.0.0.1:3041 -allow_non_jtag

    current_hw_server 127.0.0.1:3121

    current_hw_target [get_hw_targets */xilinx_tcf/Digilent/XXXXXXXXXX]

    open_hw_target

    current_hw_device [lindex [get_hw_devices] 0]

    refresh_hw_device -update_hw_probes false [lindex [get_hw_devices] 0]   

    set_property PROBES.FILE {} [lindex [get_hw_devices] 0]

    set_property FULL_PROBES.FILE {} [lindex [get_hw_devices] 0]

    set_property PROGRAM.FILE {C:/temp/bitfile2.bit} [lindex [get_hw_devices] 0]

    program_hw_devices [lindex [get_hw_devices] 0]

    refresh_hw_device [lindex [get_hw_devices] 0]   

    exit

 

Please let me know what I need to modify in my code to avoid the error 100% of the time. 

0 Kudos
Reply
8 Replies
Xilinx Employee
Xilinx Employee
697 Views
Registered: ‎01-21-2013

Hi @KannanD,

 

There are a few posts that describe a way to simultaneously program two devices connected to two separate targets.

Does the below linked post help?

https://forums.xilinx.com/t5/FPGA-Configuration/Program-multiple-FPGAs-at-the-same-time-using-multiple-HW-USB/m-p/993881  

 

Thanks,
Wendy
Xilinx Technical Support
-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Visitor
Visitor
658 Views
Registered: ‎10-21-2020

Hi @wduffy,

That post does not help me for the following reasons.

1. I want to do load bit files in parallel across multiple devices programmatically and not manually.

2. The sample code I pasted in the question is able to program the devices in parallel, but I get target locked error frequently.

Is there anything that needs to be tweaked in my code snippet to achieve 100% success when programming bit files?

0 Kudos
Reply
Xilinx Employee
Xilinx Employee
638 Views
Registered: ‎01-21-2013

Hi @KannanD,

 

Apologies, I didn't realize the setup worked for you some of the time. 

Do you see the error often, would it be 50% of the time or less/more?

When you experience the error, is it the case that it was successfully running the programming operation, or at a different stage? I'm interested to discover at what point in the process you receive the error. 

Would it be possible to send the log from Vivado when you experience the error?

When the cables are recognized, I'm assuming the cable names differ?

 

Thanks,
Wendy
Xilinx Technical Support
-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Visitor
Visitor
632 Views
Registered: ‎10-21-2020

Hi @wduffy,

Thanks for your quick response.

The error occurs 10% to 20% of the time. Here is the error I am getting when "open_hw_target" statement is executed in the TCL file.

Targets(s) ", jsn-JTAG-SMT2-XXXXXXXXjsn-JTAG-HS3-XXXXXXXX" may be locked by another hw_server.

There are two different devices each connected to the computer using dedicated cables. So, they have different serial numbers.

I pasted my program and tcl script in this post above. I hope that gives you an idea about the sequence of steps that are executed. 

Please let me know if you require any additional information.

Note: My program works well without any error 100% of the time if the devices are programmed one after the other. 

0 Kudos
Reply
Moderator
Moderator
589 Views
Registered: ‎02-09-2017

Hi @KannanD,

 

@wduffy and I discussed about this issue earlier today.

It seems like your script is correct. You are creating different hw_server and cs_server ports for each Vivado instance, which is the correct way of doing it.

I believe you can resolve this issue by using the jtag-port-filter command when starting the hw_server, as described in the document Vivado Programming and Debugging - UG908 (v2020.1), pg. 396.

with this command, the hw_server will only connect to cables matching the provided serial number, for instance:

hw_server -e "set jtag-port-filter Xilinx/DLC10/0000128f515601"

 

The command above will only allow hw_server to connect and communicate to devices that are connected via a DLC10 cable with serial number 0000128f515601.

So in your case, I suggest you modify the script#1 with the serial number for cable#1, and similarly for the script#2.

Could you please try this approach and let us know if that works for you?

Thank you,

Andre Guerrero

Product Applications Engineer

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Visitor
Visitor
325 Views
Registered: ‎10-21-2020

@anunesgu @wduffy - Apologies for delayed response.

XXXXXXXX and YYYYYYYYYYYY in my sample script represent the last portion of the serial number. The documentation says that the filter string can contain any portion of the serial number. So I chose the last portion (0000128f515601 in the above example) which is unique and I believe it should work.

Please let me know if I need to try something else. 

 

0 Kudos
Reply
Visitor
Visitor
120 Views
Registered: ‎10-21-2020

@wduffy @anunesgu - Can you please tell me if you have any updates for me?

0 Kudos
Reply
Moderator
Moderator
78 Views
Registered: ‎02-09-2017

Hi @KannanD ,

Yes, I believe that using just the XXXXXXXX  of the cable number should work.

Did you have the chance to try it? Please let us know if it still doesn't work.

Thank you,

Andre Guerrero

Product Applications Engineer

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Reply