cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Adventurer
Adventurer
1,989 Views
Registered: ‎04-01-2015

ZCU111 Fan Oddities

Jump to solution

The FPGA fan on the ZCU111 is handled quite differently than other Xilinx boards that I've used.  First there's the issue that the FPGA has no way of monitoring nor controlling the fan speed -- the fan tach and PWM controls are taken only to a MAX6643 fan controller. 

The wiring of the fan controller seems rather odd as well.  The FANFAIL_B line is tied to the FULLSPD line.  So when the fan has not failed these lines will be pulled high and the fan will operate at full speed.   So what's even the point of having the FPGA temperature sensor (DXP/DXN) connected?   

Could it be that the wiring of FANFAIL_B to FULLSPD may have been an error?

Precluding the ability for an application to monitor and report fan speed seems like an oversight, too. 

0 Kudos
1 Solution

Accepted Solutions
Highlighted
757 Views
Registered: ‎02-22-2018

Re: ZCU111 Fan Oddities

Jump to solution

First, I missed the pullup resistor.  It's there in the schematic, and my eyes went right over it.  duh!

Second, I've verified that a software fix will work.  I'm running Ubuntu linux on the board.  The typical configuration file for devices has this in it:

&i2c0 {
	status = "okay";
	clock-frequency = <400000>;

	tca6416_u22: gpio@20 {
		compatible = "ti,tca6416";
		reg = <0x20>;
		gpio-controller; /* interrupt not connected */
		#gpio-cells = <2>;
		/*
		 * IRQ not connected
		 * Lines:
		 * 0 - MAX6643_OT_B
		 * 1 - MAX6643_FANFAIL_B
		 * 2 - MIO26_PMU_INPUT_LS
		 * 4 - SFP_SI5382_INT_ALM
		 * 5 - IIC_MUX_RESET_B
		 * 6 - GEM3_EXP_RESET_B
		 * 10 - FMCP_HSPC_PRSNT_M2C_B
		 * 11 - CLK_SPI_MUX_SEL0
		 * 12 - CLK_SPI_MUX_SEL1
		 * 16 - IRPS5401_ALERT_B
		 * 17 - INA226_PMBUS_ALERT
		 * 3, 7, 13-15 - not connected
		 */
	};

From that I realized that the i2c for this device is being handled by the gpio controller.  This can be accessed using the linux command line through the device files in /sys/class/gpio, as explained here:

https://www.ics.com/blog/gpio-programming-using-sysfs-interface

I ran this command:

ross@rf:~$ ls -l /sys/class/gpio/
total 0
--w------- 1 root root 4096 Mar 21 19:24 export
lrwxrwxrwx 1 root root    0 Mar 21 19:24 gpiochip309 -> ../../devices/platform/amba/ff020000.i2c/i2c-0/0-0020/gpio/gpiochip309
lrwxrwxrwx 1 root root    0 Mar 21 19:24 gpiochip325 -> ../../devices/platform/amba/ff0a0000.gpio/gpio/gpiochip325
lrwxrwxrwx 1 root root    0 Mar 21 19:24 gpiochip499 -> ../../devices/platform/amba_pl@0/a00c1000.gpio/gpio/gpiochip499
lrwxrwxrwx 1 root root    0 Mar 21 19:24 gpiochip507 -> ../../devices/platform/amba_pl@0/a00c0000.gpio/gpio/gpiochip507
--w------- 1 root root 4096 Mar 21 19:24 unexport

This shows me that gpiochip309 is an i2c device.  I can check if it's the right device as follows:

root@rf:~# cat /sys/class/gpio/gpiochip309/label 
tca6416
root@rf:~# 

It indeed is the tca6416 chip.  I believe there's only one of these, so it's the one.

Next, we need to access the pin.  We make control of it available and then set it to an output like so:

root@rf:~# echo 310 > /sys/class/gpio/export 
root@rf:~# echo "out" >/sys/class/gpio/gpio310/direction 

That was all I needed to do.  The fan stopped whining at max speed.  What's the 310?  It's is I/O number 1 of the chip that I want to change, and the chip base is 309 (from its name "gpiochip309").  So 310 is I/O 1 of the chip.  Apparently 0 is the default output value of the pin, so as soon as I changed it to an output a 0 was sent out that fixed the fan.  Now that this pin is set to an output you can also change the output value of the pin with commands like these:

root@rf:~# echo 1 >/sys/class/gpio/gpio310/value 
root@rf:~# echo 0 >/sys/class/gpio/gpio310/value 

Setting to a 1 gives terrible fan noise, and setting to a 0 gives a more sane fan speed.

This info lets you set the fan to a sane speed when running Linux.

There may be additional software fixes needed, if *anything* else is actually monitoring this pin.  Is anyone aware of any software monitoring that's being performed to check for a failed fan and take some action?  I'm guessing there isn't any monitoring of this sort.  My Linux seems to be running along fine with the normal fan operation.

I'd like to also be able to apply this fix when booting from jtag and *not* running Linux.  Is there a way to set i2c registers from jtag?  I've never had to do this.  Perhaps someone can help with how to set pin 1 of U22 (TCA6416A) to an output and set the output to zero via i2c controlled from jtag.

View solution in original post

17 Replies
Highlighted
Xilinx Employee
Xilinx Employee
1,914 Views
Registered: ‎06-21-2018

Re: ZCU111 Fan Oddities

Jump to solution

Hi norume,

Thanks for the heads up! I don't have a ZCU111 with me, but we'll check whether the wiring is correct on our side.

Regards,

Andres

 

0 Kudos
Highlighted
Adventurer
Adventurer
1,910 Views
Registered: ‎04-01-2015

Re: ZCU111 Fan Oddities

Jump to solution

Thanks.  I hope that the next version of this board, should there be one, provides the FPGA some mechanism for monitoring the fan speed.  Either a fan tachometer signal to the FPGA or perhaps a more capable fan controller chip with an I2C connection to the FPGA.

0 Kudos
Xilinx Employee
Xilinx Employee
1,898 Views
Registered: ‎06-21-2018

Re: ZCU111 Fan Oddities

Jump to solution

Hi norume,

I'll pass the information regarding the FPGA being able to control the fan to the design team, especially since it was available in previous boards.

However, the FANFAIL and FULLSPD being tied together seems to be correct, since both signals are active-low.

It's the Datasheet's Typical Operating Circuit (page 15):

https://datasheets.maximintegrated.com/en/ds/MAX6643-MAX6645.pdf

Thanks,

Andres

 

 

0 Kudos
Highlighted
Adventurer
Adventurer
1,886 Views
Registered: ‎04-01-2015

Re: ZCU111 Fan Oddities

Jump to solution

Yeah, I think that the example circuits on pages 11 and 15 imply something that isn't really true.

The pin table on page 4 has:

6   FULLSPD    Active-High Logic Input. When pulled high, the fan runs at 100% duty cycle.

 

And from the text on page 7:

FULLSPD

The MAX6643 features a FULLSPD input. Pulling FULLSPD high forces PWM_OUT to 100% duty cycle. The FULLSPD input allows a microcontroller to force the fan to full speed when necessary.  By connecting FANFAILto an inverter, the MAX6643 can force other fans to 100% in multifan systems, or for an over-temperature condition (by connecting OT inverter to FULLSPD).

 

And the selection guide on page 14 shows that the MAX6643 has an active-high FULLSPD pin.

AFAICT the existence of an active-low FULLSPD pin is in the mind of the data sheet writer only!  There may have been plans for a polarity-selection pin somewhere, or perhaps another member of the chip family with an active-low FULLSPD signal, but everything in the data sheet indicates that the MAX6643 FANSPD pin is active-high.

Highlighted
Xilinx Employee
Xilinx Employee
1,870 Views
Registered: ‎06-21-2018

Re: ZCU111 Fan Oddities

Jump to solution

You clearly had read through the datasheet more thoroughly than I had!

I'll inquire the board designer about this, but your explanation seems accurate.

Thanks,

Andres

 

0 Kudos
Highlighted
Moderator
Moderator
1,823 Views
Registered: ‎10-19-2011

Re: ZCU111 Fan Oddities

Jump to solution

Unless there is something that is critically broken, we don't have a new revision in the works for this board.

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Highlighted
Adventurer
Adventurer
1,815 Views
Registered: ‎04-01-2015

Re: ZCU111 Fan Oddities

Jump to solution

Depends on your threshold of "criticality" I guess.  A stuck at full speed fan and no way for an application to directly detect fan health meets my threshold, but I realize that spinning a new revision of the board represents a pretty significant cost.

0 Kudos
Highlighted
Advisor
Advisor
1,410 Views
Registered: ‎02-12-2013

Re: ZCU111 Fan Oddities

Jump to solution

The datasheet seems to indicate that therre is a MAX6643A variant with active low FULLSPD.

By the way, this error is also present in the ZCU104 eval board.  That's too bad because the sound from the little fan on that board makes it unusable around people.

Please do this correctly in future boards.

----------------------------------------
DSP in hardware and software
-----------------------------------------
Highlighted
Observer
Observer
1,399 Views
Registered: ‎07-05-2016

Re: ZCU111 Fan Oddities

Jump to solution

I hope that the fix to this issue involves more than just a MAX6643 variant.  I really miss the ability for my applications to know and reoport the fan speed.  A proper fix, IMHO, would be to either move the fan control and tachometer lines back to the FPGA or to use a fan controller with an I2C interface readable by the FPGA.

Highlighted
Xilinx Employee
Xilinx Employee
1,398 Views
Registered: ‎06-21-2018

Re: ZCU111 Fan Oddities

Jump to solution

Hi pedro_uno,

The problem was originated by Maxim's datasheet showing a "Typical Operating Circuit" not consistent with the pin polarity. Unfortunately, it wasn't caught on the initial revisions.

It has been noted and document so we can avoid it in future Eval Kits.

Thanks for the feedback!

Andres

 

0 Kudos
Highlighted
864 Views
Registered: ‎02-22-2018

Re: ZCU111 Fan Oddities

Jump to solution

Is there a circuit-board mod that can be used to fix this issue, that won't compromise board cooling?

For example, can I break the connection to the FULLSPD line?  Or break the connection to FULLSPD and then tie it to ground?

0 Kudos
Highlighted
Adventurer
Adventurer
859 Views
Registered: ‎04-01-2015

Re: ZCU111 Fan Oddities

Jump to solution

That's an interesting idea.  I'm not able to get to a board right now to see if the required board surgery is practical.  It certainly would address the noise issue caused by the stuck-at-high fan. 

0 Kudos
Highlighted
788 Views
Registered: ‎02-22-2018

Re: ZCU111 Fan Oddities

Jump to solution

The MAX6643 has all its pins visible.  FANFAIL_B is open drain, so connecting it directly to ground shouldn't cause a problem.  Thus bridging pin 6 (FULLSPD) to pin 7 (GND) should bring both FULLSPD and FANFAIL_B low, and disable FULLSPD altogether.  That would be a really simple fix.

HOWEVER, it appears that FANFAIL_B also goes to pin 5 of U22, which is a TCA6416A.  This is an I2C chip, which is probably used here for monitoring FANFAIL_B?  Some software that's monitoring this pin might get seriously upset if FANFAIL_B is low, since it will think the fan has failed.  Or maybe that software doesn't even exist, so nothing will care. How to tell? Maybe someone from Xilinx knows whether anything monitors this pin?

If pin of U22 is actually monitored, a more complicated hardware fix might be required, perhaps breaking a trace (if the trace is even visible).

It's odd that there is no pullup resistor.  Perhaps pin 5 of U22 has an internal pullup?

I have to seriously look at U22, and how it is controlled in software.  It looks like its pins can be configured either as inputs or outputs.  So perhaps pin 5 could be configured as an output and brought low through I2C -- meaning there may be a purely software fix to this fan issue.

It's important to know if any Xilinx software is monitoring this pin and will cause issues with some of these possible fixes.

Highlighted
758 Views
Registered: ‎02-22-2018

Re: ZCU111 Fan Oddities

Jump to solution

First, I missed the pullup resistor.  It's there in the schematic, and my eyes went right over it.  duh!

Second, I've verified that a software fix will work.  I'm running Ubuntu linux on the board.  The typical configuration file for devices has this in it:

&i2c0 {
	status = "okay";
	clock-frequency = <400000>;

	tca6416_u22: gpio@20 {
		compatible = "ti,tca6416";
		reg = <0x20>;
		gpio-controller; /* interrupt not connected */
		#gpio-cells = <2>;
		/*
		 * IRQ not connected
		 * Lines:
		 * 0 - MAX6643_OT_B
		 * 1 - MAX6643_FANFAIL_B
		 * 2 - MIO26_PMU_INPUT_LS
		 * 4 - SFP_SI5382_INT_ALM
		 * 5 - IIC_MUX_RESET_B
		 * 6 - GEM3_EXP_RESET_B
		 * 10 - FMCP_HSPC_PRSNT_M2C_B
		 * 11 - CLK_SPI_MUX_SEL0
		 * 12 - CLK_SPI_MUX_SEL1
		 * 16 - IRPS5401_ALERT_B
		 * 17 - INA226_PMBUS_ALERT
		 * 3, 7, 13-15 - not connected
		 */
	};

From that I realized that the i2c for this device is being handled by the gpio controller.  This can be accessed using the linux command line through the device files in /sys/class/gpio, as explained here:

https://www.ics.com/blog/gpio-programming-using-sysfs-interface

I ran this command:

ross@rf:~$ ls -l /sys/class/gpio/
total 0
--w------- 1 root root 4096 Mar 21 19:24 export
lrwxrwxrwx 1 root root    0 Mar 21 19:24 gpiochip309 -> ../../devices/platform/amba/ff020000.i2c/i2c-0/0-0020/gpio/gpiochip309
lrwxrwxrwx 1 root root    0 Mar 21 19:24 gpiochip325 -> ../../devices/platform/amba/ff0a0000.gpio/gpio/gpiochip325
lrwxrwxrwx 1 root root    0 Mar 21 19:24 gpiochip499 -> ../../devices/platform/amba_pl@0/a00c1000.gpio/gpio/gpiochip499
lrwxrwxrwx 1 root root    0 Mar 21 19:24 gpiochip507 -> ../../devices/platform/amba_pl@0/a00c0000.gpio/gpio/gpiochip507
--w------- 1 root root 4096 Mar 21 19:24 unexport

This shows me that gpiochip309 is an i2c device.  I can check if it's the right device as follows:

root@rf:~# cat /sys/class/gpio/gpiochip309/label 
tca6416
root@rf:~# 

It indeed is the tca6416 chip.  I believe there's only one of these, so it's the one.

Next, we need to access the pin.  We make control of it available and then set it to an output like so:

root@rf:~# echo 310 > /sys/class/gpio/export 
root@rf:~# echo "out" >/sys/class/gpio/gpio310/direction 

That was all I needed to do.  The fan stopped whining at max speed.  What's the 310?  It's is I/O number 1 of the chip that I want to change, and the chip base is 309 (from its name "gpiochip309").  So 310 is I/O 1 of the chip.  Apparently 0 is the default output value of the pin, so as soon as I changed it to an output a 0 was sent out that fixed the fan.  Now that this pin is set to an output you can also change the output value of the pin with commands like these:

root@rf:~# echo 1 >/sys/class/gpio/gpio310/value 
root@rf:~# echo 0 >/sys/class/gpio/gpio310/value 

Setting to a 1 gives terrible fan noise, and setting to a 0 gives a more sane fan speed.

This info lets you set the fan to a sane speed when running Linux.

There may be additional software fixes needed, if *anything* else is actually monitoring this pin.  Is anyone aware of any software monitoring that's being performed to check for a failed fan and take some action?  I'm guessing there isn't any monitoring of this sort.  My Linux seems to be running along fine with the normal fan operation.

I'd like to also be able to apply this fix when booting from jtag and *not* running Linux.  Is there a way to set i2c registers from jtag?  I've never had to do this.  Perhaps someone can help with how to set pin 1 of U22 (TCA6416A) to an output and set the output to zero via i2c controlled from jtag.

View solution in original post

Highlighted
673 Views
Registered: ‎02-22-2018

Re: ZCU111 Fan Oddities

Jump to solution

After trying the software fan fix for a while and verifying there are no issues, I tried the hardware fix.  Just make a solder bridge between pins 6 and 7 of the MAX6643, and the fan is fixed.  This is the before picture:

Before the hardware fixBefore the hardware fix

Below is the picture after the fix:

Fixed by bridging pins 6 and 7 with solder.Fixed by bridging pins 6 and 7 with solder.

It seems to be working great so far.  It also shouldn't be too hard to remove the solder bridge if it turns out there is some software that monitors that pin and gets upset.  Right now, I don't think there are any issues, and so I'm golden. 

The fan now works like the documentation says -- it goes high speed for a few seconds at powerup, then settles down to a much quieter and more relaxing speed.

Highlighted
Adventurer
Adventurer
520 Views
Registered: ‎04-01-2015

Re: ZCU111 Fan Oddities

Jump to solution

The software fix is working great here.  My applications use the standalone support so I'm confident that there's nothing else attempting to use the line in question as an input.

Thanks very much for an elegant solution to this issue.

Highlighted
Xilinx Employee
Xilinx Employee
498 Views
Registered: ‎06-21-2018

Re: ZCU111 Fan Oddities

Jump to solution

Thank you both for your input! Can you mark the solution as Accepted Solution so others can benefit from it?

Best regards,
Andres

0 Kudos