MOD-IO looses i2c connection after power cycle of controller

[kleurbleur]

Whenever the usb connection to the controller (in my case an ESP32-PoE-ISO-IND) is disconnected, the MOD-IO module can't be found anymore on i2C (Error 263). After a power cycle of the MOD-IO module it can be found again. If the USB connection does stay connected and the controller is reset via the button it does find the MOD-IO module. This is also tested on an ESP32-PoE-IND and is with the supplied code for Arduino.

Is this intended behaviour or am I missing something? 

[LubOlimex]

I think it is somehow related to powering order and voltage levels. Probably when ESP32-PoE-IND is unpowered MOD-IO attempts to power it via the I2C lines with some parasitic power. Explore that case. As a start is there anything else in the setup that can keep MOD-IO powered (fully or partially) when ESP32-PoE-IND is unpowered?

[kleurbleur]

This also happens after uploading new firmware to the ESP32's. In that case there is no power loss I think? At least there is not a physical cable connection change. In the case of the ESP32-PoE-ISO-IND it is also powered via PoE when the USB connection is disconnected.

The MOD-IO has it's own power supply (24VDC) and is powered throughout all the processen I describe. The MOD-IO will be used in a permanent installation where access is difficult and power cycling the MOD-OI is not an option.

[LubOlimex]

What are the options to avoid power cycling ESP32-PoE-ISO? Can you try if the problem persists if you attach Li-Po battery to ESP32-PoE-ISO?

[kleurbleur]

None, the installation will be powered off sometimes. If the MOD-IO is not intended to be used after a power cycle or firmware upload to the ESP32 controller it is connected to, let me know and I will take the MOD-IO out of the installation.

[LubOlimex]

I believe the issue is not in the MOD-IO or its design. I am not sure if any other board in place of MOD-IO would behave differently.

I believe it is a typical problem with two interconnected boards each of which has a power supply. What happens is - when you remove the power supply from the ESP32 board the MOD-IO2 powers partially the ESP32 board via the UEXT. So when you re-apply the supply to ESP32 the levels are improper and the whole ESP32 board doesn't work.

Is it an option to disconnect the physical connection between MOD-IO and ESP32-POE-ISO when you remove the USB? Maybe if you use a longer 10 pin ICD cable this is accomplishable? So every time MOD-IO remains powered but ESP32-POE-ISO gets unpowered you also remove the UEXT connection between them and then you connect everything in such order:

- MOD-IO is still powered and we can't reset it
- Apply USB or power to ESP32-POE-ISO
- Apply the UEXT connection

It is important to power ESP32 before applying the UEXT connection.

[LubOlimex]

Of course much more sophisticated solutions with extra circuitry to prevent that power flowing from MOD-IO to ESP32-POE-ISO is possible but would require significantly more effort.

But we have to first confirm what is the issue 100%. There is slight chance it can be some software issue related to the I2C library that hangs the whole I2C bus when one device gets disconnected during some event.

[kleurbleur]

Ok, I think I get it. So in the case when both loose their power the connection loss won't happen? Because that will be the case in the final product.

How I work with it now, is that I power cycle the MOD-IO after uploading to the ESP32. This is a bit cumbersome during debugging but doable, for the final product when it needs to power on and off by itself this is not an option. The weird thing is that both are powered in this process. The MOD-IO via it's 24V supply and the ESP32 via the USB and the PoE connection.

As an alternative I could also install a relay or mosfet which power cycles the MOD-IO whenever the ESP32 gets the i2c error. But this feels like a hacky workaround.

I will try what you suggested in your previous post.

[LubOlimex]

Hmmm, I didn't consider that PoE is enabled during this. Might be something else, like ground difference, which connector on MOD-IO do you use for the hardware connections the male or the female? Do you use a cable between ESP32-POE-ISO and MOD-IO (since if you use the female it can be directly plugged into ESP32-POE-ISO)?

[kleurbleur]

The MOD-IO is connected via the female connector (the one next the analog inputs, I get confused with male or female) with supplied UEXT cable directly to the ESP32.

[kleurbleur]

Maybe important to know too is that in production the ESP32 is connected to a PoE switch which takes about 5s to startup and power the ESP32. The MOD-IO is directly on.

ps: the title of the topic is wrong since the ESP32 never is powered down, merely resets after a firmware update.

[LubOlimex]

Can you measure the difference between grounds between MOD-IO and ESP32-POE-ISO? Do it the following way:

- Fully disconnect MOD-IO from ESP32-POE-ISO (but keep all other connections and power that are typically connected to each of the two);

- Place the probes of multimeter or voltage meter between grounds of MOD-IO and ESP32-POE-ISO

What is the voltage?

[kleurbleur]

With the probe set to 2000mV reading I get fluctuating values between 0 and 50 over a span of roughly half a second.

[LubOlimex]

It is some weird behavior I am trying to replicate it here but it definitely is weird.

[kleurbleur]

Yeah, for some reason this happens now less often. Like one in three times or so? It seems to be that when the controller is longer offline (e.g. writing to it's storage during programming the controller) it happens more often.