Summary

On 9 Jul 2026 , it was reported that some The Things Indoor Gateway Pro (TTIG Pro) gateways lost connection to the LNS for some of the tenants in EU2 region. This was triggered by certificate renewal issue. Once the remediation was in place, the affected gateways started reconnecting.

Impact

• Some gateways got disconnected for some of the tenants in EU2 region.
• The number of connected gateways kept declining gradually.
• Eventually, once the fix was in place, the affected gateways have reconnected and service has recovered without intervention.

Root Cause

The Let's Encrypt R12 certificate serving port 8889 expired on Jul 8 2026 at 05:54 UTC. This port is used exclusively by TTIG Pro gateways for their WebSocket connection to the LNS.

Certbot had already renewed the certificate before expiry and stored the new YR-signed certificate in the proxy_tls secret in AWS Secrets Manager. However, the secret was not synced by the GatewayProxy (Envoy) service. As a result, Envoy continued to serve the expired R12 certificate from memory, causing all TTIG Pro TLS handshakes to fail with certificate verification error.

The same expired-cert condition was present on AU1 and AS1 (no TTIG Pro gateways in those regions, so no customer impact) and was approaching on NAM1 (cert expiring Jul 14).

Resolution

The fix was applied by restarting the GatewayProxy ECS task on each affected cluster, which forced Envoy to re-read the proxy_tls secret from Secrets Manager on startup and load the already-renewed YR certificate.

Prevention / Long-term improvements

Investigate why the renewed certificate was not synchronized to Envoy and implement a permanent fix to ensure certificate updates are automatically reloaded before certificate expiry. Additionally, improve monitoring and alerting to detect certificate synchronization issues before they can impact customer traffic.