The Things Gateway Controller was down intermittently for a total duration of 56 minutes between 2025/10/16 13:12 UTC and 14:42 UTC. This also caused some gateways to disconnect from The Things Stack.
The issue started after deploying an update to The Things Gateway Controller. When the newly updated services started, all gateways try to reconnect. Due to the amount of The Things Indoor Gateway Pro gateways in the field, the number of database connections opened to connect the gateways simultaneously exceeded the maximum number of allowed database connections. For connecting one gateway, there are several parallel, independent tasks that execute database queries. Because not enough database connections were available, almost all gateways were suffering because one or more tasks could not complete, causing the gateway to disconnect. This happened in a loop.
We temporarily increased the number of allowed database connections so that all gateways could connect. In the meantime, we work on a permanent fix to cap the maximum number of database connections and put proper wait and back-off behavior in place.