The issue has been identified and fixed. Everything is back operational.
Posted Sep 11, 2020 - 19:19 CEST
Update
We are continuing to monitor for any further issues.
Posted Sep 11, 2020 - 18:45 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 11, 2020 - 18:44 CEST
Identified
We're investigating an issue with downlink not being scheduled to gateways. The cluster does not correctly keep track of downlink paths to gateways, so the Network Server cannot reach a Gateway Server to which the gateway is connected. We're on it and post progress here.
Posted Sep 11, 2020 - 17:04 CEST
Update
We are deploying a patch to `eu1`, `au1` and `nam1` clusters and are monitoring results.
Posted Sep 11, 2020 - 12:51 CEST
Update
On monitoring the status overnight, we see that the application links are stable, uplinks are being processed. However, we are noticing a drop in delivered downlinks. We've traced the possible cause of the issue and are working on applying a fix.
Posted Sep 11, 2020 - 10:56 CEST
Monitoring
We've identified the cause and have applied a fix. We're monitoring the system.
Posted Sep 10, 2020 - 17:06 CEST
Investigating
We are investigating application linking issues in the Europe (eu1) cluster of our Cloud Hosted service.
Posted Sep 10, 2020 - 14:58 CEST
This incident affected: The Things Stack Cloud (Europe 1 (eu1), North America 1 (nam1), Australia 1 (au1)).