The Things Industries
All Systems Operational

About This Site

This page shows the current status of The Things Stack Cloud services.

Cloud ? Operational
Europe 1 (eu1) ? Operational
North America 1 (nam1) ? Operational
Join Server ? Operational
Australia 1 (au1) ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
We are discontinuing the Pub/Sub functionality in The Things Stack Cloud: the external MQTT server and NATS broker support. The reason for this is that we cannot scale this properly while guaranteeing our service level. Below is a technical explanation.

This does not affect the MQTT server provided by The Things Stack Cloud. It only affects external MQTT servers via Pub/Sub.

Also, this does not affect customers using the AWS IoT Integration, even though the current integration uses MQTT under the hood. We are therefore refactoring the AWS IoT Integration to use AWS IoT Data Plane for uplink and gRPC for downlink. You will be notified in advance to upgrade your AWS IoT Integration. You will only have to update the CloudFormation template; no other migration or configuration will be necessary. The AWS IoT Integration also benefits from this, as the current fire-and-forget downlink scheduling via AWS IoT MQTT is replaced with a proper retry behavior.

Existing Pub/Sub integrations will be disabled automatically after January 5th, 2021. We will reach out to customers that are using external MQTT and NATS and guide them in the migration to webhooks. For more information about webhooks, see https://thethingsstack.io/integrations/webhooks/

---

The reason why scaling external MQTT servers is not feasible, is that all traffic is tunneled over a single TCP connection. Distributing that over multiple connections is possible, but as MQTT 3.1.1 does not support shared subscriptions, we would have to deduplicate downlink messages. On top of that, when using multiple subscribers with unique MQTT client IDs, the QoS functionality would not work. Therefore, if there is a connection interruption between The Things Stack Cloud and an external MQTT server, we risk losing downlink messages without knowing. MQTT 5 overcomes many of these challenges on paper, but adoption and stable library support is stalling. We might re-enable MQTT and NATS pub/sub in the future based on technical advancements and customer demand.
Posted on Nov 19, 09:30 CET
Past Incidents
Nov 25, 2020

No incidents reported today.

Nov 24, 2020

No incidents reported.

Nov 23, 2020

No incidents reported.

Nov 22, 2020

No incidents reported.

Nov 21, 2020

No incidents reported.

Nov 20, 2020

No incidents reported.

Nov 19, 2020
Resolved - We monitored the fixed infrastructure overnight and all metrics are back and healthy.
Nov 19, 09:30 CET
Monitoring - We identified the cause of the issue to be unintended throttling of uplink messages in the Network Server which caused a queueing of uplinks and disrupting downlink flow. We've applied a fix that ensures that this throttling is mitigated and traffic reaches the application server as intended. We're continuing to monitor the fix.
Nov 18, 21:56 CET
Identified - We experience degraded performance for downlink. We've identified the cause and will apply a fix shortly.
Nov 18, 16:42 CET
Nov 18, 2020
Nov 17, 2020
Resolved - This incident has been resolved.
Nov 17, 20:38 CET
Update - We are continuing to monitor for any further issues.
Nov 17, 17:24 CET
Monitoring - We experience degraded performance for downlink. We identified internal clustering and configuration issues as the cause. A fix has been deployed and we're monitoring the results.

This incident acts as a placeholder to follow up. At the time of writing, services should be back to normal and perform better than an hour ago.
Nov 17, 17:00 CET
Nov 16, 2020

No incidents reported.

Nov 15, 2020

No incidents reported.

Nov 14, 2020

No incidents reported.

Nov 13, 2020

No incidents reported.

Nov 12, 2020

No incidents reported.

Nov 11, 2020

No incidents reported.