We are discontinuing the Pub/Sub functionality in The Things Stack Cloud: the external MQTT server and NATS broker support. The reason for this is that we cannot scale this properly while guaranteeing our service level. Below is a technical explanation.
This does not affect the MQTT server provided by The Things Stack Cloud. It only affects external MQTT servers via Pub/Sub.
Also, this does not affect customers using the AWS IoT Integration, even though the current integration uses MQTT under the hood. We are therefore refactoring the AWS IoT Integration to use AWS IoT Data Plane for uplink and gRPC for downlink. You will be notified in advance to upgrade your AWS IoT Integration. You will only have to update the CloudFormation template; no other migration or configuration will be necessary. The AWS IoT Integration also benefits from this, as the current fire-and-forget downlink scheduling via AWS IoT MQTT is replaced with a proper retry behavior.
Existing Pub/Sub integrations will be disabled automatically after January 5th, 2021. We will reach out to customers that are using external MQTT and NATS and guide them in the migration to webhooks. For more information about webhooks, see
https://thethingsstack.io/integrations/webhooks/---
The reason why scaling external MQTT servers is not feasible, is that all traffic is tunneled over a single TCP connection. Distributing that over multiple connections is possible, but as MQTT 3.1.1 does not support shared subscriptions, we would have to deduplicate downlink messages. On top of that, when using multiple subscribers with unique MQTT client IDs, the QoS functionality would not work. Therefore, if there is a connection interruption between The Things Stack Cloud and an external MQTT server, we risk losing downlink messages without knowing. MQTT 5 overcomes many of these challenges on paper, but adoption and stable library support is stalling. We might re-enable MQTT and NATS pub/sub in the future based on technical advancements and customer demand.
Posted on
Nov 19, 09:30 CET