The communication between the Network server and the Application server was interrupted due to a bug which resulted in the messages being sent in a loop, eventually causing the nam1 routing cluster to stall.
To prevent such issues in the future, we will expand testing scenarios by using more traffic and test component failures/unavailability in our staging environment.
Posted Feb 26, 2021 - 12:55 CET
Completed
Maintenance for our eu1 and au1 clusters was completed within the planned maintenance window.
Unfortunately the updates resulted in a significant outage of our nam1 cluster, which is why we have kept this incident open. The issues on our nam1 cluster have now been resolved.
We will publish a postmortem after we have finished our investigation into the incident on our nam1 cluster, but for now everything is stable.
Posted Feb 23, 2021 - 18:31 CET
Update
We are continuing to verify the maintenance items.
Posted Feb 23, 2021 - 18:27 CET
Verifying
Verification is currently underway for the maintenance items.
Posted Feb 23, 2021 - 13:01 CET
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Feb 23, 2021 - 11:00 CET
Scheduled
During this maintenance window we will upgrade The Things Stack Cloud to v3.11.1.
We do expect some short service interruptions during this update.
Here is the changelog since the current version v3.10.9:
### Added
- Reset functionality in Network Server, which resets session context and MAC state (see `ttn-lw-cli end-devices reset` command). For OTAA all data is wiped and device must rejoin, for ABP session keys, device address and downlink queue are preserved, while MAC state is reset. - Store and retrieve Gateway Claim Authentication Code from database. - This requires a database schema migration (`ttn-lw-stack is-db migrate`) because of the added column. - This uses the same encryption key set using the `is.gateways.encryption-key-id` configuration option. - Improved handling of connection issues in the Console, as well as automatic reconnects. - Helpful details for synthetic meta events in the data view of the Console. - Support field mask paths in Storage Integration API requests. - CUPS redirection. - This requires a database schema migration (`ttn-lw-stack is-db migrate`) because of the added columns. - Configuration option (`is.user-registration.enabled`) to enable or disable user registrations. - Missing CLI commands for getting single API keys or collaborators for entities. - New Account App for authentication, authorization and account related functionality. - This introduces various UX improvements and new designs around e.g. user login, registration and the "forgot password" flow. - Integrate Device Repository. - Device Repository component to integrate [Device Repository](https://github.com/TheThingsNetwork/lorawan-devices) with The Things Stack. See the `dr` configuration section. - The Device Repository database is bundled automatically into Docker release images. See the `ttn-lw-stack dr-db init` command to manually fetch the latest changes. - Device repository service to the JavaScript SDK. - Choosing array representation for end device session keys as well as gateway EUI. - Storage integration panel in the Console.
### Changed
- Network Server does not store `recent_uplinks`, `recent_adr_uplinks` and `recent_downlinks` anymore. - Improved Network Server downlink task performance. - Authorized Application Registry is changed to Authorized Entity Registry. - This requires a database schema migration (`ttn-lw-stack dcs-db migrate`) to migrate existing claims to the new registry. - Improved Network Server matching performance. - Network Server matching mapping in the database. - This requires a database migration (`ttn-lw-stack ns-db migrate`). - Sending a non-empty implicitly specified field disallowed field will now cause RPCs to fail. E.g. if RPC supports paths `A` and `A.B`, sending value with `A.C` non-empty and field mask `A` would result in an error. - Improved content of emails sent by the Identity Server. - Stricter validation of the maximum length of string fields, binary fields, lists and maps. - Frequency plans and webhook templates are now included in Docker images, and used by default, instead of fetching directly from GitHub. - JavaScript entrypoints changed from `oauth.css` and `oauth.js` to `account.css` and `account.css`. Note: For deployments using custom frontend bundles (e.g. via `--is.oauth.ui.js-file`), the filenames have to be updated accordingly as well. - Authentication Providers and External Users are now hard deleted by default. - This requires a database migration (`ttn-lw-stack is-db migrate`).
### Removed
- Application Server linking. The Network Server now pushes data to the cluster Application Server instead. - Applications which desire to handle payload decryption within their own domains should disable payload decryption at application or device level and decrypt the payload on their own end. - While not backwards compatible, the decision to remove linking was heavily motivated by scalability concerns - the previous linking model scales poorly when taking high availability and load balancing concerns into account. - The option to disable CUPS per gateway `gcs.basic-station.require-explicit-enable`.
### Fixed
- Network Server DevStatusReq scheduling conditions in relation to frame counter value. - Missing `authentication`, `remote_ip` and `user_agent` fields in events when using event backends other than `internal`. - Handling of `DLChannelReq` if dependent `NewChannelReq` was previously rejected. - Login after user registration leading to dead-end when originally coming from the Console. - Frame counter display of end devices on initial page load in the Console. - AU915-928 data rate indexes in Regional Parameter specification versions below 1.0.2b.
Posted Feb 16, 2021 - 11:28 CET
This scheduled maintenance affected: The Things Stack Cloud (Europe 1 (eu1), North America 1 (nam1), Australia 1 (au1)).