On August 13th, we observed a few isolated occurrences of queries to our in-memory data store timing out. While we don't have a definitive root cause for the issue, we have been able to simulate similar cases with high query volume in our testing environments, and are making the following remediations to both hopefully prevent this issue from occurring again as well as mitigate it in the event that it does.

First, we'll be upgrading the instance class of the machines this database runs on, which uses a more modern AWS hypervisor and processor architecture.

Additionally, we'll be bolstering certain commands with retry and circuit breaker logic, such that in the rare case that this does happen again, the command will be immediately retried to a point, while allowing the queries to be skipped entirely if the database is truly unavailable.

Lastly, we'll be tweaking our slow query logs in order to ensure we keep an eye on performance of this resource so that we can continue to tune our queries and ensure they don't place an undue amount of stress on this database.

We apologize for any inconvenience this may have caused.