Inconsistent On Premise Device Activation
Incident Report for Singlewire Software


Last week, a small number of our customers were affected by a change we made that caused notifications sent to particular device groups to fail to reach their intended on-premises recipients. Although we resolved the problem quickly, sending notifications consistently and accurately is the core of our product, and we realize the impact this had on some of you. We’re very sorry, and we want you to know what went wrong and what we’re doing to prevent it from happening again.

What went wrong?

Device groups allow you to group together on-premises recipients in several ways: by selecting them individually; selecting multiple other, existing groups to combine; or constructing matching rules. Individual devices that would otherwise be included can also be specifically excluded. The logic we use in determining if devices should match is:

(device is in individual additions) OR (device is in any included other existing group AND device matches rules) AND NOT (device is in individual exclusions)

In our release two weeks ago, we made a change to individually added IP speakers and InformaCast Desktop Notifiers that made them behave as if they were rules, rather than individual additions. In doing so, we neglected to consider specific case where a device group had both other existing groups included and specific speakers or InformaCast Desktop Notifiers added. Unfortunately, the result was that neither devices in other existing groups nor the individual speaker nor desktop notifier additions were included.

We have extensive tests meant to catch problems like this, but in the case of this particular problem, while we had separate tests for individual additions, existing device groups, and matching rules, we didn’t have one for this specific combination.

What are we doing about it?

In addition to the fix we released, we’re adding additional tests for device groups, both for this specific case and for other permutations of our logic. These will ensure that the next time we need to tweak this code for whatever reason, we won’t be able to re-introduce this problem or a similar one.

We also realize that the logic of how devices are included is complex and can be difficult to understand. We’re working on changes to the Device Group Details page of our Administration Console to make the interaction between these inclusion methods clearer, both for you and us.

Posted Nov 12, 2019 - 17:02 CST

The deployed fix is working as expected. Should you experience any issues, please reach out to our support team. Thanks!
Posted Nov 08, 2019 - 14:00 CST
The fix has been successfully deployed. We will continue to monitor the situation and update this incident as appropriate.
Posted Nov 08, 2019 - 13:42 CST
A fix has been identified and scheduled for release at Friday Nov 8th at 13:00 CST (Friday Nov 8th at 19:00 UTC).

No outage is expected for this patch. We will continue to update this incident during deployment.
Posted Nov 08, 2019 - 12:08 CST
The scheduled patch deployment has been delayed. We will update this new incident when we have more information.
Posted Nov 07, 2019 - 17:36 CST
A fix has been identified and scheduled for release at Thursday Nov 7th at 18:30 CST (Friday Nov 8th at 12:30 UTC).

No outage is expected for this patch. We will continue to update this incident during deployment.
Posted Nov 07, 2019 - 16:45 CST
A workaround has been documented by our engineers in the Singlewire Community. The procedure can be found here:
Posted Nov 06, 2019 - 17:08 CST
Notifications sent to Device Groups consisting of both "Filtered with Device Groups" and Individually added Speakers and/or Desktop Notifiers may not reach all intended on premise devices.

The root cause has been identified and a fix is in progress.
Posted Nov 06, 2019 - 16:21 CST
This incident affected: InformaCast Mobile API.