Errors Sending Some Notifications from Message Templates with Images/Audio
Incident Report for Singlewire Software
Postmortem

Last Friday, we received several reports of organizations being unable to send notifications from our administration console. We traced the problem to missing metadata for files attached to some Message Templates, which had gotten erroneously deleted by a cleanup script. We immediately disabled the script and made a change to the administration console to allow sending notifications without that metadata. We also started work on a script to restore the metadata that had been removed.

Yesterday morning, several organizations reported a similar but slightly different issue, which again resulted in the administration console being unable to send some notifications. Our investigation revealed that the missing metadata was again the root cause. We had originally planned to deploy and run the metadata restoration script next week, but due to the second issue, we accelerated that work, and successfully deployed and ran the restoration script yesterday afternoon. We expect that this should prevent future similar issues.

The cleanup script that initially caused these erroneous deletions remains disabled while we correct its logic to avoid such errors in the future. We're sorry for the impact these issues may have had on your organization.

Posted Nov 11, 2022 - 15:17 CST

Resolved
This incident has been resolved.
Posted Nov 10, 2022 - 15:51 CST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 10, 2022 - 15:13 CST
Update
We're running a script to repair affected Message Templates. This may take several hours; we will post an update when it is complete or in 4 hours.
Posted Nov 10, 2022 - 12:00 CST
Update
We have identified a fix and are working to test and deploy it. Expect another update within an hour.
Posted Nov 10, 2022 - 11:00 CST
Identified
We've identified the problem regarding message templates and are currently testing a fix.
Posted Nov 10, 2022 - 10:31 CST
Investigating
We're received several reports of organizations being unable to send notifications from specific message templates. Unchecking the audio and/or image file associated with the message template, or re-uploading those files, seems to correct the problem. We're investigating the cause and will post an update within an hour.
Posted Nov 10, 2022 - 09:57 CST
This incident affected: InformaCast Mobile API and InformaCast Fusion Administrative WebConsole.