Client Data Retrieval Not Returning Data
Incident Report for AWAIR
Postmortem

On the morning of October 16th, 2019, we noticed a data processing delay on the server that we use for Awair services (Dashboard, Mobile App, and Developer APIs). Although all Awair services showed a “disconnected” status or did not return data, the problem was not the connection, but the server’s inability to process and display data.

By 12pm (PST) on October 17th, we were able to resolve all data processing bottlenecks and retrieve the vast majority of the data that was collected during the 27-hour backlog period. As we implemented those fixes on October 17th, however, we were unable to capture data during a select few periods:

  • 4:05am to 4:15am PT (10 minutes maximum)
  • 5:40am to 6:00am PT (20 minutes maximum)
  • 2:45pm to 3:20pm PT (35 minutes maximum)

In the Awair app and Omni Dashboard, you can expect to see your missing air quality data from October 16th and 17th restored, with the exception of the three gap periods listed above.

Thank you for your patience over the last couple days. Please reach out to us if we can offer further assistance at hello@getawair.com.

Posted Oct 18, 2019 - 15:40 PDT

Resolved
All systems are back to normal.
Posted Oct 17, 2019 - 16:00 PDT
Update
Historical data has been reprocessed and backfilled. A secondary issue with devices appearing offline in client facing interfaces is being addressed. Devices are online.
Posted Oct 17, 2019 - 15:07 PDT
Monitoring
Historical data from trend lines and APIs are starting to backfill from backend component queues. Continuing to monitor until data is finished processing.
Posted Oct 17, 2019 - 11:25 PDT
Identified
Identified issue with an upstream service provider. Working with them to find a resolution.
Posted Oct 16, 2019 - 16:15 PDT
Update
We are continuing to investigate this issue.
Posted Oct 16, 2019 - 12:06 PDT
Investigating
We are currently investigating this issue.
Posted Oct 16, 2019 - 06:55 PDT
This incident affected: Dashboard, Developer APIs, OAuth 2.0, and App.