Line graphs temporarily missing data points across products

Incident Report for AWAIR

Resolved

All data pipelines for averaging sensor data are running stably.
Posted Jun 26, 2019 - 21:11 PDT

Monitoring

All Dataflow jobs have been migrated to the stable region and missing average data have been populated retroactively. Monitoring to see if the pipelines run stably across time.
Posted Jun 26, 2019 - 20:01 PDT

Update

Parts of the average sensor data missing from line graphs have been restored. We are still working on backfilling the rest of the missing average data.
Posted Jun 26, 2019 - 18:13 PDT

Update

Still working to migrate the Dataflow jobs to a different region to mitigate the impact of GCP's outage.
Posted Jun 26, 2019 - 18:05 PDT

Update

We are going to migrate our jobs to be in different GCP region to get around the issue that GCP is having. We'll provide an update shortly.
Posted Jun 26, 2019 - 16:30 PDT

Identified

Google Cloud Platform (GCP)'s status page (https://status.cloud.google.com/) showing that they are experiencing issues with creating new Dataflow jobs which impacts AWAIR's ability to generate average data based on raw data gathered from devices. We are actively monitoring to get updates from GCP team.
Posted Jun 26, 2019 - 16:26 PDT

Investigating

Line graphs that show sensor data are temporarily failing to display certain time periods due to Google Cloud Platform's outage with Dataflow job scheduling. We are actively working with Google Cloud Team to mitigate the impact of this incident.
Posted Jun 26, 2019 - 16:25 PDT
This incident affected: Enterprise Dashboard and Mobile Apps (Awair Business App).