Line graphs temporarily missing data points across products
Incident Report for AWAIR
Resolved
All data pipelines for averaging sensor data are running stably.
Posted Jun 26, 2019 - 21:11 PDT
Monitoring
All Dataflow jobs have been migrated to the stable region and missing average data have been populated retroactively. Monitoring to see if the pipelines run stably across time.
Posted Jun 26, 2019 - 20:01 PDT
Update
Parts of the average sensor data missing from line graphs have been restored. We are still working on backfilling the rest of the missing average data.
Posted Jun 26, 2019 - 18:13 PDT
Update
Still working to migrate the Dataflow jobs to a different region to mitigate the impact of GCP's outage.
Posted Jun 26, 2019 - 18:05 PDT
Update
We are going to migrate our jobs to be in different GCP region to get around the issue that GCP is having. We'll provide an update shortly.
Posted Jun 26, 2019 - 16:30 PDT
Identified
Google Cloud Platform (GCP)'s status page (https://status.cloud.google.com/) showing that they are experiencing issues with creating new Dataflow jobs which impacts AWAIR's ability to generate average data based on raw data gathered from devices. We are actively monitoring to get updates from GCP team.
Posted Jun 26, 2019 - 16:26 PDT
Investigating
Line graphs that show sensor data are temporarily failing to display certain time periods due to Google Cloud Platform's outage with Dataflow job scheduling. We are actively working with Google Cloud Team to mitigate the impact of this incident.
Posted Jun 26, 2019 - 16:25 PDT
This incident affected: Dashboard and App.