Outage in Runs & Experiments

Incident Report for Grid AI

Resolved

We have applied an update to all clusters and service is now fully restored.
Posted Sep 30, 2021 - 17:59 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 30, 2021 - 17:08 UTC

Update

We are continuing to work on a fix for this issue.
Posted Sep 30, 2021 - 16:12 UTC

Identified

We have identified an issue in which Runs & Experiments are unable to run due to a failure in an upcoming feature. All clusters are affected, including Grid Cloud and BYOC. We will be turning the feature off and updating all clusters.
Posted Sep 30, 2021 - 16:12 UTC
This incident affected: Grid Service.