Cloud execution disruption due to AWS outage

Incident Report for k6

Resolved

This incident has been resolved.

Posted Dec 08, 2021 - 01:09 CET

Update

We are seeing significant recovery of the us-east-1 AWS region. EC2 service and related APIs are now available.
Starting tests in "amazon:us:ashburn" load zone is again possible. To mitigate further disruptions in this region, we have changed the default load zone to "amazon:us:columbus" for tests that don't explicitly specify the load zone.
The execution logs (console.log messages) are not yet available.
All other systems are operational.

Posted Dec 07, 2021 - 23:40 CET

Update

We are continuing to monitor for any further issues.

Posted Dec 07, 2021 - 23:13 CET

Update

We are continuing to monitor for any further issues.

Posted Dec 07, 2021 - 23:12 CET

Monitoring

We are seeing early signs of recovery of the us-east-1 AWS region. EC2 service and related APIs are slowly becoming available.

Starting tests in "amazon:us:ashburn" load zone is still not possible. We strongly recommend using "amazon:us:columbus" or any other supported load zone. See full list https://k6.io/docs/cloud/creating-and-running-a-test/cloud-tests-from-the-cli/#load-zones

When creating tests from web UI (test builder, script editor), please change the default from `ashburn` to `columbus`. When starting tests from CLI, specify "options.ext.loadimpact.distribution" to explicitly request a different load zone. Example available here: https://k6.io/docs/cloud/creating-and-running-a-test/cloud-tests-from-the-cli/#cloud-execution-options

We are continuing to monitor the situation, but we are cautiously optimistic.

Posted Dec 07, 2021 - 23:11 CET

Update

The system is unable to initialize cloud tests due to a major AWS outage.

Tests started locally with cloud streaming continue to work (`k6 run -o cloud script.js`).

For more info and current status, please check: https://status.aws.amazon.com/

Posted Dec 07, 2021 - 18:58 CET

Identified

The outage is caused by issues with AWS services, for more info and current status please check: https://status.aws.amazon.com/

Posted Dec 07, 2021 - 18:04 CET

Investigating

We are currently investigating an issue which causes k6 Cloud test runs to be stuck in initialising state.

Posted Dec 07, 2021 - 17:43 CET

This incident affected: Test Runners ("k6 cloud script.js" remote execution mode, "k6 run -o cloud script.js" streaming execution mode) and api.k6.io Cloud REST API, Load zones AWS.