Past Incidents

A maintenance is scheduled on Monday 2019-02-18 at 11:00 UTC (12:00 noon, Paris time (CET)); it will affect the main API, deployments and GIT repositories.

The maintenance will last at least 5 minutes but no more than 20 minutes.

Different parts of the system will be affected throughout this maintenance, please wait until the end of the maintenance before reporting any issues you may be having.

EDIT 11:00 UTC: The maintenance will start in a few minutes. Deployments and GIT repositories will be unavailable. The console might report an "unknown" or not up-to-date state for applications. This is expected.

EDIT 11:05 UTC: Maintenance is starting, deployments are down and so are GIT repositories (push actions will be rejected)

EDIT 11:09 UTC: Deployments are available again. Push actions on GIT repositories are still disabled.

EDIT 11:10 UTC: Our main API is entering read-only mode. 500 errors might appear during this time.

EDIT 11:11 UTC: Git repositories are now available. You might need to clear your DNS cache to be able to push again.

EDIT 11:20 UTC: Our main API should be fully available again. We are looking if everything looks fine.

EDIT 11:23 UTC: Everything is looking fine. The maintenance is over. You might experience git push errors up until 45 minutes. To avoid that, please clear your DNS cache.

We are experiencing issues on one hypervisors which can impact addons.

EDIT 20:50UTC: we are hard rebooting the hypervisor.

EDIT 20:55UTC: the hypervisor is up, the addons hosted on it are starting.

EDIT 20:58UTC: fixed.

Some deployments are having a delay to start.

10:27 UTC: The issue is now fixed

Applications in the console are reporting an "unknown" state.

We are investigating the issue. Deployments are stopped until we find the root cause.

EDIT 16:35 UTC: Applications state should now be OK. Deployments are still stopped until we figure out the issue.

EDIT 16:40 UTC: Problem has been identified. We will resume deployments in a few minutes. All deployments action were queued and will be consumed.

EDIT 16:42 UTC: Deployments are enabled again. It may take a few minutes before your actions are handled. We consider this incident over.

We have an abnormal amount of 503 errors served by our reverse proxies. We are investigating.

EDIT 11:31 UTC: Cause has been identified, we are currently fixing the issue on our reverse proxies.

EDIT 11:34 UTC: All reverse proxies now have a consistent state. The issue is fixed.

The issue happened after a configuration error made during a manual operation on some of the reverse proxies. Applications that redeployed since 11:08 UTC were impacted by that issue. Other applications were fine. The changes were rollbacked and will again be tested thoroughly on our test infrastructure.

The main API is having issues with its databases connections, it only ever replies with a 500 error to most requests.

EDIT 07:11 UTC: The issue is resolved