Past Incidents

A maintenance is scheduled on Monday 2019-02-18 at 11:00 UTC (12:00 noon, Paris time (CET)); it will affect the main API, deployments and GIT repositories.

The maintenance will last at least 5 minutes but no more than 20 minutes.

Different parts of the system will be affected throughout this maintenance, please wait until the end of the maintenance before reporting any issues you may be having.

EDIT 11:00 UTC: The maintenance will start in a few minutes. Deployments and GIT repositories will be unavailable. The console might report an "unknown" or not up-to-date state for applications. This is expected.

EDIT 11:05 UTC: Maintenance is starting, deployments are down and so are GIT repositories (push actions will be rejected)

EDIT 11:09 UTC: Deployments are available again. Push actions on GIT repositories are still disabled.

EDIT 11:10 UTC: Our main API is entering read-only mode. 500 errors might appear during this time.

EDIT 11:11 UTC: Git repositories are now available. You might need to clear your DNS cache to be able to push again.

EDIT 11:20 UTC: Our main API should be fully available again. We are looking if everything looks fine.

EDIT 11:23 UTC: Everything is looking fine. The maintenance is over. You might experience git push errors up until 45 minutes. To avoid that, please clear your DNS cache.

We have an abnormal amount of 503 errors served by our reverse proxies. We are investigating.

EDIT 11:31 UTC: Cause has been identified, we are currently fixing the issue on our reverse proxies.

EDIT 11:34 UTC: All reverse proxies now have a consistent state. The issue is fixed.

The issue happened after a configuration error made during a manual operation on some of the reverse proxies. Applications that redeployed since 11:08 UTC were impacted by that issue. Other applications were fine. The changes were rollbacked and will again be tested thoroughly on our test infrastructure.

The main API is having issues with its databases connections, it only ever replies with a 500 error to most requests.

EDIT 07:11 UTC: The issue is resolved