Duration: 1 hour and 15 minutes

Affected Components: Managed Deployment Workers, Self-Hosted Deployment Workers

Mar 10, 10:00:00 GMT+0
Investigating - We are investigating reports of failed AWS deployments for customers using OIDC-based authentication. Our team is actively looking into this..

Mar 10, 10:20:00 GMT+0
Identified - We have identified the root cause of the OIDC AWS deployment failures. A recent platform update introduced a change that is incompatible with some AWS IAM OIDC identity provider configurations. Our team is working on a fix..

Mar 10, 11:15:00 GMT+0
Resolved - The issue affecting OIDC-based AWS deployments has been resolved. All systems are operating normally. No data was lost or compromised. We apologize for the inconvenience..

Mar 10, 11:00:00 GMT+0
Monitoring - A fix has been deployed for OIDC-based AWS deployment failures. We are monitoring to confirm full recovery. Customers using access keys or non-AWS providers were not affected. If you continue to experience issues, contact [support@env0.com](mailto:support@env0.com)..

Affected Components: API

Jan 13, 12:08:38 GMT+0
Resolved - The issue was caused by a deployment mechanism failure, where one of the services did not deploy successfully while other services attempted to call newly introduced functions..

Jan 13, 10:43:49 GMT+0
Monitoring - We identified the issue and applied a quick fix. The system has returned to normal operation. We are continuing to monitor for any additional errors..

Jan 13, 10:29:15 GMT+0
Identified - The issue has been identified and a fix is being implemented..

Affected Components: API, Management Console

Jan 7, 16:57:32 GMT+0
Resolved - This incident has been resolved..

Jan 7, 15:41:10 GMT+0
Monitoring - We are continuing to monitor for any further issues..

Jan 7, 15:39:47 GMT+0
Monitoring - We're identified some issues regarding Authentication errors, we've implemented a fix and we're monitoring the situation.

Affected Components: OpenTofu Module Registry

Nov 18, 15:08:15 GMT+0
Resolved - This incident has been resolved..

Nov 18, 13:39:36 GMT+0
Monitoring - OpenTofu Module Registry is unavailable because of Cloudflare outage..

Affected Components: API, Management Console

Oct 20, 22:45:08 GMT+0
Resolved - This incident has been resolved..

Oct 20, 16:01:14 GMT+0
Monitoring - We are continuing to monitor for any further issues..

Oct 20, 16:01:00 GMT+0
Monitoring - We are currently monitoring a widespread service disruption affecting our system. This incident is caused by a significant, ongoing network connectivity outage in AWS's US-EAST-1 Region, which is impacting multiple services including EC2, DynamoDB, and SQS. Our team is in contact with our provider and actively monitoring the situation. We will post an update as soon as AWS provides more information on their recovery..

Affected Components: API, Management Console

Oct 20, 12:32:16 GMT+0
Resolved - This incident is now resolved. Our cloud provider, AWS, has confirmed recovery for the underlying SQS issue that was affecting our deployment triggers. We have verified that all services, including PR-Plan and CD deployments, are now operating normally and all queued jobs have been processed..

Oct 20, 11:45:40 GMT+0
Monitoring - We have narrowed down the scope of the performance degradation. We have confirmed that **PR-Plan and CD deployments** are not triggered. All other deployment triggers, including **UI, API, PR Comment, and scheduled tasks, are working normally.** This issue remains a direct result of the ongoing AWS outage in the US-EAST-1 Region. We are continuing to monitor the situation and will post an update as soon as performance for PR-Plan and CD deployments is restored..

Oct 20, 10:57:05 GMT+0
Monitoring - Our core deployment service is currently experiencing degraded performance. Users may experience significant delays or errors when triggering new deployments. This is a direct result of the ongoing AWS outage affecting EC2 instances in the US-EAST-1 Region. Our team is actively monitoring the situation and AWS's recovery progress. We will post an update as soon as we see performance for our deployment service return to normal..

Oct 20, 09:51:26 GMT+0
Monitoring - We are seeing significant signs of recovery. Our cloud provider, AWS, has reported that most requests are now succeeding and they are working through a backlog. Our internal end-to-end tests confirm that our application is back online and operating as expected. We are moving this incident to "Monitoring" as we continue to observe the situation to ensure full stability and recovery. We will post a final update once we are confident the issue is fully resolved..

Oct 20, 07:05:31 GMT+0
Investigating - We are currently investigating increased error rates. We have identified a potential service disruption with our cloud provider, AWS, that we suspect is the root cause. Our team is actively monitoring the situation and working to confirm the full impact..

Affected Components: API, Management Console

Aug 26, 16:35:37 GMT+0
Resolved - The incident has been resolved, environment management and deployments are operational.

Aug 26, 16:17:55 GMT+0
Monitoring - The fix has been applied and we'll continue to monitor the situation.

Aug 26, 16:12:47 GMT+0
Identified - We've identified the issue and the fix is currently being applied.

Aug 26, 14:51:05 GMT+0
Investigating - We're investigating environment management and slowness in the env0 system, which occurred earlier today and started again.

Affected Components: API, Management Console

Aug 26, 13:35:42 GMT+0
Resolved - Resolved, we'll continue to investigate and monitor.

Aug 26, 12:53:30 GMT+0
Investigating - Slowness and other issues had dropped, we're monitoring the situation.

Aug 26, 12:35:26 GMT+0
Investigating - We're currently investigating environment management and slowness in the env0 system.

Affected Components: Management Console

Aug 21, 12:38:46 GMT+0
Resolved - This incident has been resolved..

Aug 21, 12:21:54 GMT+0
Monitoring - We are currently monitoring the situation. The rate of delayed deployments in env0 was greatly reduced, and we are currently monitoring to make sure that everything is back to normal.

Aug 21, 11:51:36 GMT+0
Identified - Please perform a hard refresh of your page as soon as possible..

Aug 21, 11:09:31 GMT+0
Identified - Our team has identified the issue. Please perform a hard refresh of your page as soon as possible. We apologize for the inconvenience..

Aug 21, 10:10:14 GMT+0
Investigating - We are currently investigating an issue where deployments in the env0 app are not starting, are delayed, or are failing with an error..

Affected Components: Management Console

Jul 28, 11:15:20 GMT+0
Resolved - This incident has been resolved..

Jul 28, 10:58:16 GMT+0
Monitoring - We are continuing to monitor for any further issues..

Jul 28, 10:46:23 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

Jul 28, 10:31:07 GMT+0
Investigating - We are currently investigating this issue..

Affected Components: Management Console

Jul 28, 09:36:06 GMT+0
Resolved - We've identified some load on our cluster earlier, it has since subsided, and the issue is now resolved..

Jul 28, 09:09:41 GMT+0
Investigating - We are continuing to investigate this issue..

Jul 28, 09:08:35 GMT+0
Investigating - We are currently investigating this issue..

Affected Components: Management Console

May 22, 17:10:34 GMT+0
Resolved - This incident has been resolved..

May 22, 17:01:10 GMT+0
Investigating - We are currently investigating a sporadic issue with creating deployments in the env0 SaaS. env0 agents are currently unaffected..

Affected Components: Management Console

May 22, 12:30:50 GMT+0
Resolved - This incident has been resolved..

May 22, 12:24:58 GMT+0
Monitoring - We've have resolved the issue and are currently monitoring the fix. The env0 SaaS should be fully operational.

May 22, 12:01:57 GMT+0
Investigating - We are currently investigating a sporadic issue with creating deployments in the env0 SaaS. env0 agents are currently unaffected..

Affected Components: Management Console

Mar 26, 18:30:18 GMT+0
Resolved - This incident has been resolved..

Mar 26, 16:07:42 GMT+0
Identified - A recent docker image upgrade caused a dependency issue, our teams are working on a fix..

Affected Components: API, Management Console

Feb 23, 11:56:57 GMT+0
Resolved - Outage resolved..

Feb 23, 11:49:23 GMT+0
Monitoring - Revert is done. Monitoring....

Feb 23, 11:40:22 GMT+0
Identified - Issue identified in one of our services. Revert in progress (~20m).

Feb 23, 11:36:42 GMT+0
Investigating - We are continuing to investigate this issue. Auto revert is upcoming..

Feb 23, 11:36:32 GMT+0
Investigating - We are seeing several errors when customers access the app. The issue is in investigation. Auto revert is upcoming..

Affected Components: Management Console

Jan 19, 10:32:55 GMT+0
Resolved - This incident has been resolved..

Jan 19, 10:11:41 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

Jan 19, 09:31:12 GMT+0
Investigating - We noticed that deployments are not getting started, our team is working on a fix for the issue..

Affected Components: API, Management Console

Jan 14, 06:21:28 GMT+0
Resolved - This incident has been resolved..

Jan 14, 06:03:14 GMT+0
Investigating - We are currently investigating this issue..

Affected Components: API, Management Console

Jan 14, 05:42:20 GMT+0
Resolved - This incident has been resolved..

Jan 14, 05:38:31 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

Jan 14, 05:13:16 GMT+0
Investigating - We are currently investigating this issue..

Affected Components: API, Management Console

Dec 1, 10:58:59 GMT+0
Resolved - This incident has been resolved..

Dec 1, 10:46:39 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

Dec 1, 10:26:29 GMT+0
Identified - We noticed some timeout errors in our system. Our team is on it..

Affected Components: API, Management Console

Nov 25, 12:05:26 GMT+0
Resolved - This incident has been resolved..

Nov 25, 11:44:13 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

Nov 25, 11:42:22 GMT+0
Identified - We identified some timeout errors in our system, our team is working on it..

Sep 29, 14:22:20 GMT+0
Resolved - This incident has been resolved..

Sep 29, 13:42:07 GMT+0
Identified - The issue has been identified and a fix is being implemented..

Affected Components: Management Console

Sep 16, 07:34:48 GMT+0
Resolved - This incident has been resolved..

Sep 16, 07:14:29 GMT+0
Monitoring - We are continuing to monitor for any further issues..

Sep 16, 07:12:30 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

Sep 16, 06:34:55 GMT+0
Identified - We have identified and are working to resolve an issue when trying to deploy custom flows fetched from GitLab Enterprise Edition. We will roll out a fix shortly..

Affected Components: API, Management Console

Jul 18, 10:14:15 GMT+0
Resolved - This incident has been resolved..

Jul 18, 09:37:12 GMT+0
Identified - We currently having some issues completing deployment when using git refs (commit hashes) in place of branch name. The issue has been identified and a fix is being issued..

Affected Components: Management Console

Jun 24, 10:44:13 GMT+0
Resolved - This incident has been resolved..

Jun 24, 10:37:34 GMT+0
Identified - We've identified an issue with accessing the UI of env0 - our team is working on rolling out a fix.

Affected Components: Management Console

May 22, 11:29:13 GMT+0
Resolved - This incident has been resolved..

May 22, 11:18:15 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

May 22, 10:32:45 GMT+0
Identified - We have identified an issue with how we report git providers status checks. We are looking into rolling out a fix within the next 30 minutes.

Duration: 5 hours

Affected Components: Management Console, API

Jul 22, 17:02:53 GMT+0
Completed - The scheduled maintenance has been completed..

Jul 22, 15:01:27 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Jul 22, 13:59:57 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Jul 22, 09:26:56 GMT+0
Identified - We apologize, we've had an issue with automatic messaging for this incident. The incident was marked as completed, however the maintenance window and actions didn't start yet. Maintenance will go through as planned, today from 3PM UTC to 8PM UTC.

Duration: 5 hours

Affected Components: Management Console, API

Jul 22, 08:01:07 GMT+0
Completed - The scheduled maintenance has been completed..

Jul 22, 03:00:06 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Jul 22, 02:00:03 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Jul 5, 10:45:07 GMT+0
Identified - We will be undergoing scheduled maintenance during this time..

Jul 5, 10:43:46 GMT+0
Identified - We are writing to inform you that due to customers requests, we have updated the scheduled database upgrade window as follows: Date: Saturday, July 22, 2023 Time: 3:00 PM UTC to 8:00 PM UTC We apologize for any inconvenience this may cause. Thank you, env0 team.

Jul 2, 09:00:59 GMT+0
Identified - We will be performing scheduled maintenance on our database on Sunday, July 23rd, 2023, from 7:00 am to 12:00 pm UTC. This maintenance is necessary to upgrade our database, which will improve performance and security. During this time, the platform will be offline. You will not be able to login to the application, or interact with the API. Any flows triggered by git actions or similar, will also be disabled during this time. We apologize for any inconvenience this may cause, in the meantime, if you have any questions or concerns, please do not hesitate to contact us. Thank you for your understanding. Sincerely, env0 team.

Duration: 4 hours

Affected Components: Management Console, API

Jan 21, 21:56:48 GMT+0
Completed - The scheduled maintenance has been completed..

Jan 21, 18:00:07 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Jan 21, 17:00:27 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Jan 5, 13:51:38 GMT+0
Identified - We are scheduling a planned downtime on Saturday 21 January at 1800 UTC / 1100 EST in order to complete important env0 Platform maintenance. This maintenance window may last up to four (4) hours. During this time, you will have no access to the env0 Platform via UI or API, and previously scheduled deployments will not run. We will notify our users when the env0 Platform is available again. Please contact us should you have any questions or concerns..

Duration: 30 minutes

Affected Components: Management Console

Oct 20, 11:06:55 GMT+0
Completed - The scheduled maintenance has been completed..

Oct 20, 10:46:20 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Oct 20, 10:45:11 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Oct 20, 10:43:53 GMT+0
Identified - We will be undergoing scheduled maintenance during this time..

Oct 20, 10:42:58 GMT+0
Identified - We are rolling out a fix for terraform private module registry. During this time the deployments may fail. Sorry for inconvenience..