tag:status.aviator.co,2005:/historyAviator Status - Incident History2024-03-28T03:20:15ZAviatortag:status.aviator.co,2005:Incident/196066232024-01-13T21:01:03Z2024-01-13T21:01:03ZScheduled maintenance for Aviator<p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>21:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>23:26</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance to perform minor upgrades to our infrastructure. We do not expect significant impact of the Aviator service during that time.</p>tag:status.aviator.co,2005:Incident/191237472023-11-15T02:50:00Z2023-11-15T02:50:00ZWebserver and APIs have degraded performance<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>02:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>02:42</var> UTC</small><br><strong>Monitoring</strong> - It appears to be a Google Cloud outage that has auto-recovered. We are still monitoring the traffic for any continued impact.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>02:29</var> UTC</small><br><strong>Investigating</strong> - The webserver and API are unhealthy, the management dashboard might show errors and some APIs would error out. Investigating.</p>tag:status.aviator.co,2005:Incident/190130722023-11-03T20:07:20Z2023-11-03T20:07:20ZElevated error rates on GitHub API<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>20:07</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>19:37</var> UTC</small><br><strong>Monitoring</strong> - The API calls have recovered, we are just chugging along the backlog in queues, should be caught up in next 10-15 mins.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>18:52</var> UTC</small><br><strong>Identified</strong> - We are seeing degraded performance of GitHub APIs failing due to authentication issues, and are monitoring the situation closely. The queues will have degraded performance due to this incident.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>18:52</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>18:51</var> UTC</small><br><strong>Monitoring</strong> - We are seeing degraded performance of GitHub APIs failing due to authentication issues, and are monitoring the situation closely. The queues will have degraded performance due to this incident.</p>tag:status.aviator.co,2005:Incident/189943152023-11-02T01:46:54Z2023-11-02T01:46:54ZElevated error rates from GitHub API<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>01:46</var> UTC</small><br><strong>Resolved</strong> - Error rates have dropped considerably in the last hour. Marking this as resolved.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>23:09</var> UTC</small><br><strong>Identified</strong> - We are currently experiencing elevated error rates from GitHub API causing degraded performance in the background queues.</p>tag:status.aviator.co,2005:Incident/184156672023-09-16T20:04:18Z2023-09-16T20:04:18ZScheduled maintenance for Aviator<p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>20:04</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>18:02</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>01:44</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance to improve Aviator processing capacity</p>tag:status.aviator.co,2005:Incident/185051422023-09-14T22:27:56Z2023-09-14T22:27:56ZWeb application dashboard is down<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>22:27</var> UTC</small><br><strong>Resolved</strong> - The web application dashboard is back online. Everything is operating normally.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>22:20</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the web application server outage.<br />The queue operation should be unaffected with this outage.</p>tag:status.aviator.co,2005:Incident/181676452023-08-17T00:38:32Z2023-08-17T00:38:32Zbackground queues are slow<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>00:38</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>23:53</var> UTC</small><br><strong>Monitoring</strong> - The background queues should be back to normal now. We will continue to monitor the state for the next hour.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>23:40</var> UTC</small><br><strong>Investigating</strong> - We are investigating slowness in the background queues to process the jobs. You may see show responses for actions on GitHub.</p>tag:status.aviator.co,2005:Incident/175065822023-06-18T20:01:03Z2023-06-18T20:01:03ZScheduled maintenance for Aviator (1/2)<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>20:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>22:47</var> UTC</small><br><strong>Scheduled</strong> - We will updating our database indexes in two shots. The dashboard and queues may be temporarily impacted during this time.</p>tag:status.aviator.co,2005:Incident/172006032023-05-21T18:44:18Z2023-05-21T18:44:18ZScheduled maintenance for Aviator<p><small>May <var data-var='date'>21</var>, <var data-var='time'>18:44</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>23:44</var> UTC</small><br><strong>Scheduled</strong> - We will be updating our network configurations, the dashboard and background queues may be impacted during the window.</p>tag:status.aviator.co,2005:Incident/169575292023-04-25T03:21:25Z2023-04-25T03:21:25ZElevated error rates on GitHub API<p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>03:21</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>00:30</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>00:29</var> UTC</small><br><strong>Update</strong> - Looks like we are still seeing elevated errors rates, also reported by Github. Please also take a look at githubstatus.com</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>00:29</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>00:29</var> UTC</small><br><strong>Investigating</strong> - Looks like we are still seeing elevated errors rates, also reported by Github. Please also take a look at githubstatus.com</p>tag:status.aviator.co,2005:Incident/169560102023-04-24T23:58:54Z2023-04-24T23:58:54ZElevated error rates on GitHub API<p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>23:58</var> UTC</small><br><strong>Resolved</strong> - Things are back to normal.</p><p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>20:44</var> UTC</small><br><strong>Monitoring</strong> - We are experiencing elevated error rates from GitHub API. This may result in degraded performance with the queues. We are monitoring the health and will continue providing updates here.</p>tag:status.aviator.co,2005:Incident/169061672023-04-23T19:12:49Z2023-04-23T19:12:49ZScheduled maintenance for Aviator<p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>19:12</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>23:34</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing a scheduled maintenance at this time. The dashboard and background queues may be unavailable during this window.</p>tag:status.aviator.co,2005:Incident/163003932023-03-04T23:37:51Z2023-03-04T23:37:51ZScheduled maintenance for Aviator<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>23:37</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>02:31</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing a scheduled maintenance at this time. The dashboard and background queues may be unavailable during this time.</p>tag:status.aviator.co,2005:Incident/161676672023-02-16T19:38:46Z2023-02-16T19:38:46ZExperiencing slow web page loads<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>19:38</var> UTC</small><br><strong>Resolved</strong> - Everything should be back to normal, any webhook events that we missed should have also been processed now.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>18:59</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.aviator.co,2005:Incident/117522132022-10-15T21:00:30Z2022-10-15T21:00:30ZScheduled maintenance for Aviator<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>18:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>18:09</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing system upgrade during this time. The Aviator services may be intermittently unavailable during this time.</p>tag:status.aviator.co,2005:Incident/108146052022-08-20T20:00:31Z2022-08-20T20:00:31ZScheduled maintenance for Aviator<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>18:22</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing schedule maintenance to upgrade our servers and dependencies. The queues and dashboard may be impacted intermittently during the window.</p>tag:status.aviator.co,2005:Incident/104533902022-07-09T21:02:33Z2022-07-09T21:02:33ZScheduled maintenance for Aviator<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>21:02</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>23:29</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance to upgrade our dependencies and monitoring systems.</p>tag:status.aviator.co,2005:Incident/100055192022-05-21T22:00:41Z2022-05-21T22:00:41ZScheduled maintenance for MergeQueue Dashboard<p><small>May <var data-var='date'>21</var>, <var data-var='time'>22:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>19:02</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>19:49</var> UTC</small><br><strong>Scheduled</strong> - We will be going under scheduled maintenance. There should be no interruptions in the queueing behavior of MergeQueue but the Management dashboard may be inaccessible during the window.</p>