CI misses and slowdown today

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

CI misses and slowdown today

Antoine Musso-3
Hello,

The CI might have missed some events from 9:26 UTC until 14:04 UTC. That
is due to a configuration change that has not been worked as expected
and caused some patches to be entirely ignored.

https://phabricator.wikimedia.org/T105474
https://phabricator.wikimedia.org/T222605

Some patches might thus be missing a CI report and would need to be
tested again (either by voting Code-Review +1 or by commenting 'recheck').


Unrelated is that since 14:30 a large amount of job have been triggered.
  The sole way I have found to solve it is to hard restart Zuul but that
would flush all its of its state.  I am investigating and will come up
with an enhancement later this week.


--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI misses and slowdown today

Antoine Musso-3
On 06/05/2019 17:51, Antoine Musso wrote:

> Hello,
>
> The CI might have missed some events from 9:26 UTC until 14:04 UTC. That
> is due to a configuration change that has not been worked as expected
> and caused some patches to be entirely ignored.
>
> https://phabricator.wikimedia.org/T105474
> https://phabricator.wikimedia.org/T222605
>
> Some patches might thus be missing a CI report and would need to be
> tested again (either by voting Code-Review +1 or by commenting 'recheck').
>
>
> Unrelated is that since 14:30 a large amount of job have been triggered.
>   The sole way I have found to solve it is to hard restart Zuul but that
> would flush all its of its state.  I am investigating and will come up
> with an enhancement later this week.

Hello,

The queue has been drained and CI behave properly now. An incident
report will follow later on.

--
Antoine Musso



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI misses and slowdown today

Antoine Musso-3
On 06/05/2019 18:26, Antoine Musso wrote:

> On 06/05/2019 17:51, Antoine Musso wrote:
>> Hello,
>>
>> The CI might have missed some events from 9:26 UTC until 14:04 UTC.
>> That is due to a configuration change that has not been worked as
>> expected and caused some patches to be entirely ignored.
>>
>> https://phabricator.wikimedia.org/T105474
>> https://phabricator.wikimedia.org/T222605
>>
>> Some patches might thus be missing a CI report and would need to be
>> tested again (either by voting Code-Review +1 or by commenting
>> 'recheck').
>>
>>
>> Unrelated is that since 14:30 a large amount of job have been
>> triggered.   The sole way I have found to solve it is to hard restart
>> Zuul but that would flush all its of its state.  I am investigating
>> and will come up with an enhancement later this week.
>
> Hello,
>
> The queue has been drained and CI behave properly now. An incident
> report will follow later on.

Hello,

There were two different issues but both causing similar effect to our
user. Hence there is a single incident report:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20190506-zuul

cheers,

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l