CI response time slow today

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

CI response time slow today

Greg Grossmeier-2
Hello all,

I want to apologize for the current slowness in our CI stack. Everything
is working fine in the sense that nothing is broken.

We simply did not adequately prepare for the switch to php5.5 being the
new minimum version for MediaWiki. This resulted in a large influx of
jobs on a set of our Jenkins executers that was not a large enough pool.
Timo and Antoine have created more executers to help mitigate [0].

More explicitly: due to package version availability in Ubuntu, our
php5.3 jobs run on our pool of Ubuntu Precis hosts, while the
php5.5/hhvm jobs run on our pool of Ubuntu Trusty hosts.

Both sets of jobs are in our pipeline to migrate to the new Nodepool
system that allows burstable availability of executers.

Again, apologies for the delays but know that the jobs will complete
just not as fast as you are accustomed to.

Greg

[0] https://phabricator.wikimedia.org/T126423

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI response time slow today

Antoine Musso-3
Le 10/02/2016 22:06, Greg Grossmeier a écrit :

> Hello all,
>
> I want to apologize for the current slowness in our CI stack. Everything
> is working fine in the sense that nothing is broken.
>
> We simply did not adequately prepare for the switch to php5.5 being the
> new minimum version for MediaWiki. This resulted in a large influx of
> jobs on a set of our Jenkins executers that was not a large enough pool.
> Timo and Antoine have created more executers to help mitigate [0].
>
> More explicitly: due to package version availability in Ubuntu, our
> php5.3 jobs run on our pool of Ubuntu Precis hosts, while the
> php5.5/hhvm jobs run on our pool of Ubuntu Trusty hosts.
>
> Both sets of jobs are in our pipeline to migrate to the new Nodepool
> system that allows burstable availability of executers.
>
> Again, apologies for the delays but know that the jobs will complete
> just not as fast as you are accustomed to.
>
> Greg
>
> [0] https://phabricator.wikimedia.org/T126423

Hello,

As a follow, yesterday we created new instance turns out 2GB memory is
not enough to run some of the MediaWiki jobs.  Since yesterday you might
have encountered errors such as:

* Access denied for user 'jenkins_mw0'@'localhost'
* Disk I/O Error
* File that went away mysteriously
* Long delays in jobs

etc

A few minutes ago I have deleted all those 2GB slaves and spawned bigger
ones.  Provisioning is ongoing as I am writing. Will get them pooled in
a couple/three hours.


I do not have proof: but PHP 5.5 on Trusty might have its share of issue
as well when stress tests with our test suites.   After all we never
massively ran tests against 5.5 ;-}

cheers and be careful!

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

CI response time back to normal (was Re: CI response time slow today)

Greg Grossmeier-2
In reply to this post by Greg Grossmeier-2
<quote name="Greg Grossmeier" date="2016-02-10" time="13:06:03 -0800">
> Hello all,
>
> I want to apologize for the current slowness in our CI stack. Everything
> is working fine in the sense that nothing is broken.

Quick update on this issue.

CI response times should be back to normal (or much closer to normal)
after we addressed most of the issues with this yesterday (by adding
more Ubuntu Trusty Jenkins slaves, the migration to Nodepool is a longer
term goal).

Thank you for your understanding,

Greg

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l