CI and cross repository dependencies

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

CI and cross repository dependencies

Antoine Musso-3
Hello,

We often have the case of a change to an extension depending on a
pending patch to MediaWiki core.  I have upgraded our CI scheduler -
Zuul - a couple weeks ago and it now supports marking dependencies even
in different repositories.


Why does it matter?  To make sure the dependency is fulfilled one
usually either:

* CR-2 the patch until dependent change is merged
* write a test that exercise the required patch in MediaWiki.

With the first solution (lack of test), once both are merged, nothing
prevent one from cherry picking a patch without its dependent patch.
For example for MediaWiki minor releases or Wikimedia deployment branches.

When a test covers the dependency, it will fail until the dependent one
is merged which is rather annoying.


Zuul now recognizes the header 'Depends-On' in git messages, similar to
'Change-Id' and 'Bug'.  'Depends-On' takes as parameter a change-id and
multiple ones can be added.

When a patch is proposed in Gerrit, Zuul looks for Gerrit changes
matching the 'Depends-On' and verify whether any are still open. In such
a case, it will craft references for the open patches so all the
dependencies can be tested as if they got merged.


Real world example
------------------

The ContentTranslation extension is tested with the Wikidata one and was
not passing the test.  Wikidata created a patch and we did not want to
merge it until we confirm the ContentTranslation one is passing properly.

The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/
Change-Id: I0312c23628d706deb507b5534b868480945b6163

On ContentTranslation we indicated the dependency:
https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG
+ Depends-On: I0312c23628d706deb507b5534b868480945b6163

Which is the Wikidata patch.


Zuul:
* received the patch for ContentTranslation
* looked up the change-id and found the Wikidata
* created git references in both repo to point to the proper patches

Jenkins:
* zuul-cloner cloned both repos and fetched the references created by
the Zuul service
* run tests
* SUCCESS

That confirmed us the Wikidata patch was actually fixing the issue for
ContentTranslation. Hence we CR+2 both and all merged fine.


Please take a moment to read upstream documentation:

http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies

Wikidata/ContentTranslation task:
https://phabricator.wikimedia.org/T118263


--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Florian Schmidt
That sounds really great and very useful! Thanks for this info :)

Best,
Florian

-----Original-Nachricht-----
Betreff: [Wikitech-l] CI and cross repository dependencies
Datum: 2015-11-19T14:19:27+0100
Von: "Antoine Musso" <[hidden email]>
An: "[hidden email]" <[hidden email]>

Hello,

We often have the case of a change to an extension depending on a
pending patch to MediaWiki core.  I have upgraded our CI scheduler -
Zuul - a couple weeks ago and it now supports marking dependencies even
in different repositories.


Why does it matter?  To make sure the dependency is fulfilled one
usually either:

* CR-2 the patch until dependent change is merged
* write a test that exercise the required patch in MediaWiki.

With the first solution (lack of test), once both are merged, nothing
prevent one from cherry picking a patch without its dependent patch.
For example for MediaWiki minor releases or Wikimedia deployment branches.

When a test covers the dependency, it will fail until the dependent one
is merged which is rather annoying.


Zuul now recognizes the header 'Depends-On' in git messages, similar to
'Change-Id' and 'Bug'.  'Depends-On' takes as parameter a change-id and
multiple ones can be added.

When a patch is proposed in Gerrit, Zuul looks for Gerrit changes
matching the 'Depends-On' and verify whether any are still open. In such
a case, it will craft references for the open patches so all the
dependencies can be tested as if they got merged.


Real world example
------------------

The ContentTranslation extension is tested with the Wikidata one and was
not passing the test.  Wikidata created a patch and we did not want to
merge it until we confirm the ContentTranslation one is passing properly.

The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/
Change-Id: I0312c23628d706deb507b5534b868480945b6163

On ContentTranslation we indicated the dependency:
https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG
+ Depends-On: I0312c23628d706deb507b5534b868480945b6163

Which is the Wikidata patch.


Zuul:
* received the patch for ContentTranslation
* looked up the change-id and found the Wikidata
* created git references in both repo to point to the proper patches

Jenkins:
* zuul-cloner cloned both repos and fetched the references created by
the Zuul service
* run tests
* SUCCESS

That confirmed us the Wikidata patch was actually fixing the issue for
ContentTranslation. Hence we CR+2 both and all merged fine.


Please take a moment to read upstream documentation:

http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies

Wikidata/ContentTranslation task:
https://phabricator.wikimedia.org/T118263


--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Amir E. Aharoni
In reply to this post by Antoine Musso-3
Wonderful, thank you, this should be very useful.


--
Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
http://aharoni.wordpress.com
‪“We're living in pieces,
I want to live in peace.” – T. Moore‬

2015-11-19 15:19 GMT+02:00 Antoine Musso <[hidden email]>:

> Hello,
>
> We often have the case of a change to an extension depending on a
> pending patch to MediaWiki core.  I have upgraded our CI scheduler -
> Zuul - a couple weeks ago and it now supports marking dependencies even
> in different repositories.
>
>
> Why does it matter?  To make sure the dependency is fulfilled one
> usually either:
>
> * CR-2 the patch until dependent change is merged
> * write a test that exercise the required patch in MediaWiki.
>
> With the first solution (lack of test), once both are merged, nothing
> prevent one from cherry picking a patch without its dependent patch.
> For example for MediaWiki minor releases or Wikimedia deployment branches.
>
> When a test covers the dependency, it will fail until the dependent one
> is merged which is rather annoying.
>
>
> Zuul now recognizes the header 'Depends-On' in git messages, similar to
> 'Change-Id' and 'Bug'.  'Depends-On' takes as parameter a change-id and
> multiple ones can be added.
>
> When a patch is proposed in Gerrit, Zuul looks for Gerrit changes
> matching the 'Depends-On' and verify whether any are still open. In such
> a case, it will craft references for the open patches so all the
> dependencies can be tested as if they got merged.
>
>
> Real world example
> ------------------
>
> The ContentTranslation extension is tested with the Wikidata one and was
> not passing the test.  Wikidata created a patch and we did not want to
> merge it until we confirm the ContentTranslation one is passing properly.
>
> The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/
> Change-Id: I0312c23628d706deb507b5534b868480945b6163
>
> On ContentTranslation we indicated the dependency:
> https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG
> + Depends-On: I0312c23628d706deb507b5534b868480945b6163
>
> Which is the Wikidata patch.
>
>
> Zuul:
> * received the patch for ContentTranslation
> * looked up the change-id and found the Wikidata
> * created git references in both repo to point to the proper patches
>
> Jenkins:
> * zuul-cloner cloned both repos and fetched the references created by
> the Zuul service
> * run tests
> * SUCCESS
>
> That confirmed us the Wikidata patch was actually fixing the issue for
> ContentTranslation. Hence we CR+2 both and all merged fine.
>
>
> Please take a moment to read upstream documentation:
>
>
> http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies
>
> Wikidata/ContentTranslation task:
> https://phabricator.wikimedia.org/T118263
>
>
> --
> Antoine "hashar" Musso
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Sam Smith
This is great! A huge thanks to everyone involved.

-Sam

On Thu, Nov 19, 2015 at 5:26 PM, Amir E. Aharoni <
[hidden email]> wrote:

> Wonderful, thank you, this should be very useful.
>
>
> --
> Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
> http://aharoni.wordpress.com
> ‪“We're living in pieces,
> I want to live in peace.” – T. Moore‬
>
> 2015-11-19 15:19 GMT+02:00 Antoine Musso <[hidden email]>:
>
> > Hello,
> >
> > We often have the case of a change to an extension depending on a
> > pending patch to MediaWiki core.  I have upgraded our CI scheduler -
> > Zuul - a couple weeks ago and it now supports marking dependencies even
> > in different repositories.
> >
> >
> > Why does it matter?  To make sure the dependency is fulfilled one
> > usually either:
> >
> > * CR-2 the patch until dependent change is merged
> > * write a test that exercise the required patch in MediaWiki.
> >
> > With the first solution (lack of test), once both are merged, nothing
> > prevent one from cherry picking a patch without its dependent patch.
> > For example for MediaWiki minor releases or Wikimedia deployment
> branches.
> >
> > When a test covers the dependency, it will fail until the dependent one
> > is merged which is rather annoying.
> >
> >
> > Zuul now recognizes the header 'Depends-On' in git messages, similar to
> > 'Change-Id' and 'Bug'.  'Depends-On' takes as parameter a change-id and
> > multiple ones can be added.
> >
> > When a patch is proposed in Gerrit, Zuul looks for Gerrit changes
> > matching the 'Depends-On' and verify whether any are still open. In such
> > a case, it will craft references for the open patches so all the
> > dependencies can be tested as if they got merged.
> >
> >
> > Real world example
> > ------------------
> >
> > The ContentTranslation extension is tested with the Wikidata one and was
> > not passing the test.  Wikidata created a patch and we did not want to
> > merge it until we confirm the ContentTranslation one is passing properly.
> >
> > The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/
> > Change-Id: I0312c23628d706deb507b5534b868480945b6163
> >
> > On ContentTranslation we indicated the dependency:
> > https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG
> > + Depends-On: I0312c23628d706deb507b5534b868480945b6163
> >
> > Which is the Wikidata patch.
> >
> >
> > Zuul:
> > * received the patch for ContentTranslation
> > * looked up the change-id and found the Wikidata
> > * created git references in both repo to point to the proper patches
> >
> > Jenkins:
> > * zuul-cloner cloned both repos and fetched the references created by
> > the Zuul service
> > * run tests
> > * SUCCESS
> >
> > That confirmed us the Wikidata patch was actually fixing the issue for
> > ContentTranslation. Hence we CR+2 both and all merged fine.
> >
> >
> > Please take a moment to read upstream documentation:
> >
> >
> >
> http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies
> >
> > Wikidata/ContentTranslation task:
> > https://phabricator.wikimedia.org/T118263
> >
> >
> > --
> > Antoine "hashar" Musso
> >
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Brian Gerstle
Nice work!  Is this at all related to upstream/downstream Jenkins jobs?

On Thu, Nov 19, 2015 at 12:38 PM, Sam Smith <[hidden email]> wrote:

> This is great! A huge thanks to everyone involved.
>
> -Sam
>
> On Thu, Nov 19, 2015 at 5:26 PM, Amir E. Aharoni <
> [hidden email]> wrote:
>
> > Wonderful, thank you, this should be very useful.
> >
> >
> > --
> > Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
> > http://aharoni.wordpress.com
> > ‪“We're living in pieces,
> > I want to live in peace.” – T. Moore‬
> >
> > 2015-11-19 15:19 GMT+02:00 Antoine Musso <[hidden email]>:
> >
> > > Hello,
> > >
> > > We often have the case of a change to an extension depending on a
> > > pending patch to MediaWiki core.  I have upgraded our CI scheduler -
> > > Zuul - a couple weeks ago and it now supports marking dependencies even
> > > in different repositories.
> > >
> > >
> > > Why does it matter?  To make sure the dependency is fulfilled one
> > > usually either:
> > >
> > > * CR-2 the patch until dependent change is merged
> > > * write a test that exercise the required patch in MediaWiki.
> > >
> > > With the first solution (lack of test), once both are merged, nothing
> > > prevent one from cherry picking a patch without its dependent patch.
> > > For example for MediaWiki minor releases or Wikimedia deployment
> > branches.
> > >
> > > When a test covers the dependency, it will fail until the dependent one
> > > is merged which is rather annoying.
> > >
> > >
> > > Zuul now recognizes the header 'Depends-On' in git messages, similar to
> > > 'Change-Id' and 'Bug'.  'Depends-On' takes as parameter a change-id and
> > > multiple ones can be added.
> > >
> > > When a patch is proposed in Gerrit, Zuul looks for Gerrit changes
> > > matching the 'Depends-On' and verify whether any are still open. In
> such
> > > a case, it will craft references for the open patches so all the
> > > dependencies can be tested as if they got merged.
> > >
> > >
> > > Real world example
> > > ------------------
> > >
> > > The ContentTranslation extension is tested with the Wikidata one and
> was
> > > not passing the test.  Wikidata created a patch and we did not want to
> > > merge it until we confirm the ContentTranslation one is passing
> properly.
> > >
> > > The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/
> > > Change-Id: I0312c23628d706deb507b5534b868480945b6163
> > >
> > > On ContentTranslation we indicated the dependency:
> > > https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG
> > > + Depends-On: I0312c23628d706deb507b5534b868480945b6163
> > >
> > > Which is the Wikidata patch.
> > >
> > >
> > > Zuul:
> > > * received the patch for ContentTranslation
> > > * looked up the change-id and found the Wikidata
> > > * created git references in both repo to point to the proper patches
> > >
> > > Jenkins:
> > > * zuul-cloner cloned both repos and fetched the references created by
> > > the Zuul service
> > > * run tests
> > > * SUCCESS
> > >
> > > That confirmed us the Wikidata patch was actually fixing the issue for
> > > ContentTranslation. Hence we CR+2 both and all merged fine.
> > >
> > >
> > > Please take a moment to read upstream documentation:
> > >
> > >
> > >
> >
> http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies
> > >
> > > Wikidata/ContentTranslation task:
> > > https://phabricator.wikimedia.org/T118263
> > >
> > >
> > > --
> > > Antoine "hashar" Musso
> > >
> > >
> > > _______________________________________________
> > > Wikitech-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Antoine Musso-3
In reply to this post by Sam Smith
Le 19/11/2015 18:38, Sam Smith a écrit :
> This is great! A huge thanks to everyone involved.
>
> -Sam

To be fair, I have not done much.  The feature has been written by James
E. Blair for the OpenStack Fundation.

On my side, I merely carefully reviewed six months worth of code and
made sure the feature wasn't going to over load our Gerrit :-}


--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Antoine Musso-3
In reply to this post by Brian Gerstle
Le 19/11/2015 18:51, Brian Gerstle a écrit :
> Nice work!
>
> Is this at all related to upstream/downstream Jenkins jobs?

The Zuul system does not rely at all on Jenkins upstream/downstream. One
can think of Jenkins as a huge collections of independent shell scripts
that are waiting to be executed with appropriate parameters.

OpenStack has a spec to get rid of Jenkins entirely and instead have
Zuul create an Ansible play book to run on a machine.  But really that
is another topic.


To elaborate a bit more:

Gerrit does support dependencies between changes, but only in the same
repository and branch. You can see that in the Gerrit web interface, and
Gerrit will refuse to merge a change for which the parent is not merged yet.

Zuul does the same but independently from Gerrit. It is merely filling
the gap of Gerrit lacks of cross repositories dependencies.

When a change is voted +2, it is enqueued in 'gate-and-submit'.  Zuul
immediately verify whether the dependencies are either merged or ahead
in the queue, else it will reject the change and report back in Gerrit.

So if you have change A and change B depending on A. You +2 A then B and
the queue is:

 A <-- B (depend on A)

A is processed (no dependency)
For B, Zuul find the dependency A ahead and thus it is processed.

If A fails the tests, B tests are automatically cancelled and the change
dequeued.  Zuul knows B depends on A.

Assuming all changes are merged by Zuul (via CR+2), Zuul dependency
comes on top of Gerrit and nicely enforce dependencies.



--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Brian Gerstle
>
> Jenkins as a huge collections of independent shell scripts
> that are waiting to be executed with appropriate parameters.


Right that made me wonder... but then you said:

OpenStack has a spec to get rid of Jenkins entirely and instead have
> Zuul create an Ansible play book to run on a machine.  But really that
> is another topic.


Makes sense that we should drop Jenkins if we're not leveraging it's
features.

Congrats again on contributing back to the OSS community!

On Thu, Nov 19, 2015 at 2:27 PM, Antoine Musso <[hidden email]> wrote:

> Le 19/11/2015 18:51, Brian Gerstle a écrit :
> > Nice work!
> >
> > Is this at all related to upstream/downstream Jenkins jobs?
>
> The Zuul system does not rely at all on Jenkins upstream/downstream. One
> can think of Jenkins as a huge collections of independent shell scripts
> that are waiting to be executed with appropriate parameters.
>
> OpenStack has a spec to get rid of Jenkins entirely and instead have
> Zuul create an Ansible play book to run on a machine.  But really that
> is another topic.
>
>
> To elaborate a bit more:
>
> Gerrit does support dependencies between changes, but only in the same
> repository and branch. You can see that in the Gerrit web interface, and
> Gerrit will refuse to merge a change for which the parent is not merged
> yet.
>
> Zuul does the same but independently from Gerrit. It is merely filling
> the gap of Gerrit lacks of cross repositories dependencies.
>
> When a change is voted +2, it is enqueued in 'gate-and-submit'.  Zuul
> immediately verify whether the dependencies are either merged or ahead
> in the queue, else it will reject the change and report back in Gerrit.
>
> So if you have change A and change B depending on A. You +2 A then B and
> the queue is:
>
>  A <-- B (depend on A)
>
> A is processed (no dependency)
> For B, Zuul find the dependency A ahead and thus it is processed.
>
> If A fails the tests, B tests are automatically cancelled and the change
> dequeued.  Zuul knows B depends on A.
>
> Assuming all changes are merged by Zuul (via CR+2), Zuul dependency
> comes on top of Gerrit and nicely enforce dependencies.
>
>
>
> --
> Antoine "hashar" Musso
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Jon Robson
This is great. Just tried it out on
https://gerrit.wikimedia.org/r/#/c/238992/ and hopefully it will soon
be passing. This should hopefully help bubble up important dependent
patches that might be overlooked if developers are not watching
certain repositories and simply ignoring patches to Jenkins -2ing
which can be for a variety of reasons :)

Now I just need to get out of the habit of writing "Dependency:" in
favour of "Depends-On:" :)


On Thu, Nov 19, 2015 at 2:51 PM, Brian Gerstle <[hidden email]> wrote:

>>
>> Jenkins as a huge collections of independent shell scripts
>> that are waiting to be executed with appropriate parameters.
>
>
> Right that made me wonder... but then you said:
>
> OpenStack has a spec to get rid of Jenkins entirely and instead have
>> Zuul create an Ansible play book to run on a machine.  But really that
>> is another topic.
>
>
> Makes sense that we should drop Jenkins if we're not leveraging it's
> features.
>
> Congrats again on contributing back to the OSS community!
>
> On Thu, Nov 19, 2015 at 2:27 PM, Antoine Musso <[hidden email]> wrote:
>
>> Le 19/11/2015 18:51, Brian Gerstle a écrit :
>> > Nice work!
>> >
>> > Is this at all related to upstream/downstream Jenkins jobs?
>>
>> The Zuul system does not rely at all on Jenkins upstream/downstream. One
>> can think of Jenkins as a huge collections of independent shell scripts
>> that are waiting to be executed with appropriate parameters.
>>
>> OpenStack has a spec to get rid of Jenkins entirely and instead have
>> Zuul create an Ansible play book to run on a machine.  But really that
>> is another topic.
>>
>>
>> To elaborate a bit more:
>>
>> Gerrit does support dependencies between changes, but only in the same
>> repository and branch. You can see that in the Gerrit web interface, and
>> Gerrit will refuse to merge a change for which the parent is not merged
>> yet.
>>
>> Zuul does the same but independently from Gerrit. It is merely filling
>> the gap of Gerrit lacks of cross repositories dependencies.
>>
>> When a change is voted +2, it is enqueued in 'gate-and-submit'.  Zuul
>> immediately verify whether the dependencies are either merged or ahead
>> in the queue, else it will reject the change and report back in Gerrit.
>>
>> So if you have change A and change B depending on A. You +2 A then B and
>> the queue is:
>>
>>  A <-- B (depend on A)
>>
>> A is processed (no dependency)
>> For B, Zuul find the dependency A ahead and thus it is processed.
>>
>> If A fails the tests, B tests are automatically cancelled and the change
>> dequeued.  Zuul knows B depends on A.
>>
>> Assuming all changes are merged by Zuul (via CR+2), Zuul dependency
>> comes on top of Gerrit and nicely enforce dependencies.
>>
>>
>>
>> --
>> Antoine "hashar" Musso
>>
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>
>
> --
> EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
> IRC: bgerstle
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Jon Robson
* http://jonrobson.me.uk
* https://www.facebook.com/jonrobson
* @rakugojon

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Marko Obrovac
Thanks a ton, Antoine, for this! It has just made our lives sooo much
easier for properly testing https://gerrit.wikimedia.org/r/#/c/254086/ :)

On 2 December 2015 at 18:44, Jon Robson <[hidden email]> wrote:

> This is great. Just tried it out on
> https://gerrit.wikimedia.org/r/#/c/238992/ and hopefully it will soon
> be passing. This should hopefully help bubble up important dependent
> patches that might be overlooked if developers are not watching
> certain repositories and simply ignoring patches to Jenkins -2ing
> which can be for a variety of reasons :)
>
> Now I just need to get out of the habit of writing "Dependency:" in
> favour of "Depends-On:" :)
>

True. My favourite line was:

Note: depends on <changeset-id>

In my case, I just need to drop "note:" and change the casing :P

Cheers,
Marko


>
> On Thu, Nov 19, 2015 at 2:51 PM, Brian Gerstle <[hidden email]>
> wrote:
> >>
> >> Jenkins as a huge collections of independent shell scripts
> >> that are waiting to be executed with appropriate parameters.
> >
> >
> > Right that made me wonder... but then you said:
> >
> > OpenStack has a spec to get rid of Jenkins entirely and instead have
> >> Zuul create an Ansible play book to run on a machine.  But really that
> >> is another topic.
> >
> >
> > Makes sense that we should drop Jenkins if we're not leveraging it's
> > features.
> >
> > Congrats again on contributing back to the OSS community!
> >
> > On Thu, Nov 19, 2015 at 2:27 PM, Antoine Musso <[hidden email]>
> wrote:
> >
> >> Le 19/11/2015 18:51, Brian Gerstle a écrit :
> >> > Nice work!
> >> >
> >> > Is this at all related to upstream/downstream Jenkins jobs?
> >>
> >> The Zuul system does not rely at all on Jenkins upstream/downstream. One
> >> can think of Jenkins as a huge collections of independent shell scripts
> >> that are waiting to be executed with appropriate parameters.
> >>
> >> OpenStack has a spec to get rid of Jenkins entirely and instead have
> >> Zuul create an Ansible play book to run on a machine.  But really that
> >> is another topic.
> >>
> >>
> >> To elaborate a bit more:
> >>
> >> Gerrit does support dependencies between changes, but only in the same
> >> repository and branch. You can see that in the Gerrit web interface, and
> >> Gerrit will refuse to merge a change for which the parent is not merged
> >> yet.
> >>
> >> Zuul does the same but independently from Gerrit. It is merely filling
> >> the gap of Gerrit lacks of cross repositories dependencies.
> >>
> >> When a change is voted +2, it is enqueued in 'gate-and-submit'.  Zuul
> >> immediately verify whether the dependencies are either merged or ahead
> >> in the queue, else it will reject the change and report back in Gerrit.
> >>
> >> So if you have change A and change B depending on A. You +2 A then B and
> >> the queue is:
> >>
> >>  A <-- B (depend on A)
> >>
> >> A is processed (no dependency)
> >> For B, Zuul find the dependency A ahead and thus it is processed.
> >>
> >> If A fails the tests, B tests are automatically cancelled and the change
> >> dequeued.  Zuul knows B depends on A.
> >>
> >> Assuming all changes are merged by Zuul (via CR+2), Zuul dependency
> >> comes on top of Gerrit and nicely enforce dependencies.
> >>
> >>
> >>
> >> --
> >> Antoine "hashar" Musso
> >>
> >>
> >> _______________________________________________
> >> Wikitech-l mailing list
> >> [hidden email]
> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >>
> >
> >
> >
> > --
> > EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
> > IRC: bgerstle
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Jon Robson
> * http://jonrobson.me.uk
> * https://www.facebook.com/jonrobson
> * @rakugojon
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
Marko Obrovac, PhD
Senior Services Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Jon Robson
This doesn't seem to work for https://gerrit.wikimedia.org/r/#/c/238992/
Any ideas why? Is it because the Gather patch is on a different branch
or because I'm not using the commit message right?


On Wed, Dec 2, 2015 at 2:36 PM, Marko Obrovac <[hidden email]> wrote:

> Thanks a ton, Antoine, for this! It has just made our lives sooo much
> easier for properly testing https://gerrit.wikimedia.org/r/#/c/254086/ :)
>
> On 2 December 2015 at 18:44, Jon Robson <[hidden email]> wrote:
>
>> This is great. Just tried it out on
>> https://gerrit.wikimedia.org/r/#/c/238992/ and hopefully it will soon
>> be passing. This should hopefully help bubble up important dependent
>> patches that might be overlooked if developers are not watching
>> certain repositories and simply ignoring patches to Jenkins -2ing
>> which can be for a variety of reasons :)
>>
>> Now I just need to get out of the habit of writing "Dependency:" in
>> favour of "Depends-On:" :)
>>
>
> True. My favourite line was:
>
> Note: depends on <changeset-id>
>
> In my case, I just need to drop "note:" and change the casing :P
>
> Cheers,
> Marko
>
>
>>
>> On Thu, Nov 19, 2015 at 2:51 PM, Brian Gerstle <[hidden email]>
>> wrote:
>> >>
>> >> Jenkins as a huge collections of independent shell scripts
>> >> that are waiting to be executed with appropriate parameters.
>> >
>> >
>> > Right that made me wonder... but then you said:
>> >
>> > OpenStack has a spec to get rid of Jenkins entirely and instead have
>> >> Zuul create an Ansible play book to run on a machine.  But really that
>> >> is another topic.
>> >
>> >
>> > Makes sense that we should drop Jenkins if we're not leveraging it's
>> > features.
>> >
>> > Congrats again on contributing back to the OSS community!
>> >
>> > On Thu, Nov 19, 2015 at 2:27 PM, Antoine Musso <[hidden email]>
>> wrote:
>> >
>> >> Le 19/11/2015 18:51, Brian Gerstle a écrit :
>> >> > Nice work!
>> >> >
>> >> > Is this at all related to upstream/downstream Jenkins jobs?
>> >>
>> >> The Zuul system does not rely at all on Jenkins upstream/downstream. One
>> >> can think of Jenkins as a huge collections of independent shell scripts
>> >> that are waiting to be executed with appropriate parameters.
>> >>
>> >> OpenStack has a spec to get rid of Jenkins entirely and instead have
>> >> Zuul create an Ansible play book to run on a machine.  But really that
>> >> is another topic.
>> >>
>> >>
>> >> To elaborate a bit more:
>> >>
>> >> Gerrit does support dependencies between changes, but only in the same
>> >> repository and branch. You can see that in the Gerrit web interface, and
>> >> Gerrit will refuse to merge a change for which the parent is not merged
>> >> yet.
>> >>
>> >> Zuul does the same but independently from Gerrit. It is merely filling
>> >> the gap of Gerrit lacks of cross repositories dependencies.
>> >>
>> >> When a change is voted +2, it is enqueued in 'gate-and-submit'.  Zuul
>> >> immediately verify whether the dependencies are either merged or ahead
>> >> in the queue, else it will reject the change and report back in Gerrit.
>> >>
>> >> So if you have change A and change B depending on A. You +2 A then B and
>> >> the queue is:
>> >>
>> >>  A <-- B (depend on A)
>> >>
>> >> A is processed (no dependency)
>> >> For B, Zuul find the dependency A ahead and thus it is processed.
>> >>
>> >> If A fails the tests, B tests are automatically cancelled and the change
>> >> dequeued.  Zuul knows B depends on A.
>> >>
>> >> Assuming all changes are merged by Zuul (via CR+2), Zuul dependency
>> >> comes on top of Gerrit and nicely enforce dependencies.
>> >>
>> >>
>> >>
>> >> --
>> >> Antoine "hashar" Musso
>> >>
>> >>
>> >> _______________________________________________
>> >> Wikitech-l mailing list
>> >> [hidden email]
>> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> >>
>> >
>> >
>> >
>> > --
>> > EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
>> > IRC: bgerstle
>> > _______________________________________________
>> > Wikitech-l mailing list
>> > [hidden email]
>> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>
>>
>> --
>> Jon Robson
>> * http://jonrobson.me.uk
>> * https://www.facebook.com/jonrobson
>> * @rakugojon
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>
>
> --
> Marko Obrovac, PhD
> Senior Services Engineer
> Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Jon Robson
* http://jonrobson.me.uk
* https://www.facebook.com/jonrobson
* @rakugojon

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: CI and cross repository dependencies

Antoine Musso-3
Le 03/12/2015 02:17, Jon Robson a écrit :
> This doesn't seem to work for https://gerrit.wikimedia.org/r/#/c/238992/
> Any ideas why? Is it because the Gather patch is on a different branch
> or because I'm not using the commit message right?

Hello,

Indeed the patch from Gather dev branch is not found, because the job
consider the branch to be 'master'.

Long read, one feel free to copy to the wiki!


context
=======

https://gerrit.wikimedia.org/r/#/c/238992/
MobileFrontend patch against master

It relies on some Javascripts introduced in Gather dev branch by:
https://gerrit.wikimedia.org/r/#/c/250826/

The job mediawiki-extensions-qunit on MobileFrontend fail.  That job
clones a subset of extensions deployed on Wikimedia cluster and run the
whole QUnit test suite.


What Zuul does?
===============


Prepare patches
---------------

When the patch for MobileFrontend is received, Zuul detects it depends
on the Gather change and thus build the dependency chain.  It then ask
for zuul-merger to craft merge commits for each patches against the tip
of their branches and tag them. The output from the merge system is:


CreateZuulRef dev/Zfb21011d8e154089a0657b3a419d739e on Gather
CreateZuulRef master/Zfb21011d8e154089a0657b3a419d739e on MobileFrontend

The refs are prefixed with the branch the patch has been merged against.
Note the ideas are the same, that is the unique identifier for this
specific build.

The references are exposed in git under refs/zuul/*


Retrieval of refs
-----------------

Zuul triggers the builds with the parameters:

ZUUL_REF=refs/zuul/master/Zfb21011d8e154089a0657b3a419d739e
ZUUL_BRANCH=master

'master' because that is the tip of MobileFrontend.


zuul-cloner takes care of cloning the repositories and checking out the
patch / branch.  The order of lookups are:

1) Zuul reference for the indicated branch
2) Zuul reference for the master branch
3) The tip of the indicated branch
4) The tip of the master branch

Ref:
http://docs.openstack.org/infra/zuul/cloner.html#clone-order

Where "indicated branch" is ZUUL_BRANCH or 'master'.


When looking at the zuul-cloner output:

Creating repo mediawiki/extensions/Gather
   upstream repo has branch master
   does not have ref refs/zuul/master/Zfb21011d8e154089a0657b3a419d739e
   Falling back to branch master
   Checking out remotes/origin/master

The master/Z... ref is not found for Gather cause the patch was againt
dev branch and the actual ref is dev/Z... So it fallback to master and
your patch is not included.


Creating repo mediawiki/extensions/MobileFrontend from upstream
   upstream repo has branch master
   Fetched ref refs/zuul/master/Zfb21011d8e154089a0657b3a419d739e


What??
------

Since MobileFrontend@master is tested with Gather@master, one would need
the patch to dev to be cherry picked to master and that would work.

The same issue will occur for Gather@dev, other repositories will use
the tip of master since dev branches do not exists.


You can override the branch either for all projects (--branch) or on a
per project basis (--project-branch).  So in theory, we could have a
second job that properly fetch from dev, but the other job will still fail.


What do I do now?
-----------------

Once your Gather patch in 'dev' is reviewed and merged, you can cherry
pick against master (the Change-Id will be the same). Then 'recheck' the
MobileFrontend patch which should be happy.  You can +2 it, it is not
going to merge till the change to Gather is merged in master.

Then, we can cut the wmf branches, deploy on the cluster with high
confident and process with the next feature :-}

cheers,

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l