Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

John Mark Vandenberg
On Fri, May 29, 2015 at 2:09 AM, Brian Wolff <[hidden email]> wrote:
> Shouldnt such [pywikibot] tests be run against beta wiki not testwiki?

Primarily this hasnt happened because pywikibot doesnt have a family
file for the beta wiki, but that is our issue, so I've done a simple
test run on beta wiki

four initial beta wiki problems:

- no https

(not nice - that means test accounts must be created and accessed
using passwords that are sent in essentially cleartext - so sharing
passwords with the same account name on the real wikis is a security
risk)

- invalid siteinfo interwiki map; includes entries to wikis that do
not exist.  That means pywikibot cant parse wikitext links, as it cant
distinguish between iwprefix: and namespace: (pywikibot team might be
able to reduce the impact of this wiki config problem)

- paramInfo failure that breaks pywikibot's dynamic api detection;
pywikibot batch loads the paraminfo for all modules, for example
issuing the following API as part of the initialisation, and it is a
503 Service Unavailable
http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=abusefiltercheckmatch|abusefilterchecksyntax|abusefilterevalexpression|abusefilterunblockautopromote|antispoof|block|bouncehandler|centralauthtoken|centralnoticebannerchoicedata|centralnoticequerycampaign|checktoken|cirrus-config-dump|cirrus-mapping-dump|cirrus-settings-dump|clearhasmsg|compare|createaccount|cxconfiguration|cxdelete|cxpublish|delete|deleteglobalaccount|echomarkread|echomarkseen|edit|editlist|editmassmessagelist|emailuser|expandtemplates|fancycaptchareload|featuredfeed|feedcontributions|feedrecentchanges|feedwatchlist|filerevert|flowthank|globalblock|globaluserrights|help|imagerotate|import|jsonconfig|languagesearch|login|logout|managetags|massmessage|mobileview|move|opensearch&maxlag=5&format=json

A little digging shows that the problematic module is fancycaptchareload

http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=fancycaptchareload&maxlag=5&format=json

"Our servers are currently experiencing a technical problem. This is
probably temporary and should be fixed soon. Please try again in a few
minutes.

If you report this error to the Wikimedia System Administrators,
please include the details below.
Request: GET http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=fancycaptchareload&maxlag=5&format=json,
from 127.0.0.1 via deployment-cache-text02 deployment-cache-text02
([127.0.0.1]:3128), Varnish XID 855090368
Forwarded for: [your IP], 127.0.0.1
Error: 503, Service Unavailable at Fri, 29 May 2015 07:57:09 GMT "

Could someone investigate this / fix that module?

Pywikibot devs could also help reduce the impact of this type of
problem in the future, by falling back from batch mode fetch to
loading individual modules in order to skip buggy modules.

- no SUL with the real wikis

(probably the best choice given no https on the beta cluster, but it
complicates adding beta wiki to our existing Travis-CI test matrix
which includes real wikis)

actual results at
https://travis-ci.org/jayvdb/pywikibot-core/builds/64532033

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

John Mark Vandenberg
On Fri, May 29, 2015 at 3:14 PM, John Mark Vandenberg <[hidden email]> wrote:

> ..
> - paramInfo failure that breaks pywikibot's dynamic api detection;
> pywikibot batch loads the paraminfo for all modules, for example
> issuing the following API as part of the initialisation, and it is a
> 503 Service Unavailable
> http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=abusefiltercheckmatch|abusefilterchecksyntax|abusefilterevalexpression|abusefilterunblockautopromote|antispoof|block|bouncehandler|centralauthtoken|centralnoticebannerchoicedata|centralnoticequerycampaign|checktoken|cirrus-config-dump|cirrus-mapping-dump|cirrus-settings-dump|clearhasmsg|compare|createaccount|cxconfiguration|cxdelete|cxpublish|delete|deleteglobalaccount|echomarkread|echomarkseen|edit|editlist|editmassmessagelist|emailuser|expandtemplates|fancycaptchareload|featuredfeed|feedcontributions|feedrecentchanges|feedwatchlist|filerevert|flowthank|globalblock|globaluserrights|help|imagerotate|import|jsonconfig|languagesearch|login|logout|managetags|massmessage|mobileview|move|opensearch&maxlag=5&format=json
>
> A little digging shows that the problematic module is fancycaptchareload
>
> http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=fancycaptchareload&maxlag=5&format=json
>
> "Our servers are currently experiencing a technical problem. This is
> probably temporary and should be fixed soon. Please try again in a few
> minutes.
>
> If you report this error to the Wikimedia System Administrators,
> please include the details below.
> Request: GET http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=fancycaptchareload&maxlag=5&format=json,
> from 127.0.0.1 via deployment-cache-text02 deployment-cache-text02
> ([127.0.0.1]:3128), Varnish XID 855090368
> Forwarded for: [your IP], 127.0.0.1
> Error: 503, Service Unavailable at Fri, 29 May 2015 07:57:09 GMT "
>
> Could someone investigate this / fix that module?

After pywikibot devs cleared some other test breakages (our problems)
we see this problem is also affecting the test wikis, which I assume
means this bug will be affecting real wikis soon, so I've created an
Unbreak Now! ticket for it and cc'd @greg.

https://phabricator.wikimedia.org/T100775

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

Alex Monk
In reply to this post by John Mark Vandenberg
On 29 May 2015 at 09:14, John Mark Vandenberg <[hidden email]> wrote:

> - no https
>
> (not nice - that means test accounts must be created and accessed
> using passwords that are sent in essentially cleartext - so sharing
> passwords with the same account name on the real wikis is a security
> risk)
>
It's risky anyway, do you know who has access to the beta cluster? It's not
considered secure and you do not need any NDA or anything to get access -
if you are using a real password on beta, change it. It's in labs.
https://phabricator.wikimedia.org/T50501 is about HTTPS on beta.


> - no SUL with the real wikis
>
> (probably the best choice given no https on the beta cluster, but it
> complicates adding beta wiki to our existing Travis-CI test matrix
> which includes real wikis)

Beta cluster will never get access to CentralAuth passwords in production.
Maybe via OpenID or something, but not proper SUL with production wikis.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

Jeremy Baron
On May 29, 2015 09:10, "Alex Monk" <[hidden email]> wrote:
> On 29 May 2015 at 09:14, John Mark Vandenberg <[hidden email]> wrote:
> It's risky anyway, do you know who has access to the beta cluster? It's
not
> considered secure and you do not need any NDA or anything to get access -
> if you are using a real password on beta, change it. It's in labs.

+1. we don't really need HTTPS on beta to secure beta. Rogue sysops and
other vandals aren't the end of the world. rather we need HTTPS so that we
can do tests with HTTPS.

Don't share passwords between beta and any other site unless you also don't
care about the other site. (e.g. you could share if both passwords are so
weak as abcde12345)

-Jeremy
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

John Mark Vandenberg
In reply to this post by John Mark Vandenberg
On Fri, May 29, 2015 at 3:14 PM, John Mark Vandenberg <[hidden email]> wrote:
> On Fri, May 29, 2015 at 2:09 AM, Brian Wolff <[hidden email]> wrote:
>> Shouldnt such [pywikibot] tests be run against beta wiki not testwiki?
>
> Primarily this hasnt happened because pywikibot doesnt have a family
> file for the beta wiki, but that is our issue, so I've done a simple
> test run on beta wiki

I've create a task for this little project:

https://phabricator.wikimedia.org/T100796

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

Brad Jorsch (Anomie)
In reply to this post by John Mark Vandenberg
On Fri, May 29, 2015 at 7:01 AM, John Mark Vandenberg <[hidden email]>
wrote:

> On Fri, May 29, 2015 at 3:14 PM, John Mark Vandenberg <[hidden email]>
> wrote:
> > ..
> > - paramInfo failure that breaks pywikibot's dynamic api detection;
> > pywikibot batch loads the paraminfo for all modules, for example
> > issuing the following API as part of the initialisation, and it is a
> > 503 Service Unavailable
> >
> http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=abusefiltercheckmatch|abusefilterchecksyntax|abusefilterevalexpression|abusefilterunblockautopromote|antispoof|block|bouncehandler|centralauthtoken|centralnoticebannerchoicedata|centralnoticequerycampaign|checktoken|cirrus-config-dump|cirrus-mapping-dump|cirrus-settings-dump|clearhasmsg|compare|createaccount|cxconfiguration|cxdelete|cxpublish|delete|deleteglobalaccount|echomarkread|echomarkseen|edit|editlist|editmassmessagelist|emailuser|expandtemplates|fancycaptchareload|featuredfeed|feedcontributions|feedrecentchanges|feedwatchlist|filerevert|flowthank|globalblock|globaluserrights|help|imagerotate|import|jsonconfig|languagesearch|login|logout|managetags|massmessage|mobileview|move|opensearch&maxlag=5&format=json
> >
> > A little digging shows that the problematic module is fancycaptchareload
> >
> >
> http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=fancycaptchareload&maxlag=5&format=json
> >
> > "Our servers are currently experiencing a technical problem. This is
> > probably temporary and should be fixed soon. Please try again in a few
> > minutes.
> >
> > If you report this error to the Wikimedia System Administrators,
> > please include the details below.
> > Request: GET
> http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfo&modules=fancycaptchareload&maxlag=5&format=json
> ,
> > from 127.0.0.1 via deployment-cache-text02 deployment-cache-text02
> > ([127.0.0.1]:3128), Varnish XID 855090368
> > Forwarded for: [your IP], 127.0.0.1
> > Error: 503, Service Unavailable at Fri, 29 May 2015 07:57:09 GMT "
> >
> > Could someone investigate this / fix that module?
>
> After pywikibot devs cleared some other test breakages (our problems)
> we see this problem is also affecting the test wikis, which I assume
> means this bug will be affecting real wikis soon, so I've created an
> Unbreak Now! ticket for it and cc'd @greg.
>
> https://phabricator.wikimedia.org/T100775
>

https://gerrit.wikimedia.org/r/#/c/214610/

I'll very likely go ahead and backport it to 1.26wmf8 as soon as it's
merged and I confirm Beta is fixed.



--
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l