Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

Bryan Davis
On Thu, Feb 4, 2016 at 8:20 AM, MZMcBride <[hidden email]> wrote:
> Federico Leva (Nemo) wrote:
>>Login pretty much never does what I expect nowadays, but I'm not sure my
>>expectations are correct so I can't identify actual bugs.
>
> There are various open tasks in Phabricator about user sessions currently,
> such as <https://phabricator.wikimedia.org/T124440>. Being unexpectedly
> logged out lately has been a bit annoying, though I don't know if it's
> related to the Performance team or some other team.

The origin of the unexpected logouts falls on the AuthManager project
and specifically the SessionManager component that rolled out in
1.27.0-wmf.11 [0]. We had various issues related to the session
handling changes including a bug that was overloading the storage
capacity of the Redis servers that store session data [1] and two
other issues which required rolling the wikis back to 1.27.0-wmf.10
[2][3].

Both rollbacks were accompanied by a run of the
"resetGlobalUserTokens.php" maintenance script which updates each
user's CentralAuth records in such a way that their authentication
session will be considered invalid the next time it is used on a wiki.
This was done from an abundance of caution point of view concerning
possible issues with sessions that had been issued by the
SessionManager software. The reset script is not fast [4], so session
invalidation has slowly worked its way across the CentralAuth user
table.

Part of the enhancements that are being applied in order to bring
SessionManager back to production with 1.27.0-wmf.13 is a new config
setting that can be used to give us a nearly instant switch to throw
to invalidate all active sessions. This setting is actually included
in 1.27.0-wmf.12, but the configuration on the Wikimedia cluster has
not been changed to make use of it yet. Invalidating all user sessions
is not something we plan to do for fun certainly, but there have been
in the past (and likely will be in the future) software and
configuration issues that necessitate the use of that heavy hammer
approach.


[0]: https://phabricator.wikimedia.org/T123451
[1]: https://phabricator.wikimedia.org/T125267
[2]: https://wikitech.wikimedia.org/wiki/Incident_documentation/20160123-SessionManagerRolloutFailure
[3]: https://tools.wmflabs.org/sal/log/AVKZtfQXW8txF7J0uNE2
[4]: https://phabricator.wikimedia.org/T124861

Bryan
--
Bryan Davis              Wikimedia Foundation    <[hidden email]>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

Federico Leva (Nemo)
No, this is not what I'm talking about. My problems span multiple weeks
or months and I reiterate my need for a document outlining the expected
behaviour.

Nemo

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

Brad Jorsch (Anomie)
On Thu, Feb 4, 2016 at 4:00 PM, Federico Leva (Nemo) <[hidden email]>
wrote:

> No, this is not what I'm talking about. My problems span multiple weeks or
> months and I reiterate my need for a document outlining the expected
> behaviour.


Off the top of my head, it goes something like this on WMF wikis.

   1. You submit the login form on xxwiki. The response sets a bunch of
   cookies and redirects you to Special:CentralLogin/start on loginwiki.
   2. Loginwiki sets some cookies and redirects you to
   Special:CentralLogin/complete on xxwiki.
   3. xxwiki updates the cookies and redirects you to the returnto page.
   4. The returnto page will have a number of <img> tags for 1x1 images to
   try to log you in to the other domains in the cluster. It'll also have one
   to try to update the cookies on loginwiki.

The final set of cookies includes xxwikiSession, xxwikiUserID, and
xxwikiUserName locally, and centralauth_Session, centralauth_User, and (if
you checked "remember me") centralauth_Token set on the whole domain. For
most domains the whole domain is like ".wikipedia.org", while for stuff
under wikimedia.org it's the third level like "commons.wikimedia.org".

Even if nothing below works, you *should* be logged in on xxwiki and
loginwiki now.

The 1x1 <img> tags work like this, when they work. They can fail if the
browser blocks 1x1 images or third-party cookies. If any step fails due to
not having the right cookies, it'll just stop there and serve the
transparent PNG.

   1. The <img> tag points to Special:CentralAutoLogin/start on the target
   wiki. This will redirect to Special:CentralAutoLogin/checkLoggedIn on
   loginwiki.
   2. Loginwiki will redirect back to
   Special:CentralAutoLogin/createSession on the target wiki. Unless it thinks
   you're logged out, of course.
   3. The target wiki will set a session cookie and redirect to
   Special:CentralAutoLogin/validateSession on loginwiki.
   4. Loginwiki will redirect back to Special:CentralAutoLogin/setCookies
   on the target wiki.
   5. The target wiki will set all the relevant cookies and serve a
   transparent 1x1 PNG. Now you should be logged in when you visit any wiki on
   the domain.

When you visit a wiki, aren't logged in, and don't have the "I already did
this" token set in local storage, it does something much like the 1x1 <img>
flow except with a <script> tag via ResourceLoader. In the last step, it
will serve some JS that either replaces the user bar at the top of the
screen or shows a popup telling you to reload. If any step fails, it serves
either a do-nothing script or one that sets the "I already did this" token.
This also happens when you visit Special:UserLogin while logged out even if
"I already did this" is set.

Hope that helped. There's probably little point in reporting bugs against
1.27.0-wmf.12, wait for wmf.13 next Thursday in case they're fixed there.


--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

Federico Leva (Nemo)
Thanks Brad for the implementation explanatin but I'd rather need a
document outlining the expectations for user-facing functionality.

I just had to separately and manually login three times (on Meta, then
on Wikipedia, then on MediaWiki.org) in order to be logged in on most
wikis. I don't find any page explaining whether it's normal for global
login to require entering my password 3+ times. If it's expected and
documented, I can stop worrying.

Nemo

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

Gergo Tisza
On Fri, Jun 17, 2016 at 9:15 AM, Federico Leva (Nemo) <[hidden email]>
wrote:

> Thanks Brad for the implementation explanatin but I'd rather need a
> document outlining the expectations for user-facing functionality.
>
> I just had to separately and manually login three times (on Meta, then on
> Wikipedia, then on MediaWiki.org) in order to be logged in on most wikis. I
> don't find any page explaining whether it's normal for global login to
> require entering my password 3+ times. If it's expected and documented, I
> can stop worrying.


The expectation for single sign-on is that you sign-on a single time and
then you are signed on everywhere. If that's not what happens, feel free to
file a bug, ideally with a HAR file or something equivalent.

(I think there is one scenario when you need to login manually: when
visiting a wiki that's not included in the autologin domains while being
logged out, then logging in elsewhere, then visiting it again. The "user is
logged out" cookie gets set on the first visit, so on the second one no
autologin is attempted. But that should be very rare as Meta, Commons, and
all the not *.wikimedia.org wikis are autologin domains, so this can only
happen on exotic projects like outreach.wikimedia.org.)
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

Brad Jorsch (Anomie)
For anyone who missed it four months ago, this subthread is in response to
https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084714.html

On Fri, Jun 17, 2016 at 6:30 AM, Gergo Tisza <[hidden email]> wrote:

> On Fri, Jun 17, 2016 at 9:15 AM, Federico Leva (Nemo) <[hidden email]>
> wrote:
>
> > I just had to separately and manually login three times (on Meta, then on
> > Wikipedia, then on MediaWiki.org) in order to be logged in on most
> wikis. I
> > don't find any page explaining whether it's normal for global login to
> > require entering my password 3+ times. If it's expected and documented, I
> > can stop worrying.
>
> The expectation for single sign-on is that you sign-on a single time and
> then you are signed on everywhere. If that's not what happens, feel free to
> file a bug, ideally with a HAR file or something equivalent.
>

Note: A "HAR file" is a log of the HTTP requests and responses. Without
some kind of log like this your bug is unlikely to receive much feedback
beyond a request to provide one, since the process depends heavily on the
details of the HTTP requests.

But first, check your cookie settings. If you have third-party cookies
blocked, single sign-on just isn't going to work because it has to rely on
third-party cookies to function. An "Only allow from sites I've visited"
setting may work, depending on how exactly the browser implements it.

Specifically, if the browser either ignores the cookies sent during the
redirect to loginwiki or refuses to send them back to loginwiki for the
<img> or <script> checks, single sign-on just isn't going to work for you.
Similarly, if it ignores the cookies sent by the <img> tag for the other
domains (or an ad blocker or privacy blocker blocks the 1x1 images), those
other domains are going to have to rely on the first-visit script to log
you in and will be subject to the caveat that if you visited them before
logging in the first-visit script won't run.


--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Authentication session unwanted across wikis (was Recent authentication session instability)

Purodha Blissenbach
In reply to this post by Gergo Tisza
On 17.06.2016 12:30, Gergo Tisza wrote:

> The expectation for single sign-on is that you sign-on a single time
> and
> then you are signed on everywhere. If that's not what happens, [...]

If this is what happens, and I not want it, what can I do?

It is not rare that I need to be logged in with two identities at
two sites, such as admin here and non-admin there.


Purodha


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Authentication session unwanted across wikis (was Recent authentication session instability)

Sylvain Boissel
Hi,

You can use two browsers, or two browser sessions.

Sylvain

2016-06-17 18:04 GMT+02:00 Purodha Blissenbach <[hidden email]>:

> On 17.06.2016 12:30, Gergo Tisza wrote:
>
> The expectation for single sign-on is that you sign-on a single time and
>> then you are signed on everywhere. If that's not what happens, [...]
>>
>
> If this is what happens, and I not want it, what can I do?
>
> It is not rare that I need to be logged in with two identities at
> two sites, such as admin here and non-admin there.
>
>
> Purodha
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




--
*Sylvain Boissel*
ADMINISTRATEUR SYSTÈME ET RÉSEAUX
*WIKIMÉDIA FRANCE*


*Tél +33 1 42 36 97 72*
*www.wikimedia.fr <http://www.wikimedia.fr/>*
*40 rue de Cléry, **75002 Paris*
<http://www.openstreetmap.org/node/691082430#map=19/48.86814/2.34683>

*Imaginez un monde où chaque personne sur la planète aurait librement accès
à la totalité du savoir humain. C'est notre engagement. Aidez Wikimedia
France à en faire une réalité <https://dons.wikimedia.fr>.*
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

Federico Leva (Nemo)
In reply to this post by Brad Jorsch (Anomie)
Thanks for pointing out this expectation. Please place this goal
statement somewhere on the wiki.

I've sent bug reports with HAR files before, but it always proved to be
a lot of effort for zero gain. Perhaps there are specific tips to debug
such issues in a way that makes bug reporting effective?

It's of course possible that my browser's settings are interfering, but
the mentioned hypothesis sound like something that would prevent
crosswiki login in all cases, not just on some domains.

Nemo

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Authentication session unwanted across wikis (was Recent authentication session instability)

Purodha Blissenbach
In reply to this post by Sylvain Boissel
Hi,
This sound complicated and error prone.
Can we not simply have an off-switch in the user settings?
Purodha

On 17.06.2016 19:32, Sylvain Boissel wrote:

> Hi,
>
> You can use two browsers, or two browser sessions.
>
> Sylvain
>
> 2016-06-17 18:04 GMT+02:00 Purodha Blissenbach
> <[hidden email]>:
>
>> On 17.06.2016 12:30, Gergo Tisza wrote:
>>
>>> The expectation for single sign-on is that you sign-on a single
>>> time and
>>> then you are signed on everywhere. If that's not what happens,
>>> [...]
>>
>> If this is what happens, and I not want it, what can I do?
>>
>> It is not rare that I need to be logged in with two identities at
>> two sites, such as admin here and non-admin there.
>>
>> Purodha
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> --
>
> SYLVAIN BOISSEL
>
> ADMINISTRATEUR SYSTÈME ET RÉSEAUX
> _WIKIMÉDIA FRANCE_
>
> Tél +33 1 42 36 97 72
>
> _WWW.WIKIMEDIA.FR [1]_
> _40 rue de Cléry, __75002 Paris_ [2]
>
> _Imaginez un monde où chaque personne sur la planète aurait
> librement accès à la totalité du savoir humain. C'est notre
> engagement. Aidez Wikimedia France à en faire une réalité [3]._
>
> Links:
> ------
> [1] http://www.wikimedia.fr/
> [2] http://www.openstreetmap.org/node/691082430#map=19/48.86814/2.34683
> [3] https://dons.wikimedia.fr

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Authentication session unwanted across wikis (was Recent authentication session instability)

Brad Jorsch (Anomie)
On Jun 18, 2016 2:37 AM, "Purodha Blissenbach" <[hidden email]>
wrote:
>
> This sound complicated and error prone.
> Can we not simply have an off-switch in the user settings?

Such a switch would not be impossible, but the added complexity in what is
already an extremely complex system would likely be more prone to bugs and
other issues (which could easily affect *all* users) than the few people
with this need just using multiple browsers or browser profiles.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Recent authentication session instability (was Re: Update from the Wikimedia Performance Team)

Brad Jorsch (Anomie)
In reply to this post by Federico Leva (Nemo)
On Jun 17, 2016 7:30 PM, "Federico Leva (Nemo)" <[hidden email]> wrote:
>
> Thanks for pointing out this expectation. Please place this goal
statement somewhere on the wiki.
>
> I've sent bug reports with HAR files before, but it always proved to be a
lot of effort for zero gain. Perhaps there are specific tips to debug such
issues in a way that makes bug reporting effective?

I don't know which bugs those might have been, but in my experience SUL
bugs like this often come down to cookie handling so seeing exactly which
cookies are being sent is important. Other kinds of bugs don't so often.

> It's of course possible that my browser's settings are interfering, but
the mentioned hypothesis sound like something that would prevent crosswiki
login in all cases, not just on some domains.

I recently cleaned up some bad settings in my browser that were blocking
third-party cookies for some domains where I needed to allow them.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Authentication session unwanted across wikis (was Recent authentication session instability)

Purodha Blissenbach
In reply to this post by Brad Jorsch (Anomie)
On 19.06.2016 16:10, Brad Jorsch (Anomie) wrote:

> On Jun 18, 2016 2:37 AM, "Purodha Blissenbach"
> <[hidden email]> wrote:
>>
>> This sound complicated and error prone.
>> Can we not simply have an off-switch in the user settings?
>
> Such a switch would not be impossible, but the added complexity in
> what is already an extremely complex system would likely be more prone
> to bugs and other issues (which could easily affect *all* users) than
> the few people with this need just using multiple browsers or browser
> profiles.

The unified account login has been a pain my ass since the very earliest
discussions about it. I knew, it would not work well, and indeed, it
does
not work well.

You argue like a car maker that now offers automatic car only,
regardless of their customer base and regardless of the increased
fuel consumption, wear, etc.

I suggest, if you make an additional feature changing basic behavior,
have it as an addition, not a replacement. If you must make it a
default,
you must add an option to switch it off.

Purodha

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Authentication session unwanted across wikis (was Recent authentication session instability)

Andre Klapper-2
On Sun, 2016-06-19 at 16:48 +0200, Purodha Blissenbach wrote:
> The unified account login has been a pain my ass since the very earliest
> discussions about it. I knew, it would not work well, and indeed, it 
> does not work well.

Could you please point to bug reports which cover specific aspects so
that we can understand what "does not work well" for you?

> You argue like a car maker that now offers automatic car only,
> regardless of their customer base and regardless of the increased
> fuel consumption, wear, etc.

The 'customer base' asked for Single User Login:
https://meta.wikimedia.org/w/index.php?title=Wikimedia_Forum&diff=next&oldid=3619272

If your car with Single User Login increases fuel consumption, can you
please point to specific issues with steps to reproduce the problem?

> I suggest, if you make an additional feature changing basic behavior,
> have it as an addition, not a replacement. If you must make it a 
> default, you must add an option to switch it off.

Anomie already explained why adding options and if/then/else code
complexity creates huge software maintenance costs over the years...

andre
--

Andre Klapper | Wikimedia Bugwrangler
http://blogs.gnome.org/aklapper/

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l