Enforcing ORES limit of parallel connections per IP

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Enforcing ORES limit of parallel connections per IP

Amir Sarabadani
Hello,
ORES has policy that no more than four parallel connections per IP should
be requesting the service. We weren't enforcing this policy and that caused
several outages before.

As of today if there is more than four connections per IP, the responses
after fourth one will be throttled and will be slower, and after seventh
connection, ORES just responds with immediate time out error.

This might affect CloudVPS and Toolforge users who connect to ORES from
there but there are several IPs coming from the cloud and looking at
request statistics, this limit is bigger than the usage. But if you are
having trouble using ORES in the cloud, let me know so I add those IPs to
the whitelist.

This limit only applies to external connections and it should not affect
internal requests coming from mediawiki or the change propagation service.

For more information, please read https://phabricator.wikimedia.org/T160692
and file a task against ORES project in phabricator in case anything is
wrong.

Happy scoring!
Best
--
Amir Sarabadani
Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de

Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/

Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Enforcing ORES limit of parallel connections per IP

Chad
Wouldn't it make to send a 429 instead of 408?

-Chad

On Sep 19, 2018 1:37 PM, "Amir Sarabadani" <[hidden email]>
wrote:

Hello,
ORES has policy that no more than four parallel connections per IP should
be requesting the service. We weren't enforcing this policy and that caused
several outages before.

As of today if there is more than four connections per IP, the responses
after fourth one will be throttled and will be slower, and after seventh
connection, ORES just responds with immediate time out error.

This might affect CloudVPS and Toolforge users who connect to ORES from
there but there are several IPs coming from the cloud and looking at
request statistics, this limit is bigger than the usage. But if you are
having trouble using ORES in the cloud, let me know so I add those IPs to
the whitelist.

This limit only applies to external connections and it should not affect
internal requests coming from mediawiki or the change propagation service.

For more information, please read https://phabricator.wikimedia.org/T160692
and file a task against ORES project in phabricator in case anything is
wrong.

Happy scoring!
Best
--
Amir Sarabadani
Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de

Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/

Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Enforcing ORES limit of parallel connections per IP

Amir Sarabadani
Yes, thank you for pointing out. I made
https://phabricator.wikimedia.org/T204956

Best

On Thu, 20 Sep 2018 at 07:10, Chad <[hidden email]> wrote:

> Wouldn't it make to send a 429 instead of 408?
>
> -Chad
>
> On Sep 19, 2018 1:37 PM, "Amir Sarabadani" <[hidden email]>
> wrote:
>
> Hello,
> ORES has policy that no more than four parallel connections per IP should
> be requesting the service. We weren't enforcing this policy and that caused
> several outages before.
>
> As of today if there is more than four connections per IP, the responses
> after fourth one will be throttled and will be slower, and after seventh
> connection, ORES just responds with immediate time out error.
>
> This might affect CloudVPS and Toolforge users who connect to ORES from
> there but there are several IPs coming from the cloud and looking at
> request statistics, this limit is bigger than the usage. But if you are
> having trouble using ORES in the cloud, let me know so I add those IPs to
> the whitelist.
>
> This limit only applies to external connections and it should not affect
> internal requests coming from mediawiki or the change propagation service.
>
> For more information, please read
> https://phabricator.wikimedia.org/T160692
> and file a task against ORES project in phabricator in case anything is
> wrong.
>
> Happy scoring!
> Best
> --
> Amir Sarabadani
> Software Engineer
>
> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> Tel. (030) 219 158 26-0
> http://wikimedia.de
>
> Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> Wissens frei teilhaben kann. Helfen Sie uns dabei!
> http://spenden.wikimedia.de/
>
> Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> Körperschaften I Berlin, Steuernummer 27/029/42207.
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Amir Sarabadani
Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de

Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/

Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l