Quantcast

A few questions

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

A few questions

richard Cavell
Hi all and TIA.

I usually run my bot with maxlag = 5 seconds.

When it attempts a
communication with Wikipedia, if it receives Error:maxlag or HTTP 503 ,
it retries up to 5 times with 10 seconds between each try.  If it still
can't get through, it aborts.  At the time I'm writing this, about 3pm
Friday afternoon in Australia, my bot is routinely aborting due to 5
maxlag or 503s in a row.

I am usually logged into #wikipedia and #wikipedia-en on Freenode, and
have noticed that many times I and other Australians will complain that
Wikipedia is very slow and timing out for human edits, when Americans
claim that they have no such problem at that time.

Currently I'm running the bot with maxlag = 10 seconds, and it's having
no problems now.

Is the maxlag parameter something that will change if I'm connecting
 from Australia?  May I be allowed some leniency for being on the other
side of the world, or does it not work like that?  If maxlag > 5
seconds, does that mean I should stop wasting the server's time and run
my bot some other time?

Richard



_______________________________________________
Wikibots-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikibots-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: A few questions

Scot Wilcoxon
I don't know why Australia might be having trouble, unless the
Christchurch earthquake irritated a cable.

The way I have my bots avoid irritating the servers is to back off very
quickly, and slowly speed back up.  Starting with a minimum of 5 seconds
between updates...  If there is no reply after 30 seconds, the bot
sleeps for 60 seconds before retrying.  Another failure increases the
sleep time to 120 seconds.  A successful update decreases the sleep time
by 10 seconds, until the minimum 5 seconds is reached.  So a slow period
can slow the bot to several minutes between attempts, and it then takes
many minutes to return to normal.

On Thu, 2011-03-03 at 23:44 -0500, [hidden email] wrote:

> Hi all and TIA.
>
> I usually run my bot with maxlag = 5 seconds.
>
> When it attempts a
> communication with Wikipedia, if it receives Error:maxlag or HTTP 503 ,
> it retries up to 5 times with 10 seconds between each try.  If it still
> can't get through, it aborts.  At the time I'm writing this, about 3pm
> Friday afternoon in Australia, my bot is routinely aborting due to 5
> maxlag or 503s in a row.
>
> I am usually logged into #wikipedia and #wikipedia-en on Freenode, and
> have noticed that many times I and other Australians will complain that
> Wikipedia is very slow and timing out for human edits, when Americans
> claim that they have no such problem at that time.
>
> Currently I'm running the bot with maxlag = 10 seconds, and it's having
> no problems now.
>
> Is the maxlag parameter something that will change if I'm connecting
>  from Australia?  May I be allowed some leniency for being on the other
> side of the world, or does it not work like that?  If maxlag > 5
> seconds, does that mean I should stop wasting the server's time and run
> my bot some other time?
>
> Richard
>
>
>
> _______________________________________________
> Wikibots-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikibots-l



_______________________________________________
Wikibots-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikibots-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: A few questions

Russell Blau
"Scot Wilcoxon" <[hidden email]> wrote:

>
> The way I have my bots avoid irritating the servers is to back off very
> quickly, and slowly speed back up.  Starting with a minimum of 5 seconds
> between updates...  If there is no reply after 30 seconds, the bot
> sleeps for 60 seconds before retrying.  Another failure increases the
> sleep time to 120 seconds.  A successful update decreases the sleep time
> by 10 seconds, until the minimum 5 seconds is reached.  So a slow period
> can slow the bot to several minutes between attempts, and it then takes
> many minutes to return to normal.
>
> On Thu, 2011-03-03 at 23:44 -0500, [hidden email] wrote:
>> Hi all and TIA.
>>
>> I usually run my bot with maxlag = 5 seconds.
>>
>> When it attempts a
>> communication with Wikipedia, if it receives Error:maxlag or HTTP 503 ,
>> it retries up to 5 times with 10 seconds between each try.  If it still
>> can't get through, it aborts.  At the time I'm writing this, about 3pm
>> Friday afternoon in Australia, my bot is routinely aborting due to 5
>> maxlag or 503s in a row.
>>

The algorithm I've used in working on pywikipediabot is, upon receiving a
maxlag error, to have the bot sleep for min(5, lag / 2) seconds, where 'lag'
is the lag reported in the error message.  If there is a long replication
lag, then it is going to take longer for the database slaves to catch up
than with a short lag, so there is no point in rechecking every 5 seconds.
Aborting after 5 maxlag errors is purely a design choice; if you don't mind
having your bot wait minutes or hours for the lag to subside, you can do
that too.

Russ


_______________________________________________
Wikibots-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikibots-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: A few questions

Platonides
In reply to this post by richard Cavell
[hidden email] wrote:

> Hi all and TIA.
>
> I usually run my bot with maxlag = 5 seconds.
>
> When it attempts a
> communication with Wikipedia, if it receives Error:maxlag or HTTP 503 ,
> it retries up to 5 times with 10 seconds between each try.  If it still
> can't get through, it aborts.  At the time I'm writing this, about 3pm
> Friday afternoon in Australia, my bot is routinely aborting due to 5
> maxlag or 503s in a row.
>
> I am usually logged into #wikipedia and #wikipedia-en on Freenode, and
> have noticed that many times I and other Australians will complain that
> Wikipedia is very slow and timing out for human edits, when Americans
> claim that they have no such problem at that time.
>
> Currently I'm running the bot with maxlag = 10 seconds, and it's having
> no problems now.
>
> Is the maxlag parameter something that will change if I'm connecting
>  from Australia?  May I be allowed some leniency for being on the other
> side of the world, or does it not work like that?  If maxlag > 5
> seconds, does that mean I should stop wasting the server's time and run
> my bot some other time?
>
> Richard

The lag only affects the server themselves. It just means that at that
time the lag was between 5 and 10 seconds for several minutes.

You probably wan to take a look at
http://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb=
to see what's going on.


_______________________________________________
Wikibots-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikibots-l
Loading...