Dump process does not work

classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Dump process does not work

Andreas Meier-2
Since today the dump process does not work correctly. It is running, but
without any success

Best regards

Andim


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Brion Vibber-3
El 5/1/09 5:51 PM, Andreas Meier escribió:
> Since today the dump process does not work correctly. It is running, but
> without any success

Tomasz is on it... we've upgraded the machine they run on and it needs
some more tweaking. :)

-- brion

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Tomasz Finc-2
Brion Vibber wrote:

> El 5/1/09 5:51 PM, Andreas Meier escribió:
>> Since today the dump process does not work correctly. It is running, but
>> without any success
>
> Tomasz is on it... we've upgraded the machine they run on and it needs
> some more tweaking. :)
>
> -- brion
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Indeed. The backup job was missing the php normalize library. Putting
that into place now. Then I'll see if there is any db weirdness.

--tomasz

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Russell Blau-2
"Tomasz Finc" <[hidden email]> wrote in message
news:[hidden email]...
Brion Vibber wrote:
>> El 5/1/09 5:51 PM, Andreas Meier escribió:
>>> Since today the dump process does not work correctly. It is running, but
>>> without any success
>>
>> Tomasz is on it... we've upgraded the machine they run on and it needs
>> some more tweaking. :)
>>
> Indeed. The backup job was missing the php normalize library. Putting
> that into place now. Then I'll see if there is any db weirdness.

But, on the bright side, every database in the system now has a dump that
was completed within the last nine hours (roughly).  When's the last time
you could say *that*?  :-)






_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Tomasz Finc-2
Russell Blau wrote:

> "Tomasz Finc" <[hidden email]> wrote in message
> news:[hidden email]...
> Brion Vibber wrote:
>>> El 5/1/09 5:51 PM, Andreas Meier escribió:
>>>> Since today the dump process does not work correctly. It is running, but
>>>> without any success
>>> Tomasz is on it... we've upgraded the machine they run on and it needs
>>> some more tweaking. :)
>>>
>> Indeed. The backup job was missing the php normalize library. Putting
>> that into place now. Then I'll see if there is any db weirdness.
>
> But, on the bright side, every database in the system now has a dump that
> was completed within the last nine hours (roughly).  When's the last time
> you could say *that*?  :-)

Mwhaha .. that would be awesome if it was actually useful data. The
libs, binaries and configs have all been fixed. I've run a couple of
batch jobs for the small wikis [tokiponawiktionary, emlwiki] and am
running [afwiki] right now to try a bigger data set. No issues so far
past the main page not noticing them finishing.

After afwiki finishes up I'll remove the failed runs as they don't
provide us with any useful data. Will set the worker to begin processing
after that. Plus I'll actually document the setup.

--tomasz

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Roan Kattouw-2
In reply to this post by Russell Blau-2
2009/5/1 Russell Blau <[hidden email]>:
> But, on the bright side, every database in the system now has a dump that
> was completed within the last nine hours (roughly).  When's the last time
> you could say *that*?  :-)
>
Yes, and those dumps are 20-byte .gz files. Oops.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Chad
In reply to this post by Tomasz Finc-2
Document the dump process? You must be joking :)

-Chad

On May 1, 2009 3:24 PM, "Tomasz Finc" <[hidden email]> wrote:

Russell Blau wrote: > "Tomasz Finc" <[hidden email]> wrote in message >
news:49FB3CA6.90602@wi...
Mwhaha .. that would be awesome if it was actually useful data. The
libs, binaries and configs have all been fixed. I've run a couple of
batch jobs for the small wikis [tokiponawiktionary, emlwiki] and am
running [afwiki] right now to try a bigger data set. No issues so far
past the main page not noticing them finishing.

After afwiki finishes up I'll remove the failed runs as they don't
provide us with any useful data. Will set the worker to begin processing
after that. Plus I'll actually document the setup.

--tomasz _______________________________________________ Wikitech-l mailing
list Wikitech-l@lists....
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Platonides
In reply to this post by Roan Kattouw-2
Roan Kattouw wrote:
> 2009/5/1 Russell Blau <[hidden email]>:
>> But, on the bright side, every database in the system now has a dump that
>> was completed within the last nine hours (roughly).  When's the last time
>> you could say *that*?  :-)
>
> Yes, and those dumps are 20-byte .gz files. Oops.
>
> Roan Kattouw (Catrope)

Completed and with an outstanding compression rate!
;)


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Tomasz Finc-2
In reply to this post by Tomasz Finc-2
Tomasz Finc wrote:

> Russell Blau wrote:
>> "Tomasz Finc" <[hidden email]> wrote in message
>> news:[hidden email]...
>> Brion Vibber wrote:
>>>> El 5/1/09 5:51 PM, Andreas Meier escribió:
>>>>> Since today the dump process does not work correctly. It is running, but
>>>>> without any success
>>>> Tomasz is on it... we've upgraded the machine they run on and it needs
>>>> some more tweaking. :)
>>>>
>>> Indeed. The backup job was missing the php normalize library. Putting
>>> that into place now. Then I'll see if there is any db weirdness.
>> But, on the bright side, every database in the system now has a dump that
>> was completed within the last nine hours (roughly).  When's the last time
>> you could say *that*?  :-)
>
> Mwhaha .. that would be awesome if it was actually useful data. The
> libs, binaries and configs have all been fixed. I've run a couple of
> batch jobs for the small wikis [tokiponawiktionary, emlwiki] and am
> running [afwiki] right now to try a bigger data set. No issues so far
> past the main page not noticing them finishing.
>
> After afwiki finishes up I'll remove the failed runs as they don't
> provide us with any useful data. Will set the worker to begin processing
> after that. Plus I'll actually document the setup.

afwiki finished just fine and all subsequent wiki's have been happy. The
only issue left is that the version of 7za on Ubuntu 8.04 ignores the
system umask and decides that 600 is good enough for everyone. This is
fixed in 4.58 and I've requested a backport from the ubuntu folks at

https://bugs.edge.launchpad.net/hardy-backports/+bug/370618

In the mean time I've forced a chmod of 644 into the dumps script.

--tomasz

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Andreas Meier-2
Tomasz Finc schrieb:

>Tomasz Finc wrote:
>> After afwiki finishes up I'll remove the failed runs as they don't
>> provide us with any useful data. Will set the worker to begin processing
>> after that. Plus I'll actually document the setup.
>
> afwiki finished just fine and all subsequent wiki's have been happy. The
> only issue left is that the version of 7za on Ubuntu 8.04 ignores the
> system umask and decides that 600 is good enough for everyone. This is
> fixed in 4.58 and I've requested a backport from the ubuntu folks at
>
> https://bugs.edge.launchpad.net/hardy-backports/+bug/370618
>
> In the mean time I've forced a chmod of 644 into the dumps script.
>
> --tomasz

Last year and in the begining of this year there were 5 dump processes
at the same time. Now there are only two. With 5 running processes it ws
possible to have a dump of each project once a month, but with 2 this is
not possible. The system seems to be stable now, so can you increase the
number of running jobs to 5?

Best regards

Andim


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Tomasz Finc-2
Andreas Meier wrote:

> Tomasz Finc schrieb:
>> Tomasz Finc wrote:
>>> After afwiki finishes up I'll remove the failed runs as they don't
>>> provide us with any useful data. Will set the worker to begin processing
>>> after that. Plus I'll actually document the setup.
>> afwiki finished just fine and all subsequent wiki's have been happy. The
>> only issue left is that the version of 7za on Ubuntu 8.04 ignores the
>> system umask and decides that 600 is good enough for everyone. This is
>> fixed in 4.58 and I've requested a backport from the ubuntu folks at
>>
>> https://bugs.edge.launchpad.net/hardy-backports/+bug/370618
>>
>> In the mean time I've forced a chmod of 644 into the dumps script.
>>
>> --tomasz
>
> Last year and in the begining of this year there were 5 dump processes
> at the same time. Now there are only two. With 5 running processes it ws
> possible to have a dump of each project once a month, but with 2 this is
> not possible. The system seems to be stable now, so can you increase the
> number of running jobs to 5?

This has now been upped to 12 jobs being run concurrently in order to
catch up. No outstanding issues that have surfaced yet.

-tomasz



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

David Gerard-2
In reply to this post by Russell Blau-2
2009/5/1 Russell Blau <[hidden email]>:

> But, on the bright side, every database in the system now has a dump that
> was completed within the last nine hours (roughly).  When's the last time
> you could say *that*?  :-)


A good en:wp dump is the sort of thing warranting announcement on the
tech blog. Heck, the *main* blog.


- d.

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Anthony-73
On Mon, May 4, 2009 at 8:52 PM, David Gerard <[hidden email]> wrote:

> 2009/5/1 Russell Blau <[hidden email]>:
>
> > But, on the bright side, every database in the system now has a dump that
> > was completed within the last nine hours (roughly).  When's the last time
> > you could say *that*?  :-)
>
>
> A good en:wp dump is the sort of thing warranting announcement on the
> tech blog. Heck, the *main* blog.


Why don't you make up the press release, David?
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Chad
Might wait for them to actually complete properly.

-Chad

On May 4, 2009 9:05 PM, "Anthony" <[hidden email]> wrote:

On Mon, May 4, 2009 at 8:52 PM, David Gerard <[hidden email]> wrote: >
2009/5/1 Russell Blau <ru...
Why don't you make up the press release, David?

_______________________________________________ Wikitech-l mailing list
[hidden email]...
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump process does not work

Anthony-73
That wouldn't be nearly as fun to watch.

On Mon, May 4, 2009 at 9:56 PM, Chad <[hidden email]> wrote:

> Might wait for them to actually complete properly.
>
> -Chad
>
> On May 4, 2009 9:05 PM, "Anthony" <[hidden email]> wrote:
>
> On Mon, May 4, 2009 at 8:52 PM, David Gerard <[hidden email]> wrote: >
> 2009/5/1 Russell Blau <ru...
> Why don't you make up the press release, David?
>
> _______________________________________________ Wikitech-l mailing list
> [hidden email]...
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Dump throughput

erikzachte
In reply to this post by Tomasz Finc-2
Tomasz, the amount of dump power that you managed to activate is impressive.
136 dumps yesterday, today already 110 :-) Out of 760 total.
Of course there are small en large dumps, but this is very encouraging.

Erik Zachte



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump throughput

Russell Blau-2
"Erik Zachte" <[hidden email]> wrote in message
news:002d01c9cd8d$3355beb0$9a013c10$@com...
> Tomasz, the amount of dump power that you managed to activate is
> impressive.
> 136 dumps yesterday, today already 110 :-) Out of 760 total.
> Of course there are small en large dumps, but this is very encouraging.
>

Yes, thank you Tomasz for your attention to this.  The commonswiki process
looks like it *might* be dead, by the way.

Russ




_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump throughput

Gerard Meijssen-3
In reply to this post by erikzachte
Hoi,
This is the kind of news that will make many people happy. Obviously what
every one is waiting for is the en.wp to finish .. :) But it is great to
have many moments to be happy.
thanks,
       GerardM

2009/5/5 Erik Zachte <[hidden email]>

> Tomasz, the amount of dump power that you managed to activate is
> impressive.
> 136 dumps yesterday, today already 110 :-) Out of 760 total.
> Of course there are small en large dumps, but this is very encouraging.
>
> Erik Zachte
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump throughput

Bilal Abdul Kader
Hi Tomasz,
Any ideas about a fresher dump of enwiki-meta-pages-history?

bilal


>
> 2009/5/5 Erik Zachte <[hidden email]>
>
> > Tomasz, the amount of dump power that you managed to activate is
> > impressive.
> > 136 dumps yesterday, today already 110 :-) Out of 760 total.
> > Of course there are small en large dumps, but this is very encouraging.
> >
> > Erik Zachte
> >
> >
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Dump throughput

Tomasz Finc-2
In reply to this post by Russell Blau-2
Russell Blau wrote:

> "Erik Zachte" <[hidden email]> wrote in message
> news:002d01c9cd8d$3355beb0$9a013c10$@com...
>> Tomasz, the amount of dump power that you managed to activate is
>> impressive.
>> 136 dumps yesterday, today already 110 :-) Out of 760 total.
>> Of course there are small en large dumps, but this is very encouraging.
>>
>
> Yes, thank you Tomasz for your attention to this.  The commonswiki process
> looks like it *might* be dead, by the way.

Don't think so as I actively see it being updated. It's currently set to
to finish it's second to last step on 2009-05-06 02:53:21.

No one touch anything while its still going ;)


--tomasz


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
12