slow diffs

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

slow diffs

Travis Derouin-2
Hi,

We've had some complaints off and on about diffs taking a long time.
This happens at random times, doesn't happen often and clears itself
up after awhile. When this happens,  the load on the Apache servers
seem low and the database seems ok.

We did seem to have this problem if users had the Google toolbar
installed a few months ago, but it appears that the problem is back
and unrelated.

I wonder if the job queue has anything to do with it, sometimes it
get's up to over 10,000 on us, but usually is at 0. I haven't been
able to check what it's at when users experience this problem.

Does anyone have any ideas on how to debug, or what the possible
causes might be? We're still running 1.6.7.

Thanks,
Travis
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: slow diffs

Tim Starling
Travis Derouin wrote:

> Hi,
>
> We've had some complaints off and on about diffs taking a long time.
> This happens at random times, doesn't happen often and clears itself
> up after awhile. When this happens,  the load on the Apache servers
> seem low and the database seems ok.
>
> We did seem to have this problem if users had the Google toolbar
> installed a few months ago, but it appears that the problem is back
> and unrelated.
>
> I wonder if the job queue has anything to do with it, sometimes it
> get's up to over 10,000 on us, but usually is at 0. I haven't been
> able to check what it's at when users experience this problem.
>
> Does anyone have any ideas on how to debug, or what the possible
> causes might be? We're still running 1.6.7.

The PHP difference engine is slow, especially where lots of distinct
changes were made. Processing time can easily run into the minutes. It
could be that the complaints arise from such diffs, and that it appears
to be random because when you view the diff again, it is cached and thus
fast.

The solution to this is to install the C++ extension "wikidiff2", which
is fast for any input text.

The other possibility is that diffs are slow for reasons not directly
related to diffing. Display of diff pages requires loading the revision
text, which is uncached in 1.6.7, and also rendering the whole page to
HTML for display underneath the diff, which can take several seconds.
However, if this were the problem, you would expect other operations
such as page saves to suffer as well.

-- Tim Starling

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: slow diffs

Travis Derouin-2
Thanks Tim.

Is there any documentation for wikidiff2 other than the readme file?
I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)
and am not sure how to configure it - what settings do I change to
enable it?

Thanks,
Travis


On 10/19/06, Tim Starling <[hidden email]> wrote:

> Travis Derouin wrote:
> > Hi,
> >
> > We've had some complaints off and on about diffs taking a long time.
> > This happens at random times, doesn't happen often and clears itself
> > up after awhile. When this happens,  the load on the Apache servers
> > seem low and the database seems ok.
> >
> > We did seem to have this problem if users had the Google toolbar
> > installed a few months ago, but it appears that the problem is back
> > and unrelated.
> >
> > I wonder if the job queue has anything to do with it, sometimes it
> > get's up to over 10,000 on us, but usually is at 0. I haven't been
> > able to check what it's at when users experience this problem.
> >
> > Does anyone have any ideas on how to debug, or what the possible
> > causes might be? We're still running 1.6.7.
>
> The PHP difference engine is slow, especially where lots of distinct
> changes were made. Processing time can easily run into the minutes. It
> could be that the complaints arise from such diffs, and that it appears
> to be random because when you view the diff again, it is cached and thus
> fast.
>
> The solution to this is to install the C++ extension "wikidiff2", which
> is fast for any input text.
>
> The other possibility is that diffs are slow for reasons not directly
> related to diffing. Display of diff pages requires loading the revision
> text, which is uncached in 1.6.7, and also rendering the whole page to
> HTML for display underneath the diff, which can take several seconds.
> However, if this were the problem, you would expect other operations
> such as page saves to suffer as well.
>
> -- Tim Starling
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: slow diffs

Tim Starling-2
Travis Derouin wrote:
> Thanks Tim.
>
> Is there any documentation for wikidiff2 other than the readme file?

I doubt it.

> I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)

We seem to be using swig version 1.3.21, you might want to try something
similar.

> and am not sure how to configure it - what settings do I change to
> enable it?

"make install" will install the shared library into the PHP extensions
directory. Then in LocalSettings.php, put:

$wgExternalDiffEngine = 'wikidiff2';

That will tell MW to use wikidiff2, and if necessary, to do a dl() to load
the extension on demand. Apparently dl() is deprecated these days, so to be
completely pure and true you might want to avoid it by putting:

extension=php_wikidiff2.so

in php.ini.

-- Tim Starling

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: slow diffs

Travis Derouin-2
Upgrading swig seemed to get me by the first part. Which version of
GCC did you use? I'm getting compilation errors as well:

g++ -O3 `php-config --includes` -shared -fPIC -o php_wikidiff2.so
wikidiff2.cpp wikidiff2_wrap.cpp
DiffEngine.h: In member function `void _DiffEngine<T>::diff(const
   std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
   std::allocator<_CharT> >&, Diff<T>&) [with T = std::string]':
DiffEngine.h:560:   instantiated from `Diff<T>::Diff(const
std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
std::allocator<_CharT> >&) [with T = std::string]'
wikidiff2.cpp:15:   instantiated from here
DiffEngine.h:196: error: `assert' undeclared (first use this function)
DiffEngine.h:196: error: (Each undeclared identifier is reported only once for
   each function it appears in.)
DiffEngine.h: In member function `void _DiffEngine<T>::diff(const
   std::vector<T, std::allocator<_CharT> >&, const std::vector<T,

Thanks.



On 10/25/06, Tim Starling <[hidden email]> wrote:

> Travis Derouin wrote:
> > Thanks Tim.
> >
> > Is there any documentation for wikidiff2 other than the readme file?
>
> I doubt it.
>
> > I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)
>
> We seem to be using swig version 1.3.21, you might want to try something
> similar.
>
> > and am not sure how to configure it - what settings do I change to
> > enable it?
>
> "make install" will install the shared library into the PHP extensions
> directory. Then in LocalSettings.php, put:
>
> $wgExternalDiffEngine = 'wikidiff2';
>
> That will tell MW to use wikidiff2, and if necessary, to do a dl() to load
> the extension on demand. Apparently dl() is deprecated these days, so to be
> completely pure and true you might want to avoid it by putting:
>
> extension=php_wikidiff2.so
>
> in php.ini.
>
> -- Tim Starling
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: slow diffs

Travis Derouin-2
Hey Tim,

It seems like if

#include <assert.h>

is added to DiffEngine.h, it takes care of this problem.


Thanks,
Travis


On 10/25/06, Travis Derouin <[hidden email]> wrote:

> Upgrading swig seemed to get me by the first part. Which version of
> GCC did you use? I'm getting compilation errors as well:
>
> g++ -O3 `php-config --includes` -shared -fPIC -o php_wikidiff2.so
> wikidiff2.cpp wikidiff2_wrap.cpp
> DiffEngine.h: In member function `void _DiffEngine<T>::diff(const
>    std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
>    std::allocator<_CharT> >&, Diff<T>&) [with T = std::string]':
> DiffEngine.h:560:   instantiated from `Diff<T>::Diff(const
> std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
> std::allocator<_CharT> >&) [with T = std::string]'
> wikidiff2.cpp:15:   instantiated from here
> DiffEngine.h:196: error: `assert' undeclared (first use this function)
> DiffEngine.h:196: error: (Each undeclared identifier is reported only once for
>    each function it appears in.)
> DiffEngine.h: In member function `void _DiffEngine<T>::diff(const
>    std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
>
> Thanks.
>
>
>
> On 10/25/06, Tim Starling <[hidden email]> wrote:
> > Travis Derouin wrote:
> > > Thanks Tim.
> > >
> > > Is there any documentation for wikidiff2 other than the readme file?
> >
> > I doubt it.
> >
> > > I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)
> >
> > We seem to be using swig version 1.3.21, you might want to try something
> > similar.
> >
> > > and am not sure how to configure it - what settings do I change to
> > > enable it?
> >
> > "make install" will install the shared library into the PHP extensions
> > directory. Then in LocalSettings.php, put:
> >
> > $wgExternalDiffEngine = 'wikidiff2';
> >
> > That will tell MW to use wikidiff2, and if necessary, to do a dl() to load
> > the extension on demand. Apparently dl() is deprecated these days, so to be
> > completely pure and true you might want to avoid it by putting:
> >
> > extension=php_wikidiff2.so
> >
> > in php.ini.
> >
> > -- Tim Starling
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > http://mail.wikipedia.org/mailman/listinfo/wikitech-l
> >
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: slow diffs

Travis Derouin-2
Hey Tim,

I have some more questions about wikidiff2. I installed it, added it
to LocalSettings.php, and ran some benchmarks on diff pages, it
doesn't seem to give me any improvement, at least on our development
server. Loading 183 diff consecutive pages takes about 5 minutes using
both wikidiff2 and the default diff.

Is there any way to tell wikidiff2 is being used and is configured
properly? Or is there a better way benchmark the module? I seem to be
able to call  wikidiff2_do_diff manually from a test page.

Or is it possible that the benefits are only realized when there's
some substantial load on the server?

Any ideas would be appreciated. Thanks.

Travis


On 11/2/06, Travis Derouin <[hidden email]> wrote:

> Hey Tim,
>
> It seems like if
>
> #include <assert.h>
>
> is added to DiffEngine.h, it takes care of this problem.
>
>
> Thanks,
> Travis
>
>
> On 10/25/06, Travis Derouin <[hidden email]> wrote:
> > Upgrading swig seemed to get me by the first part. Which version of
> > GCC did you use? I'm getting compilation errors as well:
> >
> > g++ -O3 `php-config --includes` -shared -fPIC -o php_wikidiff2.so
> > wikidiff2.cpp wikidiff2_wrap.cpp
> > DiffEngine.h: In member function `void _DiffEngine<T>::diff(const
> >    std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
> >    std::allocator<_CharT> >&, Diff<T>&) [with T = std::string]':
> > DiffEngine.h:560:   instantiated from `Diff<T>::Diff(const
> > std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
> > std::allocator<_CharT> >&) [with T = std::string]'
> > wikidiff2.cpp:15:   instantiated from here
> > DiffEngine.h:196: error: `assert' undeclared (first use this function)
> > DiffEngine.h:196: error: (Each undeclared identifier is reported only once for
> >    each function it appears in.)
> > DiffEngine.h: In member function `void _DiffEngine<T>::diff(const
> >    std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
> >
> > Thanks.
> >
> >
> >
> > On 10/25/06, Tim Starling <[hidden email]> wrote:
> > > Travis Derouin wrote:
> > > > Thanks Tim.
> > > >
> > > > Is there any documentation for wikidiff2 other than the readme file?
> > >
> > > I doubt it.
> > >
> > > > I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)
> > >
> > > We seem to be using swig version 1.3.21, you might want to try something
> > > similar.
> > >
> > > > and am not sure how to configure it - what settings do I change to
> > > > enable it?
> > >
> > > "make install" will install the shared library into the PHP extensions
> > > directory. Then in LocalSettings.php, put:
> > >
> > > $wgExternalDiffEngine = 'wikidiff2';
> > >
> > > That will tell MW to use wikidiff2, and if necessary, to do a dl() to load
> > > the extension on demand. Apparently dl() is deprecated these days, so to be
> > > completely pure and true you might want to avoid it by putting:
> > >
> > > extension=php_wikidiff2.so
> > >
> > > in php.ini.
> > >
> > > -- Tim Starling
> > >
> > > _______________________________________________
> > > Wikitech-l mailing list
> > > [hidden email]
> > > http://mail.wikipedia.org/mailman/listinfo/wikitech-l
> > >
> >
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l