What type of language is the wikitext ?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

What type of language is the wikitext ?

Kaartic Sivaraam
Dear all,

Quoting from my previous post,

    “Currently the syntax highlighter of the Wikipedia android app seems
    to be slow except on high-end devices. It has been proposed to
    change the implementation to provide users with a better
    (streamlined) experience while editing[1].”


I recently came to know from a reply to that post [2], that wikitext is
not a "regular language"[3]. I wanted to know what kind of language
wikitext is to ensure that the algorithm for syntax highlighting does
the right work. Is wikitext a "Context Free Language" or is it something
else?


Links
  -----
[1]: https://phabricator.wikimedia.org/T164936
[2]: https://lists.wikimedia.org/pipermail/mediawiki-l/2017-June/046627.html
[3]: https://en.wikipedia.org/wiki/Regular_language

---
Regards,
Kaartic

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What type of language is the wikitext ?

יגאל חיטרון
Hi. Any computer language that has parentheses isn't context free. So if
you are not in Assembly, it's enough.
About context-free - it's tricky question. I wouldn't call it context free
because of some macro expansions, for example, the pipe ([[abc (def)|]] ->
[[abc (def)|abc]]). The problem I don't know if it will really disturb you,
because highlighting should not care about this.
The first problem of context-free, templates, should not bother you either,
because you do not have template expansion on the page is highlighted, only
in "runtime". Even "subst" mechanism does not work before saving.
I tried now to find some wikitext syntax constructs that created a "really"
context sensitive problems, as text power (find ww for some w), but had not
found anything.
Any other oppinions?
Igal (User:IKhitron)


2017-07-05 16:10 GMT+03:00 Kaartic Sivaraam <[hidden email]>
:

> Dear all,
>
> Quoting from my previous post,
>
>    “Currently the syntax highlighter of the Wikipedia android app seems
>    to be slow except on high-end devices. It has been proposed to
>    change the implementation to provide users with a better
>    (streamlined) experience while editing[1].”
>
>
> I recently came to know from a reply to that post [2], that wikitext is
> not a "regular language"[3]. I wanted to know what kind of language
> wikitext is to ensure that the algorithm for syntax highlighting does the
> right work. Is wikitext a "Context Free Language" or is it something else?
>
>
> Links
>  -----
> [1]: https://phabricator.wikimedia.org/T164936
> [2]: https://lists.wikimedia.org/pipermail/mediawiki-l/2017-June/
> 046627.html
> [3]: https://en.wikipedia.org/wiki/Regular_language
>
> ---
> Regards,
> Kaartic
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What type of language is the wikitext ?

Gabriel Wicke-3
Here is an older blog post describing some of the issues in parsing
wikitext, including examples:
https://blog.wikimedia.org/2013/03/04/parsoid-how-wikipedia-catches-up-with-the-web/

On Wed, Jul 5, 2017 at 6:31 AM, יגאל חיטרון <[hidden email]> wrote:

> Hi. Any computer language that has parentheses isn't context free. So if
> you are not in Assembly, it's enough.
> About context-free - it's tricky question. I wouldn't call it context free
> because of some macro expansions, for example, the pipe ([[abc (def)|]] ->
> [[abc (def)|abc]]). The problem I don't know if it will really disturb you,
> because highlighting should not care about this.
> The first problem of context-free, templates, should not bother you either,
> because you do not have template expansion on the page is highlighted, only
> in "runtime". Even "subst" mechanism does not work before saving.
> I tried now to find some wikitext syntax constructs that created a "really"
> context sensitive problems, as text power (find ww for some w), but had not
> found anything.
> Any other oppinions?
> Igal (User:IKhitron)
>
>
> 2017-07-05 16:10 GMT+03:00 Kaartic Sivaraam <kaarticsivaraam91196@gmail.
> com>
> :
>
> > Dear all,
> >
> > Quoting from my previous post,
> >
> >    “Currently the syntax highlighter of the Wikipedia android app seems
> >    to be slow except on high-end devices. It has been proposed to
> >    change the implementation to provide users with a better
> >    (streamlined) experience while editing[1].”
> >
> >
> > I recently came to know from a reply to that post [2], that wikitext is
> > not a "regular language"[3]. I wanted to know what kind of language
> > wikitext is to ensure that the algorithm for syntax highlighting does the
> > right work. Is wikitext a "Context Free Language" or is it something
> else?
> >
> >
> > Links
> >  -----
> > [1]: https://phabricator.wikimedia.org/T164936
> > [2]: https://lists.wikimedia.org/pipermail/mediawiki-l/2017-June/
> > 046627.html
> > [3]: https://en.wikipedia.org/wiki/Regular_language
> >
> > ---
> > Regards,
> > Kaartic
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
Gabriel Wicke
Principal Engineer, Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What type of language is the wikitext ?

יגאל חיטרון
In reply to this post by יגאל חיטרון
Sorry in the first line, of course, "Any computer language that has
parentheses isn't regular".
Igal (User:IKhitron)


2017-07-05 16:31 GMT+03:00 יגאל חיטרון <[hidden email]>:

> Hi. Any computer language that has parentheses isn't context free. So if
> you are not in Assembly, it's enough.
> About context-free - it's tricky question. I wouldn't call it context free
> because of some macro expansions, for example, the pipe ([[abc (def)|]] ->
> [[abc (def)|abc]]). The problem I don't know if it will really disturb you,
> because highlighting should not care about this.
> The first problem of context-free, templates, should not bother you
> either, because you do not have template expansion on the page is
> highlighted, only in "runtime". Even "subst" mechanism does not work before
> saving.
> I tried now to find some wikitext syntax constructs that created a
> "really" context sensitive problems, as text power (find ww for some w),
> but had not found anything.
> Any other oppinions?
> Igal (User:IKhitron)
>
>
> 2017-07-05 16:10 GMT+03:00 Kaartic Sivaraam <kaarticsivaraam91196@gmail.
> com>:
>
>> Dear all,
>>
>> Quoting from my previous post,
>>
>>    “Currently the syntax highlighter of the Wikipedia android app seems
>>    to be slow except on high-end devices. It has been proposed to
>>    change the implementation to provide users with a better
>>    (streamlined) experience while editing[1].”
>>
>>
>> I recently came to know from a reply to that post [2], that wikitext is
>> not a "regular language"[3]. I wanted to know what kind of language
>> wikitext is to ensure that the algorithm for syntax highlighting does the
>> right work. Is wikitext a "Context Free Language" or is it something else?
>>
>>
>> Links
>>  -----
>> [1]: https://phabricator.wikimedia.org/T164936
>> [2]: https://lists.wikimedia.org/pipermail/mediawiki-l/2017-June/
>> 046627.html
>> [3]: https://en.wikipedia.org/wiki/Regular_language
>>
>> ---
>> Regards,
>> Kaartic
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What type of language is the wikitext ?

Kaartic Sivaraam
In reply to this post by Kaartic Sivaraam
On Wed, 5 Jul 2017 07:48:18 -0700, Gabriel Wicke <[hidden email]> wrote,
> Here is an older blog post describing some of the issues in parsing
> wikitext, including examples:
> https://blog.wikimedia.org/2013/03/04/parsoid-how-wikipedia-catches-up-with-the-web/
>
Thanks! Will look into it.

--
Kaartic

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What type of language is the wikitext ?

Kaartic Sivaraam
In reply to this post by Kaartic Sivaraam
On Wed, 5 Jul 2017 16:31:44 +0300, יגאל חיטרון <[hidden email]> wrote,

> Hi. Any computer language that has parentheses isn't context free. So
> if
> you are not in Assembly, it's enough.
> About context-free - it's tricky question. I wouldn't call it context
> free
> because of some macro expansions, for example, the pipe ([[abc
> (def)|]] ->
> [[abc (def)|abc]]). The problem I don't know if it will really
> disturb you,
> because highlighting should not care about this.
> The first problem of context-free, templates, should not bother you
> either,
> because you do not have template expansion on the page is
> highlighted, only
> in "runtime". Even "subst" mechanism does not work before saving.
> I tried now to find some wikitext syntax constructs that created a
> "really"
> context sensitive problems, as text power (find ww for some w), but
> had not
> found anything.
> Any other oppinions?
>
Thanks for the reply. So from what I could get does it mean that as far
as syntax-highlighting is concerned 'wikitext' could be considered to
be something close to a context-free language. As a result of, syntax
highlighting could be done with a program that works like a PDA.
Correct me if I'm wrong.

--
Kaartic

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What type of language is the wikitext ?

יגאל חיטרון
Yes, but again, it's my opinion only.
Igal


2017-07-10 17:30 GMT+03:00 Kaartic Sivaraam <[hidden email]>
:

> On Wed, 5 Jul 2017 16:31:44 +0300, יגאל חיטרון <[hidden email]>
> wrote,
> > Hi. Any computer language that has parentheses isn't context free. So
> > if
> > you are not in Assembly, it's enough.
> > About context-free - it's tricky question. I wouldn't call it context
> > free
> > because of some macro expansions, for example, the pipe ([[abc
> > (def)|]] ->
> > [[abc (def)|abc]]). The problem I don't know if it will really
> > disturb you,
> > because highlighting should not care about this.
> > The first problem of context-free, templates, should not bother you
> > either,
> > because you do not have template expansion on the page is
> > highlighted, only
> > in "runtime". Even "subst" mechanism does not work before saving.
> > I tried now to find some wikitext syntax constructs that created a
> > "really"
> > context sensitive problems, as text power (find ww for some w), but
> > had not
> > found anything.
> > Any other oppinions?
> >
> Thanks for the reply. So from what I could get does it mean that as far
> as syntax-highlighting is concerned 'wikitext' could be considered to
> be something close to a context-free language. As a result of, syntax
> highlighting could be done with a program that works like a PDA.
> Correct me if I'm wrong.
>
> --
> Kaartic
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What type of language is the wikitext ?

Kaartic Sivaraam
On Mon, 2017-07-10 at 17:30 +0300, יגאל חיטרון wrote:
> Yes, but again, it's my opinion only.
> Igal
>
I get that :) It seems your opinion is not wrong from seeing the
current syntax highlighter implementation[1]. It does seems to imitate
a PDA when details are ignored.

[1]: https://github.com/wikimedia/apps-android-wikipedia/blob/master/app/src/main/java/org/wikipedia/edit/richtext/SyntaxHighlighter.java#L148

--
Kaartic

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Loading...