<translate> tags are a usability nightmare for editing on mediawiki.org

[Brion Vibber-4]

Lots of pages on mediawiki.org are pretty much uneditable because they're
strewn with <translate> spanning multiple paragraphs that make VisualEditor
completely unusable and the source editor very difficult to use.

I'm trying to update documentation, and I'm seriously thinking of regexing
out all the <translate> stuff so I can actually edit the wiki pages.

This is probably not a good thing.


https://phabricator.wikimedia.org/T131516

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[C. Scott Ananian]

Proper VE/Parsoid support for the <translate> extension has been wanted for
a while now.  It hasn't made a list of quarterly goals yet.  Help wanted,
certainly.
  --scott
On Apr 1, 2016 12:32 PM, "Brion Vibber" <[hidden email]> wrote:

> Lots of pages on mediawiki.org are pretty much uneditable because they're
> strewn with <translate> spanning multiple paragraphs that make VisualEditor
> completely unusable and the source editor very difficult to use.
>
> I'm trying to update documentation, and I'm seriously thinking of regexing
> out all the <translate> stuff so I can actually edit the wiki pages.
>
> This is probably not a good thing.
>
>
> https://phabricator.wikimedia.org/T131516
>
> -- brion
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-4]

On Fri, Apr 1, 2016 at 7:35 PM, C. Scott Ananian <[hidden email]>
wrote:

> Proper VE/Parsoid support for the <translate> extension has been wanted for
> a while now.  It hasn't made a list of quarterly goals yet.  Help wanted,
> certainly.
>

This is a HUGELY important dogfooding goal; we need to be able to edit the
wiki about our own product!

Anything we can do to help out?

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-4]

So based on notes from Niklas on the bug, it looks like there are two main
problems:

1) VE treats <translate>...</translate> chunks as uneditable extension
blobs.

2) Some/many pages use <translate>...</translate> incorrectly: spanning
paragraphs, markup level boundaries, etc.


So problem 1) can be improved in two ways:
1a) Have VE treat <translate>...</translate> as more or less like plaintext
or a div or something
1b) Much fancier sub-document editing

Fixing problem 2) means refactoring broken pages in ways that presumably
would require at least some of their translations to be re-created.

I don't know whether 2) is caused by problems in the translation interface,
or by people further editing the pages without a detailed understanding of
how the <translate> and <!--T:123--> markup bits work, or a combination of
both.

-- brion



On Fri, Apr 1, 2016 at 7:39 PM, Brion Vibber <[hidden email]> wrote:

> On Fri, Apr 1, 2016 at 7:35 PM, C. Scott Ananian <[hidden email]>
> wrote:
>
>> Proper VE/Parsoid support for the <translate> extension has been wanted
>> for
>> a while now.  It hasn't made a list of quarterly goals yet.  Help wanted,
>> certainly.
>>
>
> This is a HUGELY important dogfooding goal; we need to be able to edit the
> wiki about our own product!
>
> Anything we can do to help out?
>
> -- brion
>
>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[rupert THURNER-2]

On Apr 1, 2016 18:32, "Brion Vibber" <[hidden email]> wrote:
>
> Lots of pages on mediawiki.org are pretty much uneditable because they're
> strewn with <translate> spanning multiple paragraphs that make
VisualEditor
> completely unusable and the source editor very difficult to use.
>
> I'm trying to update documentation, and I'm seriously thinking of regexing
> out all the <translate> stuff so I can actually edit the wiki pages.
>
> This is probably not a good thing.
>
>
> https://phabricator.wikimedia.org/T131516

This is not only on mediawiki.org but also on meta. I d not unhappy if this
replaced by nothing.

Rupert
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Gregory Varnum-2]

I think improving VE to understand translate, and expanding the training/documentation for translate are the best next steps. This is also a problem on Meta-Wiki.

Removing the translate setup from the pages seems problematic as we're trying to make info available in languages beyond just English.

-greg

_______________
Sent from my iPhone - a more detailed response may be sent later.

> On Apr 1, 2016, at 9:49 AM, Brion Vibber <[hidden email]> wrote:
>
> So based on notes from Niklas on the bug, it looks like there are two main
> problems:
>
> 1) VE treats <translate>...</translate> chunks as uneditable extension
> blobs.
>
> 2) Some/many pages use <translate>...</translate> incorrectly: spanning
> paragraphs, markup level boundaries, etc.
>
>
> So problem 1) can be improved in two ways:
> 1a) Have VE treat <translate>...</translate> as more or less like plaintext
> or a div or something
> 1b) Much fancier sub-document editing
>
> Fixing problem 2) means refactoring broken pages in ways that presumably
> would require at least some of their translations to be re-created.
>
> I don't know whether 2) is caused by problems in the translation interface,
> or by people further editing the pages without a detailed understanding of
> how the <translate> and <!--T:123--> markup bits work, or a combination of
> both.
>
> -- brion
>
>
>
>> On Fri, Apr 1, 2016 at 7:39 PM, Brion Vibber <[hidden email]> wrote:
>>
>> On Fri, Apr 1, 2016 at 7:35 PM, C. Scott Ananian <[hidden email]>
>> wrote:
>>
>>> Proper VE/Parsoid support for the <translate> extension has been wanted
>>> for
>>> a while now.  It hasn't made a list of quarterly goals yet.  Help wanted,
>>> certainly.
>>
>> This is a HUGELY important dogfooding goal; we need to be able to edit the
>> wiki about our own product!
>>
>> Anything we can do to help out?
>>
>> -- brion
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[C. Scott Ananian]

Arlo actually implemented the Parsoid side of this:
https://phabricator.wikimedia.org/T50891
We still need VE support: https://phabricator.wikimedia.org/T55974
 --scott

On Fri, Apr 1, 2016 at 1:00 PM, Gregory Varnum <[hidden email]>
wrote:

> I think improving VE to understand translate, and expanding the
> training/documentation for translate are the best next steps. This is also
> a problem on Meta-Wiki.
>
> Removing the translate setup from the pages seems problematic as we're
> trying to make info available in languages beyond just English.
>
> -greg
>
> _______________
> Sent from my iPhone - a more detailed response may be sent later.
>
> > On Apr 1, 2016, at 9:49 AM, Brion Vibber <[hidden email]> wrote:
> >
> > So based on notes from Niklas on the bug, it looks like there are two
> main
> > problems:
> >
> > 1) VE treats <translate>...</translate> chunks as uneditable extension
> > blobs.
> >
> > 2) Some/many pages use <translate>...</translate> incorrectly: spanning
> > paragraphs, markup level boundaries, etc.
> >
> >
> > So problem 1) can be improved in two ways:
> > 1a) Have VE treat <translate>...</translate> as more or less like
> plaintext
> > or a div or something
> > 1b) Much fancier sub-document editing
> >
> > Fixing problem 2) means refactoring broken pages in ways that presumably
> > would require at least some of their translations to be re-created.
> >
> > I don't know whether 2) is caused by problems in the translation
> interface,
> > or by people further editing the pages without a detailed understanding
> of
> > how the <translate> and <!--T:123--> markup bits work, or a combination
> of
> > both.
> >
> > -- brion
> >
> >
> >
> >> On Fri, Apr 1, 2016 at 7:39 PM, Brion Vibber <[hidden email]>
> wrote:
> >>
> >> On Fri, Apr 1, 2016 at 7:35 PM, C. Scott Ananian <
> [hidden email]>
> >> wrote:
> >>
> >>> Proper VE/Parsoid support for the <translate> extension has been wanted
> >>> for
> >>> a while now.  It hasn't made a list of quarterly goals yet.  Help
> wanted,
> >>> certainly.
> >>
> >> This is a HUGELY important dogfooding goal; we need to be able to edit
> the
> >> wiki about our own product!
> >>
> >> Anything we can do to help out?
> >>
> >> -- brion
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

--
(http://cscott.net)
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-4]

On Friday, April 1, 2016, C. Scott Ananian <[hidden email]> wrote:

> Arlo actually implemented the Parsoid side of this:
> https://phabricator.wikimedia.org/T50891


<3


> We still need VE support: https://phabricator.wikimedia.org/T55974


 Cool, will concentrate my efforts there. :)

I hope we can figure out a nice intermediate behavior that keeps the
annotations and doesn't act too surprising to the user. I can't dedicate
huge amounts of time to it but would be happy to help where I can.

-- brion

 --scott

>
> On Fri, Apr 1, 2016 at 1:00 PM, Gregory Varnum <[hidden email]
> <javascript:;>>
> wrote:
>
> > I think improving VE to understand translate, and expanding the
> > training/documentation for translate are the best next steps. This is
> also
> > a problem on Meta-Wiki.
> >
> > Removing the translate setup from the pages seems problematic as we're
> > trying to make info available in languages beyond just English.
> >
> > -greg
> >
> > _______________
> > Sent from my iPhone - a more detailed response may be sent later.
> >
> > > On Apr 1, 2016, at 9:49 AM, Brion Vibber <[hidden email]
> <javascript:;>> wrote:
> > >
> > > So based on notes from Niklas on the bug, it looks like there are two
> > main
> > > problems:
> > >
> > > 1) VE treats <translate>...</translate> chunks as uneditable extension
> > > blobs.
> > >
> > > 2) Some/many pages use <translate>...</translate> incorrectly: spanning
> > > paragraphs, markup level boundaries, etc.
> > >
> > >
> > > So problem 1) can be improved in two ways:
> > > 1a) Have VE treat <translate>...</translate> as more or less like
> > plaintext
> > > or a div or something
> > > 1b) Much fancier sub-document editing
> > >
> > > Fixing problem 2) means refactoring broken pages in ways that
> presumably
> > > would require at least some of their translations to be re-created.
> > >
> > > I don't know whether 2) is caused by problems in the translation
> > interface,
> > > or by people further editing the pages without a detailed understanding
> > of
> > > how the <translate> and <!--T:123--> markup bits work, or a combination
> > of
> > > both.
> > >
> > > -- brion
> > >
> > >
> > >
> > >> On Fri, Apr 1, 2016 at 7:39 PM, Brion Vibber <[hidden email]
> <javascript:;>>
> > wrote:
> > >>
> > >> On Fri, Apr 1, 2016 at 7:35 PM, C. Scott Ananian <
> > [hidden email] <javascript:;>>
> > >> wrote:
> > >>
> > >>> Proper VE/Parsoid support for the <translate> extension has been
> wanted
> > >>> for
> > >>> a while now.  It hasn't made a list of quarterly goals yet.  Help
> > wanted,
> > >>> certainly.
> > >>
> > >> This is a HUGELY important dogfooding goal; we need to be able to edit
> > the
> > >> wiki about our own product!
> > >>
> > >> Anything we can do to help out?
> > >>
> > >> -- brion
> > > _______________________________________________
> > > Wikitech-l mailing list
> > > [hidden email] <javascript:;>
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email] <javascript:;>
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
>
>
>
> --
> (http://cscott.net)
> _______________________________________________
> Wikitech-l mailing list
> [hidden email] <javascript:;>
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Isarra Yos]

Is there any way to just flag pages to not get translated? These tags
are basically the main reason I've given up on documenting any of my
extensions/skins.

On 01/04/16 16:31, Brion Vibber wrote:

> Lots of pages on mediawiki.org are pretty much uneditable because they're
> strewn with <translate> spanning multiple paragraphs that make VisualEditor
> completely unusable and the source editor very difficult to use.
>
> I'm trying to update documentation, and I'm seriously thinking of regexing
> out all the <translate> stuff so I can actually edit the wiki pages.
>
> This is probably not a good thing.
>
>
> https://phabricator.wikimedia.org/T131516
>
> -- brion
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Jon Robson]

I'm not too aware of the history, but is there good reason we do not have
en.mediawiki.org etc?
The Translate tag has always seemed like a hack that I've never quite
understood.
The translate tag causes lots of issues on mobile (impacting usability and
performance) due to not playing well with the rest of the language
ecosystem.



On Sun, Apr 3, 2016 at 11:00 AM, Isarra Yos <[hidden email]> wrote:

> Is there any way to just flag pages to not get translated? These tags are
> basically the main reason I've given up on documenting any of my
> extensions/skins.
>
>
> On 01/04/16 16:31, Brion Vibber wrote:
>
>> Lots of pages on mediawiki.org are pretty much uneditable because they're
>> strewn with <translate> spanning multiple paragraphs that make
>> VisualEditor
>> completely unusable and the source editor very difficult to use.
>>
>> I'm trying to update documentation, and I'm seriously thinking of regexing
>> out all the <translate> stuff so I can actually edit the wiki pages.
>>
>> This is probably not a good thing.
>>
>>
>> https://phabricator.wikimedia.org/T131516
>>
>> -- brion
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

--
Jon Robson
* http://jonrobson.me.uk
* https://www.facebook.com/jonrobson
* @rakugojon
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Adam Wight-2]

On Apr 3, 2016 1:30 AM, "Jon Robson" <[hidden email]> wrote:
> The Translate tag has always seemed like a hack that I've never quite
understood.

+1. Couldn't we use Parsoid data tags to identify paragraphs? It seems like
that would lend itself to an incremental migration.

-Adam
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[rupert THURNER-2]

On Sun, Apr 3, 2016 at 4:13 PM, Adam Wight <[hidden email]> wrote:
> On Apr 3, 2016 1:30 AM, "Jon Robson" <[hidden email]> wrote:
>> The Translate tag has always seemed like a hack that I've never quite
> understood.
>
> +1. Couldn't we use Parsoid data tags to identify paragraphs? It seems like
> that would lend itself to an incremental migration.
>

i tried it once, at
https://meta.wikimedia.org/wiki/Grants:IdeaLab/Inspire/de. if you look
at the page its a mix of english and german. it shows 96% translated.
if you look at https://meta.wikimedia.org/w/index.php?title=Special:Translate&group=page-Grants%3AIdeaLab%2FInspire&action=page&filter=&language=de
it seems complete. and if you want to edit the original page i feel
the user experience nightmarish. the reusability factor to translate
articles from de.wikipedia to en.wikipedia or the other way round is
zero. if this mess goes away i'd be happy :)

rupert

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Niklas Laxström]

2016-04-03 11:29 GMT+03:00 Jon Robson <[hidden email]>:
> The Translate tag has always seemed like a hack that I've never quite
> understood.

I am happy to direct to our documentation [1] anyone who asks, or
explain if the documentation is not sufficient.

The word hack can have both positive and negative meanings and it is
unclear what do you mean with it here.

> The translate tag causes lots of issues on mobile (impacting usability and
> performance) due to not playing well with the rest of the language
> ecosystem.

I was under the impression that mobile does not work with the wikitext
directly. Translate tags never appear in the parsed wikitext output.
Can you please point me to the tasks in Phabricator where these are
explained? I am especially curious about the part about "rest of the
language ecosystem" because having worked on many parts of it, I don't
feel the same way.

[1] https://www.mediawiki.org/wiki/Help:Extension:Translate

  -Niklas

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[rupert THURNER-2]

On Sun, Apr 3, 2016 at 9:48 PM, Niklas Laxström
<[hidden email]> wrote:
> 2016-04-03 11:29 GMT+03:00 Jon Robson <[hidden email]>:
>> The Translate tag has always seemed like a hack that I've never quite
>> understood.
>
> I am happy to direct to our documentation [1] anyone who asks, or
> explain if the documentation is not sufficient.
>
> The word hack can have both positive and negative meanings and it is
> unclear what do you mean with it here.

lol - negative :) but i did not make that connection to translatewiki.
how many approaches to "translate" exist now? one for translating
articles? a translate extension? something in wikidata? makes 3?

rupert

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Niklas Laxström]

To Brion and other people who think the page translation markup is
annoying and a usability issue: As the (then volunteer) developer who
created it, I can only agree.

The way page translations currently works, which is extensively
documented at [0], is the result of lots of experimenting with what
works and what does not. When developing this feature, I got some
first hand experience working with the MediaWiki parser, which was not
always easy.

Yes, the wikipage translation feature can and should be improved as
technology progresses: we now have a visual editor which did not exist
when this feature was developed. However, as shown by statistics [1],
these issues do not prevent the feature from being used to translate
thousands of pages, including the weekly tech news. The markup issue
is not a reason to stop using this feature.

For this quarter the Language team is going to address high priority
issues in Translate that can prevent proper use of the page
translation feature [2]. Our team is small and has a huge scope of
work. Only with help of others it is possible to proceed faster and
cover more ground.

To remind us about our visions: we should "make efforts to support the
translation of key documents into multiple languages" [3] and we
should "provide the essential infrastructure for the support and
development of multilingual wiki projects" [4].

I do not wish to spend my time arguing repeatedly for these goals.
Instead I want to work on making them happen. My request is that we
(especially us English speakers and developers) accept some
inconvenience when necessary to support multilingualism. [5]

[0] https://www.mediawiki.org/wiki/Help:Extension:Translate
[1] https://www.mediawiki.org/wiki/Wikimedia_Language_engineering/Reports/2016-March#Usage_data_2
[2] https://phabricator.wikimedia.org/project/profile/1854/
[3] https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Guiding_Principles
[4] https://wikimediafoundation.org/wiki/Mission_statement
[5] Not limited to this thread, see
https://gerrit.wikimedia.org/r/#/c/214893/ and
https://phabricator.wikimedia.org/T39797 for examples which are stuck
for a long time.

  -Niklas

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Subramanya Sastry]

Niklas and the language team: thanks for your efforts in enabling
translation features. They are truly important and necessary.

As for the topic of hacks, I feel wikitext's history has been one where
people have stepped in to address critical issues / needs that existed
at the time with whatever resources were available at that time to get
stuff done. They have not necessarily been the most elegant solutions
always. So, I think wikitext's history is one of hacks in the best
possible sense of the term demonstrating resourcefulness to get things
done even while, when looked at from a technical lens, some of those
have also been hacks in the negative sense. I don't think there is
anything controversial saying that.

As a member of the parsing team, I have been interested in making
incremental progress in addressing this technical debt in wikitext
markup. We have been slowly trying to nudge wikitext towards semantics
that are more structural rather than being purely string-based. We have
been making RFC proposals towards this end. Now that Parsoid's utility
has been established, we have energy and mental space freed up to think
beyond the immediate problem of supporting visual editing. I believe we
have a very powerful change-management tool in Parsoid and we should try
to leverage it as such.

Anyway, let us think through what might be good ways of solving this
translation problem. Some high-level questions in that direction:

1. Given the success of CX, and the increasing use of VE, is explicit
wikitext markup still necessary for enabling translation?

2. Identifying document fragments for translation is another instance of
the same problem of associating metadata with document fragments *across
edits*. Citations, content-translation, comments-as-documentation,
authorship information,
maintaining-association-between-translated-fragments, etc. are all
different instances of this problem. Translate extension seems to be
using comments in wikitext markup as one way to maintain this
cross-edit-association. Maybe there is value in thinking about this
problem broadly.

3. Are there incremental steps we can take to start deprecating the
existing <translate> extension solution and move towards some structural
metadata solution? Given that translate extension is not used on *all
wikis*, looking at the wikis where translation is enabled, is there a
possibility of making this a VE-only / CX-only feature? CX, for example,
is already doing work to maintain association between translated
fragments across document edits.

Subbu.

On 04/04/2016 07:13 AM, Niklas Laxström wrote:

> To Brion and other people who think the page translation markup is
> annoying and a usability issue: As the (then volunteer) developer who
> created it, I can only agree.
>
> The way page translations currently works, which is extensively
> documented at [0], is the result of lots of experimenting with what
> works and what does not. When developing this feature, I got some
> first hand experience working with the MediaWiki parser, which was not
> always easy.
>
> Yes, the wikipage translation feature can and should be improved as
> technology progresses: we now have a visual editor which did not exist
> when this feature was developed. However, as shown by statistics [1],
> these issues do not prevent the feature from being used to translate
> thousands of pages, including the weekly tech news. The markup issue
> is not a reason to stop using this feature.
>
> For this quarter the Language team is going to address high priority
> issues in Translate that can prevent proper use of the page
> translation feature [2]. Our team is small and has a huge scope of
> work. Only with help of others it is possible to proceed faster and
> cover more ground.
>
> To remind us about our visions: we should "make efforts to support the
> translation of key documents into multiple languages" [3] and we
> should "provide the essential infrastructure for the support and
> development of multilingual wiki projects" [4].
>
> I do not wish to spend my time arguing repeatedly for these goals.
> Instead I want to work on making them happen. My request is that we
> (especially us English speakers and developers) accept some
> inconvenience when necessary to support multilingualism. [5]
>
> [0] https://www.mediawiki.org/wiki/Help:Extension:Translate
> [1] https://www.mediawiki.org/wiki/Wikimedia_Language_engineering/Reports/2016-March#Usage_data_2
> [2] https://phabricator.wikimedia.org/project/profile/1854/
> [3] https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Guiding_Principles
> [4] https://wikimediafoundation.org/wiki/Mission_statement
> [5] Not limited to this thread, see
> https://gerrit.wikimedia.org/r/#/c/214893/ and
> https://phabricator.wikimedia.org/T39797 for examples which are stuck
> for a long time.
>
>    -Niklas
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Niklas Laxström]

2016-04-04 16:00 GMT+03:00 Subramanya Sastry <[hidden email]>:
> Niklas and the language team: thanks for your efforts in enabling
> translation features. They are truly important and necessary.

And I want to thank you for your positive and constructive approach
for solving this issue.


> 1. Given the success of CX, and the increasing use of VE, is explicit
> wikitext markup still necessary for enabling translation?

I am actually eager to try out non mark-up approach for Translate's
page translation feature. I am not aware of any fundamental reason for
not being able to work without additional wikitext mark-up. But I
would like to clarify some things about the relation of Translate
(specifically its page translation feature) and Content Translation
(CX).

The use case for CX is very different from Translate's page
translation. Former supports one-time "free form" translation from one
language to one language. The latter supports maintaining
content-preserving translations from one language to hundreds of
languages so that translations can be kept in sync when the source
text changes.

Along the same line the occasional suggestions about replacing either
extension with the other one is not practical. My opinion is that the
way to bring these closer together is to take the best parts of both
(and also others like VE) and combine them to produce tools which are
tailored for each use case and which are consistent in appearance and
functionality. Translate does so much more that it is easier to add a
new translation interface to Translate than it is to re-implement all
the backend tracking functionality in CX.


> 2. Identifying document fragments for translation is another instance of the
> same problem of associating metadata with document fragments *across edits*.
> Citations, content-translation, comments-as-documentation, authorship
> information, maintaining-association-between-translated-fragments, etc. are
> all different instances of this problem. Translate extension seems to be
> using comments in wikitext markup as one way to maintain this
> cross-edit-association. Maybe there is value in thinking about this problem
> broadly.

Definitely. As we have discussed earlier, the definition of what is a
"breaking change" does vary between the different use cases, and in
fact is left to a humans (translation admins) to decide in the page
translation feature. But the other approach of each tool implementing
it from scratch is not ideal either.

One thing to remember here is that there are two goals with the
current mark-up. The association of content across edits is only one
of them (these are the T-comment you refer to). The other goal is to
specify what is and what is not translatable. For example, one
frequent use case is to mark for translation only the image captions
but not rest of the image wikitext mark-up. I am hopeful that some
heuristics with additional tools for manual correction would reach
good results for this goal.

Yet another thing related to this is that we will want to support
WYSIWYG/HTML translation in Translate. As you might know already,
Special:Translate has two modes: the list view and the page view. Page
view should be replaced with interface not much unlike CX, but we also
need some support for the list view.


> 3. Are there incremental steps we can take to start deprecating the existing
> <translate> extension solution and move towards some structural metadata
> solution? Given that translate extension is not used on *all wikis*, looking
> at the wikis where translation is enabled, is there a possibility of making
> this a VE-only / CX-only feature? CX, for example, is already doing work to
> maintain association between translated fragments across document edits.

What to do with the translate tags is a delicate question and whether
to stop supporting them altogether is a question that has lots of
dependencies such as how long there will exist third party wikis
(where Translate is also used a lot) that prefer to use wikitext
instead of VE and parsoid or alternatives. See also my thoughts to
your first question.

Right now my answer is no: we cannot make this implementation of the
feature VE or CX only. My preference is to start developing a parallel
mark-up-less system and to provide migration tools. This new system
can depend on VE and parsoid and other modern functionality. Its
implementation will require cross team planning and collaboration.

  -Niklas

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Jon Robson-2]

On Sun, Apr 3, 2016 at 10:48 PM, Niklas Laxström <[hidden email]>
wrote:

> 2016-04-03 11:29 GMT+03:00 Jon Robson <[hidden email]>:
> > The Translate tag has always seemed like a hack that I've never quite
> > understood.
>
> I am happy to direct to our documentation [1] anyone who asks, or
> explain if the documentation is not sufficient.
>
> The word hack can have both positive and negative meanings and it is
> unclear what do you mean with it here.
>

FWIW I didn't mean this in a negative way. Hack is always a neutral word in
my vocabulary :-). Possibly a more descriptive term would have been
"oddity". I've personally had to push changes in MobileFrontend that felt
wrong, but were necessary due to the environmental constraints I worked in.
(Side note: In return have to put up with insulting comments such as
"MobileFrontend is an abonimation" from people that haven't taken the time
to understand why things are done the way they are). I've ignored those
people and still consider the work I've done as useful and important, just
as I value the work everyone has put into our language support.

The least I can do is elaborate on my brevity earlier (sent via mobile
during hackathon :-)) I think Subbu has covered very well my feelings
around this so I don't have much to add to what he's said.

My original concern however was not around the documentation,  but how we
have 2 mechanisms for doing languages - create a separate site or use the
translate tag. It wasn't clear to me which predates the other and why that
decision was made. The advantage of a separate site is that it is the
simplest possible way to translate.. through no effort whatsoever an editor
can translate an article on a mobile device for example by navigating to
their project and clicking the edit button. On the other hand
Special:Translate doesn't work [1] and the current plan is to make it
redirect to desktop which is disappointing and I'd guess loses us lots of
potential editors (myself included).

>
> > The translate tag causes lots of issues on mobile (impacting usability
> and
> > performance) due to not playing well with the rest of the language
> > ecosystem.
>
> I was under the impression that mobile does not work with the wikitext
> directly. Translate tags never appear in the parsed wikitext output.
> Can you please point me to the tasks in Phabricator where these are
> explained? I am especially curious about the part about "rest of the
> language ecosystem" because having worked on many parts of it, I don't
> feel the same way.
>
> [1] https://www.mediawiki.org/wiki/Help:Extension:Translate

When we experimented with the first versions of the editor, we found that
edits significantly increased when we loaded sections rather than the
entire article - so less wikitext leads to greater number of edits. The
translate tag makes the wikitext more confusing and bloats it when
translations exist, but I've not investigated so this would need more
investigation.

To take a more concrete example of impact on mobile - we've made the mobile
skin play nicely with language interwiki links (we've even dedicated this
entire quarter to improving language switching on mobile web [2] ). On the
other hand, the languages tag does not work the same way as an interwiki
link. It does it's own thing which is sadly suffering from usability issues
on mobile [3].

The point being that when we don't consolidate systems that do similar
things, we are losing opportunities to benefit from things elsewhere or
waste engineer time fixing 2 identical problems.

The main issues I have are more from an architecture perspective. I would
expect a function along the lines of
ArticlePage->getLanguages() to exist and be agnostic to where languages
come from - translate tag or interwiki link for consumers such as skins and
apps.

I hope my views on this are a little clearer to you now and apologies for
putting you on the defensive if I did.

I'd love to see our new language switcher compatible with the output of the
translate tag and the translation mechanisms available on mobile phones, so
readers can view translations and edit seamlessly around our projects.




>
>
>   -Niklas
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

[1] https://phabricator.wikimedia.org/T102922
[2] https://phabricator.wikimedia.org/T121919
[3] https://phabricator.wikimedia.org/T106361
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Ryan Kaldari-2]

On Fri, Apr 1, 2016 at 10:49 AM, Brion Vibber <[hidden email]> wrote:

> 2) Some/many pages use <translate>...</translate> incorrectly: spanning
> paragraphs, markup level boundaries, etc.
>

The documentation says to do it that way:
https://www.mediawiki.org/wiki/Help:Extension:Translate/Page_translation_example
Is that incorrect?
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber-4]

On Mon, Apr 4, 2016 at 5:21 PM, Niklas Laxström <[hidden email]>
wrote:

> The use case for CX is very different from Translate's page
> translation. Former supports one-time "free form" translation from one
> language to one language. The latter supports maintaining
> content-preserving translations from one language to hundreds of
> languages so that translations can be kept in sync when the source
> text changes.
>

*nod* one big difficulty with doing a CX-like model for documentation pages
is in synchronizing updates...

I think that in combination with good diff/changeset displays, the CX model
can be extended to support updating translations for edited pages when
we're translating the way we do documentation: with one master copy,
usually English, that has translated 'clones' in other languages. It would
likely be much harder to support back-and-forth edits for multilingual
pages that aren't required to be kept in sync with a master, as with
Wikipedia articles in different languages.

It would be a little different from how <translate> blocks work in that
when new text is added and hasn't been translated yet, you don't see the
updated English text on your localized page. But we can automate a marker
that says the English page has been updated and can link you back to it.

> > 2. Identifying document fragments for translation is another instance of
> the
> > same problem of associating metadata with document fragments *across
> edits*.
> > Citations, content-translation, comments-as-documentation, authorship
> > information, maintaining-association-between-translated-fragments, etc.
> are
> > all different instances of this problem. Translate extension seems to be
> > using comments in wikitext markup as one way to maintain this
> > cross-edit-association. Maybe there is value in thinking about this
> problem
> > broadly.
>
> Definitely. As we have discussed earlier, the definition of what is a
> "breaking change" does vary between the different use cases, and in
> fact is left to a humans (translation admins) to decide in the page
> translation feature. But the other approach of each tool implementing
> it from scratch is not ideal either.
>

*nod* with a master->translation model, this is something that can be
human-assisted; note that good heuristics in diff detection can already do
things like detect moved paragraphs, which should help.

> One thing to remember here is that there are two goals with the
> current mark-up. The association of content across edits is only one
> of them (these are the T-comment you refer to). The other goal is to
> specify what is and what is not translatable. For example, one
> frequent use case is to mark for translation only the image captions
> but not rest of the image wikitext mark-up. I am hopeful that some
> heuristics with additional tools for manual correction would reach
> good results for this goal.
>
> Yet another thing related to this is that we will want to support
> WYSIWYG/HTML translation in Translate. As you might know already,
> Special:Translate has two modes: the list view and the page view. Page
> view should be replaced with interface not much unlike CX, but we also
> need some support for the list view.
>

+1 on all this. :)

For marking untranslatable bits, I suspect it may be better to default all
text to translatable except where code blocks are explicitly marked. We can
perhaps provide a markup like <nowiki> for <notranslate> as well, which
should be something that can be integrated in both source and visual
editors.

> > 3. Are there incremental steps we can take to start deprecating the
> existing
> > <translate> extension solution and move towards some structural metadata
> > solution? Given that translate extension is not used on *all wikis*,
> looking
> > at the wikis where translation is enabled, is there a possibility of
> making
> > this a VE-only / CX-only feature? CX, for example, is already doing work
> to
> > maintain association between translated fragments across document edits.
>
> What to do with the translate tags is a delicate question and whether
> to stop supporting them altogether is a question that has lots of
> dependencies such as how long there will exist third party wikis
> (where Translate is also used a lot) that prefer to use wikitext
> instead of VE and parsoid or alternatives. See also my thoughts to
> your first question.
>
> Right now my answer is no: we cannot make this implementation of the
> feature VE or CX only. My preference is to start developing a parallel
> mark-up-less system and to provide migration tools. This new system
> can depend on VE and parsoid and other modern functionality. Its
> implementation will require cross team planning and collaboration.
>

+1

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l