Heads-up: WMF engineering process improvement meetings

[Erik Moeller-4]

Hi folks,

this is a heads-up that Tomasz, Rob, Alolita, CT, Brion, Roan[TBC],
Howie and I will be meeting with ThoughtWorks (
http://en.wikipedia.org/wiki/ThoughtWorks ) Thursday and Friday. Rob
and I also got a download from Tim ahead of time.

ThoughtWorks has done great work with the fundraising developers
(Arthur, Katie, Kaldari) already in helping protect that team from
distractions, prioritize its work, and ensure that
roles/responsibilities are clearly understood vis-a-vis the
fundraising folks in the community department. So, we're now exploring
a deeper engagement with ThoughtWorks on problem-solving across the
engineering organization.

Following initial conversations, the purpose of the meeting tomorrow
is to do a deep-dive into a specific problem set: the code review,
deployment and release management process. We'll be digging into
questions like:

1) What are the best methods to ensure we keep up with the backlog
while still maintaining a good clip of WMF priority development;
2) What's a realistic deployment and release cycle to shoot for (for
trunk, extensions, branches);
3) How do we dissipate key skills more widely among both staff and
volunteers (e.g. deployment, security reviews);
4) How/when can we split "big hairy projects" with integration issues
into more manageable chunks;
5) What other high priority improvements do we need to prioritize
(e.g. increased test coverage, improvements to the testing frameworks,
het-deploy, staging environments, etc.)

We're intentionally keeping this first meeting at a manageable size to
have a high face-to-face throughput and to explore where ThoughtWorks
can best help us. But I'm very much intending to make our thinking
public, and to form clear and visible groups around core problems
we're tackling, just as we have been doing with all WMF engineering
projects. So, I'll keep you posted, and if you have thoughts that
you'd like to post ahead of time, please do it onlist or offlist :-)

Thanks,
Erik

--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[MZMcBride-2]

Erik Moeller wrote:

> Following initial conversations, the purpose of the meeting tomorrow
> is to do a deep-dive into a specific problem set: the code review,
> deployment and release management process. We'll be digging into
> questions like:
>
> 1) What are the best methods to ensure we keep up with the backlog
> while still maintaining a good clip of WMF priority development;
> 2) What's a realistic deployment and release cycle to shoot for (for
> trunk, extensions, branches);
> 3) How do we dissipate key skills more widely among both staff and
> volunteers (e.g. deployment, security reviews);
> 4) How/when can we split "big hairy projects" with integration issues
> into more manageable chunks;
> 5) What other high priority improvements do we need to prioritize
> (e.g. increased test coverage, improvements to the testing frameworks,
> het-deploy, staging environments, etc.)
>
> We're intentionally keeping this first meeting at a manageable size to
> have a high face-to-face throughput and to explore where ThoughtWorks
> can best help us. But I'm very much intending to make our thinking
> public, and to form clear and visible groups around core problems
> we're tackling, just as we have been doing with all WMF engineering
> projects. So, I'll keep you posted, and if you have thoughts that
> you'd like to post ahead of time, please do it onlist or offlist :-)

Thanks very much for the heads-up. Posting the minutes or any notes from the
meeting on Meta-Wiki or mediawiki.org would be fantastic.

I might say that one more point to focus on specifically is to how to
leverage volunteer development (this is hinted at in some of your five
points). There are _a lot_ of people who are capable of coding in PHP and
who are willing to donate their time and talents, but Wikimedia/MediaWiki
code development has chased them off, generally through neglect (patches
sitting, review sitting, etc.). If there are ways to specifically look at
that, it would be an enormous benefit to Wikimedia/MediaWiki, I think.

Much like the wikis themselves, you can rely on people to do good work if
there's a support structure in place. The only caution is that, of course,
if patches and commits increase without additional review/deployment
support, you just make the problem worse (backlogs and frustration grow).
Still, it seems that a huge issue is that volunteer time and talent is being
wasted, so I think that's a fairly high priority (or at least should be).

I'm really glad that these issues are being looked at. I know that a lot of
people in the community (end-users and people related to the software
development) have become frustrated with the speed of code
review/development/time-to-new-features, so any step in a better direction
is fantastic. :-)

MZMcBride



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Brion Vibber]

On Thu, Jul 7, 2011 at 12:40 PM, MZMcBride <[hidden email]> wrote:

> Thanks very much for the heads-up. Posting the minutes or any notes from
> the
> meeting on Meta-Wiki or mediawiki.org would be fantastic.
>

We're taking notes on an etherpad for now; some process flowcharts are being
done on a physical wall so once they're distilled onto the wiki w/ photos it
should all be accessible together.

I might say that one more point to focus on specifically is to how to
> leverage volunteer development (this is hinted at in some of your five
> points). There are _a lot_ of people who are capable of coding in PHP and
> who are willing to donate their time and talents, but Wikimedia/MediaWiki
> code development has chased them off, generally through neglect (patches
> sitting, review sitting, etc.). If there are ways to specifically look at
> that, it would be an enormous benefit to Wikimedia/MediaWiki, I think.
>

This is a big thing we want to solve -- current processes are very vague &
laggy for a lot of stuff and only keep a fast deployment cycle for a few
WMF-driven projects; this isn't because we're mean but because those are the
only things that have internal drivers within WMF pushing them strongly
enough; how to make sure other ongoing work gets the attention it needs too
is very much on our minds.

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Alec Conroy-2]

On Thu, Jul 7, 2011 at 12:40 PM, MZMcBride <[hidden email]> wrote:

> I might say that one more point to focus on specifically is to how to
> leverage volunteer development (this is hinted at in some of your five
> points). There are _a lot_ of people who are capable of coding in PHP and
> who are willing to donate their time and talents, but Wikimedia/MediaWiki
> code development has chased them off, generally through neglect (patches
> sitting, review sitting, etc.). If there are ways to specifically look at
> that, it would be an enormous benefit to Wikimedia/MediaWiki, I think.

+1!

There's an enormous pool of volunteer developers out there who would
gladly work for us, non-stop, if we can find a way to let them.  For
many things, our templating language can be lot harder to work with
than PHP-- but despite its difficulty, look at how many useful
advanced templates have been developed without us even having to ask
for them.

Anyone who can make advanced templates can almost certainly handle
PHP.  The reason templates flourish while development flounders is
"Openness"--- templating is essentially an open platform, WMF
development is most certainly not an open platform.

Volunteer developers will do ridiculous amounts of work for us,
innovating in ways we can't even imagine.   Google's most popular
program is it's "20% time" that allows them to spend one day a week
working on whatever they want.

People want to innovate, just like people want to improve our
projects' content.  They will work for free-- but they have to know
they'll  be able to actually use their innovation themselves, and most
have to know they can share it with others if it's popular.  Most
developers won't work for free only to have a third party decide
whether it's sufficiently meritorious for its use to be allowed or
not.

Right now, there's system in place to allow me to initiate, develop,
implement, and share a feature without having to deal with a lot of
read tape and permission-getting.  If I want a Wikipedia that's a
little different in some way, I have to implement on the  client-side
or I literally have to make my own fork of Wikipedia, that involved
buying a domain name, setting up a host, raising money for it / paying
for it, etc etc etc.   A huge nightmare full of work that developers
don't enjoy.

"Be Bold" hasn't been applied to the development or new projects yet.
Right now, "Be Bold" is for an edits, not innovation.
Right now, "Be Bold" is for new articles, not new projects.

We meed to figure out how to allow developer innovations instantly,
automatically,  in real time.  But we also have to make sure those
innovations don't affect the user experience for third-parties.

Once we get such a platform, development can take off.  Until then,
development will mostly be driven by third-party mediawiki project and
paid staff--  both good to have, but orders of magnitude smaller than
the size of the volunteer developer population that is going
un-tapped.

Alec

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l