Investigating building an apps content service using RESTBase and Node.js

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Investigating building an apps content service using RESTBase and Node.js

Bryan Davis
On Wed, Feb 4, 2015 at 4:00 PM, Brion Vibber <[hidden email]> wrote:

> I think the way we'd want to go is roughly to have a *partnership between*
> the Services and Mobile teams produce and maintain the service.
>
> (Note that the state of the art is that Mobile Apps are using Mobile Web's
> MobileFrontend extension as an intermediate API to aggregate & format page
> data -- which basically means Max has done the server-side API work for
> Mobile Apps so far.)
>
> I'd expect to see Max and/or someone else from the Mobile team
> collaborating with the Services team to create what y'all need:
> 1) something that does what Mobile Apps needs it to...
> 2) and can be maintained like Services needs it to.
>
> In general I'm in favor of more ad-hoc project-specific teams rather than
> completely siloing every service to the Services group, or every mobile UI
> to the Mobile group.

+1. This is the only thing that will scale in my opinion. "Full stack"
teams involving design, front end, back end, ops, release, project
management, and testing resources should be formed to work on vertical
slices of functionality ("features" or "products") that are
prioritized by the entire organization. Thinking that some team can be
called on to fulfill all of the cross-feature/product needs is
madness. Services is 3 people, MediaWiki-Core is 9 people (minus
standing obligations like security and performance reviews). Teams of
this size cannot be expected to service all the "backend" needs of the
myriad product/feature verticals that are under the WMF umbrella. If
we don't have enough people to staff projects this way we are trying
to do too many things at once. (Which I'm pretty sure is actually the
case.)

Bryan
--
Bryan Davis              Wikimedia Foundation    <[hidden email]>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Investigating building an apps content service using RESTBase and Node.js

Dan Garry
In reply to this post by Brion Vibber-4
On 4 February 2015 at 15:00, Brion Vibber <[hidden email]> wrote:
>
> In general I'm in favor of more ad-hoc project-specific teams rather than
> completely siloing every service to the Services group, or every mobile UI
> to the Mobile group.
>

Agreed. This also ensures that the service exactly meets the functional
requirements, no more and no less.

Dan

--
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Investigating building an apps content service using RESTBase and Node.js

Tim Starling-2
In reply to this post by Marko Obrovac
On 04/02/15 16:59, Marko Obrovac wrote:
> For v1, however, we plan to
> provide only logical separation (to a certain extent) via modules which can
> be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> provide them with routing, monitoring, caching and authorisation out of the
> box.

We already have routing, monitoring and caching in Varnish. That seems
like a good place to implement further service routing to me.

<https://git.wikimedia.org/blob/operations%2Fpuppet/4a7f5ce62d9cdd1ace20ca6c489cbdb538503750/templates%2Fvarnish%2Fmisc.inc.vcl.erb>

It's simple to configure, has excellent performance and scalability,
and monitoring and a distributed logging system are already implemented.

It doesn't have authorisation, but I thought that was going to be in a
separate service from RESTBase anyway.

-- Tim Starling


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Investigating building an apps content service using RESTBase and Node.js

Yuri Astrakhan-2
Tim, I like Varnish's vcl flexibility, but not the debugging aspects.
Still, +1, but could you elaborate on how you see:

* Services communicate with each other - via varnish as well or directly?
* Do you see routing varnish layer as non-caching, only to forward request
to the second tear service-specific varnish instance that will actually
perform caching?
* Will there be any problems separating services into separate, non
mutually trusted clusters?
* service management - varnish would only indicate overall health of the
service. What about service-specific real time monitored values (if
needed), logs, controlling service instances. Any thoughts on generalizing
that infrastructure?

Lastly, semi related - my service will also need a node.js lib to access MW
api. How should we manage common libs like that?

Thanks!
On Feb 6, 2015 4:15 AM, "Tim Starling" <[hidden email]> wrote:

> On 04/02/15 16:59, Marko Obrovac wrote:
> > For v1, however, we plan to
> > provide only logical separation (to a certain extent) via modules which
> can
> > be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> > provide them with routing, monitoring, caching and authorisation out of
> the
> > box.
>
> We already have routing, monitoring and caching in Varnish. That seems
> like a good place to implement further service routing to me.
>
> <
> https://git.wikimedia.org/blob/operations%2Fpuppet/4a7f5ce62d9cdd1ace20ca6c489cbdb538503750/templates%2Fvarnish%2Fmisc.inc.vcl.erb
> >
>
> It's simple to configure, has excellent performance and scalability,
> and monitoring and a distributed logging system are already implemented.
>
> It doesn't have authorisation, but I thought that was going to be in a
> separate service from RESTBase anyway.
>
> -- Tim Starling
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Investigating building an apps content service using RESTBase and Node.js

Bryan Davis
On Thu, Feb 5, 2015 at 9:00 PM, Yuri Astrakhan <[hidden email]> wrote:
> Lastly, semi related - my service will also need a node.js lib to access MW
> api. How should we manage common libs like that?

Libraries published on npm using semantic versioning and imported to
the individual services as needed.

Bryan
--
Bryan Davis              Wikimedia Foundation    <[hidden email]>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Investigating building an apps content service using RESTBase and Node.js

Giuseppe Lavagetto
In reply to this post by Marko Obrovac
On Wed, Feb 4, 2015 at 6:59 AM, Marko Obrovac <[hidden email]> wrote:

> On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling <[hidden email]>
> wrote:
>
>> I don't really understand why you want it to be integrated with
>> RESTBase. As far as I can tell (it is hard to pin these things down),
>> RESTBase is a revision storage backend and possibly a public API for
>> that backend.
>
>
> Actually, RESTBase's logic applies to the Mobile Apps case quite naturally.
> When a page is fetched and transformed, it can be stored so that consequent
> requests can simply retrieve the transformed document form storage.
>
>

Ok, so in this vision RESTBase is a caching layer for revisions

>> I thought the idea of SOA was to separate concerns.
>> Wouldn't monitoring, caching and authorization would be best done as a
>> node.js library which RESTBase and other services use?
>>
>
> Good point. Ideally, what we would need to do is provide the right tools to
> developers to create services, which can then be placed "strategically"
> around DCs (in cooperation with Ops, ofc). For v1, however, we plan to
> provide only logical separation (to a certain extent) via modules which can
> be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> provide them with routing, monitoring, caching and authorisation out of the
> box. The good point here is that this 'modularisation' eases the transition
> to a more-decomposed orchestration SOA model. Going in that direction,
> however, requires some prerequisites to be fulfilled, such as [1].
>

So, now RESTBase has become a router and an auth provider as well?
(Gabriel already clarified me that "monitoring" means that RESTbase
will expose its own metrics for that specific service, so this is not
monitoring of the service at all, rather accounting).

I need some clarification at this point - what is RESTBase really
going to do? I'm asking because when I read "RESTBase will provide
them with routing, [...] and authorisation" I immediately think of a
request router and a general on-wiki auth provider. And we already
have both, and re-doing them in RESTBase would be plainly wrong.

Maybe you intend very specific things when you say "routing" and not
request routing, which is what everybody here will think of.
And when you say "auth" you mean that RESTBase implements an auth
schema for its clients, so that no client can access data from another
one. If this is the case, I have some further questions: is this going
to be RBAC? Which permissions models are you implementing? Are we sure
it is what we will need? And foremost: will this be exposable to
external consumers? will it be able to hook up to our traditional wiki
auth scheme?

Can you please expand a bit on those concepts? Or a lot of confusion,
uncertainty and doubt will spread amongst your fellow engineers,
resulting in an hostile view of what may be a perfectly well designed
software.

Cheers,

Giuseppe

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Investigating building an apps content service using RESTBase and Node.js

Giuseppe Lavagetto
In reply to this post by Marko Obrovac
On Wed, Feb 4, 2015 at 6:59 AM, Marko Obrovac <[hidden email]> wrote:

> On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling <[hidden email]>
> wrote:
>
>> I don't really understand why you want it to be integrated with
>> RESTBase. As far as I can tell (it is hard to pin these things down),
>> RESTBase is a revision storage backend and possibly a public API for
>> that backend.
>
>
> Actually, RESTBase's logic applies to the Mobile Apps case quite naturally.
> When a page is fetched and transformed, it can be stored so that consequent
> requests can simply retrieve the transformed document form storage.
>
>

Ok, so in this vision RESTBase is a caching layer for revisions

>> I thought the idea of SOA was to separate concerns.
>> Wouldn't monitoring, caching and authorization would be best done as a
>> node.js library which RESTBase and other services use?
>>
>
> Good point. Ideally, what we would need to do is provide the right tools to
> developers to create services, which can then be placed "strategically"
> around DCs (in cooperation with Ops, ofc). For v1, however, we plan to
> provide only logical separation (to a certain extent) via modules which can
> be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> provide them with routing, monitoring, caching and authorisation out of the
> box. The good point here is that this 'modularisation' eases the transition
> to a more-decomposed orchestration SOA model. Going in that direction,
> however, requires some prerequisites to be fulfilled, such as [1].
>

So, now RESTBase has become a router and an auth provider as well?
(Gabriel already clarified me that "monitoring" means that RESTbase
will expose its own metrics for that specific service, so this is not
monitoring of the service at all, rather accounting).

I need some clarification at this point - what is RESTBase really
going to do? I'm asking because when I read "RESTBase will provide
them with routing, [...] and authorisation" I immediately think of a
request router and a general on-wiki auth provider. And we already
have both, and re-doing them in RESTBase would be plainly wrong.

Maybe you intend very specific things when you say "routing" and not
request routing, which is what everybody here will think of.
And when you say "auth" you mean that RESTBase implements an auth
schema for its clients, so that no client can access data from another
one. If this is the case, I have some further questions: is this going
to be RBAC? Which permissions models are you implementing? Are we sure
it is what we will need? And foremost: will this be exposable to
external consumers? will it be able to hook up to our traditional wiki
auth scheme?

Can you please expand a bit on those concepts? Or a lot of confusion,
uncertainty and doubt will spread amongst your fellow engineers,
resulting in an hostile view of what may be a perfectly well designed
software.

Cheers,

Giuseppe

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Investigating building an apps content service using RESTBase and Node.js

Nuria Ruiz
In reply to this post by James Douglas
> In general I'm in favor of more ad-hoc project-specific teams rather than
completely siloing every service to the Services group, or every mobile UI
to the Mobile group.=
Agreed, as long as everyone deploying services communicates through
services team so there is no duplication of solutions.
We should have 1 routing layer, standard monitoring and standard caching as
it is been pointed out before. Specifics of services might be different but
it seems the core mission of service team to provide common infrastructure,
right?.
That means that the 1st service we develop will be more work for services
team than subsequent services. *Ideally* the mobile team should develop the
service without having to solve (for example) any caching problems that are
not mobile specific.





On Wed, Feb 4, 2015 at 3:04 PM, James Douglas <[hidden email]>
wrote:

> > In general I'm in favor of more ad-hoc project-specific teams rather than
> completely siloing every service to the Services group, or every mobile UI
> to the Mobile group.
>
> I strongly agree.  Based on experience on both sides of this spectrum, I
> recommend (when feasible) favoring feature teams over functional teams.
>
> On Wed, Feb 4, 2015 at 3:00 PM, Brion Vibber <[hidden email]>
> wrote:
>
> > I think the way we'd want to go is roughly to have a *partnership
> between*
> > the Services and Mobile teams produce and maintain the service.
> >
> > (Note that the state of the art is that Mobile Apps are using Mobile
> Web's
> > MobileFrontend extension as an intermediate API to aggregate & format
> page
> > data -- which basically means Max has done the server-side API work for
> > Mobile Apps so far.)
> >
> > I'd expect to see Max and/or someone else from the Mobile team
> > collaborating with the Services team to create what y'all need:
> > 1) something that does what Mobile Apps needs it to...
> > 2) and can be maintained like Services needs it to.
> >
> > In general I'm in favor of more ad-hoc project-specific teams rather than
> > completely siloing every service to the Services group, or every mobile
> UI
> > to the Mobile group.
> >
> > -- brion
> >
> > On Wed, Feb 4, 2015 at 2:29 PM, Corey Floyd <[hidden email]>
> wrote:
> >
> > > On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke <[hidden email]>
> > > wrote:
> > >
> > > > Regarding general-purpose APIs vs. mobile: I think mobile is in some
> > > ways a
> > > > special case as their content transformation needs are closely
> coupled
> > > with
> > > > the way the apps are presenting the content. Additionally, at least
> > until
> > > > SPDY is deployed there is a strong performance incentive to bundle
> > > > information in a single response tailored to the app's needs. One
> > > strategy
> > > > employed by Netflix is to introduce a second API layer
> > > > <
> > > >
> > >
> >
> http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html
> > > > >
> > > > on
> > > > top of the general content API to handle device-specific needs. I
> think
> > > > this is a sound strategy, as it contains the volatility in a separate
> > > layer
> > > > while ensuring that everything is ultimately consuming the
> > > general-purpose
> > > > API. If the need for app-specific massaging disappears over time, we
> > can
> > > > simply shut down the custom service / API end point without affecting
> > the
> > > > general API.
> > > >
> > >
> > >
> > > I can definitely understand that motivation for providing mobile
> specific
> > > service layer - so if the services team wants to implement the API in
> > this
> > > way and support that architecture, I am totally on board.
> > >
> > > My remaining hesitation here is that from the reading of this proposal,
> > the
> > > mobile team is the owner of implementing this service, not the services
> > > team (Maybe I am misreading?).
> > >
> > > This leads me to ask questions like:
> > > Why is the mobile apps team investigating which is the best server side
> > > technology? That seems outside of our domain knowledge.
> > > Who will be responsible for maintaining this code?
> > > Who will be testing it to make sure that is performant?
> > >
> > > I'm new, so maybe these answers are obvious to others, but to me they
> > seem
> > > fuzzy when responsibilities are divided between two teams.
> > >
> > > I would propose that this be a project that the Services Team owns. And
> > > that the Mobile Apps Team defines specs on what they need the new
> service
> > > to provide.
> > > _______________________________________________
> > > Wikitech-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
12