operating systems and web servers

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

operating systems and web servers

River Tarnell-7
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

So, this is not an announcement of any intent to change anything, I just
want to get an idea of how people feel about two things we could,
perhaps, change in the future:

1. ZWS to Apache on the web server
2. Solaris to FreeBSD on login servers

#2 depends on #1, so it seems sensible to discuss both together.

I don't have any strong opinion about either of these myself, but I
would like to hear what users think.

ZWS to Apache:

I know it annoyed people when we moved from Apache to ZWS initially,
because rewrite rules had to be redone, some .htaccess stuff stopped
working, etc.  At the time we were using mod_suphp for per-user (suexec)
PHP scripts, and it was extremely inefficient; the system spent most of
its time doing nothing.  ZWS allowed us to fix the problem cheaply (no
new hardware required), and I think for most users it worked just the
same.

Since then, two things have changed: firstly, Apache with mod_fcgid now
has decent FastCGI support, and with only a little work could be made to
support suexec PHP as well.  Secondly, ZWS is now in maintenance, and
won't see any further development (so it might be better to switch now,
rather than wait until one month before ZWS support ends entirely and be
forced to switch).

With that in mind, it makes sense to consider moving back to Apache.  
The main downside is that rewrite rules would have to be converted back
to Apache format (mod_rewrite).  OTOH, .htaccess features missing from
ZWS would be available again (I don't know if anyone actually needs
this, but I believe at least a few users have complained about missing
features.)

Solaris to FreeBSD:

Of the two changes, I think this one would actually be the less
disruptive.  For users, nearly everything would stay the same: we
already provide the GNU userland ('ls', etc) by default (and would
continue to do so) and the third-party software in /opt/ts would be
identical, as would cronie, SGE, Perl/Python/..., etc.  

Software-wise, since nothing would really change, I don't see any
particular advantages for users.  Disadvantages: 'ps -eaf' would stop
working ;-) and anyone with locally-compiled software (C/C++, or XS Perl
modules, etc.) would need to recompile them.

For us (admins), the main advantage is reduced maintenance overhead:
FreeBSD releases a new minor version about once a year, and supports
each for 2 years; each release branch only gets very infrequent updates
for security or errata.  In comparison, there is a new Solaris update
every 6 months, and during yesterday's maintenance we installed 358 (!)
separate patches.  Oracle doesn't provide a security-updates-only
release, and it's difficult to mix-and-match patches (e.g. to only get
security patches).  

This doesn't directly affect users, but fewer OS changes should lead to
less lengthy / disruptive maintenance and less frequent reboots.  OTOH,
I don't know if this has a noticeable impact on users at the moment, and
the previous maintenance was the first for ~170 days.

        - river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (SunOS)

iEYEARECAAYFAk3thyIACgkQIXd7fCuc5vJjsgCfYfbW28r+eW3Lr6L1T+BrzGV/
v4YAn3lkZ7+YfG8ajKBQRx7DBQRKhfnS
=lcFN
-----END PGP SIGNATURE-----

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: operating systems and web servers

Daniel Kinzler
hi river

i know we have talkedabout this before, but for the benefit of other users, let
me ask the obvious question: why FreeBSD and not Linux?

-- daniel

On 07.06.2011 04:04, River Tarnell wrote:

> Hi,
>
> So, this is not an announcement of any intent to change anything, I just
> want to get an idea of how people feel about two things we could,
> perhaps, change in the future:
>
> 1. ZWS to Apache on the web server
> 2. Solaris to FreeBSD on login servers
>
> #2 depends on #1, so it seems sensible to discuss both together.
>
> I don't have any strong opinion about either of these myself, but I
> would like to hear what users think.
>
> ZWS to Apache:
>
> I know it annoyed people when we moved from Apache to ZWS initially,
> because rewrite rules had to be redone, some .htaccess stuff stopped
> working, etc.  At the time we were using mod_suphp for per-user (suexec)
> PHP scripts, and it was extremely inefficient; the system spent most of
> its time doing nothing.  ZWS allowed us to fix the problem cheaply (no
> new hardware required), and I think for most users it worked just the
> same.
>
> Since then, two things have changed: firstly, Apache with mod_fcgid now
> has decent FastCGI support, and with only a little work could be made to
> support suexec PHP as well.  Secondly, ZWS is now in maintenance, and
> won't see any further development (so it might be better to switch now,
> rather than wait until one month before ZWS support ends entirely and be
> forced to switch).
>
> With that in mind, it makes sense to consider moving back to Apache.
> The main downside is that rewrite rules would have to be converted back
> to Apache format (mod_rewrite).  OTOH, .htaccess features missing from
> ZWS would be available again (I don't know if anyone actually needs
> this, but I believe at least a few users have complained about missing
> features.)
>
> Solaris to FreeBSD:
>
> Of the two changes, I think this one would actually be the less
> disruptive.  For users, nearly everything would stay the same: we
> already provide the GNU userland ('ls', etc) by default (and would
> continue to do so) and the third-party software in /opt/ts would be
> identical, as would cronie, SGE, Perl/Python/..., etc.
>
> Software-wise, since nothing would really change, I don't see any
> particular advantages for users.  Disadvantages: 'ps -eaf' would stop
> working ;-) and anyone with locally-compiled software (C/C++, or XS Perl
> modules, etc.) would need to recompile them.
>
> For us (admins), the main advantage is reduced maintenance overhead:
> FreeBSD releases a new minor version about once a year, and supports
> each for 2 years; each release branch only gets very infrequent updates
> for security or errata.  In comparison, there is a new Solaris update
> every 6 months, and during yesterday's maintenance we installed 358 (!)
> separate patches.  Oracle doesn't provide a security-updates-only
> release, and it's difficult to mix-and-match patches (e.g. to only get
> security patches).
>
> This doesn't directly affect users, but fewer OS changes should lead to
> less lengthy / disruptive maintenance and less frequent reboots.  OTOH,
> I don't know if this has a noticeable impact on users at the moment, and
> the previous maintenance was the first for ~170 days.
>
> - river.

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette


_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: operating systems and web servers

Jeremy Baron

(Take with some grains of salt: never used FreeBSD or Solaris shells myself and am not a ts-admin. Also, this is a cell phone and I shouldn't even be awake now!)

Also, what do you lose with FreeBSD?

so far I can think of:
* zfs dedupe
* there's no longer a single "goto" company to get support from (guessing...) so you lose that annual cost but it may be non trivial to find the right hacker to find/fix a problem in an emergency. of course much of that support is likely to come for free (a guess)
* you're still a different platform from the rest of wikimedia (basically the foundation) and so lose economy of scale
* you mentioned SGE would still be available but it may be unmaintained (last I checked its enwp article, it said oracle was close sourcing it) just to keep in mind, I wouldn't stay just for SGE

things you lose with linux vs. FreeBSD:
* zfs is gone and (assuming lvm2 + traditional RAID vs. zfs) generally thin storage provisioning and snapshotting is much more limited and wasteful and fragile/more room for error. also most storage expansions will end up w/ transition periods that have *no* redundancy.
* zones/jails

I haven't kept up with the status in the last ~8 months but debian-kfreebsd may have matured enough to warrant a look. (See #debian-kbsd on oftc)

-Jeremy

On Jun 7, 2011 2:58 AM, "Daniel Kinzler" <[hidden email]> wrote:

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: operating systems and web servers

River Tarnell-7
In reply to this post by Daniel Kinzler
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Daniel Kinzler:
> why FreeBSD and not Linux?

Why Linux and not FreeBSD, or Solaris, or OpenServer, or MP-RAS, or ...?  
I've asked this many times in the past (usually when someone says "I
want Linux") and never had a real answer that I can remember.[0]

The way I see it, choice of operating system has very little impact on
users because users don't, generally, interact with the operating system.  
There are a few specific cases which make a difference (I can think of
cron, 'ps' and top/prstat), but apart from that, the software users
actually use is the stuff in /opt/ts, which is independent of operating
system.[1]

For servers other than login servers (like databases) there are
additional considerations, since those make a lot more use of OS
features, like storage management.  I've previously considered and
rejected FreeBSD for use as a database server (in favour of staying with
Solaris + VxVM).  

To me, FreeBSD seems to be an ideal candidate for a platform to layer
/opt/ts on top of: it provides a base operating system which is
reliable, has a good feature set in areas which only the OS can provide
(e.g. DTrace, ZFS, auditing), and it's simple to install and maintain.

        - river.

[0] I know some people are concerned about use of proprietary software
on the Toolserver, but that's a separate issue, and in any case FreeBSD
and Linux are equally open source.

[1] This assumes that all the software we provide works on FreeBSD, of
course.  I haven't verified this, because we provide ~650 packages and
it would take many hours, so there's no point doing it unless we
actually decide to change OS.  However, I don't foresee any problems.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (SunOS)

iEYEARECAAYFAk3t5ikACgkQIXd7fCuc5vLokACfUbRk0fIc6yU6tdXaS+oTr26/
0G8AoMGuJ149lizDh9KMO426taRJVDr8
=Hom2
-----END PGP SIGNATURE-----

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: operating systems and web servers

River Tarnell-7
In reply to this post by Jeremy Baron
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeremy Baron:
> Also, what do you lose with FreeBSD?
 
> so far I can think of:
> * zfs dedupe

ZFS version 28, including dedup, was integrated into FreeBSD head
(9-CURRENT) in February.  I imagine it will make it back into 8.x
eventually, but although dedup is nice on paper, I'm not sure it's
actually very useful. You need so much memory to cache the DDT (since
putting it on disk kills performance) that it's usually cheaper to just
buy more storage.

It's also unlikely that we would ever migrate the /home NFS server to
FreeBSD (at least in the near-term future), since it lacks the HA
clustering and volume management that Solaris has.  (This is an example
of an area where Linux *does* have an advantage over FreeBSD.)

It's *also* unlikely that we would ever put /home on ZFS, whatever OS we
used.  I don't mind ZFS for root, and I use it at home (on both FreeBSD
and Solaris) without problems, but we've had some unfortunate issues
with it at the TS that have made me a bit wary.

> * there's no longer a single "goto" company to get support from
> (guessing...) so you lose that annual cost but it may be non trivial to find
> the right hacker to find/fix a problem in an emergency. of course much of
> that support is likely to come for free (a guess)

This is true, and is something that came up the last time we discussed
this internally.  I'm not too worried about this; the login server
environment doesn't really tax the OS, and I can't think of many (any?)
problems we've had there that would require vendor support.  

There are places like iXsystems <http://www.ixsystems.com/bsdsupport>
which offer third-party support for FreeBSD.  I don't have any
experience with that, or any idea how it compares price-wise.

> * you're still a different platform from the rest of wikimedia (basically
> the foundation) and so lose economy of scale

Not sure what economies we could gain here.  The Toolserver is
completely separate from Wikimedia's infrastructure (except that we use
the same colo), and both of us prefer it that way.  IOW, even if we
used Linux, TS systems could not become just another Wikimedia server.

> * you mentioned SGE would still be available but it may be unmaintained
> (last I checked its enwp article, it said oracle was close sourcing it) just
> to keep in mind, I wouldn't stay just for SGE

SGE is dead on every platform, including Solaris.  The replacement is
either Oracle Grid Engine (which costs money), or one of the various
forks of the open-source SGE (which should work on both FreeBSD and
Solaris).  We're still using SGE because I want to see which (if any)
fork gains acceptable before switching.

> things you lose with linux vs. FreeBSD:
> * zfs is gone and (assuming lvm2 + traditional RAID vs. zfs) generally thin
> storage provisioning and snapshotting is much more limited and wasteful and
> fragile/more room for error. also most storage expansions will end up w/
> transition periods that have *no* redundancy.

ZFS snapshots are nice, but OTOH ZFS can't remove storage from a pool or
do any kind of online relayout, both of which are standard features in
VxVM (and I assume in Linux LVM as well, although I have no experience
of it.)

In any case we don't store data on login servers.

> * zones/jails

Linux has lxc and OpenVZ, which are both very similar to zones, as well
as Xen or KVM for full virtualisation.

> I haven't kept up with the status in the last ~8 months but debian-kfreebsd
> may have matured enough to warrant a look. (See #debian-kbsd on oftc)

Never really saw the point of that.  If you want Linux, use Linux.  (I
guess at the moment it's the only way for Linux users to get a useful
ZFS?)

        - river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (SunOS)

iEYEARECAAYFAk3t7cAACgkQIXd7fCuc5vIBUACfcCivOSjgrnbZRSRu3W7lc+Bq
TpgAn0xAsVMmFkCnaUQTx165O6nZKyRs
=d221
-----END PGP SIGNATURE-----

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: operating systems and web servers

Marc-Andre
In reply to this post by River Tarnell-7
On 07/06/2011 4:49 AM, River Tarnell wrote:
>
> Why Linux and not FreeBSD, or Solaris, or OpenServer, or MP-RAS, or ...?
> I've asked this many times in the past (usually when someone says "I
> want Linux") and never had a real answer that I can remember.[0]

Availability of expertise.

Just here at Ubisoft, for instance, there are about 25 Linux sysadmins
(ranging from intermediate to guru-level).  Of all of us, there are only
five that have Solaris or *BSD experience, and only *two* who have both.

Whatever caused this originally, Linux ended up with a much bigger
mindshare than the BSD did.  You'll find more users and sysadmins with
CentOS or Ubuntu experience than all other unices combined once we old
farts start retiring.

-- Coren / Marc


_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: operating systems and web servers

Aryeh Gregor
In reply to this post by River Tarnell-7
On Tue, Jun 7, 2011 at 5:22 AM, River Tarnell
<[hidden email]> wrote:
> ZFS snapshots are nice, but OTOH ZFS can't remove storage from a pool or
> do any kind of online relayout, both of which are standard features in
> VxVM (and I assume in Linux LVM as well, although I have no experience
> of it.)

Linux LVM can do anything you want online, in my experience: grow and
shrink logical volumes, move logical volumes between physical volumes
within the same volume group, and add or remove physical volumes, for
instance.  I've found that moving physical volumes is much slower than
it should be, though, and I've had bad experiences once or twice
moving a PV that's being actively used (system hang, but no data loss
and the operation continued automatically after power-cycling and
completed successfully).

md isn't quite so good about what it can do online, but it's getting
better recently.  It can add or remove devices from RAID1/5/6 online,
and even change the RAID level online sometimes.  If it doesn't do
what you want, and you have enough storage attached, you can always
create the new md device and move the old one's contents to it online
using LVM, possibly leaving one or both RAID arrays degraded while you
do so to get the extra space necessary.

ext4 can grow online if you grow the device it's sitting on, but it
can't shrink online, and (AFAIK) nor can any other high-profile Linux
filesystem except btrfs.

So in practice, on Linux you can do all your I/O-related stuff online
except shrinking filesystems.  This is usually enough, but the
inability to shrink filesystems online can occasionally be a pain.
btrfs is slated to fix this, in addition to solving world hunger and
curing cancer.

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette