stat1002 and stat1003 deprecated. Please use new stat boxes

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

stat1002 and stat1003 deprecated. Please use new stat boxes

Andrew Otto
Hi all!

tl;dr: Stop using stat100[23] by September 1st.

We’re finally replacing stat1002 and stat1003.  These boxes are out of
warranty, and are running Ubuntu Trusty, while most of the production fleet
is already on Debian Jessie or even Debian Stretch.

stat1005 is the new stat1002 replacement.  If you have access to stat1002,
you also have access to stat1005.  I’ve copied over home directories from
stat1002.

stat1006 is the new stat1003 replacement.  If you have access to stat1003,
you also have access to stat1006.  I’ve copied over home directories from
stat1003.

I have not migrated any personal cron jobs running on stat1002 or
stat1003.  I need your help for this!

Both of these boxes are running Debian Stretch.  As such, packages that
your work depends on may have upgraded.  Please log into the new boxes and
try stuff out!  If you find anything that doesn’t work, please let me know
by commenting on https://phabricator.wikimedia.org/T152712.

Please be fully migrated to the new nodes by September 1st.  This will give
us enough time to fully decommission stat1002 and stat1003 by the end of
this quarter.

I’ve only done a single rsync of home directories.  If there is new data on
stat1002 or stat1003 that you want rsynced over, let me know on the ticket.

A few notes:
- stat1002 used to have /a.  This has been removed in favor of /srv.  /a no
longer exists.
- Home directories are now much larger.  You no longer need to create
personal directories in /srv.
- /tmp is still small, so please be careful.  If you are running long jobs
that generate temporary data, please have those jobs write into your home
directory, rather than /tmp.
- We might implement user home directory quotas in the future.

Thanks all!  I’ll send another email in about a months time to remind you
of the impending deadline of Sept 1.

-Andrew Otto
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: stat1002 and stat1003 deprecated. Please use new stat boxes

Tilman Bayer
Thanks Andrew! Having wiki documentation pages for each of the new servers
would be great (like we currently have
https://wikitech.wikimedia.org/wiki/Stat1003 and
https://wikitech.wikimedia.org/wiki/Stat1002 ). Also, I'm not sure if/where
their SSH fingerprints have been published already (cf.
https://phabricator.wikimedia.org/T162972 ).

On Tue, Jul 18, 2017 at 10:31 AM, Andrew Otto <[hidden email]> wrote:

> Hi all!
>
> tl;dr: Stop using stat100[23] by September 1st.
>
> We’re finally replacing stat1002 and stat1003.  These boxes are out of
> warranty, and are running Ubuntu Trusty, while most of the production fleet
> is already on Debian Jessie or even Debian Stretch.
>
> stat1005 is the new stat1002 replacement.  If you have access to stat1002,
> you also have access to stat1005.  I’ve copied over home directories from
> stat1002.
>
> stat1006 is the new stat1003 replacement.  If you have access to stat1003,
> you also have access to stat1006.  I’ve copied over home directories from
> stat1003.
>
> I have not migrated any personal cron jobs running on stat1002 or
> stat1003.  I need your help for this!
>
> Both of these boxes are running Debian Stretch.  As such, packages that
> your work depends on may have upgraded.  Please log into the new boxes and
> try stuff out!  If you find anything that doesn’t work, please let me know
> by commenting on https://phabricator.wikimedia.org/T152712.
>
> Please be fully migrated to the new nodes by September 1st.  This will
> give us enough time to fully decommission stat1002 and stat1003 by the end
> of this quarter.
>
> I’ve only done a single rsync of home directories.  If there is new data
> on stat1002 or stat1003 that you want rsynced over, let me know on the
> ticket.
>
> A few notes:
> - stat1002 used to have /a.  This has been removed in favor of /srv.  /a
> no longer exists.
> - Home directories are now much larger.  You no longer need to create
> personal directories in /srv.
> - /tmp is still small, so please be careful.  If you are running long jobs
> that generate temporary data, please have those jobs write into your home
> directory, rather than /tmp.
> - We might implement user home directory quotas in the future.
>
> Thanks all!  I’ll send another email in about a months time to remind you
> of the impending deadline of Sept 1.
>
> -Andrew Otto
>
>
>


--
Tilman Bayer
Senior Analyst
Wikimedia Foundation
IRC (Freenode): HaeB
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: stat1002 and stat1003 deprecated. Please use new stat boxes

Andrew Otto
In reply to this post by Andrew Otto
> Please be fully migrated to the new nodes by September 1st.

Some other issues are accelerating this timeline for stat1002.

All /home directories from stat1002 have been synced to stat1005.  In
addition, non-productionized data that was in /a has been synced to
stat1005 in /srv/stat1002-a.

I have also stopped various user cron jobs that were accessing /a on
stat1002.  If you had something running there that isn’t puppetized, please
migrate it to stat1005 as soon as you can.

stat1002 will remain online at least until September 1st, but we need to
unmount the drive that contains both /home and /a partitions. /a and /home
on stat1002 (as they are now) will no longer be accessible early next week.

If you can’t find something on stat1005 that you were used to finding on
stat1002, please let me know and I will help you find it.

Thanks!  Apologies in advance if this inconveniences anyone.

- Andrew Otto
  Systems Engineer
  Wikimedia Foundation


On Tue, Jul 18, 2017 at 1:31 PM, Andrew Otto <[hidden email]> wrote:

> Hi all!
>
> tl;dr: Stop using stat100[23] by September 1st.
>
> We’re finally replacing stat1002 and stat1003.  These boxes are out of
> warranty, and are running Ubuntu Trusty, while most of the production fleet
> is already on Debian Jessie or even Debian Stretch.
>
> stat1005 is the new stat1002 replacement.  If you have access to stat1002,
> you also have access to stat1005.  I’ve copied over home directories from
> stat1002.
>
> stat1006 is the new stat1003 replacement.  If you have access to stat1003,
> you also have access to stat1006.  I’ve copied over home directories from
> stat1003.
>
> I have not migrated any personal cron jobs running on stat1002 or
> stat1003.  I need your help for this!
>
> Both of these boxes are running Debian Stretch.  As such, packages that
> your work depends on may have upgraded.  Please log into the new boxes and
> try stuff out!  If you find anything that doesn’t work, please let me know
> by commenting on https://phabricator.wikimedia.org/T152712.
>
> Please be fully migrated to the new nodes by September 1st.  This will
> give us enough time to fully decommission stat1002 and stat1003 by the end
> of this quarter.
>
> I’ve only done a single rsync of home directories.  If there is new data
> on stat1002 or stat1003 that you want rsynced over, let me know on the
> ticket.
>
> A few notes:
> - stat1002 used to have /a.  This has been removed in favor of /srv.  /a
> no longer exists.
> - Home directories are now much larger.  You no longer need to create
> personal directories in /srv.
> - /tmp is still small, so please be careful.  If you are running long jobs
> that generate temporary data, please have those jobs write into your home
> directory, rather than /tmp.
> - We might implement user home directory quotas in the future.
>
> Thanks all!  I’ll send another email in about a months time to remind you
> of the impending deadline of Sept 1.
>
> -Andrew Otto
>
>
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: stat1002 and stat1003 deprecated. Please use new stat boxes

Andrew Otto
> /a and /home on stat1002 (as they are now) will no longer be accessible
early next week.

Welp, it turns out we need to accelerate this too.  I’ve got to turn
stat1002 off today.  So!  Get it at while it is good! :)

I will be shutting stat1002 late this evening US east coast time.




On Thu, Jul 27, 2017 at 1:57 PM, Andrew Otto <[hidden email]> wrote:

> > Please be fully migrated to the new nodes by September 1st.
>
> Some other issues are accelerating this timeline for stat1002.
>
> All /home directories from stat1002 have been synced to stat1005.  In
> addition, non-productionized data that was in /a has been synced to
> stat1005 in /srv/stat1002-a.
>
> I have also stopped various user cron jobs that were accessing /a on
> stat1002.  If you had something running there that isn’t puppetized, please
> migrate it to stat1005 as soon as you can.
>
> stat1002 will remain online at least until September 1st, but we need to
> unmount the drive that contains both /home and /a partitions. /a and /home
> on stat1002 (as they are now) will no longer be accessible early next week.
>
> If you can’t find something on stat1005 that you were used to finding on
> stat1002, please let me know and I will help you find it.
>
> Thanks!  Apologies in advance if this inconveniences anyone.
>
> - Andrew Otto
>   Systems Engineer
>   Wikimedia Foundation
>
>
> On Tue, Jul 18, 2017 at 1:31 PM, Andrew Otto <[hidden email]> wrote:
>
>> Hi all!
>>
>> tl;dr: Stop using stat100[23] by September 1st.
>>
>> We’re finally replacing stat1002 and stat1003.  These boxes are out of
>> warranty, and are running Ubuntu Trusty, while most of the production fleet
>> is already on Debian Jessie or even Debian Stretch.
>>
>> stat1005 is the new stat1002 replacement.  If you have access to
>> stat1002, you also have access to stat1005.  I’ve copied over home
>> directories from stat1002.
>>
>> stat1006 is the new stat1003 replacement.  If you have access to
>> stat1003, you also have access to stat1006.  I’ve copied over home
>> directories from stat1003.
>>
>> I have not migrated any personal cron jobs running on stat1002 or
>> stat1003.  I need your help for this!
>>
>> Both of these boxes are running Debian Stretch.  As such, packages that
>> your work depends on may have upgraded.  Please log into the new boxes and
>> try stuff out!  If you find anything that doesn’t work, please let me know
>> by commenting on https://phabricator.wikimedia.org/T152712.
>>
>> Please be fully migrated to the new nodes by September 1st.  This will
>> give us enough time to fully decommission stat1002 and stat1003 by the end
>> of this quarter.
>>
>> I’ve only done a single rsync of home directories.  If there is new data
>> on stat1002 or stat1003 that you want rsynced over, let me know on the
>> ticket.
>>
>> A few notes:
>> - stat1002 used to have /a.  This has been removed in favor of /srv.  /a
>> no longer exists.
>> - Home directories are now much larger.  You no longer need to create
>> personal directories in /srv.
>> - /tmp is still small, so please be careful.  If you are running long
>> jobs that generate temporary data, please have those jobs write into your
>> home directory, rather than /tmp.
>> - We might implement user home directory quotas in the future.
>>
>> Thanks all!  I’ll send another email in about a months time to remind you
>> of the impending deadline of Sept 1.
>>
>> -Andrew Otto
>>
>>
>>
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: stat1002 and stat1003 deprecated. Please use new stat boxes

Andrew Otto
This is done!  stat1002 is offline.

Thanks all!

On Thu, Jul 27, 2017 at 5:27 PM, Andrew Otto <[hidden email]> wrote:

> > /a and /home on stat1002 (as they are now) will no longer be accessible
> early next week.
>
> Welp, it turns out we need to accelerate this too.  I’ve got to turn
> stat1002 off today.  So!  Get it at while it is good! :)
>
> I will be shutting stat1002 late this evening US east coast time.
>
>
>
>
> On Thu, Jul 27, 2017 at 1:57 PM, Andrew Otto <[hidden email]> wrote:
>
>> > Please be fully migrated to the new nodes by September 1st.
>>
>> Some other issues are accelerating this timeline for stat1002.
>>
>> All /home directories from stat1002 have been synced to stat1005.  In
>> addition, non-productionized data that was in /a has been synced to
>> stat1005 in /srv/stat1002-a.
>>
>> I have also stopped various user cron jobs that were accessing /a on
>> stat1002.  If you had something running there that isn’t puppetized, please
>> migrate it to stat1005 as soon as you can.
>>
>> stat1002 will remain online at least until September 1st, but we need to
>> unmount the drive that contains both /home and /a partitions. /a and /home
>> on stat1002 (as they are now) will no longer be accessible early next week.
>>
>> If you can’t find something on stat1005 that you were used to finding on
>> stat1002, please let me know and I will help you find it.
>>
>> Thanks!  Apologies in advance if this inconveniences anyone.
>>
>> - Andrew Otto
>>   Systems Engineer
>>   Wikimedia Foundation
>>
>>
>> On Tue, Jul 18, 2017 at 1:31 PM, Andrew Otto <[hidden email]> wrote:
>>
>>> Hi all!
>>>
>>> tl;dr: Stop using stat100[23] by September 1st.
>>>
>>> We’re finally replacing stat1002 and stat1003.  These boxes are out of
>>> warranty, and are running Ubuntu Trusty, while most of the production fleet
>>> is already on Debian Jessie or even Debian Stretch.
>>>
>>> stat1005 is the new stat1002 replacement.  If you have access to
>>> stat1002, you also have access to stat1005.  I’ve copied over home
>>> directories from stat1002.
>>>
>>> stat1006 is the new stat1003 replacement.  If you have access to
>>> stat1003, you also have access to stat1006.  I’ve copied over home
>>> directories from stat1003.
>>>
>>> I have not migrated any personal cron jobs running on stat1002 or
>>> stat1003.  I need your help for this!
>>>
>>> Both of these boxes are running Debian Stretch.  As such, packages that
>>> your work depends on may have upgraded.  Please log into the new boxes and
>>> try stuff out!  If you find anything that doesn’t work, please let me know
>>> by commenting on https://phabricator.wikimedia.org/T152712.
>>>
>>> Please be fully migrated to the new nodes by September 1st.  This will
>>> give us enough time to fully decommission stat1002 and stat1003 by the end
>>> of this quarter.
>>>
>>> I’ve only done a single rsync of home directories.  If there is new data
>>> on stat1002 or stat1003 that you want rsynced over, let me know on the
>>> ticket.
>>>
>>> A few notes:
>>> - stat1002 used to have /a.  This has been removed in favor of /srv.  /a
>>> no longer exists.
>>> - Home directories are now much larger.  You no longer need to create
>>> personal directories in /srv.
>>> - /tmp is still small, so please be careful.  If you are running long
>>> jobs that generate temporary data, please have those jobs write into your
>>> home directory, rather than /tmp.
>>> - We might implement user home directory quotas in the future.
>>>
>>> Thanks all!  I’ll send another email in about a months time to remind
>>> you of the impending deadline of Sept 1.
>>>
>>> -Andrew Otto
>>>
>>>
>>>
>>
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: stat1002 and stat1003 deprecated. Please use new stat boxes

Andrew Otto
In reply to this post by Andrew Otto
Hi all!  Just an update:  We plan to decommission stat1003 next week.

I’ll be sure to run a final home directory rsync from stat1003 -> stat1006
before we do.


On Tue, Jul 18, 2017 at 1:31 PM, Andrew Otto <[hidden email]> wrote:

> Hi all!
>
> tl;dr: Stop using stat100[23] by September 1st.
>
> We’re finally replacing stat1002 and stat1003.  These boxes are out of
> warranty, and are running Ubuntu Trusty, while most of the production fleet
> is already on Debian Jessie or even Debian Stretch.
>
> stat1005 is the new stat1002 replacement.  If you have access to stat1002,
> you also have access to stat1005.  I’ve copied over home directories from
> stat1002.
>
> stat1006 is the new stat1003 replacement.  If you have access to stat1003,
> you also have access to stat1006.  I’ve copied over home directories from
> stat1003.
>
> I have not migrated any personal cron jobs running on stat1002 or
> stat1003.  I need your help for this!
>
> Both of these boxes are running Debian Stretch.  As such, packages that
> your work depends on may have upgraded.  Please log into the new boxes and
> try stuff out!  If you find anything that doesn’t work, please let me know
> by commenting on https://phabricator.wikimedia.org/T152712.
>
> Please be fully migrated to the new nodes by September 1st.  This will
> give us enough time to fully decommission stat1002 and stat1003 by the end
> of this quarter.
>
> I’ve only done a single rsync of home directories.  If there is new data
> on stat1002 or stat1003 that you want rsynced over, let me know on the
> ticket.
>
> A few notes:
> - stat1002 used to have /a.  This has been removed in favor of /srv.  /a
> no longer exists.
> - Home directories are now much larger.  You no longer need to create
> personal directories in /srv.
> - /tmp is still small, so please be careful.  If you are running long jobs
> that generate temporary data, please have those jobs write into your home
> directory, rather than /tmp.
> - We might implement user home directory quotas in the future.
>
> Thanks all!  I’ll send another email in about a months time to remind you
> of the impending deadline of Sept 1.
>
> -Andrew Otto
>
>
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: stat1002 and stat1003 deprecated. Please use new stat boxes

Andrew Otto
Alright, I just ran a final rsync of /home on stat1003 -> stat1006.  Only
files that did not yet exist on stat1006, or have a newer modification time
on stat1003 were copied over.

We will begin the decommission of stat1003 tomorrow.



On Mon, Aug 28, 2017 at 12:02 PM, Andrew Otto <[hidden email]> wrote:

> Hi all!  Just an update:  We plan to decommission stat1003 next week.
>
> I’ll be sure to run a final home directory rsync from stat1003 -> stat1006
> before we do.
>
>
> On Tue, Jul 18, 2017 at 1:31 PM, Andrew Otto <[hidden email]> wrote:
>
>> Hi all!
>>
>> tl;dr: Stop using stat100[23] by September 1st.
>>
>> We’re finally replacing stat1002 and stat1003.  These boxes are out of
>> warranty, and are running Ubuntu Trusty, while most of the production fleet
>> is already on Debian Jessie or even Debian Stretch.
>>
>> stat1005 is the new stat1002 replacement.  If you have access to
>> stat1002, you also have access to stat1005.  I’ve copied over home
>> directories from stat1002.
>>
>> stat1006 is the new stat1003 replacement.  If you have access to
>> stat1003, you also have access to stat1006.  I’ve copied over home
>> directories from stat1003.
>>
>> I have not migrated any personal cron jobs running on stat1002 or
>> stat1003.  I need your help for this!
>>
>> Both of these boxes are running Debian Stretch.  As such, packages that
>> your work depends on may have upgraded.  Please log into the new boxes and
>> try stuff out!  If you find anything that doesn’t work, please let me know
>> by commenting on https://phabricator.wikimedia.org/T152712.
>>
>> Please be fully migrated to the new nodes by September 1st.  This will
>> give us enough time to fully decommission stat1002 and stat1003 by the end
>> of this quarter.
>>
>> I’ve only done a single rsync of home directories.  If there is new data
>> on stat1002 or stat1003 that you want rsynced over, let me know on the
>> ticket.
>>
>> A few notes:
>> - stat1002 used to have /a.  This has been removed in favor of /srv.  /a
>> no longer exists.
>> - Home directories are now much larger.  You no longer need to create
>> personal directories in /srv.
>> - /tmp is still small, so please be careful.  If you are running long
>> jobs that generate temporary data, please have those jobs write into your
>> home directory, rather than /tmp.
>> - We might implement user home directory quotas in the future.
>>
>> Thanks all!  I’ll send another email in about a months time to remind you
>> of the impending deadline of Sept 1.
>>
>> -Andrew Otto
>>
>>
>>
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l