Keeping server software up to date

keeping_server_sw_up_to_date.md

Keeping Server Software Up To Date

The Stack

1. OS
2. Managed Packages
3. Unmanaged Packages
4. Application Frameworks
5. Application Libraries
6. Application Code

From the OS up to app code, each layer that is out of date makes each higher layer harder to update. For example, if the OS is badly out of date, the package manager's repositories may not contain the latest packages. If the OS has reached EOL, the repositories may no longer be online. We encountered this recently when centos 4 reached EOL.

Set Goals

Primary goal: improve security

For the highest quality semantically versioned projects, with versions greater than or equal to 1.0.0, upgrade decisions are easy.

1. patch-level - apply immediately
   • semver: only backwards compatible bug fixes
   • these usually contain critical security fixes
2. minor-version - apply within one month
   • semver:
     • stability fixes
     • new backwards-compatible features
     • deprecations
3. major-version
   • before EOL - with or without client approval/budget
   • when requested by a developer - with client approval + budget

For known violators of semver, upgrade decisions are more difficult. Also, some projects are semantically versioned but are not of the highest quality. These projects may have a reputation for being unstable during the first one or two patch levels after a minor version release.

1. ruby
   • four level versioning: maj-min-tiny-patch
   • patch versions (fourth level) aim to be backwards compatible
   • tiny versions contain new features
2. MacOS
   • three level versioning, but fails to increment major version
   • The first one or two patch levels after a minor version release are often unstable.

Delegate Responsibility

Responsibility for upgrading app frameworks and libraries has already been delegated to the developers. Responsibility for the OS and packages has not been delegated.

We can either concentrate the responsibility to a single individual or spread it among many. If a single staffperson is interested in this role, that may be most efficient. Then again, spreading the responsibility among multiple people will broaden their skills.

Either way, it must be clear that upgrades are a primary responsibility, on par with new features and bugs. Non-billable maintenance must have the same priority as billable work, and this must be reflected in goal charts and other metrics.

Supervise

In general, little supervision is necessary, especially if a single staffperson is responsible for all OS and package upgrades. If responsibility is distributed widely among developers, oversight becomes more important.

Your ideas for upgrading software based on semver make sense in the Ruby world. I'm not sure how well they apply to managed packages. In my experience, operating systems' package managers usually don't allow you specify things like "only patch-level upgrades". They just update you to the most recent version of the package released for the OS version. However, the release policies are usually pretty conservative, and disruptive changes are often given a new package name (e.g. python-2.7 vs python-2.6), so updating managed packages usually has a pretty low risk level.

I'd like to see a broad outline of the areas we need to consider before we delve deep into any one area. Areas that I see are OS, Managed Packages, Unmanaged Packages, Application Frameworks, Application Libraries, and Application Code.