What is it

You want to upgrade your Project Server 4. This consists of two phases:

• upgrade of the Debian Linux operating system and the PS4 base software
• upgrade of all your stacks

This upgrades the operating system within its major version. If you want to move to the next major version (a "Distribution Upgrade") see specific instructions in the HowTo section.

Note: There is no need to update both OS and stacks at the same time. While it is always recommended to keep your software up-to-date, stacks and OS version are only loosely connected to each other. Specifically this means that you can run different OS levels in your stacks and can also have a different OS level on your host machine. This is one of the promises of container technology.

Prerequisites

Have a Backup available

Please make a backup of your current system state especually if your system has some "production" character, i.e. you have to have it up and running again after a short downtime period. Every PS4 installation is unique in the set of stacks and their concrete configuration; so no one can foresee whether things will run smooth of whether issues turn up.

In cloud environments the usual cloud snapshot feature is a great way for such a task because

• both backup and restore is pretty fast (especially as we often deal with tens or even hundreds of gigabytes)
• it will uncompromisingly restore your system back to the exact same state.

Your cloud provider will charge some money for the additional storage needed by the snapshot but as you usually keep this data for a pretty short time span this should not be something which keeps you from using it.

While not absolutely necessary it is wise to do the snapshot when the VM is powered down because you thenwill have all in-memory application data settled into the files on disk.

Make sure you have access to the package servers

To accomplish the upgrade the new packages have to be downloaded from the Internet. Should your outgoing net traffic be constrained make sure that the upgrade scripts are able to access the resources they need:

All communication uses the HTTP and HTPS protocols.

Check what will happen

Especially if your last upgrade is long ago take the time to check what has changed in your applications. A docker-stack status will show you what is installed (column Installed) and what is current (column Available). The version consists of two parts separated by a dash. The first part tells you the application version, the second part is the PS4 stack version (a date). At least when you see a major release in the application version (i.e. the first of the usually two or three numbers comprising the version got incremented) you should consult the applications website to find out whether incompatible changes were made which might need a reaction by you. 

You can find out about the website URL on the corresponding stack page.

Additionally, if your application has a database container, also check its version. Mostly this will be a PostgreSQL database. A major release will always force the database to get reloaded, i.e. all your tables are unloaded to SQL statements, the database will be recreated and the data is loaded back into the tables. You have to force this step by providing the force option to the docker-stack upgrade command: It will do a backup of the stack, then upgrade and then a restore which will reload the database. This has to be done for all stacks which need such a DB upgrade.

To find out about database container run docker-ps. The last column NAME shows the container name while the second column IMAGE tells you the image name whose last part (after the colon) is the version number. The PostgreSQL database container has a NAME ending in -postgresql and the image is std-postgresql in most cases. In the following example the Keycloak stack has a DB container named iam-keycloak-postgresql whose PostgreSQL version is 12.3 (some columns removed for brevity):

Consider rehearsing the upgrade

It may pay off to first excercise the upgrade on a copy of your production system before applying it to this production system - a complete system upgrade is a non-trivial task. Again you can expect that the clood makes the task easier: Cloud providers often do not just offer a snapshot feature but also allow you to set up a second system based on the snapshot (a "clone system"). This simply means that you restore the snapshot to another VM than the one were you made it. See How to clone a PS4 server for more information.

After you have cloned the server do a quick check to make sure that application work as expected. If you detect deviations check docker ps to see whether containers are up and running or if a container is trapped in a restart loop.

Avoid getting lost or missing steps

If your PS4 application consists of more than two or three stacks it makes sense to prepare a list to check off all necessary steps. One gets lost too easy especially if things do not run as expected.

How to achieve it

Execute all commands on the PS4 host (i.e. not in one of the containers) and as user root. If you want to have an additional checkpoint on the way fell free to create additional snapshots.

1. Check for free disk space (df -h). If the disk is filled up then free some space by removing files or increasing the disk partition.
   
2. Have a look into the logs of your applications (docker logs <container-name>) to see whether errors or warnings are displayed. Sometimes applications emit such messages but they do not do any harm and have no impact on the user. If you know about it you avoid being misled by these messages if you check the logs again after the upgrade.
   
3. Upgrade operating system and PS4 base: psa upgrade. (If you are a bit into Linux and Debian: This basically does an apt update; apt upgrade).
   
4. Execute a docker-stack command, i.e. docker-stack ls to list your stacks. If there are any migration tasks in the upgraded PS4 base they would get triggered now (usually there are no such tasks).
   
5. Upgrade stacks: For each stack:

• If the stack has no database or if there is no version jump to a new major database version: docker-stack upgrade <stackname>.

• If there is a switch to a new major database version: docker-stack upgrade force <stackname>. Depending on the amount of data in the database this may take some time.

• Issue a docker ps command and check the STATUS column for this stack's container. You should see a "Up xx seconds" or "Up xx minutes" message. If the status tells that the container is restarting over and over again the application fails to start. Checking the logs (see next step) should give you more insight.

• Check the stack log for any noticeable problems. Don't get nervous in case it takes longer until the application has started - often one has no good idea of how long it takes to launch e.g. an application server or a wiki system especially if it is loaded with data and may additionally execute some automated migration tasks. If you detect any abnormalities try to resove them before proceeding. Remember that you can return to your initial state by restoring the snapshot it case you run out of time or think that problem resolution may take longer.

• Do a short functional test of the application. This could also trigger a UI-centric migration procedure which appliations sometimes offer.

Good to know

• A stack upgrade never removes your data as it is stored in Docker volumes and not in the container's file system.

• In case of issues do screenshots. This makes things easier if you need our assistance.

• The upgrade will always target the latest available stack version. If necessary you can choose a different version: In the upgrade command append the stack name with an at-sign followed by the stack version, e.g. docker-stack upgrade vcs-gitea@1.11.5-20200515. To find out about available versions look at the stack page and click on Most recent changes near the end of the page.