How to upgrade from Ubuntu 20.04 LTS to Ubuntu 22.04 LTS

We're in the process of upgrading some of our older uptime satellite servers that power our Uptime Monitoring from Ubuntu 20.04 to Ubuntu 22.04 LTS. This is mostly a smooth process, but in case anyone's ever in a similar position, these are the steps we're taking to complete the upgrades.

We should note: there's a lot of ways to upgrade servers, and the safest approach is to setup a brand new machine, move over your data, test things & then migrate your DNS records to the new server. Our cloud providers however do not allow the existing public IP addresses to move from one server to another. Since a lot of clients have whitelisted our IPs to get (unrestricted/un-ratelimited) access, we have to preserve them. For this reason, we're preferring in-place upgrades, where the IPs stay the same.

Backup the server first

A lot of things can go wrong with in-place upgrades obviously, so it's best to be safe. You'll be installing a new Linux kernel, replace just about every package on the system (including glibc and openssl), and if anything goes wrong, your machine won't boot.

Ensure you have a back-up or a snapshot in place that you can restore from. This is also a good opportunity to test some of your disaster recovery practices, since you'll be taking snapshots & rolling back to them.

Updates first

I always like to ensure that the machine we'll be upgrading is in the best state possible, before the update. That's why I'll update & reboot the machine first, to get a fresh system.

$ apt update
$ apt upgrade
$ reboot

After that, you should be running the latest Kernel on your Ubuntu 20.04 LTS.

Perform the upgrade to Ubuntu 22.04 LTS

This is the type of step that is safest to perform through a console screen or VNC display. Ideally, you head over to your cloud provider of choice and open up the console there and login to the system as the local root user. This way, even if your SSH session(s) disconnect, you still have server access.

Because this process went incredibly smooth with our servers (and we have the fallbacks in place of snapshots/backups), we opted for the SSH approach instead.

Step 1: open a screen/tmux via SSH

Your SSH session might disconnect throughout the upgrades, so it's safe to start the process in a separate shell that you can (try to) reconnect to later, should it disconnect.

$ screen -S ubuntu_grade

You can detach yourself from the shell by hitting CTRL+A and then D.

Step 2: start the upgrade

In your shell, type:

$ cat /etc/issue
Ubuntu 20.04.6 LTS

This will confirm you're on Ubuntu 20.04 LTS.

Next, start the upgrade:

$ do-release-upgrade

In this process, you'll be prompted to make several decisions. I'll mark some of these below with a little explainer to each.

Third party sources disabled

Some third party entries in your sources.list were disabled. You can
re-enable them after the upgrade with the 'software-properties' tool
or your package manager.

To continue please press [ENTER]

You likely have 3rd party repos (ie: to get the latest PHP or Python versions). To avoid any interference of packages, those specific repos will be disabled during the upgrade. This is fine.

You might, as one of the first steps, encounter this message:

Invalid package information

After updating your package information, the essential package
'ubuntu-minimal' could not be located. This may be because you have
no official mirrors listed in your software sources, or because of
excessive load on the mirror you are using. See /etc/apt/sources.list
for the current list of configured software sources.
In the case of an overloaded mirror, you may want to try the upgrade
again later.

This happened to us for the Digital Ocean VMs. I believe it's because Digital Ocean includes some of their own reposities with their mirrors in the distributions, and by default those get disabled during the upgrade. To avoid that, restart the command with:

$ RELEASE_UPGRADER_ALLOW_THIRD_PARTY=1 do-release-upgrade

Once done, the upgrade will restart.

14 packages are going to be removed. 145 new packages are going to be
installed. 893 packages are going to be upgraded.

You have to download a total of 826 M. This download will take about
2 minutes with your connection.

Installing the upgrade can take several hours. Once the download has
finished, the process cannot be canceled.

 Continue [yN]  Details [d]

Obviously, we'll install new packages, so hit y to continue.

 Restart services during package upgrades without asking?                                                                                   
     <Yes>     <no>

In our case, each server we're upgrading gets taken out of rotation temporarily for us to perform the maintenance on. We're fine having the upgrader auto-restart services when needed, so we choose <Yes>.

Throughout the process you're going to get a lot of questions like these;

Configuration file '/etc/crontab'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** crontab (Y/I/N/O/D/Z) [default=N] ?

You made modifications to a file that the system upgrade would like to overwrite. These have to be decided upon on a case-by-case basis. After all, you know your system best. In the above example, we have custom cronjobs per server, so we'll choose N (the default value) to keep our version of the file.

Whenever possible, I'll opt for answer I where the default package version can get used, but you need to know the scenario's in which you can use that.

One of those scenario's for instance is sysctl.conf;

Configuration file '/etc/sysctl.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** sysctl.conf (Y/I/N/O/D/Z) [default=N] ?

If you've not made any changes there (and it's unlikely you did), best to pick option I here for the package defaults. This is the file that is used to override default kernel parameter values. And since you'll be upgrading several kernel versions higher, I find it safest to take whatever the package maintainer opted to include in there. They probably know kernel tuning more than I do.

Note: some of these prompts will be pure text/cli, some will be in an ncurses-like UI. The UI might be different, but the questions/answers and logic to deal with them are the same in all scenario's.

System upgrade is complete.

Restart required

To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.

Continue [yN]

Now's the moment of truth: hit y to reboot!

Step 3: verify the upgrade

Once the server has rebooted, log in and verify the upgrade is completed:

$ cat /etc/issue
Ubuntu 22.04.4 LTS

Step 4: cleanup obsolete packages

There might still be a few packages on the system that you can now cleanup.

$ apt autoremove

This will prompt you to uninstall packages that seem obsolete, but the decision is ultimately yours.

Have a look at the installed packages and see if any of these are no longer needed (think: old PHP versions).

$ apt list --installed

You're good!

If all went well, you've now upgraded your machine successfully to the latest LTS version (at time of writing).

Now it's up to you to validate that your application(s) are still functional.