podman provides an integrated auto-update mechanism, which I recently applied to all of my containers running on feldspaten. The mechanism is nice, but at first appears a bit counter-intuitive. In this post ’m trying to explain how you can use the mechanism and what to watch out for. I’ll start with the trap I think most people fall for.
# auto-update is natural but not intuitive
Most people’s path to containers starts with docker, which by itself is an amazing tool. I see podman as an improvement/modernization step of docker because it can run daemonless, rootless and thereby integrates well with systemd. The last part is in my experience a stumbling block, most people fall over at first.
When you deply your applications with docker the typical approach is to use Docker Compose. This allows for easy deployment of your application, consisting of multiple containers and removes the necessity of writing long uncomprehensible command line arguments with tons of -v, -e parameters. This looks honestly ugly and Docker-Compose does a good job in making this building block more accessible.
The consequence of Docker Compose is that most people use podman containers the same way as they use docker containers. You first create the container, and then you figure a way out, how to restart the container on every reboot. And this approach does not work with podman auto-update, because it requires this process to be upside-down … Wait upside-down? … What do I mean with that?
The canonical way of starting podman containers at boottime is the creation of custom systemd units for them. This is cool and allows to have daemonless, independent containers running. podman itself provides a handy way of creating those system units, e.g. here for a new nginx container:
# podman create --name nginx -p 80:80 -v /var/www:/usr/share/nginx/html docker.io/library/nginx:latest
# podman generate systemd nginx
This generates the following output:
# container-c69183be0ce6fd025f04f34ab8e7f8c1abadcd3fa901b116489191df022b480c.service
# autogenerated by Podman 3.4.4
# Sat Jul 9 10:45:47 CEST 2022
[Unit]
Description=Podman container-c69183be0ce6fd025f04f34ab8e7f8c1abadcd3fa901b116489191df022b480c.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/var/run/containers/storage
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman start c69183be0ce6fd025f04f34ab8e7f8c1abadcd3fa901b116489191df022b480c
ExecStop=/usr/bin/podman stop -t 10 c69183be0ce6fd025f04f34ab8e7f8c1abadcd3fa901b116489191df022b480c
ExecStopPost=/usr/bin/podman stop -t 10 c69183be0ce6fd025f04f34ab8e7f8c1abadcd3fa901b116489191df022b480c
PIDFile=/var/run/containers/storage/btrfs-containers/c69183be0ce6fd025f04f34ab8e7f8c1abadcd3fa901b116489191df022b480c/userdata/conmon.pid
Type=forking
[Install]
WantedBy=default.target
Now you can still tune this unit, perhaps call the container by name instead by it’s id (using the -n parameter) but ultimately this ends up in /etc/systemd/system and you can then enable this system unit - voila! - Your container is up and running as a proper systemd unit at startup. Nice.
Now comes the problem: a necessary requirement for podman auto-update is that systemd units CREATE the podman container at startup. From the documentation page:
[…] The systemd units are expected to be generated with podman-generate-systemd –new, or similar units that create new containers in order to run the updated images. Systemd units that start and stop a container cannot run a new image.
This means, that the typical process of how people approach containers would not work. Most people first create the containers and then try to make the container restart and then want them to also auto-update. podman auto-update however requires the user to think the other way around - you need to make a systemd unit which creates the container for you at startup. This is the part that I believe is counter-intuitive at first, although if you think from the systems perspective is is the more natural process flow.
Enough talk, let’s do this.
# Create a proper systemd unit for podman with auto-update
So, in order for podman auto-update to work, you will need two things:
- a systemd unit that creates the container
- label the container with
io.containers.autoupdate=image
A good template for the former can be generated from podman itself:
podman generate systemd --new nginx
# container-c69183be0ce6fd025f04f34ab8e7f8c1abadcd3fa901b116489191df022b480c.service
# autogenerated by Podman 3.4.4
# Sat Jul 9 10:48:25 CEST 2022
[Unit]
Description=Podman container-c69183be0ce6fd025f04f34ab8e7f8c1abadcd3fa901b116489191df022b480c.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --cgroups=no-conmon --rm --sdnotify=conmon -d --replace --name nginx -p 80:80 -v /var/www:/usr/share/nginx/html docker.io/library/nginx:latest
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=default.target
CAVE: This does not hold yet the --label io.containers.autoupdate=image argument that you will need to add to ExecStart (remember to add arguments before the image name). For your convenience I provide the modified systemd unit as template here:
The ExecStart is the magical part. Edit this to your needs, add the necessary environment variables and volumes there and you’re good. The systemd unit will create a new podman container with your configuration at each startup, and the PODMAN_SYSTEMD_UNIT variable allows podman auto-update to figure out, which systemd unit is associated to which container. This is necessary to pull the updated image and then restart the systemd unit.
The neat part is, that once you have created your systemd unit, you can store this elsewhere and use this as your deployment recipe for your podman containers. On a fresh machine, you can just dump them to /etc/systemd/system/, do a systemd daemon-reload, enable/start the service and you’re done. podman will pull the images at startup if they are not present, so the only thing you need to care about is the systemd unit. This is awesome, because you only have to care about one piece for the system configuration of your containers.
After setting up your container, you can simply run podman auto-update to automatically update your containers with new versions of the underlying image.
# podman auto-update
UNIT CONTAINER IMAGE POLICY UPDATED
podman-nginx.service 8f7bdccf5576 (nginx) docker.io/library/nginx:latest registry false
This checks if a new image is available, and if so, pulls the new image and restarts the associated systemd unit.
The cherry on top is to enable the podman-auto-update.timer unit, so this happens automatically once per day.
systemctl enable --now podman-auto-update.timer
And that’s it! It is a little bit of a configuration effort but once it’s in place, podman auto-update will automagically update all of your containers once new images are available. After you automated this one, there is nothing that holds you back from that 4-week camping trip to Scandinavia, you always dreamed about. At least the excuse that you need to update your containers is not valid anymore ;-)
Update 2022-02-07: This post made it in the top-10 of Hacker News today. Whoohoo!!