podman - "Found incomplete layer" error

Scroll down to Solution if you found this page via Google and are in need for help.

In the process of migrating a couple of services to MicroOS I somehow managed it to completely jam my podman container engine:

# podman image ls
WARN[0000] Found incomplete layer "236fcd368394d7094f40012a131c301d615722e60b25cb459efa229a7242041b", deleting it 
Error: stat /var/lib/containers/storage/btrfs/subvolumes/236fcd368394d7094f40012a131c301d615722e60b25cb459efa229a7242041b: no such file or directory

Basically every podman command resulted in this error, even listing the images was not possible. sigh.
A quick Google search didn’t reveal anything helpful, so I did what every random dude on the internet does next: I asked Reddit for help.

But I was so fortunate that this seem to be one of those issues that are rare enough that nobody has encountered it before. another sigh

Ultimately I could solve it myself by just removing the containers directory (/var/lib/containers). Quick surgical operations, let’s just dump the broken stuff and let podman recreate it with new shiny configuration and cache files, fresh from the upstream repositories.

Solution

In short: I just removed the whole /var/lib/containers directory.

CAVE: This will remove all of your container volumes (unless stored elsewhere). Don’t do this if you have volume data, that you want to keep! You have been warned.

CREATE A BACKUP BEFORE PROCEEDING - All your container volumes are lost irrecoverably. Don’t trust a random dude from the internet with advice on your data. Have a backup.

WARNING: All your container volumes in /var/lib/containers will be destroyed!
If you don't have all volumes on custom locations, you might loose your data. Have a backup!

# umount /var/lib/containers/storage/btrfs
# rm -rf /var/lib/containers
# mkdir -p /var/lib/containers/storage/btrfs

Done. I restarted my podman systemd units, and they just pulled a new image, mounted the data volumes from /srv and I had everything back online in no time. Pew, dodged a bullet.

How to get there

By grepping for the layer 236fcd368394d7094f40012a131c301d615722e60b25cb459efa229a7242041b I figured, that the layer and container information are stored in two json files:

  • /var/lib/containers/storage/btrfs-containers/containers.json
  • /var/lib/containers/storage/btrfs-layers/layers.json

Since all of my container data is stored in /srv, I could risk an attempt to just remove the directory and start with an empty podman cache/directory. So I created a VM snapshot and then went ahead to just kill the directory.

It worked nicely, except for the /var/lib/containers/storage/btrfs being busy, but a simple unmount and another rm -r later the directory was gone. I re-created the /var/lib/containers/storage/btrfs directory in case podman expects it to be there when creating subvolume mounts and then started the systemd services again.

What was the culprit?

I’m not sure, the issue happens while upgrading the Hypervisor that involved a couple of reboots. My working hypothesis is that podman got interrupted while doing a image pull in such a way, that the json file was out of sync with the btrfs filesystem, and this made podman stumble.

I’m not certain enough to file a bug for it but could resolve the issue, which for me at this point is more important, since I want to continue with the ongoing migration process.