Scroll down to Solution if you found this page via Google and are in need for help.
In the process of migrating a couple of services to MicroOS I somehow managed it to completely jam my
podman container engine:
# podman image ls WARN Found incomplete layer "236fcd368394d7094f40012a131c301d615722e60b25cb459efa229a7242041b", deleting it Error: stat /var/lib/containers/storage/btrfs/subvolumes/236fcd368394d7094f40012a131c301d615722e60b25cb459efa229a7242041b: no such file or directory
podman command resulted in this error, even listing the images was not possible. sigh.
A quick Google search didn’t reveal anything helpful, so I did what every random dude on the internet does next: I asked Reddit for help.
But I was so fortunate that this seem to be one of those issues that are rare enough that nobody has encountered it before. another sigh
Ultimately I could solve it myself by just removing the containers directory (
/var/lib/containers). Quick surgical operations, let’s just dump the broken stuff and let
podman recreate it with new shiny configuration and cache files, fresh from the upstream repositories.
If you’re having this issue right now: remove the whole
CAVE: This will remove all of your container volumes (unless stored elsewhere). Don’t do this if you have volume data, that you want to keep! You have been warned.
CREATE A BACKUP BEFORE PROCEEDING - All your container volumes are lost irrecoverably. Don’t trust a random dude from the internet with advice on your data. Have a backup.
WARNING: All your container volumes in /var/lib/containers will be destroyed! If you don't have all volumes on custom locations, you might loose your data. Have a backup! # umount /var/lib/containers/storage/btrfs # rm -rf /var/lib/containers # mkdir -p /var/lib/containers/storage/btrfs
Done. I restarted my
podman systemd units, and they just pulled a new image, mounted the data volumes from
/srv and I had everything back online in no time. Pew, dodged a bullet.
Use the overlay storage driver to avoid the issue from arising
To avoid this issue in the first place: Use the
overlay storage driver instead of
# Default Storage Driver driver = "overlay"
At least until the issue is solved in
How to get there
By grepping for the layer
236fcd368394d7094f40012a131c301d615722e60b25cb459efa229a7242041b I figured, that the layer and container information are stored in two json files:
Since all of my container data is stored in
/srv, I could risk an attempt to just remove the directory and start with an empty podman cache/directory. So I created a VM snapshot and then went ahead to just kill the directory.
It worked nicely, except for the
/var/lib/containers/storage/btrfs being busy, but a simple unmount and another
rm -r later the directory was gone. I re-created the
/var/lib/containers/storage/btrfs directory in case
podman expects it to be there when creating subvolume mounts and then started the systemd services again.
What was the culprit?
I’m not sure, the issue happens while upgrading the Hypervisor that involved a couple of reboots. My working hypothesis is that
podman got interrupted while doing a image pull in such a way, that the
json file was out of sync with the
btrfs filesystem, and this made
I’m not certain enough to file a bug for it but could resolve the issue, which for me at this point is more important, since I want to continue with the ongoing migration process.
I filed #16882 for this issue. I was suggested to use the
overlay driver instead of
btrfs and this resolved the problem for me. No issues in over two months now.