Linux Technology Projects

Persistent container environment with podmansh

podmansh executes a login shell within a podman container and allows you to have a containerized platform for users to login. It’s documentation relies on Quadlet provided containers, which is great in general, but comes with one caveat: Containers are ephemeral. I wanted to have a custom sandbox for my user, where I can get even local root, but because everything is contained in a container, it never touches the base system. If a user (read: me) borks their local container, the actual OS is not affected. Plus, every user can get their own container with the system of their liking. phoenix can use Tumbleweed, neo can use Fedora, grisu can use Debian, all on the same host.

In this blog post describe how I managed to get a persistent openSUSE Tumbleweed container environment for phoenix every time I ssh onto one of my servers using podmansh. There’s a TLDR and a detailled description top to bottom, so that you can follow the process and stop at any point, when you feel like you know what’s going on. podmansh is conceptionally very simple and that’s what makes it elegant, but also a bit tricky for my use case.

TL;DR

Note: phoenix is my user and it has UID 1000. I will use the Tumbleweed container from registry.opensuse.org/opensuse/tumbleweed.

Replace the configuration accordingly to your own needs. Don’t be afraid to try.

As root:

  • Create a podmansh service starting the podmansh container. This service needs to be started as user phoenix:
# /etc/systemd/user/podmansh.service
[Unit]
Description=podmansh container
After=local-fs.target
RequiresMountsFor=%t/containers

[Service]
Type=oneshot
ExecStartPre=/usr/bin/mkdir -p %h/data
ExecStartPre=-/usr/bin/podman create --name=podmansh -ti --init --user %U -w=%h --userns=keep-id -v %h/data:%h:Z -v %h/.ssh:%h/.ssh --hostname %H registry.opensuse.org/opensuse/tumbleweed sleep infinity
ExecStart=/usr/bin/podman start podmansh
RemainAfterExit=yes

[Install]
RequiredBy=default.target

Alternative: Remove the podman create ... from the unit and run this locally yourself.

  • Edit the volume mounts (-v in podman create ...) to your liking. And the container image. And whatever else you want to modify.
  • Enable the systemd linger, so that the user services are started at boot time
# loginctl enable-linger 1000

As phoenix:

  • Enable the services:
$ systemctl --user daemon-reload
$ systemctl --user enable --now podmansh.service
  • Ensure if the podmansh container is running, otherwise check why it doesn’t start (podman log podmansh might help you)
$ podman ps
CONTAINER ID  IMAGE                                             COMMAND         CREATED      STATUS      PORTS       NAMES
79e7a706c0a0  registry.opensuse.org/opensuse/tumbleweed:latest  sleep infinity  7 hours ago  Up 7 hours              podmansh
  • Modify the container to your needs (e.g. set root password, install sudo, …)
$ podman exec -ti --user=0 podmansh /bin/bash

# zypper ref && zypper -n in vim sudo && visudo

As root again (on the host, not in the container!):

  • Change the login shell to podmansh:
# chsh -s /usr/bin/podmansh phoenix

Now the next time the user phoenix connects via ssh to that machine, the user will be sent to their own podmansh container. Neat!

Top to bottom

podmansh is just a wrapper around executing podman exec -ti podmansh /bin/sh. This means upon login, podman tries to just attach (not start or run!) to the running podmansh container and throw/jail the user therein. It’s just awesome how little it needs to be functional, this is elegant by its own simplicity.

In practise this means all you need to have is a running podmansh container with /bin/sh in it. From podman 5.1 onwards you will be able to configure the container name and shell, for now it’s hardcoded but that’s in my humble opinion just a minor issue.

The documentation provides you with an example using Quadlet in /etc/containers/systemd/users/${USERID}/podmansh.container. This is also elegant in it’s own way because this will automatically spin up a new container whenever a user session is started, and also tear it down afterwards. The downside is that Quadlet only supports ephemeral containers, so you will get a new one every session and everything you did in the last session is forgotten to the streams of time.

Having an ephemeral session container has it’s advantages. But if you’re a lazy bastard as I am, then you might want to have a persistent container because it avoids me having to setup a CI somewhere that creates a container with everything I need/want to have in it. If I want to have a new package, I just install it and it will be there the next day when I login again. I could also setup a CI that builds my own container, but that appears overkill to me for this particular task. Your mileage might vary.

My goal is to have a persistent Tumbleweed container for my phoenix user, but with local sudo rights so I can install whatever I need within that container without ever coming near the actual OS. If I ever bork the container, I can just throw it away and start new. The host OS won’t be bothered with whatever unholy mess phoenix has made in his own sandbox container.

To reach that goal all we need is to make sure, that we have a podmansh container running as user phoenix when phoenix logs in. I solved this goal by creating a user systemd unit, that starts the container. This together with using a systemd linger that starts a user session at boot time works nice and you have your user container ready when you login. They will be restarted after a reboot.

So, let’s first create a user systemd service:

# /etc/systemd/user/podmansh.service
[Unit]
Description=podmansh container
After=local-fs.target
RequiresMountsFor=%t/containers

[Service]
ExecStartPre=/usr/bin/mkdir -p %h/data
ExecStartPre=-/usr/bin/podman create --name=podmansh -ti --init --user %U -w=%h --userns=keep-id -v %h/data:%h:Z -v %h/.ssh:%h/.ssh --hostname %H registry.opensuse.org/opensuse/tumbleweed sleep infinity
Type=oneshot
ExecStart=/usr/bin/podman start podmansh
RemainAfterExit=yes

[Install]
RequiredBy=default.target

The only really important part is that the container needs to have the name podmansh. Everything else you can (and should) modify to your needs. Change the mount points, the hostname, permissions, …. Whatever you like. It’s your environment.

Mind the ExecStartPre=-/usr/bin/podman create. You can leave out this command entirely and just create the container. I like this because this means the container would be automatically re-created if it doesn’t exist. The - at the beginning tells systemd, that it’s ok if this command fails - that’s necessary because the container will exist after the first run, and creating a container with a name that already exists results in an error (unless you use --replace).

If you place this as /etc/systemd/user/podmansh.service this will be available to all users. You can customize this on a per user level by putting it into /etc/systemd/user/${USERID}/podmansh.service.

I want to be able to modify the ssh keys from within the container as well, that’s why the volume mount %h/.ssh:%h/.ssh is included as well.

One should be in principle be able to mount the whole home directory directly using -v %h:%h. I decided against that, because there are certain parts like the container and systemd configuration that I want to keep apart from the container data file, in case I ever need to change that. That’s why the container home directory is actually located in ~/data. This is how I like it. You can (and should), of course, configure it however you like.

Ok, now we have the service and we can start it. Run the following commands as user phoenix:

$ systemctl --user daemon-reload
$ systemctl --user enable --now podmansh.service

And now check if the podmansh container is running:

$ podman ps
CONTAINER ID  IMAGE                                             COMMAND         CREATED      STATUS      PORTS       NAMES
79e7a706c0a0  registry.opensuse.org/opensuse/tumbleweed:latest  sleep infinity  7 hours ago  Up 7 hours              podmansh

Note: Don’t do sudo -u phoenix -s. This will not work and give you cryptic crun errors (at least it did for me). It’s best to login directly on the system or ssh to the system (without podmansh in place).

This works for now, but there’s one problem: The container won’t be started after a reboot. Only after a user session is established. To start the container at boottime, you can enable a user lingering - this establishes a session at boot time and it then starts your container. As root you can do this for user phoenix (UID 1000):

# loginctl enable-linger 1000

At this point all the pieces are in principle there and you could switch the shell to /usr/bin/podmansh (or wherever podmansh is located in your distribution). Careful however,because you might still need to do some manual configuration on the container before it’s ready for you.

I use a fresh Tumbleweed container, where there is no root password set or sudo installed. And since I configured the systemd service to drop me as user phoenix and not as root, I couldn’t run commands as root. I resolved this by dropping me a root bash and then installing and configuring sudo. Because the container is persistent, I only need to do this once.

So, on the host as user phoenix you can spawn a bash (or whatever) shell as root (--user=0) and do those final modifications:

$ podman exec -ti --user=0 podmansh /bin/bash

(now you're in the container, run the following there)

# zypper ref && zypper -n in vim sudo && visudo
...
#Defaults targetpw   # ask for the password of the target user i.e. root
#ALL   ALL=(ALL) ALL   # WARNING! Only use this together with 'Defaults targetpw'!
phoenix ALL=(ALL:ALL) NOPASSWD: ALL
...

Now we’re ready to switch the user shell to podmansh. On the host (not in the container), as root, you can switch the login shell for phoenix to use podmansh:

# chsh -s /usr/bin/podmansh phoenix

And now, every time I login onto that server as phoenix, I have my own persistent Tumbleweed container, where I can get local root privileges without risking to bork the host OS.

And I find this beautiful.

References

Credits

  • Petr Lautrbach with very helpful feedback on the podman mailing list. His answer pointed me in the right direction. Thanks!