Securing the Trantor Star Bridge: Backrest and Crowdsec

In a previous post, I detailed the results of my foray into FCOS, Podman Quadlets, and Pangolin, and mentioned the next step of taking backups of the setup. In addition to that, I ended up taking the plunge into setting up Crowdsec as well; this post serves as the continuation outlining what went into that.

Backrest

With Backrest, I immediately ran into a major blocker: because the environment is rootless and running with UserNS=auto, while I could mount the different container volumes into the Backrest container, it couldn't read any of the files that weren't globally readable, which ended up being a lot of the files. I didn't want to muck around with the filesystem permissions that the applications themselves set, so I needed another way. Luckily, the restic documentation had the key:

Examples — restic 0.18.0-dev documentation

In that doc, they provide the Linux capability necessary for Backrest to read files that aren't globally readable: CAP_DAC_READ_SEARCH

I also needed SecurityLabelDisable=true for it to read files properly. After combining these elements, I reached a .container that looks something like this:

[Container]
AutoUpdate=registry
Environment=BACKREST_DATA=/app/data
Environment=BACKREST_CONFIG=/app/config/config.json
Environment=XDG_CACHE_HOME=/cache
Environment=TMPDIR=/tmp
PublishPort=127.0.0.1:9898:9898
Image=docker.io/garethgeorge/backrest:latest
Volume=backrest.volume:/app:Z,U

SecurityLabelDisable=true
AddCapability=CAP_DAC_READ_SEARCH

Volume=/host/path/to/container-vols:/bak:ro

[Service]
Restart=always

[Install]
WantedBy=default.target

backrest.container

There's a reason why Backrest is binding its web UI port to the loopback interface - and that has everything to do with Crowdsec and what needed to be done there.

Crowdsec

The actual Quadlet-ization of Crowdsec was not expressly difficult:

[Container]
AutoUpdate=registry
Environment="COLLECTIONS=crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules"
Environment=ENROLL_INSTANCE_NAME=pangolin-crowdsec
Environment=PARSERS=crowdsecurity/whitelists
Environment=ENROLL_TAGS=podman
Image=docker.io/crowdsecurity/crowdsec:latest

Pod=fosrl.pod

HealthCmd="cscli capi status"
HealthInterval="10s"
HealthRetries="15"
HealthTimeout="10s"

Label="traefik.enable=false"

Volume=crowdsec-config.volume:/etc/crowdsec:Z,U
Volume=crowdsec-db.volume:/var/lib/crowdsec/data:Z,U
Volume=traefik-logs.volume:/var/log/traefik:z,ro

Exec=-t

[Service]
Restart=always

crowdsec.container

What I would come to find, however, was that this was not enough to achieve its true/complete functionality. The reason for this is outlined in @eriksjolund's excellent doc on Podman networking:

The entry of interest is "pasta + custom network", which is how my Pod was configured - with this setup, source addresses were not being preserved, meaning Traefik's access logs were only showing internal container network addresses and not legitimate external IPs. What this meant for Crowdsec was that it was unable to discern who was actually accessing resources, and was allowing everything under its policy that internal networks are trusted. I quickly realized this after doing a test ban on my IP, only to find that I was still able to access resources.

The consensus online seemed to be that socket activation was the de facto way to deal with this conundrum. Luckily, Erik had a repo for exactly that:

Unfortunately, after following the steps to create sockets with the proper file descriptors and attaching them in a bunch of different arrangements (e.g. to the .pod, which is where network config typically takes place, as well as directly to traefik.container), I was unsuccessful in getting socket activation to work with the Pod. All browser as well as curl requests to the Pod would time out, despite the Traefik container being satisfied with the .socket's being attached to the .pod. After raising this with Erik on Discord, it would seem they reached a similar wall.

At this point, I begrudgingly opted for the Network=pasta route outlined in Erik's networking doc, which I had been trying to avoid because it meant giving up custom container networks. This was problematic for the setup at the time, because that's how Pangolin was communicating with Backrest and Pocket ID, which were on the same host. For Pocket ID, I opted to move it back inside the Newt tunnel, and for Backrest, I opted to bind its web UI port to the host loopback interface as previously outlined. To then access the Backrest UI this way, I just needed to add a little snippet to the pasta network config:

[Pod]
Network=pasta:-T,9898:9898

[...]

Snippet of Pangolin .pod

This uses --tcp-ns to map a port from the container to the host, as outlined here:

I initially had Backrest set up without authentication because it was behind Pangolin's auth; with this arrangement, however, I opted to enable Backrest's authentication to protect the locally exposed port. Funnily enough this is how I discovered that Crowdsec was finally working - after reaching Backrest via Pangolin/Traefik, and getting the password wrong multiple times, I got banned!

Now my setup not only has backups going, but is also more secure!