Building the Trantor Star Bridge: FCOS, Quadlets, and Pangolin

Pangolin has been all the craze in the community recently, but I’d been avoiding it due to the need for a VPS and the potential complexity it would add. However, after peeking its documentation and realizing how affordable a small VPS is, I finally decided to jump off the deep end and give it a try.

For this project, I wanted something “different” for the base OS. With how small the VPS was, how public this was going to be, and the mainly containerized workloads I would be running on it, I decided I would tackle another technology I’d been avoiding: Fedora CoreOS. It fit the bill for everything I wanted for this project:

Immutable root filesystem
SELinux enforcing
Minimal packages
Container-first philosophy

What turned me off to it initially was its Butane/Ignition configuration system, which seemed unwieldy for configuring a whole OS. My misconception was: “I have to configure everything from storage to networking to package groups in this config file??”

While that isn’t necessarily a bad thing, especially for IaC reproducible setups, I was hoping for a more straightforward experience that focused more on containers. This ended up actually being the case, because as it turns out, FCOS handles all of that for you by default, and really the only requirement to get a running system is a user with either an SSH key or password.

This realization was an absolute game changer, because it reframed the dilemma from configuring every knob and button in Ignition, to only the customizations I needed on top. I was able to stand up a running FCOS fairly quickly this way, and iterated on my Butane configuration as I began to tweak the OS and build out the containers.

The RedHat family is pretty outspoken on using Podman for containers, so for this project, I decided to lean into that philosophy and use Quadlets to deploy Pangolin. I had used Quadlet .container files briefly in the past, but for this project, I also needed to delve into .network, .volume, and .pod files. This ended up looking like the following:

[Pod]
Network=fosrl.network
PublishPort=80:80
PublishPort=443:443
PublishPort=51820:51820/udp
UserNS=auto

fosrl.pod

[Network]
Driver=bridge

fosrl.network

[Volume]
Driver=local

pangolin-config, traefik-config, and traefik-letsencrypt .volume files

[Container]
AutoUpdate=registry
Image=docker.io/fosrl/pangolin:latest
HealthCmd="curl -f http://localhost:3001/api/v1/"
HealthInterval="3s"
HealthRetries="15"
HealthTimeout="3s"
Pod=fosrl.pod
Volume=pangolin-config.volume:/app/config

[Service]
Restart=always

[Install]
WantedBy=default.target

pangolin.container

[Unit]
Requires=pangolin.container
After=pangolin.container

[Container]
AutoUpdate=registry
AddCapability=NET_ADMIN SYS_MODULE
Exec=--reachableAt=http://localhost:3003 --generateAndSaveKeyTo=/var/config/key --remoteConfig=http://localhost:3001/api/v1/gerbil/get-config --reportBandwidthTo=http://localhost:3001/api/v1/gerbil/receive-bandwidth
Image=docker.io/fosrl/gerbil:latest
Pod=fosrl.pod
Volume=pangolin-config.volume:/var/config

[Service]
Restart=always

[Install]
WantedBy=default.target

gerbil.container

[Unit]
Requires=pangolin.container
After=pangolin.container

[Container]
AutoUpdate=registry
Environment=PORKBUN_API_KEY=<key>
Environment=PORKBUN_SECRET_API_KEY=<key>
Exec="--configFile=/etc/traefik/traefik_config.yml"
Image=docker.io/traefik:latest
Pod=fosrl.pod
Volume=traefik-config.volume:/etc/traefik:ro
Volume=traefik-letsencrypt.volume:/letsencrypt

[Service]
Restart=always

[Install]
WantedBy=default.target

traefik.container

Some notes on the install:

The .pod setup was nice to organize and orchestrate the entire Pangolin stack, but also had the added bonus of putting them all in the same network namespace, which seemed to be required for Gerbil and Traefik as per the Pangolin Docker Compose file.
I opted for persistent container volumes rather than host paths for more seamless SELinux handling.
UserNs=auto was a nice touch to further lock down the containers and volumes.

At the end of all my iteration, my Butane config grew into the following:

variant: fcos
version: 1.6.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - <public key>
    - name: runner
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: <hostname>
    - path: /etc/systemd/zram-generator.conf
      mode: 0644
      contents:
        inline: |
          # This config file enables a /dev/zram0 device with the default settings
          [zram0]
    - path: /etc/sysctl.d/90-net.ipv4.ip_unprivileged_port_start.conf
      contents:
        inline: |
          net.ipv4.ip_unprivileged_port_start = 80
    - path: /var/lib/systemd/linger/runner
      mode: 0644
    - path: /home/runner/.config/containers/systemd/fosrl.network
      mode: 0644
      contents:
        inline: |
          [Network]
          Driver=bridge
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers/systemd/fosrl.pod
      mode: 0644
      contents:
        inline: |
          [Pod]
          Network=fosrl.network
          PublishPort=80:80
          PublishPort=443:443
          PublishPort=51820:51820/udp
          UserNS=auto
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers/systemd/pangolin-config.volume
      mode: 0644
      contents:
        inline: |
          [Volume]
          Driver=local
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers/systemd/traefik-config.volume
      mode: 0644
      contents:
        inline: |
          [Volume]
          Driver=local
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers/systemd/traefik-letsencrypt.volume
      mode: 0644
      contents:
        inline: |
          [Volume]
          Driver=local
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers/systemd/pangolin.container
      mode: 0644
      contents:
        inline: |
          [Container]
          AutoUpdate=registry
          Image=docker.io/fosrl/pangolin:latest
          HealthCmd="curl -f http://localhost:3001/api/v1/"
          HealthInterval="3s"
          HealthRetries="15"
          HealthTimeout="3s"
          Pod=fosrl.pod
          Volume=pangolin-config.volume:/app/config

          [Service]
          Restart=always

          [Install]
          WantedBy=default.target
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers/systemd/gerbil.container
      mode: 0644
      contents:
        inline: |
          [Unit]
          Requires=pangolin.container
          After=pangolin.container
          
          [Container]
          AutoUpdate=registry
          AddCapability=NET_ADMIN SYS_MODULE
          Exec=--reachableAt=http://localhost:3003 --generateAndSaveKeyTo=/var/config/key --remoteConfig=http://localhost:3001/api/v1/gerbil/get-config --reportBandwidthTo=http://localhost:3001/api/v1/gerbil/receive-bandwidth
          Image=docker.io/fosrl/gerbil:latest
          Pod=fosrl.pod
          Volume=pangolin-config.volume:/var/config

          [Service]
          Restart=always

          [Install]
          WantedBy=default.target
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers/systemd/traefik.container
      mode: 0644
      contents:
        inline: |
          [Unit]
          Requires=pangolin.container
          After=pangolin.container
          
          [Container]
          AutoUpdate=registry
          Exec="--configFile=/etc/traefik/traefik_config.yml"
          Environment=PORKBUN_API_KEY=<key>
          Environment=PORKBUN_SECRET_API_KEY=<key>
          Image=docker.io/traefik:latest
          Pod=fosrl.pod
          Volume=traefik-config.volume:/etc/traefik:ro
          Volume=traefik-letsencrypt.volume:/letsencrypt

          [Service]
          Restart=always

          [Install]
          WantedBy=default.target
      user:
        name: runner
      group:
        name: runner
  directories:
    - path: /home/runner/.config
      mode: 0755
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers
      mode: 0755
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/containers/systemd
      mode: 0755
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/systemd
      mode: 0755
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/systemd/user
      mode: 0755
      user:
        name: runner
      group:
        name: runner
    - path: /home/runner/.config/systemd/user/default.target.wants
      mode: 0755
      user:
        name: runner
      group:
        name: runner

nathira-fcos.bu

I'm pretty happy with the setup, as well as the new reality of not having to open ports at home to expose resources.

After configuring/tweaking Pangolin to my liking, the next step will be to back it up...