Introduction

As a brief preface for this post as I lament about being a terrible writer in the sense that I never do write with this being my first post (although if you just want to know how I setup my server stack and how, feel free to skip to Setting up Tunnels). I’ve been home-labbing for at least 5 years at this point and I’ve never once written anything talking about it. However, I feel like I’m at the stage that it feels important to document some of my decisions and challenges I face so that, in the future, I know where I am and what lead me there.

A purpose?

I’m not particularly a fan of large companies, especially those that like to enter a space, do like kings and make themselves de-facto by providing most services for free with the express intent to get people hooked and, thus, become paying customers. With outage after outage, I found myself asking more often how it is that more people aren’t moving away. Their QA is compromised, the dashboard is falling apart and they’re committing shady practices around AI crawlers. Of course there are genuine benefits of having a service that large, but those benefits are immediately dumped by the wayside when a large number cyber attacks are performed using Cloudflare services. All this said however, I must admit that I’m hardly doing this for the reasons above; I just enjoy tinkering.

The old way

Aside from using Cloudflare for DNS, I was utilising Nginx as a reverse proxy on my home system, and also on a VPS. I like Nginx, but configuring it can be quite wordy even when using includes. I was also quite unhappy with the fact that my infrastructure was practically split in two and drastically inconsistent, needing to manage different machines depending on where a service was located, along with managing Cloudflare’s certificates for multiple domains. At the very least with Cloudflare, I was able to open the standard HTTP ports on my home network and have the firewall only allow traffic through for Cloudflare’s IP addresses.

A reverse proxy replacement

Caddy is younger than Nginx but it is relatively mature and provides a much better experience for getting started. For my purposes, it’s exactly what I need and its Caddyfile is much simpler than Nginx’s configuration. Other helpful things it provides are proxy headers, websocket upgrades and TLS management by default, the latter meaning I can switch away from application or Cloudflare managed TLS certificates and let Caddy provide those certs for me.

Structure

As I mentioned before, I have two hosts and so my goal was to have a single reverse proxy for both. In order to achieve this, I needed to connect the two machines together in some way. I’m familiar with Wireguard and have been using it to stay connected to my home network for some years now but, I felt for this, I wanted something that demands less thought. I decided Tailscale would be a good fit since it gives me the Wireguard protocol I love along with a few additional bells and whistles, and the piece of mind of not having to write several config files for each and every system… And, of course in my typical fashion, I also installed Headscale since I don’t like the idea of having an external control server for internal services.

Setting up tunnels

I must note first, the order of this post in no way represents the incoherent nature of the actual event. However, in hindsight, I can provide a clearer picture of what would have happened if I were an organised person. Before starting with Tailscale, it’s important to make sure sysctl is properly configured to forward traffic.

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

And you may need to check the status of the TUN/TAP driver depending on your system. For example, for LXCs in Proxmox (and maybe elsewhere too), you need to specify usage of the driver in the LXC configuration file.

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

Installing Headscale is a fairly straightforward process, I got it from COPR the package of which works out of the box, although you may want to set your own DNS domain using .home.arpa or .internal. Next is Tailscale where --login-server needs to be used to make it aware of the Headscale instance. I also needed --accept-routes on both machines and --advertise-routes on the home machine to allow Caddy to access services running on my home server via IP address. These settings can be changed after running tailscale up but the correct login server needs to be set initially.

Enter containers

Caddy is not difficult to work with, so naturally I don’t have much to say about the Caddyfile and its features. For most scenarios, certificates are managed automatically out of the box with letsencrypt via https and will do basically what you expect; pointing hosts to files or other servers. But since I want to run it in a container and use non-standard modules, I have to build my own image. It’s further complicated by the fact I want to change as little about the bare system as possible and I use a RHEL distro. Docker is not available by default on RHEL, instead providing Red Hat’s own Podman. Now I don’t hate Podman, I actually like the fact that it’s daemon-less, however that comes with a caveat. Since Docker uses a daemon, it can restart your containers on boot, for Podman it gets a bit more complicated. The other issue I had to consider was retaining source IP addresses on incoming connections, which would get replaced by the internal Podman network address. This problem is solved both by using Quadlets and systemd socket activation.

The Quadlets took me quite a long time to get my head around, longer than I’d like to admit, but once it’s all in place it’s not too bad. All the necessary components go into ~/.config/containers/systemd (apart from the socket which lives in ~/.config/systemd/user) and come together very similar to how a docker-compose file does, but systemd flavoured. For my configuration, I landed on a volume for each Caddy and CrowdSec to use, a shared logs folder so CrowdSec, as read-only, can read the Caddy logs to make its decisions. I also needed a network so the two could communicate and the two container files. I would also need to mount the necessary config files and provide secrets via a .env file. It is also necessary to perform loginctl enable-linger <username> in order to allow the containers to run without a login session.

For the socket, I need 80/tcp, 443/tcp and 443/udp for http3. This does mean that I need to enable unprivileged users to bind from port 80 rather than the usual 1024. Pretty easy to do by setting net.ipv4.ip_unprivileged_port_start to 80 with sysctl. If you’re worried about the implications of this, I’m not. The resulting socket file looks like this.

[Socket]
BindIPv6Only=both

ListenStream=[::]:80
ListenStream=[::]:443
ListenDatagram=[::]:443

[Install]
WantedBy=sockets.target

And caddy’s container file looks something like this

[Unit]
Description=Caddy
Wants=caddy.network caddy-config.volume caddy-data.volume caddy-logs.volume
After=caddy.socket
Requires=caddy.socket

[Service]
ExecReload=podman exec caddy /usr/bin/caddy reload --config /etc/caddy/Caddyfile

[Container]
Image=localhost/caddy:latest
ContainerName=caddy
Exec=/usr/bin/caddy run --config /etc/caddy/Caddyfile
EnvironmentFile=%h/caddy/.env
Network=caddy.network
NoNewPrivileges=true
ReadOnly=true
Volume=%h/caddy/Caddyfile:/etc/caddy/Caddyfile:Z
Volume=%h/caddy/static:/static:Z,ro
Volume=caddy-config.volume:/config:Z
Volume=caddy-data.volume:/data:Z
Volume=caddy-logs.volume:/var/log/caddy

The use of socket activation is also where the DNS provider comes into play. When Caddy tries to fire a http-01 challenge, the hostname cannot be resolved and will fail. To mitigate this, I instead use the DNS-01 challenge. There are a few security implications with this method, but if you’re not worried, neither am I. (a small note to tell you that TextEdit says: “The word “am” may not agree with the rest of the sentence” and recommends the brilliant options “is” or “are”) It’s pretty easy to do this in the global section of the Caddyfile with the cert_issuer directive or per entry if you would prefer. I also set default binds on the sockets for the various http methods.

All together I end up something like this:

{
        email {env.EMAIL}
        admin unix//run/admin.sock
        auto_https disable_redirects

        default_bind fd/4 {
                protocols h1 h2
        }

        default_bind fdgram/5 {
                protocols h3
        }

        cert_issuer acme {
                dns hetzner {env.HETZNER_API_TOKEN}
                disable_http_challenge
        }

        crowdsec {
                api_url http://crowdsec:8080
                api_key {env.CROWDSEC_API_KEY}
                ticker_interval 15s
                appsec_url http://crowdsec:7422
        }

        log {
                output file /var/log/caddy/access.log {
                        roll_size 30MiB
                        roll_keep 5
                        roll_keep_for 14d
                }
        }
}

(common) {
        header /* {
                -Server
        }
        log
}

http:// {
        import common

        bind fd/3 {
                protocols h1
        }
        redir https://{host}{uri}
}

www.noodle.moe, noodle.moe {
        import common

        route {
                crowdsec

                root /static/moe/dist
                file_server
        }

        handle_errors 404 {
                rewrite /404.html
                file_server
        }
}

www.example.com, example.com {
	import common

	route {
		crowdsec

		reverse_proxy 192.168.150.40:3000
	}
} 

Fin

After a needlessly painful two days, I have successfully made the switch away from Cloudflare and unified my server configuration. I’m still yet to explore configuring Anubis and Appsec did not work when I tried it, so that too requires investigation. For more information about the methods I used, you can find out more about socket activation here and about configuring crowdsec here. Note that if you plan to use the CrowdSec web console, don’t; it doesn’t support the Caddy bouncer and is also not very good in general.