Luka Dekanozishvili - Blog posts

DMARC is not that hard

2026-09-27T00:00:00+00:00

What's DMARC?

Domain based Message Authentication, Reporting or Conformance, or DMARC for short, tells receiving mailservers how to handle mail that fails SPF (who's allowed to send mail on behalf of this email?), DKIM (did this mail really originate from the domain?), or both, and can provide reporting to help you discover misconfigurations. If delivering a mail fails, or a mailserver rejects a mail because SPF or DKIM failed (most likely because of spam/scams), if configured, a DMARC report would be sent to the offender. Oftentimes, even if DMARC passes, an email is still sent to the reporting address to signify success.

The reporting address is specified in a DNS record, e.g.:

_dmarc.example.com. IN TXT "v=DMARC1; p=reject; rua=mailto:dmarc-reports@example.com"

Fun fact: IN stands for internet.

DMARC reports are in XML, and aren't really human-readable. Take a look at this mess:

<?xml version="1.0" encoding="UTF-8" ?>
<feedback>
  <report_metadata>
    <org_name>lukadeka.com</org_name>
    <email>noreply-dmarc@lukadeka.com</email>
    <report_id>...</report_id>
    <date_range>
      <begin>...</begin>
      <end>...</end>
    </date_range>
  </report_metadata>
  <policy_published>
    <domain>lukadeka.com</domain>
    <adkim>r</adkim>
    <aspf>r</aspf>
    <p>reject</p>
    <sp>reject</sp>
    <fo>0</fo>
  </policy_published>
  <record>
    <row>
      <source_ip>...</source_ip>
      <count>2</count>
      <policy_evaluated>
        <disposition>reject</disposition>
        <dkim>fail</dkim>
        <spf>fail</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <envelope_from>...</envelope_from>
      <header_from>lukadeka.com</header_from>
    </identifiers>
    <auth_results>
      <spf>
        <domain>...</domain>
        <scope>helo</scope>
        <result>none</result>
      </spf>
      <spf>
        <domain>...</domain>
        <scope>mfrom</scope>
        <result>fail</result>
      </spf>
    </auth_results>
  </record>
  <record>
    ...
  </record>
</feedback>

Instead, they're meant to be processed by automatic aggregation tools. Free third party services exist, but Stalwart Mail (the mailserver I use) has a built-in tool. This is a DMARC report received by my mailserver from Google (they love sending me DMARC reports):

How could you mess this up?

Now that we understand DMARC, we can look at what happened many times in my mailbox.

My mailserver sends DMARC reports to other mailservers whenever I send an email and it gets delivered. It's surprising how many people misconfigure their mailservers where either the address doesn't exist, or doesn't accept incoming mails (that's the whole point!). In both cases I get a mail from either my mailserver's mailer daemon or their mailserver that delivery failed.

I use a catch-all email setup, so I receive (almost) all mails sent at @lukadeka.com, and these failures end up in my mailbox.

I even received an automatic mail from a "Technology Leader" from SAP, telling me they were in India, and working from a different timezone:

"Thank you for your message. I am travelling and will be working from the Bangalore office from August 24 to August 28 during India hours."

I wonder why the "Technology Leader" never checked why DMARC reports constantly land in their mailbox.

Other domains included:

ashbyhq.com - Job application platform
cresta.ai - Where I applied for a job
gozauber.com - Where I applied for a job

This gave me an idea, to write a follow-up email informing them of the misconfiguration, and offering that I could fix it if they were to hire me, but people don't get the best first-impression when they're told they messed up.

All in all, seeing these mails when I wake up just makes me laugh a little, and I wanted to share that experience.

Building an app-unaware multi-app proxy: One domain, many backends

2026-07-01T00:00:00+00:00

Preface

At my job we provide software, and install sensors for buses/trains/trams that count passengers. The software aggregates the data, allowing it to be analyzed. One of the web-apps communicates directly with all of the sensors of each vehicle of the organization, gathering status, logs, and miscellaneous data. These sensors all have webservers that can be accessed via an IP, e.g. in the browser.

We have different types of sensors, many of which aren't manufactured inhouse, so we don't have control of the web-apps. This appeared to be a nuisance when we wanted the customers to be able to access the sensor applications along with us.

Proposed solutions

The obvious idea was to grant access to the sensors via a VPN. While being the simplest solution, this was unfortunately rejected because it required each customer to install a VPN client, our provided certificates, and would cause possible complexity for our IT, along with ever-updating ACLs.

The following idea to "just use a reverse-proxy" wouldn't work since the customers had way too many sensors to manually maintain one-by-one with a reverse-proxy like nginx. Even though we have many different kinds of sensors, none of them supported being proxied from a specific path. This means we'd need a (sub)domain for each sensor, with an authentication guard behind each of them. Managing this across teams would mean: for each modified vehicle/sensor or login identity, the customer would have to submit a ticket, we'd have to forward it to IT, and wait for them to manually update it. This system only scales well on opposite day.

The IT tried to make their proxy work, and hardcoded paths from the same domain to the sensors, but this broke the apps that weren't designed to be proxied from somewhere other than the root of the domain (/). The "authentication guard" I mentioned before was only done via IP whitelisting, which, given the dynamic nature of IP addresses, is insecure as well.

So, what other solution was there? Naturally, I did what any other reasonable engineer would do: I promised my team to write a custom reverse-proxy inside our Java Spring app that handles proxying the requests to the right sensor based on path variables while the apps aren't aware they're being proxied - so I did just that.

Challenges

Let's assume the API path is:

https://example.com/api/v1/proxy/123/

Which proxies to the 123rd app, which is running at e.g. http://10.0.0.0/.

A request to:

https://example.com/api/v1/proxy/123/index.html

should land at:

http://10.0.0.0/index.html.

Since the apps aren't aware they're being proxied, they make assumptions that the website is running at the root. This causes unintended behavior:

Root-relative fetches in the HTML like src="/style.css" resolve to wrong paths like https://example.com/style.css
Elements like CSS can make network requests that land at the wrong API path
Digest authentication can't be proxied since it was designed to prevent Man-In-The-Middle (MITM) attacks
Javascript updates the DOM to inject elements that can make more requests
Javascript can redirect the user to a wrong location, e.g. if it detects the browser isn't on a specific path
Apps can use relative paths with a backwards traversal like src="../image.jpeg" even though they're running on root, and break the API path

Here's how I addressed those:

Solutions in detail

Fixing these, and more, required iteration, and creative solutions.

1. Root-relative fetches in the HTML like src="/style.css" resolve to wrong paths like https://example.com/style.css

2. Elements like CSS can make network requests that land at the wrong API path

To start, traditional reverse-proxies like nginx already handle these cases - rewriting HTML is trivial. I decided to make all src/href/action attributes have absolute paths, e.g. src="/api/v1/proxy/123/style.css", while making sure 3rd party domains or other paths like these aren't rewritten:

javascript:, mailto:john@example.com, {{ ... }} (Angular variable)

3. Digest authentication can't be proxied since it was designed to prevent Man-In-The-Middle (MITM) attacks

Here's a TLDR on how Digest auth works:

The server sends the client a realm (predetermined string) and a nonce (random number)
The client generates an MD5 hash (response) using that nonce, the username, password, the request URI path, and other stuff
The server knows all of these in advance and validates the hash

Here's a sample processed Authorization header by the browser sent to the server:

Digest
username="secret_agent_007",
realm="example.com",
nonce="k6a95db6b55k5lrw5yezq3p2lu4l6k2m",
uri="/",
response="g0s8qt3n5kf588fm0efib2f4o4g0x2yv"

If Digest auth were to be proxied, the browser would compute an incorrect hash because of the API path in the uri field.

While I could've gone for Basic auth, and computed the hashes inside the proxy, I decided against that approach. This would've added more complexity, and would've made this part of the proxy stateful.

Instead, I utilized the app's database, which already stores the credentials for the sensors. So now, if a 401 UNAUTHORIZED is received for any request to the sensor, it's retried using the credentials from the DB using Digest auth.

Since Digest auth acts as a "session token" of sorts, I put that token in a cookie DIGEST_AUTH that I set in the browser. This way, if the proxy receives a request with that cookie, it assumes the user is authenticated and constructs the correct Authorization header.

This comes with a drawback - if the credentials are ever changed, the proxy will stop working until the credentials in the database are manually updated. In our case this can be done simply, and the tradeoff is definitely worth it.

4. Javascript updates the DOM to inject elements that can make more requests

Handling this was a bit tricky. Since Angular does this aggressively, ignoring or hardcoding this wouldn't work. The only approach that worked was monkeypatching JavaScript by inlining a script tag in the initial HTML. The Javascript listens to updates, and makes sure the paths are generated correctly.

5. Javascript can redirect the user to a wrong location, e.g. if it detects the browser isn't on a specific path

Inside the same <script> tag, I inlined a class window.__PROXY_LOCATION__, a mock of window.location, along with its methods. Afterward, I added a Regex inside the proxy that replaces window.location, document.location, location etc. safely with my mock class in the Javascript. This is necessary because the proxy doesn't run any Javascript and can't intervene during execution-time. The mock class makes sure that redirects, so href, pathname, assign, replace, and so on are handled with consideration of the proxy prefix (/api/v1/proxy/{SENSOR_ID}/).

6. Apps can use relative paths with a backwards traversal like src="../image.jpeg" even though they're running on root, and break the API path

While regular relative imports get handled correctly, ones that try a backwards traversal (../) while still being on root break the API path, e.g. http://10.0.0.0/image.jpeg. The backwards traversal here has no good reason to exist, since ../image.jpeg and ./image.jpeg mean the same thing on the root.

To handle network requests that might've slipped through the Javascript patching, or paths with backwards traversal, I added a Service Worker which is fetched for each page.

The Service Worker intercepts all network requests, and double-checks that requests to the API path have the correct path. E.g. if a resource ../image.jpeg was resolved to

https://example.com/api/v1/proxy/image.jpeg

instead of the correct

https://example.com/api/v1/proxy/123/image.jpeg,

the Service Worker would try to fix the path by doing a greedy find-and-replace of the API path.

While this isn't a perfect solution, and it doesn't work for backwards-relative paths with too many ../s, there weren't many options to handle apps that put ../ at the root for the fun of it.

Other challenges

While there were other challenges as well, the aforementioned ones were the most interesting. I felt like I wasn't going to overcome some of them, and there wasn't going to be an elegant (or even acceptable) solution that didn't include hardcoding. Nevertheless I didn't give up and managed to get the unorthodox proxy working piece-by-piece.

Other challenges included:

Proxying WebSockets (stateful)
Handling multipart/x-mixed-replace streams in a non-blocking way
Cookie path rewriting
Injecting a <base> tag in the HTML, and dealing with apps that also inject one
Upgrading 3rd party HTTP requests to HTTPS implicitly
Maintaining browser interactivity
Testing that one change didn't break other sensors

Conclusion

While the app isn't perfect, it's still incredible that it works, and is more-or-less generalizable for every application type. It was a really interesting challenge, and I learnt a lot about network protocols, proxies, browsers, frontend/backend interactions, SSL, and many more.

Special thanks to my online friend who works at Netflix for helping me with the design and my roadblocks in the Java implementation.

Introducing HexName - a free DNS service

2026-02-04T00:00:00+00:00

Introduction

After 4.5 months of work, I'm proud to finally announce the launch of my project HexName, the backend/frontend/infrastructure of which I built on my own.

The motivation

When I first started out with homelabbing, I eventually wanted to make my services available to the public via port-forwarding. To not have to remember my dynamic IP every time it updated, I used No-IP, a Dynamic DNS (DDNS) service that lets you periodically update a DNS record to point to dynamic IP, like example.hopto.org.

The problem with No-IP was that you had to log in and confirm your domain every 30 days, otherwise it would be deactivated. I found this really annoying.

I then discovered DuckDNS, a similar service that lets you register up to 5 subdomains like example.duckdns.org.

After having used their services for quite a while, I can say that their service is slow and unreliable. The DNS resolution frequently times out, and you end up having to retry until it eventually resolves, or access your server via an IP (which defeats the whole purpose of DDNS).

Additionally, both providers only allow you to publish a single DNS record per domain, disallowing any fun/advanced DNS configurations.

There's also afraid.org which has been operated since 2001 and serves multiple thousand DNS queries per second. They also offer DDNS with subdomain registration up to 5 subdomains for free, but with an added twist - people can "submit" their domains to let other people register subdomains off of it.

There is an obvious problem with this though, as addressed in the FAQ:

Q: How can I protect myself from using a domain that may disappear from freedns.afraid.org and/or become "broken"?

A: "... If you wish to use a shared domain in the system for any sort of long term use (like email/web services) then it is recommended you use one of the domains owned by me personally ..."

This kind of defeats the whole purpose of the idea.

Additionally, a couple of things stood out to me that I didn't like about the service:

The password length is limited to 4-16 characters (4 characters in 2026, really? We're still doing that?). I usually generate much longer passwords with my password manager
The registration form didn't include the correct attributes in the HTML for my browser/password manager to save/autofill the login credentials automatically
Every action (registering a subdomain/domain) required a captcha that I didn't find particularly easy to solve
There was no possibility to register subdomains off of the subdomain, e.g. mail.example.mooo.com

This inspired me to provide a better DNS service where users had freedom to do anything they wanted with their domain - run a mailserver, host various services - even change the nameservers, and all that without annoying the user, for free, forever.

Development cycle

Since I'm currently studying computer science full time at a university, as well as working a part-time job, I didn't have much time to work on the project. Nevertheless I decided to try and dedicate my free time to the project.

Backend

I started working on the backend for the project in the middle of October. I decided this was a great opportunity to learn a programming language I had heard great things about - Rust. I have to say, I absolutely loved the experience and the community behind it, providing great support and packages.

While developing the backend, I had to integrate it with PowerDNS's authoritative nameserver, the DNS server I settled on to run. The API wasn't straightforward, forcing me to write helper functions to interact with it properly, but eventually I got it working.

After many iterations, refactors, and switching the routing framework, I finished working on the backend two months after starting, afterward only implementing after-thought features and bugfixes.

Frontend

Since I had no previous experience with frontend development (except building this stunning portfolio website), I did a lot of research beforehand, comparing different technologies I could use.

I settled on SvelteKit for routing/reactivity, along with TailwindCSS, and a component library for it called DaisyUI.

DaisyUI significantly helped with the initial design of the website, thanks to which I didn't get stuck for hours trying to tweak login forms and buttons.

After three months, I finished the frontend as well. I decided to make the source-code publically available at my personal Git repo.

Since it's my first real experience with frontend development, there are glaring issues with the website, but overall I'm proud of how it turned out. You can check out a live demo at hexname.com.

Deployment

Being familiar with homelabbing and hosting my services with NixOS, the deployment was a breeze. I got another VPS from Hetzner Cloud and installed NixOS on it.

After playing around with oci-containers and making sure the services can reliably interact with each other, I set up logical replication with Postgres so that every change would also be reflected on the second nameserver in real-time, which I'm running on my personal Hetzner instance (the one serving this blog).

Other than that, I set up incremental backups of the database to be executed every 10 minutes to ensure no data is lost. Because I'm running/backing up the data between two VPSes in different regions, technically I'm running multi-region.

Additionally, to send/receive emails for my project, I set up another mailserver in two days, which gets a 10/10 score on Mail-Tester.

For monitoring, I use Uptime-Kuma to make sure the services stay up-and-running.

The NixOS configuration for the entire deployment is also available at my Git repo.

The product

As of now, HexName offers the registration of 20 total subdomains off of the following domains:

hexname.com
loves-beer.com
dickdns.org (you can tell why I went with this last one)

As well as 200 DNS records per subdomain from the supported record types:

A
AAAA
CNAME
TXT
MX
NS
SRV

Additionally, HexName offer a DDNS service for A records, allowing users to update the IP periodically, just like the DDNS providers mentioned before.

Practically, all of this means that users can register domains like george.loves-beer.com, use them to host their services (like game servers), blogs, and even receive emails with them (e.g. contact@george.loves-beer.com).

Credits

While learning Rust for the backend, and Svelte, I had a lot of questions about language syntax, semantics, as well as general programming concepts. Fortunately, I'm in a very helpful online community where they were able to assist me along the journey.

Doesn't matter what you're learning, being around experienced people is a significant help.

Moving forward

Since I don't expect to make any money off this project, I made it solely for the sake of wanting to offer a better free service, as well as a learning opportunity.

I plan to support/maintain the project for a long time as well as update the programs/system periodically. I don't expect this to take too much time since I'm familiar with the technologies used for deployment and I went with a minimalistic approach.

Currently, the yearly costs of the project are:

€49.8 for the VPS (ns1.hexname.com)
$14.98 for the domain hexname.com
$14.98 for the domain loves-beer.com
$12.98 for the domain dickdns.org

So 101.0$/year or 8.5$/month in total.

If you'd like to support the project, feel free to contact me here.

Rescuing NixOS stuck on Stage 1 boot

2025-10-27T00:00:00+00:00

Introduction

One of the main selling points of NixOS is that it allows you to have many different binaries of the same program at the same time. Along with program isolation and dependency management, this enables the user to have many of the so-called NixOS generations at the same time.

How it works is simple: when applying any change to the system, like installing a program or changing an option declaratively, you have to rebuild the system. You can either switch to it immediately like so:

sudo nixos-rebuild switch

or switch to it only when you reboot the machine:

sudo nixos-rebuild boot

If the rebuild succeeds, it creates a new generation. The user is still able to switch to a previous generation though, and this comes in very handy when either a rebuild succeeds but the system is left unusable. Or if something hangs/breaks after a reboot, you are able to boot into a different generation.

The issue

I have a service enabled that garbage-collects previous generations older than 2 weeks:

nix.gc = {
  automatic = true;
  dates = "Fri *-*-* 04:00:00";
  options = "--delete-older-than 14d";
};

This means, if I were to make a change that broke the booting sequence of my system, and I only noticed it 2 weeks after, I'll be left with an unbootable system. While this might sound scary, it doesn't happen often. Unluckily for me, this is exactly what happened.

My issue was that the system was waiting indefinitely for the kernel module vfio_pci to load, which wasn't responding. After making sure there were no timeouts in place, I rebooted the system into a NixOS minimal installation image and began diagnosing.

Fixing the issue

I'm using zfs for my boot drive, so the first step was to mount it.

# Escalate your privileges
sudo su

# Load the zfs kernel module
modprobe zfs

# List all available pools (that are not imported)
zpool import

# Show all available datasets
zfs list

Output:

NAME                           USED  AVAIL  REFER  MOUNTPOINT
nvmepool                      51.3G   406G    96K  none
nvmepool/home                  156M   406G   156M  legacy
nvmepool/nix                  16.7G   406G  16.7G  legacy
nvmepool/root                 11.8M   406G  11.8M  legacy
nvmepool/var                  1.35G   406G  1.35G  legacy

Here, nvmepool is the name of my zpool.

Now create the directories to mount the datasets on:

mkdir -p /mnt/home /mnt/nix /mnt/root /mnt/var

and mount them:

mount -t zfs nvmepool/root /mnt
mount -t zfs nvmepool/home /mnt/home
mount -t zfs nvmepool/nix /mnt/nix

Note: nvmepool/root refers to the / directory, also known as the root directory, and not to the /root/ directory, which is the root user's home directory.

The boot directory also has to be mounted explicitly for NixOS to be able to build generations, so find out which is your boot partition:

$ lsblk --fs
NAME        FSTYPE     FSVER LABEL    UUID                FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1
├─nvme0n1p1 vfat       FAT32 boot     6445-6413            906.6M    11% /boot
└─nvme0n1p2 zfs_member 5000  nvmepool 6733023062829592015

Here, look for vfat, or for the EFI or ESP label. The boot partition shouldn't be larger than a few gigabytes.

Afterward, mount it:

mount /dev/nvme0n1p1 /mnt/boot

In my case, I also had to mount these directories:

mount --rbind /dev /mnt/dev
mount --rbind /sys /mnt/sys
mount --rbind /proc /mnt/proc

Finally, enter the mounted system:

nixos-enter --root /mnt

Make sure networking works (for cache.nixos.org):

ping google.com

Then, navigate to your NixOS config directory, edit and revert the changes manually, and execute:

nixos-rebuild boot

Or this, if you're using flakes:

nixos-rebuild boot --flake .#

If that succeeds, exit the chrooted environment:

exit

and export the zpool cleanly:

zpool export nvmepool

Note: the last step isn't really necessary if you don't have this NixOS option set: boot.zfs.forceImportRoot = true;

Finally reboot the system (and make sure not to boot into the live image). Afterward, you should be able to boot into your system normally.

My research paper on Functional Paradigms and LISP

2025-09-19T00:00:00+00:00

Preface

In the Summer semester 2025 I signed up to a seminar worth 3 ECTS points at the Technical University Berlin since attending one seminar is mandatory for a Bachelor's degree in Computer Science.

Since seminar slots are hard to get for this exact reason, my university usually hosts a "meta-page" where you can rank your top choices and get assigned a seminar.

So I got assigned the seminar Open Distributed Systems (ODS).

Introduction

The whole point of a seminar is to learn how to do research and gather first-hand experiences at writing your first scientific paper.

The process was split up like this:

We had a few introductory classes
Then we were instructed to write an extended abstract (max. two pages) based on a research paper from their curated list
After which we had to anonymously give feedback to two other research papers using a template
After we received the feedback, we had to prepare a 10 minute presentation, present it, and answer questions from the audience for an additional 5 minutes
And in the end, after considering all the feedback, we had to submit 5-6 pages of the final research paper

All the while having 4 classes in total on how to do research, write a good research paper, how to present, the formalities, etc.

Topic

I picked the paper Recursive functions of symbolic expressions and their computation by machine, part I by John McCarthy. I chose this topic because my friend was recommending Lisps, and I wanted to learn more about Functional Programming (FP) in general.

I ended up phrasing my title like this:

"Functional paradigms in LISP and its contribution to reliable, maintainable software"

And my research question like this:

"In what way do the functional programming paradigms introduced in LISP contribute to the development of more reliable and maintainable software?"

I mostly talked about the benefits of FP over imperative, how it supports Test-Driven Development (TDD), and what the benefits of enforcing static types as well as pure functions has.

I also argued that FP is very useful when a language rewrite is necessary, and when the priorities shift to codebase maintainability, readability, and minimizing runtime errors - since a lack thereof accounts to over 50% of project costs in practice:

"...a common misconception among beginner software developers is that designing and writing efficient programs is the most important task of the job. In practice, incorrect, unreadable, and hard to maintain software accounts to over fifty percent of the project costs. Tremendous resources could be saved if the focus were shifted to readability, reliability and correctness, but the reality shows that most businesses favor rapid development and software that just works in the present."

The paper

Click here to open the PDF in a new tab.

This browser does not support PDFs. Please try opening it in a new tab, or download the PDF to view it.

Shortcomings

As stated in the paper, I didn't have enough time, and was unable to find solid evidence to support all my claims. Because of this, I had to resort to logical reasoning, which wasn't as scientific as it should've been for a research paper, which I acknowledged here:

"Despite a lot of research being FP-centric, finding solid empirical evidence in industrial and enterprise contexts proved to be challenging. Due to a limited availability of published enterprise data, we relied on logical reasoning where empirical validation was unavailable."

Overall, I made it harder for myself by choosing such a broad topic, and trying to argue against a very popular paradigm.

Because I focused on FP, I didn't cover Lisps enough. I failed to mention and explain Lisp's macro system, which is its defining characteristic, which allows code to programmatically modify other parts of the code.

Conclusion

Overall, I learned a lot about FP and the history of LISP by doing the research, as well as languages like Clojure, and BEAM-based languages like Erlang and Elixir. The experience gained in doing research will also come in handy for my Bachelor thesis.

If you'd like to cite my paper, here is the BibTeX:

@misc{FunctionalParadigmsAndLisp,
  author = {Luka Dekanozishvili},
  title = {Functional paradigms in LISP and its contribution to reliable, maintainable software},
  howpublished = {\url{https://lukadeka.com/resources/functional-paradigms-and-lisp.pdf}},
  email = {research@lukadeka.com},
  school = {Technical University Berlin},
  address = {Berlin, DE},
  year = {2025},
  month = {September}
}

Setting up Netbird with Zitadel on NixOS

2025-08-28T00:00:00+00:00

Preface

Deploying Netbird's management server on NixOS proved particularly difficult since no step-by-step guide was provided for NixOS. I had to piece things together by reading documentation from many different documentation sources, including source code, and experimenting.

Since the options at search.nixos.org were well documented, I was able to get a working setup after some trial and error, and debugging.

Introduction

NetBird is an open-source VPN management platform built on top of WireGuard making it easy to create secure private networks for your organization or home.

Setup

Netbird requires an Identity Provider for authentication/authorization. The supported self-hosted options are:

Zitadel
Keycloak
Authentik

Zitadel

Configuration

Here's an example config for Zitadel, along with its database:

let
  domain = "example.com";
in
{
  services.zitadel = {
    enable = true;
    openFirewall = true;

    masterKeyFile = "/path/to/zitadel/master_key";
    extraStepsPaths = [ "/path/to/zitadel/admin_steps" ];
    extraSettingsPaths = [ "/path/to/zitadel/settings" ];

    tlsMode = "external";
    settings = {
      Port = 39995;
      ExternalPort = 443;
      ExternalDomain = "auth.${domain}";
      Database = {
        postgres = {
          Host = "127.0.0.1";
          Port = 5432;
          Database = "zitadel";
          MaxOpenConns = 15;
          MaxIdleConns = 10;
          MaxConnLifetime = "1h";
          MaxConnIdleTime = "5m";
        };
      };
    };
  };

  # Postgres database for Zitadel
  virtualisation.oci-containers.containers.zitadel-db = {
    image = "postgres:17";
    ports = [ "5432:5432" ];
    environmentFiles = [
      "/path/to/zitadel/postgres_env"
    ];
    volumes = [
      "/var/lib/zitadel-db:/var/lib/postgresql/data"
    ];
  };

  networking.firewall.allowedTCPPorts = [ 80 443 ];

  # Ensure the mounted directory for the database exists
  system.activationScripts.makeZitadelDir = lib.stringAfter [ "var" ] ''
    mkdir -p /var/lib/zitadel-db
  '';

  # Proxy the SSO provider
  services.nginx.enable = true;
  services.nginx.virtualHosts."auth.${domain}" = {
    forceSSL = true;
    enableACME = true;
    locations."/" = {
      proxyPass = "http://127.0.0.1:39995";
      proxyWebsockets = true;
      extraConfig = ''
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
      '';
    };
  };

  security.acme = {
    acceptTerms = true;
    defaults.email = "you@example.com";
  };
}

Environment variables

Generate the master_key with:

tr -dc A-Za-z0-9 </dev/urandom | head -c 32

Create and populate the admin_steps file:

FirstInstance:
  InstanceName: Zitadel
  Org:
    Human:
      UserName: John
      FirstName: John
      LastName: Lackland
      DisplayName: John
      Password: make-sure-this-is-secure
      PasswordChangeRequired: false
      Email:
        Address: you@example.com
        Verified: true

Create and populate the settings file:

Database:
  postgres:
    User:
      Username: zitadel
      Password: make-sure-this-is-secure
      SSL:
        Mode: disable
    Admin:
      Username: postgres
      Password: make-sure-this-is-secure
      SSL:
        Mode: disable

The postgres user is used as a oneshot to initialize the zitadel user after the initial database creation.

Create and populate the postgres_env file:

POSTGRES_USER=postgres
POSTGRES_PASSWORD=set-the-same-password-as-in-settings
POSTGRES_DB=postgres

Registration

Afterward, log in with your Admin account at auth.example.com, and follow Netbird's documentation on how to configure Zitadel.

Make sure to put netbird.example.com as the domain in the Redirect Settings and not auth.example.com.

Put the ClientSecret from Zitadel in the client_secret file, and hold on to the Client ID.

The Client ID is not a secret so it's okay to hardcode it in the NixOS configuration.

Netbird

Afterward, set up Netbird:

{ config, lib, ... }:

let
  domain = "example.com";
  netbirdDomain = "netbird.${domain}";
  clientId = "<YOUR_CLIENT_ID_FROM_ZITADEL>";
in
{
  imports = [ ./zitadel.nix ];

  services.netbird.server = {
    enable = true;
    enableNginx = true;
    domain = netbirdDomain;

    coturn = {
      enable = true;
      domain = netbirdDomain;
      passwordFile = "/path/to/netbird/turn_password";
    };

    signal = {
      enable = true;
      enableNginx = true;
      domain = netbirdDomain;
    };

    dashboard = {
      enable = true;
      enableNginx = true;
      domain = netbirdDomain;
      settings = {
        AUTH_AUTHORITY = "https://auth.${domain}";
        AUTH_CLIENT_ID = clientId;
        AUTH_AUDIENCE = clientId;
      };
    };

    management = {
      enable = true;
      enableNginx = true;
      domain = netbirdDomain;
      turnDomain = netbirdDomain;
      singleAccountModeDomain = netbirdDomain;
      oidcConfigEndpoint = "https://auth.${domain}/.well-known/openid-configuration";

      settings = {
        Signal.URI = "${netbirdDomain}:443";

        HttpConfig.AuthAudience = clientId;
        IdpManagerConfig.ClientConfig.ClientID = clientId;
        DeviceAuthorizationFlow.ProviderConfig = {
          Audience = clientId;
          ClientID = clientId;
        };
        PKCEAuthorizationFlow.ProviderConfig = {
          Audience = clientId;
          ClientID = clientId;
        };

        TURNConfig = {
          Secret._secret = "/path/to/netbird/turn_password";
          CredentialsTTL = "12h";
          TimeBasedCredentials = false;
          Turns = [
            {
              Password._secret = "/path/to/netbird/turn_password";
              Proto = "udp";
              URI = "turn:${netbirdDomain}:3478";
              Username = "netbird";
            }
          ];
        };
        Relay = {
          Addresses = [ "rels://${netbirdDomain}:33080" ];
          CredentialsTTL = "24h";
          Secret._secret = "/path/to/netbird/relay_secret";
        };
        DataStoreEncryptionKey._secret = "/path/to/netbird/data_store_encryption_key";
      };
    };
  };

  # Make the env available to the systemd service
  systemd.services.netbird-management.serviceConfig = {
    EnvironmentFile = "/path/to/netbird/setup.env";
  };

  # Override ACME settings to get a cert
  services.nginx.virtualHosts = lib.mkMerge [
    {
      "${netbirdDomain}" = {
        enableACME = true;
        forceSSL = true;
      };
    }
  ];

  # Run the Netbird relay with TLS to allow relaying over TCP
  virtualisation.oci-containers.containers.netbird-relay = {
    image = "netbirdio/relay:latest";
    ports = [
      "33080:33080"
    ];
    volumes = [
      "/var/lib/acme/${netbirdDomain}/:/certs:ro"
    ];
    environment = {
      NB_LOG_LEVEL = "info";
      NB_LISTEN_ADDRESS = ":33080";
      NB_EXPOSED_ADDRESS = "rels://${netbirdDomain}:33080";
      NB_TLS_CERT_FILE = "/certs/fullchain.pem";
      NB_TLS_KEY_FILE = "/certs/key.pem";
    };
    environmentFiles = [
      "/path/to/netbird/relay_secret_container"
    ];
  };

  networking.firewall.allowedTCPPorts = [ 80 443 3478 10000 33080 ];
  networking.firewall.allowedUDPPorts = [ 3478 5349 33080 ];
  networking.firewall.allowedUDPPortRanges = [{
    from = 40000;
    to = 40050;
  }]; # TURN ports
}

Environment files

Generate the turn_password and data_store_encryption_key with:

openssl rand -base64 32

Generate the relay_secret with:

openssl rand -base64 32 | sed 's/=//g'

And also put it in the relay_secret_container file for the Podman container:

NB_AUTH_SECRET=the-same-secret-as-in-relay-secret

Create and populate the setup.env file:

NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT="https://auth.example.com/.well-known/openid-configuration"
NETBIRD_USE_AUTH0=false # Since we're using Zitadel
NETBIRD_AUTH_CLIENT_ID="<YOUR_CLIENT_ID_FROM_ZITADEL>"
NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api"
NETBIRD_AUTH_AUDIENCE="<YOUR_CLIENT_ID_FROM_ZITADEL>"
NETBIRD_AUTH_REDIRECT_URI="/auth"
NETBIRD_AUTH_SILENT_REDIRECT_URI="/silent-auth"

NETBIRD_AUTH_DEVICE_AUTH_PROVIDER="hosted"
NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID="<YOUR_CLIENT_ID_FROM_ZITADEL>"

NETBIRD_MGMT_IDP="zitadel"
NETBIRD_IDP_MGMT_CLIENT_ID="netbird"
NETBIRD_IDP_MGMT_CLIENT_SECRET="<YOUR_CLIENT_SECRET_FROM_ZITADEL>"
NETBIRD_IDP_MGMT_EXTRA_MANAGEMENT_ENDPOINT="https://netbird.example.com/management/v1"
NETBIRD_MGMT_IDP_SIGNKEY_REFRESH=true

NETBIRD_DOMAIN="netbird.example.com"
NETBIRD_DISABLE_LETSENCRYPT=true # Since Netbird is behind nginx
NETBIRD_MGMT_API_PORT=443
NETBIRD_SIGNAL_PORT=443
TURN_MIN_PORT=40000
TURN_MAX_PORT=40050

Cloud providers

Many cloud providers like Hetzner Cloud use stateless firewalls (since they're cheaper to run than SPI firewalls). These can interfere with Netbird's operation.

If your VPS provider uses a stateless firewall, you have to open up the required dynamic ports that Netbird uses in the cloud provider's UI.

To see which ports you need to open, run:

sudo cat /proc/sys/net/ipv4/ip_local_port_range

See additional configurations for cloud providers.

Additional configuration

To enable the Netbird client, set:

  services.netbird.enable = true;

Troubleshooting

Setup issues

The generated configuration for the management interface is stored in /var/lib/netbird-mgmt/management.json. Verify this file if you suspect the environment variables aren't being applied properly, or to check if you're correctly overriding the defaults with services.netbird.server.management.settings.

You can check if the TURN/STUN and Relay servers are working properly with the online tester Trickle ICE. A simple guide for troubleshooting is provided here by Netbird.

Client issues

Run netbird status -d to check the details of the client as well as available relays. This also shows the connection status with other peers.

A more detailed description of diagnosing client issues can be found in Netbird's documentation.

How to cope with losing a public IP as a homelabber

2025-08-24T00:00:00+00:00

Preface

This post explains how IP routing, NATs and CGNATs (or double-NATs) work, discusses different strategies for homelabbers to bypass CGNAT restrictions, and focuses on Netbird as a solution.

This post is mainly intended for homelabbers, beginner network engineers, or curious system or DevOps administrators.

Introduction

I used to have my own residential IP address before June 2025, and exposing services to the internet was trivial. I only needed to punch holes through my router (i.e., port forwarding), and the services were accessible from anywhere.

Note that port forwarding is a potential attack surface, and should not be done carelessly. I heeded this warning and ensured I followed proper security practices (like zero-trust approaches, IP blocklisting, etc.).

In June 2025 my ISP (Internet Service Provider) decided to install their routers for everyone in the building. Given the guaranteed downtime, they decided to kill two birds with one stone and rob everyone of their public IP addresses, including me. We were put behind a CGNAT (Carrier Grade Network Address Translation).

A CGNAT functions like a NAT we all know and love: the router. It hands out private IP addresses that are unroutable from the WAN (Wide Area Network) to its hosts. When a host wants to access a website, the router changes the source IP address of the host (i.e., the private one) to the WAN IP address the router has received. This address is commonly a public-facing one. The router keeps track of which port (from its ephemeral port range) was used to send out the request, and returns the resources back to the correct host.

Since IPv4 addresses started running out and getting more expensive, NATs were introduced to allow multiple hosts to share the same IPv4 address.

I mentioned before that a router's WAN IP address doesn't have to be public. This is exactly where a CGNAT comes in: it hands out private IP addresses to the hosts (so routers), and translates them to public ones, just like a run-of-the-mill router would. This is why a network behind a CGNAT is commonly called a double-NAT.

So like hosts on the LAN can share a public IP address, different LANs on different routers behind a CGNAT can share one as well. Commonly a pool of public IP addresses is used.

Since I couldn't forward ports on the CGNAT to point to my router, which would then point to my home-server, I needed a different approach than port forwarding.

Solution?

Since I was already using WireGuard to self-host VPNs to my home-servers, I decided to buy a cheap VPS, and "just use WireGuard", as many people recommended in forums, to expose my services.

I set up a WireGuard server on the VPS and a WireGuard client on my home-server that would always keep a connection with the server. This way, the home-server was reachable from only the VPS.

Once I made another subnet for the VPN, and forwarded the ports I wanted (so ports 80 and 443 for HTTP and HTTPS) from the VPS to the home-server, the setup worked great, and was reliable.

Only after I wanted to access the server from outside my home network did I encounter weird issues: SSH didn't work after forwarding the SSH port for my home-server. After debugging for a long time, and failing to find a solution online, I gave up.

Had the SSH connection worked, I still wouldn't've been satisfied. Every time I wanted to forward a port, using that port on the VPS would've been impossible. This was especially problematic for the HTTPS port (443) since it meant I could only serve websites from one machine.

While not sounding like a huge deal, I didn't want a suboptimal setup when I had heard of better alternatives, so I started investigating.

Considerations

Before I could consider my options to continue exposing services to the internet, I needed to write down my requirements for the software/service:

Ability to connect to my home network from anywhere
Ability to act like an exit node (common VPN functionality)
Allow me to expose any service I want on any port
Integrate well with other servers I have
Uncomplicated networking
Allow me to only expose what I need to (zero-trust)
Free and open-source software only, ideally self-hostable
As few external third-party dependencies as possible
Good documentation
Simple and reliable

I considered Cloudflare tunnels, which allow you to expose anything you want using their proxies. I ultimately decided against it since it violates 7. and 8., and privacy reasons, even though it's free.

I then considered Tailscale, which is a mesh VPN based on WireGuard. While a popular choice for connecting many hosts on different networks together, and having a free tier, their control-server is closed-source, and not self-hostable. This is not a huge privacy deal though since the control server only connects hosts to each other, and optionally acts as a relay for end-to-end encrypted data when the connection can't be established. I also didn't really like the mobile app.

Headscale is an alternative free and open-source self-hostable Tailscale-compatible control server (that was a lot of words), whose integral contributor also works at Tailscale. I briefly tried it out, but ultimately decided against using it since they don't support exit nodes, and don't plan on adding the functionality anytime soon. The documentation also wasn't comprehensive and I had trouble getting it to run properly on NixOS, my server operating system of choice.

My second to final consideration was Zrok which is based on OpenZiti. I believe this would have been a solid choice, but I couldn't figure out how my network architecture would look like, and it ended up being too complicated to run on NixOS.

This is where I came across Netbird.

Introducing Netbird

Similar to Tailscale, Netbird is a mesh VPN based on WireGuard that allows devices to connect to each other peer-to-peer or fall back to a relay if NAT traversal fails.

To answer all my requirements, Netbird allows you to:

Connect to a network with at least one Netbird peer (with support for high-availability with redundant peers)
Use any peer as an exit node
Proxy services very easily using private Netbird IPs
Support for Linux, Android, Windows, Mac, iOS, and even OpenWRT routers
Subnets are well supported, as well as DNS management, and different client profiles
Supports zero-trust principles using groups, ACLs (Access Control Lists) with posture checks (minimum client requirements, IP filtering, etc.)
Self-hostable, free and open-source software. They also offer a managed free tier
The easy-to-comprehend management interface shows which peer is online/offline, and no critical feature seems to be lacking
Great documentation at docs.netbird.io, along with an active, friendly and helpful community of developers, and administrators
The management interface is easy to comprehend, and doesn't confuse the user with unnecessary options

That being said, I bought a cheap VPS, and deployed the management server there. From that point on, all I had to do was install the Netbird client on all my devices. Thankfully this is as easy as copy pasting a simple install script:

curl -fsSL https://pkgs.netbird.io/install.sh | sh

and for NixOS, adding one line to my declarative config:

  services.netbird.enable = true;

The main benefit of Netbird was the hugely simplified management of peers, how easy ACLs were to set up, how straightforward the setup was, and how it allows me to have a clear overview of the status of the peers. Also compared to the setup with WireGuard I had before, peer-to-peer connections are now possible instead of always relaying everything. Not to mention, I really like the design of the management interface (if you couldn't tell (and haven't seen how it looks like)).

Conclusion

Deploying Netbird on NixOS took some time since I couldn't find a guide for NixOS specifically, and had to go off reading documentation at many different places, reading source code, and experimenting. Eventually I managed to get it up and running, and decided to write a guide on setting up Netbird for NixOS to spare potential headaches.

Netbird has been working great ever since the initial deployment.

Since the VPS has a publicly routable IP (how else would you access it?), all that's needed is to proxy the desired traffic to the home-server via the Netbird IP, and that's all she wrote.

Setting up Seafile on NixOS

2024-12-03T00:00:00+00:00

Preface

After I decided I wanted to set up Seafile, a file server, I found little documentation for NixOS.

After experimenting with the options, and asking the maintainer for help, I finally got a working configuration, and decided to write a wiki entry as well to help future NixOS newcomers.

This entry is also available on wiki.nixos.org and nixos.wiki.

Introduction

Seafile is a file-hosting software system with a simple web interface and client applications for file access. Seafile's functionality is similar to file-hosting services such as Dropbox and Google Drive.

As opposed to Nextcloud, Seafile offers simpler user administration and better file-server performance.

Setup

Minimal configuration of Seafile:

  services.seafile = {
    enable = true;

    adminEmail = "admin@example.com";
    initialAdminPassword = "change this later!";

    ccnetSettings.General.SERVICE_URL = "https://seafile.example.com";

    seafileSettings = {
      fileserver = {
        host = "unix:/run/seafile/server.sock";
      };
    };
  };

Use nginx to serve Seafile from a unix socket:

  services.nginx.virtualHosts."seafile.example.com" = {
    forceSSL = true;
    enableACME = true;
    locations = {
      "/" = {
        proxyPass = "http://unix:/run/seahub/gunicorn.sock";
        extraConfig = ''
          proxy_set_header   Host $host;
          proxy_set_header   X-Real-IP $remote_addr;
          proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header   X-Forwarded-Host $server_name;
          proxy_read_timeout  1200s;
          client_max_body_size 0;
        '';
      };
      "/seafhttp" = {
        proxyPass = "http://unix:/run/seafile/server.sock";
        extraConfig = ''
          rewrite ^/seafhttp(.*)$ $1 break;
          client_max_body_size 0;
          proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_connect_timeout  36000s;
          proxy_read_timeout  36000s;
          proxy_send_timeout  36000s;
          send_timeout  36000s;
        '';
      };
    };
  };

Additional configuration

  services.seafile = {
    # ...
    seafileSettings = {
      quota.default = "50"; # Amount of GB allotted to users
      history.keep_days = "14"; # Remove deleted files after 14 days

      fileserver = {
        host = "unix:/run/seafile/server.sock";
        web_token_expire_time = 18000; # Expire the token in 5h to allow longer uploads
      };
    };

    # Enable weekly collection of freed blocks
    gc = {
      enable = true;
      dates = [ "Sun 03:00:00" ];
    };
  };

To change the directory of the database, create the directory with the appropriate permissions, chown -R seafile:seafile it and set:

  services.seafile = {
    # ...
    dataDir = "/path/seafile/data";
  };

Troubleshooting

The initialAdminPassword is set only once when the server is first initialized. Any changes to it afterward will have no effect on it. If you cannot log in for the first time, delete /var/lib/seafile/data, remove the Seafile configuration from your config, rebuild, re-add it, and the password will be set in the next rebuild.

Logs for Seafile and SeaHub (Seafile's web interface) are stored in /var/log/seafile/server.log and /var/log/seafile/seahub.log respectively.

For additional options, refer to the Seafile Admin Manual.