yapping about self-hosting, arch linux, networking

boring

Dec 16, 2024

intro

If you are wondering what the vacuous and cavernous hole my chest is occupied with, the month's flavour is an unhealthy hyper-fixation on self-hosting and networking. (No, I am not feigning interest in the creative pursuits of others to further my own; I mean actual networking with computers and stuff.)

I have this autonomous negative thought, which reoccurs constantly, and when I blink, its impression is only more intensely embossed: His Majesty's Police, in an attempt to suppress the free flow of information and uphold the spiritual cuckoldry of the Nanny State™, bursts into my domicile with the intention of seizing every single large language model and mirror of the Arch Wiki that's installed on my NAS.

This final act of tyranny is my tipping point, the entropic conclusion accrued from a lifetime of psychic damage. This is my Killdozer, my own Michael Douglas in Falling (1993).

The unlimited resources afforded by the state make intransigence a guaranteed death sentence, but my postulation is as follows, is not dying a free man the strongest act of resistance? Liberated from the flesh, my soul is sovereign.

They oblige me, rendered from serf to spirit they make me an example, I am ventilated in a Waco-style siege, levelling the entire soviet style brutalist council estate to get to one 4-bay NAS. Sycophantic mouthpieces of the media gaslight in their sophistry nitpicking around the semantics of necessity when inquired about the force levied and whether or not it was 'proportionate'.

An internal investigation is conducted and the administration has found itself to be not guilty. Sealed in a tomb of bureaucracy, It is all part and parcel, well oiled the machine operates as usual.

on distro-hopping

I digress, so please forgive me, but it was non-negotiable as I simply had no other avenues to gain your attention or the proficiency to seamlessly segue into my self-hosting origin story. My interest in self-hosting emerged from a primitive desire to construct a basic media server out of a Raspberry Pi 3 that I purchased around ~2016. Running Raspbian was also my first tangible experience with Linux, outside of maybe running a few commands in the terminal of a MacBook.

Although I managed to set up Plex and connect it to an external WD Red drive, I wouldn't say I truly understood Linux; I simply copied and pasted commands from a tutorial until I reached the desired outcome. This was, however, a formative experience, but the real learning began when I encountered issues that the tutorial didn’t cover. Essentially, this acted as a pipeline, providing the sliver of confidence needed to combat the intimidation of the Linux ecosystem and begin my journey of distro-hopping. Eventually, I "reverted" (Mashallah) to using Linux as my daily driver on both my desktop and ThinkPad. This also paved the way for pursuing a more robust server solution in the form of a Synology NAS after experiencing the loss of a 2TB Seagate external drive that held a bunch of sentimental data.

My growing concern regarding Microsoft Windows was its telemetry: the frequency and intrusiveness of this data collection raise privacy concerns. It appears to increase with every version, and the dark patterns present within its architecture mean that even if you manage to disable it, it’s only temporary as all it takes is one update to reverse everything—unless you're highly vigilant and committed to playing this endless cat-and-mouse game. Additionally, in Windows’ effort to capitalize on the AI trend, there has been a shift towards a system of AI telemetry. This new system appears significantly more aggressive and intelligent than its predecessor, although it is optional. I am not interested in normalizing the idea of a GPT wrapper that monitors my activities. In my opinion, such a concept seems invasive and concerning. It’s unreasonable—and genuinely concerning—to expect users to consent to be surveilled by an AI system, especially when they have the option to opt-out.

Furthermore, it felt like the amount of money I had been spending on computer components didn't accurately correlate with the performance I should be able to achieve. I felt that I was simply paying to offset the performance impact these new "features" introduced. This sentiment made me feel like even being part of the Windows ecosystem was somehow wasteful, as it antiquated hardware far before its time.

Although ARM and x64 are different processors, considering the capabilities of the Raspberry Pi, it is a living example that a device can be simultaneously powerful while also not being demanding. Rather than an idealistic concept or pipe dream. I had rapidly been approaching an inflection point where philosophically and pragmatically it did not seem feasible to continue using Windows.

I distro-hopped this year, trying out Debian and Linux Mint but ultimately returned to Windows. It was not until I installed the Arch-based distro EndeavourOS on my ThinkPad that I began to find a distro that worked for me. However, I eventually had no choice but to install Arch as my EndeavourOS installation had a weird glitch where my password would randomly be refused, even after running:

su
failock --user pry --reset

In the 5 months of running Arch Linux, I probably reinstalled the operating system at least 5 times, and every time was as equally demoralising. It is psychologically unsustainable to be partaking in this bizarre form of self-flagellation unless you maybe possess a degradation kink or other unspecified masochistic preferences. So now I have resorted to making an image of the machine every few weeks that I just restore every time I break Arch.

The idea that people make distros their entire personality does possess an element of truth, but I feel it is more apt to say that an individual's personality makes their distro. Like are you seriously surprised that an older woman would opt to use Debian, one of the oldest and most stable Linux distributions? She didn’t pick the distro; the distro picked her.

Conversely, the idea that this androgynous trans femme runs Arch, one of the most bleeding-edge and customizable distros, just makes sense. Being trans is like ricing a computer with a different architecture.

Chicken/egg type shit fr

You quickly learn that when using self-hosting solutions or Linux, you're essentially transferring the financial cost of ease of use into an investment of your time, sanity, and patience. I have been self-hosting for around 3 years now, I got introduced to docker when figuring out what would be the most painless way of starting a Minecraft server and it eventually snowballed, I think once someone becomes introduced to docker they get one shotted and must containerise everything.

As of now the only containers I'm running, are Open Web-UI, Portainer and a Soulseek server.
My self-hosting pursuits were more around creating this walled garden and not relying on getting nickeled and dimed from subscription services.

E-mail

Once you've developed a taste for self-hosting and gained experience with hosting web servers, the next logical step is email. However, to anyone considering hosting their mail server, be warned: this is not for the faint of heart. It requires significantly more configuration than running an nginx server and for good reason.

Email is an old protocol that has been exploited by malicious actors since its inception. The cost of sending spam emails is extremely low, making it an attractive option for those looking to advertise. For instance, if you send 1000 spam emails and one person buys the product, that's a hit rate of 0.1%. Although this may seem like a low success rate, the cost of setting up and sending those emails is a fraction of a penny.

What's even cheaper than sending 1000 emails from your mail server?

Sending 1000 emails from YOUR mail server, or sending 1000 emails from a mail server that pretends to be someone else's email server to avoid detection.

These glaring vulnerabilities are why hacking a mail server is so valuable because the potential return is invaluable, I cannot emphasise enough how dire an incorrect or suboptimal configuration of an email server is. If you misconfigure an email server it can have dire consequences for the reputation of your domain/IP moving on and if you're self-hosted you probably will have to get a new ISP.

So checks and balances had to be implemented to stop this form of abuse.

How Senders Are Authenticated

To prove the legitimacy of a sender there are three records that you have to configure with your DNS server:

SPF (Sender Policy Framework)
By verifying that the sending IP address is authorised to send emails on behalf of your domain, SPF helps prevent spammers from using your domain name in the "From" field. Emails failing this check are often marked as spam or rejected by receiving servers.
DKIM (DomainKeys Identified Mail)
DKIM ensures that the content of an email has not been altered during transit. If the digital signature is valid, it confirms that the email comes from a legitimate source and hasn't been tampered with. Emails with invalid signatures are often flagged as spam or rejected.
DMARC (Domain-based Message Authentication, Reporting & Conformance) records
DMARC provides a policy layer on top of SPF and DKIM. It instructs receiving mail servers on how to handle emails that fail either SPF or DKIM checks, thereby strengthening the overall security posture against email spoofing and phishing attempts.

Misconfiguration of these records will have your emails not reach their destination and your IP might also be flagged for spam. Which is an instant gg.

These 3 records, help to prove that you are the sender, but regardless if you fail to adequately secure your server and someone compromises it and uses it to spam you will still be blacklisted.

So let's say you have all these configured and you decide to send an email, it should arrive safely, right?

Wrong.

Simply sending an email from an IP that belongs to the domain is enough to get you blacklisted.

The final step before you're able to send an email is setting a PTR record that points your IP to your domain. This is only possible by speaking to your ISP. The reason why you need this record is because email servers perform a reverse DNS lookup on your domain name. A reverse DNS lookup is a DNS query for the domain name associated with a given IP address.

This accomplishes the opposite of the more commonly used forward DNS lookup, in which the DNS system is queried to return an IP address.

Reverse lookups are commonly used by email servers. Email servers check and see if an email message came from a valid server before bringing it onto their network. Many email servers will reject messages from any server that does not support reverse lookups or from a server that is highly unlikely to be legitimate. Spammers often use IP addresses from hijacked machines, which means there will be no PTR record. Or, they may use dynamically assigned IP addresses that lead to server domains with highly generic names.

There is no other way of getting this PTR record it can only be assigned by your ISP, thankfully there was no extra charge, but regardless I had to call up my ISP every day for 3 days before they approved it.

Once approved you also have to check your logs constantly, I think since owning my NAS there have been over 7000 attempts from people trying to get in, you must make sure you view these logs regularly and make sure your network is secure.

The cool thing tho is once you have this set up is that every domain you buy will have a working email, provided you set the MX record of your mail server as the mail server that has the PTR record.

Network Flow

The final frontier in my self-hosting journey was making sure my network was secure, I also wanted to put my Raspberry Pi to use as my NAS made it obsolete.

Okay, so I have this setup where my router is using Pi-hole as the primary DNS server for all connected devices. Pi-hole is set up with some specific upstream servers, and there's also Tailscale involved, along with a NAS running Unbound. It's a bit complex, but I'll try to make sense of it step by step.

First, let's understand what each component does:

Pi-hole: It's a network-wide ad block that protects your devices from advertising by blocking nasty internet trackers. Essentially, it acts as a DNS server that can filter out ads by not resolving domains known to serve ads.
Tailscale: This is a network connectivity tool that allows devices to connect securely over the internet as if they were on the same local network, essentially your own VPN. It has its own DNS server, which handles Tailscale-related lookups.
NAS running Unbound: Unbound is a validating, recursive, and caching DNS resolver. Running it on a NAS means that you have a local DNS server that can resolve domain names.

Router (192.168.1.1)
    |
    v
Pi-hole (192.168.1.3) 
    |-----------------------------------|
    |                                   |
    v                                   v
Tailscale (100.x.x.x)          NAS/Unbound (192.168.1.2)
    |                                   |
    v                                   v
Global DNS                      Local DNS (Split DNS)
                                + DHCP Server for Network

Pi-hole is the primary DNS server for all devices connected through the router.
Pin-hole's upstream servers are:
Tailscale's global DNS server (100.100.100.100): This handles Tailscale-related lookups and ensures connectivity for devices on the Tailscale network.
- NAS running Unbound (as DNS 2): Acts as a secondary resolver for redundancy and local queries.
Tailscale DNS:
Pi-hole is configured as the global resolver for Tailscale devices, which means that ad-blocking is enabled for devices on the Tailscale network.
- Split DNS: The NAS (running Unbound) is configured as a Tailscale split DNS nameserver for specific domain queries, like .tailscale or local domains.

So, how does this all work together?

Let's consider two scenarios:

For regular devices on the LAN:
These devices send their DNS queries to Pi-hole.
- Pi-hole checks its block list and either blocks the request if it's an ad or resolves the domain otherwise.
- For resolving domains, Pi-hole forwards the query to its upstream servers.
- Non-Tailscale queries might be sent to general upstream resolvers like Cloudflare or Quad9, but in this setup, it's specified to send Tailscale-related queries to Tailscale’s DNS server (100.100.100.100).
  - Additionally, local queries could be sent to Unbound on the NAS if needed.
For Tailscale devices:
These devices are configured to use Pi-hole as their global nameserver.
- This means that for regular internet queries, Pi-hole handles them, applying its ad-blocking filters.
- However, for specific domain queries (like .tailscale or other local domains), Tailscale is set up to use split DNS, routing these queries to Unbound on the NAS.

So, in essence, for Tailscale devices:

General internet traffic is still filtered through Pi-hole for ad-blocking.
Specific local domain queries are handled by Unbound on the NAS, providing quick and accurate resolution for those domains.

This setup is designed to provide a balance between ad-blocking for general web usage and efficient resolution of local or Tailscale-specific domains.

Thanks for reading, I hope you enjoyed reading this as I found this to be possibly the most boring post I think I have ever made

Take care

Regards,

pry