Metric</th>	Plausible.io</th>	Logs + GoAccess</th>	Δ factor</th></tr></thead>
Visitors</td>	32.1k</td>	76.9k</td>	x2.4</td></tr>
Pageviews</td>	44.5k</td>	468.6k</td>	x10.5</td></tr>
Bandwidth</td>	-</td>	16.6 GiB</td>	-</td></tr> </tbody></table> Just as Marko noticed, logs show much higher numbers of visitors and pageviews, likely due to crawlers and bots that get noticed in the logs but do not run javascript and therefore are not picked up by Plausible.</p> I could compare other metrics like referrers and top pages, but again, I suggest you read the post on the Plausible blog</a>.</p> I'd like to add that the logs can provide some information about bandwidth usage and which files are downloaded the most. This would allow you to make informed decisions when optimizing caching and file loading. Plausible can't help you with this data, one needs logs for this.</p> Qualitative data</h3> The experience with Plausible was more convenient than with GoAccess, as the website of the former loads in seconds whilst the latter took 3 minutes to process the logs and generate the results.</p> Conclusion</h2> Both methods have advantages and disadvantages. Plausible gives fast and precise results but potentially impacts page load (although minimally). Server logs don't impact page load, can provide bandwidth stats but inflate numbers due to traffic noise generated by search engines, crawlers and bots. Personally, I will continue using both for the foreseeable future.</p> Methodology</h2> Plausible</h3> Visit the Plausible.io</a> website and simply look at the website's stats.</p> Caddy logs</h3> Logs were collected using the following snippet in the Caddyfile:</p> `log { </span> output file /var/log/caddy/access.log { </span> roll_size 100MiB </span> roll_keep 10 </span> roll_keep_for 2160h </span> } </span>} </span></code></pre>` GoAccess</h3> As GoAccess cannot read Caddy logs directly, a small bash script is needed:</p> today_date</span>=</span>$</span>(</span>date</span> -u</span> +"%Y-%m-%d"</span>) </span>today_date</span>=</span>$</span>(</span>date</span> -u --date</span>=</span>"$</span>today_date</span> -30 day" +"%Y-%m-%d"</span>) </span>today_ts</span>=</span>$</span>(</span>date</span> -d </span>$</span>today_date</span> +%</span>s) </span> </span>goaccess </span><</span>(</span>zcat</span> -f</span> logs/access</span>* \| </span>jq</span> --raw-output </span>' </span> .request.remote_addr \|= .[:-6] \| </span> select(.request.remote_addr != "1.1.1.1") \| </span> select(.request.remote_addr != "2.2.2.2") \| </span> select(.ts >= '</span>$today_ts</span>') \| </span> [ </span> .common_log, </span> .request.headers.Referer[0] // "-", </span> .request.headers."User-Agent"[0], </span> .duration </span> ] \| @csv'</span>) </span>\ </span> --log-format</span>=</span>'"%h - - [%d:%t %^] ""%m %r %H"" %s %b","%R","%u",%T'</span> --time-format</span>=</span>'%H:%M:%S'</span> --date-format</span>=</span>'%d/%b/%Y' </span></code></pre> This was adapted from the bash script described by Alessandro</a> in this blog post</a>.</p> Optimizing the website's load performance 2020-06-05T22:47:21+00:00 My old webhosting</h2> When I started making websites back in 2010-ish (maybe even earlier than that, I don't remember), I used shared hosting as I did not have the slightest clue about how a server worked, let alone set one up for web hosting. About two-three years ago, I switched to Cloudways</a>, allowing one to host websites on a virtual private server (VPS) and still not require any actual knowledge about the inner workings of a server.</p> My new webhosting</h2> However, I've been managing my own private server for almost two years so I felt confident I could do the hosting myself. Hip as I am (I am not), I decided to go with a Caddy server</a> as a Docker container</a> on a VPS hosted by DigitalOcean</a>. For the low-traffic websites I currently maintain, this is largely sufficient.</p> I am in the process of moving each website one by one to the new hosting solution. It was time for this very website, yarmo.eu</a> and I thought to myself:</p> I should actually check if I gain any website load performance by moving to this new solution.</p> </blockquote> When asking the Fediverse, 70% predicted Caddy would perform better than Cloudways</a>.</p> Let's get testing</h2> I decided to use WebPageTest.org</a> to measure load perfomance. For each case described below, three measurements were performed and the median measurement is displayed and analyzed.</p> Cloudways</h3> First, a baseline measurement of my existing Cloudways solution.</p> Cloudways - overview</em></p> Cloudways - rating</em></p> Cloudways - waterfall</em></p> So the server returns the first byte of information after 480 milliseconds. Now, I should tell you that my website is based on Phug</a>, the PHP port of pug.js templating</a>. The page is rendered in real-time and apparently, that takes a little over 300 ms.</p> It is worth noting that any other metric is then dependent on how the website is programmed. Once Cloudways has sent over the data, it no longer has any influence on load performance.</p> The website if fully loaded after 923 ms. Good to know. About a second to wait for my website to load.</p> Over on the waterfall, we see a bunch of files being downloaded simultaneously after the HTML page is loaded. The largest asset to load is the profile picture.</p> Wait, what is that `F</code> over on the rating? Security is NOT in order! As it turns out, Cloudways does not handle security-related HTTP headers for you… I did not know that! They` `recommend setting these headers in a .htaccess file</a>.</p>` `Let this be a reminder to all of you: test your websites. One might learn a few tricks.</p>` `Anyway, can Caddy do better?</p>` Caddy</h3> Caddy - overview</em></p> Caddy - rating</em></p> Caddy - waterfall</em></p> Well, as it turns out, it's largely the same performance. First byte arrived after 459 ms, but I've ran it a few times and there's really little difference between Cloudways and Caddy.</p> BUT! Learning from my previous mistakes, I configured Caddy to set up all the correct headers and won't you look at that, A+</code> on the security score!</p> So that's it then?</p> Well, not really… I learned that my website wasn't running optimally because I forgot some basic HTTP headers. Did I forget more? In other words, can I do even better than this?</p> I've tried a lot of things, I'll just narrow it down to the two most important findings.</p> Caddy - inline most of it</h3> As it turned out, I had a few small SVG icons and some CSS files. I tried rendering them into the HTML page, so the data would be sent on the first data transmission and no separate requests were needed. For good measure, I also minified the CSS files which, for one file, reduced the size by 30%!</p> Caddy+inline - overview</em></p> Caddy+inline - rating</em></p> Caddy+inline - waterfall</em></p> On the waterfall above, you can clearly see the dank-mono.css</code> was not inlined but I tried multiple configurations, there was no real gain as the image also needed to load and took longer anyway. So, all in all, inlining the SVG and CSS content did little in this case.</p> Also, note the regression from A+</code> to A</code> on the security score. There was one header I couldn't quite get working properly so I had to disable that one, other than that, it's working better than it ever has.</p> What drew my attention for the final step was the B</code>. My server responds in within 480 ms and that it still not good enough for you, WebPageTest? Ok, have it your way.</p> What takes my server so long to respond? Well, obviously, it must be the templating. Can I improve the template? Perhaps. But as it turns out, I don't have to! Ever heard of caching?</p> As described on their website</a>, PHUG has support for caching and even calling an optimized version of their renderer. So I applied both caching and optimized rendering.</p> Caddy - PHUG optimization</h3> Caddy+PHUG - overview</em></p> Caddy+PHUG - rating</em></p> Caddy+PHUG - waterfall</em></p> Well, there it is!!! First byte of data arrived after a mere 173 ms, website is useable in less than half a second and all scores are A</code>!</p> That's the result I was hoping for. Now on my todo list:</p> Optimize the profile picture further or try SVG</li> Get all HTTP headers perfect</li> </ul> Any comments or recommendations/optimization? Please let me know</a>!</p> The Case of the Missing Entropy 2020-06-05T12:14:57+00:00 In computing, entropy is the randomness collected by an operating system or application for use in cryptography or other uses that require random data. (Source: wikipedia</a>)</p> </blockquote> Docker, are you still there?</h2> It all started when I got myself a new VPS server for serving web content. I have a more-than-capable server at home but I'd rather not use it for "uptime-sensitive" use-cases, the odd crash still takes it down from time to time. I know, a CDN…</p> Sticking with what I'm comfortable with, I decided to go with a docker setup with only a few containers:</p> caddy</a> as web server and reverse proxy</li> php-fpm</a> as PHP processor</li> a couple of others with minor significance</li> </ul> As per usual, I write my docker-compose.yaml</code> and it's all set. But not this time. At times, when I would change something in the yaml file and run docker-compose up -d</code>, it would do it immediately, as I would expect from all the times I've run it on my homelab. But sometimes, it would wait a minute or longer and only execute then.</p> I accepted this behavior a few times, but at some point, it had to be dealt with.</p> Investigating</h2> I noticed a few things. First, it did not seem to be due to a lack of computing resources. My Grafana</a> dashboard (with InfluxDB</a> as backend and Telegraf</a> as agent) clearly showed me that CPU usage was about 1% and RAM was about 30% full. No excessive DISK IO or NETWORK IO. So we are not overwhelming the system!</p> Additionally, while it was waiting to execute, I could open a new SSH connection and do other stuff. With one exception: any docker-related command would not execute.</p> Final clue: I could not ctrl-c my way out of a pending docker command execution, but if I would close the terminal, open a new one, connect via SSH and run any new docker command, it would still wait.</p> Final final clue: a minute later, I could run docker commands left, right and center without a single problem. Another minute later, it might do the whole waiting again. It was very… "Random". Wink, wink…</p> Have you figured it out yet? I hadn't.</p> Researching</h2> With this information, I was confident enough to start searching online and I came across this github issue</a> fairly quickly:</p> "docker-compose often takes a long time to do anything"</p> </blockquote> That sounds about right!</p> A few comments in, it was suggested</a> to run the following command: cat /proc/sys/kernel/random/entropy_avail</code>.</p> On my VPS, this returned 52</code>. Whoopsie…</p> Entropy</h2> For those of you who don't know what (computing) entropy is, here's the wikipedia article</a> for it. In short: computers are terrible at coming up with random numbers (just like humans! Topic for another day), which many applications require for their proper working.</p> Our operating systems have a clever way to solve this: take all input that is NOT generated by the computer itself and use that as "randomness". For example, a computer doesn't know in advance how you are going to move the cursor or which keyboard button you will press. The operating system takes these inputs, processes them to "extract the randomness" and stores it in the entropy pool</code>.</p> Any application needing some randomness can request some random data from the entropy pool</code>. Maintaining sufficient entropy is therefore a challenge in itself: process enough random data to keep up with the demand.</p> Apparently, Docker is an application that requires randomness. But how have I never encountered this issue before?</p> VPS and entropy</h2> On both my desktop computer and homelab, the entropy available is around 4000</code>, which is perfect. They are able to maintain this entropy because of all the sources of randomness available to them. Mouse and keyboard inputs, processes running in the background, etc.</p> Now, let's take the VPS as a counter example. These things are made to be fully reproducible: every time you boot one up, they are expected to run in the same way. They are also very sealed off from the host system for security reasons: I cannot read core temperature values for my VPS. They don't have "true hardware", they get portions of hardware, shared with other VPS instances. Except for my SSH connection, the VPS has no mouse or keyboard inputs.</p> In other words, VPSs are severely lacking in sources of entropy. That is why the entropy available was only 52</code> and why docker stalled: it had to wait for sufficient randomness to occur.</p> More information on VPS and entropy from DigitalOcean</a>.</p> The remedy: haveged</h2> There is a way to remedy the situation: haveged</a>. Having only discovered it last night, I do not fully understand it yet but from what I have read, it is a pseudorandom number generator (PRNG) that fills the entropy pool</code> with "pseudorandomness". Installing haveged</code> immediately solved my issue, all docker commands were running instantly again.</p> Can you tell when I installed haveged?</em></p> Caveat: pseudorandomness</h2> There is a downside to this: PRNGs are NOT random. Wikipedia article on PRNGs</a>. PRNGs generate numbers that appear random but are fully deterministic: run the exact same algorithm twice, and you'll get the same "random" numbers. Therefore, VPSs may not be the perfect solutions to perform entropy-heavy tasks such as cryptography: a cryptographic key generated with pseudorandom numbers is far less secure than one generated with truly random numbers.</p> LunaSea: FOSS FTW 2020-05-22T19:47:38+00:00 #100DaysToOffload >> 2020-05-22 >> 023/100</code></p> Out with the old</h2> A couple of weeks ago, I finally discovered a FOSS alternative for nzb360, a great app for managing Plex, Radarr, Sonarr, etc. I wish to have kept using nzb360, but unfortunately, the app relies too heavily on Google Services and though I have paid for it, I can no longer use it as my LineageOS phone can't process purchases made on official Google Android phones.</p> In with the new</h2> Named LunaSea</a>, it can do anything it should, (manage Sonarr, Radarr, Lidarr, NZB clients), it looks fantastic, it's available for both Google Android and iPhone and, of course, it's FOSS</a>.</p> Only thing I'm missing is a donation button. And a fediverse account :)</p> Introduction to PiHole 2020-05-10T23:24:04+00:00 #100DaysToOffload >> 2020-05-10 >> 016/100</code></p> PiHole</a> is almost ubiquitously present on every list of services people could/should selfhost. And rightfully so, it is easy to set up and extremely useful on a daily basis. It blocks ads on almost all websites on all the devices in your home without the necessity of installing anything on them. It will also stop some devices from communicating with their parent companies behind your back.</p> How it works</h2> To understand how PiHole does its thing, we need a quick introduction into how DNS works, the system that makes sure we can visit websites even if they are located on the other side of the world. The problem DNS solves is that the URL you use to visit a website doesn't tell your device anything about the physical location or IP address of the server that hosts the website.</p> If you wish to visit a website, say yarmo.eu</a>, you enter that address in the top bar and hit enter. Your browser will then ask your router to get this website for you. If this is the first time you visit this website, your router doesn't know yet where the server is located, so it asks a DNS server in geographical proximity, usually the DNS server of your ISP.</p> If this DNS server knows the IP address of the server, it will be relayed back to your device which will now ask that server directly for the content of the website. If the DNS server doesn't have this information, it will ask another and so forth until the IP address of the host server is found.</p> As we established above, your router contains a DNS router. However, this can almost always be delegated to another DNS server in your home. That's where PiHole comes in. Instead of your router trying to figure out where the website server is located, it will ask PiHole to do so.</p> But PiHole has a trick up its sleeve: it has a built-in database of hundreds of thousands of URLs that are associated with ads and when they are requested, PiHole simply ignores them.</p> So you want to visit coolsite.com</code>? Fine, PiHole will get you that website. Now, coolsite.com</code> suddenly wants to load an ad from ads.gafam.com</code>? The computer asks the router, the router asks PiHole, PiHole knows this URL is used to serve ads and will block that request, giving you a website without ads. Awesome!</p> Something you want to say?</h2> Meanwhile, you are listening to music using a wireless speaker in your living room from a made-up brand "NOSON". What you don't know is that this device is continuously sending messages to the company containing information about the music you play and more. PiHole knows this and as soon as the speaker requests to send a message to metrics.noson.com</code>, PiHole says no.</p> That's how PiHole blocks ads AND protects your privacy.</p> Dedicated hardware</h2> Dedicating hardware to PiHole is advised but the hardware can be as simple as a Raspberry Pi Zero</a>. The reason it is advised to use dedicated hardware is because if your PiHole crashes, there's no more internet in the home until you get the PiHole working again. It way to prevent this situation from happening is to always have two PiHoles running on separate hardware and telling the router about both PiHoles.</p> Second DNS server?</h2> Oh, and while we're on the subject: do not put any "fallback" DNS servers like Google's or Cloudflare's in the second DNS server field on your router. Unfortunately, it doesn't work like a fallback, all routers will simply divide the workload over the two DNS servers. This means that if an outside DNS server is put in second place, it will receive DNS calls even if the PiHole is fully functional.</p> Having a proper DNS fallback server is difficult to set up, so best would be to use two different PiHole instances. Unless, of course, you don't mind a small period of internet loss and you are always nearby to fix the situation.</p> Caveats</h2> Unfortunately, ads on video platforms like YouTube will not be blocked. This is because they serve the ads on the same domains as they serve the main content, meaning that they don't have a ads.youtube.com</code> or something similar. Therefore, PiHole cannot block the ads. As there are a few of these edge cases, it is always recommended to use PiHole in conjunction with on-device ad blockers like uBlock Origin</a>.</p> Final words</h2> Really, there are few reasons to not get PiHole into your home and the benefits vastly outweigh the challenges (IMHO). It is also a great start on a journey of selfhosting more services and realising that one can be independent of major corporations to some degree.</p> Traefik migrated to v2 2020-05-09T23:01:34+00:00 #100DaysToOffload >> 2020-05-09 >> 015/100</code></p> Last september, traefik</a> received its big version 2 update</a>. I was very excited about TCP routers and the newly implemented middlewares. Can't have been more than a few days later that I tried to migrate my homelab to the new version. I remember being annoyed by the lack of a proper migration guide. Sure, it's possible that I didn't look good enough, but I searched for it for a few days without results. I tried using the new documentation and failed, everything crashed and could not get it working. As I did not have the time to do much more extensive research and also, I needed the selfhosted services on a daily basis, so I left it.</p> Until today. The whole migration took me a little over three hours and I learned quite a bit on the way. Also, the migration guide</a> has helped quite a bit. If this was updated since September, great article. If not, still a great article and I really did not take the appropriate amount of time to prepare my migration.</p> Easy steps</h2> The first thing I did was a general search-and-replace for the docker labels (both routers and services). What was traefik.frontend.rule=Host:xyz</code> now is traefik.http.routers.router0.rule=Host(``xyz``)</code>. What was traefik.port=80</code> now is traefik.http.services.service0.loadbalancer.server.port=80</code>. Quite a bit longer and more cumbersome, but in the end, more extensible.</p> The traefik.docker.network=xyz</code> is now unnecessary in most cases as you can define a default network in the traefik.toml</code> file. Speaking of which, you can now work with a YAML file. It's not to everyone's taste, but I will switch to it in the future when I have some more time.</p> The traefik.toml</code> still needed quite the makeover, but everything is will explained in the migration guide</a> and the reference page</a>. Content-wise, I changed little, it's just that the syntax is different. Notable changes are the domains for which certificates are needed now are declared in the entrypoints</code> section instead of the acme</code> section and the file</code> section can no longer include the "router/service declarations", they belong in a separate file.</p> Pitfalls</h2> Doing this resulted in a non-functional state. Two different possibilities: either the container could not be redirected to the correct service resulting in a 404, or the redirect was correct but without correct certificate. That first possibility was mostly on me: containers are now no longer exposed by default and I forgot to add traefik.enable=true</code>. Mind you, I always set traefik.enable=false</code> when I didn't want a container exposed and still do.</p> However this did not solve the issue for all the containers. I suspect there's still some trickery I need to do in case of using multiple routers. I tried explicitly specifying the service</code> for the different routers but that wasn't the solution.</p> As for the other issue, the solution was simple but finding the source was quite hard: as it turns out, I renamed the certificateResolver</code> to something other than default</code>. If such is the case, then containers will NOT automatically use it for their certificates. Adding traefik.http.routers.router0.tls=true</code> and traefik.http.routers.router0.tls.certresolver=mycertresolver</code> to each container solves this issue.</p> Todo</h2> One thing I haven't got working yet is using the providers.file</code> provider. I tried to mimic the container labels but to no avail. Yet.</p> Update 2020-05-12</strong>: I fixed the providers.file</code> issue. Remember kids, always read the documentation well. It turns out, I missed the line that starts with </code> in the code below.</p> [</span>http</span>.</span>services</span>] </span> [</span>http</span>.</span>services</span>.</span>Service01</span>] </span> [</span>http</span>.</span>services</span>.</span>Service01</span>.</span>loadBalancer</span>] </span> [[http.services.Service01.loadBalancer.servers]] </span> </span>url </span>= </span>"foobar" </span></code></pre> Varken: Plex monitoring solution 2020-05-05T21:49:58+00:00 #100DaysToOffload >> 2020-05-05 >> 011/100</code></p> Today, I discovered Varken</a>, a neat solution to monitor your Plex ecosystem (including Sonarr, Radarr, etc.) and store the data in your InfluxDB instance. This solution is a great addition as I can now make Grafana or Chronograf dashboardz encompassing both server metrics and Plex metrics. The reason this is important is that I have a relatively low-power server (NUC) and a single stream on Plex can have a noticeable impact on the CPU usage.</p> Varken requires a Tautulli</a> instance to collect the data from as well as a MaxMind</a> API key which unfortunately isn't optional. I run all software mentioned in this post in separate docker containers.</p> Also, 11th post for #100DaysToOffload today and 11 is my lucky number :)</p> Selfhost email… But should you? 2020-05-03T19:46:47+00:00 #100DaysToOffload >> 2020-05-03 >> 009/100</code></p> Yesterday, I wrote about how you can</strong> selfhost your very own email server</a>. Shortly after publishing the post, it was pointed out to me that are very reasonable drawbacks to doing this. So today, let me give you my answer to the question of whether you should</strong> selfhost your email.</p> Relying on hardware</h2> Firstly, I mentioned in that article that although I have two domains on my selfhosted server, I still fall back to a protonmail address for the most important stuff like banking and governmental services. So what's the point of selfhosting then, if I do use third-party email addresses? Well, what I failed to mention was that in the long term, yes, I do want everything selfhosted.</p> When I started my email hosting adventure, I was very cautious. Only months before did I start my own homelab and as it turned out, that had the tendency to crash every so often, making it a no-go for email hosting. I resorted to use a VPS while getting my homelab sorted out. This worked great and still does, but at the time, you can imagine I was still discovering the DNS parameters, the reputation handling (more on this later). Also, what were the consequences of running a VPS for 24/7? I could not commit to using the selfhosted email for anything more than experimentation.</p> Fastforward about a year and the VPS has held up greatly, the email software has never crashed or acted against my expectations. It has received regular updates and never failed once during one. Meanwhile, my homelab has proven to be extremely reliable and with the upcoming hardware upgrade, I expect even less irregularities than I do now. Soon enough, when all the stars align and I figure out how to make recoveries as fast on my local hardware as I can on a VPS (make a new instance based on a daily snapshot and voilà!), my email server will be transferred to my homelab and I will use it for everything.</p> The pain of administering an email server</h2> Secondly, I've also heard of people not resorting to selfhost their email because of the fragility of the underlying processes and if one thing is slightly out of tune, the whole email server stops working. Although I've toyed around with most of the individual processes like dovecot at the beginning to understand what they do and how they work, I haven't touched a single one of them in almost a year. mailcow.email</a> is just that good. I've played around with the settings and it won't stop working. Meanwhile, I get an antivirus, spam monitoring, those handy "+topic" email filtering. I'd like to try out Mail-in-a-Box</a>, mostly because it is also recommended by PrivacyTools</a> but I have no incentive too. My current solution just works great for me.</p> The reputation of a server</h2> Lastly, I need to address a IMO bigger problem: reputation. If other servers don't trust you, your emails may easily be thrown into the spam folder of the recipient or even rejected. The main reason for this is to fight spam: mass email spammers usually operate from unknown IP addresses. Unfortunately, this hurts the selfhosters. So, before you have even installed your email server software, you are already mistrusted by simply not using the big servers like Gmail and Hotmail. And indeed, when I started, most of my emails landed in spam.</p> This got greatly improved simply by using an email relay; in my case, mailgun</a>. These paid-for (but often with free tier) services are a lot more trusted since mailgun will do spam prevention on their end, so letting them send your emails for you is a great improvement. And even with the recently reduced free tier</a>, I don't send nearly enough emails to come close to the free tier quota.</p> However, it still happens that my emails are treated as spam so I often do follow ups via other channels of communication. Another issue may be that the IP address you were given already has a bad reputation caused by a previous owner: this is difficult to find out and even harder to fix. DDG-ing improve email server reputation</code> yields many articles but read a handful of them and soon you'll realise it's really really hard to improve it. There's no central repository, no forms. Getting a mistrusted IP address can quickly suck all the fun out of having your own email server.</p> Answering the question</h2> So, should you</strong>? This depends on how willing you are to be independent of third-party email services and how much you are willing to put up with. I started naïvely and had to answer this question along the way while experimenting. By now, my personal answer to this question is: yes. I see the benefits and drawbacks. I'm not sure if it's the usage of mailgun, or me sending mails to family and friends and then asking them to tell their services to not mark it as spam, but most of my emails are properly received nowadays. Also, I have managed to improve my infrastructure, I can rely on the hardware (and soon on the emergency recovery mechanisms) and will soon migrate my email server so it's nicely at home.</p> Hosting your own email server is not easy and requires your full dedication. And with many upcoming trusted and privacy friendly email services</a>, it may not always be the right tool for the job.</p> Selfhost email 2020-05-02T16:20:35+00:00 #100DaysToOffload >> 2020-05-02 >> 008/100</code></p> Yes, you can selfhost email. And you should, if and ONLY if you feel comfortable with maintaining a linux server. I'm not a pro at all, but I've been doing it for almost two years, I know where to find my logs, I know how to find the correct answers on stackoverflow and troubleshoot a less-than-functional system.</p> So don't start with this, but eventually, soon enough, you can selfhost your email.</p> Because email is important to me, I have chosen to not host it at home, any network issue could prevent emails from coming in. Granted, the sending server will usually retry for 24 hours until the email is actually received on your side so small errors are forgiven, but still, I've opted for a dedicated droplet on digitalocean, though any VPS will do.</p> And then, follow the instructions on mailcow.email</a> and you're set. SSH in once a week on your VPS to run the updater. The administration side has plenty of features for advanced administration of the email server and the included webclient is the awesome SOGo.</p> If you want to make sure your email server is as trusted by other servers as possible, your emails are sent as securely as possible and your experience with other email clients is as smooth as possible, please check out my post on email server DNS settings</a>.</p> With all that being said, I still use a protonmail address for critical websites and services like governmental services and banking, because whatever happens, I need to make sure that I really receive these emails. On my selfhosted email server, I use two domains: one which I share with the world and with websites for logins and one that I keep private and only use for direct communication with other people. I have yet to experience a single minute of outage, credits to digitalocean and the people behind mailcow.</p> ---UPDATE---</p> After a fair comment</a> on the fediverse, I have written a follow-up post</a> to address a few more critical points like server reputation and how I deal with that.</p> The People's Web 2020-04-30T08:53:37+00:00 #100DaysToOffload >> 2020-04-30 >> 006/100</code></p> The day has long past that we should have started worrying about the openness of our web. It was only a matter of time before censorship and individual tracking would seep into the web of the western world. It pained me to see the levels of state interference in foreign countries, but at least, to me, that was something I only read about in the news, it was then unimaginable that this would happen in the short term in the "free world".</p> Now, in these trying times, states and big corporations may see it more than justified to track individuals and adjust the information we receive. I do not question their motives: "we are in this together". It takes the world to fight this pandemic, and the next one, and the one after that.</p> But the question on a lot of minds is: "how do we come back from this?" The hardest part is for the people to accept they are being tracked and their information filtered, so big corporations did it with shady and hidden tactics to avoid this confrontation. We are past this: YouTube has announced it will ban all content that does not conform to WHO and countries everywhere are building apps to track our health and social interactions. Again, this could prove to be what humanity needs right now, but what about after? Is there even an after?</p> Fortunately, we all have the power to make a few changes to improve our online well-being: change social networks, don't rely on corporations, self-host as much as you can. I will be dedicating a large number of posts on this topic. Self-hosting is not hard, it just takes a little effort to get started.</p> As a friendly reminder, this website (blog included) has no tracking whatsoever: I do not care who you are, where you are from or with how many you are reading this post.</p> Selfhosted email: DNS records 2020-02-18T18:39:54+00:00 When selfhosting email, an essential element to get right are the DNS records. Some are absolutely mandatory for email to work, some build trust and some just make life easier. Here's an overview of how I set up DNS for my personal mail server.</p> My setup</h2> I have a VPS server running mailcow</a> and two domains: one linking to the mail server, admin page and the web client (let's call this mail.server.domain</code>), and the other just being a email domain used in the email address (let's call this public.domain</code>, so the email address would be hi@public.domain</code>). This way, even if you know the email domain, you don't directly know server domain. Granted, one could find this out by looking at a few DNS records.</p> The benefit is that I can host email for multiple domains, as long as they all point to mail.server.domain</code> by using the correct DNS records.</p> Please note that using a single domain is just as easy, as mail.server.domain</code> and public.domain</code> will simply be the same. Another scenario for which you could use the 2-domain setup is when you want your email address to be hi@public.domain</code> (without subdomain) but wish to put the mail server (and/or web client) on mail.public.domain</code> (with subdomain).</p> I use DigitalOcean</a> as my VPS and DNS server.</p> Mandatory DNS records for server.domain</h2> A records</h3> A records link domain names to IP addresses. When you want to use the admin page or web client provided by your email selfhosting software, the browser needs to know the IP address of the server/VPS and that is what the A record is used for. A records are not used by mail servers when sending or receiving emails.</p> TYPE HOSTNAME VALUE TTL </span>A mail.server.domain 1.2.3.4 3600 </span></code></pre> Mandatory DNS records for public.domain</h2> MX records</h3> MX records tell other mail servers where to actually send the emails. In my case, my email address is hi@public.domain</code> but my mail server is located at mail.server.domain</code>. Other mail servers look at the address, see public.domain</code> and will assume this is our mail server. We use MX records to direct the emails to mail.server.domain</code> instead.</p> TYPE HOSTNAME VALUE PRIORITY TTL </span>MX public.domain mail.server.domain 1 14400 </span></code></pre> Optional DNS records for public.domain</h2> A and MX records is all you need to get a functional email address. However, for ease of use and good reputation/trust, a few additional DNS records are recommended.</p> SRV records (ease of use)</h3> SRV records are used to link specific protocols to specific domains and ports. Just like how MX records tell other mail servers to direct their mails to your mail server on a different domain, the same must be done for mail clients. Say you want to use Thunderbird (or any other mail client) to access your emails. You will log in with your address (hi@public.domain</code>) and password in Thunderbird, and it will then assume your mail server must be located at public.domain</code>. It will not find it there, warn you about this and you will have to manually enter your IMAP and SMTP server details. If you have set up SRV records, Thunderbird will automatically detect the correct server location (mail.server.domain</code>) and save you some hassle.</p> TYPE HOSTNAME VALUE PORT PRIORITY WEIGHT TTL </span>SRV _imap._tcp mail.server.domain 143 1 100 14400 </span>SRV _imaps._tcp mail.server.domain 993 1 100 14400 </span>SRV _pop3._tcp mail.server.domain 110 1 100 14400 </span>SRV _pop3s._tcp mail.server.domain 995 1 100 14400 </span>SRV _submission._tcp mail.server.domain 587 1 100 14400 </span>SRV _smtps._tcp mail.server.domain 465 1 100 14400 </span>SRV _sieve._tcp mail.server.domain 4190 1 100 14400 </span>SRV _autodiscover._tcp mail.server.domain 443 1 100 14400 </span>SRV _carddavs._tcp mail.server.domain 443 1 100 14400 </span>SRV _caldavs._tcp mail.server.domain 443 1 100 14400 </span></code></pre> TXT records (good reputation)</h3> TXT records are simply messages that provide additional information. Here, TXT records are used to tell other mail servers more about your own mail server in order to build some trust between them: these records are a useful tool against spoofing where bad actors try to impersonate you and pretend you are sending the bad emails they are sending.</p> TYPE HOSTNAME VALUE </span>TXT @ "v=spf1 mx ~all" </span>TXT dkim._domainkey "v=DKIM1;k=rsa;t=s;s=email;p=..." </span>TXT _dmarc "v=DMARC1;p=reject;rua=mailto:admin@public.domain" </span></code></pre> Detailed information on these records can be found here</a>, but in short:</p> SPF records tells other mail servers that only the specified mail server (in this case, mx</code> which points to mail.server.domain</code> via the MX record) is allowed to send emails for your email domain (in this case, public.domain</code>);</li> DKIM applies a virtual signature to all your sent emails and other mail servers use the second TXT record above to validate that signature;</li> DMARC records tell other mail servers what should happen to emails that fail the SPF and DKIM policies; the record above states these emails should be rejected and a notification sent to admin@public.domain</code>.</li> </ul> The DKIM record contains a cryptographic key (replaced above by ...</code>). In my case, this key was generated for me by mailcow and is unique for each email domain.</p> Please note that having the above TXT records does not guarantee that other servers trust you immediately: your emails are still likely to end up in spam folders at first. Using intermediaries like mailgun</a> can help with avoiding the spam folder. More on that in a later blog post.</p> References</h2> Mailcow has their own recommended DNS records guide</a> which, in conjunction with their admin page, should make setting up DNS records a breeze.</p> This guide</a> has a lot of in-depth information about the SPF, DKIM and DMARC records (TXT records above).</p>

Yarmo - selfhosting

Matrix Synapse: migrating from Cloudron to ansible

2022-06-11T10:37:13+00:00

Maybe, like me, you tried to selfhost a Matrix Synapse</a> server, miserably failed because it's just not quite that easy and then settled for Cloudron's Synapse app</a> because it just works. Pay a bit more, worry a little less.

Sure, it works but you still introduced a middle man in your pristine homelab setup and the itch never goes away to get rid of it.

Time for round 2.

The ansible playbook</h2>
The last few years, I have been slowly learning how to work with ansible</a> and ever since I found the matrix-docker-ansible-deploy</a> project, I knew this would be my chance for redemption.
So, I found myself a spare machine (SBC, NUC, VPS, all should work), installed Debian 11, enabled SSH and went to work. Here's what I did to finally make the Cloudron migration happen.

Decommissioning Cloudron</h2>
!Read through the whole guide first! Understanding why you do certain things will help you do those things correctly or do them do differently: just because I did them a certain way doesn't mean it's the best way for everyone.
First, I moved some important files from the Cloudron server to my new server. There are two folders of interest, both can be obtained by visiting the Cloudron admin panel, going to the settings for the Matrix-Synapse app and clicking on `Storage</code>.`
`In the /home/yellowtent/appsdata/<APP-ID></code> folder, you'll find the postgresqldump</code> file that was generated during the last Cloudron app backup — so make sure to run a backup right before migrating to have the latest data!`
`In the /mnt/data/apps/<APP-NAME>/data</code> folder, you'll find the all-importand media_store</code> folder.`
`Additionally, in the /mnt/data/apps/<APP-NAME>/configs</code> folder, you will find the homeserver.yaml</code> config file for the Matrix Synapse server — use this for inspiration for the new one but more importantly, make a note of the database/args/user</code> value. There is also the signing.key</code> file.`
`Using your method of choice (rsync, wormhole…), copy the postgresqldump</code> file, the media_store</code> folder and the signing.key</code> over to the new server in — for example — the /migration</code> folder.`
`This is the moment to power down the Cloudron machine (or simply the app if you wish to keep Cloudron running) and update the` `DNS records</a>.`

Fixing the database dump</h2>
Before we touch ansible, we need to go our postgresqldump</code> file and replace all instances of the previous database user (the one you found in the homeserver.yaml</code> on the Cloudron server, it should look like user1a2b3c4d5e6f7g8h9i</code>) with matrix</code>.</p>
According to the Importing an existing Postgres database</a>, this should be synapse</code> and not matrix</code>. I ran into permission issues when using synapse</code> but that may have been due to a configuration error I made elsewhere. Feel free to attempt with synapse</code>.</p>
Setting up the new server with ansible</h2>
I am going to skip the majority of the ansible process here, namely the Configuring the Ansible playbook</a> part and only focus on what is relevant when migrating from a Cloudron instance. There is a special Importing an existing Postgres database</a> but I had to change the steps a bit to make them work for me.</p>
I followed the Configuring the Ansible playbook</a> steps to obtain my ansible configuration file.</p>
In the end, I added to vars.yml</code>:</p>
# Set up synapse database connection
</span>matrix_synapse_database_user</span>: </span>'matrix'
</span>matrix_synapse_database_password</span>: </span>'SUPERSECRETPASSWORD'
</span>matrix_synapse_database_database</span>: </span>'matrix'
</span>
</span># Set up postgres
</span>matrix_postgres_connection_username</span>: </span>'matrix'
</span>matrix_postgres_connection_password</span>: </span>'SUPERSECRETPASSWORD'
</span>matrix_postgres_db_name</span>: </span>'matrix'
</span></code></pre>
If you chose in the previous section to try and import the database dump into the synapse</code> database instead of the matrix</code>, make sure to update those values here.</p>
Let's let ansible set up everything on the server:</p>
ansible-playbook</span> -i</span> inventory/hosts setup.yml</span> --tags</span>=</span>setup-all</span> -K
</span></code></pre>
Do not start just yet! First, let's import the database dump:</p>
ansible-playbook</span> -i</span> inventory/hosts setup.yml </span>\
</span>  --extra-vars</span>=</span>'server_path_postgres_dump=/migration/postgresqldump' </span>\
</span>  --tags</span>=</span>import-postgres</span> -K
</span></code></pre>
If there weren't any errors, let's import the media_store</code> folder as well:</p>
ansible-playbook</span> -i</span> inventory/hosts setup.yml </span>\
</span>  --extra-vars</span>=</span>'server_path_media_store=/migration/media_store/' </span>\
</span>  --tags</span>=</span>import-synapse-media-store</span> -K
</span></code></pre>
If still no errors, great! Now let's take a look at that signing key we copied over. What I should have done was follow the instructions here</a> to add the previous signing key to the list of old signing keys. But I didn't know of these instructions when performing the setup, so I simply used the old key to overwrite the signing key ansible had generated and stored in /matrix/synapse/config/matrix.your.domain.signing.key</code>. Not as elegant, I admit, but it works.</p>
It's time to start the server (and run the setup again for the new signing key):</p>
ansible-playbook</span> -i</span> inventory/hosts setup.yml</span> --tags</span>=</span>setup-all,start</span> -K
</span></code></pre>
Done!</p>
Again, I skipped a lot of important steps like setting up a reverse proxy — this playbook includes Nginx</a> — so make sure to read all the documentation</a> to end up with a fully functional instance.</p>
Something went wrong</h3>
Surely, something went wrong in one of the steps above, it happens.</p>
If something went wrong during the importing of the postgresqldump</code>, you can't just repeat the step as postgres will now complain that some import steps were already performed (see the Importing an existing Postgres database</a> guide).</p>
I followed the exact steps they propose. So, on the new server, run:</p>
systemctl</span> stop matrix-postgres
</span>rm</span> -rf</span> /matrix/postgres/data/</span>*
</span>systemctl</span> start matrix-postgres
</span></code></pre>
Then, back on the ansible controller, run:</p>
ansible-playbook</span> -i</span> inventory/hosts setup.yml </span>\
</span>  --tags</span>=</span>setup-postgres</span> -K
</span></code></pre>
You now have an empty database ready for a fresh import!</p>
Checking the import process succeeded</h3>
To check if the data was imported correctly, here's how to log into the database and query the user table:</p>
/usr/local/bin/matrix-postgres-cli
</span># List the databases
</span>\l
</span># Connect to the matrix database
</span>\c </span>matrix
</span># Query the users table
</span>select </span>*</span> from users</span>;
</span></code></pre>
If that last query returns the list of users you expect to see, we should be good! Well, almost.</p>
Resetting passwords</h2>
For some explanation that is as of yet beyond me, the old passwords won't work. That is, we haven't changed the passwords during the import process, but we also cannot log in as the server will now complain the password is incorrect.</p>
Luckily, we can simply reset a user's password to fix this:</p>
ansible-playbook</span> -i</span> inventory/hosts setup.yml </span>\
</span>  --extra-vars</span>=</span>'username=USER password=SUPERSECRETPASSWORD' </span>\
</span>  --tags</span>=</span>update-user-password</span> -K
</span></code></pre>
This user can now log in again.</p>
If like me, you host a Synapse server for a small number of people that trust you, this step is not an issue. If you host for a lot of people, I am not really sure how to proceed. Hopefully someone can help out here, explain what step I missed to make passwords work after the migration and I can update the guide accordingly.</p>
Conclusion</h2>
There you have it, this is what I did to obtain a working fresh Synapse server with all the data from the Cloudron server. Cloudron was nice to work with but I am glad I have everything working again without the need for a middle man.</p>
So now… dendrite when?</p>



InfluxDB 2 migration
2022-05-20T18:36:41+00:00
What is InfluxDB?</h2>
InfluxDB</a> is the MIT licensed</a> time series database</a> of my monitoring stack of choice, the so-called TICK stack</a>, consisting of Telegraf</a> (data collection agent), InfluxDB</a> (time series database), Chronograf</a> (charts and dashboard interface) and Kapacitor</a> (data processing engine).</p>
My history with InfluxDB</h2>
I started my homelab somewhere in 2017-2018. When about a year in I faced some random crashes and reboots which I did not manage to understand or fix, I first turned to netdata</a> before settling with the TICK stack.</p>
While I did eventually solve the random crashes, an obvious issue surfaced: if I want to monitor a crashing server, I probably shouldn't host the monitoring stack on that same server.</p>
And so, my setup eventually converged to the one I use today: a homelab and a few VPSs, all monitored by a TICK stack installed on a dedicated VPS.</p>
InfluxDB 2.0</a> was released in November of 2020 and I remember this moment clearly: I updated my InfluxDB docker container, I noticed everything immediately broke, I read up on what a gargantuan update this 2.0 release was, I said "nope" and immediately reverted to 1.8.x.</p>
Yesterday, I decided it was finally time to sit down and calmly move to InfluxDB 2(.2.0).</p>
InfluxDB 2, or the Death of the TICK stack</h2>
Please refer release notes for InfluxDB 2</a> to see for yourself what has changed.</p>
One of the most notable changes is that InfluxDB 2 now has a interface with graphs and dashboards (replacing Chronograf's functionality) and can send alerts based on data (which previously required Kapacitor). While I am sure Chronograf and Kapacitor can still be used together with InfluxDB 2, they no longer need to and indeed, I have now removed these services from my monitoring stack.</p>
InfluxDB 2, the migration</h2>
The InfluxDB 2 dockerhub page</a> has clear instructions on how to migrate from 1.x to 2.x when using containers and so I did. And while the container logs showed the migration process was successful, I had this weird issue where when I would try to then log in on the web interface, I was greeted with a "fresh install" screen, suggesting the InfluxDB instance was in fact not aware of the migration.</p>
And indeed, after following the instructions of the "fresh install" screen, everything was working fine but no trace of any migrated data :(</p>
If my historical InfluxDB data was in any way valuable to me, I would have put more effort into migrating all of it to InfluxDB 2.</p>
But it wasn't.</p>
InfluxDB 2, the non-migration</h2>
And thus, in a bit of an anticlimax to this whole situation, I rm -rf</code>ed the whole data directory and started anew with a fresh InfluxDB 2 instance. No more Chronograf, no more Kapacitor. All servers have updated Telegraf configurations and I am currently rebuilding the dashboards in InfluxDB's web interface.</p>
And I also still need to rebuild my Grafana dashboards to make use of Flux instead of InfluxQL. I still have things to do before I can fully enjoy my monitoring solution again.</p>


playerctl: get currently playing music
2022-05-03T14:35:49+00:00
TLDR</h2>
To get information about music currently playing on the computer, run:</p>
playerctl</span> metadata</span> --all-players --format </span>'{{ status }}: {{ artist }} - {{ title }}'
</span></code></pre>
A wide variety of media players are MPRIS-enabled and can be queried using the above command, including media players running in the browser such as Airsonic</a>.</p>
playerctl on man.archlinux.org</a></p>
Explanation</h2>
For a long time, I've been using Airsonic</a> as my media player, handy if you are into selfhosted services and despise streaming platforms with a passion.</p>
Since I also do live streaming</a> and like to play ambient music in the background, I attempted to build a little "Now playing" widget for on stream.</p>
Sadly, no luck: Airsonic doesn't have an API.</p>
After a lot of trying out different clients and hosting additional services, I settled on syncing my music collection from the NAS to my computer and then playing music through mopidy, as I could then query the currently playing music using MPD:</p>
# Snippet will be added soon, currently away from computer
</span></code></pre>
Not my favorite solution but it works.</p>
In comes this guy DeavidSedice</a> and he tells me I can get that same information from most existing media players with a single line of bash:</p>
# The command as he sent it to me
</span>playerctl</span> metadata</span> --all-players --format </span>'{{ status }}: {{ artist }} - {{ title }}               ' </span>| </span>grep</span> Playing </span>| </span>tr </span>'\n' ' '
</span></code></pre>
Why did I never hear about this before? A life changer! And it even works with Airsonic running in the browser.</p>
This tool makes it quite trivial to write a bash script that loops the command, writes the output to a text file and have OBS display it on stream.</p>


Traefik: managing both HTTP and HTTPS connections separately
2020-07-28T00:00:00+00:00
Introduction</h2>
If you have used the wonderful Traefik</a> before to route all your traffic to different docker containers or other services, you'll know how easy it is to add SSL certificates and secure HTTPS connections, simple and free thanks to their integration with Let's Encrypt</a>.</p>
You will first need the following lines in the traefik.toml</code> file:</p>
[</span>entryPoints</span>]
</span>  [</span>entryPoints</span>.</span>web</span>]
</span>    </span>address </span>= </span>":80"
</span>    [</span>entryPoints</span>.</span>web</span>.</span>http</span>]
</span>      [</span>entryPoints</span>.</span>web</span>.</span>http</span>.</span>redirections</span>]
</span>        [</span>entryPoints</span>.</span>web</span>.</span>http</span>.</span>redirections</span>.</span>entryPoint</span>]
</span>          </span>to </span>= </span>"websecure"
</span>          </span>scheme </span>= </span>"https"
</span>
</span>  [</span>entryPoints</span>.</span>websecure</span>]
</span>    </span>address </span>= </span>":443"
</span></code></pre>
To learn more about this setup using traefik.yml</code> or the CLI, please check out the documentation</a>.</p>
What these lines do is make sure that Traefik listens to the correct ports, 80</code> for HTTP connections (or web</code>) and 443</code> for HTTPS connections (or websecure</code>). Because we only want secure connections coming from the internet, Traefik is instructed to always redirect web</code> entrypoint connections to the websecure</code> entrypoint. Easy and safe!</p>
Now, let us get a SSL certificate and secure those HTTPS connections:</p>
[</span>certificatesResolvers</span>.</span>myresolver</span>.</span>acme</span>]
</span>  </span>email </span>= </span>"test@example.com"
</span>  </span>storage </span>= </span>"acme.json"
</span>  [</span>certificatesResolvers</span>.</span>myresolver</span>.</span>acme</span>.</span>httpChallenge</span>]
</span>    </span>entryPoint </span>= </span>"web"
</span></code></pre>
Setting up the certificateResolver</code> is only half the process, now you need to make sure your docker container uses it by appending a few labels to docker-compose.yml</code>:</p>
mycontainer</span>:
</span>  </span>image</span>: someimage
</span>  </span>labels</span>:
</span>    </span>- traefik</span>.</span>http</span>.</span>routers</span>.</span>router0</span>.</span>rule</span>=</span>Host(`example.com`)
</span>    </span>- traefik</span>.</span>http</span>.</span>routers</span>.</span>router0</span>.</span>tls</span>=</span>true
</span>    </span>- traefik</span>.</span>http</span>.</span>routers</span>.</span>router0</span>.</span>tls</span>.</span>certresolver</span>=</span>myresolver
</span></code></pre>
To learn more about this setup using swarm, kubernetes or rancher, please check out the documentation</a>.</p>
You don't often get so much functionality out of a few lines of code: the service provided by mycontainer</code>, whether it's a Nextcloud</a> container, a Wallabag</a> container or any other service you might enjoy using, is now available by visiting example.com with HTTPS enabled automatically ensuring a secure connection now and in the future as Traefik manages the certificate renewal for you.</p>
Once you start hosting 20, 30, maybe 40 containers and making them accessible to yourself, to friends and family or the entire world through the internet, Traefik really starts to make your self-hosting life a whole lot more enjoyable.</p>
The problem: home connections</h2>
There's one use-case where the "HTTPS everything" credo becomes counter-productive: services that should only be accessible via the home network.</p>
In my home, I monitor a lot of services and devices using Telegraf</a> which sends the metrics to an InfluxDB</a> database to be analyzed by Grafana</a>, a setup also known as the TIG stack</a>. All these services are hosted on a server running inside my home.</p>
The thing is, I don't need access to my Grafana dashboards outside my home. In fact, I don't even want</em> those dashboards and the data they display available on the internet. So, I can just make up a domain that doesn't exist, say "grafana.lan", ensure that my home's DNS resolver (this could be your router or, in my case, PiHole</a>) sends any grafana.lan</code> requests to my server instead of to the internet and just make my Grafana container respond to grafana.lan</code> requests using Traefik labels:</p>
grafana</span>:
</span>  </span>image</span>: grafana/grafana
</span>  </span>labels</span>:
</span>    </span>- traefik</span>.</span>http</span>.</span>routers</span>.</span>grafana</span>.</span>rule</span>=</span>Host(`grafana.lan`)
</span>    </span>- traefik</span>.</span>http</span>.</span>services</span>.</span>grafana</span>.</span>loadbalancer</span>.</span>server</span>.</span>port</span>=</span>3000
</span></code></pre>
I added a server.port</code> label to make sure Traefik forwards all requests to the correct port that Grafana listens to, in this case 3000</code>.</p>
I removed the TLS-related labels since I won't be needing HTTPS! Inside my home, I can just visit http://grafana.lan</code> since I won't be using the dangerous internet connections outside my home.</p>
Simple, right?</p>
Well, no. We have a problem, because this setup won't work.</p>
Redirect HTTP to HTTPS</h2>
Remember the first lines we added to traefik.toml</code>? The ones that redirected all HTTP requests to the HTTPS entrypoint? Well, every time we want to visit http://grafana.lan</code>, it will be redirected to https://grafana.lan</code>. In the best case scenario, your browser will display a few flashy warnings that you are about to enter a website over an unsecured connection, with a hidden button that allows you to proceed anyway. In the worst case scenario, your browser will not allow you anywhere close to your own container because there is a certificate problem.</p>
So, the problem is that there is no certificate for grafana.lan</code>? Well, simple! Let's just add back the TLS labels and have a secure connection within the home!</p>
Again, no can do. A domain needs to exist and be accessible from the internet in order for a SSL certificate to be issued. Since neither criteria are applicable to our grafana.lan</code>, we can't get a SSL certificate.</p>
Luckily, there is a solution.</p>
The solution: domain-specific HTTP redirection</h2>
What we need to do is instruct Traefik to not redirect all</strong> HTTP connections to HTTPS, but only</strong> those that we can access from the internet. The connections that stay in the house should remain HTTP connections.</p>
Remember these lines?</p>
[</span>entryPoints</span>.</span>web</span>.</span>http</span>]
</span>  [</span>entryPoints</span>.</span>web</span>.</span>http</span>.</span>redirections</span>]
</span>    [</span>entryPoints</span>.</span>web</span>.</span>http</span>.</span>redirections</span>.</span>entryPoint</span>]
</span>      </span>to </span>= </span>"websecure"
</span>      </span>scheme </span>= </span>"https"
</span></code></pre>
These need to be removed from traefik.toml</code> so that you keep the basic entrypoint definitions:</p>
[</span>entryPoints</span>]
</span>  [</span>entryPoints</span>.</span>web</span>]
</span>    </span>address </span>= </span>":80"
</span>
</span>  [</span>entryPoints</span>.</span>websecure</span>]
</span>    </span>address </span>= </span>":443"
</span></code></pre>
Next, we need to add a so-called dynamic file</code> configuration. Add the following line to the [providers]</code> section in traefik.toml</code>:</p>
[</span>providers</span>.</span>file</span>]
</span>  </span>filename </span>= </span>"/path/to/traefik_dynamic.toml"
</span>  </span>watch </span>= </span>true
</span></code></pre>
Create the file /path/to/traefik_dynamic.toml</code> and add the following content:</p>
[</span>http</span>]
</span>  [</span>http</span>.</span>middlewares</span>]
</span>    [</span>http</span>.</span>middlewares</span>.</span>redirect_to_https</span>]
</span>      [</span>http</span>.</span>middlewares</span>.</span>redirect_to_https</span>.</span>redirectscheme</span>]
</span>        </span>scheme </span>= </span>"https"
</span>
</span>  [</span>http</span>.</span>routers</span>]
</span>    [</span>http</span>.</span>routers</span>.</span>web_redir</span>]
</span>      </span>rule </span>= </span>"HostRegexp(`example.com`, `{subdomain:[a-z]+}.example.com`)"
</span>      </span>entryPoints </span>= [</span>"web"</span>]
</span>      </span>middlewares </span>= [</span>"redirect_to_https"</span>]
</span>      </span>service </span>= </span>"api@internal"
</span></code></pre>
This code instructs Traefik to add a new middleware</a> whose only responsibility is to redirect HTTP requests to HTTPS requests.</p>
The next few lines applies this middleware</code> to a router</a>. The trick here is that this router is only applied to requests that target example.com</code> and all its subdomains. You could add as many domains as you need to get them to always redirect to HTTPS. Our home domain grafana.lan</code> is not in the list, therefore the middleware</code> will not be applied to it and the connection will remain HTTP!</p>
Note</strong>: the router points to the service api@internal</code>. This is done because a router always needs a service even when, as in this case, the service will never be used because the connection is redirected to HTTPS and after that, this router is no longer used. If you don't like using api@internal</code> for this purpose, you could always run a harmless whoami</a> container and point the router to that service.</p>
Conclusion</h2>
This Traefik setup takes a bit more effort but allows you to gracefully handle two different types of connections.</p>
Connections coming from the internet are secured via SSL certificates and will always happen over HTTPS.</p>
Connections coming from within the house stay within the house, they happen over HTTP and you can feel a lot safer knowing the internet no longer has direct access to that container.</p>


Quick comparison: Plausible vs logs
2020-07-13T23:19:39+00:00
About a month ago, I started collecting website usage data using both Plausible.io</a> and logs generated by Caddyserver</a>, my reverse proxy. The goal was to compare the data sources, just like Marko Saric</a> did in a post on the Plausible blog</a>.</p>
Here's a quick overview of the results. For more details, read the post mentioned above, the results are nearly identical and Marko does a great job explaining the results.</p>
Results</h2>
Quantitative data</h3>
The table below summarizes key metrics computed by both Plausible and GoAccess</a> (based on Caddyserver logs). Data used was collected between June 13th and July 13th.</p>
Metric</th> Plausible.io</th> Logs + GoAccess</th> Δ factor</th></tr></thead>

Visitors</td> 32.1k</td> 76.9k</td> x2.4</td></tr>
Pageviews</td> 44.5k</td> 468.6k</td> x10.5</td></tr>
Bandwidth</td> -</td> 16.6 GiB</td> -</td></tr>
</tbody></table>
Just as Marko noticed, logs show much higher numbers of visitors and pageviews, likely due to crawlers and bots that get noticed in the logs but do not run javascript and therefore are not picked up by Plausible.</p>
I could compare other metrics like referrers and top pages, but again, I suggest you read the post on the Plausible blog</a>.</p>
I'd like to add that the logs can provide some information about bandwidth usage and which files are downloaded the most. This would allow you to make informed decisions when optimizing caching and file loading. Plausible can't help you with this data, one needs logs for this.</p>
Qualitative data</h3>
The experience with Plausible was more convenient than with GoAccess, as the website of the former loads in seconds whilst the latter took 3 minutes to process the logs and generate the results.</p>
Conclusion</h2>
Both methods have advantages and disadvantages. Plausible gives fast and precise results but potentially impacts page load (although minimally). Server logs don't impact page load, can provide bandwidth stats but inflate numbers due to traffic noise generated by search engines, crawlers and bots. Personally, I will continue using both for the foreseeable future.</p>
Methodology</h2>
Plausible</h3>
Visit the Plausible.io</a> website and simply look at the website's stats.</p>
Caddy logs</h3>
Logs were collected using the following snippet in the Caddyfile:</p>
log {
</span>    output file /var/log/caddy/access.log {
</span>        roll_size 100MiB
</span>        roll_keep 10
</span>        roll_keep_for 2160h
</span>    }
</span>}
</span></code></pre>
GoAccess</h3>
As GoAccess cannot read Caddy logs directly, a small bash script is needed:</p>
today_date</span>=</span>$</span>(</span>date</span> -u</span> +"%Y-%m-%d"</span>)
</span>today_date</span>=</span>$</span>(</span>date</span> -u --date</span>=</span>"$</span>today_date</span> -30 day" +"%Y-%m-%d"</span>)
</span>today_ts</span>=</span>$</span>(</span>date</span> -d </span>$</span>today_date</span> +%</span>s)
</span>
</span>goaccess </span><</span>(</span>zcat</span> -f</span> logs/access</span>* | </span>jq</span> --raw-output </span>'
</span>   .request.remote_addr |= .[:-6] |
</span>   select(.request.remote_addr != "1.1.1.1") |
</span>   select(.request.remote_addr != "2.2.2.2") |
</span>   select(.ts >= '</span>$today_ts</span>') |
</span>   [
</span>      .common_log,
</span>      .request.headers.Referer[0] // "-",
</span>      .request.headers."User-Agent"[0],
</span>      .duration
</span>   ] | @csv'</span>) </span>\
</span>   --log-format</span>=</span>'"%h - - [%d:%t %^] ""%m %r %H"" %s %b","%R","%u",%T'</span> --time-format</span>=</span>'%H:%M:%S'</span> --date-format</span>=</span>'%d/%b/%Y'
</span></code></pre>
This was adapted from the bash script described by Alessandro</a> in this blog post</a>.</p>


Optimizing the website's load performance
2020-06-05T22:47:21+00:00
My old webhosting</h2>
When I started making websites back in 2010-ish (maybe even earlier than that, I don't remember), I used shared hosting as I did not have the slightest clue about how a server worked, let alone set one up for web hosting. About two-three years ago, I switched to Cloudways</a>, allowing one to host websites on a virtual private server (VPS) and still not require any actual knowledge about the inner workings of a server.</p>
My new webhosting</h2>
However, I've been managing my own private server for almost two years so I felt confident I could do the hosting myself. Hip as I am (I am not), I decided to go with a Caddy server</a> as a Docker container</a> on a VPS hosted by DigitalOcean</a>. For the low-traffic websites I currently maintain, this is largely sufficient.</p>
I am in the process of moving each website one by one to the new hosting solution. It was time for this very website, yarmo.eu</a> and I thought to myself:</p>

I should actually check if I gain any website load performance by moving to this new solution.</p>
</blockquote>
When asking the Fediverse, 70% predicted Caddy would perform better than Cloudways</a>.</p>
Let's get testing</h2>
I decided to use WebPageTest.org</a> to measure load perfomance. For each case described below, three measurements were performed and the median measurement is displayed and analyzed.</p>
Cloudways</h3>
First, a baseline measurement of my existing Cloudways solution.</p>


Cloudways - overview</em></p>


Cloudways - rating</em></p>


Cloudways - waterfall</em></p>
So the server returns the first byte of information after 480 milliseconds. Now, I should tell you that my website is based on Phug</a>, the PHP port of pug.js templating</a>. The page is rendered in real-time and apparently, that takes a little over 300 ms.</p>
It is worth noting that any other metric is then dependent on how the website is programmed. Once Cloudways has sent over the data, it no longer has any influence on load performance.</p>
The website if fully loaded  after 923 ms. Good to know. About a second to wait for my website to load.</p>
Over on the waterfall, we see a bunch of files being downloaded simultaneously after the HTML page is loaded. The largest asset to load is the profile picture.</p>
Wait, what is that F</code> over on the rating? Security is NOT in order! As it turns out, Cloudways does not handle security-related HTTP headers for you… I did not know that! They recommend setting these headers in a .htaccess file</a>.</p>
Let this be a reminder to all of you: test your websites. One might learn a few tricks.</p>
Anyway, can Caddy do better?</p>
Caddy</h3>


Caddy - overview</em></p>


Caddy - rating</em></p>


Caddy - waterfall</em></p>
Well, as it turns out, it's largely the same performance. First byte arrived after 459 ms, but I've ran it a few times and there's really little difference between Cloudways and Caddy.</p>
BUT! Learning from my previous mistakes, I configured Caddy to set up all the correct headers and won't you look at that, A+</code> on the security score!</p>
So that's it then?</p>
Well, not really… I learned that my website wasn't running optimally because I forgot some basic HTTP headers. Did I forget more? In other words, can I do even better than this?</p>
I've tried a lot of things, I'll just narrow it down to the two most important findings.</p>
Caddy - inline most of it</h3>
As it turned out, I had a few small SVG icons and some CSS files. I tried rendering them into the HTML page, so the data would be sent on the first data transmission and no separate requests were needed. For good measure, I also minified the CSS files which, for one file, reduced the size by 30%!</p>


Caddy+inline - overview</em></p>


Caddy+inline - rating</em></p>


Caddy+inline - waterfall</em></p>
On the waterfall above, you can clearly see the dank-mono.css</code> was not inlined but I tried multiple configurations, there was no real gain as the image also needed to load and took longer anyway. So, all in all, inlining the SVG and CSS content did little in this case.</p>
Also, note the regression from A+</code> to A</code> on the security score. There was one header I couldn't quite get working properly so I had to disable that one, other than that, it's working better than it ever has.</p>
What drew my attention for the final step was the B</code>. My server responds in within 480 ms and that it still not good enough for you, WebPageTest? Ok, have it your way.</p>
What takes my server so long to respond? Well, obviously, it must be the templating. Can I improve the template? Perhaps. But as it turns out, I don't have to! Ever heard of caching?</p>
As described on their website</a>, PHUG has support for caching and even calling an optimized version of their renderer. So I applied both caching and optimized rendering.</p>
Caddy - PHUG optimization</h3>


Caddy+PHUG - overview</em></p>


Caddy+PHUG - rating</em></p>


Caddy+PHUG - waterfall</em></p>
Well, there it is!!! First byte of data arrived after a mere 173 ms, website is useable in less than half a second and all scores are A</code>!</p>
That's the result I was hoping for. Now on my todo list:</p>

Optimize the profile picture further or try SVG</li>
Get all HTTP headers perfect</li>
</ul>
Any comments or recommendations/optimization? Please let me know</a>!</p>







The Case of the Missing Entropy
2020-06-05T12:14:57+00:00

In computing, entropy is the randomness collected by an operating system or application for use in cryptography or other uses that require random data. (Source: wikipedia</a>)</p>
</blockquote>
Docker, are you still there?</h2>
It all started when I got myself a new VPS server for serving web content. I have a more-than-capable server at home but I'd rather not use it for "uptime-sensitive" use-cases, the odd crash still takes it down from time to time. I know, a CDN…</p>
Sticking with what I'm comfortable with, I decided to go with a docker setup with only a few containers:</p>

caddy</a> as web server and reverse proxy</li>
php-fpm</a> as PHP processor</li>
a couple of others with minor significance</li>
</ul>
As per usual, I write my docker-compose.yaml</code> and it's all set. But not this time. At times, when I would change something in the yaml file and run docker-compose up -d</code>, it would do it immediately, as I would expect from all the times I've run it on my homelab. But sometimes, it would wait a minute or longer and only execute then.</p>
I accepted this behavior a few times, but at some point, it had to be dealt with.</p>
Investigating</h2>
I noticed a few things. First, it did not seem to be due to a lack of computing resources. My Grafana</a> dashboard (with InfluxDB</a> as backend and Telegraf</a> as agent) clearly showed me that CPU usage was about 1% and RAM was about 30% full. No excessive DISK IO or NETWORK IO. So we are not overwhelming the system!</p>
Additionally, while it was waiting to execute, I could open a new SSH connection and do other stuff. With one exception: any docker-related command would not execute.</p>
Final clue: I could not ctrl-c my way out of a pending docker command execution, but if I would close the terminal, open a new one, connect via SSH and run any new docker command, it would still wait.</p>
Final final clue: a minute later, I could run docker commands left, right and center without a single problem. Another minute later, it might do the whole waiting again. It was very… "Random". Wink, wink…</p>
Have you figured it out yet? I hadn't.</p>
Researching</h2>
With this information, I was confident enough to start searching online and I came across this github issue</a> fairly quickly:</p>

"docker-compose often takes a long time to do anything"</p>
</blockquote>
That sounds about right!</p>
A few comments in, it was suggested</a> to run the following command: cat /proc/sys/kernel/random/entropy_avail</code>.</p>
On my VPS, this returned 52</code>. Whoopsie…</p>
Entropy</h2>
For those of you who don't know what (computing) entropy is, here's the wikipedia article</a> for it. In short: computers are terrible at coming up with random numbers (just like humans! Topic for another day), which many applications require for their proper working.</p>
Our operating systems have a clever way to solve this: take all input that is NOT generated by the computer itself and use that as "randomness". For example, a computer doesn't know in advance how you are going to move the cursor or which keyboard button you will press. The operating system takes these inputs, processes them to "extract the randomness" and stores it in the entropy pool</code>.</p>
Any application needing some randomness can request some random data from the entropy pool</code>. Maintaining sufficient entropy is therefore a challenge in itself: process enough random data to keep up with the demand.</p>
Apparently, Docker is an application that requires randomness. But how have I never encountered this issue before?</p>
VPS and entropy</h2>
On both my desktop computer and homelab, the entropy available is around 4000</code>, which is perfect. They are able to maintain this entropy because of all the sources of randomness available to them. Mouse and keyboard inputs, processes running in the background, etc.</p>
Now, let's take the VPS as a counter example. These things are made to be fully reproducible: every time you boot one up, they are expected to run in the same way. They are also very sealed off from the host system for security reasons: I cannot read core temperature values for my VPS. They don't have "true hardware", they get portions of hardware, shared with other VPS instances. Except for my SSH connection, the VPS has no mouse or keyboard inputs.</p>
In other words, VPSs are severely lacking in sources of entropy. That is why the entropy available was only 52</code> and why docker stalled: it had to wait for sufficient randomness to occur.</p>
More information on VPS and entropy from DigitalOcean</a>.</p>
The remedy: haveged</h2>
There is a way to remedy the situation: haveged</a>. Having only discovered it last night, I do not fully understand it yet but from what I have read, it is a pseudorandom number generator (PRNG) that fills the entropy pool</code> with "pseudorandomness". Installing haveged</code> immediately solved my issue, all docker commands were running instantly again.</p>


Can you tell when I installed haveged?</em></p>
Caveat: pseudorandomness</h2>
There is a downside to this: PRNGs are NOT random. Wikipedia article on PRNGs</a>. PRNGs generate numbers that appear random but are fully deterministic: run the exact same algorithm twice, and you'll get the same "random" numbers. Therefore, VPSs may not be the perfect solutions to perform entropy-heavy tasks such as cryptography: a cryptographic key generated with pseudorandom numbers is far less secure than one generated with truly random numbers.</p>


LunaSea: FOSS FTW
2020-05-22T19:47:38+00:00
#100DaysToOffload >> 2020-05-22 >> 023/100</code></p>
Out with the old</h2>
A couple of weeks ago, I finally discovered a FOSS alternative for nzb360, a great app for managing Plex, Radarr, Sonarr, etc. I wish to have kept using nzb360, but unfortunately, the app relies too heavily on Google Services and though I have paid for it, I can no longer use it as my LineageOS phone can't process purchases made on official Google Android phones.</p>
In with the new</h2>
Named LunaSea</a>, it can do anything it should, (manage Sonarr, Radarr, Lidarr, NZB clients), it looks fantastic, it's available for both Google Android and iPhone and, of course, it's FOSS</a>.</p>
Only thing I'm missing is a donation button. And a fediverse account :)</p>


Introduction to PiHole
2020-05-10T23:24:04+00:00
#100DaysToOffload >> 2020-05-10 >> 016/100</code></p>
PiHole</a> is almost ubiquitously present on every list of services people could/should selfhost. And rightfully so, it is easy to set up and extremely useful on a daily basis. It blocks ads on almost all websites on all the devices in your home without the necessity of installing anything on them. It will also stop some devices from communicating with their parent companies behind your back.</p>
How it works</h2>
To understand how PiHole does its thing, we need a quick introduction into how DNS works, the system that makes sure we can visit websites even if they are located on the other side of the world. The problem DNS solves is that the URL you use to visit a website doesn't tell your device anything about the physical location or IP address of the server that hosts the website.</p>
If you wish to visit a website, say yarmo.eu</a>, you enter that address in the top bar and hit enter. Your browser will then ask your router to get this website for you. If this is the first time you visit this website, your router doesn't know yet where the server is located, so it asks a DNS server in geographical proximity, usually the DNS server of your ISP.</p>
If this DNS server knows the IP address of the server, it will be relayed back to your device which will now ask that server directly for the content of the website. If the DNS server doesn't have this information, it will ask another and so forth until the IP address of the host server is found.</p>
As we established above, your router contains a DNS router. However, this can almost always be delegated to another DNS server in your home. That's where PiHole comes in. Instead of your router trying to figure out where the website server is located, it will ask PiHole to do so.</p>
But PiHole has a trick up its sleeve: it has a built-in database of hundreds of thousands of URLs that are associated with ads and when they are requested, PiHole simply ignores them.</p>
So you want to visit coolsite.com</code>? Fine, PiHole will get you that website. Now, coolsite.com</code> suddenly wants to load an ad from ads.gafam.com</code>? The computer asks the router, the router asks PiHole, PiHole knows this URL is used to serve ads and will block that request, giving you a website without ads. Awesome!</p>
Something you want to say?</h2>
Meanwhile, you are listening to music using a wireless speaker in your living room from a made-up brand "NOSON". What you don't know is that this device is continuously sending messages to the company containing information about the music you play and more. PiHole knows this and as soon as the speaker requests to send a message to metrics.noson.com</code>, PiHole says no.</p>
That's how PiHole blocks ads AND protects your privacy.</p>
Dedicated hardware</h2>
Dedicating hardware to PiHole is advised but the hardware can be as simple as a Raspberry Pi Zero</a>. The reason it is advised to use dedicated hardware is because if your PiHole crashes, there's no more internet in the home until you get the PiHole working again. It way to prevent this situation from happening is to always have two PiHoles running on separate hardware and telling the router about both PiHoles.</p>
Second DNS server?</h2>
Oh, and while we're on the subject: do not put any "fallback" DNS servers like Google's or Cloudflare's in the second DNS server field on your router. Unfortunately, it doesn't work like a fallback, all routers will simply divide the workload over the two DNS servers. This means that if an outside DNS server is put in second place, it will receive DNS calls even if the PiHole is fully functional.</p>
Having a proper DNS fallback server is difficult to set up, so best would be to use two different PiHole instances. Unless, of course, you don't mind a small period of internet loss and you are always nearby to fix the situation.</p>
Caveats</h2>
Unfortunately, ads on video platforms like YouTube will not be blocked. This is because they serve the ads on the same domains as they serve the main content, meaning that they don't have a ads.youtube.com</code> or something similar. Therefore, PiHole cannot block the ads. As there are a few of these edge cases, it is always recommended to use PiHole in conjunction with on-device ad blockers like uBlock Origin</a>.</p>
Final words</h2>
Really, there are few reasons to not get PiHole into your home and the benefits vastly outweigh the challenges (IMHO). It is also a great start on a journey of selfhosting more services and realising that one can be independent of major corporations to some degree.</p>


Traefik migrated to v2
2020-05-09T23:01:34+00:00
#100DaysToOffload >> 2020-05-09 >> 015/100</code></p>
Last september, traefik</a> received its big version 2 update</a>. I was very excited about TCP routers and the newly implemented middlewares. Can't have been more than a few days later that I tried to migrate my homelab to the new version. I remember being annoyed by the lack of a proper migration guide. Sure, it's possible that I didn't look good enough, but I searched for it for a few days without results. I tried using the new documentation and failed, everything crashed and could not get it working. As I did not have the time to do much more extensive research and also, I needed the selfhosted services on a daily basis, so I left it.</p>
Until today. The whole migration took me a little over three hours and I learned quite a bit on the way. Also, the migration guide</a> has helped quite a bit. If this was updated since September, great article. If not, still a great article and I really did not take the appropriate amount of time to prepare my migration.</p>
Easy steps</h2>
The first thing I did was a general search-and-replace for the docker labels (both routers and services). What was traefik.frontend.rule=Host:xyz</code> now is traefik.http.routers.router0.rule=Host(``xyz``)</code>. What was traefik.port=80</code> now is traefik.http.services.service0.loadbalancer.server.port=80</code>. Quite a bit longer and more cumbersome, but in the end, more extensible.</p>
The traefik.docker.network=xyz</code> is now unnecessary in most cases as you can define a default network in the traefik.toml</code> file. Speaking of which, you can now work with a YAML file. It's not to everyone's taste, but I will switch to it in the future when I have some more time.</p>
The traefik.toml</code> still needed quite the makeover, but everything is will explained in the migration guide</a> and the reference page</a>. Content-wise, I changed little, it's just that the syntax is different. Notable changes are the domains for which certificates are needed now are declared in the entrypoints</code> section instead of the acme</code> section and the file</code> section can no longer include the "router/service declarations", they belong in a separate file.</p>
Pitfalls</h2>
Doing this resulted in a non-functional state. Two different possibilities: either the container could not be redirected to the correct service resulting in a 404, or the redirect was correct but without correct certificate. That first possibility was mostly on me: containers are now no longer exposed by default and I forgot to add traefik.enable=true</code>. Mind you, I always set traefik.enable=false</code> when I didn't want a container exposed and still do.</p>
However this did not solve the issue for all the containers. I suspect there's still some trickery I need to do in case of using multiple routers. I tried explicitly specifying the service</code> for the different routers but that wasn't the solution.</p>
As for the other issue, the solution was simple but finding the source was quite hard: as it turns out, I renamed the certificateResolver</code> to something other than default</code>. If such is the case, then containers will NOT automatically use it for their certificates. Adding traefik.http.routers.router0.tls=true</code> and traefik.http.routers.router0.tls.certresolver=mycertresolver</code> to each container solves this issue.</p>
Todo</h2>
One thing I haven't got working yet is using the providers.file</code> provider. I tried to mimic the container labels but to no avail. Yet.</p>

Update 2020-05-12</strong>: I fixed the providers.file</code> issue. Remember kids, always read the documentation well. It turns out, I missed the line that starts with *</code> in the code below.</p>
[</span>http</span>.</span>services</span>]
</span>    [</span>http</span>.</span>services</span>.</span>Service01</span>]
</span>        [</span>http</span>.</span>services</span>.</span>Service01</span>.</span>loadBalancer</span>]
</span>*           [[http.services.Service01.loadBalancer.servers]]
</span>                </span>url </span>= </span>"foobar"
</span></code></pre>


Varken: Plex monitoring solution
2020-05-05T21:49:58+00:00
#100DaysToOffload >> 2020-05-05 >> 011/100</code></p>
Today, I discovered Varken</a>, a neat solution to monitor your Plex ecosystem (including Sonarr, Radarr, etc.) and store the data in your InfluxDB instance. This solution is a great addition as I can now make Grafana or Chronograf dashboardz encompassing both server metrics and Plex metrics. The reason this is important is that I have a relatively low-power server (NUC) and a single stream on Plex can have a noticeable impact on the CPU usage.</p>
Varken requires a Tautulli</a> instance to collect the data from as well as a MaxMind</a> API key which unfortunately isn't optional. I run all software mentioned in this post in separate docker containers.</p>
Also, 11th post for #100DaysToOffload today and 11 is my lucky number :)</p>


Selfhost email… But should you?
2020-05-03T19:46:47+00:00
#100DaysToOffload >> 2020-05-03 >> 009/100</code></p>
Yesterday, I wrote about how you can</strong> selfhost your very own email server</a>. Shortly after publishing the post, it was pointed out to me that are very reasonable drawbacks to doing this. So today, let me give you my answer to the question of whether you should</strong> selfhost your email.</p>
Relying on hardware</h2>
Firstly, I mentioned in that article that although I have two domains on my selfhosted server, I still fall back to a protonmail address for the most important stuff like banking and governmental services. So what's the point of selfhosting then, if I do use third-party email addresses? Well, what I failed to mention was that in the long term, yes, I do want everything selfhosted.</p>
When I started my email hosting adventure, I was very cautious. Only months before did I start my own homelab and as it turned out, that had the tendency to crash every so often, making it a no-go for email hosting. I resorted to use a VPS while getting my homelab sorted out. This worked great and still does, but at the time, you can imagine I was still discovering the DNS parameters, the reputation handling (more on this later). Also, what were the consequences of running a VPS for 24/7? I could not commit to using the selfhosted email for anything more than experimentation.</p>
Fastforward about a year and the VPS has held up greatly, the email software has never crashed or acted against my expectations. It has received regular updates and never failed once during one. Meanwhile, my homelab has proven to be extremely reliable and with the upcoming hardware upgrade, I expect even less irregularities than I do now. Soon enough, when all the stars align and I figure out how to make recoveries as fast on my local hardware as I can on a VPS (make a new instance based on a daily snapshot and voilà!), my email server will be transferred to my homelab and I will use it for everything.</p>
The pain of administering an email server</h2>
Secondly, I've also heard of people not resorting to selfhost their email because of the fragility of the underlying processes and if one thing is slightly out of tune, the whole email server stops working. Although I've toyed around with most of the individual processes like dovecot at the beginning to understand what they do and how they work, I haven't touched a single one of them in almost a year. mailcow.email</a> is just that good. I've played around with the settings and it won't stop working. Meanwhile, I get an antivirus, spam monitoring, those handy "+topic" email filtering. I'd like to try out Mail-in-a-Box</a>, mostly because it is also recommended by PrivacyTools</a> but I have no incentive too. My current solution just works great for me.</p>
The reputation of a server</h2>
Lastly, I need to address a IMO bigger problem: reputation. If other servers don't trust you, your emails may easily be thrown into the spam folder of the recipient or even rejected. The main reason for this is to fight spam: mass email spammers usually operate from unknown IP addresses. Unfortunately, this hurts the selfhosters. So, before you have even installed your email server software, you are already mistrusted by simply not using the big servers like Gmail and Hotmail. And indeed, when I started, most of my emails landed in spam.</p>
This got greatly improved simply by using an email relay; in my case, mailgun</a>. These paid-for (but often with free tier) services are a lot more trusted since mailgun will do spam prevention on their end, so letting them send your emails for you is a great improvement. And even with the recently reduced free tier</a>, I don't send nearly enough emails to come close to the free tier quota.</p>
However, it still happens that my emails are treated as spam so I often do follow ups via other channels of communication. Another issue may be that the IP address you were given already has a bad reputation caused by a previous owner: this is difficult to find out and even harder to fix. DDG-ing improve email server reputation</code> yields many articles but read a handful of them and soon you'll realise it's really really hard to improve it. There's no central repository, no forms. Getting a mistrusted IP address can quickly suck all the fun out of having your own email server.</p>
Answering the question</h2>
So, should you</strong>? This depends on how willing you are to be independent of third-party email services and how much you are willing to put up with. I started naïvely and had to answer this question along the way while experimenting. By now, my personal answer to this question is: yes. I see the benefits and drawbacks. I'm not sure if it's the usage of mailgun, or me sending mails to family and friends and then asking them to tell their services to not mark it as spam, but most of my emails are properly received nowadays. Also, I have managed to improve my infrastructure, I can rely on the hardware (and soon on the emergency recovery mechanisms) and will soon migrate my email server so it's nicely at home.</p>
Hosting your own email server is not easy and requires your full dedication. And with many upcoming trusted and privacy friendly email services</a>, it may not always be the right tool for the job.</p>


Selfhost email
2020-05-02T16:20:35+00:00
#100DaysToOffload >> 2020-05-02 >> 008/100</code></p>
Yes, you can selfhost email. And you should, if and ONLY if you feel comfortable with maintaining a linux server. I'm not a pro at all, but I've been doing it for almost two years, I know where to find my logs, I know how to find the correct answers on stackoverflow and troubleshoot a less-than-functional system.</p>
So don't start with this, but eventually, soon enough, you can selfhost your email.</p>
Because email is important to me, I have chosen to not host it at home, any network issue could prevent emails from coming in. Granted, the sending server will usually retry for 24 hours until the email is actually received on your side so small errors are forgiven, but still, I've opted for a dedicated droplet on digitalocean, though any VPS will do.</p>
And then, follow the instructions on mailcow.email</a> and you're set. SSH in once a week on your VPS to run the updater. The administration side has plenty of features for advanced administration of the email server and the included webclient is the awesome SOGo.</p>
If you want to make sure your email server is as trusted by other servers as possible, your emails are sent as securely as possible and your experience with other email clients is as smooth as possible, please check out my post on email server DNS settings</a>.</p>
With all that being said, I still use a protonmail address for critical websites and services like governmental services and banking, because whatever happens, I need to make sure that I really receive these emails. On my selfhosted email server, I use two domains: one which I share with the world and with websites for logins and one that I keep private and only use for direct communication with other people. I have yet to experience a single minute of outage, credits to digitalocean and the people behind mailcow.</p>
---UPDATE---</p>
After a fair comment</a> on the fediverse, I have written a follow-up post</a> to address a few more critical points like server reputation and how I deal with that.</p>


The People's Web
2020-04-30T08:53:37+00:00
#100DaysToOffload >> 2020-04-30 >> 006/100</code></p>
The day has long past that we should have started worrying about the openness of our web. It was only a matter of time before censorship and individual tracking would seep into the web of the western world. It pained me to see the levels of state interference in foreign countries, but at least, to me, that was something I only read about in the news, it was then unimaginable that this would happen in the short term in the "free world".</p>
Now, in these trying times, states and big corporations may see it more than justified to track individuals and adjust the information we receive. I do not question their motives: "we are in this together". It takes the world to fight this pandemic, and the next one, and the one after that.</p>
But the question on a lot of minds is: "how do we come back from this?" The hardest part is for the people to accept they are being tracked and their information filtered, so big corporations did it with shady and hidden tactics to avoid this confrontation. We are past this: YouTube has announced it will ban all content that does not conform to WHO and countries everywhere are building apps to track our health and social interactions. Again, this could prove to be what humanity needs right now, but what about after? Is there even an after?</p>
Fortunately, we all have the power to make a few changes to improve our online well-being: change social networks, don't rely on corporations, self-host as much as you can. I will be dedicating a large number of posts on this topic. Self-hosting is not hard, it just takes a little effort to get started.</p>
As a friendly reminder, this website (blog included) has no tracking whatsoever: I do not care who you are, where you are from or with how many you are reading this post.</p>


Selfhosted email: DNS records
2020-02-18T18:39:54+00:00
When selfhosting email, an essential element to get right are the DNS records. Some are absolutely mandatory for email to work, some build trust and some just make life easier. Here's an overview of how I set up DNS for my personal mail server.</p>

My setup</h2>
I have a VPS server running mailcow</a> and two domains: one linking to the mail server, admin page and the web client (let's call this mail.server.domain</code>), and the other just being a email domain used in the email address (let's call this public.domain</code>, so the email address would be hi@public.domain</code>). This way, even if you know the email domain, you don't directly know server domain. Granted, one could find this out by looking at a few DNS records.</p>
The benefit is that I can host email for multiple domains, as long as they all point to mail.server.domain</code> by using the correct DNS records.</p>
Please note that using a single domain is just as easy, as mail.server.domain</code> and public.domain</code> will simply be the same. Another scenario for which you could use the 2-domain setup is when you want your email address to be hi@public.domain</code> (without subdomain) but wish to put the mail server (and/or web client) on mail.public.domain</code> (with subdomain).</p>
I use DigitalOcean</a> as my VPS and DNS server.</p>
Mandatory DNS records for server.domain</h2>
A records</h3>
A records link domain names to IP addresses. When you want to use the admin page or web client provided by your email selfhosting software, the browser needs to know the IP address of the server/VPS and that is what the A record is used for. A records are not used by mail servers when sending or receiving emails.</p>
TYPE  HOSTNAME            VALUE    TTL
</span>A     mail.server.domain  1.2.3.4  3600
</span></code></pre>
Mandatory DNS records for public.domain</h2>
MX records</h3>
MX records tell other mail servers where to actually send the emails. In my case, my email address is hi@public.domain</code> but my mail server is located at mail.server.domain</code>. Other mail servers look at the address, see public.domain</code> and will assume this is our mail server. We use MX records to direct the emails to mail.server.domain</code> instead.</p>
TYPE  HOSTNAME       VALUE               PRIORITY  TTL
</span>MX    public.domain  mail.server.domain  1         14400
</span></code></pre>
Optional DNS records for public.domain</h2>
A and MX records is all you need to get a functional email address. However, for ease of use and good reputation/trust, a few additional DNS records are recommended.</p>
SRV records (ease of use)</h3>
SRV records are used to link specific protocols to specific domains and ports. Just like how MX records tell other mail servers to direct their mails to your mail server on a different domain, the same must be done for mail clients. Say you want to use Thunderbird (or any other mail client) to access your emails. You will log in with your address (hi@public.domain</code>) and password in Thunderbird, and it will then assume your mail server must be located at public.domain</code>. It will not find it there, warn you about this and you will have to manually enter your IMAP and SMTP server details. If you have set up SRV records, Thunderbird will automatically detect the correct server location (mail.server.domain</code>) and save you some hassle.</p>
TYPE  HOSTNAME            VALUE               PORT  PRIORITY  WEIGHT  TTL
</span>SRV   _imap._tcp          mail.server.domain  143   1         100     14400
</span>SRV   _imaps._tcp         mail.server.domain  993   1         100     14400
</span>SRV   _pop3._tcp          mail.server.domain  110   1         100     14400
</span>SRV   _pop3s._tcp         mail.server.domain  995   1         100     14400
</span>SRV   _submission._tcp    mail.server.domain  587   1         100     14400
</span>SRV   _smtps._tcp         mail.server.domain  465   1         100     14400
</span>SRV   _sieve._tcp         mail.server.domain  4190  1         100     14400
</span>SRV   _autodiscover._tcp  mail.server.domain  443   1         100     14400
</span>SRV   _carddavs._tcp      mail.server.domain  443   1         100     14400
</span>SRV   _caldavs._tcp       mail.server.domain  443   1         100     14400
</span></code></pre>
TXT records (good reputation)</h3>
TXT records are simply messages that provide additional information. Here, TXT records are used to tell other mail servers more about your own mail server in order to build some trust between them: these records are a useful tool against spoofing where bad actors try to impersonate you and pretend you are sending the bad emails they are sending.</p>
TYPE  HOSTNAME         VALUE
</span>TXT   @                "v=spf1 mx ~all"
</span>TXT   dkim._domainkey  "v=DKIM1;k=rsa;t=s;s=email;p=..."
</span>TXT   _dmarc           "v=DMARC1;p=reject;rua=mailto:admin@public.domain"
</span></code></pre>
Detailed information on these records can be found here</a>, but in short:</p>

SPF records tells other mail servers that only the specified mail server (in this case, mx</code> which points to mail.server.domain</code> via the MX record) is allowed to send emails for your email domain (in this case, public.domain</code>);</li>
DKIM applies a virtual signature to all your sent emails and other mail servers use the second TXT record above to validate that signature;</li>
DMARC records tell other mail servers what should happen to emails that fail the SPF and DKIM policies; the record above states these emails should be rejected and a notification sent to admin@public.domain</code>.</li>
</ul>
The DKIM record contains a cryptographic key (replaced above by ...</code>). In my case, this key was generated for me by mailcow and is unique for each email domain.</p>
Please note that having the above TXT records does not guarantee that other servers trust you immediately: your emails are still likely to end up in spam folders at first. Using intermediaries like mailgun</a> can help with avoiding the spam folder. More on that in a later blog post.</p>
References</h2>
Mailcow has their own recommended DNS records guide</a> which, in conjunction with their admin page, should make setting up DNS records a breeze.</p>
This guide</a> has a lot of in-depth information about the SPF, DKIM and DMARC records (TXT records above).</p>

Methodology</h2>

Plausible</h3>
Visit the Plausible.io</a> website and simply look at the website's stats.</p>

Optimizing the website's load performance

Let's get testing</h2>
I decided to use WebPageTest.org</a> to measure load perfomance. For each case described below, three measurements were performed and the median measurement is displayed and analyzed.</p>

The Case of the Missing Entropy

LunaSea: FOSS FTW

In with the new</h2>
Named LunaSea</a>, it can do anything it should, (manage Sonarr, Radarr, Lidarr, NZB clients), it looks fantastic, it's available for both Google Android and iPhone and, of course, it's FOSS</a>.</p>
Only thing I'm missing is a donation button. And a fediverse account :)</p>

Introduction to PiHole

`Final words</h2> Really, there are few reasons to not get PiHole into your home and the benefits vastly outweigh the challenges (IMHO). It is also a great start on a journey of selfhosting more services and realising that one can be independent of major corporations to some degree.</p>`

Traefik migrated to v2

Varken: Plex monitoring solution

Selfhost email… But should you?

Selfhost email

The People's Web

Selfhosted email: DNS records

Mandatory DNS records for public.domain</h2>

Optional DNS records for public.domain</h2>
A and MX records is all you need to get a functional email address. However, for ease of use and good reputation/trust, a few additional DNS records are recommended.</p>

`References</h2> Mailcow has their own recommended DNS records guide</a> which, in conjunction with their admin page, should make setting up DNS records a breeze.</p> This guide</a> has a lot of in-depth information about the SPF, DKIM and DMARC records (TXT records above).</p>`

Yarmo - selfhosting

Matrix Synapse: migrating from Cloudron to ansible

Conclusion</h2>
There you have it, this is what I did to obtain a working fresh Synapse server with all the data from the Cloudron server. Cloudron was nice to work with but I am glad I have everything working again without the need for a middle man.</p>
So now… dendrite when?</p>

InfluxDB 2 migration

playerctl: get currently playing music

Traefik: managing both HTTP and HTTPS connections separately

Quick comparison: Plausible vs logs

Yarmo - selfhosting

Matrix Synapse: migrating from Cloudron to ansible

Conclusion</h2> There you have it, this is what I did to obtain a working fresh Synapse server with all the data from the Cloudron server. Cloudron was nice to work with but I am glad I have everything working again without the need for a middle man.</p> So now… dendrite when?</p>

InfluxDB 2 migration

playerctl: get currently playing music

TLDR</h2> To get information about music currently playing on the computer, run:</p>

Traefik: managing both HTTP and HTTPS connections separately

Quick comparison: Plausible vs logs

Methodology</h2>

Optimizing the website's load performance

The Case of the Missing Entropy

LunaSea: FOSS FTW

Introduction to PiHole

Traefik migrated to v2

Varken: Plex monitoring solution

Selfhost email… But should you?

Selfhost email

The People's Web

Selfhosted email: DNS records

Mandatory DNS records for public.domain</h2>

Optional DNS records for public.domain</h2> A and MX records is all you need to get a functional email address. However, for ease of use and good reputation/trust, a few additional DNS records are recommended.</p>

Conclusion</h2>
There you have it, this is what I did to obtain a working fresh Synapse server with all the data from the Cloudron server. Cloudron was nice to work with but I am glad I have everything working again without the need for a middle man.</p>
So now… dendrite when?</p>

Optional DNS records for public.domain</h2>
A and MX records is all you need to get a functional email address. However, for ease of use and good reputation/trust, a few additional DNS records are recommended.</p>