Yarmo - homelab

InfluxDB 2 migration

2022-05-20T18:36:41+00:00

What is InfluxDB?</h2>
InfluxDB</a> is the MIT licensed</a> time series database</a> of my monitoring stack of choice, the so-called TICK stack</a>, consisting of Telegraf</a> (data collection agent), InfluxDB</a> (time series database), Chronograf</a> (charts and dashboard interface) and Kapacitor</a> (data processing engine).</p>

My history with InfluxDB</h2>
I started my homelab somewhere in 2017-2018. When about a year in I faced some random crashes and reboots which I did not manage to understand or fix, I first turned to netdata</a> before settling with the TICK stack.</p>
While I did eventually solve the random crashes, an obvious issue surfaced: if I want to monitor a crashing server, I probably shouldn't host the monitoring stack on that same server.</p>
And so, my setup eventually converged to the one I use today: a homelab and a few VPSs, all monitored by a TICK stack installed on a dedicated VPS.</p>
InfluxDB 2.0</a> was released in November of 2020 and I remember this moment clearly: I updated my InfluxDB docker container, I noticed everything immediately broke, I read up on what a gargantuan update this 2.0 release was, I said "nope" and immediately reverted to 1.8.x.</p>
Yesterday, I decided it was finally time to sit down and calmly move to InfluxDB 2(.2.0).</p>

InfluxDB 2, or the Death of the TICK stack</h2>
Please refer release notes for InfluxDB 2</a> to see for yourself what has changed.</p>
One of the most notable changes is that InfluxDB 2 now has a interface with graphs and dashboards (replacing Chronograf's functionality) and can send alerts based on data (which previously required Kapacitor). While I am sure Chronograf and Kapacitor can still be used together with InfluxDB 2, they no longer need to and indeed, I have now removed these services from my monitoring stack.</p>

InfluxDB 2, the migration</h2>
The InfluxDB 2 dockerhub page</a> has clear instructions on how to migrate from 1.x to 2.x when using containers and so I did. And while the container logs showed the migration process was successful, I had this weird issue where when I would try to then log in on the web interface, I was greeted with a "fresh install" screen, suggesting the InfluxDB instance was in fact not aware of the migration.</p>
And indeed, after following the instructions of the "fresh install" screen, everything was working fine but no trace of any migrated data :(</p>
If my historical InfluxDB data was in any way valuable to me, I would have put more effort into migrating all of it to InfluxDB 2.</p>
But it wasn't.</p>

InfluxDB 2, the non-migration</h2>
And thus, in a bit of an anticlimax to this whole situation, I `rm -rf</code>ed the whole data directory and started anew with a fresh InfluxDB 2 instance. No more Chronograf, no more Kapacitor. All servers have updated Telegraf configurations and I am currently rebuilding the dashboards in InfluxDB's web interface.</p>`
`And I also still need to rebuild my Grafana dashboards to make use of Flux instead of InfluxQL. I still have things to do before I can fully enjoy my monitoring solution again.</p>`


Friendly reminder to clean your NUC's fan
2020-06-08T13:35:15+00:00
Intel NUCs</a> make for some great low-entry-barrier low-power-consumption servers and homelabs. I have three NUCs at home, two of which have played a server role. They span several generations: a 5i3, a 7i7 and a 8i5.</p>
And they all have one thing in common: sooner or later, their fans clog up with dust, they heat up, they make more noise and perform worse.</p>
If you haven't cleaned the fan in a while, your best bet is to open the NUC up and clean the fan and the exhaust.</p>
To prevent having to open a NUC up too often, I bought a few cans of compressed air and regularly blow air through the device. I'm also looking into placing air filters near the air intake.</p>


Can you tell when compressed air was applied to the NUC?</em></p>


Introduction to PiHole
2020-05-10T23:24:04+00:00
#100DaysToOffload >> 2020-05-10 >> 016/100</code></p>
PiHole</a> is almost ubiquitously present on every list of services people could/should selfhost. And rightfully so, it is easy to set up and extremely useful on a daily basis. It blocks ads on almost all websites on all the devices in your home without the necessity of installing anything on them. It will also stop some devices from communicating with their parent companies behind your back.</p>
How it works</h2>
To understand how PiHole does its thing, we need a quick introduction into how DNS works, the system that makes sure we can visit websites even if they are located on the other side of the world. The problem DNS solves is that the URL you use to visit a website doesn't tell your device anything about the physical location or IP address of the server that hosts the website.</p>
If you wish to visit a website, say yarmo.eu</a>, you enter that address in the top bar and hit enter. Your browser will then ask your router to get this website for you. If this is the first time you visit this website, your router doesn't know yet where the server is located, so it asks a DNS server in geographical proximity, usually the DNS server of your ISP.</p>
If this DNS server knows the IP address of the server, it will be relayed back to your device which will now ask that server directly for the content of the website. If the DNS server doesn't have this information, it will ask another and so forth until the IP address of the host server is found.</p>
As we established above, your router contains a DNS router. However, this can almost always be delegated to another DNS server in your home. That's where PiHole comes in. Instead of your router trying to figure out where the website server is located, it will ask PiHole to do so.</p>
But PiHole has a trick up its sleeve: it has a built-in database of hundreds of thousands of URLs that are associated with ads and when they are requested, PiHole simply ignores them.</p>
So you want to visit coolsite.com</code>? Fine, PiHole will get you that website. Now, coolsite.com</code> suddenly wants to load an ad from ads.gafam.com</code>? The computer asks the router, the router asks PiHole, PiHole knows this URL is used to serve ads and will block that request, giving you a website without ads. Awesome!</p>
Something you want to say?</h2>
Meanwhile, you are listening to music using a wireless speaker in your living room from a made-up brand "NOSON". What you don't know is that this device is continuously sending messages to the company containing information about the music you play and more. PiHole knows this and as soon as the speaker requests to send a message to metrics.noson.com</code>, PiHole says no.</p>
That's how PiHole blocks ads AND protects your privacy.</p>
Dedicated hardware</h2>
Dedicating hardware to PiHole is advised but the hardware can be as simple as a Raspberry Pi Zero</a>. The reason it is advised to use dedicated hardware is because if your PiHole crashes, there's no more internet in the home until you get the PiHole working again. It way to prevent this situation from happening is to always have two PiHoles running on separate hardware and telling the router about both PiHoles.</p>
Second DNS server?</h2>
Oh, and while we're on the subject: do not put any "fallback" DNS servers like Google's or Cloudflare's in the second DNS server field on your router. Unfortunately, it doesn't work like a fallback, all routers will simply divide the workload over the two DNS servers. This means that if an outside DNS server is put in second place, it will receive DNS calls even if the PiHole is fully functional.</p>
Having a proper DNS fallback server is difficult to set up, so best would be to use two different PiHole instances. Unless, of course, you don't mind a small period of internet loss and you are always nearby to fix the situation.</p>
Caveats</h2>
Unfortunately, ads on video platforms like YouTube will not be blocked. This is because they serve the ads on the same domains as they serve the main content, meaning that they don't have a ads.youtube.com</code> or something similar. Therefore, PiHole cannot block the ads. As there are a few of these edge cases, it is always recommended to use PiHole in conjunction with on-device ad blockers like uBlock Origin</a>.</p>
Final words</h2>
Really, there are few reasons to not get PiHole into your home and the benefits vastly outweigh the challenges (IMHO). It is also a great start on a journey of selfhosting more services and realising that one can be independent of major corporations to some degree.</p>


My homelab crashed, time for a break?
2020-05-07T19:53:23+00:00
#100DaysToOffload >> 2020-05-07 >> 013/100</code></p>
It's not the first time my homelab has crashed and it won't be the last time. Something with the hard drives. I'll figure it out, no doubt. But despite needing it for various services throughout my daily routine, I have decided to let the homelab rest for a few days maybe.</p>
It has been running almost non-stop since I started it about two years ago, I never made major changes, always gradually improved upon it. Now, the time may have come to take a hard look at what I started with, what I ended up with, learn a few valuable lessons and perhaps start over. My homelab could use a 2.0 moment.</p>


IMPUC #1 · Homelab overview
2020-01-05T09:32:21+00:00
"In My Particular Use Case" (or IMPUC) is a series of short posts describing how I setup my personal homelab, what worked, what failed and what techniques I eventually was able to transfer to an academic setting for my PhD work.</p>

Why a homelab?</h2>
I started my homelab about a year after I started my PhD. My academic work was challenging in a technical way, with new data generated every day, managing raw data, processed data, metadata. I built a number of tools that would aid me on my daily basis for my work but I needed a place to just try out every technology I could possibly need for my job. It eventually turned out that the homelab was destined to do far greater things than simply serve as a testbed but that's how it started and what provided me the knowledge and experience to solve important issues in my academic work.</p>
The central server</h2>
So one day, I ordered myself an Intel nuc with a 5th generation i3 processor, 8 GB of RAM, an m.2 drive and got started. Container solutions caught my attention before I even had the machine so I first installed docker and later, docker-compose. This setup hasn't changed a bit today as it still allows me to launch new services very easily by changing a single yaml file with minimal impact on the hosting machine. The first things I would install were several databases and gitea, a self hosted git service. The services sit behind a reverse proxy (traefik) to allow them to be accessed by using (sub)domains. Configuration of the machine is managed by a folder of dotfiles backed up in a git repo and stow</code>ed as necessary, but I am currently looking into ansible for this purpose. A 4-bay JBOD USB3 device provides the storage that the nuc then also (partly) makes available over the local network via smb.</p>
The peripheral Pi's</h2>
Floating around the central server are several raspberry pi's. Back when I first started, the central server would sometimes crash or soft-lock and since my entire system monitoring system (telegraf+influxdb+grafana) was also installed on there, there was not a whole lot of investigating and fixing I could immediately do. Now, the central server and the pi's all run telegraf and a single pi now hosts the influxdb+grafana stack and only that. Another pi is acting as a media center (Kodi) and finally, two redundant pi's function as DNS forwarders (Pi-Hole), one of which also hosts my VPN solution (wireguard).</p>
The out-of-house computing</h2>
I have two permanent VPS's running: a website server (Cloudways) and a mail server (mailcow). Both could be hosted on the central server but as long as I cannot guarantee a perfectly stable Internet connection (which my house does not have) nor stable computing (personal budget issue), I choose to host these outside of the house.</p>
Final words</h2>
Thanks for reading this, more posts will come soon explaining with more depth some of the elements described above. If you have questions, you can find several ways to contact on yarmo.eu</a>.</p>