AI-Powered Homelab

Introduction

In this episode, Josh is joined by Henrik Rexed, Cloud Native Advocate at Dynatrace, CNCF Ambassador, and the creator of the Is It Observable YouTube channel. Henrik has spent more than fifteen years in performance engineering and observability, and for the past five years has focused on Kubernetes, OpenTelemetry, and the open source cloud native ecosystem. He is also, by his own admission, someone who cannot wait for a machine when he needs one.

The home lab started as a workaround. An internal reorganization cut off his cloud budget overnight, and rather than wait for a new one, Henrik bought two cheap mini PCs. That grew into a five-node Proxmox cluster, two NAS boxes, a dedicated Mac Studio running local models around the clock, and a Tailscale bridge so the whole thing follows him to conferences. Along the way, he found something the cloud had been hiding from him: private registries, DNS restrictions, and security constraints are what real enterprises actually deal with, and you only learn them the hard way.

The second half turns to agents. Henrik walks through a setup where OpenClaw acts as his CTO, Paperclip models his agents as an organization with roles, tickets, and a defined workflow from product manager to brainstormer to architect to coder to reviewer, and a separate machine holds the code. He is candid about what is not working, particularly token consumption and memory management, and explains why he instruments his agents the same way he would instrument any production system. Also covered: why he built his own InfluxDB receiver, what OCB is and when to use it, hardware regrets, and the RAM crisis.

Episode Highlights

• [00:01:51] Seven years maintaining the most starred repo in the Apache Software Foundation
• [00:02:24] The origin story: a reorg cut the cloud budget, so he bought two mini PCs
• [00:03:57] Kubernetes feature flags and API server tracing, the thing cloud providers would not let him do
• [00:05:09] What the cloud hides: private registries, DNS, and the sweating and swearing part
• [00:07:19] The hardware tour: mini PCs, TrueNAS, a NAS backing up a NAS, and a Mac Studio for LLMs
• [00:09:05] The RAID mistake, and why the second NAS exists
• [00:10:45] Why he would rebuild the whole lab on Minisforum machines
• [00:12:52] Five node Proxmox cluster, cluster API templates, MetalLB, and Tailscale
• [00:15:37] The conference disaster: calling home to ask his wife if the machines are on
• [00:18:58] Why etcd has to stay on local disks, and living with 2.5 gigabit
• [00:20:44] The AI stack: GPU operator, Tesla T4, and a Mac Studio running Ollama
• [00:22:57] OpenClaw as CTO, and why he did not want it writing code
• [00:24:03] Paperclip: agents as an org chart, from product manager to DevOps engineer
• [00:27:11] Where it falls down: token consumption, context size, and memory management
• [00:28:58] Why the infrastructure came first, and the party guest who forgets you at breakfast
• [00:31:04] Discovering BMAD, and the jump in code quality that followed
• [00:35:26] Six hours in traffic and a day of skiing, managed entirely from WhatsApp
• [00:37:16] Credential hygiene: dedicated users, scoped SSH keys, no shared tokens
• [00:38:29] Monitoring the lab, and putting Tetragon in front of OpenClaw
• [00:42:06] Why he wrote his own InfluxDB receiver, and what OCB is for
• [00:53:24] What the home lab content online gets wrong: nobody covers day two
• [00:58:01] The biggest advantage: a switch, and a daughter with her own VMs

Episode Transcript

Josh Lee [00:00:11]

Hello, hello, hello, and welcome to another episode of Unevenly Distributed. I am your host Josh Lee, open source advocate at Altinity, and today I am joined by my good friend Henrik Rexed from Dynatrace. Hey Henrik.

Henrik Rexed [00:00:24]

Hey, very happy to be here and good to see you again.

Josh Lee [00:00:29]

Yeah. Just quickly introduce yourself, and what you do and what you get up to.

Henrik Rexed [00:00:34]

So my name is Henrik Rexed, like you mentioned. I’m a cloud native advocate working at Dynatrace, five years now. It’s been almost a week. Five years and a week.

Josh Lee [00:00:46]

Exciting.

Henrik Rexed [00:00:47]

Yeah, thank you. And I’m focused pretty much internally on Kubernetes, OpenTelemetry, all the cloud native things that are happening around. So my natural choice is to turn more on the open source side of the game. So I’m doing a lot of digging on a lot of projects. And I like to share, and I like to learn and like to share. And that’s why I have this YouTube channel, where usually when I do some topics I then produce an episode out of that, on Is It Observable.

Josh Lee [00:01:25]

And it’s been one of my favorites. It’s been there. Sorry. It’s one of my favorites.

Henrik Rexed [00:01:29]

Thank you. Thank you. Yeah, thank you.

And yeah, so I’m still trying to build content, and of course, you know, maybe in the YouTube game you still need to be consistent, but sometimes you also have to do some work. You have to find the right balance.

Josh Lee [00:01:43]

Yes. Well, this is sort of the impetus for this episode, because we’re both home lab enthusiasts, and maybe this relates to all of these experiments that you’re running for your YouTube channel.

So yeah, today we’re going to dig into using home labs for various things, but also running local AI, I think, is something that people are very interested in. Especially as pricing changes and things like this happen, and we’re all trying to figure this out.

Henrik Rexed [00:02:13]

Yeah.

Josh Lee [00:02:14]

So I guess to start, I always like to start with the problem statement. You have a home lab. It’s pretty extensive. We’ll get into what all you have. But what was the problem you were trying to solve when you started building your home lab?

Henrik Rexed [00:02:24]

It’s an excellent question. So the story behind, in fact, let’s get back a few months ago. So a few months ago I had an internal reorg change, and all the cloud spends were related to a specific entity within the company I’m working for. And then suddenly they decided to change, and as a result of that, when we changed org, the people didn’t want to pay for our cloud bills, or say okay, so I’m going to get rid of those people. And then obviously the new organization had to reopen some budget.

And so when that happened, I was heavily relying on cloud, and I say, how am I going to work now, because I have no access? And then I say okay. And the problem with me, that’s something I’ve been working on since I’m born, I’m not a very patient guy. So if I need something and I don’t have it, then I will do it.

So as a result of that, I say okay, and then also one thing, so then I started to invest with a home lab, saying, I just need that. Which was a big mistake. Usually you say, I just need that. No, you wake up the next day and say, I need bigger, and then you need bigger. So, promise, I love home lab, but it’s the best way of losing money, or wasting money to be honest. But it’s amazing.

But the thing that was interesting is that I was testing a lot of Kubernetes stuff, and there was a couple of features related to Kubernetes where you need to have control on the control plane to enable feature flags, to enable things. And I was always frustrated when I was using cloud providers, because I didn’t have that option, or I had to go through creating a VM. I mean, it’s kind of cumbersome.

So as soon as I got my first home lab, that was the first thing I was going to do. I was enabling all those feature flags from Kubernetes, testing the tracing on the API server, testing the tracing on the kubelet, to see what it was actually generating for us, and then seeing how I can do things. It was great. So far, to be honest, now I think there’s no cloud spend from my end since I got my home lab done. But there is some electricity spend on my home now, which is the other downside.

Josh Lee [00:04:57]

Yeah. You’re subsidizing your employer a little bit there, I guess. So it sounds like it’s something you’re actually relying on to run experiments. What else are you using it for?

Henrik Rexed [00:05:09]

So everything that I have, what I’m building on, what I’m working on, is based on the home lab. It’s 100%.

So I started small. As I said, I had two machines just for small Kubernetes needs. And then I quickly realized that I needed more RAM and more CPU. So then I extended by adding more nodes. So I use Proxmox under the scenes everywhere. And then I wanted to move to AI. So then I started to build a node that was GPU- ready.

And then you start to have those types of problems where you say, I needed a private Docker registry. I need my own DNS within the machines. I need my own Vault. And then suddenly small pieces pop up everywhere.

And I think it’s fantastic, because when you’re just dealing with the cloud, you’re doing your experiments or your things as a developer advocate, but you’re missing the complexity that a real enterprise, a real company, will go through. They want to use a private registry, they have some DNS restrictions, they have some security in place. And those things you never – I mean, you touch it, but not really with the sweating and the swearing part.

Josh Lee [00:06:44]

I like that expression.

Henrik Rexed [00:06:46]

Yes. And I think that’s a great way, because you learn, and then you try to figure out how you can do things. And obviously a home lab is great, but doing a lot of manual things, does it make sense? Because you want to be efficient. So you try to bring the automations as much as you can into the game.

Josh Lee [00:07:06]

That makes a lot of sense. Okay, let’s take a step back. You said it’s always growing, so I know this is not a permanent answer, but what do you have? What’s in your stack right now, hardware-wise?

Henrik Rexed [00:07:19]

So hardware-wise, I started with two mini PCs, which were very cheap. I went to Amazon, and I bought, so it’s basically a Chinese-based brand. Two mini PCs which was about, I think, 16 gigs of RAM each, with a CPU that was fair enough for that specific use case.

And then when I started running Kubernetes, I saw I had some issues. So then I invested in a bigger machine, an upgrade that has 64 gigs and has more threads on the CPU side, and I picked the AMD chipset for that. And when I had that third machine, then I said, okay, I’m good now.

And then I was not. And then I added another machine because I needed even more threads and more memory. So I added another 64 gig mini PC in the rack.

And then I started to be worried about the space, the backups and everything. So then I purchased a small machine and I used TrueNAS. And then with TrueNAS I had a couple of discs I added there. And for TrueNAS you don’t need a big instance. It could be very cheap. You just need a disk and put the TrueNAS operating system, and then I connected my TrueNAS as a disk storage for Proxmox. So then I was able to basically get the data out, I mean the discs, the ISOs, and everything around Proxmox.

And then what happened is that I figured out recently that the discs that I purchased, because you want to play with the budget, were great, but I didn’t do a RAID where you can have redundancy. So I didn’t put any, it’s basically RAID zero. So very, very stupid. The best is to at least have a mirroring approach, so if one disc fails you still have a backup, which is what you expect from a NAS.

So then what I did is I purchased a separate NAS. I have a big NAS with NAS discs, a big one, really really big. And this one is a backup. So I have a task that backs up the tasks. No, I have a weekly backup. So I had a daily but I shifted it to weekly. I have a weekly backup of all my stuff on Proxmox, and then I have a scheduled task that every Sunday takes the backups and pushes it to the NAS. So in case my NAS dies, I have another NAS. That was my idea.

And then on the NAS, what I’m doing, I also do the project that I’m working on with the agents, that’s on the other NAS. So my NAS is like the central reference of everything, the other one is more backup for Proxmox and stuff like this, and images and disk images and stuff like this.

And then when I moved to the agent and LLM world, then I realized, I need a GPU. How can I do that? And then this is where I went online. I searched a lot. And then I found the Minisforum MS-02.

I think if I had to rethink my home lab, I would redesign everything with Minisforum machines, because you can buy them with just the chipset and the motherboard and then you can customize the rest. Especially the MS-02 Ultra that has two PCI slots. You have bigger places to put a GPU. I mean, since I bought this machine, I realized that there’s a different quality build. Better cooling. I mean everything is well designed. So next time, yeah, again, I will probably go on the Minisforum route.

The only downside is that when I did that machine, I bought the RAM before the RAM crisis. I was very lucky. I was, honestly, I don’t know, because now I wanted to do the upgrade and I want to have another machine, another instance. And I was saying I’m going to go with the MS-02 Ultra and buy another GPU card and everything. But when I looked at the price of the RAM, I realized that I will spend more on the RAM than on the rest of the machine. I say, doesn’t make sense. So I said okay, let’s pause, let’s try to do what I can do with what I have for now, and then figure it out.

But then when I did this first release of the LLM stuff, I bought online a Tesla T4 from Nvidia, which is great. 16 gigs of VRAM is great, but you cannot do big models. Quite limited. So then I realized, okay, for my needs, I need a bigger machine. And then this is where I purchased a dedicated machine just for LLM. So I purchased the Mac Studio, and this is just running 24/7 hosting Ollama with big models, and I’m just using it on various workloads that I have on my Proxmox.

Josh Lee [00:12:52]

Yeah, lot of money. I have, the RAM in my laptop is now worth more than the laptop.

I have so many questions. Wow, where to even start? So you’re running Proxmox and TrueNAS. Are you running your Proxmox nodes as a cluster?

Henrik Rexed [00:13:10]

So I have a Proxmox cluster with five nodes with all that stuff. So all the images and all the settings are in the NAS. So then if I need to spin up whatever, for redundancy or something, it doesn’t rely on the local storage. So that works super well.

And then what I’ve done is that, because I do a lot of Kubernetes stuff, I have a few clusters that are running 24/ 7. So I have the cluster API management cluster and I use the cluster API. So I already have some pre-made templates, with a very quick command, kubectl apply on the main cluster. Boom, it spins up the machines, defines the control plane, adds the worker nodes. And then I just have to add the kubeconfig locally on my machine and then I can reach out to the cluster. So that is fantastic.

And then I was using a lot of MetalLB for exposing. So MetalLB, for those not aware, is that if you’re doing any ingress or Gateway API and you need to expose your IP address out of your cluster, then what I’ve done is I allocate predefined IP addresses in my local network, and then basically when you request a load balancer IP or service or Gateway API, it will basically pull out from those IPs that you have reserved and it will be exposed from the cluster. So that works perfectly well.

One thing that I didn’t mention is that because I’m traveling, I mean, as you do for conferences, and because now it’s my way to do things, then I have a specific machine with Tailscale. And that machine is basically a way of having access to my local network. I have a private network with all the machines, and then wherever I am I can connect, and then I have a routing rule. So then I can basically use my machine, it’s like I’m working from home. It doesn’t make any difference. I can access Proxmox, I can access whatever I have in my environment, and that is fantastic.

The one thing is that I have a few machines that don’t boot after a shutdown. And it happened, I was going to a conference, it was in the Middle East, and then I arrived, I need to change things and everything, and I realized, boom, I’m not able to access the machines anymore. And then I call my wife, please please please check, are they running, are they running? No, everything is dark.

So yeah, this I need to improve, that approach, at least on the machines that don’t have the bootable stuff. I need to figure out a way to do this, to make it reliable, at least make sure that they always restart whatever happens. So then I’m not jumping into those critical situations.

Josh Lee [00:16:20]

I’ve run into that exact same issue with mini PCs, where the BIOS doesn’t have an auto-restart or a start on power loss setting. And that’s another reason why I, like you, I’ve come to really like the Minisforum machines, because they tended to have those BIOS settings.

Henrik Rexed [00:16:34]

I mean, the Minisforum is basically a server friendly mini PC and it has all the features that you expect from it. So I think, well done Minisforum, because you know in the industry there’s the Apple Mac lovers. I think I’m starting to be a Minisforum lover.

Josh Lee [00:16:58]

Yes. Although on the RAM front it’s interesting, because the Apple RAM tax is less severe now because of this RAM crisis.

Henrik Rexed [00:17:07]

Yeah. But that’s true, that is. Because if you buy, especially if you buy old machines. But what I realize is that the new versions of the Mac, if you look at my specs, because for work I have a MacBook Pro M3 Pro. If you look at the spec of the M3 Pro compared to the M5 Pro that released, I said, what happened with the RAM? Did they divide by two the RAM quantity that’s available? They did. The machine has way, way, way less memory.

And then for the price, okay, I know the RAM is expensive, but I would, if I was an end user, I will never buy an M5. I will go online and check if there’s no M2, M3, because at the end I will have a bigger setup and do whatever I want with the machines.

Josh Lee [00:18:01]

I’m not sure if you cut out or if the platform cut out there. Hello.

Henrik Rexed [00:18:06]

Yeah, I’m here.

Josh Lee [00:18:07]

I think you’re okay. Maybe my iPad is just struggling. It’s okay. We can edit this. The beauty of not recording live.

Henrik Rexed [00:18:19]

Let me check.

Josh Lee [00:18:21]

Okay, I think we’re back in sync.

Henrik Rexed [00:18:25]

Yeah, I’m online.

Josh Lee [00:18:27]

Okay, great. Don’t know what happened there. So I lost what we were talking about, the M3 and the M5 and the memory. We’ll see what happens on the recording with that.

So okay, I have more questions. You’re using TrueNAS for your VM discs?

Henrik Rexed [00:18:46]

Yes.

Josh Lee [00:18:47]

But you’re using mini PCs.

Henrik Rexed [00:18:50]

Yeah.

Josh Lee [00:18:51]

What kind of networking? I have been afraid to go down this route, because all of my mini PCs have gigabit connections and I thought it would be too slow.

Henrik Rexed [00:18:58]

So there are two things, depending on what I need to do. So for example, for etcd for Kubernetes, and for LLM, I try to use local discs on the machine where it’s running the actual VM, because they need fast.

I realized after a few tests that etcd, I mean if you have two nodes that’s fine. etcd is filling up and the disk speed with the network is fine. But then every moment you add more nodes, then it starts to do more stuff. And then sometimes when you do a big apply with a lot of workload that is scheduled in Kubernetes, then suddenly the cluster is not responding, and then you have to restart, and it’s just a nightmare. So this is where I shifted, where etcd is going to be local.

But for the rest, most of the VM discs are in TrueNAS. And I have a router, so all the machines are going to the router, and then basically they have their own local network. But yeah, I’m limited to 2.5 gigs. It’s not 10 gigs. But that’s something I could improve, maybe, if I redesign this entire setup for sure.

Josh Lee [00:20:26]

Yes. Because the upgrades never end.

Henrik Rexed [00:20:29]

This is true.

Josh Lee [00:20:31]

Okay, let’s talk about the AI workloads, because I think that’s the most exciting part for a lot of people here. Although I do want to dive back into what you were mentioning about using cluster API and Kubernetes as well. But yeah, so what are you using for running your AI workloads, and how are you connecting to them?

Henrik Rexed [00:20:44]

So pure AI workloads, I have one cluster of Kubernetes with the Nvidia GPU operator. So I have one node from that cluster that gets access to the card. I use a local, so it’s a small model, it’s an 8B model, but I’m usually using it for inference, for Whisper, for stuff like this, because it’s limited.

Then when it comes to more coding tasks or more complex reasoning things, I use my Mac Studio. So I expose the Mac Studio on the local network. It’s part of the DNS, and then I’ve tested several models and then I selected the one that makes more sense for me as of now. Maybe it’s going to change, I mean things are moving so fast, so it’s changing.

So I’m using a Qwen 3.5 122B on this machine. And the other one that I kind of liked in terms of responses was the GPT-OSS 120B. That was really good. And now, since yesterday, they released Qwen 3.6, I’m waiting for Qwen 3.6 to be a size that I can host on my machine. But I saw the difference with Qwen 3.5. So I’m really intending to wait and then upgrade, get rid of Qwen 3.5, and use Qwen 3.6.

Because honestly, I mean, Claude is amazing, Anthropic models are amazing, but the difference between Qwen 3.6 and Anthropic from the price perspective is insane. It’s really insane, to be honest.

Josh Lee [00:22:55]

Yeah.

Henrik Rexed [00:22:57]

And then, so I, like everyone who is watching this AI buzz, AI world, I felt very weak, and I went into the route of OpenClaw. But I said, I have a home lab and I’m going to be smart.

So I did a VM. So I have a VM that has some discs, it has just limited hardware, so it’s not a big VM. It has OpenClaw on it. And I connected it, so I have a fallback. So I’m connected to my Mac Studio, so in case I’m rate-limited, it goes back to my Mac Studio, so then it’s always up and running.

And then what I did is, I didn’t want him to code. I wanted him as my product manager, my boss. No, not my boss, he is my CTO. I’m talking to the CTO.

And so then, which means I have another machine which has access to my NAS where I store all the coding stuff and everything, and I have Claude Code installed with my subscription. And then I installed on top of that Paperclip. I don’t know if you use Paperclip or

Josh Lee [00:24:09]

I don’t know it. No.

Henrik Rexed [00:24:11]

So Paperclip is, to simplify, you create an org, you create an organization, you define skills for specific agents. So I have my testing architect, I have my architect, I have my observability architect, I have the coder of course, I have the reviewer, I have the product manager, I have the brainstormer. So I have different agents that are there for different purposes.

And then I define a flow of work. So I have an idea. It goes through the product manager. It goes through the brainstormer. The brainstormer validates, goes to the architect. The architect says great. Then he shifts it to the sprint planner. So then he takes one ticket and then splits it into several epics and stories and sprint planning. And then once that is ready he shares it with the product manager to see how they’re going to work, and then they distribute the work to the developer.

Once the developer finishes his task, he asks for the code review. Code review shares the output. If it’s good, we resume, otherwise it goes back to the coder. If it’s good it’s approved, it goes to the testing engineer. Testing engineer approves. And then I have a DevOps engineer at the end that does the GitHub workflow, the CI, and everything.

So I create the workflow and then Paperclip. So at the end it’s one UI, one solution that is in the machine where my Claude subscription is installed. Paperclip is on, and then at the end all those different agents load the code. That’s the only thing it does.

And what I’m doing is that I have a specific workflow, and then you create tickets, you create projects. So I have different projects where I’m working, and then I create tickets, or I request OpenClaw, say hey, I have that idea, blah blah blah, create the project, ask the brainstormer, blah blah. And then once they have done the analysis, they split, and they create sub-tickets. It’s a bit like Jira.

So then, because when I was using OpenClaw I was very hyped, but I was like, give me a status. Because when you’re traveling and you’re on WhatsApp, you want to figure out what’s going on. They have a UI, but the UI you don’t want to expose out there. So then you keep asking, what’s the status? What’s the status?

And here with Paperclip, you basically have tickets. So you can say, okay, this is done. Okay, they are running on this. Okay, things are moving. And then if things are not moving, you say, hey, you go to your CTO, OpenClaw in my case, say, come on, make your team work.

But it’s, I mean, I like it, but from a token consumption perspective now, after I’ve been using it intensively the last few weeks, I think the token consumption and optimization is not ideal. So I’m working on an observability plugin on this project, because I want to fine-tune the agents. Because Paperclip adds this notion where they share the context into a ticket. So then when you give a task, the first thing the agent does, I have a task, and then he goes to Paperclip, looks at the task, okay, grabs it. So then the context when they start working is kind of big.

But I liked this approach because I was using a methodology called BMAD a lot in the past. So BMAD is like an agile for agents, but it was basically in code. So one instance, and then you shift the skills, and you reset the sessions and reset the context and everything.

But when I discovered Paperclip, great, I’m going to take the BMAD methodology and move it to Paperclip. Which I did, but I think I’m consuming more tokens than expected. So I’m currently rethinking how I can make myself more efficient in terms of token consumption, of course, but also with the hardware that I have. But it’s fun. It’s a never-ending game.

Josh Lee [00:28:38]

It’s a never-ending game, absolutely. And in my experience the momentum builds, right? Once the setup you just described, I’m even a little bit overwhelmed, and I’m sure some of our listeners are overwhelmed by all of the complexity that you just described. But where did you start? How did you iterate to that point? Because I’m guessing you didn’t start with all of this.

Henrik Rexed [00:28:58]

I mean, because I started with the infrastructure. So having a home lab lets you, I mean of course if you have available resources, but you have no excuse to say, I need a machine. No, you have it. So you can say, okay, let’s. And then you can fail. You don’t care. You kill the machine and you start from scratch. And that is, I think, an amazing layer for testing, trying, investigating topics. And I’m so happy that I took that route, to be honest. I’m so happy that we had this cloud limitation in the past that let me invest in this. And also it upgraded so much my skills in terms of networking, on security, because you touch so many different areas.

But coming to the agent setup, I started with the natural way of coding, and I was so frustrated when your context is almost big, too big enough, you’re saturated. Then you create a new session. It has some context because you try to store in MD files, but still, I like to compare it like you go to a party and you discover one guy and it’s like your buddy. You spend an amazing party, amazing evening. It’s like a crazy night. You wake up, you go to breakfast, you see that guy and the guy is not even looking at you. He doesn’t even recognize you at breakfast, and you say, what? I mean, come on, we had so much fun yesterday.

And it’s the same thing. So I was frustrated to ask the same thing on and on and on, and sometimes I was yelling in front of it. And so then I started to say, how can I improve it?

So that’s why I discovered this BMAD methodology, and from the moment I shifted to BMAD, I was astonished by the quality of the code produced. I mean, you can vibe code and then you have a lot of issues and then you go through those issues. But here it’s like the first release, you maybe have one bug, but a small bug, not like a structural bug. And I was like, wow, if you are structuring your agents to do things and you follow a real method, wow, the quality is so amazing.

And then, like I said, I wanted to keep track of things that I was doing. And then also I think I’m in an era where instead of being the prompter, I want to delegate the tasks. I want to do something else. So I want to reach out to an agent, say this is what I want, ask me as many questions as you want, and you are going to drive it. And this is where I’m trying to go more today.

And it’s far from being perfect, so I’m trying to fine-tune things. One thing that I try to improve is the memory management, because I realize that I’m over consuming tokens. So I’m trying to find solutions. There is a project that was released called mem palace, for memory. I’m going to try it out locally, and then I think I’m probably going to make a separate machine. So I want to basically have some right sets of storage and solutions that manage memory for agents. So again, with the home lab it’s very easy to test it out. So it’s not a big thing.

Josh Lee [00:32:58]

That’s, wow, this is so cool. I’m going to have to try out all of these things and try and put some of these tools that you’re mentioning into the show notes. Right now I’m very primitive, I think, comparatively. I’m just using Claude Code myself on the command line to work on projects.

And I’m finding that if it’s a green field, if it’s a new thing and it’s small, I can kind of do a good one-shot. But then if I want it to edit something that already exists, or a larger project, I have to babysit it a lot more. So are you finding that with these methodologies that you’re using, that it can apply edits even on large complex projects?

Henrik Rexed [00:33:33]

Yeah. So since I started using it heavily, because I did a few contributions on open source projects, but I wanted to make sure that the quality I was getting out was okay. And then since I started to implement this, I started to realize, I can do. I started to do contributions to homeg, to symposium, to kagent, to agent gateway, to agent evals, to create different MCPs. And I say, I’ve been very productive.

What I would say is that, before Paperclip, when I was using BMAD, I created skills for OpenClaw where OpenClaw can connect to the machine where I have Claude Code and run tmux sessions, and he is driving the prompts. So I was doing this a long time, and then when I discovered Paperclip I tried to shift this approach with Paperclip.

I would say that when I was using BMAD with this skill, the quality was amazing. The quality was super great. And then I shifted to Paperclip, and I realized that sometimes when they create tickets, the agents create tickets on the wrong project.

Josh Lee [00:34:52]

Yeah.

Henrik Rexed [00:34:56]

And also, yeah, sometimes the quality. Yeah, that’s why I’m trying to improve, because with Paperclip maybe it adds a great layer, but I think it loses some things. There are some downsides. So I want to improve that area.

But yeah, when I had this OpenClaw skill using BMAD, so he knows BMAD, OpenClaw, he is preparing, and then he’s able to, he is basically interacting, because BMAD is a lot of questions from the agents, and he’s able to drive. And that was amazing.

And I remember I was on a week skiing, and we were driving to the ski resort, and there was a huge traffic. We spent six hours in the car waiting, not moving. And we were chatting with the kids and everything, and suddenly I was in my project, and I started to chat with OpenClaw, and I said I want this. I said okay, fair enough, let’s do it.

And then in the morning I woke up early before the kids and everything, and I started to do the brainstorming with OpenClaw. So I installed the skills, I created a separate machine for Claude Code on my home lab, and then I was with WhatsApp. So he had all the context, and I was skiing, and then we were waiting for the kids. I took my phone, say what’s the status? Sprint, etc, etc…. And then I was back home after one day of skiing, everything was done, it was pushed on GitHub. There were a few hiccups there on the GitHub actions, but the core was there. And I was saying, that is awesome. Wow, that is awesome. And this is where we’re heading to, huh?

Okay. And then I say, whoa, so if you pay me to go skiing and manage projects, I can do that. I can do that.

Josh Lee [00:36:44]

This is, you’re living the dream. This is amazing. I think someone I saw in my feed, like, I don’t want AI to create art. I want AI to do my laundry and dishes so I can create art. And it sounds like that’s what you have, a little bit.

Henrik Rexed [00:36:56]

Yeah.

Josh Lee [00:36:57]

You’re doing the creative parts, the fun parts, and letting the AI handle the boring parts.

Henrik Rexed [00:37:03]

Yeah. Well, yeah. Also, as you said, there’s always some babysitting, to be honest.

Josh Lee [00:37:08]

That’s, yes. Yes. So how are you giving OpenClaw – are you giving the agents your GitHub keys, push access? How do you manage access?

Henrik Rexed [00:37:16]

No. So I’m very concerned about this. What I’m doing is I’m connecting to the machines and then running the git command. So then once the git is configured, they can do their own commits.

But I’m a bit scared of pushing any tokens on any channels that interact. So I try to, when he needs, for example for SSH access, on my Mac Studio I create a dedicated user with specific rights, create an SSH key, and then I put the SSH key on the machine, so then he has access. Same thing from the other ones. I try to reduce as much as possible sharing credentials with agents, or at least I enable them access without sharing details.

Josh Lee [00:38:08]

So you’re creating separate accounts for them basically, and giving them their own identities. They’re not masquerading as you.

Henrik Rexed [00:38:14]

Yeah.

Josh Lee [00:38:15]

Yeah, that sounds safe. Okay. What are you using? Let’s talk about observability. We’re both observability aficionados. How are you monitoring all of this?

Henrik Rexed [00:38:29]

So when I started with Proxmox, I wanted to have a clear picture on what’s going on. Proxmox gives you a lot of insights. And you did a live stream, you talked about it. So they have a native monitoring for InfluxDB.

And then I said okay, great. Then I looked at the collector. Collector has InfluxDB, but it was not designed to grab that data. So what I did is I created an InfluxDB receiver that is actually connecting to InfluxDB, and then it’s trying to discover the metrics exposed by InfluxDB, and then based on a structure it’s recreating the metrics as we want, and I’m pushing it back to Dynatrace.

And then, because I’m a Dynatrace employee, I have my own tenant and everything, which is awesome. And then I have my own dashboard for Proxmox where I can see the machine utilizations, any potential errors. If I have some things failing, I will be alerted. So I have a collector running there. Same thing for the cluster API. I have a specific cluster API dashboard. So I collect everything.

And then for OpenClaw, what I did is I put Tetragon. I installed Tetragon on OpenClaw, and I was looking for, because when I started there were so many feeds about security concerns about OpenClaw. So I said okay, I’m going to create rules on specific file access and specific things, and I want to be alerted. So then I have a collector as well.

I built also for OpenClaw the observability plugin, because when I started there was something, but it was not great. So I created one that was more, you can see all the interaction you have with the chat, the LLM calls and everything. So I’m using that plugin. Now they have improved it. I need to figure out if it’s as good as my plugin, because I didn’t want to lose traction, because I have a dashboard that relies on the data that I’m producing for OpenClaw. I can see the token consumption, the number of agents running, things like this.

Yeah, I try to put as much as possible the observability part, so I can always have an eye, especially on the agent. I think that it’s a fantastic technology, but you are a human that will configure an agent to do things, and this is where it gets messy. So if we do the wrong prompt, or the wrong skills, or the wrong configuration agent, whatever MD, then it is going to work but it will be less efficient.

So I think observability is clearly how you can improve. So try to evaluate things. I’m trying to contribute to this project called agent evals that solo.io released. They’re mainly running on traces. I’m trying to push them to go to metrics and logs as well, but for the moment they want to keep on traces. But I think it’s a good way. So I think something that checks the efficiency of your agents, and then at least for now be alerted, maybe in the future something that tries to improve it. That would be even better.

Josh Lee [00:41:56]

Did I understand correctly that you built your own InfluxDB receiver for Proxmox?

Henrik Rexed [00:42:04]

Yeah.

Josh Lee [00:42:05]

What drove you to do that?

Henrik Rexed [00:42:06]

Because I said, okay, great, InfluxDB. And then I say, okay, I’m going to look at the InfluxDB receiver that is in the collector, and it didn’t work. I was not able to use it. And then I looked at the design, and the design was made for another use case than mine. And I said okay, so I think my use case, maybe it’s very niche, so let’s build a new one.

And it seems that people, I’m not the only one that has this specific use case, so maybe, I don’t know, either we should merge both receivers, either we should mark it differently. But the one I built is like, it connects and then tries to discover the measurements exposed in InfluxDB and then recreate metrics with the right dimensions.

And that’s how I decided to build it, because my initial goal was to say, Proxmox is cool, but I don’t want to be just looking at Proxmox every day. I want to have something that keeps me alerted.

Josh Lee [00:43:20]

Yeah. Yeah. Absolutely. Is that open source? Is that publicly available?

Henrik Rexed [00:43:26]

It’s on my GitHub repo.

Josh Lee [00:43:28]

So it’s, have to check that out.

Henrik Rexed [00:43:29]

It’s a receiver. So you will have to use OCB. So I’ve built, in this repo, OCB instructions to build it. I added a very basic collector, but if you want to get more you should add this receiver. But I didn’t maintain it, so maybe I need to. Yeah, it reminds me that I need to upgrade the OpenTelemetry version so it’s compiled aligned with the latest collector version. So then you can also add all the, but for now it’s like a couple of months old. So I should have done that. Sorry.

Josh Lee [00:44:05]

Nice. Yep. I’m happy to check that out and help contribute. For those who are not familiar, what’s OCB?

Henrik Rexed [00:44:14]

So OCB is a utility called the OpenTelemetry Collector Builder. So the idea, it’s going to be a utility that will generate an OpenTelemetry collector binary, and it takes as an input a YAML file where you say, I need those extensions, here is the link to the repo, and this is the version. I need this receiver, this processor.

So you basically define what they are, because if you use contrib, you will end up with 70-plus plugins, and usually you only need 10, 12 megs max, I would say. So instead of having a very big binary, you can define which ones you need to use and which versions, and it will generate the binary. And also you can do that from a Docker perspective. So you can use OCB as a library in Docker to rebuild it.

And Bindplane did a CI action that does the same thing, where you could basically build with a YAML structure file the collector that you need in GitHub actions. So I think once you’re using collectors heavily, I think building your own distro makes sense. Especially if you’re using four plugins, it doesn’t make sense to have 70-plus plugins.

Josh Lee [00:45:31]

I’m currently using contrib but I need to get on that bandwagon of building my own. Definitely, especially with the memory crisis being what it is. Every megabyte counts, right?

Henrik Rexed [00:45:43]

Yeah.

Josh Lee [00:45:45]

So okay, you mentioned Tetragon, huge fan of Isovalent and the eBPF tools that they make. Earlier you mentioned you’re using MetalLB. Have you tried Cilium?

Henrik Rexed [00:46:00]

I mean, I use MetalLB because I wanted to have, so MetalLB is not a CNI. It’s going to be something that will expose in your local network an IP address. So it’s a way of getting an IP address for your ingress.

So I have used Cilium, I love it. So far, to be honest, because I was using it mainly in the cloud, with cloud providers. Since I’m in my local lab, I’m using a very simple CNI. I never tried to use it again. I use a lot of Tetragon, to use their eBPF to detect any suspicious activity that could happen on the kernel. But Cilium on my cluster I have not implemented so much.

But I think MetalLB and Cilium could be quite, there’s no overlap, because Cilium will manage basically the network in the cluster, and MetalLB will be more for the outside of the cluster.

Josh Lee [00:47:13]

Makes sense. Which brings up another question then: which distributions of Kubernetes are you running?

Henrik Rexed [00:47:18]

So I’m trying to keep to the latest. The only thing, and here is a call for the maintainers, I think I will have to contribute there. Is that when you use the cluster API you need providers, and if you’re a Proxmox user, you need the Proxmox provider.

The Proxmox provider, as of now, uses old APIs from the cluster API. So meaning that you cannot upgrade to the latest cluster API. You need to be aligned with the version that’s compatible with the Proxmox provider. With that said, it means that for now you’re limited to version 1.33. And if you want to go to 1.34, we need to have the Proxmox provider upgraded.

And I’m checking every month, to be honest, to see if the provider is upgraded, because I really want to, it’s like there are a lot of releases coming in, and I want to basically play with that, and then I’m limited so far due to the limitation of the Proxmox provider. So I think I need to reach out to the maintainers to see where they are, or I will try to put my hands there. Especially, I have a Proxmox environment, so maybe it makes sense, I could test it out, and maybe I could contribute there. I will see.

Josh Lee [00:48:48]

Cool. Are you running any alternate distributions ever, like k0s, K3s, Talos?

Henrik Rexed [00:48:53]

So I started with K3s, and then I said I don’t need it, I don’t want it. I want to have the real community experience, you know, the one that makes you swear, sweat, and learn. So I went that route, that’s why I took that path.

Josh Lee [00:49:12]

Yeah, that’s, if it were easy, we wouldn’t do it, right?

Henrik Rexed [00:49:19]

And yeah, also the thing is, the home lab, the game is to try and fail. So let’s take what is available and try to do it. If it’s too easy then why should I do it on the home lab? I mean of course, sometimes you want to make things easier, but sometimes it’s a way of also learning things.

Josh Lee [00:49:40]

Yeah. Okay. We talked a lot about how you’re using these agents and AI models for development. I’m curious, are you using them for any of the DevOps and the management of the home lab and the VMs themselves, setting up the observability pipelines, things like that?

Henrik Rexed [00:49:54]

So I have this observability agent that I have built. But for me it’s more one agent that is part of the development.

Because when I started, I realized quickly that when you’re asking Claude or whatever to instrument, technically he will instrument, but if you look at the spans that are produced, and the metrics and everything, then you realize, ah, there are a lot of issues on the trace context propagation, there are issues with the semantics not being respected. There’s a lot of things that are bad.

And so I said, okay, let’s create a skill. And also I wanted to have a skill that configures the collector with my expertise. So then I built that skill, and now I rely on this guy a lot. Especially on all the projects I’m building, I try to ask him as much as possible to make sure that the semantics are respected, that we always use the latest SDKs, and that if there is any collector to be built, he will do it.

So I’m using it more in this direction. And I mentioned I’m using this DevOps engineer, so he is really responsible more on the GitHub side, to do the right GitHub actions, to build the right artifacts, to make them exposed. So then when I’m in the home lab, I have an image of reference that I can use in my home lab. So I’m trying to use that as much as possible.

But your point of having something that checks the machines and reports, I love it, and I think I’m going to investigate. Did you try to do stuff like this?

Josh Lee [00:51:47]

Yeah, my Claude agent has an Ansible inventory and access to all of my machines, both the nodes and the VMs. I’m not sure how I feel about that from a security point of view, but from a productivity point of view, it’s working really well.

Henrik Rexed [00:52:02]

Yeah, that’s a good point. Because when I use OpenClaw and everything, I was very cautious, I mean I was worried about him having access to everything. So that’s why I was trying to control.

But the at least what I want is something reporting back to OpenClaw, so then I have a WhatsApp message. Hey, so then I can take actions, especially when I have a backup that fails. When a backup fails in Proxmox, then your image is not available anymore. And then usually you always discover that when you need a machine. So then it would be great to have this mechanism that at least sends me a WhatsApp message through OpenClaw. So yeah, it’s a good point. I will try to build that skill.

Josh Lee [00:52:55]

That sounds really cool.

Henrik Rexed [00:52:57]

Thanks.

Josh Lee [00:52:58]

Yeah. And if you’re open sourcing that and we can collaborate on those, that sounds awesome. Okay. I mean, we’re going to have to talk about this more whenever we see each other, over many beers, because this topic is endless. But we’re running up on time. So I want to ask you a few more spicy questions and then we’ll wrap things up.

So first I’ll ask, what’s something that you think people believe about all of what we’ve been talking about that’s wrong or misguided?

Henrik Rexed [00:53:24]

One of the things is, I think that when you go the home lab route, I am a YouTube consumer, so I go to YouTube, and I think there’s a lot of things missing. And I was wondering, I want to do a dedicated channel about this, because I enjoy talking about it and building content, to basically have something that is replacing that gap.

Because there’s a lot of very basic onboarding. So, let’s install Proxmox, and then the rest, nothing. And then on the cluster API, same thing. I did an episode about it, but I should do more about how you manage it, because it’s something that you will face.

And also the other thing that I was thinking is Harbor. I use it because I have a Harbor registry that caches all the images, because when you are running on your own machine, you’re not Google. So you can pull a lot of images, and you will see that you will be rate-limited by any registry. And then you’ll say. So that’s why Harbor is a very good solution. At least you cache the images, and then you’re not hitting the rate limit.

So there are a lot of tips here and there where nobody is talking about it, and then you discover that. And I think this is where I said, I need to do more on this angle, because I think it’s going to be useful.

I think there are a lot of people covering the hardware piece, which I love, explaining, I built that NAS with a Raspberry Pi. It’s impressive. But then they never go to the, okay, I have a home lab, then maintain it. And this is clearly missing. Home lab is great, but there are also some extra tasks that need to be done, like updating the machines, the packages, the releases.

Josh Lee [00:55:28]

Yeah, it’s not the sexy part, those day two operations.

Henrik Rexed [00:55:34]

No, that’s why I thought that your agents go and update everything. I mean, if I can remove that from my shoulders, it would be great.

Josh Lee [00:55:42]

Okay. If you had to delete half of your setup, what would you keep?

Henrik Rexed [00:55:46]

If I had to delete half of my setup, what would I keep? I will probably delete the first machine that I bought. I think the hardware configuration is not extendable. So I cannot open the machine and add extra discs. I cannot boot in case of. So I think, if I was able to go back in the past, I would say no, don’t click on buy and purchase right now. Take this machine instead. This is what I would do.

I think I’m happy with what I have. I have nothing against that. And also, when I bought my NAS, I bought a few discs. I would probably, these two NAS have more discs available – but I still have the other NAS, so it’s not a big deal, but this is something that I would like to improve.

And also you mentioned one point – the networking piece. So you want to go cost-effective when you start, because obviously it’s going to be a big check that you’re going to have to put on the table. So yeah, definitely, if I could have the choice to upgrade to a router or to a switch that supports a high-speed network, having machines that have two Ethernet ports, yeah, I would love to do that. And now, yeah, I have that, so let’s use it. But the upgrades are based on the price and the priority, so I have a list of upgrades, but I will do it slowly.

Josh Lee [00:57:31]

Yeah, definitely. That makes a lot of sense. I’ve heard the expression buy once, cry once.

Henrik Rexed [00:57:36]

Yeah, but it’s, I mean, it’s like always, you start, there’s nothing where you buy from scratch the best thing. You always say, it’s too expensive. And then at the end you have spent more money than if you had bought at the beginning the right stuff. That’s, yeah, that’s what it is.

Josh Lee [00:57:54]

Yeah. Okay. Last question. What’s the biggest advantage you think you get from your home lab?

Henrik Rexed [00:58:01]

I think I mentioned it several times. For me, it’s like I have a switch. I want to do things, I can do it. My daughter, one of my daughters, wants to go into the engineering route.

Josh Lee [00:58:16]

Incredible!

Henrik Rexed [00:58:18]

Yeah. And I said, you know what, so I created VMs with Ubuntu, her developer laptop and everything. So then she can play, she can connect to these machines easily.

Yeah, I think it just gives you so many options. And then if you want to do vibe coding and test things and create things and host them, you can do it. You don’t have to pay for any cloud subscriptions and be cautious about that. I think that’s amazing.

The only thing that needs to be resolved, I think, is I still even if you have local models, you still rely on those big models, because the models that you can host are way, way, way far from what they are able to host. So I’m trying to figure out: how can I improve this? How can I improve this? And I don’t have an answer yet, but I would love to have this resolved.

Josh Lee [00:59:19]

But yeah, I think we’re all waiting for this to become more affordable.

Henrik Rexed [00:59:23]

Yeah.

Josh Lee [00:59:24]

Okay. Anything you’d like to promote? Anything you’ve got coming up? Any conferences you’ll be at?

Henrik Rexed [00:59:31]

I will be at Open Source Summit, talking about OpenClaw, by the way, and how I secure it with Tetragon. I also show the observability plugin that I’ve built. And I will also talk about another project on the Open Observability Summit, that will be in Minneapolis as well. So those are the next conferences I will be at.

I will also be in Budapest in three weeks from now, or four weeks from now, for an observability conference. But otherwise, stay tuned on Is It Observable. So I’m releasing a few episodes about KubeCon. It’s going to be more interview style, but I’m preparing some content about a couple of things not related to the home lab, but on things that I’ve been playing with in my home lab.

Josh Lee [01:00:19]

Nice. Okay. Well, thank you, Henrik. This has been an awesome conversation, and yeah, lots of fun.