❯ nslookup minecraft-0.minecraft.tristan
Server:		192.168.1.1
Address:	192.168.1.1#53

Name:	minecraft-0.minecraft.tristan
Address: 10.100.0.1

Had a fun breakthrough this morning on the cluster. I had been trying for a while to expose the CoreDNS to the home network. Like anything DNS, there were a number of hurdles to clear along the way.

Until now I had been adding static domain entries at the gateway, which works fine but just feels like too much of an easy way out. Also, having to ask for a new entry every time you pop open a new service is a bit of a pain, especially when you’re just messing around and have no idea which ones you’re even going to keep in the end.

So, the first order of business is CoreDNS. It’s already dynamically providing DNS inside the cluster. It’s exposed as a ClusterIP service:

❯ kubectl -n kube-system get svc
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP
kube-dns       ClusterIP      172.17.0.10      <none>     

You can query it from your node, and it’ll give you a response. If you have a service exposed, you should be able to find a record for service_name.namespace.svc.cluster.local.

❯ nslookup minecraft-0.minecraft.svc.cluster.local 172.17.0.10
Server:		172.17.0.10
Address:	172.17.0.10#53

Name:	minecraft-0.minecraft.svc.cluster.local
Address: 172.17.236.254

Now we have a few problems. First, CoreDNS is exposed via a ClusterIP (here, 172.17.0.10), which is only accessible from inside the cluster. Second, if you query it, it will give you another 172 address, which again is no good from outside the cluster! Also, the .svc.cluster.local domain is kinda clunky.

I came across this Reddit post, which also covers MetalLB, but I just picked out what I needed for CoreDNS, namely setting up the k8s_external plugin, which is a matter of editing its configmap with kubectl -n kube-system edit cm coredns and adding:

k8s_external tristan {                                                                                                                       
   headless                                                                                                                                   
}     

I added the headless keyword because we have a number of StatefulSets and those are headless. The above entry adds the tristan domain to CoreDNS, and when it is queried for service_name.namespace.tristan, it will give the external IP addresses:

❯ nslookup minecraft-0.minecraft.tristan 172.17.0.10
Server:		172.17.0.10
Address:	172.17.0.10#53

Name:	minecraft-0.minecraft.tristan
Address: 10.100.0.1

Progress! However we still have that 172 address for CoreDNS, so we’ll need to expose that through MetalLB:

apiVersion: v1
kind: Service
metadata:
  name: kube-dns-ext
  namespace: kube-system
  annotations:
    metallb.universe.tf/allow-shared-ip: "DNS"
spec:
  type: LoadBalancer
  ports:
  - port: 53
    name: "udp"
    targetPort: 53
    protocol: UDP
  - port: 53
    name: "tcp"
    targetPort: 53
    protocol: TCP
  selector:
    k8s-app: kube-dns

❯ kubectl get svc
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP  
kube-dns       ClusterIP      172.17.0.10      <none>      
kube-dns-ext   LoadBalancer   172.17.233.207   10.100.0.5 

So, now we can query a useful ip (here, 10.100.0.5) and get useful responses!

❯ nslookup minecraft-0.minecraft.tristan 10.100.0.5
Server:		10.100.0.5
Address:	10.100.0.5#53

Name:	minecraft-0.minecraft.tristan
Address: 10.100.0.1

There was more work to do at this point though, because I needed to set up the home gateway to forward queries for the tristan domain to CoreDNS. Coincidentally, Ubiquiti just added that functionality in their latest update:

So now we don’t even need to specify CoreDNS and can just send queries to our gateway:

❯ nslookup minecraft-0.minecraft.tristan
Server:		192.168.1.1
Address:	192.168.1.1#53

Name:	minecraft-0.minecraft.tristan
Address: 10.100.0.1

I have a confession: I skipped pretty much all of hardware virtualization and dove straight into containerization and orchestration. I had been doing just fine till now. My ENCOR Lab Manual arrived.

I had purchased a license to Cisco Modeling Labs during Black Friday and hadn’t really used it till now. You’ve got a few options – namely bare-metal or VM. I do have a few viable machines for bare metal installation, but that would have been too easy, right?

VMware Workstation was easy enough to get running on Nunu, our shared family server. I did run into a day’s worth of issues becuase I was trying to get it working through SSH X11 tunneling. The CML image wanted some swap space, which, unfortunately, is disabled on Nunu for the sake of Kubernetes. The alternative was to enable reserved memory from the host, which required root permissions.

This was where I ran into the “never run a GUI as root” issue because VMWare Workstation has a graphical console. I still don’t get what was going on but I think because I was SSH’ed in, pkexec was losing the DISPLAY variable. Eventually I installed TigerVNC on Nunu and my laptop and VNCed in as myself, which let me run it with the right privileges.

CML is now working properly and I finally get to do something with it. Things like these are why I try to keep as close as possible to CLI and text-based work. Whenever there’s an environment to build out and work in, I always find myself fighting with the environment instead of working with it.

Anyway, time to stop whining and get to work. I’ve got these in my .zshrc to bring it up and down:

alias cml-start='ssh nunu vmrun -T player start /home/jay/vmware/cml2_p_2.8.0-6_amd64-32/cml2_p_2.8.0-6_amd64-32.vmx nogui'
alias cml-stop='ssh nunu vmrun -T player stop /home/jay/vmware/cml2_p_2.8.0-6_amd64-32/cml2_p_2.8.0-6_amd64-32.vmx nogui'

One of several undocumented achievements this year, and probably my proudest, was this speed/tach for EA’s F1 2024. I managed to pull in the whole family to make this happen, with some amazing contributions from each member.

Work had sent me to Cisco Live in June, where there was a pretty sizable McLaren F1 presence since Cisco are one of their major sponsors. Zak Brown and Oscar Piastri were there for one of the major keynotes.

Anyway, there was a Splunk demo booth at the show floor with some racing simulators set up. I tried my hand and managed to get second place at the time (not that huge of an achievement since it was pretty early in the day. I checked back in a few hours later and the best time was several seconds faster than the one I had posted). On the sides of the booth, Splunk was demoing a plugin that tracked telemetry data streaming out from the game. It saw everything from tire temperatures to steering and throttle/braking inputs, but for the purposes of Cisco Live, it had a simple map of the Montreal track with a heatmap of speeds that players were traveling at.

The main takeaway from this was, there was some manner of data being streamed out from the game. Some further research brought me to a forum post where there was a pretty formal looking pdf file detailing the spec.

Apparently, when set to do so, the game puts out a stream of UDP packets to a port and destination of your choosing. The payload of each packet is a bunch of unstructured binary information to pick apart. This was where I enlisted my wife, who deals with data streams in her day job. I showed her the doc from EA, and she knew exactly what to do with it. At this point it was only a passing “Hey wouldn’t it be neat” sort of thing but a few hours later she came back and told me she had written a preliminary parser for the telemetry stream, and from there it was up to me to figure out what to do with that data.

I had to tweak the code a bit because I really just wanted to focus on the speed and tachometer metrics (you really get everything out of this stream, including car pitch/roll, track temperature, individual tire temperatures, etc.), but my contribution to this all was on the network side. The only way to test the code was to capture a live stream, and the only way to have a live stream was to be (ostensibly) playing the game. For the first few iterations I would have to play the game while my wife watched the printlines to the console, but I realized I could simulate this with a packet capture.

Wireshark has a replay feature where you can give it a recording of packets and it will send those back out on the network at your request. This is where things like replay attacks happen. Anyway, I recorded the UDP stream of myself driving around Suzuka, and had Wireshark replay that over multicast on the network. From there, whoever was interested could listen in and test their parser.

So, now we have some relevant data. What next? Our kid has a Circuit Playground Express that he noodles around with now and then. With the CRICKIT attachment it can control a few servos, which we would use for the gauges. The only problem was, it had no network capabilities. That was where the Feather M0 came in. It has wifi, but no CircuitPython support, which was key for having our kid involved. So, I had to configure the Feather to talk to the CircuitPlayground via a serial stream over the alligator clips.

Our kid was in charge of the servo code. His task was to write a function with two inputs, and move the servos to the right spots on the gauges. I have a huge regret here in that we did not preserve the code because we were just coding it live on the CircuitPlayground. Along the way we had a handy little test and calibration suite too. It was relatively simple, but the code was entirely his and we were all extremely proud of it. Unfortunately it was wiped when we started our next project, which I may get around to posting here.

In any case, it works! Here’s us testing it. He did a great job on that chicane too! Github repo for the project here.

He was 8 years old at the time of the video.

Uh oh.

Now that it’s almost the end of the year I reminded myself that one of my 2024 New Years Resolution was to document my accomplishments more, and this blog was supposed to be the medium. I got a few good ones at the beginning of the year but like all my New Years resolutions I fell off at around April.

Since the last post I’ve reinstalled my OS a few times, taking my home folder along with me, which for the most part has been working just fine.

…until now.

I had to reinstall Ruby to get Jekyll going again. Unfortunately what I got was the above error, along with a whole day’s worth of other errors as I fell into a edit-check-retry loop.

Eventually I came across this line in the Arch page for Ruby:

A sudo pacman -S ruby-sdlib finally cleared up the last of it. I’m actually not sure if that was what started everything in the first place but it was the end of several hours of reinstalling and updating and uninstalling Ruby and Gems and Jekyll and editing files all over the place.

When Arch says to read the manual, you really can’t skim the manual because of things like this where there’s a line that says “you could do this thing if you needed to, and chances are you do need it, but we’re going to leave it up to you.”

I’ve finally finished this project and it’s working exactly as intended. My only regret is that the two LED displays are slightly crooked. I had taken for granted that they would somehow align themselves on the circuitboard but I only realized that wasn’t the case after I had soldered the 14 pins.

It’s extremely visible and readable from across the living room.

In any case, another one in the books. It’s in Fahrenheit for now because the numbers were bigger. A single byte would cover up to 256 if I only wanted to display three-digit integers, but I wanted four digits of precision, so I went on a quest to learn how to send two bytes at a time, which turned out to be less than trivial. Send-a-byte/read-a-byte is pretty simple but sending an array of bytes required I reacquaint myself with C arrays and memcpy(). I had gotten too comfortable with Python’s list slices and completely forgotten the syntax for pointers.

Python and Arduino code for this project is here