SSM and socat Port Forwarding to Access Private VPC Resources

AWS System Manager Session Manager added the port forwarding feature, announced in this blog post back in 2019. In this post I’ll show you how to leverage SSM and socat port forwarding to access systems in a private subnet that don’t have the SSM agent installed.

You’ll use an SSM agent enabled EC2 instance as an initial target for the ssm port forward session. On this instance, you’ll run socat as a relay for the incoming TCP session to the other instance that does not have the SSM agent.

What is socat?

To quote the official man page, socat (SOcket CAT) is a multipurpose relay. It is a command line tool that establishes two bidirectional byte streams and transfers data between them.

You can use it to connect all sorts of channels. For example:

  • files
  • pipes
  • devices
  • sockets, such as TCP, UDP, IPv4, etc
  • SSL sockets
  • programs

SSM and socat Port Forwarding Example

In my example I have an AWS EMR (Elastic Map Reduce) master node running a web dashboard for ganglia in a private VPC subnet.

I don’t want to add a bastion host / jump box or provide SSH access from the public net.

SSM would provide a nice way for me to connect a remote session, or port forward using IAM authentication and negating the need for any ingress security group rules, but only if I had the SSM agent available on this instance.

Seeing as though the EMR master node is not SSM agent enabled, and I can’t use SSM port forwarding directly to this instance, we could use an interim machine with SSM as a jump box.

Example Configuration

Here is how I configured port forwarding in my use case to access ganglia on a private instance EMR node.

  • The EC2 instance with SSM agent must have an IAM policy attached that allows the relevant ssm access. The blog post linked above has instructions. In a nutshell though, most standard Amazon AMIs include the ssm agent. Your EC2 instance profile should include the required actions too. The AmazonSSMManagedInstanceCore managed policy includes these.
  • Install socat on the SSM agent enabled interim machine the private subnet. For this I connected an SSM session to get shell access and ran sudo yum install -y socat
  • Now I needed to open a source channel for the SSM port forward aws cli command to connect, and connect that source to the destination of the EMR master node running ganglia.
socat TCP4-LISTEN:8080,fork,reuseaddr TCP4:10.0.4.149:80

The command listens on port 8080, and forwards TCP to the EMR node, 10.0.4.149 on port 80. Importantly, the command uses fork and reuseaddr to allow multiple connections.

  • Next is to use the AWS CLI ssm start-session command to start a port forwarding session to the interim instance with the SSM agent running. Grab the Instance ID for the EC2 machine and:
aws ssm start-session --target {your-instance-id-here} --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["8080"],"localPortnumber":["8089"]}'
ssm and socat port forwarding in action

If you setup socat correctly to listen on port 8080, then the connection should be opened and accepted.

Now you can simply open a web browser locally and direct it to http://localhost:8089/ganglia to access ganglia on the remote EMR master node.

Accessing EMR cluster memory stats via the remote port forwarded session.

Closing

AWS SSM is a useful tool to get access to instances in a secure, audited fashion without needing to open up risky SSH access or other remote ports to the public internet.

When constrained and needing a jump across to an instance without the SSM agent you can leverage tools to help. Socat is one such tool that can facilitate this within the private network.

SpotiPod – Spotify Streaming Device from an iPod Classic

spotipod classic

I recently came across the sPot: Spotify in a 4th-gen iPod (2004) project on hackaday.io by Guy Dupont. This post is my go at building Guy’s project from the ground up – the SpotiPod.

Here is my version, up and running, albeit with some hardware leaking out the side for now…

The main components required in terms of hardware are:

  • iPod Classic (4th gen) – at least the device case and clickwheel components
  • A Raspberry Pi Zero W
  • 2″ LCD display (see note further on about an alternative)
  • 3.7V 1000mah LiPo battery
  • Adafruit boost module (to boost 3.7V to 5V required by the Pi, LCD, etc…
  • Adafruit USB charge controller module

Once it’s all connected and configured, you’ll be able to load up your own Spotify playlists and libraries, browse them, and play them all from your iPod Classic device over a bluetooth speaker or remote system.

The SpotiPod build almost complete.
Some wire reduction still required before I can completely close it up.

SpotiPod High-level Build

I won’t be getting into the low-level parts of the build, so for specific details I would recommend viewing Guy’s project log and branching off the posts there.

Initial Raspberry Pi Zero W setup with 2″ LCD display

I started off loading Raspbian Lite OS onto an 8GB microSD card and booting up a barebones Pi Zero W. The W version is important because it has WiFi and Bluetooth on the board.

My first task was to configure SSH to start automatically on boot. Useful when I have no display to start with. I also added some of the required software components to begin testing and messing around with, including redis server, compiling the click c++ application that Guy wrote to interface with the iPod’s clickwheel, and setting up some of the X11 components for a basic UI.

Soldering up the connection for the 2″ LCD display was my first hardware connection task. My soldering iron might come out once in a year, so I’m a bit of a novice in this area, but I managed the task on my first go.

Connecting the LCD to Raspberry Pi Zero via Composite Connector
Connecting the LCD to Raspberry Pi Zero via Composite Connector
testing the LCD
First LCD test successful

I posted more details here on the 2″ LCD connection and configuration in software.

Connecting the Charge Controller and Voltage Boost Modules

Next up I focused on getting the power delivery working. The goal was to be able to charge the LiPo battery via USB and have the circuit supply 5V to all SpotiPod components where required.

I familiarised myself with the Adafruit module documentation and pinout diagrams before connecting the circuit up.

  • Adafruit Powerboost 1000 module
  • Adafruit Charge Controller

Here is a useful diagram to follow, posted by Kakoub on the hackaday.io project page, re-hosted here in case the imgur upload ever disappears:

Click for a larger, clearer version

So after getting power delivery and connections soldered in place, and precariously placing all the components away from eachother to prevent shorts, here is where I was at:

SpotiPod Powerboost and Charge modules connected
Powerboost and Charge modules connected

Testing the Click wheel and Software

The click wheel connectivity is made easier with an 8 Pin FPC cable breakout board. I hooked this up next, soldering the 4 wires required for power and data.

The click wheel ribbon connector could then snap into the breakout board ribbon connector. This is by far the most delicate part of the build in my opinion. The ribbon cable is super thin and delicate.

Connecting the 8 pin FPC breakout board. My soldering skills slowly coming back with a bit of practice…

Testing the interface was a case of compiling the clickwheel program with gcc over an SSH connection then executing it with everything connected.

If wired up correctly, the program will output touch and click data to stdout.

iPod Classic click wheel testing over SSH

Gutting the old iPod Classic

I managed to snag an old iPod Classic 4th gen off eBay for about £20. It wasn’t working, but had the two bits I needed – the chassis, and the clickwheel.

Breaking out my trusty iFixit essentials toolkit, I set about opening it up to remove the unecessary components.

The method I found to work was to pry it open on the left and right edges using the pry tool. Once you can get into one of the edges, slide the tool around, and things get easier.

Insulating the case and components

With most of the electronics connected, I began insulating things with polyimide tape. Most importantly, the metal iPod case. I put down about three layers of tape and tried to cover all parts as best I could.

Insulating the iPod classic shell for the SpotiPod build.

Installing into the iPod Classic Case

This is the tricky part. It’s quite difficult to squeeze all the SpotiPod hardware in.

I started out by strengthening all my soldering connections with a bit of hot glue.

adding hot glue to the soldered connections for the SpotiPod
Adding hot glue to the soldered connections

After a bit of arranging, squeezing, and coercing, everything fits… Mostly!

Things are still not perfect though. I need to reduce my wire lengths before I can get the case to fully close.

For now though, everything works and I have a fully functional SpotiPod!

Tips and Tricks

I’ve put together a list of things that might help if you do this yourself. These are bits that I recall getting snagged on:

  • Make sure you setup all the X11 and software dependencies correctly. Getting Openbox and the frontend application to launch on start up can be tricky and this is crucial. Pay attention to your /etc/X11/xinit/xinitrc and /etc/xdg/openbox/autostart configurations.
  • You don’t have to use the more expensive Adafruit composite LCD display. Ricardo’s build at RSFlightronics uses a much cheaper LCD and some creative approaches to get display output working.
  • Watch out for the click wheel ribbon orientation when you connect it to the breakout board!
  • Use thin and short length wires for connections where possible. Not too short though as it is useful to be able to open the device up and put the two halves side-by-side.
  • Make sure you have a Spotify Premium subscription. I can’t remember exactly, but I’m sure that creating your own app to get your client and secret keys, or some of the scopes required for the app will only work on Premium. (It might have even been spotify connect).
  • You’ll need to configure your own Spotify App using the Spotify Developer portal. Keep your client and secret keys safe to yourself. Remember to setup environment variables with these that the openbox session can access.
  • The frontend/UI application has a hardcoded reference to the Spotify Connect device as “Spotifypod”. Keep things simple by setting your raspotify configuation to use this name too, otherwise you need to update the code too.
  • If you’re struggling to get the software side working at first, it can really help to setup VNC while you debug things. This allows you to get a desktop environment on the Pi Zero and execute scripts or programs in an x session as openbox would.

Thanks again to Guy Dupont for his excellent SpotiPod project and idea. Putting this all together really makes for a fun and rewarding hardware/software hacking experience.

Tiny 2″ TFT Composite Video on Raspberry Pi Zero

This weekend I wanted to test composite video on the Raspberry Pi Zero. I had a Raspberry Pi Zero W and a NTSC/PAL (Television) TFT Display – 2.0″ Diagonal from Adafruit.

This display is ridiculously small. It’s quite something to boot up Raspbian with the PIXEL desktop environment on a Pi Zero with this little 2″ display.

Hardware Wire Up

The display only needs 4 connections.

  • Power (I’m using 5v from a Adafruit Power Boost 1000 basic module)
  • Ground (ground connected to pin 14 / ground on Pi Zero and negative on the boost module)
  • Positive (yellow) from TFT display board to TV on the Pi Zero
  • White from the TFT display board to the other pin, next to the TV pin.
connecting composite video on raspberry pi zero with the TV pin

Here’s what everything looked like after connecting the boost module (3.7v to 5v conversion), the battery charge controller, LiPo, and 2″ TFT display.

boost module, battery charge controller, LiPO, and TFT display all connected to the Raspberry Pi Zero

Powering up and Configuration

Power up and enjoy the tiny display outputting the boot up sequence.

The display is meant to run at 320×240, so after booting up I edited /etc/config.txt to set this up along with some overscan tweaks.

sudo nano /etc/config.txt

Set the following in the /boot/config.txt file:

# these overscan settings are what worked well for me
overscan_left=-26
overscan_right=-26
overscan_top=-16
overscan_bottom=-24

framebuffer_width=320
framebuffer_height=240

Although it is really expensive for what it is, the 2.0″ TFT display is great for small electronics projects that call for full display output. It’s simple and easy to connect, and doesn’t take up too much space either.

Cheap Minecraft Server in AWS with Docker and Traefik

minecraft-like figure on the grass

According to the Minecraft Realms plan pricing page, you can get a realms server at around £5.59 per month. You get some nice conveniences there but… I refuse to pay much at all when I can throw some infrastructure together myself in the cloud to create the ultimate cheap Minecraft server.

Considering my Docker instance running Traefik hosts another 3 or 4 of my personal services along with a Minecraft server, then this solution only costs me around £1.50 a month.

I chose to go with a single AWS EC2 instance that runs Docker. Minecraft runs in a container and sits alongside other personal websites and services that I host there too.

I use Traefik to route traffic coming in to this single host for various TCP ports as well as HTTP(s) on different hostnames. This essentially levels up the cost savings even further as I don’t need multiple EC2 instances (one for each service), and I don’t even need to pay for something like an application or network load balancer, as Traefik does this for me.

A Quick Review of Alternatives

There are other alternatives to consider if you’re looking for a cheap Minecraft server, so don’t take this as being the only option. Here is what I’ve used in the past before settling on my current solution:

  • Minecraft on a dedicated cloud VM. If you just want a dedicated Minecraft VM in the cloud, then DigitalOcean is a good, cheap option. You can also get fairly cheap instances Vultr.
  • Running Minecraft on my own personal Raspberry Pi Kubernetes Cluster. I was even able to expose it over the internet for friends to play on by leveraging a Pi device as a dedicated router. I then used port forwarding to get it working through my double NAT setup. The ARM container was a little slow as a server for more than 2 or 3 players on Raspberry Pi hardware though.
  • Minecraft Server on a home PC / Workstation, with port forwarding to allow other players to connect. This is not ideal, especially on Windows machines or systems that you don’t want to leave running 24/7 as you would for a dedicated server.
  • Various other Minecraft-as-a-service providers. These are decent options in some cases. However for me price and control are important, and I much prefer to self host in this case.

Cheap Minecraft Server in AWS EC2 with Traefik

I used my Cheap Traefik EC2 Docker Hosting solution as the base. You can read that article to get access to the CDK resources required to deploy it yourself.

The cost benefits to using this particular recipe are:

  • EC2 Graviton2 ARM based processor – slightly cheaper to run than Intel and AMD. The downside is more limited software choices. You need to make sure you use ARM compatible packages or Docker images.
  • Spot instance – this has massive savings over a normal lifecycle EC2 instance. The downside is that it can be terminated at any time with only a couple of minutes of notice. When using these you need to make sure you have good data persistence that is not local to the EC2 instance. I personally use a mounted EFS volume. It is re-attached to a new instance from the autoscaling group if the old instance is terminated.

If you don’t use the CDK solution I mentioned above, then alternatively deploy yourself an EC2 instance. Give it an elastic IP address, set up the Security Group ingress rules accordingly, and get shell access. First thing you’ll want to install is Docker, then you’re pretty much good to go.

Minecraft Docker Image

I found a great Minecraft Docker image that is well maintained and has the correct ARM image builds for use on Graviton2 hardware. Check out itzg/minecraft-server. There are other arch builds there that’ll run on just about any other platform.

Docker Compose Service

If you use docker-compose, then here is the simple service definition to get things running.

version: "3"

networks:
  web:
    external: true
  internal:
    external: false

services:
  mc:
    image: itzg/minecraft-server:2021.1.0-multiarch-latest
    environment:
      EULA: "TRUE"
      VERSION: "1.16.5"
      ENABLE_AUTOPAUSE: "TRUE"
      OVERRIDE_SERVER_PROPERTIES: "TRUE"
      MAX_TICK_TIME: "-1"
      TYPE: "BUKKIT"
    labels:
      - traefik.tcp.routers.mc.rule=HostSNI(`*`)
      - traefik.port=25565
    networks:
      - web
    volumes:
      - /data/mc:/data

The docker-compose definition will run a Docker container using the latest multiarch image (which will run on ARM devices). When starting, the container will prepare and run a Minecraft 1.16.5 server. It will also use Bukkit and enable auto pause. The game server does not tick over when there are no players connected.

Traefik Configuration

In the docker-compose definition above, you might have noticed the container labels. The labels prefixed with traefik are used to inform Traefik of how to route network traffic.

the cheap minecraft server uses a Traefik TCP router with HostSNI *
The TCP router using HostSNI on *

In our case, TCP connections are required on port 25565 and HostSNI is used to route those coming in for * (all hosts). The TCP connections on port 25565 go to Traefik, and based on this rule, directed to the Minecraft container.

There is one limitation to be aware of here, and that is that you can only use HostSNI with * for connections that do not use TLS. This is because Server Name Indication (SNI) is an extension of the TLS protocol.

I don’t believe Minecraft supports TLS in any case though. It just means that you won’t be able to have more than one Minecraft server container using the same port running on the single Docker host.

Finishing Off Configuration

Lastly, you might want to point a convenient Host record (A record) to your AWS EC2 Elastic IP address. For example: yourmcserver.example.com -> 1.2.3.4.

All being well, you should now be able to find and connect to your server.

minecraft server listing

WSL2 GUI X-Server Using VcXsrv

wsl2 gui desktop

I needed to set up a WSL2 GUI recently on my machine (WSL2 running uBuntu 20.04.1 LTS). I found a guide that runs through the process but found that a few tweaks needed to be made. Specifically, the communication to VcXsrv was being blocked by Windows Firewall.

There were also a couple of extra tweaks needed for audio passthrough using PulseAudio and setting a windowed resolution.

Setting up a WSL2 GUI X-Server in Windows

Start by installing xfce4 and goodies.

sudo apt install xfce4 xfce4-goodies

If you’re running Kali you should use:

sudo apt install kali-desktop-xfce

During the install you’ll be prompted about which display manager to use. This is up to you, though I personally chose lightdm.

Download this .zip package which contains VcXsrv and PulseAudio along with some configuration and a shortcut to launch.

Extract it to the root of your C:\ drive. You should end up with contents under C:\WSL VcXsrv.

WSL2 GUI vcxsrv package contents

Run the vcxsrv-64.1.20.8.1.installer.exe installer in this folder, choosing defaults for the install.

Once installed, you’ll want to enable High DPI scaling for VcXsrv in Windows.

  • Navigate to C:\Program Files\VcXsrv
  • Right-click xlaunch.exe and go to Compatibility
  • Click Change high DPI settings and choose Override high DPI scaling behavior. Ensure Application is in the dropdown.

Next, edit the startWSLVcXsrv.bat batch file and change the last line that reads ubuntu.exe run to one of:

  • ubuntu2004.exe run in the case you are using uBuntu 20.04 from the Microsoft Store for WSL
  • ubuntu1804.exe run if you are using uBuntu 18.04 from the Microsoft Store for WSL
  • ubuntu.exe run for when you are using standard uBuntu from the Microsoft Store for WSL
  • kali.exe run if you installed Kali-Linux from the Microsoft Store for WSL

Pin the WSL VcXsrv shortcut somewhere convenient like the taskbar.

Opening Windows Firewall for VcXsrv and PulseAudio

Next you need to allow inbound traffic to Windows for VcXsrv and PulseAudio.

Open Windows Defender Firewall with Advanced Security and add two new Inbound Rules as follows:

  • Type: Program
  • Program path: %ProgramFiles%\VcXsrv\vcxsrv.exe for VcXsrv and %SystemDrive%\WSL VcXsrv\pulseaudio-1.1\bin\pulseaudio.exe for PulseAudio
  • Allow the connection
  • Profile: Domain, Private
  • Name: vcxsrv or pulseaudio depending which rule you are adding

I personally added the following to ExtraParams under the XLaunch node of config.xlaunch. This sets windowed mode to 1920×1080 for monitor #1 on my machine.

-screen 0 1920x1080@1

Viewing your WSL2 GUI

With all of that setup out of the way, you should be able to simply launch VcXsrv from the pinned shortcut and everything should work.

Try it out and you should get Desktop up and running for your WSL2 environment.

WSL2 gui example with audio settings open

PulseAudio passthrough should also be available if you check your sound / volume settings. Try an audio test using alsa-utils:

sudo apt install alsa-utils
speaker-test

Kudos to this guide on reddit for most of the setup instructions. As mentioned before, I needed to configure my firewall and also added some tweaks for windowed mode.

Troubleshooting

If you find your VcXsrv Server display window is blank when launching, try the following:

  • Double-check your firewall rule is allowing inbound connections for vcxsrv.exe for the domain and private scopes.
  • With the black X-server / display window from VcXsrv still open, launch a WSL shell separately, and run the following to set your DISPLAY environment variable:
export DISPLAY=$(grep -m 1 nameserver /etc/resolv.conf | awk '{print $2}'):0

This takes the IP address of your host machine (conveniently used as a nameserver in your WSL Linux environment for DNS lookups) and sets it as the Display remote location (with :0 for the display number appended).

Now, try to launch a xfce4 session with:

xfce4-session

If all goes to plan, the session should target your machine where VcXsrv Server is running and your display window should come to life with your WSL environment desktop.

ZFS ARC Assisting Frequent, Random IO from Web Traffic

ZFS includes some great features that help deliver lightning fast read speeds for storage. Two such features are Adapative Replacement Cache (ARC), and a second possible cache tier, L2ARC. Together these features can dramatically decrease the number of reads required from magnetic storage.

Usually ZFS will assign all but 1GB of system memory to ARC. When files are retrieved they are cached in system RAM dedicated to ARC. This allows them to be retrieved very quickly the next time they are read.

A second tier of cache named L2ARC is also possible. This is where you can dedicate solid state disks to act as another tier of cache. L2ARC only really makes sense when you are fronting slower magnetic spinning disks with cache, and you won’t have enough RAM to support a large enough ARC either. L2ARC doesn’t make a lot of sense if you are already running storage pools across solid state drives.

Another point to consider is that if your requirements mandate a large ARC size, system memory will start to get very expensive. If you are using magnetic storage tiers, then using SSD for L2ARC becomes a much cheaper option than simply dumping piles of money into more memory.

My personal FreeNAS server has 16GB of RAM, with much of that dedicated to ARC.

Analyzing ZFS ARC hit rates for Web Traffic to this Blog

I’ve been running this blog on my personal Kubernetes Raspberry Pi Cluster, which has its storage provisioned via NFS from my FreeNAS storage server.

The storage and cache breakdown is:

  • 4 x 3TB SATA spinning disks, RAIDZ1
  • 2 x 250GB SSD SATA disks, ZFS Mirror
  • 16GB RAM, around 4GB used for jails and VMs, the remaining for ARC
  • No L2ARC. NFS storage is provided from the SSD based storage pool

This blog has a bunch of static files along with it’s WordPress installation, plus a MySQL database. Both of these sets of files are provisioned from NFS on the ZFS mirror storage pool, utilizing Kubernetes PVs.

The SSD storage is already quite fast, but the cache features of ZFS really help here when serving frequent random IO generated from web traffic. Just take a look at these two charts:

ZFS ARC hit ratio with ARC requests chart.
ARC Hit Ratio and ARC Requests. Notice the really high (almost 100%) ARC hit ratio.

Instead of constantly reading from SSDs and causing unecessary wear, web traffic file requests are mostly served from ARC.

From these two graphs, I can tell that right now 16GB of system memory is perfectly fine for my home storage and web traffic serving needs. ARC is handling these workloads perfectly, with a very high cache hit ratio too.

Secondly, I certainly don’t need L2ARC. Web content is already sitting on SSD storage. Additionally, even if files were on magnetic storage, the main ARC would still be able to serve almost all web traffic.

Conclusion

ZFS ARC impresses all around. On it’s busiest day of web traffic last year, this blog saw 12000 sessions, and around 13500 page views in just a few hours. ZFS ARC happily served the site’s storage needs from system memory.

In fact, the ARC hit ratio actually increased to just over 99% at this point in time!

ZFS ARC hit ratio above 99%
Over 99% ARC hit ratio