Using Node.js Streams to Create a Toy Version of jq

To play around with Node.js streams, I made a simple ‘toy’ version of jq, a handy command line JSON processor.

To be clear, the real jq utility is very lightweight and has a ton of functionality. In this post I’ll be mimicking some of it’s functionality and using Node.js to do so.

This will of course result in a bloated tool with way more bundled in than what we actually need.

That being said, it’ll be a useful exercise to go through to learn a little bit about Node.js streams.

In the github repository, you’ll see how easy it is to hack together a very simple command-line tool to process data streamed in from stdin.

node.js stream transform example with PowerShell pipeline processing.
JSON going in to the tool, being filtered, and then converted to an object in PowerShell through the pipeline.

Node.js Streams

The stream documentation describes them as as:

A stream is an abstract interface for working with streaming data in Node.js. The stream module provides an API for implementing the stream interface.

For this example I’ll be jumping straight into the Transform Streamstream.Transform. This is a duplex stream, where the input would usually be related to the output in some way.

Transforming Input from stdin

The basic use of a Transform stream in Node.js (to process input from stdin) looks like this:

const {Transform} = require('stream')
const TransformStream = new Transform;
TransformStream._transform = (chunk, encoding, callback) => {
    // do something with the chunk
    console.log(chunk.toString().toUpperCase());
    callback();
}

process.stdin.pipe(TransformStream);

Toy jq Utility With Transform Streams

I created a ‘toy’ version of jq using a Node.js Transform stream. It’s a very quickly hacked together example, so don’t expect it to do everything that jq can do. I’m also fully aware that the real jq utility is a very lightweight tool and that doing this in the Node.js runtime adds a lot of unecessary bloat!

This is purely for demonstration purposes.

Packaging up the Node.js app with pkg, we get a platform specific binary called toyjq.

Examples

Using node.js streams - toyjs usage examples

Pretty print input JSON

cat ./example.json | toyjq-linux

{
  name: 'directoryobject',
  path: '/path/to/directoryobject',
  type: 'Directory',
  children: [ { foo: 'bar' }, { foo: 'bar1' } ]
}

Output the `type` field only from input JSON:

cat ./example.json | toyjq-linux

cat ./example.json | toyjq-linux '.type'

"Directory"

Output just the `name` and `children` fields in the input JSON:

cat ./example.json | toyjq-linux '{name, children}'

{
  name: 'directoryobject',
  children: [ { foo: 'bar' }, { foo: 'bar1' } ]
}

Assuming now the Windows platform version of toyjq, and using a PowerShell cmdlet for this example…

Select the children array, convert it to an object in PowerShell and then select the last item in that object array:

(cat .\example.json | .\toyjq-win.exe '.children' | ConvertFrom-Json).foo | Select -Last 1

bar1

The above examples show how you can easily process data from one input pipeline (stdin in this case) and send it along through the pipeline using Node.js streams.

You can find the example toyjq app on my GitHub repository.

SSM and socat Port Forwarding to Access Private VPC Resources

AWS System Manager Session Manager added the port forwarding feature, announced in this blog post back in 2019. In this post I’ll show you how to leverage SSM and socat port forwarding to access systems in a private subnet that don’t have the SSM agent installed.

You’ll use an SSM agent enabled EC2 instance as an initial target for the ssm port forward session. On this instance, you’ll run socat as a relay for the incoming TCP session to the other instance that does not have the SSM agent.

What is socat?

To quote the official man page, socat (SOcket CAT) is a multipurpose relay. It is a command line tool that establishes two bidirectional byte streams and transfers data between them.

You can use it to connect all sorts of channels. For example:

  • files
  • pipes
  • devices
  • sockets, such as TCP, UDP, IPv4, etc
  • SSL sockets
  • programs

SSM and socat Port Forwarding Example

In my example I have an AWS EMR (Elastic Map Reduce) master node running a web dashboard for ganglia in a private VPC subnet.

I don’t want to add a bastion host / jump box or provide SSH access from the public net.

SSM would provide a nice way for me to connect a remote session, or port forward using IAM authentication and negating the need for any ingress security group rules, but only if I had the SSM agent available on this instance.

Seeing as though the EMR master node is not SSM agent enabled, and I can’t use SSM port forwarding directly to this instance, we could use an interim machine with SSM as a jump box.

Example Configuration

Here is how I configured port forwarding in my use case to access ganglia on a private instance EMR node.

  • The EC2 instance with SSM agent must have an IAM policy attached that allows the relevant ssm access. The blog post linked above has instructions. In a nutshell though, most standard Amazon AMIs include the ssm agent. Your EC2 instance profile should include the required actions too. The AmazonSSMManagedInstanceCore managed policy includes these.
  • Install socat on the SSM agent enabled interim machine the private subnet. For this I connected an SSM session to get shell access and ran sudo yum install -y socat
  • Now I needed to open a source channel for the SSM port forward aws cli command to connect, and connect that source to the destination of the EMR master node running ganglia.
socat TCP4-LISTEN:8080,fork,reuseaddr TCP4:10.0.4.149:80

The command listens on port 8080, and forwards TCP to the EMR node, 10.0.4.149 on port 80. Importantly, the command uses fork and reuseaddr to allow multiple connections.

  • Next is to use the AWS CLI ssm start-session command to start a port forwarding session to the interim instance with the SSM agent running. Grab the Instance ID for the EC2 machine and:
aws ssm start-session --target {your-instance-id-here} --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["8080"],"localPortnumber":["8089"]}'
ssm and socat port forwarding in action

If you setup socat correctly to listen on port 8080, then the connection should be opened and accepted.

Now you can simply open a web browser locally and direct it to http://localhost:8089/ganglia to access ganglia on the remote EMR master node.

Accessing EMR cluster memory stats via the remote port forwarded session.

Closing

AWS SSM is a useful tool to get access to instances in a secure, audited fashion without needing to open up risky SSH access or other remote ports to the public internet.

When constrained and needing a jump across to an instance without the SSM agent you can leverage tools to help. Socat is one such tool that can facilitate this within the private network.

SpotiPod – Spotify Streaming Device from an iPod Classic

spotipod classic

I recently came across the sPot: Spotify in a 4th-gen iPod (2004) project on hackaday.io by Guy Dupont. This post is my go at building Guy’s project from the ground up – the SpotiPod.

Here is my version, up and running, albeit with some hardware leaking out the side for now…

The main components required in terms of hardware are:

  • iPod Classic (4th gen) – at least the device case and clickwheel components
  • A Raspberry Pi Zero W
  • 2″ LCD display (see note further on about an alternative)
  • 3.7V 1000mah LiPo battery
  • Adafruit boost module (to boost 3.7V to 5V required by the Pi, LCD, etc…
  • Adafruit USB charge controller module

Once it’s all connected and configured, you’ll be able to load up your own Spotify playlists and libraries, browse them, and play them all from your iPod Classic device over a bluetooth speaker or remote system.

The SpotiPod build almost complete.
Some wire reduction still required before I can completely close it up.

SpotiPod High-level Build

I won’t be getting into the low-level parts of the build, so for specific details I would recommend viewing Guy’s project log and branching off the posts there.

Initial Raspberry Pi Zero W setup with 2″ LCD display

I started off loading Raspbian Lite OS onto an 8GB microSD card and booting up a barebones Pi Zero W. The W version is important because it has WiFi and Bluetooth on the board.

My first task was to configure SSH to start automatically on boot. Useful when I have no display to start with. I also added some of the required software components to begin testing and messing around with, including redis server, compiling the click c++ application that Guy wrote to interface with the iPod’s clickwheel, and setting up some of the X11 components for a basic UI.

Soldering up the connection for the 2″ LCD display was my first hardware connection task. My soldering iron might come out once in a year, so I’m a bit of a novice in this area, but I managed the task on my first go.

Connecting the LCD to Raspberry Pi Zero via Composite Connector
Connecting the LCD to Raspberry Pi Zero via Composite Connector
testing the LCD
First LCD test successful

I posted more details here on the 2″ LCD connection and configuration in software.

Connecting the Charge Controller and Voltage Boost Modules

Next up I focused on getting the power delivery working. The goal was to be able to charge the LiPo battery via USB and have the circuit supply 5V to all SpotiPod components where required.

I familiarised myself with the Adafruit module documentation and pinout diagrams before connecting the circuit up.

  • Adafruit Powerboost 1000 module
  • Adafruit Charge Controller

Here is a useful diagram to follow, posted by Kakoub on the hackaday.io project page, re-hosted here in case the imgur upload ever disappears:

Click for a larger, clearer version

So after getting power delivery and connections soldered in place, and precariously placing all the components away from eachother to prevent shorts, here is where I was at:

SpotiPod Powerboost and Charge modules connected
Powerboost and Charge modules connected

Testing the Click wheel and Software

The click wheel connectivity is made easier with an 8 Pin FPC cable breakout board. I hooked this up next, soldering the 4 wires required for power and data.

The click wheel ribbon connector could then snap into the breakout board ribbon connector. This is by far the most delicate part of the build in my opinion. The ribbon cable is super thin and delicate.

Connecting the 8 pin FPC breakout board. My soldering skills slowly coming back with a bit of practice…

Testing the interface was a case of compiling the clickwheel program with gcc over an SSH connection then executing it with everything connected.

If wired up correctly, the program will output touch and click data to stdout.

iPod Classic click wheel testing over SSH

Gutting the old iPod Classic

I managed to snag an old iPod Classic 4th gen off eBay for about £20. It wasn’t working, but had the two bits I needed – the chassis, and the clickwheel.

Breaking out my trusty iFixit essentials toolkit, I set about opening it up to remove the unecessary components.

The method I found to work was to pry it open on the left and right edges using the pry tool. Once you can get into one of the edges, slide the tool around, and things get easier.

Insulating the case and components

With most of the electronics connected, I began insulating things with polyimide tape. Most importantly, the metal iPod case. I put down about three layers of tape and tried to cover all parts as best I could.

Insulating the iPod classic shell for the SpotiPod build.

Installing into the iPod Classic Case

This is the tricky part. It’s quite difficult to squeeze all the SpotiPod hardware in.

I started out by strengthening all my soldering connections with a bit of hot glue.

adding hot glue to the soldered connections for the SpotiPod
Adding hot glue to the soldered connections

After a bit of arranging, squeezing, and coercing, everything fits… Mostly!

Things are still not perfect though. I need to reduce my wire lengths before I can get the case to fully close.

For now though, everything works and I have a fully functional SpotiPod!

Tips and Tricks

I’ve put together a list of things that might help if you do this yourself. These are bits that I recall getting snagged on:

  • Make sure you setup all the X11 and software dependencies correctly. Getting Openbox and the frontend application to launch on start up can be tricky and this is crucial. Pay attention to your /etc/X11/xinit/xinitrc and /etc/xdg/openbox/autostart configurations.
  • You don’t have to use the more expensive Adafruit composite LCD display. Ricardo’s build at RSFlightronics uses a much cheaper LCD and some creative approaches to get display output working.
  • Watch out for the click wheel ribbon orientation when you connect it to the breakout board!
  • Use thin and short length wires for connections where possible. Not too short though as it is useful to be able to open the device up and put the two halves side-by-side.
  • Make sure you have a Spotify Premium subscription. I can’t remember exactly, but I’m sure that creating your own app to get your client and secret keys, or some of the scopes required for the app will only work on Premium. (It might have even been spotify connect).
  • You’ll need to configure your own Spotify App using the Spotify Developer portal. Keep your client and secret keys safe to yourself. Remember to setup environment variables with these that the openbox session can access.
  • The frontend/UI application has a hardcoded reference to the Spotify Connect device as “Spotifypod”. Keep things simple by setting your raspotify configuation to use this name too, otherwise you need to update the code too.
  • If you’re struggling to get the software side working at first, it can really help to setup VNC while you debug things. This allows you to get a desktop environment on the Pi Zero and execute scripts or programs in an x session as openbox would.

Thanks again to Guy Dupont for his excellent SpotiPod project and idea. Putting this all together really makes for a fun and rewarding hardware/software hacking experience.

Tiny 2″ TFT Composite Video on Raspberry Pi Zero

This weekend I wanted to test composite video on the Raspberry Pi Zero. I had a Raspberry Pi Zero W and a NTSC/PAL (Television) TFT Display – 2.0″ Diagonal from Adafruit.

This display is ridiculously small. It’s quite something to boot up Raspbian with the PIXEL desktop environment on a Pi Zero with this little 2″ display.

Hardware Wire Up

The display only needs 4 connections.

  • Power (I’m using 5v from a Adafruit Power Boost 1000 basic module)
  • Ground (ground connected to pin 14 / ground on Pi Zero and negative on the boost module)
  • Positive (yellow) from TFT display board to TV on the Pi Zero
  • White from the TFT display board to the other pin, next to the TV pin.
connecting composite video on raspberry pi zero with the TV pin

Here’s what everything looked like after connecting the boost module (3.7v to 5v conversion), the battery charge controller, LiPo, and 2″ TFT display.

boost module, battery charge controller, LiPO, and TFT display all connected to the Raspberry Pi Zero

Powering up and Configuration

Power up and enjoy the tiny display outputting the boot up sequence.

The display is meant to run at 320×240, so after booting up I edited /etc/config.txt to set this up along with some overscan tweaks.

sudo nano /etc/config.txt

Set the following in the /boot/config.txt file:

# these overscan settings are what worked well for me
overscan_left=-26
overscan_right=-26
overscan_top=-16
overscan_bottom=-24

framebuffer_width=320
framebuffer_height=240

Although it is really expensive for what it is, the 2.0″ TFT display is great for small electronics projects that call for full display output. It’s simple and easy to connect, and doesn’t take up too much space either.

Cheap Minecraft Server in AWS with Docker and Traefik

minecraft-like figure on the grass

According to the Minecraft Realms plan pricing page, you can get a realms server at around £5.59 per month. You get some nice conveniences there but… I refuse to pay much at all when I can throw some infrastructure together myself in the cloud to create the ultimate cheap Minecraft server.

Considering my Docker instance running Traefik hosts another 3 or 4 of my personal services along with a Minecraft server, then this solution only costs me around £1.50 a month.

I chose to go with a single AWS EC2 instance that runs Docker. Minecraft runs in a container and sits alongside other personal websites and services that I host there too.

I use Traefik to route traffic coming in to this single host for various TCP ports as well as HTTP(s) on different hostnames. This essentially levels up the cost savings even further as I don’t need multiple EC2 instances (one for each service), and I don’t even need to pay for something like an application or network load balancer, as Traefik does this for me.

A Quick Review of Alternatives

There are other alternatives to consider if you’re looking for a cheap Minecraft server, so don’t take this as being the only option. Here is what I’ve used in the past before settling on my current solution:

  • Minecraft on a dedicated cloud VM. If you just want a dedicated Minecraft VM in the cloud, then DigitalOcean is a good, cheap option. You can also get fairly cheap instances Vultr.
  • Running Minecraft on my own personal Raspberry Pi Kubernetes Cluster. I was even able to expose it over the internet for friends to play on by leveraging a Pi device as a dedicated router. I then used port forwarding to get it working through my double NAT setup. The ARM container was a little slow as a server for more than 2 or 3 players on Raspberry Pi hardware though.
  • Minecraft Server on a home PC / Workstation, with port forwarding to allow other players to connect. This is not ideal, especially on Windows machines or systems that you don’t want to leave running 24/7 as you would for a dedicated server.
  • Various other Minecraft-as-a-service providers. These are decent options in some cases. However for me price and control are important, and I much prefer to self host in this case.

Cheap Minecraft Server in AWS EC2 with Traefik

I used my Cheap Traefik EC2 Docker Hosting solution as the base. You can read that article to get access to the CDK resources required to deploy it yourself.

The cost benefits to using this particular recipe are:

  • EC2 Graviton2 ARM based processor – slightly cheaper to run than Intel and AMD. The downside is more limited software choices. You need to make sure you use ARM compatible packages or Docker images.
  • Spot instance – this has massive savings over a normal lifecycle EC2 instance. The downside is that it can be terminated at any time with only a couple of minutes of notice. When using these you need to make sure you have good data persistence that is not local to the EC2 instance. I personally use a mounted EFS volume. It is re-attached to a new instance from the autoscaling group if the old instance is terminated.

If you don’t use the CDK solution I mentioned above, then alternatively deploy yourself an EC2 instance. Give it an elastic IP address, set up the Security Group ingress rules accordingly, and get shell access. First thing you’ll want to install is Docker, then you’re pretty much good to go.

Minecraft Docker Image

I found a great Minecraft Docker image that is well maintained and has the correct ARM image builds for use on Graviton2 hardware. Check out itzg/minecraft-server. There are other arch builds there that’ll run on just about any other platform.

Docker Compose Service

If you use docker-compose, then here is the simple service definition to get things running.

version: "3"

networks:
  web:
    external: true
  internal:
    external: false

services:
  mc:
    image: itzg/minecraft-server:2021.1.0-multiarch-latest
    environment:
      EULA: "TRUE"
      VERSION: "1.16.5"
      ENABLE_AUTOPAUSE: "TRUE"
      OVERRIDE_SERVER_PROPERTIES: "TRUE"
      MAX_TICK_TIME: "-1"
      TYPE: "BUKKIT"
    labels:
      - traefik.tcp.routers.mc.rule=HostSNI(`*`)
      - traefik.port=25565
    networks:
      - web
    volumes:
      - /data/mc:/data

The docker-compose definition will run a Docker container using the latest multiarch image (which will run on ARM devices). When starting, the container will prepare and run a Minecraft 1.16.5 server. It will also use Bukkit and enable auto pause. The game server does not tick over when there are no players connected.

Traefik Configuration

In the docker-compose definition above, you might have noticed the container labels. The labels prefixed with traefik are used to inform Traefik of how to route network traffic.

the cheap minecraft server uses a Traefik TCP router with HostSNI *
The TCP router using HostSNI on *

In our case, TCP connections are required on port 25565 and HostSNI is used to route those coming in for * (all hosts). The TCP connections on port 25565 go to Traefik, and based on this rule, directed to the Minecraft container.

There is one limitation to be aware of here, and that is that you can only use HostSNI with * for connections that do not use TLS. This is because Server Name Indication (SNI) is an extension of the TLS protocol.

I don’t believe Minecraft supports TLS in any case though. It just means that you won’t be able to have more than one Minecraft server container using the same port running on the single Docker host.

Finishing Off Configuration

Lastly, you might want to point a convenient Host record (A record) to your AWS EC2 Elastic IP address. For example: yourmcserver.example.com -> 1.2.3.4.

All being well, you should now be able to find and connect to your server.

minecraft server listing

WSL2 GUI X-Server Using VcXsrv

wsl2 gui desktop

I needed to set up a WSL2 GUI recently on my machine (WSL2 running uBuntu 20.04.1 LTS). I found a guide that runs through the process but found that a few tweaks needed to be made. Specifically, the communication to VcXsrv was being blocked by Windows Firewall.

There were also a couple of extra tweaks needed for audio passthrough using PulseAudio and setting a windowed resolution.

Setting up a WSL2 GUI X-Server in Windows

Start by installing xfce4 and goodies.

sudo apt install xfce4 xfce4-goodies

If you’re running Kali you should use:

sudo apt install kali-desktop-xfce

During the install you’ll be prompted about which display manager to use. This is up to you, though I personally chose lightdm.

Download this .zip package which contains VcXsrv and PulseAudio along with some configuration and a shortcut to launch.

Extract it to the root of your C:\ drive. You should end up with contents under C:\WSL VcXsrv.

WSL2 GUI vcxsrv package contents

Run the vcxsrv-64.1.20.8.1.installer.exe installer in this folder, choosing defaults for the install.

Once installed, you’ll want to enable High DPI scaling for VcXsrv in Windows.

  • Navigate to C:\Program Files\VcXsrv
  • Right-click xlaunch.exe and go to Compatibility
  • Click Change high DPI settings and choose Override high DPI scaling behavior. Ensure Application is in the dropdown.

Next, edit the startWSLVcXsrv.bat batch file and change the last line that reads ubuntu.exe run to one of:

  • ubuntu2004.exe run in the case you are using uBuntu 20.04 from the Microsoft Store for WSL
  • ubuntu1804.exe run if you are using uBuntu 18.04 from the Microsoft Store for WSL
  • ubuntu.exe run for when you are using standard uBuntu from the Microsoft Store for WSL
  • kali.exe run if you installed Kali-Linux from the Microsoft Store for WSL

Pin the WSL VcXsrv shortcut somewhere convenient like the taskbar.

Opening Windows Firewall for VcXsrv and PulseAudio

Next you need to allow inbound traffic to Windows for VcXsrv and PulseAudio.

Open Windows Defender Firewall with Advanced Security and add two new Inbound Rules as follows:

  • Type: Program
  • Program path: %ProgramFiles%\VcXsrv\vcxsrv.exe for VcXsrv and %SystemDrive%\WSL VcXsrv\pulseaudio-1.1\bin\pulseaudio.exe for PulseAudio
  • Allow the connection
  • Profile: Domain, Private
  • Name: vcxsrv or pulseaudio depending which rule you are adding

I personally added the following to ExtraParams under the XLaunch node of config.xlaunch. This sets windowed mode to 1920×1080 for monitor #1 on my machine.

-screen 0 1920x1080@1

Viewing your WSL2 GUI

With all of that setup out of the way, you should be able to simply launch VcXsrv from the pinned shortcut and everything should work.

Try it out and you should get Desktop up and running for your WSL2 environment.

WSL2 gui example with audio settings open

PulseAudio passthrough should also be available if you check your sound / volume settings. Try an audio test using alsa-utils:

sudo apt install alsa-utils
speaker-test

Kudos to this guide on reddit for most of the setup instructions. As mentioned before, I needed to configure my firewall and also added some tweaks for windowed mode.

Troubleshooting

If you find your VcXsrv Server display window is blank when launching, try the following:

  • Double-check your firewall rule is allowing inbound connections for vcxsrv.exe for the domain and private scopes.
  • With the black X-server / display window from VcXsrv still open, launch a WSL shell separately, and run the following to set your DISPLAY environment variable:
export DISPLAY=$(grep -m 1 nameserver /etc/resolv.conf | awk '{print $2}'):0

This takes the IP address of your host machine (conveniently used as a nameserver in your WSL Linux environment for DNS lookups) and sets it as the Display remote location (with :0 for the display number appended).

Now, try to launch a xfce4 session with:

xfce4-session

If all goes to plan, the session should target your machine where VcXsrv Server is running and your display window should come to life with your WSL environment desktop.