Custom Kubernetes Webhook Token Authentication with Github (a NodeJS implementation)

Introduction

Recently I was tasked with setting up a couple of new Kubernetes clusters for a team of developers to begin transitioning an older .NET application over to .NET Core 2.0. Part of my this work lead me down the route of trying out some different authentication strategies.

I ended on RBAC being a good solution for our needs allowing for nice role based permission flexibility, but still needed a way of handling authentication for users of the Kubernetes clusters. One of the options I looked into here was to use Kubernetes’ support for webhook token authentication.

Webhook token authentication allows a remote service to authenticate with the cluster, meaning we could hand off some of the work / admin overhead to another service that implements part of the solution already.

Testing Different Solutions

I found an interesting post about setting up Github with a custom webhook token authentication integration and tried that method out. It works quite nicely and some good benefits as discussed in the post linked before, but summarised below:

  • All developers on the team already have their own Github accounts.
  • Reduces admin overhead as users can generate their own personal tokens in their Github account and can manage (e.g. revoke/re-create) their own tokens.
  • Flexible as tokens can be used to access Kubernetes via kubectl or the Dashboard UI from different machines
  • An extra one I thought of – Github teams could potentially be used to group users / roles in Kubernetes too (based on team membership)

As mentioned before, I tried out this custom solution which was written in Go and was excited about the potential customisation we could get out of it if we wanted to expand on the solution (see my last bullet point above).

However, I was still keen to play around with Kubernetes’ Webhook Token Authentication a bit more and so decided to reimplement this solution in a language I am more familiar with. .NET Core would have been a good candidate, but I didn’t have a lot of time at hand and thought doing this in NodeJS would be quicker.

With that said, I rewrote the Github Webhook Token Authenticator service in NodeJS using a nice lightweight node alpine base image and set things up for Docker builds. Basically readying it for deployment into Kubernetes.

Implementing the Webhook Token Authenticator service in NodeJS

The Webhook Token Authentication Service simply implements a webhook to verify tokens passed into Kubernetes.

On the Kubernetes side you just need to deploy the DaemonSet with this authenticator docker image, run your API servers with RBAC enabled

Create a DaemonSet to run the NodeJS webhook service on all relevant master nodes in your cluster.

Here is the DaemonSet configuration already setup to point to the correct docker hub image.

Deploy it with:

kubectl create -f .\daemonset.yaml

Use the following configurations to start your API servers with:

authentication-token-webhook-config-file
authentication-token-webhook-cache-ttl

Update your cluster spec to add a fileAsset entry and also point to the authentication token webhook config file that will be put in place by the fileAsset using the kubeAPIServer config section.

You can get the fileAsset content in my Github repository here.

Here is how the kubeAPIServer and fileAssets sections should look once done. (I’m using kops to apply these configurations to my cluster in this example).

You can then set up a ClusterRole and ClusterRoleBinding along with usernames that match your users’ actual Github usernames to set up RBAC permissions. (Going forward it would be great to hook up the service to work with Github teams too.)

Here is an example ClusterRole that provides blanket admin access as a simple example.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: youradminsclusterrole
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
  - nonResourceURLs: ["*"]
    verbs: ["*"]

Hook up the ClusterRole with a ClusterRoleBinding like so (pointing the user parameter to the name of your github user account you’re binding to the role):

kubectl create clusterrolebinding yourgithubusernamehere-admin-binding --clusterrole=youradminsclusterrole --user=yourgithubusernamehere

Don’t forget to create yourself (in your Github account), a personal access token. Update your .kube config file to use this token as the password, or login to the Kubernetes Dashboard UI and select “Token” as the auth method and drop your token in there to sign in.

The auth nodes running in the daemonset across cluster API servers will handle the authentication via your newly configured webhook authentication method, go over to Github, check that the token belongs to the user in the ClusterRoleBinding (of the same github username) and then use RBAC to allow access to the resources specified in your ClusterRole that you bound that user to. Job done!

For more details on how to build the NodeJS Webhook Authentication Docker image and get it deployed into Kubernetes, or to pull down the code and take a look, feel free to check out the repository here.

Finding Vendor specific VIBs on ESXi hosts with PowerCLI

 

Example showing VIBs loaded on a host with a search of the vendor name "VMware"

 

The other day I was trying to find a list of Custom VIBs (VMware Installation Bundles) that were installed on an ESXi host. The reason was that I just wanted to verify that the VIB had actually installed correctly or not. I threw the query out on Twitter and of course @alanrenouf had a solution in next to no time.

So Alan’s solution is to use the Get-EsxCli cmdlet and specify the host name using the VMHost parameter. After that, he simply uses the code property “software” to gain access to the list of VIBs on the host. E.g.

$ESXCLI = Get-EsxCli -VMHost esxi-01.noobs.local
$ESXCLI.software.vib.list()

I have used esxcli on its own before but didn’t realise that PowerCLI had this cmdlet built in to interface with hosts in the same way that esxcli would. This is a great solution and means you can fetch so much more in this regard.

To filter things down a bit more and find the exact match for the Dell OMSA VIB I was looking out for, I used a where clause looking for a match for “dell” on the Vendor property:

$ESXCLI.software.vib.list() | Select AcceptanceLevel,ID,InstallDate,Name,ReleaseDate,Status,Vendor,Version | Where {$_.Vendor -match "dell"}

Thanks again to Alan Renouf for pointing out the use of Get-EsxCli and for providing an example!

 

Create a custom Welcome Message / DCUI Splash screen for your ESXi Hosts with esxcli

 

Messing around with ESXCLI the other day I came across the commands to get / set the welcome message for your ESXi host. By default you are greeted with the following familiar splash screen for your ESXi hosts:

 

Standard DCUI Splash Screen / Welcome Message

 

Well, with esxcli, it turns out you can very easily change this to your own custom welcome message / splash screen. I personally don’t see any practical gains by doing this, but just thought I would show the command to set it. Using esxcli, enter the following (of course the bit between quotes is up to you):

 

esxcli system welcomemsg set -m="My Custom Welcome Screen. Press F2 to Customize System/View Logs or F12 to Shutdown/Restart."

 

 

To see what the current custom message is set to simply use the same command, but this time use “get” –

 

esxcli system welcomemsg get

 

Finally, to set it back to the default (No welcome message), simply set your welcome message to nothing –

 

esxcli system welcomemsg set -m=

 

Here is an example of a custom welcome message for an ESXi host in my lab I tested this with:

 

 

How to create a PowerShell / PowerCLI profile & add Functions for later use

 

Ever wondered how to set yourself up a PowerShell or PowerCLI profile? Or how to go about saving useful Functions that you have created or picked up elsewhere to your profile for use in new sessions? Here I’ll detail the basics of PowerShell or PowerCLI session profiles and show you how to set one up, as well as how to save your first Function to this profile for use in future sessions.

 

Creating a PowerShell (or PowerCLI) profile is a great idea if you are considering spending any decent amount of time in the shell or executing scripts.  They allow you to customise your PowerShell / PowerCLI environment and apply useful functions or elements to every new PowerShell / PowerCLI session you start, without the need to reload these items yourself each time. A few examples of what you can add to your profile are functions, variables, cmdlets and snap-ins. This makes life so much easier when scripting or working on your latest bit of automation.

 

To begin with, you can see the currently set profile path in your session by typing $profile in the shell. If you would then like to edit your profile, try issuing the command “notepad $profile”. This will (if it exists) open your profile in notepad to edit. If it doesn’t exist, you’ll get an error when notepad tries to load indicating so. If this is the case, or you would like to create a new PowerShell profile for use in PowerShell or PowerCLI, use the last bit of script listed in the next section to create a profile and get started with customising your environment.

 

Show your profile path or open it up for editing in notepad:

$profile
notepad $profile

 

Create a new PowerShell profile for the current user if one does not already exist, then open it up for editing in notepad:

if (!(test-path $profile)) 
           {new-item -type file -path $profile -force}
notepad $profile

 

Once you have your profile created and ready for editing, try add a few useful Functions or variable declarations to get yourself going, then launch a new PowerShell or PowerCLI session to try them out. Here is a quick example of a function you can create to test your profile out in PowerCLI:

 

function VMInfo {
	param(
	[parameter(Mandatory = $true)]
    	[string[]]$VMName)
	Get-VM $VMName
}

 

Save your profile with the above function added, then start a new session of PowerCLI. Connect to your vCenter server using Connect-VIServer and then try type in “VMInfo aVMname” (where aVMname = the name of an actual VM in your environment). As you may have noticed, this function is just doing the same job that the Get-VM cmdlet already does for you, and will simply return the info about a VM specified. Here is a quick run down of the function above. The function is declared first of all (Name of the function). Then we open curly brackets to start the code. The param section then defines whether a parameter is mandatory (needed) or not, and what type of variable that parameter is (in our case we used a string), as well as giving the parameter a variable name ($VMName). The last bit does the actual work to run the Get-VM cmdlet on the parameter ($VMName) passed to the function. Here’s the output when running the new function against a VM called “SEAN-DC01” in my PowerCLI session:

 

 

You should now be able to see how useful this can become – you can quickly add new functions, variables, etc to your PowerShell profile for future use in new sessions. Try think of a few useful ones for cmdlets that you use often on a day to day basis and add these into your profile. You won’t regret it in the long run!