Quick and Easy Local NPM Registry With Verdaccio and Docker

container storage

Sometimes it can be useful to be able to npm publish libraries or projects you’re working on to a local npm registry for use in other development projects.

This post is a quick how-to showing how you can get up and running with a private, local npm registry using Verdaccio and docker compose.

Verdaccio claims it is a zero config required NPM registry, and that is pretty much correct. You can have it up and running in under 5 minutes. Here’s how:

Local NPM Registry Quick Start

Clone verdaccio docker-examples and then change directory into the docker-examples/docker-local-storage-volume directory.

git clone https://github.com/verdaccio/docker-examples.git
cd docker-examples/docker-local-storage-volume

This particular sample docker-compose configuration gives you a locally run verdaccio instance along with persistence via local volume mount.

From here you can be up and running by simply issuing the following docker-compose command:

docker-compose up -d

However if you do want to make a few tweaks to the configuration, simply load up the conf/config.yaml file in your editor.

I wanted to change the max_body_size to a higher value to allow for larger npm packages to be published locally, so I added:

max_body_size: 500mb

If you haven’t yet started the local docker container, start it up with docker-compose up.

Usage

Now all you need to do is configure your local npm settings to use verdaccio on http://localhost:4873. This is default host and port that verdaccio is configured to listen on when running in docker locally).

Then add an npm user for local development:

npm adduser --registry http://localhost:4873

To use your new registry at a project level, you can create a .npmrc file in your local projects with the following content:

@shogan:registry=http://localhost:4873

Of course replace the scope of @shogan with the package scope of your choosing.

To publish a module / package locally:

npm publish --registry http://localhost:4873

Other Examples

There are lots more verdaccio samples and configurations that you can use in the docker-examples repository. Take a look to find these, including Kubernetes resources to deploy if you prefer running there for a local development setup.

Also refer to the verdaccio configuration page for more examples and documentation on the possible config options.

AWS CodeBuild local with Docker

AWS have a handy post up that shows you how to get CodeBuild local by running it with Docker here.

Having a local CodeBuild environment available can be extremely useful. You can very quickly test your buildspec.yml files and build pipelines without having to go as far as push changes up to a remote repository or incurring AWS charges by running pipelines in the cloud.

I found a few extra useful bits and pieces whilst running a local CodeBuild setup myself and thought I would document them here, along with a summarised list of steps to get CodeBuild running locally yourself.

Get CodeBuild running locally

Start by cloning the CodeBuild Docker git repository.

git clone https://github.com/aws/aws-codebuild-docker-images.git

Now, locate the Dockerfile for the CodeBuild image you are interested in using. I wanted to use the ubuntu standard 3.0 image. i.e. ubuntu/standard/3.0/Dockerfile.

Edit the Dockerfile to remove the ENTRYPOINT directive at the end.

# Remove this -> ENTRYPOINT ["dockerd-entrypoint.sh"]

Now run a docker build in the relevant directory.

docker build -t aws/codebuild/standard:3.0 .

The image will take a while to build and once done will of course be available to run locally.

Now grab a copy of this codebuild_build.sh script and make it executable.

curl -O https://gist.githubusercontent.com/Shogan/05b38bce21941fd3a4eaf48a691e42af/raw/da96f71dc717eea8ba0b2ad6f97600ee93cc84e9/codebuild_build.sh
chmod +x ./codebuild_build.sh

Place the shell script in your local project directory (alongside your buildspec.yml file).

Now it’s as easy as running this shell script with a few parameters to get your build going locally. Just use the -i option to specify the local docker CodeBuild image you want to run.

./codebuild_build.sh -c -i aws/codebuild/standard:3.0 -a output

The following two options are the ones I found most useful:

  • -c – passes in AWS configuration and credentials from the local host. Super useful if your buildspec.yml needs access to your AWS resources (most likely it will).
  • -b – use a buildspec.yml file elsewhere. By default the script will look for buildspec.yml in the current directory. Override with this option.
  • -e – specify a file to use as environment variable mappings to pass in.

Testing it out

Here is a really simple buildspec.yml if you want to test this out quickly and don’t have your own handy. Save the below YAML as simple-buildspec.yml.

version: 0.2

phases:
  install:
    runtime-versions:
      java: openjdk11
    commands:
      - echo This is a test.
  pre_build:
    commands:
      - echo This is the pre_build step
  build:
    commands:
      - echo This is the build step
  post_build:
    commands:
      - bash -c "if [ /"$CODEBUILD_BUILD_SUCCEEDING/" == /"0/" ]; then exit 1; fi"
      - echo This is the post_build step
artifacts:
  files:
    - '**/*'
  base-directory: './'

Now just run:

./codebuild_build.sh -b simple-buildspec.yml -c -i aws/codebuild/standard:3.0 -a output /tmp

You should see the script start up the docker container from your local image and ‘CodeBuild’ will start executing your buildspec steps. If all goes well you’ll get an exit code of 0 at the end.

aws codebuild test run output from a local Docker container.

Good job!

This post contributes to my effort towards 100DaysToOffload.

Streamlining your Kubernetes development process with Draft (and Helm)

Draft is a tool built for developers who do their dev work against a Kubernetes environment (whether it be a live cluster of a Minikube instance).

It really helps speed up development time by helping out with the code -> build -> run -> test dev cycle. It does this by scaffolding out a Dockerfile and Helm Chart template pack customised for your app with a single command and then by building and deploying your application image to your Kubernetes environment with a second.

Setting up Draft and a basic .NET Core Web API project

First off, make sure you have already set up your kubectl configuration to be able to talk to your Kubernetes cluster, and have also setup and configured Helm.

Set the Draft binary up in a known system path on your machine after downloading it from the Draft Releases page.

Run draft init to initialise Draft. It’ll drop it’s configuration in a subdirectory of your user profile directory called .draft.

Create a new .NET Core 2.1 ASP.net project and select Web API as the type.

Open a shell and navigate over to the root project directory of your new .NET Core 2.1 app. E.g. cd solution\projectname

Run draft create to setup Draft with your new project. This is where the Draft magic happens. Essentially, Draft will:

  • Detect your application code language. (In this case csharp)
  • Create a Dockerfile for your app
  • Set up a Helm chart and necessary template structure to easily deploy your app into Kubernetes direct from your development machine

You should see output similar to this:

PS C:\git\draftdotnetcorewebapi\draftdotnetcorewebapi> draft create
--> Draft detected JSON (97.746232%)
--> Could not find a pack for JSON. Trying to find the next likely language match...
--> Draft detected XML (1.288026%)
--> Could not find a pack for XML. Trying to find the next likely language match...
--> Draft detected csharp (0.914658%)
--> Ready to sail

At this point you could run draft up and if you have a container registry setup for Draft on your machine already, it would build and push your Docker image and then deploy your app into Kubernetes. However, if you don’t yet have a container registry setup for Draft you’ll need to do that first.

draft config set registry docker.io/yourusernamehere

PS, just make sure your local development machine has credentials setup for your container registry. E.g. Docker Hub.

Run your app with Draft (and help from Helm)

Now run draft up

PS C:\git\draftdotnetcorewebapi\draftdotnetcorewebapi> draft up
Draft Up Started: 'draftdotnetcorewebapi': 01CH1KFSSJWDJJGYBEB3AZAB01
draftdotnetcorewebapi: Building Docker Image: SUCCESS ⚓  (45.0376s)
draftdotnetcorewebapi: Pushing Docker Image: SUCCESS ⚓  (10.0875s)
draftdotnetcorewebapi: Releasing Application: SUCCESS ⚓  (3.3175s)
Inspect the logs with `draft logs 01CH1KFSSJWDJJGYBEB3AZAB01`

Awesome. Draft built your application into a Docker image, pushed that image up to your container registry and then released your application using the Helm Chart it scaffolded for you when you initially ran draft create.

Take a look at Kubernetes. Your application is running.

kubectl get deployments

PS C:\git\draftdotnetcorewebapi\draftdotnetcorewebapi> kubectl get deployments
NAME                           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
draftdotnetcorewebapi-csharp   1         1         1            1           7m

Iterating on your application

So your app is up and running in Kubernetes, now what?

Let’s make some changes to the Helm chart to get it deploying using a LoadBalancer (or NodePort if you’re using Minikube). Let’s also add a new Api Controller called NamesController that simply returns a JSON array of static names with a GET request.

using Microsoft.AspNetCore.Mvc;

namespace draftdotnetcorewebapi.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class NamesController : ControllerBase
    {
        [HttpGet]
        public ActionResult<IEnumerable> Get()
        {
            return new string[] { "Wesley", "Jean-Luc", "Damar", "Guinan" };
        }
    }
}

Change your charts/csharp/values.yaml file to look like this (use NodePort if you’re trying this out with Minikube):

using Microsoft.AspNetCore.Mvc;
# Default values for c#.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
  pullPolicy: IfNotPresent
service:
  name: dotnetcore
  type: LoadBalancer
  externalPort: 8080
  internalPort: 80
resources:
  limits:
    cpu: 1
    memory: 256Mi
  requests:
    cpu: 250m
    memory: 256Mi
ingress:
  enabled: false

Run draft up again. Your app will get built and released again. This time you’ll have a LoadBalancer service exposed and your updated application with the new API endpoint will be available within seconds.

This time however Draft was clever enough to know that it didn’t need a new Helm release. Using Helm, it determined that an existing release was already in place and instead did a helm upgrade underneath the covers. Test it for yourself with a helm list

PS C:\git\draftdotnetcorewebapi\draftdotnetcorewebapi> helm list
NAME                            REVISION        UPDATED                         STATUS          CHART                           NAMESPACE
draftdotnetcorewebapi           2               Wed Jun 27 23:10:11 2018        DEPLOYED        csharp-v0.1.0                   default

Check the service’s External IP / URL and try it out by tacking on /api/names on the end to try out the new Names API endpoint.

PS C:\git\draftdotnetcorewebapi\draftdotnetcorewebapi>PS C:\git\draftdotnetcorewebapi\draftdotnetcorewebapi> kubectl get service draftdotnetcorewebapi-csharp -o wide
NAME                           TYPE           CLUSTER-IP     EXTERNAL-IP                                                               PORT(S)          AGE       SELECTOR
draftdotnetcorewebapi-csharp   LoadBalancer   100.66.92.87   aezzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.us-east-2.elb.amazonaws.com   8080:31381/TCP   32m       app=draftdotnetcorewebapi-csharp

Draft clean up

To take your app down and delete the Helm release, simply issue a draft delete on the command line.

PS C:\git\draftdotnetcorewebapi\draftdotnetcorewebapi> helm list
NAME                            REVISION        UPDATED                         STATUS          CHART                           NAMESPACE
draftdotnetcorewebapi           2               Wed Jun 27 23:10:11 2018        DEPLOYED        csharp-v0.1.0                   default

Check the service’s External IP / URL and try it out by tacking on /api/names on the end to try out the new Names API endpoint.

PS C:\git\draftdotnetcorewebapi\draftdotnetcorewebapi> draft delete
app 'draftdotnetcorewebapi' deleted

That’s all there is to it.

Draft really helps ease the monotony and pain of setting up a new project and getting it all working with Docker and Kuberenetes. It vastly improves your development cycle times too. Check it out and start using it to save time!

Deploying a simple linked container web app with Docker

This is a simple guide on how to deploy a multi-container ‘linked’ web app using Docker.

If you have not yet installed or set up a Docker host to run the containers on, here is my guide on setting up a basic uBuntu 16.04 Docker host VM.

The ‘web app’ we’ll be looking at how to deploy will consist of two basic components – a MySQL database for the back-end, and a simple PHP script for the ‘web front-end’ which simply connects to the MySQL container and displays some info from a database table.

simple-web-app-linked-diagram

For the MySQL container we’ll be using the official Docker repo ‘mysql-server’ image, and for our web front-end, we’ll be creating our own Docker image using a custom Dockerfile we’ll craft ourselves, based on an uBuntu 15.04 image.

This means we’ll be covering the following Docker basics:

  • Running docker containers
  • Linking docker containers (more secure than exposing ports directly)
  • Creating custom docker images using a Dockerfile
  • Building a custom image

Start off by creating a new directory in your home directory called ‘web01’ to create and store the Dockerfile we’ll using to build our custom web front-end image. Then create an emtpy file called ‘Dockerfile’ in this directory and edit it using your favourite text editor. I’m using nano for this.

 

 

This is what your new Dockerfile should look like:

 

The commands do the following:

  • FROM – tells docker build to base this image build on the ubuntu:15.04 image
  • RUN – strings a few apt-get commands together to install apache, php5, and a few other tools like curl. This is important, as every RUN command in a Dockerfile creates a new image layer, and we don’t want our image to contain too many layers.
  • The last RUN command grabs the content from a gist I created which is a basic PHP script, and places it in the /var/www/html directory in the container, then deletes the default index.html file that apache places there. This is the script that will connect to our MySQL container and display some basic info (our basic ‘web app’).
  • EXPOSE – exposes port 80 so we can map this to our Docker host and access the website outside of the container.
  • CMD – runs the apache2 service with PID 1 when the container starts.

Now you can build the Dockerfile and create your own custom image, which is what will be used to start the web container later.

Use the following build command to build the new image from your custom Dockerfile

docker build -t=”web01image” ~/web01/Dockerfile

Run ‘docker images’ after the build completes and you should see the new image listed:

docker-images

Next, you’ll run a new container using the official mysql-server image from the Docker repository. You won’t yet have this image locally, but the command will automatically download the image for you.

docker run –name db01 -e MYSQL_ROOT_PASSWORD=MyRootPassword -d mysql/mysql-server:latest

Note that I’ve called my container ‘db01’ and given it a root password of ‘MyRootPassword’. The -e parameter specifies that an environment variable called MYSQL_ROOT_PASSWORD inside the container should be given the value of ‘MyRootPassword’. The MySQL container then uses this environment variable to setup the root user for MySQL when the container starts.

Now that the database container is up and running (verify by running ‘docker ps’ to check its running), you can deploy the custom web container using your image you created above. In this docker run command, you’ll also link  the web container to the db01 container you previously started up using the –link parameter. This is important to link the two containers.

The web container will be given environment variables with information telling it about the networking config of the DB container. These environment variables will then be access by the simple web PHP script to tell it where to find the database server, and what credentials to use to connect.

docker run –name=web01 –link=db01:mysql -d -p=80:80 web01image

Important: notice that in the –link parameter, the name of the database/MySQL container is specified. Make sure you use the exact name you gave your MySQL database container here – this ensures that the linking of the two containers is correct. The last ‘web01image’ bit specifies to base the container you are running off of the newly built ‘web01image’.

The -p parameter maps the exposed port 80 in the container to port 80 on the docker host, so you’ll be able to access the website by using http://dockerhost:80

Check that the new web container and previously created MySQL container are running by using the ‘docker ps’ command.

docker-ps-output

Out of interest, this is what the PHP script looks like (this is what is downloaded and placed on the web container as a RUN build step in the Dockerfile you created above):

You can see the environment variables that the PHP script grabs (top of the script) to establish the database connection from the docker container. These environment variables are what are created and populated by linking the web container to the db container using the –link parameter.

Lastly, you may want to create a sample database, table and some data for the simple ‘web app’ to display after it connects to the database container. Issue the following ‘docker exec’ command, which will add the sample database, create a sample table, and add some sample data.

Make sure you change the ‘MyRootPassword’ bit to whatever root MySQL password you chose when you ran the MySQL container above, and ensure you run exec against the name of the MySQL container you chose (I used db01). Keep the database name and the rest of the command intact, as the PHP script relies on these staying the same.

docker exec db01 mysql -u root -pMyRootPassword -e “create database testdb1; use testdb1; CREATE TABLE events (id INT NOT NULL PRIMARY KEY AUTO_INCREMENT, name VARCHAR(20), signup_date DATE); INSERT INTO events (id,name,signup_date) VALUES (NULL, ‘MySpecialEvent’, ‘2016-06-11’);”

Finally, browse to http://dockerhostnameorip and you should see the simple PHP script display some basic info, stating it was able to connect to the MySQL server and display the sample data in the database.

simple-php-web-app-display

Setting up a basic uBuntu 16.04 Docker host VM

I’ve used this process multiple times to create quick Docker host VMs running on VMware Workstation in my home lab. It is important to note that although I’m using VMware Workstation, the type 2 hypervisor you use here is fairly unimportant. You could just as well use VirtualBox, or Fusion for this purpose.

Download the latest uBuntu 16.04 LTS server ISO from: http://www.ubuntu.com/download/server (I believe 16.04 comes only in 64-bit, but make sure its 64-bit)

Create a new Virtual Machine for your Docker host using your type 2 hypervisor software (Workstation in my case).

Give the VM following hardware/spec:

  • OS – Linux/uBuntu 64bit
  • 1 or 2 vCPUs
  • 512 MB RAM
  • 9GB disk
  • 2 x vNICs (1st is set to the default NAT option and the 2nd should be set to Host-only)

Here is my VM’s setup:

vm-hardware-docker-host

Attach the uBuntu ISO and start the VM up.

Install a standard uBuntu OS using the text based installer, and just be sure to also install OpenSSH server when prompted for features to install. After the install completes, reboot, login with your user account you created during install, run ‘ifconfig’ to check the assigned IP address, and then use your favourite SSH client to connect to that IP. Using a PuTTy session will just make copy/pasting commands into your uBuntu VM easier.

Now you’ll install docker – the package includes both the docker server and client.

 

Run the commands above in sequence, and after the apt-get install docker-engine at the end, run ‘sudo service docker status’ to check that docker is running. You should see it listed as Active (running)

● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-07-06 21:20:39 BST; 2h 0min ago

Run a quick ‘docker info’ command to ensure that you get information back from Docker and that everything looks OK.

docker-info