Simple Content Delivery Network (CDN) using Amazon AWS (S3 + CloudFront)

 

Content Delivery Networks

Having a content delivery network has many benefits for your users or clients. One of the most obvious reasons of having a CDN, is the ability to serve up content to your users from multiple (often the most optimal) locations.  Users access files that originate from one original source location, but the content is delivered by the closest location(s), often with the lowest latency and highest possible speed.

Using Amazon CloudFront, you can share dynamic, static, or even streamed content to users (including full websites), using Amazon’s global network of edge locations. This means that content can be served to users at the highest possible speeds, with the lowest possible latencies. In this blog post, I will cover the steps you need to take to deploy a basic CDN using Amazon AWS. For this purpose, we will leverage a combination of Amazon S3 + CloudFront.

 

Setting up Amazon S3

Amazon S3 (Amazon Simple Storage Service) is essentially Amazon’s “storage for the Internet”, and as explained above, CloudFront is a content delivery network service. As such, both products sit in Amazon’s “Storage & Content Delivery” stack.

 

  • To get started you will of course need an Amazon AWS account. Go to http://aws.amazon.com/ and register. You will need to provide credit card details, but most products have some sort of free tier that you can utilise for initial testing (usually free for up to 1 year, based on certain utilisation thresholds).
  • Once you are all signed up, you’ll need to navigate to the AWS Web Console. This is the central location you can use to manage all AWS services (among other options such as the AWS SDK and Command Line).
aws-console-example
The central, AWS Web Management Console
  • To start, we’ll need to define an origin location for our content. This is the location our original files are kept. For this purpose, we will use Amazon S3. It allows us easy access to files that we place in something Amazon call a “bucket”. I like to think of it as a folder, or container. You can have as many buckets as you wish, however each one’s name needs to be completely unique across Amazon S3. Click on “S3” under the “Storage & Content Delivery” heading of your AWS Console to get started.
  • From here, you will be greeted with a welcome page and some explanation of what S3 is. Simply click “Create Bucket” to get going.

create-bucket

 

  • Provide a unique bucket name, and specify a region to use. Regions have the benefit of allowing organisations to comply with storage regulation rules – for example, if you were storing client data that you were bound legally to keep within the UK, you would specify the Ireland region.

new-bucket

 

  • Your new bucket will appear in the S3 Management Console after being created. Simply click the name of the bucket to open it. For our simple CDN, we’ll just be serving up one single file – pretend this was a really large file that needed efficient distribution to many people – for example a large media file. At the top left, you’ll see an “Upload” button. Click this, and choose a file to upload as your test file. I will be using a simple image file. (By the way, Amazon have a service called “Amazon Import/Export”, which allows you to send really large amounts of data via post on portable media to Amazon for them to upload directly to your Amazon S3 or Glacier services).
  • Click “Start Upload” once you have chosen a file to test with.
  • After the file is finished uploading, it will appear in the console under your bucket name. (I called mine “image-for-distribution.png”).

example-file-in-bucket

 

  • Right-click the file, and choose the option “Make Public” for this test. This choice would be affected by the nature of the files you would want to deliver to users in your own configuration, but for this simple example, this is what I am choosing.
  • Right-click the file again, and choose “Properties“. Here you can get the direct, public link to your file and test access to it in your web browser. This is simple, direct access, and is not the access we are aiming for, as we will utilise our CDN with CloudFront to serve the file in our final configuration. This is just to test that the direct link is working.

aws-file-properties

 

Setting up CloudFront and your Distribution

  • Now that we know our basic file is being correctly served from Amazon S3, we’ll navigate to “CloudFront” from the main AWS Console (aws.amazon.com). A quick way to get there is by clicking the orange cube icon in the top left of your AWS page – wherever you are in the console, it’ll take you back to the main AWS console. From there just click “CloudFront“.
  • In CloudFront, we’ll want to create something called a “Distribution“. Click the “Create Distribution” button to get started.

create-distribution

 

  • Make sure you select “Download” type for the “delivery method” when asked on the next page, then click “Continue“.

cloudfront-delivery-method

 

  • We’ll now select various options for our CloudFront Distribution.
    • For “Origin Domain Name“, click the text box and you’ll see a populated list of Amazon S3 buckets. Your bucket you created earlier should feature here. Click it to select it.
    • The “Origin ID” should auto populate based on your S3 bucket name you chose.
    • If you wish to restrict users to only access your content via CloudFront URLs, and not direct by S3 URLs, then choose “Yes” for “Restrict Bucket Access“.
    • If you chose “Yes” for restricting bucket access, you’ll also need to create a “Comment” and “Grant Read Permissions” on the bucket for CloudFront’s access to the S3 bucket. Click “Yes, Update Bucket Policy” to have CloudFront get read access automatically to the S3 bucket.
    • Select “HTTP and HTTPS” for “Viewer Protocol Policy“.
    • You can customise the object caching properties if you wish, but for this example, just leave the “Default Cache Behavior Settings” on their defaults.
    • Now you can set your “Distribution Settings“. Choose “Use All Edge Locations (Best Performance)” for “Price Class“. This will ensure that all edge locations around the world are used to distribute your content in the fastest, most efficient way to your users. You could also restrict this to other groups of regions e.g. only the US and Europe for example – this would be a cheaper option, but not as efficient for all users globally.
    • Next, we can add an alternate CNAME for the distribution. This is highly recommended so that you can provide your own domain name formatted URLs to users, instead of a long, ugly default Amazon CloudFront URL. Enter something now, (for example I will use cdn.shogan.co.uk as I own the domain and can create this CNAME record myself in DNS). Once you are complete with this distribution setup, you should get the Distribution URL, and point a new CNAME record to the full URL that CloudFront assigns to your distribution.
    • Leave all other options at their defaults for now, and make sure that the last option “Distribution State” is “Enabled“, then click the “Create Distribution” button at the very bottom.

example-distribution-settings1 example-distribution-settings2

  • Your Distribution should now be created. Use the Navigation menu on the left side of the screen and click “Distribution” to see a list of your CloudFront Distributions.

cloudfront-distributions

 

  • At first the “Status” will show “InProgress“. After a few minutes this should change to “Deployed“.
  • In the mean time, look for your “Domain Name” that this Distribution has been assigned, and go and create a CNAME record pointing the CNAME you specified when creating this distribution, to the domain name. For example, you may have something like dxxxxxxxxxm.cloudfront.net. In my case, I specified a CNAME of cdn.shogan.co.uk, so I will create a CNAME record linking these together.

 

Testing

Once your CNAME record is created, type in your new CNAME record, followed by a forward slash, and then the name of the file you originally uploaded to your S3 bucket that is linked to by this CloudFront distribution. For example, my file was called “file-for-distribution.png” and my CNAME record I made is cdn.shogan.co.uk. So to utilise my CloudFront CDN, I would simply access the file as “cdn.shogan.co.uk/image-for-distribution.png”. If your DNS takes a while to apply/propagate, then you can simply use the CloudFront domain name assigned to your Distribution (for example dxxxxxxxxxxm.cloudfront.net/yourfilename.extension) to test out your distribution. Remember to ensure your distribution is in a deployed state before testing. You should now see your file served up in your web browser via your brand spanking new Amazon AWS powered CDN!

 

Conclusion

That concludes the basic setup of a Amazon S3 + CloudFront powered Content Delivery Network. I hope this was useful for some. In forthcoming blog posts I will delve into setting up custom logging and monitoring / alerting for your CDN. Please remember to like/share/tweet this post out to friends if you thought it was useful.

 

 

Setting up DNS SRV records for an Office 365 migration (on 123-reg)

I needed to setup some DNS records for an Office 365 migration earlier and was initially slightly confused translating the settings Microsoft supplied us to those needed as input on 123-reg’s Advanced DNS configuration. The MX, TXT and CNAME records were simple enough, but it was the SRV records that needed a bit of fiddling to get right.

As an example on the SRV records, MS give you something like this:

Type Service Protocol Port Weight Priority TTL Name Target
SRV _sip _tls 443 1 100 1 Hour thedomain.co.uk sipdir.online.lync.com
SRV _sipfederationtls _tcp 5061 1 100 1 Hour thedomain.co.uk sipfed.online.lync.com

123-reg give you this interface to enter SRV records yourself:

Screen Shot 2013-06-13 at 14.24.42

 

By looking at the examples you can start to understand how to translate the Service, Protocol, and Weight items that MS give you, into the 123-reg input boxes (which do not exist individually for Service, Protocol and Weight).

In the first SRV record example –

Hostname therefore becomes: _sip._tls (the Service + the Protocol with a dot (.) between)

TTL of course becomes: 3600 (1 hr)

Priority is 100

Destination (the most confusing one) becomes: 1 443 sipdir.online.lync.com. (note that it starts with Weight (1), then a space, then the port number (443), then the Target (sipdir.online.lync.com), followed by a dot (.)

That forms your complete SRV record. By entering these along with the other records you require, you should have a fully functional Office 365 setup on your custom domain name.

VMware vExpert for 2013

This is a bit of a delayed reaction to the vExpert 2013 announcements late last month, but I have been very busy and didn’t have time to finish posting the below on the day…

I woke up this morning to a flurry of tweets announcing the vExpert 2013 nominations. I was honoured to have received this title for a second year running now. Four of my colleagues at Xtravirt also received the title, as did 25 or so fellow London VMUG members. In total there were 500 or so people that received this title for 2013, out of 850 applicants. Congratulations to all who were nominated this year around!

Along with the flurry of activity on twitter came the inevitable blog posts. Two of my favourite so far are from @dawoo and @rimmergram.

Jane’s post rang true for me as one topic she covered was the perceived negativity to the announcements from some. I had also noticed a little bit of negativity from others on twitter around the announcements and the greater number of vExperts this year. Of course everyone is entitled to their own opinions. For me, I was just honoured to be able to keep the title for a second year around. This means I was able to keep at my quest for sharing knowledge in 2012 – my main platform being this blog.

Going slightly off topic, I did a look up on Google Analytics the other day, and Shogan.tech has been running for 5 years now! In that time it has received almost 300 000 page views.It started off as a really old version of WordPress running on a humble Dell Optiplex PC at one of my previous abodes, running on top of a VMware Server 2.0 VM (uBuntu Server with Apache, mySQL and PHP). After 6 months of PC issues and website outages, I decided to go the hosted route. It is a great feeling to know that many people have benefited from the content of my blog, and I hope to continue this trend throughout 2013.

For those interested, the official VMware vExpert 2013 announcement blog post and list of people awarded can be found here.

Update – where have I been?

It has been a nearly two months since I last posted blog updated in a regular fashion, which is kind of unusual for me. Unfortunately for the blog, I have had a lot going on, and blogging has had to take a hit. There is a good reason for that though, my wife and I recently purchased our first property, so we have been busy with the move, but also with completely renovating it.

We have done all of the renovation work ourselves and so far are quite proud of what we have managed to achieve!   Much more to do though, but hopefully there will be more blogging in the near future!

Here is a start to finish progress log of one room we have managed to renovate…

lounge-transformation-low