Private connectivity to the public Cloud

So you want to move an internally facing only workload to the public cloud?  But you don’t want to run your connectivity over the Internet for a number of reasons.  Those reasons could be anything from privacy, security, control or cost factors.  Or maybe you want to try and access public services via a private connection.

Most of the major public Cloud providers offer some sort of dedicated connectivity options, in this I’m going to outline the options for Amazon’s AWS and Microsoft’s Azure which are easily the most popular choices.

Disclosure note:  My current place of employment does sell connectivity to multiple cloud providers, this article only addresses this connectivity from a generic point of view and doesn’t touch on provider specific details.  There are many providers of this type of connectivity to choose from.

Before diving into the options there are probably a few things that need to be made clear when deciding to go down this path:

  • If you want a secure connection, it is up to you to encrypt your traffic – neither of the options available from AWS or Azure encrypt your traffic.  So you can either encrypt traffic at the application level such as using HTTPS or alternatively over an encrypted VPN on top of the connection that you push all of your traffic down.
  • If you’re already accessing your Cloud environment over the internet by using a VPN already, you might wonder what is the point of private connectivity?  Well a lot of the time the economics (aka cost) if you’re pushing any meaningful amount of traffic will make it absolutely worthwhile.
  • To elaborate on the above point, Cloud providers charge a lot more for ingress/egress traffic that arrives in their Cloud via the Internet than they do via private connectivity connections
  • Currently there is no native IPv6 support available – you could technically run an overlay network – but that would be completely up to you to configure and support (basically you don’t want to do this).

Amazon AWS

First cab off the rank is Amazon’s AWS cloud, by far the largest cloud provider by a very large margin currently.  For those of us in Australia, there is one region in Sydney that is split into multiple availability zones (3 as of writing).  The AWS naming for their cloud connectivity product is Direct Connect, with this being available via two methods of connectivity – via a layer 3 provider over shared physical infrastructure or via a physically dedicated layer 2 connection.

Unless you have a really large bandwidth requirement (i.e. more than 500Mbit) for your Cloud connectivity, then it is generally most economical to purchase access through a layer 3 provider – unless you happen to be co-located within the same datacentre as one of the AWS connectivity points.  Now there are some caveats with how AWS do things and some limitations to be aware of:

  • Direct Connect connects to a single VPC – the VPC being a virtual container for your computing resources such as EC2 instances etc.
    • Now if you have a complex environment with multiple VPCs that you want accessible via Direct Connect one method of achieving this without purchasing multiple Direct Connects is to create a “transport” VPC that you don’t run services in and then setup VPC peering connections with your other VPCs and appropriate routing to direct traffic around.
  • Direct Connect establishes a BGP session to propagate routes – there is a maximum limit of 100 routes that will be accepted from your network into this session, if you exceed it the BGP session will drop and there goes your connectivity.  So if you have lots of routes, you should try consolidating them in some manner – layer 3 providers will often provide you options for this, ranging from just sending the default route ( or other things such as sending the RFC1918 address space consolidated into a few routes.
  • Redundancy?  Well your AWS VPC is available across availability zones (AZs), but you’d better hope that your layer 3 provider has redundancy through being connected at more than one of the connectivity locations for the specific region – so that they don’t go with an AZ failure (presuming its a whole datacentre failure).
  • As mentioned above, if you want more than 500Mbit of bandwidth you will need to stop using layer 3 provider and have a dedicated layer 2 physical connection setup, this is of course quite costly – this of course unless you pay for two connections will not have any redundancy in respect of AZs like a layer 3 provider connection typically would.
  • You will only be able to access privately addressable services over Direct Connect, so that means services that you spin up such as EC2 instances, with services on public IP address space will still go via the Internet
    • There is a caveat to this, whereby you can publish endpoints for certain services into your VPC (e.g. S3) so that traffic for these endpoints is presented privately into your VPC and doesn’t get classified as Internet traffic (also important from a cost point of view regardless of having Direct Connect).

Microsoft Azure

Now Microsoft do things a little bit different, whilst providing more options they do also have some added complexity with some of those options.  Once again talking with respect to Australia, Microsoft have two Azure separate locations locally, with those being Australia South East (Sydney) and Australia South (Melbourne).  The Azure connectivity option is called ExpressRoute and differs quite a lot from the AWS Direct Connect option.

For starters, ExpressRoute has multiple peering options – effectively with this meaning that certain services are accessible via different peering mechanisms.  These peering options are as follows:

  • Private peering:  Used if you want to access privately spun up resources within your Azure environment
  • Public peering:  Used for accessing services available via public IP addresses (except Office365!)
  • Microsoft peering:  Used for accessing the Office365 resources (there is a massive caveat to this one though, which I’ll explain later)

Now this is where things get a little bit different as well, with the private peering option you choose which VNets (virtual networks) you want to attach to the ExpressRoute service – once you have done this the route for that VNet will be advertised back into your network via BGP.  Do note though that the bandwidth you purchase is shared between the 3 peering types – you can of course purchase multiple ExpressRoute services if you want.

Now as above, I’ll go into some of the limitations related to private peering:

  • The default limit is 1000 routes from your network advertised into Azure (yes, 10 times that of AWS)
  • You can only connect 10 VNets to an ExpressRoute service
  • VNets connected to the same ExpressRoute service can inadvertently route traffic between them WITHOUT setting up VNet peering which is the normal method of connecting VNets – this is a by product of the fact that your private peering session is a VRF on Microsoft’s Enterprise Edge (MSEE) routers, so when the traffic makes it up onto the physical kit prior to being passed over to your layer 3 provider the traffic can hairpin back into another VNet by being routed by the MSEE.  This is an important thing to know from a security/firewalling point of view, and I’m not sure it is well documented that this happens!

The thing is that its not that simple, as Microsoft also have some additional options on top of this you can pick too:

  • Premium add-on:  Now this is a bit weird, because it enables a bunch of features that I’d consider unrelated for the most part:
    • This increases the number of routes you can push into Azure from 1000 (default) to 10000
    • This allows inter-region traffic flow, e.g. if you had VNets and services in Singapore for instance, you can access these services via ExpressRoute also if you have premium and Microsoft will transit that traffic over their network for you.
    • This add-on is also required if you want to enable Microsoft peering.
  • Unmetered data transfer:  Instead of paying more Mb of data transfer (still very low, especially in comparison to Internet traffic), you can opt for unmetered if you’re doing very high volumes of traffic.

Now onto public peering, this one is quite simple, the following services are accessible over it:

  • Azure storage, SQL databases, web sites and public VIPs that you assign to Azure resources yourself.

But here is the difficult part, to access these services you need to be coming from public IP address space (i.e. not RFC1918 address ranges) to access these services – which means you will need to SNAT traffic from devices on RFC1918 address space to public IP address space prior to it arriving in Microsoft’s network.  This public IP address space then should also NOT be advertised to the actual Internet, otherwise you will end up with asymmetric routing.

Now onto Microsoft peering, this one is not simple at all, it is very similar in nature to public peering above, except way more messed up to setup.  Now as I mentioned above, there is a massive caveat to this one and its as follows:

  • Not all services required to access Office365 can be accessed via Microsoft peering.  Which means that you still need Internet access in combination with Microsoft peering.

These services not accessible via Microsoft peering include (but aren’t limited to) DNS services (i.e. resolving names), the CDN loadable resources (e.g. images etc), Office365 downloads and a handful of other things.  So if you had this grand plan to have a network without Internet access and still use Office365, forget about it, it isn’t possible.

You will also need to SNAT traffic in exactly the same manner as you do with public peering.  The other complication that you can throw into this is if you’re trying to do some sort of hybrid environment, such as having part on-premise Exchange and part on Office365, or other services hybrid such as Sharepoint and a few others such as ADFS with AAD.  This requires connectivity BACK from Microsoft so that this will ingress to your network over ExpressRoute for these resources to work.  Basically in summary, just don’t do Microsoft peering unless you know what you’re doing and have a damn good reason for doing it.

To support that idea of only doing Microsoft peering if you know what you’re doing, Microsoft currently only allow it to be provisioned through a manual approval process whereby you need to submit a detailed plan and network design.  This doesn’t apply to private or public peering types, just to Microsoft peering.

So there you have it.

If you’re keen to learn more, I’d suggest hitting up the following pages from each of the respective Cloud provider’s web sites as a starting point:



This entry was posted in Blog posts and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s