OpenVPN on AWS via CloudFormation

Want to automatically deploy an OpenVPN EC2 instance, running on Amazon Linux 2, to AWS? One that auto-produces an OVPN configuration file that is compatible with OpenVPN Connect, on both Android and iOS? This article is a breakdown of the following CloudFormation template repo:

ScriptAutomate/openvpn-cfn: Deploy OpenVPN to AWS via CloudFormation and Amazon Linux 2

Feel free to fork it, star it, test it, and comment on it! Want an in-depth look at how it works? This post helps breakdown how I patched up a non-functional CloudFormation template.

NOTE: This worked much more like a learning tool, and I'd rather recommend something like AlgoVPN when it comes to deploying and managing your own VPN in AWS.

TL;DR

I gave an old template a major facelift, bringing about a new version of a template blogged about by Linux Academy. If you want more information about Linux Academy, and how I found their original CFN template, scroll to the very bottom of the post. Otherwise, here is a mini quick-start from the bash shell.

Quick-Start

Want to give this a test run? This is about what the repo README describes. It would be best to refer to the GitHub repository in the future, as it will be most up-to-date.

# First step is to download:
# git clone git@github.com:ScriptAutomate/openvpn-cfn.git
# cd openvpn-cfn

SSHKEYPAIR='labkeypair'
AWS_PROFILE='labcreds'

# Create SSH key pair, and save contents of SSH key
aws ec2 create-key-pair \
  --key-name $SSHKEYPAIR \
  --profile $AWS_PROFILE \
  --query 'KeyMaterial' \
  --output text > $SSHKEYPAIR.pem
chmod 400 $SSHKEYPAIR.pem # Prereq .pem file permissions required by ssh

# Deploy the CFN template
aws cloudformation deploy \
  --template-file cfn-openvpn.yaml \
  --stack-name openvpn-personal \
  --profile $AWS_PROFILE \
  --parameter-overrides SSHKeyName=$SSHKEYPAIR \
  --capabilities CAPABILITY_IAM

# If wanting to SSH into the OpenVPN instance
OPENVPNIP=`aws cloudformation describe-stacks \
  --stack-name openvpn-personal \
  --profile $AWS_PROFILE \
  --output text \
    | grep OpenVPNEIP \
    | sed s/^.*EIP// \
    | tr -d '[:blank:]'`
ssh -i $SSHKEYPAIR.pem ec2-user@$OPENVPNIP

The VPNClientsS3Bucket created by the stack will have a client OVPN file as client/openvpn_clientuser.ovpn

That's it. Want to deep dive into more details? Keep on reading.

Problems with The Original Template

First off, the original CloudFormation template from the Linux Academy blog no longer works. I wanted to get this up and working so that I could have a personal OpenVPN server that can be connected to by laptop or phone. Want to use that coffee shop wifi? This makes it happen.

The YAML produced by Linux Academy seemed like a good starting point.

Amazon Linux 2 vs. Amazon Linux: Why Upgrade?

Why would one want to migrate to Amazon Linux 2, from Amazon Linux? Here is a breakdown quoted straight from the Amazon Linux 2 FAQs:

Amazon Linux 2 offers long-term support until June 30, 2023.

Amazon Linux 2 is available as virtual machine images for on-premises development and testing.

Amazon Linux 2 provides the systemd service and systems manager as opposed to System V init system in Amazon Linux AMI.

Amazon Linux 2 comes with an updated Linux kernel, C library, compiler, and tools.

Amazon Linux 2 provides the ability to install additional software packages through the extras mechanism.

The bolded section is what makes life a bit difficult, at a configuration level, because migrating System V init based service configurations over to Systemd can be painful. I won't go into the details as to why, because entire Linux communities have gone to war on this to the point that Linux distributions have forked off. Amazon Linux, along with most other major Linux distributions, have migrated over to Systemd.

OpenVPN tutorials, scripts, and configurations across the internet almost exclusively target non-systemd setups.

To get an EC2 instance of Amazon Linux 2, we need to target the right AMIs. What's the best path? I know what path I didn't want to do.

Let's Not Hard-Code AMI IDs for Base, Default AMIs

I have always been annoyed with hard-coding AMI IDs.

A problem with past CFN templates, and still with many present ones, was needing to include hard-coded values for AMI IDs. This wasn't just the case for custom AMIs built internally, but also with Amazon-provided base AMIs.

Amazon knew this was a pain, and they released a solution in 2018 by allowing customers to easily query for the latest Amazon Linux AMI IDs using AWS Systems Manager Parameter Store.

NOTE: Hardcoding AMIs can have the added benefit in having logged what AMI(s) successfully worked with a CloudFormation template. It may also be a requirement in organizations to use a custom AMI that has base configurations and org-specific monitoring software installed. In the case of my approach, in using the parameter store, I just wanted to share something that was both easy to share and know would be using the latest AMI when playing with in a sandbox.

Setup Latest Amazon Linux 2 AMI IDs as Default in CFN

JSON

"LatestAmiId": {
  "Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
  "Default": "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
}

YAML

Parameters:
  # Parameter store makes finding latest AMI image id easy
  # Is region locked, by default. Wherever CFN is spun up,
  #  resulting image id is specific to source region!
  # Source: https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/
  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'

No more region-specific mappings from 2017 that are woefully out-of-date! Any RegionMap references in the template now just need to be replaced by !Ref LatestAmiId and we're good. The following section is completely removed:

Mappings:
  RegionMap:
    us-east-1:
      "AMAZONLINUXAMI" : "ami-8c1be5f6" # Amazon Linux AMI 2017.09
    us-east-2:
      "AMAZONLINUXAMI" : "ami-c5062ba0" # Amazon Linux AMI 2017.09
    us-west-1:
      "AMAZONLINUXAMI" : "ami-02eada62" # Amazon Linux AMI 2017.09
    us-west-2:
      "AMAZONLINUXAMI" : "ami-e689729e" # Amazon Linux AMI 2017.09
    ca-central-1:
      "AMAZONLINUXAMI" : "ami-fd55ec99" # Amazon Linux AMI 2017.09
    ...
    ...
    ...

Now it is future-proofed for brand-new deployments. I even included comments about what's going on, which is an extra bonus of YAML vs. JSON.

Updating OpenVPN for Amazon Linux 2

We need to update the most painful part: the UserData and AWS::CloudFormation::Init sections.

This resulted in a ton of irritation of the kind that emphasizes the value in using a configuration management toolset like Ansible, Chef, SaltStack, or Puppet.

When you rely merely on UserData and cfn-init execution of shell commands, the portability of the configuration can have room for improvement. Yes, the code now moves along with the CFN template so it is portable in that sense. Though, it is permanently coupled to the template and has to be picked apart to properly decouple for separate versioning, updating, etc.

Of course, there is a cost (of both time and complexity) when going with a configuration management tool solution so I won't be extending this template to use one. Maybe in the future?

Extra Packages for Enterprise Linux (EPEL), OpenVPN, and Easy-RSA

OpenVPN isn't available in the default yum repositories. This is also an issue with easy-rsa, another requirement for setup.

Before

# Install the latest version of openvpn via the yum package manager
# Install easy-rsa via the EPEL repo
# Make a copy of the installed files to /opt/easy-rsa as our working directory
install_software:
  packages:
    yum:
      openvpn: []
  commands:
    01_install_software_install_easyrsa:
      command: "yum install easy-rsa -y --enablerepo=epel"
    02_install_software_copy_easyrsa:
      command: "cp -R /usr/share/easy-rsa/2.0 /opt/easy-rsa"

This doesn't work because, again, openvpn isn't in the default repositories. Not only that, but the --enablerepo=epel will not work because EPEL isn't installed on Amazon Linux 2 as a repository to enable. This means the easy-rsa install will fail, too.

How do we resolve these? We need to enable access to that Extra Packages for Enterprise Linux (EPEL) repo.

After

New OpenVPNVersion Parameter

Parameters:
  OpenVPNVersion:
    Type: "String"
    Default: "2.4.7"

Updated EC2OpenVPNInstance UserData

# Install and auto-enable EPEL repository
# Update existing packages
# Run CFN configSets
# Source: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-enable-epel/
UserData:
    "Fn::Base64":
      !Sub |
        #!/bin/bash
        yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
        yum update -y
        /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource EC2OpenVPNInstance --configsets myCfnConfigSet --region ${AWS::Region}
        /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource EC2OpenVPNInstance --region ${AWS::Region}

Updated install_software `AWS::CloudFormation::Init` configSet

install_software:
  packages:
    yum:
      openvpn: [ !Ref OpenVPNVersion ]
      easy-rsa: []
      yum-plugin-versionlock: []
  commands:
    01_install_software_copy_easyrsa:
      command: "cp -RL /usr/share/easy-rsa/3 /opt/easy-rsa"
    # Prevent future yum updates from updating openvpn
    02_lock_openvpn_version:
      command: "yum versionlock openvpn"

We added a little extra by also installing yum-plugin-versionlock, so that we can control what version of OpenVPN is installed. In this case, we added an additional parameter to the template called OpenVPNVersion.

Easy-RSA Changes

There is another change here: easy-rsa had a major version update when it entered into v3.x territory from v2.x. All the helper scripts have drastically changed, from names to arguments.

This meant a heavy facelift for many of the configs in myCfnConfigSet:

Metadata:
  AWS::CloudFormation::Init:
    # Our cfn-init config set rules, divided into logical sections to make reading it easier, hopefully :)
    configSets:
      myCfnConfigSet:
        - "configure_cfn"
        - "install_software"
        - "generate_easyrsa_vars"
        - "generate_secrets"
        - "generate_client"
        - "configure_server"
        - "upload_files"

You can see the differences by comparing the original template and the updated template on GitHub:

Old Template: Was originally a gist by zugdud, but has since been deleted
- They do have this repo, where the YAML eventually ended up: zugdud/openvpn_aws_cloudformation
New Template

The commands all had to be updated for new file paths associated with easy-rsa, along with a huge difference in the commands used to produce the same result of v2.x commands.

If you've been wondering how to navigate an unattended/silent configuration of openvpn with easy-rsa, the commands featured in the template may be a great place to start, with the commands run as follows:

cd /opt/easy-rsa
/opt/easy-rsa/easyrsa init-pki
/opt/easy-rsa/easyrsa --batch build-ca nopass
/opt/easy-rsa/easyrsa gen-dh
/opt/easy-rsa/easyrsa build-server-full server nopass
/opt/easy-rsa/easyrsa gen-crl
openvpn --genkey --secret pki/statictlssecret.key

One Last OpenVPN Fix for Mobile Clients

Though I was able to connect to the OpenVPN server from a desktop client without any issues, I ran into this error on OpenVPN Connect (Android):

There was an error attempting to connect to the selected server.
Error message: mbed TLS: SSL read error: X509 - Certificate verification failed, e.g. CRL, CA or signature check failed.

The OpenVPN Connect Android FAQ had a response for this, which was of zero help:

This is an error that tells you that the certificate could not be verified properly. This can occur for example if you are using an MD5 signed certificate. With such a type of certificate, the security level is so low, that the authenticity of the certificate simply cannot by any reasonable means be assured. In other words, it could very well be a fake certificate. The solution is to use a certificate not signed with MD5, but with SHA256 or better. You can find more information in the MD5 signature algorithm support section.

I had verified that my algo was of a higher level:

sudo openssl x509 -in /opt/easy-rsa/pki/ca.crt -noout -text | grep "Signature Algorithm"

# Output:
# Signature Algorithm: ecdsa-with-SHA256

NOTE: I also went with the ec algorithm to see if that was the issue, due to the default being rsa. This is of reading through where someone else had struggled with Android and iOS connects. Supposedly that helped solve their issue, but that made zero difference for me. I have since found out this wasn't the issue, and rsa should work fine. I have added a parameter to the template for either algorithm to be used.

I was banging my head against the wall until I came across someone on a forum with a similar problem from a year ago: OVPN Profile Works on Windows but not on Android

In it, a person says ns-cert-type is getting DEPRECATED as a config setting! It turns out ns-cert-type server existed in my client OVPN config and was breaking when attempting to connect from Android, but my other non-mobile clients didn't seem to care (this will change in the future when OpenVPN releases hit v2.5.x). I changed the following in the server.conf:

ns-cert-type server

To:

remote-cert-tls server

And now it works without fail from the Android client!

OpenVPN App Screenshot

OpenVPN App Screenshot Active

Replacing the Node.js 6.10 CustomResource

By default, CloudFormation stacks cannot delete an S3 bucket they created if there is any content within them.

The original template used a Node.js CustomResource in CloudFormation to help with the cleanup during a teardown. But it used the nodejs6.10, which is long gone in the AWS Lambda runtime world, and nodejs8.10 is approaching EOL.

I'm more experienced in Python, and I had seen a CFN template on GitHub that used the python3.7 runtime to achieve the same goal of deleting an S3 bucket even if it isn't empty.

The Future

What other problems still exist here? I'd like to approach all of the following:

Certificate Management: Certificates are only found locally on the EC2 instance and with a single client OVPN dumped to an S3 bucket. This means I can't simply destroy/redeploy without starting over with certificates since the client certs wouldn't work with the a new root CA setup
- What about issuing new certificates? Can we create a simplified way of doing this?
- What about revoking certificates?
Automated Patching: Update by way of replacing the instance with one running on the latest AMI. This should be achieved with an AWS Lambda monitoring for newer AMIs. Though, Amazon Linux AMIs don't often see updates so yum-cron should be setup until certificate management can be remedied
The existing AWS::IAM::Policy permissions could be more fine-grained

I think it would be interesting to approach the certificate management and updating problems with AWS Lambda and AWS Secrets Manager. If on first deployment there aren't particular certificates stored in the secrets manager, it could create them. If they already exist, it could pull the existing certificates down instead of generating new secrets from scratch. This would mean a complete ability to redeploy the instance as part of an updating process.

I may use this project as a way of learning the AWS CDK by migrating it over in the future.

Linux Academy is an excellent website for learning about Linux, containers, different cloud providers, configuration management tools, DevOps methodologies, certification prep, and more.

I am not associated with Linux Academy, but my CloudFormation template is a modified version of one developed in the following blog posts by Linux Academy: