Bastion Hosts

Nov 13, 2014Robb Wagoner

Bastion host(s) are a useful and important component of a system management infrastructure. A bastion-host, in this context, is actually more properly, but more obscurely, called a jump server. In this post I will simply use the term bastion host. It is the most commonly used term for the system’s function: a server, which has undergone security hardening steps, that is the operational and administrative control point for systems and hosts in a datacenter (or AWS Region).

On the last point about AWS Regions: I will touch on some powerful capabilities of bastion hosts, AWS Security Groups, and cross AWS account-access that we use here at Panda Strike.

The principles of using bastion hosts apply to Linux/Unix and Windows-based platforms. For the purpose of this post, however, I will only be writing in the context of operating a Linux/Unix platform. Windows bastion hosts will not be discussed.

How It Works

There are two common infrastructure deployment patterns: first, a traditional firewall-based infrastructure, complete with isolated networks and a DMZ. I call this consolidated network-layer security. Or a cloud-based infrastructure where all hosts are, in some way, attached directly to the Internet: distributed network-layer security. Amazon Web Services VPC is a hybrid of the two, in that compute resources can be on isolated networks, but network-layer security is still distributed via Security Groups.

Your network-layer security architecture should allow access to administrative services, such as SSH, backend admin web-based user interfaces, and network infrastructure (firewalls, load balancers, switches, et cetera), from only hardened bastion hosts. In a traditional datacenter-based architecture, or AWS VPC, your bastion hosts might be inside your firewall perimeter and accessible with a two-factor VPN. Or, depending on your company’s security standards, it might be accessible directly from the Internet, and logically function as a VPN-like endpoint itself. In the case of being a VPN-like endpoint, Single Packet Authorization can render your bastion host (mostly) invisible to the Internet.

This invisibility is a super power any organization needs, if it’s going to be serious about security.

Hardening The Bastion Host With Single Packet Authorization (SPA)

Since the bastion host is key to infrastructure security, it must be hardened. The topic of systems hardening is complex and has many layers. At the network level, an ideal bastion host would be inaccessible (invisible) from an unsecured network unless you are authenticated and authorized to connect. To accomplish this, you alter packet filter rules to permit traffic only from a specific IP address (yours), once you have been authenticated via strong (mutually verified) signatures. This is called Single Packet Authorization with Firewall Knock Operator (FWKNOP).

When using FWKNOP with GnuPG (recommended), an SPA packet is signed, encrypted, and then sent to the bastion host, at a specific UDP port which, if deemed authenticated, triggers NetFilter (iptables) changes. Typically, NetFilter is changed on the bastion host to permit NEW SSH traffic from the same IP which sent the authenticated packet. The rule which permits this is active, by default, for only 30 seconds. Until then, however, the Bastion Host is completely unseen, at a network level, from the Internet. This is very secure. For more background, see this Firewall Knock Operator Tutorial. In addition, a 2007 article in Linux Journal describes Firewall Knock Operator.

Setting up Firewall Knock Operator is somewhat complex. When I first started using it, one of the trickiest parts was the two-way signing of GPG keys: the server signs each operator’s ASCII-armored public key and and each operator signs the server’s public key, in kind. Again, for more detail, please consult the FWKNOP Tutorial at the Firewall Knock Operator project website.

This gets you the security you need, but the process doesn’t scale well. For example, if you had 30 operators which needed access to your bastion hosts, you would have to import and sign 30 keys on each bastion host. Using python-gnupg for automation is one way to attack this problem, and we may blog in future about how to do that.

How to get it

Firewall Knock Operator Server

Popular Linux distributions which have Firewall Knock Operator 2.0 in their respective package repositories:

Firewall Knock Operator Client

Mac (OSX): Use Homebrew: brew install fwknop

Linux: Install the client from the package repository

How to set it up

To use the Firewall Knock Operator client, I recommend having a simple bash shell script (sshbh) which maps AWS Region (or data center) to a bastion host hostname & GPG key and then calls the FWKNOP client command with the necessary arguments, followed by a short wait period, and then open the ssh connection.

sshbh:

Implementing bastion hosts on AWS

Bastion hosts work best on AWS on a per-region basis, with Elastic IP. You’ll need to set up your security groups appropriately.

Per-Region bastion hosts

Maintain per Region-bastion hosts so that access to your instances can be expressed via security groups and not IP addresses. In addition, doing so means your network-layer security in EC2 is more self-documenting. A t1.micro (or t2.micro if in a VPC) should function well for lower concurrency use.

I don’t have benchmarks, or firm recommendations, besides my own experience, but I believe one should be able to have 3-5 operators connected at the same time without troubles, unless large file transfers are being done. Using per-region bastion hosts also allows you to use a simple configuration pattern for the ssh_config (5) on your workstation, and, if you’re ever in a situation where it appears your bastion host might be compromised, your exposure is limited to a single region.

Elastic IP

Assign each bastion host an Elastic IP (EIP). Use a DNS CNAME record to the EIP public DNS name (e.g. ec1-23-45-678-910.compute-1.amazonaws.com). By always using the public DNS for AWS resources, it allows AWS DNS servers to resolve to the private IP address when queries originate from within the same Region. This is good practice for all EC2 DNS management, not just bastion hosts, because you will not be subject to data transfer charges for traffic which traverses the private IP address in the same availability zone.

Security Group

For AWS, bastion hosts will need SSH access all your servers. Your bastion host(s) should be in a Security Group called bastionhosts, or in our case, we name it pandastrike-bastionhosts (I explain why below). I also recommend that you maintain a security group that all of your hosts are a member of. Logically enough, I recommend you simply name it all. You can now grant bastionhosts access to TCP/22 (SSH) on all (instances). Do not use the IP address of your bastion host(s).

The bastionhosts Security Group should permit the following:

If you followed the FWKNOP Tutorial, the NetFilter (iptables) default-drop firewall policy on the bastion host should be configured to block any attempt to connect, probe, or scan all ports including the two ports permitted in the Security Group.

This security group advice assumes you’re using one AWS account.

Alternate setups

If you maintain multiple AWS accounts for isolation of different functional environments like production, test, development, etc. (also recommended), you do not need a bastion host per-account. I recommend the following:

EC2 Classic

Your AWS production account should be where your bastion hosts are launched and operated from. In your non-production AWS accounts you will also maintain a security group call all. In the group you will permit access to tcp/22 from <production account id>/bastionhosts. This will appear in the AWS console <production account id>/sg-XXXXXXXX (bastionhosts). Your account id can be found for many Amazon Resource Names in your account. Viewing the details of an IAM User is a good choice. You are using IAM users, right!?

This approach makes works well for us, and is powerful for our customers. Our DevOps customers grant access to their EC2 hosts from our bastion hosts in the same Region. Security Group rules can only reference other Security Groups in the same Region. And, because we prefix our name to our bastion host Security Groups (pandastrike-bastionhosts), our customers can see that we have access, which makes the Security Group rule transparent and self-documenting. Being able to annotate Security Group rules has been on my feature wishlist since I started using EC2.

VPC

If you’re using VPC, you will need to use VPC Peering. Also, your bastion hosts should be in a public subnet dedicated to your bastion hosts, since cross-VPC Security Group access must use CIDR-defined networks.

Quoting the AWS documentation:

You cannot reference a security group from the peer VPC as a source or destination for ingress or egress rules in your security group. Instead, reference CIDR blocks of the peer VPC as the source or destination of your security group’s ingress or egress rules.

Summary

Network security is incredibly important, and has been incredibly important since people first began doing business on the Internet. We’ve been telling our clients this for years, and we’re very glad to say that nobody doubts us for a minute these days.

Unfortunately, this hasn’t always been the case. Sometimes, the ease of the Internet has made people a little overconfident. But in recent years, we’ve found out that the NSA is spying on just about everybody, that hackers can easily steal your nude selfies, and, in the case of Target, that a CEO can get fired if their company’s network security is bad enough. Today, everybody realizes that security matters.

I’ve presented some rather nuanced recommendations to keeping your infrastructure secure, especially if it’s AWS-based. Maintaining good network-layer security need not adversely affect your productivity. Automation and smart ssh configuration can make a lot of this very easy. We’ll try to blog more about that in future, but for now, I hope I’ve given you a good foundation in bastion hosts.