Understanding the basics of SSH automated access management in an enterprise, focusing on the management of SSH access tokens and some recommendations from the National Institute of Standards and Technology
Hosts must be able to access other hosts in an automated fashion, often with very high privileges, for a variety of reasons, including file transfers, disaster recovery, privileged access management, software and patch management, and dynamic cloud provisioning. This is often accomplished using the Secure Shell (SSH) protocol. The SSH protocol supports several mechanisms for authentication, with public key authentication being recommended for automated access with SSH. Management of automated access requires proper provisioning, termination, and monitoring processes, just as interactive access by normal users does. However, the security of SSH-based automated access has been largely ignored to date.
Controlling access to information systems is critical for information security. Access controls exist on many levels and using many technologies. The levels include physical restrictions on access to hardware; logical controls for accessing network interfaces, hardware management ports on servers, virtualization hypervisors, operating system (OS) user accounts, and information through database systems; and logical controls implemented by applications.
Security breaches
Information security (confidentiality, integrity, and availability) is compromised if controls at any of these levels fail. Breaches at different levels have different implications. Generally, a breach at the hardware, hypervisor, OS, or database level is more serious than a breach at application level. For example, breaking into a database account on a server may permit reading, modifying, and destroying any data in a database, bypassing normal database-level controls. Breaking into a ‘root’ (administrator) account generally permits doing this to all data on all accounts on the system, plus installing deeply hidden backdoors, modifying the operating system, corrupting data, or rendering the system unbootable.
Most operating systems use user accounts as the primary unit of access control. In this document, a user account means an OS level user account unless otherwise specified. Many user accounts correspond to people (including system administrators), while service accounts are used for running application software or are used internally by the OS. It is also worth noting that many applications implement their own user accounts that do not correspond to OS level user accounts (they essentially share the service account of the application); however, such application-level accounts are generally beyond the scope of this document.
User accounts may be stored in a centralized repository (e.g., Active Directory [AD] or Lightweight Directory Access Protocol [LDAP]) or may be configured locally on a system. A user account defined locally is generally distinct on each system (separate password, separate home directory, etc.), even if the same account name is used on multiple computers, while user accounts defined in a centralized repository are often available on more than one computer and share the same home directory (on a networked file system). Service accounts are very commonly local accounts, and accounts for people are often stored in a directory.
In a very real sense, access control is the essence of information security. Other security technologies primarily exist to implement and enforce access controls, to make it harder to analyze and attack access control systems, limit the impact of actual breaches, evaluate the current state of protections, detect suspicious activity, counteract undesired activity, or help analyze what happened after the fact. The critical balance in information security is between the need to grant access and the need to limit access. Consequently, access must be provisioned (based on proper justification for that level of access) and it must be eventually terminated (e.g., when an employee leaves the role that justified the access, when a client system is decommissioned).
Automated access
Access has become increasingly automated. Examples of this automation include file transfers, disaster recovery, privileged access management, software and patch management, and dynamic cloud provisioning. This automation involves transferring data and executing commands, such as having hosts reconfigure other hosts. Thus, hosts must be able to access other hosts, often with very high privileges. Unfortunately, there has been little planning and oversight of automated machine-to-machine access control. Instead, such access has been added and configured on an ad hoc basis by system administrators, vendors, and integrators as part of other projects, without formal access control lifecycle management (e.g., standardized provisioning and termination processes, access token management (e.g., periodic password changes)). This publication explores the field of SSH-based automated access management, with a strong focus on security issues and how to best address them.
Secure shell
Secure Shell (SSH) is a protocol for logging into a remote host and executing commands on that host (e.g., administrative commands). SSH software is available for nearly every platform, and SSH is also embedded behind the scenes into a wide variety of IT, networking, and security technologies, including file transfer, systems management, identity management, and privileged access management. To summarize, the SSH protocol is widely used for remotely connecting to hosts, integrating hosts, and automating their operation. What distinguishes the SSH protocol from earlier remote administration protocols, such as telnet, remote shell (rsh), remote login (rlogin), and remote copy (rcp), is its built-in support for robust security features such as user authentication, device authentication, and transmission encryption. SSH has almost completely taken the place of these insecure remote administration protocols.
The SSH protocol has a typical client/server architecture. An SSH client application on host A initiates a connection to an SSH server application on host B. These two hosts negotiate encryption for their transmissions, then perform device authentication for the server host (host B), and finally send client authentication credentials (e.g., username and password) to the server. Assuming that this authentication succeeds, an SSH connection is said to be established between the hosts, ready for use.
There are three common use cases for SSH:
Interactive use
SSH is used by system administrators for manually managing and configuring Unix and Linux computers, networking equipment, and various other types of hosts remotely. SSH is also used for running applications remotely (particularly text-based legacy applications).
File transfers
SSH is used as the foundation of the Secure Copy (scp) and Secure File Transfer Protocol (SFTP) protocols. These protocols are used to transfer files between hosts while leveraging the security capabilities built into SSH.
Point-to-point tunneling
SSH can be used to implement a virtual private network (VPN) tunnel to protect data transmitted between two hosts. One or both of these hosts may be acting as a gateway for other hosts behind it.
SSH is frequently used for automated access for a variety of purposes, including managing large IT environments, integrating applications, and provisioning virtual machines in cloud services.
Automated access is commonly used with functional accounts, system accounts, service accounts, and other non-interactive user accounts (sometimes also called non-user accounts). Such accounts are used by operating systems, server applications (e.g., databases), and other applications for running processes. Automated access is also frequently used for file transfer functions.
Automated access may be unrestricted, allowing any commands to be executed, or may be limited to specific commands or operations, such as file transfers (perhaps limited to a specific directory). Organizations should limit automated access so that only the necessary commands can be executed and only the necessary resources can be engaged.
Authentication
The SSH protocol supports several mechanisms for authenticating users, including passwords, host-based authentication, Kerberos, and public key authentication. All these authentication methods fundamentally rely on some secret information, and when used for automated access, this secret information must be stored locally or be otherwise accessible. This section briefly discusses these forms of user authentication, focusing on their relevancy and appropriateness for automated access.
Password
There are two kinds of password authentication mechanisms in SSH: basic password authentication and keyboard-interactive authentication. Basic password authentication is a legacy method mandated by the SSH protocol standards. Keyboard-interactive authentication is used in most modern environments, and can support challenge-response authentication and one-time passwords in addition to traditional password authentication. Password authentication is commonly used for interactive users, but less commonly for automated access, although it is sometimes seen with hard-coded passwords in scripts and management systems.
Password authentication should generally not be used for automated access because hard-coded passwords may be obtained by attackers. If password authentication is used for automated access, the passwords should be rotated frequently in accordance with the organization’s password policy (which should also contain requirements such as minimum password length, minimum password complexity, etc.)
Host-based
Host-based authentication uses the server host’s host key—the key used by the client to verify the server’s identity—to authenticate the source host and to vouch for the identity of the user on the client side. A configuration file (.shosts) can be used with any user account on the server to specify which users on which hosts can log into that account without further authentication. However, host-based authentication does not permit configuring command restrictions—limits on what can be done on the server with the access. Because of this, its use for automated access is not recommended.
Kerberos
Many organizations use Kerberos or Active Directory authentication with SSH. Kerberos (usually together with LDAP, such as in Active Directory) implements single sign-on within a Windows domain or Kerberos realm, and allows user accounts to be stored in a centralized directory. In practice, Kerberos is rarely used for non-interactive accounts. While it can be configured to use keytab files or cached tickets for functional accounts for initial authentication, these approaches rely on having long-term credentials stored on the host or at least accessible to the process on the host that is obtaining tickets. These credentials can be exploited by an attacker to obtain a ticket granting ticket (TGT) for the functional account, and the attacker can then use single sign-on or other configured Kerberos-based trust relationships to gain access to other hosts or accounts that the functional account can access.
Certain widely used SSH implementations provide single sign-on within an Active Directory domain or Kerberos realm automatically by default. Such single sign-on implies that once access has been gained to one account using Kerberos, it is possible to log in to any other server that has the same account and is in the same domain (with single sign-on permitted) without further authentication. This can easily create lots of unwanted implicit trust relationships. Another concern is that currently widely used SSH implementations do not support command restrictions for Kerberos.
Because of these problems, the use of Kerberos authentication for automated access is not recommended.
Public Key
Public key authentication in SSH uses user keys or certificates to authenticate a connection. Such keys can be configured for both interactive users and processes, and they authorize the user or process to access a user account in an information system. An SSH client has a user key called an identity key, typically an RSA or DSA private key, and the server must have the corresponding public key configured as an authorized key for a user account. Any user in possession of the identity key is then allowed to log into the server to that user account and perform actions under the privileges configured for the key.
For interactive users, the identity key is usually stored on a smartcard or in a passphrase-protected file in a file system on a client device. If the identity key is protected by a passphrase, it is encrypted by a key derived from the passphrase. When SSH user keys are used for automated access, however, the identity key is usually stored as an unencrypted file (with no passphrase) in the file system. Given that the files grant access to servers, they contain sensitive data.
Many SSH implementations support configuring restrictions for authorized keys. These may be used for limiting what can be done on the server using the key (command restrictions) and for limiting the IP addresses from which the key can be used (source restrictions). Another advantage of public key authentication is that it does not create any implicit trust relationships, only expressly-defined trust relationships, and the permitted access can be reliably determined by inspecting the destination host. This is very important for being able to audit who can access what system and account; implicit trust relationships—access from server to server that is not explicitly configured—are hard to audit and easily get overlooked in security policy and audits. For these reasons, public key authentication is the recommended authentication mechanism for automated access with SSH. Public key authentication is by far the most frequently used method of configuring automated access using SSH as of this writing.
Management issues
In order to practically proceed to address SSH key management issues, a typical SSH access management or key management project involves:
• understanding the current situation, as most organizations have a substantial installed base of SSH keys that have not previously been managed
• procuring and deploying tools and establishing processes for efficient, controlled provisioning of automated SSH-based access and enforcing approvals according to policy
• remediating the existing environment, including establishing purpose for every configured authorized key, removing any authorized keys and identity keys that are not used or for which no justifiable purpose can be identified, configuring command restrictions, and rotating any remaining user keys.
An understanding of the existing environment can be obtained through audits. Tools are available that can help in the audit. Deploying the tools and establishing provisioning processes typically includes locking down keys and integrating into existing IT change control or approval systems. (The integration may be performed at a later stage, or in parallel with the remediation phase.)
In a typical remediation process, each host is taken under management using the tools, each host is monitored for a period of time (preferably several months) to identify which keys are actually used and how, unused/unneeded/rogue keys are removed, trust relationships that cross boundaries inappropriately or violate policy are removed, command restrictions (and source restrictions, if applicable) are added, and all keys are rotated. The remediation process is fairly labor-intensive. Depending on how many keys there are, how readily the owner and purpose of each key can be identified, and how much work is needed for writing change requests and waiting for maintenance windows, remediating a host may take several hours of work (tools can help a lot, but they cannot determine the purpose or owner of each key if that information hasn’t been recorded anywhere).
Generally it has been found that the cost of manual labor in a key management project is often as big or even significantly bigger than the cost of the tools for managing the keys. The choice of tools has a major impact on the amount of manual labor needed.