Storing Secrets in HashiCorp Vault

This is the second post in a series on Vault. For the previous post see the Getting Started.

Overview Link to heading

Vault has a variety of secrets engines that store, generate, or encrypt data. Basic engines will simply store and read data, while more complex engines will connect to external services and have the ability to generate dynamic credentials on demand.

Engines Link to heading

Here are some details on a few of the available engine types:

Key Value Link to heading

The most familiar, and maybe the most common engine is the key/value (kv) store. As the name implies, this engine stores arbitrary keys and values, like username=root and password=secret.

When using the key value engine, use version 2. Version 2 provides the ability to do versioning, TTLs, and other helpful features. This allows users to roll back to a previous version or at least have a record of what it was.

Clouds Link to heading

Vault offers some cloud-specific engines. Trying to manage cloud credentials is already tough as it is. What vault can do is store a main set of credentials and then dynamically generate credentials based on cloud policies that can also be time-based and revoked as necessary.

There are currently four clouds supported:

• Amazon Web Services
• Microsoft Azure
• Google Cloud
• AliCLoud.

Databases Link to heading

Similar to the clouds, the numerous database engines allow for generating database credentials dynamically based on roles. This allows services using the databases to no longer need hardcoded database values.

See the docs for an extensive list of the supported database.

Others Link to heading

The above only touches on a few of the secrets engines. There are many others that cover other technologies and platforms like Active Directory, OpenLDAP, PKI (Certificates), and more!

Secrets CLI Link to heading

To get a list of the current secrets engines run the following:

1
2
3
4
5
6
7

vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_4536baf4    per-token private secret storage
identity/     identity     identity_1ce81d16     identity store
secret/       kv           kv_9cae2ec0           key/value secret storage
sys/          system       system_f3713085       system endpoints used for control, policy and debugging

The cubbyhole secrets engine is used to store arbitrary secrets per token. Secrets stored there are tied to the lifetime of an authentication token. This means that when the token expires the corresponding cubbyhole is destroyed.

Identity is used later for access to secrets.

System is used by the system and is where policies will live.

KV Secrets Link to heading

Time to store some secrets! Below is an example which enables the kv engine v2 and stores the password to a devel mailing list:

1
2
3
4
5
6
7
8
9

$ vault secrets enable -version=2 -path=shared kv
Success! Enabled the kv secrets engine at: shared/
$ vault kv put shared/mailing_list/devel password=secret
Key              Value
---              -----
created_time     2020-08-28T18:14:03.334192987Z
deletion_time    n/a
destroyed        false
version          1

KV CLI Link to heading

Here is an overview of the CLI commands to interact with kv secrets:

The delete subcommand does a soft delete, that will not return the value during a get. A user can still undelete the value if required. The destory removes the value entirely.

Due to the additional versioning of kv pairs in version 2, rollback allows a user to go back to a previous version of a key. The version information is visible via the metadata command.

Hiding Secrets Link to heading

Having the password in shell history is less than ideal. other options include reading from stdin or even a JSON file:

1
2
3
4
5
6

# Using JSON
echo -n '{"key":"value"}' | vault kv put engine/path  -
vault kv put engine/path @data.json
# Using simple value
echo -n "value" | vault kv put engine/path key=-
vault kv put engine/path key=@data.txt

See the CLI docs for more details on hiding a password from the CLI.

Organizing Secrets Link to heading

There are no clear documents on how to organize secrets. While this is entirely dependent on a user’s specific scenario here are some ideas someone might consider.

A secrets engine can be created multiple times and each time it is created, the user needs to give it a path. This path can be any arbitrary name. Under each path, the actual secrets are stored.

Similarly, each key value secret is created under a path and then has one or more key value entries. This results in the following structures:

engine-path/secret-path key=value key=value

Domain Centric Link to heading

One possible then is to have a key-value engine for each domain or site. Then paths for different secrets can get created based on node or subset. A couple examples could look like:

Keep in mind that there are cloud specific engines which allow for the creation of dynamic credentials. These credentials also provide additional securities as they are time-based and revoked when the Vault lease expires.

Node Centric Link to heading

For users with lots of node specific credentials another option is to have each node specified as the engine path:

This makes creating policies to limit node acces are much easier in this way. Nodes can be limited to their specific path only. A shared path for non-node specific credenitals can be shared with all nodes.

Again, consider using more than the key value secrets engine whenever possible due to the additional features, like dynamic creation, that are wrapped around them.

Next Steps Link to heading

Now that some secrets are stored, it is time to look at managing access policies for accessing the secrets.