How to Encrypt Large Artifacts and Stream Data Using Vault Envelope Encryption

Introduction

HashiCorp Vault's Transit secrets engine offers encryption-as-a-service, enabling applications to encrypt and decrypt small objects like tokens or secrets without managing keys directly. However, this model falters with large artifacts or streaming workloads—transferring massive payloads to Vault creates performance bottlenecks and network overhead. To solve this, Vault's envelope encryption SDK lets you encrypt and decrypt data locally while Vault controls key management and access policies. This guide walks you through setting up and using envelope encryption for large-scale data.

What You Need

• A running Vault server (version 1.7 or later) with the Transit secrets engine enabled.
• Authentication credentials – a token, AppRole, or other method to access Vault.
• The Vault envelope encryption SDK (available for Go, Python, and Java).
• A data storage system (e.g., object store, database, or file system) to hold encrypted artifacts.
• Application code that will call the SDK for encryption and decryption.
• Basic understanding of key management and data encryption concepts.

Step-by-Step Guide

1. Step 1: Configure Vault Transit and Generate a Key

   Enable the Transit secrets engine if not already done. Create a named encryption key that Vault will use to wrap your data encryption keys (DEKs). For example, using the Vault CLI or API:

   vault secrets enable transit
   vault write -f transit/keys/my-key

   Set appropriate policies so that only authorized clients can encrypt/decrypt data keys. Save the key name—you'll reference it in your application.

2. Step 2: Initialize the Envelope Encryption SDK

   Install the SDK in your project. For Python:

   pip install hvac

   Configure it with your Vault address, authentication token, and the Transit key path. Example:

   from hvac import Client
   client = Client(url='https://vault.example.com:8200', token='s.yourtoken')
   transit_key = 'transit/keys/my-key'

   The SDK handles envelope operations—no manual key management required.

3. Step 3: Encrypt a Large Artifact Locally

   When your application needs to encrypt a large file or stream, follow these sub-steps:

   • Request a new data key from Vault Transit. The SDK calls transit/generate-data-key with your Transit key name. Vault returns a plaintext DEK and an encrypted DEK (ciphertext).
   • Encrypt your data locally using the plaintext DEK. Use any symmetric encryption algorithm (e.g., AES-256-GCM) supported by your language's crypto library. The DEK is never sent over the network again.
   • Store the encrypted data along with the encrypted DEK (EDK). Bundle them together—for example, append the EDK as a header in a file or store it as metadata in a database.

   Your application performs the bulk encryption, Vault only sees the small key request.

4. Step 4: Decrypt the Artifact When Needed

   To decrypt, a consumer (authorized client) does the following:

   • Retrieve the encrypted artifact from your storage system.
   • Extract the EDK from the artifact's metadata or header.
   • Send the EDK to Vault using the transit/decrypt endpoint. Vault returns the plaintext DEK if the client is authenticated and authorized.
   • Decrypt the data locally using the DEK and the same algorithm used during encryption.

   Again, Vault never sees the large encrypted blob—only the small encrypted key.

5. Step 5: Handle Streaming Workloads

   For continuous data streams (e.g., log shippers, IoT sensors), adapt the same pattern:

   • Generate a fresh DEK per stream session or per batch. Each DEK is used only for a limited amount of data.
   • Encrypt chunks as they arrive using the DEK. Store each chunk's EDK alongside it if you need independent decryption; otherwise, reuse the same EDK for the entire stream.
   • Rotate the DEK periodically (e.g., every 100 MB) to limit the data encrypted with one key—a good security practice.

   The SDK's lightweight key fetch means minimal processing overhead for high-throughput streams.

Tips for Success

• Use key caching – Cache DEKs for a short time to reduce Vault calls, but be aware of key reuse risks. Balance performance with security.
• Mock decryption in tests – Use a local Vault Dev server or test policies to simulate encryption without impacting production.
• Monitor Vault audit logs – Track every key generation and decryption request to detect unauthorized access.
• Set up key rotation – Periodically create new Transit keys and re-wrap DEKs to comply with security policies.
• Handle edge cases – If a DEK is lost or Vault is unreachable, have a fallback plan (e.g., retry logic or manual recovery procedures).
• Version your artifacts – Include the Transit key version in the EDK metadata to ensure decryption always uses the correct key.

By following these steps, you can secure large datasets and streaming workloads without sacrificing performance or key management control. The envelope encryption SDK bridges the gap between Vault's centralized security and your data-intensive applications.