Numerous methods exist for encrypting and utilizing storage on AWS, specifically tailored for YOUnite’s use of Postgres, Logstash, Mongo, and S3. This guide offers insights into encryption best practices for these storage options and provides recommendations for their configuration and deployment.

Postgres

The YOUnite stack uses two Postgres databases:

  1. Service name: younite-db

    • Used by:

      • YOUnite API Service

      • YOUnite Data Virtualization Service

  2. Service name: younite-notifications-db

    • Used by:

      • YOUnite Notifications Service

To create an encrypted EBS volume for Postgres within a Native Kubernetes Cluster, the process involves several steps, including setting up the required IAM permissions, using CLI commands for creating the encrypted EBS volume, and adjusting a Kubernetes spec file to mount the EBS volume for databases.

1. IAM Permissions Needed

The necessary IAM permissions for creating an encrypted EBS volume and attaching it to an instance (which is then utilized by the Kubernetes cluster) encompass:

  • ec2:CreateVolume for the creation of encrypted EBS volumes.

  • ec2:AttachVolume for attaching the EBS volume to an instance.

  • ec2:DescribeVolumes to list volumes and check their encryption status.

  • kms:CreateKey for creating a new KMS key for EBS volume encryption (this is optional if the default AWS managed key is being used).

  • kms:UseKey for using the KMS key to encrypt volumes.

A sample IAM policy might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateVolume",
        "ec2:AttachVolume",
        "ec2:DescribeVolumes"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:CreateKey",
        "kms:UseKey"
      ],
      "Resource": "*"
    }
  ]
}

2. CLI Command to Create Encrypted EBS Volume

The AWS CLI can be used to create an encrypted EBS volume. If the AWS managed KMS key is being utilized for encryption, the KmsKeyId parameter can be omitted. The availability-zone should be replaced with the appropriate value for the specific setup.

aws ec2 create-volume --size 2000 --region us-west-2 --availability-zone us-west-2a --volume-type gp2 --encrypted --kms-key-id <kms-key-id> --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=encrypted-volume-name}]'

3. Kubernetes Spec File Modifications

For mounting the encrypted EBS volume in a Postgres pod, it is crucial to ensure that the PersistentVolume (PV) and PersistentVolumeClaim (PVC) are set up correctly to utilize the encrypted volume. Since a PVC is already in place, attention should be given to configuring the corresponding PV to claim the encrypted EBS volume.

Encryption of EBS volumes is handled at the volume level, so the Kubernetes cluster does not need to be directly aware of the encryption. However, it’s important that the storageClassName is correctly set to a class that supports encryption or is set up in an environment where EBS volumes are encrypted by default.

If there’s a need to explicitly create a PersistentVolume that uses an encrypted EBS volume, it would be defined as follows, assuming the EBS volume has already been created and encrypted:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: encrypted-pv-name
spec:
  capacity:
    storage: 2000Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  awsElasticBlockStore:
    volumeID: <volume-id>
    fsType: ext4

The <volume-id> should be replaced with the actual ID of the created EBS volume. This PV would then be claimed by the existing PVC, assuming the storage class and size are compatible.

The configuration for the StatefulSet seems adequately prepared to mount this volume for Postgres, assuming the PersistentVolume and PersistentVolumeClaim are correctly set up and bound, making further changes unnecessary for this aspect.

Elasticsearch

To use an encrypted Amazon EBS (Elastic Block Store) volume for Elasticsearch within a Kubernetes cluster, you will need to follow several steps that involve configuring IAM permissions, using the AWS CLI to create an encrypted volume, and then modifying your Kubernetes deployment to utilize this encrypted volume. Below are the steps detailed for each part of the process:

1. IAM Permissions Needed

To create and manage encrypted EBS volumes and to attach them to instances, ensure the IAM role or user has the following permissions:

  • ec2:CreateVolume for the creation of encrypted EBS volumes.

  • ec2:AttachVolume for attaching the EBS volume to an instance.

  • ec2:DescribeVolumes to list volumes and check their encryption status.

  • kms:CreateKey for creating a new KMS key for EBS volume encryption (this is optional if the default AWS managed key is being used).

  • kms:Encrypt for allowing data to be encrypted under the specified KMS key, ensuring data at rest is secured.

  • kms:Decrypt for permitting decryption operations, necessary for accessing the encrypted data.

  • kms:DescribeKey for enabling the retrieval of details about a specific KMS key, important for auditing and security compliance.

  • kms:UseKey for using the KMS key to encrypt volumes.

These permissions can be attached to the IAM role of the EC2 instances managed by Kubernetes, or to the user/role executing the AWS CLI commands.

2. CLI Command to Create an Encrypted EBS Volume

To create an encrypted EBS volume using the AWS CLI, execute the following command, replacing <KmsKeyId> with your KMS key ID and <AvailabilityZone> with the desired availability zone:

aws ec2 create-volume --size 1000 --region us-west-2 --availability-zone <AvailabilityZone> --volume-type gp2 --encrypted --kms-key-id <KmsKeyId>

This command creates a 1000 GiB, gp2 (General Purpose SSD) volume that is encrypted with the specified KMS key.

3. Modifying Your Kubernetes Spec to Use the Encrypted EBS Volume

By default, YOUnite’s Kubernetes deployment, uses a PersistentVolumeClaim (PVC) to request storage. To use an encrypted EBS volume, ensure that the storage class specified in the PVC supports encryption and is configured to use it.

AWS provides the gp2 storage class, which supports encryption. However, encryption must be enabled on the storage class or on the EBS volume directly as shown in the CLI command above.

Since the YOUnite Elasticsearch spec file does not explicitly define a PersistentVolume (PV) that uses an encrypted EBS volume, you would typically rely on dynamic provisioning through the storage class. Ensure your storage class is set up for encryption by default, or modify your PVC to request an encrypted volume explicitly.

If you need to ensure that the PVC uses an encrypted volume, you might have to create a custom storage class that has encryption enabled and then reference this storage class in your PersistentVolumeClaim definition by setting storageClassName to your custom class.

4. Starting Elasticsearch with the Spec File

The Kubernetes spec provided with YOUnite sets up a basic Elasticsearch (referred to as OpenSearch in your spec) service, PVC, and a StatefulSet for deployment. To integrate the encrypted EBS volume, ensure that the PersistentVolumeClaim is correctly configured to use an encrypted storage class as discussed.

No changes are needed in the YOUnite Elasticsearch (OpenSearch) container spec file to utilize the encrypted volume, as the encryption is transparent to the application. The application writes to the volume as it would to any other volume, with AWS handling the encryption and decryption transparently.

Ensure that your Kubernetes nodes have the appropriate IAM roles attached to allow them to use the KMS key for decrypting the EBS volume when it is mounted to the pod.

This setup starts Elasticsearch using an encrypted EBS volume for storing log entries and document data securely.

Mongo

1. IAM Permissions Setup

The necessary IAM permissions are detailed below. These permissions ensure the ability to manage EBS volumes and utilize KMS for encryption purposes:

  • ec2:CreateVolume for the creation of new EBS volumes.

  • ec2:AttachVolume for attaching of EBS volumes to EC2 instances.

  • ec2:DescribeVolumes for listing and describing EBS volumes, crucial for automation and management scripts.

  • kms:Encrypt for allowing data to be encrypted under the specified KMS key, ensuring data at rest is secured.

  • kms:Decrypt for permitting decryption operations, necessary for accessing the encrypted data.

  • kms:DescribeKey for enabling the retrieval of details about a specific KMS key, important for auditing and security compliance.

  • kms:CreateGrant for allowing the creation of grants for KMS keys, facilitating fine-grained permissions management for encrypted volumes.

To apply these permissions, you can create an IAM policy and attach it to the IAM role associated with your Kubernetes nodes or the specific EC2 instances running within your cluster. This ensures that the underlying infrastructure has the necessary permissions to manage encrypted volumes and utilize AWS KMS for encryption keys, aligning with security best practices for handling sensitive data in a cloud-native deployment.

2. Creating the Encrypted EBS Volume

To create an encrypted EBS volume using the AWS CLI, use the following command, substituting <kms-key-id>, <availability-zone>, and <volume-size> with your desired KMS key ID, availability zone, and volume size in GiB, respectively:

aws ec2 create-volume --size <volume-size> --region <region> --availability-zone <availability-zone> --volume-type gp2 --encrypted --kms-key-id <kms-key-id>

Example:

aws ec2 create-volume --size 1000 --region us-west-2 --availability-zone us-west-2a --volume-type gp2 --encrypted --kms-key-id alias/my-kms-key

This command creates a 1000 GiB gp2 volume in the us-west-2a availability zone, encrypted with the specified KMS key.

3. Mounting the EBS Volume in Kubernetes for MongoDB

To mount the encrypted EBS volume in a MongoDB container, you will need to modify your Kubernetes deployment spec file. However, Kubernetes itself doesn’t directly manage AWS EBS volumes. Instead, it uses Persistent Volumes (PV) and Persistent Volume Claims (PVC) to provision and attach storage to Kubernetes pods.

First, create a PV that points to the encrypted EBS volume, and then a PVC that the MongoDB deployment can use. However, the YOUnite spec file for MongoDB does not include the volume attachment so a PVC needs to added to the MongoDB deployment and referenced in the pod’s volume mounts.

The specific steps to modify your deployment to use a PVC for MongoDB would typically include:

  1. Creating a PersistentVolume (PV) that references the encrypted EBS volume.

  2. Creating a PersistentVolumeClaim (PVC) that requests storage of the size and access modes required.

  3. Modifying the MongoDB deployment to mount the PVC.

Given the complexities and specifics of each Kubernetes cluster and AWS environment, such as the need to match the EBS volume ID in the PV definition, and considerations for access modes and storage classes, detailed code examples for these steps would require more context about the cluster and AWS setup.

Ensure your Kubernetes nodes are in the same availability zone as the EBS volume to avoid issues with mounting the volume.

S3

It is best practices to enable default encryption for S3. Ensure that default encryption is enabled for your S3 bucket. This ensures that all new objects are encrypted when stored in the bucket.

YOUnite uses an S3 bucket for the Kops state store (cluster configuration and state).

For example, the following command is used to start a cluster (it will create the state store if the state store does not exist):

export KOPS_STATE_STORE=s3://myco-state-store

kops create cluster --name=my-cluster.local --state=$KOPS_STATE_STORE --zones=us-west-2a --node-count=8 --node-size=c5n.xlarge

When using Amazon S3 to store your Kubernetes/Kops state store, encryption and access control are vital for ensuring the security and privacy of your cluster’s configuration and state. AWS provides several mechanisms for encrypting objects in S3, and when it comes to integrating with tools like Kops, you should ensure that your S3 buckets are correctly configured for both encryption and access.

1. Encrypting S3 Bucket for Kops State Store

For encryption, you can use either:

  1. S3-Managed Keys (SSE-S3): Amazon S3 handles the encryption and decryption of your data automatically.

  2. AWS KMS-Managed Keys (SSE-KMS): Provides you with more control over the encryption keys, including the ability to create, manage, and view audit logs of key usage.

To specify encryption with KMS when creating or managing your S3 bucket for Kops, you would typically set up the encryption on the bucket itself through the AWS Management Console, AWS CLI, or an AWS CloudFormation template. Kops doesn’t directly handle specifying the encryption method for the S3 bucket; instead, it will inherit the bucket’s encryption settings.

2. Specifying Keys for Reading/Writing State Store

When you specify the KOPS_STATE_STORE environment variable, Kops uses the AWS SDK to interact with the specified S3 bucket. If you’re using SSE-KMS for encryption, you need to ensure that the IAM role or user you’re using to run Kops commands has the necessary permissions to use the specified KMS key. This includes permissions to encrypt, decrypt, and access the key.

To grant Kops the ability to read/write to the S3 bucket and use the KMS key, you should:

  1. Attach an IAM policy to the IAM role/user that includes permissions for the actions for the KMS key used for encryption:

    • s3: for the relevant bucket

    • kms:Encrypt for allowing data to be encrypted under the specified KMS key, ensuring data at rest is secured.

    • kms:Decrypt for permitting decryption operations, necessary for accessing the encrypted data.

    • kms:ReEncrypt for allowing to re-encrypt data from one encryption key to another directly within KMS without the need to decrypt the data to plaintext before re-encrypting it under a new key.

    • kms:GenerateDataKey for generating a data encryption key (DEK) that can be used outside of AWS KMS to encrypt data.

    • kms:DescribeKey for enabling the retrieval of details about a specific KMS key, important for auditing and security compliance.

  2. Use AWS CLI or AWS Management Console to ensure the bucket policy and KMS key policy allow these actions.

Example Policy for S3 and KMS Access

Here’s a simplified example of an IAM policy that grants access to an S3 bucket and a KMS key:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::myco-state-store",
        "arn:aws:s3:::myco-state-store/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": [
        "arn:aws:kms:us-west-2:123456789012:key/your-kms-key-id"
      ]
    }
  ]
}

When running Kops commands, as long as the environment (AWS credentials) from which you are executing the commands has the necessary permissions, Kops will be able to read from and write to the S3 bucket, respecting the encryption settings without needing specific configuration for the keys in the Kops command line.