This page covers deployment topics specific to running YOUnite in Kubernetes inside of AWS.

Steps for deploying YOUnite in Kubernetes inside of AWS is covered in Deploying YOUnite Services Into a Kubernetes Cluster.

AWS Deployment Options

An example of the YOUnite stack deployed inside of pods in a Kubernetes cluster.

AWS Kube Arch

The following topology diagram shows which YOUnite Stack services are publicly accessible and which are not.

Missing from the diagram is the optional YOUnite DB Adaptor processing service DB which would typically reside in one of the BEHIND FIREWALL boxes.

youniteEcosystem
Note

The YOUnite API and Data Virtualization service are horizontally scalable, enabling operation across multiple Kubernetes pods.

The YOUnite stack services operate in a Single-AZ while the adaptors and source systems can operate in other AZs.

Typically, an external Identity Provider (IDP) is configured to integrate with YOUnibe. However, in the diagram provided, Keycloak is shown as a reference IDP.

High Availability

The stack can be failed-over to another AZ zone seamlessly with a momentary pause for the adaptor’s to reconnect to the new message bus instance (Active MQ).

Implementation Requirements

  1. Cross-region S3 Buckets

    • The cluster configuration is stored in S3 Buckets

  2. Highly Available Kubernetes Cluster Master:

    • Specifying multiple master nodes ensures that the control plane is fault-tolerant by distributing master nodes across different AZs.

  3. Mirrored Databases across AZs:

    • YOUnite Metadata Postgres DB

    • Elastic’s Logstash database

    • YOUnite Notification Service Postgres DB (if Notification Service is configured)

    • YOUnite DB Adaptor processing service Mongo DBs

  4. Message bus implementation:

  5. Elastic implementation:

  6. Keycloak is used as a reference IDP and not part of the standard deployment (YOUnite uses the customer’s IDP).

  7. The YOUnite Off-the-Shelf adaptor maintains DB scanning logs so that in the event of a failover, it can resume scanning from the last change-scan performed in the failed AZ.

AWS Region Support

There are no known regions that could hinder YOUnite deployment but the following must be validated for any region:

  • Compliance and legal restrictions

  • Service availability

  • Network latency

  • Resource limits

AWS IAM Privileges for YOUnite Cluster Administrators

A cluster administrator for YOUnite with Kubernetes does not require AWS root privileges for deployment or operations. The set of required privileges and an example of how to assign them is covered in the Create AWS IAM Cluster Administrator User guide.

Important
 It’s important to follow the principle of least privilege when assigning these permissions, granting only the necessary rights to perform the required tasks.

AWS Data Encryption

See Data Encryption.

Backing Up Data Stores and Configurations

Cluster Configuration Stored on the Local IT Host or Management Station

The host where the cluster is created can be a Local IT Host or Management Station (e.g host running on an AWS Workspace). Once the cluster is created, a local kubeconfig is stored on that system in file defined by the env var KOPS_STATE_STORE. Although this file can be recreated, it is best practices to back it up and the Kops, AWS and Kubernetes environment variables.

Consider using AWS S3 for secure and accessible storage.

  • First, ensure the AWS CLI is configured on your local host.

  • Then, you can use the aws s3 cp command to copy your $KUBECONFIG file and any other relevant configuration files or scripts that set your environment variables to an S3 bucket.

  • For added security, enable encryption and versioning on the S3 bucket.

  • Regularly updating these backups and testing restoration processes are also recommended to ensure your Kubernetes management capabilities are effectively preserved.

The Cluster’s Configuration / S3

The cluster’s configuration is stored in an AWS S3 bucket and therefore it is essential to have a backup strategy for it.

The backup strategy for S3 resources can include either just continuous backups, just periodic (snapshot) backups, or a combination of both, depending on your organization’s needs. Continuous backups track changes as they occur, allowing for point-in-time restores. Periodic backups are full snapshots taken at scheduled times or on-demand, with each subsequent backup being incremental.

For detailed guidance on using AWS Backup for Amazon S3, including best practices and cost considerations, please refer to the official AWS documentation on:

AWS EBS Volumes

YOUnite DB, Elasticsearch, YOUnite Notification DBs

It is recommended to use an AWS EBS store for these services.

For backing up an AWS Elastic Block Store (EBS), AWS offers several methods to ensure your data is securely backed up and can be recovered when needed. AWS Backup is the primary tool recommended for managing backups on AWS, providing a centralized and automated way to back up data across AWS services, including Amazon EBS volumes. With AWS Backup, you can create backup policies, known as backup plans, to specify your backup requirements, like the frequency of backups and the duration for retaining those backups. This service allows for tagging AWS resources for automated backup according to the defined backup plan.

Amazon EBS itself offers features conducive to data backup and recovery. EBS volumes, which provide block-level storage volumes for use with Amazon EC2 instances, are designed for data that requires quick access and long-term persistence. You can create point-in-time snapshots of EBS volumes, which are stored in Amazon S3 for durability. These snapshots can be used as the basis for creating new EBS volumes and can be copied across AWS Regions, aiding in geographical expansion, data center migration, and disaster recovery efforts. EBS supports several volume types tailored to different use cases, from transactional workloads to I/O-intensive applications.

When implementing a backup strategy for EBS volumes, it’s crucial to consider the architecture of your workload environment, the specific backup and recovery needs of your Amazon EC2 instances, and how these integrate with services like AWS Backup. Regular testing of the restore and recovery function is also essential to ensure that backups can effectively restore functionality as intende.

For detailed guidance on backing up Amazon EBS volumes, including step-by-step instructions and best practices, you can refer to the official AWS documentation:

The YOUnite-Off-the-Shelf Adaptor Processing Database

If using the YOUnite-Off-the-Shelf Adaptor with the Mongo DB processing DB enabled, then you will want to back it up as contains local data-store key information to make it easier to track federated data records.

One of the best practices for Mongo in AWS is to store the MongoDB on an EBS volume and to follow the backup practices for EBS covered above in AWS EBS Volumes.

Security Access Tokens

YOUnite uses Open ID Connect for granting access to the YOUnite API, YOUnite Notification and YOUnite Data Virtualization server.

The expiry of tokens is defined in the younite-api deployment file. See the SECURITY_ACCESS_TOKEN_VALIDITY_MINUTES property.

Enforcing time limits for API credential rotation is typically a security policy decision that can be enforced by the Identity Provider (IDP) managing the authentication.

YOUnite ships with Keycloak as reference IDP implementation. Credential rotation policies , including setting expiration for client secrets or certificates, are managed through client settings or by implementing custom policies that leverage Keycloak’s extensibility through its SPIs (Service Provider Interfaces). For specific configurations consult the official Keycloak documentation.

Prescriptive Guidance and Service Limits

Prescriptive guidance on managing AWS service limits involves proactive strategies such as documenting current limits, integrating AWS usage data, and setting up alerts for approaching limits. It also includes automating requests for limit increases to ensure seamless operations and prevent disruptions due to limit breaches. This approach helps organizations maintain optimal service performance and cost efficiency on AWS.

AWS Well-Architected Framework, includes prescriptive guidance on managing service limits effectively. This guidance encompasses strategies for documenting, monitoring, and proactively managing AWS service limits to ensure scalable, efficient, and uninterrupted service use. It also includes recommendations for using AWS tools and services to automate monitoring and alerts related to approaching or exceeding these limits. For detailed information, refer to the AWS Well-Architected Framework documentation.

Recovering YOUnite Service Software

YOUnite services are bundled in Docker images stored in YOUnite’s AWS ECR.

YOUnite Kubernetes configuration defines the Docker image by an ECR path and its version with an image tag:

image: <ecr-path>:<image tag>

Customers are provided an AWS key and secret that grant access to YOUnite' AWS ECR. The AWS key and secret are used at the Kops/Kubernetes startup to pull down the appropriate versions of YOUnite Docker images for the customer’s deployment.

Network Configuration

The default YOUnite Kubernetes cluster deployed in AWS has the following AWS network components. Deploying a YOUnite Kubernetes cluster is described in Deploying YOUnite Services into a Kubernetes Cluster (AWS).

VPCs

  • YOUnite creates a VPC that by default has the name <cluster-name>.k8s.local.

    • This VPC creates a single subnet with two route table entries:

      • Local traffic

      • External traffic

Security Groups

  • default: default VPC security group

  • nodes.<cluster-name>: Security group for Kubernetes nodes

  • k8s-elb-<id>: A Security group is created for each Kubernetes ELB - services with a public IP

    • api

    • keycloak (optional)

    • ui

    • kibana

  • launch-wizard: Security group for launch the YOUnite Kubernetes cluster into AWS

Network ACLs

Network ACL rules are automatically added by Kubernetes/Kops for allowing public facing service traffic for public facing services/subnets and restricting traffic for services residing on subnets that are private to the cluster.

Route Tables

A route table with the cluster’s name is created automatically by Kubernetes/Kops.

External Connections

Adaptors and source systems (e.g. databases or other data related services connected to adaptors) residing externaly to the cluster’s VPC require one ore more of the following network connections:

  • Internet Gateway (IGW)

  • VPN

  • AWS Direct Connect (DX)

  • VPC Peering

Load Balancers

A cluster load balancer is created automatically by Kubernetes/Kops.

Elastic IPs & CNAME Requirements

The following cluster services require public facing IPs and require entries in the deploying organization’s DNS maps:

  • api.younite

  • idp.younite

  • kibana.younite

  • ui.younite

For permanent deployments these should be static IPs. See Configuring DNS CNAME entries for more.