Native Kubernetes in AWS Background
-
Services run in Kubernetes Pods (pods)
-
A Kubernetes Node (node) runs one or more pods
-
A single node runs on a single AWS instance (1-to-1)
-
All instances in a cluster are the same instance size
Data Event Performance Factors
To illustrate, the following steps walk through a data event after a change is detected in a source system (e.g. database) and outlines performance factors for each step:
-
A change is made to a database table
-
The YOUnite Off-the-shelf Adaptors put minimal load on a source systems.
-
-
The adaptor detects the change and sends a data event to YOUnite API.
-
Performance will be affected if the existing load on the database connected to the adaptor is high
-
Change scan time
-
Processing time including updating the processing DB
-
Message bus queuing time
-
-
YOUnite API Service processes and routes the data event to other adaptors
-
Message bus de-queueing time
-
Data event processing/routing
-
Update data catalog
-
-
Message bus queuing time
-
-
Adaptor(s) receive change
-
Message bus de-queueing time
-
Update processing DB
-
Write to database
-
Sizing
All of the following suggestions are made as of AWS' product offerings as of January 1, 2024.
-
The following excludes:
-
Adaptors:
-
Adaptors can be deployed as part of the cluster or separately from the cluster
-
Adaptors can be run on-prem on their own hardware
-
For optimal performance, adaptors can be run on their own instance but multiple adaptors can be run on a single instance
-
The YOUnite Off-the-Shelf Adaptors include a Mongo DB processing service:
-
It can run on the same instance as the adaptor or on its own instance
-
A single instance of the Mongo DB can be shared between multiple adaptors
-
-
-
Data Notification Service (optional):
-
Depending on the number of notifications it must deliver and the overall load on the YOUnite deployment, it can be combined into an existing affinity group (run with any of the other YOUnite services) or part of an anti-affinity group (keeps it from running with another high-usage service).
-
-
Data Virtualization Service (optional):
-
The Data Virtualization service is horizontally scalable and can be run on 1 or 3 or more instances.
-
-
20 Million Data Events/Day
Cluster configuration
Instance (node) Size |
t3-large (2 core, 8 GiB) |
Cluster Nodes |
7 |
younite-api Pods |
4 |
Affinity Group |
younite-elastic, younite-logstash, younite-kibana, younite-ui |
Anti-Affinity Group |
younite-api, younite-db (metadata), younite-mb |
Service/Pod Configuration
| YOUnite Stack Service | requested mem | max mem | requested cpu | max cpu | storage |
|---|---|---|---|---|---|
younite-api |
4Gi |
6Gi |
1200m |
1600m |
default |
younite-db |
4Gi |
6Gi |
1200m |
1600m |
default (EBS) |
younite-mb |
4Gi |
6Gi |
1200m |
1600m |
20Gi |
younite-ui |
100Mi |
default |
default |
default |
default |
younite-ui |
100Mi |
default |
default |
default |
default |
younite-elastic |
1800Mi |
default |
default |
default |
default |
younite-logstash |
1800Mi |
default |
800m |
default |
default |
younite-kibana |
400Mi |
default |
default |
default |
default |
Keycloak (if needed) |
600Mi |
default |
default |
default |
default |
35 Million Data Events/Day
Cluster configuration
Instance (node) Size |
c5n-xlarge (4 core, 10.5 GiB) |
Cluster Nodes |
9 |
younite-api Pods |
6 |
Affinity Group |
younite-elastic, younite-logstash, younite-kibana, younite-ui |
Anti-Affinity Group |
younite-api, younite-db (metadata), younite-mb |
Service/Pod Configuration
| YOUnite Stack Service | requested mem | max mem | requested cpu | max cpu | storage |
|---|---|---|---|---|---|
younite-api |
4Gi |
6Gi |
1200m |
1600m |
default |
younite-db |
4Gi |
6Gi |
1200m |
1600m |
default (EBS) |
younite-mb |
4Gi |
6Gi |
1200m |
1600m |
20Gi |
younite-ui |
100Mi |
default |
default |
default |
default |
younite-ui |
100Mi |
default |
default |
default |
default |
younite-elastic |
1800Mi |
default |
default |
default |
default |
younite-logstash |
1800Mi |
default |
800m |
default |
default |
younite-kibana |
400Mi |
default |
default |
default |
default |
Keycloak (if needed) |
600Mi |
default |
default |
default |
default |
50 Million Data Events/Day
Cluster configuration
Instance (node) Size |
t2-xlarge (4 core, 16GiB) |
Cluster Nodes |
9 |
younite-api Pods |
6 |
Affinity Group |
younite-elastic, younite-logstash, younite-kibana, younite-ui |
Anti-Affinity Group |
younite-api, younite-db (metadata), younite-mb |
Service/Pod Configuration
| YOUnite Stack Service | requested mem | max mem | requested cpu | max cpu | storage |
|---|---|---|---|---|---|
younite-api |
4Gi |
6Gi |
1200m |
1600m |
default |
younite-db |
4Gi |
6Gi |
1200m |
1600m |
default (EBS) |
younite-mb |
4Gi |
6Gi |
1200m |
1600m |
20Gi |
younite-ui |
100Mi |
default |
default |
default |
default |
younite-ui |
100Mi |
default |
default |
default |
default |
younite-elastic |
1800Mi |
default |
default |
default |
default |
younite-logstash |
1800Mi |
default |
800m |
default |
default |
younite-kibana |
400Mi |
default |
default |
default |
default |
Keycloak (if needed) |
600Mi |
default |
default |
default |
default |
More than 50 Million
There are many AWS options that can be tested and levered to increase the data event performance beyond 50 million data events/day. Please contact your YOUnite System Integrator or Partner for more on strategies to enable YOUnite to meet your implementation requirements.