Data Virtualization Service Configuration

Overview

The YOUnite Data Virtualization Service is an optional service that adds Data Record Assembly or "Federated GET" functionality to the YOUnite Data Fabric platform.

The YOUnite Data Virtualization Service is a Spring Boot based microservice with a REST API that handles assembly requests. The REST API uses the same authentication as the REST API of the YOUnite Server.

This document describes configuration of this service. See link:Accessing-Data-Records.adoc for usage information, APIs, security, etc.

Dependencies

The YOUnite Data Virtualization Service has no additional dependencies, however, it shares two dependencies of the YOUnite Server:

PostgreSQL database
Message bus

Configuration

As a Spring Boot based application, configuration can be set using environment variables or a custom application.properties file.

Note

If adding an application.properties file, make sure it is in the same folder as the JAR file of the application, or somewhere else on the classpath. Spring prioritizes properties in an external (outside of the JAR file) application.properties file, so anything in this file will override the defaults.

YOUnite Server Configuration for Data Virtualization

Data Virtualization must be enabled in the YOUnite Server for certain functionality to work.

In the YOUnite Server set the environment variable DATA_VIRTUALIZATION_ENABLED to true. Another option is to set the value data.virtualization.enabled in application.properties to true.

Required Configuration

The following configuration values are required. Most of these are shared with the YOUnite Server. Some may have suitable default values, already, but minimally, the URLs, usernames and/or passwords of each of these services is required:

Authentication provider
Database connection
Message bus connection
Logstash connection (optional - for sending logs to elastic or other logging services)

Authentication Provider

Property	Description	Default	Example
OIDC_ISSUER	oic.issuer	Issuer URI of the Open ID Connect compliant authentication server. Normally the same as used by the YOUnite server.	http://localhost:8800/auth/realms/younite
OIDC_SSO_ATTRIBUTE	oidc.sso.attribute	Attribute that maps to the username in YOUnite, for example, email address.	email

Property

Description

Default

Example

OIDC_ISSUER

oic.issuer

Issuer URI of the Open ID Connect compliant authentication server. Normally the same as used by the YOUnite server.

http://localhost:8800/auth/realms/younite

OIDC_SSO_ATTRIBUTE

oidc.sso.attribute

Attribute that maps to the username in YOUnite, for example, email address.

Database, Message Bus and Logstash Connections

Environment	application.properties	Description	Default
SPRING_DATASOURCE_URL	spring.datasource.url	Database URL. This is the same database as used by the YOUnite server.	jdbc:postgresql://localhost:5432/younite
SPRING_DATASOURCE_USERNAME	spring.datasource.username	Database username.	younite
SPRING_DATASOURCE_PASSWORD	spring.datasource.password	Database password.	younite
SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE	spring.datasource.hikari.maximum-pool-size	Maximum database connection pool size.	10
MESSAGE_BUS_URL	message.bus.url	Message bus URL. This is the same message bus as used by the YOUnite server.	tcp://localhost:61616
MESSAGE_BUS_USERNAME	message.bus.username	Message bus username.	artemis
MESSAGE_BUS_PASSWORD	message.bus.username	Message bus password.	artemis
MESSAGE_BUS_SESSION_CACHE_SIZE	message.bus.session.cache.size	Maximum number of sessions to cache.	5
MESSAGE_BUS_VIRTUALIZATION_QUEUE_CONCURRENCY	message.bus.virtualization.queue.concurrency	Concurrent number of listeners. Expressed as a range, ie 1-5. If a single number is supplied it is the maximum number of listeners, with 1 being the minimum.	5
LOGGING_LOGSTASH_ENABLED	logging.logstash.enabled	Enable logging of requests through logstash (typically to send logs to elastic, but could be any supported destination). This should normally be the same logging configuration as the YOUnite Server.	true
LOGGING_LOGSTASH_HOST	logging.logstash.host	Logstash host name.	localhost
LOGGING_LOGSTASH_PORT	logging.logstash.port	Logstash port.	4560
LOGGING_LOGSTASH_ENABLE_SSL	logging.logstash.enable.ssl	Enable SSL for logstash.	false
LOGGING_LOGSTASH_TRUSTSTORE_LOCATION	logging.logstash.truststore.location	Location of the trust store file, if using custom SSL certificates. Must be in jks (Java KeyStore) format.
LOGGING_LOGSTASH_TRUSTSTORE_PASSWORD	logging.logstash.truststore.password	Password for the trust store file, if using custom SSL certificates.

Environment

application.properties

Description

Default

SPRING_DATASOURCE_URL

spring.datasource.url

Database URL. This is the same database as used by the YOUnite server.

jdbc:postgresql://localhost:5432/younite

SPRING_DATASOURCE_USERNAME

spring.datasource.username

Database username.

younite

SPRING_DATASOURCE_PASSWORD

spring.datasource.password

Database password.

younite

SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE

spring.datasource.hikari.maximum-pool-size

Maximum database connection pool size.

MESSAGE_BUS_URL

message.bus.url

Message bus URL. This is the same message bus as used by the YOUnite server.

tcp://localhost:61616

MESSAGE_BUS_USERNAME

message.bus.username

Message bus username.

artemis

MESSAGE_BUS_PASSWORD

message.bus.username

Message bus password.

artemis

MESSAGE_BUS_SESSION_CACHE_SIZE

message.bus.session.cache.size

Maximum number of sessions to cache.

MESSAGE_BUS_VIRTUALIZATION_QUEUE_CONCURRENCY

message.bus.virtualization.queue.concurrency

Concurrent number of listeners. Expressed as a range, ie 1-5. If a single number is supplied it is the maximum number of listeners, with 1 being the minimum.

LOGGING_LOGSTASH_ENABLED

logging.logstash.enabled

Enable logging of requests through logstash (typically to send logs to elastic, but could be any supported destination). This should normally be the same logging configuration as the YOUnite Server.

true

LOGGING_LOGSTASH_HOST

logging.logstash.host

Logstash host name.

localhost

LOGGING_LOGSTASH_PORT

logging.logstash.port

Logstash port.

4560

LOGGING_LOGSTASH_ENABLE_SSL

logging.logstash.enable.ssl

Enable SSL for logstash.

false

LOGGING_LOGSTASH_TRUSTSTORE_LOCATION

logging.logstash.truststore.location

Location of the trust store file, if using custom SSL certificates. Must be in jks (Java KeyStore) format.

LOGGING_LOGSTASH_TRUSTSTORE_PASSWORD

logging.logstash.truststore.password

Password for the trust store file, if using custom SSL certificates.

Optional Configuration

Environment	application.properties	Description	Default
SERVER_LANG	server.lang	Default language for internationalization support.	en_us
API_BASE_PATH	api.base.path	Base path of the REST API endpoints.	api
CORS_ALLOWED_ORIGINS	cors.allowed.origins	Allowed origins for CORS.	*
SPRING_DATASOURCE_*	spring.datasource.*	All other datasource configuration options. Reference here: https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#appendix.application-properties.data
SPRING_DATASOURCE_HIKARI_*	spring.datasource.hikari.*	This application uses HikariCP for connection pooling. List of options here: https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#appendix.application-properties.data
MESSAGE_BUS_UNIQUE_ID	message.bus.unique.id	Unique identifier of this node when registering with the message bus. Avoid special characters. Used as a suffix for the client ID of this node on the message bus as well as the suffix on the message bus queue for incoming messages. The value MUST be unique and CANNOT be shared by multiple nodes.	(a random UUID)

Environment

application.properties

Description

Default

SERVER_LANG

server.lang

Default language for internationalization support.

en_us

API_BASE_PATH

api.base.path

Base path of the REST API endpoints.

api

CORS_ALLOWED_ORIGINS

cors.allowed.origins

Allowed origins for CORS.

SPRING_DATASOURCE_*

spring.datasource.*

All other datasource configuration options. Reference here: https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#appendix.application-properties.data

SPRING_DATASOURCE_HIKARI_*

spring.datasource.hikari.*

This application uses HikariCP for connection pooling. List of options here: https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#appendix.application-properties.data

MESSAGE_BUS_UNIQUE_ID

message.bus.unique.id

Unique identifier of this node when registering with the message bus. Avoid special characters. Used as a suffix for the client ID of this node on the message bus as well as the suffix on the message bus queue for incoming messages.

The value MUST be unique and CANNOT be shared by multiple nodes.

(a random UUID)

High Availability and Scalability

The YOUnite Data Virtualization Service may be scaled to as many nodes as desired in a high availability or high demand environment. Each node in a HA environment has its own unique message bus queue to handle incoming responses from adaptors. When an assembly request is sent to an adaptor, the name of this queue is included so that the adaptor knows where to send this data. This makes the service infinitely horizontally scalable.

Horizontal vs Vertical Scaling

By nature, this service will never be as fast as querying a database directly. What this service does provide is the ability to query multiple adaptors and apply data governance and gold/silver/bronze adaptor priorities, which is quite something!

Horizontal Scaling

Horizontal scaling means adding more nodes, or instances, of this service, typically behind a load balancer. Horizontally scaling this application may increase the number of concurrent requests for assembly that can be handled.

Vertical Scaling

Vertical Scaling means adding more resources to existing nodes/instances (ie more CPU and/or memory). Vertically scaling this application may increase the speed at which requests are processed.

Other Considerations

Consider the resources allocated to the other services involved in data assembly, in particular:

The data source where the data is coming from
The adaptor which is querying this data source
The PostgreSQL database used by this service and the YOUnite Server
The message bus

Usage, Security, etc

See link:Accessing-Data-Records.adoc#Assembly for security and usage information.