This guide describes the biz demo and how to use some of YOUnite’s core features.

Important
Follow the instructions in Local Deployment Quick Start to start the stack and the biz demo.

Overview

The Biz demo has four adaptors, in three zones:

  • Asia Zone

    • PostgreSQL

  • North America Zone

    • PostgreSQL

  • EU Zone

    • SQL Server

    • MySQL

Each adaptor supports nine data domains:

  • customer

  • employee

  • department

  • product

  • employee_department

  • manufacturer

  • promotion

  • purchase_order

  • sale

Prerequisites

Important
Python 3 (version 3.7+) is required to run these scripts. They are not compatible with Python 2, which has been deprecated.
  • The YOUnite stack should be running with at least 12 GB of memory dedicated to Docker with 16 GB recommended (see the Local Deployment Quick Start guide).

  • Available Ports: The following ports must be free on your system:

Service Port

Mongo Express

8081

MongoDB

27017

PostgreSQL database

27018

SQL Server database

27019

MySQL database

27020

Postgres database

27021

  • Optionally, you may want to load your favorite database GUI. If you don’t have one, DBeaver is quite popular and is used in this guide.

Starting the demo

  • Verify that the stack is up and running by logging in to the user interface as the dgs or admin user. The default password is password:

    http://local.younite.org

There may be some python modules that need to be installed. If this is the case you will see an error such as:

ModuleNotFoundError: No module named 'requests'

Typically pip is used to install missing Python modules. In this case:

python3 -m pip install requests

For more on Python and pip see Installing Python Modules

Access the Underlying Databases

The demo databases can be accessed in two ways:

  1. Shell into the databases container and use one of the command line tools.

  2. Using a database GUI.

Use the following to create connections to the databases:

Connection Information

Name Database Type Database Container Port (local) Username Password

Asia

PostgreSQL

biz

asia-postgres-db

27021

postgres

mysecretpassword

Europe West

SQL Server

biz

europe-west-sql-server-db

27019

sa

Str0ngPwd!

Europe East

MySQL *

biz

europe-east-mysql-db

27020

biz

password

North America

PostgreSQL

biz

north-america-postgres-db

27018

postgres

mysecretpassword

For example, using DBeaver:

Postgres connection configuration (Asia)
Postgres connection configuration (North America)
SQL Server connection configuration
MySQL connection configuration
Note
When connecting to MySQL with DBeaver, there may be a "Public Key Retrieval not allowed" error. If this occurs, bring up the connection settings popup. To bring up the popup: From the Database Navigator right click on the MySQL Database and select Edit Connection
MySQL connection error - Goto Database navigator - Select MySQL - right click and Edit Connection
  • Select the Main tab select Edit Driver Settings

  • Select the Connection properties tab

  • Match the following settings (to add a property just click into the Name/Value table), and select OK:

MySQL connection error - select main tab - connectin properties tab - click into namve/value table and add allowPublicKeyRetrieval:true - useSSL:false

Mongo

In most instances, the YOUnite Off-the-Shelf Adaptor requires a "Processing" service that depends on MongoDB.

The processing service is automatically configured with the demo.

Mongo is deployed with mongo-express, a simple GUI to interact with it. Running locally, mongo-express can be accessed via http://localhost:8081.

For more on the Processing service and MongoDB see Processing Service Configuration in the "YOUnite Off-the-shelf Database Adaptor" guide.

Inspecting the Log Files With Docker

If there are issues with any of the containers, it is often helpful to view the Docker container’s log file:

Find the Docker container name:

docker ps

View the Docker container’s log:

docker logs <container-name>

Loading Data

After the start-biz-demo script has been run, there will be a folder below this script named biz-demo. It contains a folder called test-data with sql test files corresponding to the domains they have data for:

  • product.sql

  • department.sql

  • customer.sql

  • employee.sql

  • employee_department.sql

  • purchase_order.sql

  • sale.sql

To view them:

cd biz-demo
ls test-data

These contain either 1, 10, or 1,000 rows (or 100 rows in the case of department).

To load these files into a demo database, run the load.py script, e.g.:

python3 scripts/load.py -l -a 2 -f test-data/customer_1a.sql

This example will load the customer_1a.sql data into a locally running demo database number (#2 - Europe West - SQL Server).

The biz demo databases are numbered as follows:

  1. Asia - Postgres

  2. Europe - West SQL Server

  3. North American - Postgres

  4. Europe East - MySQL

Run load.py -h to see more on how the script is used.

For this demo lets just load ten customer records into the Asia (Postgres) database:

python3 scripts/load.py -l -a 1 -f test-data/customer_10.sql

OPTIONAL: python3 scripts/load-all-local.py is a quick script to load all the data:

OPTIONAL: The load.py script includes a "post processing" option (-p) that will do the following if enabled:

  1. Assign a sales_rep_id to customers who don’t yet have one. This will choose a random employee_id from the employee table.

  2. Create employee_department records for any employees who don’t have a record in this table. This will choose a random department_id from the department table.

  3. Assign a department_id to products who don’t yet have one. This will choose a random department_id from the products table.

Applying Data Governance and Synchronizing Data

From your browser, login into the local YOUnite UI:

http://local.younite.org/

Applying Data Governance

Once the data in the above load.sh step was loaded into the Asia Postgres database, the database’s adaptor detected the change and sent it as a data event to YOUnite. From there, YOUnite synchronized the data event to the other database adaptors.

Data governance can restrict the flow in or out of zones and/or adaptors.

To try data governance out, login to the YOUnite UI as the dgs (default password for the demo is password) and select the Asia Zone:

Select Zone

Select the Governance tab on the landing page:

Select Governance

Select the Outbound ACL tab and select ADD ACL:

Select Outbound ACL

Make the following selections:

  • SOURCE ADAPTOR select ALL

  • DOMAIN VERSION select ALL

  • DESTINATION ZONE select North America Headquarters

  • For the verbs (GET, PUT, POST, DELETE) select the ALL pulldown and choose Restrict All:

Then select ADD. This ACL will prevent all data events that originate in the Asia zone from getting routed to the North America Headquarters zone:

Save ACL

Your first ACL has been added:

Outbound ACL List

If using your database GUI and the database connections have been made, open the Asia (Postgres) database. For this example, we’ll be using DBeaver.

Note
With DBeaver, as with most other database viewers, you need to drill down in a Database Navigator to view the table data.

Using the connection information above, we have connected to the customer table in the four databases and created tabs for each:

Select Database

It is necessary to refresh the table to get the latest contents:

Refresh Database Table

Select the Asia Database and update Jewel MacKereth’s first name to Susan and select Save:

Change name in customer table

This change was made with the Postgres Database’s customer table connected to the YOUnite database adaptor in the Asia zone. The change was detected and sent as a data event to YOUnite where it was routed to only the two database adaptors in the Europe zone:

Change name in EU 1 (SQL server) customer table
Change name in EU 2 (MySQL) customer table

However, since the Asia zone had an ACL to block all data events originating in its zone to ALL adaptors in the North America zone, the North America Postgres database did not receive the change so the first name is still Jewel:

Change name in North America (Postgres) customer table

This is an example of how powerful data governance is with YOUnite.

Examine The Data Record’s Lineage

Understanding when a data record was created, updated and deleted is important for several reasons:

  • Managing data quality

  • Provides a tool to help reduce the costs associated with stale data

  • Data compliance - For customer related data, organizations will need to show when a data record was created and which systems it has been propagated to

There are two ways to get the lineage of a data record in YOUnite. The easiest way is:

Remain in the Asia Zone:

Select Zone

Select Data Access from the UI landing page:

Data Access page

Select the customer:1(FEDERATED) data domain:

Select Customer:1 Data Domain

Then select LINEAGE for any data record from the menu on the right:

Select any CUSTOMERID

The history of the data record is listed. To find the data record’s origin, look for ACTION = POST and (scrolling to the right) SEQUENCE = ORIGINATED:

CUSTOMERID 1AS7791993 lineage

The second method of retrieving a data record’s lineage is by viewing the actual logs. All requests to YOUnite and data events are logged in a central logger. More on logging and how to access and view the logs is is covered in the Logging guide.

Its important to understand that YOUnite does not log the actual data but just the events (this is important for data privacy policies).

However if logging the actual data is required see the YOUnite Temporal Adaptor guide.

Data Virtualization - Assemble Records in Real-Time

YOUnite does not store federated data records but assembles them in real-time from source systems.

Goto to the Gold and Silver Adaptor page:

Navigation Adaptors→Gold and Silver Adaptor

Then select customer:1 and set the Asia Database - Postgres as the GOLD adaptor:

Set Asia Adaptor as Gold

Return to the Data Access page and select the disk icon for CUSTOMERID 1AS7791993 (you may need to look on the second page of records to find it).

Select disk icon on master data page

The updated record is displayed. Since the Asia Postgres adaptor is the golden source, the record contains the updated name "Susan":

View firstname is set to SUSAN

Data Virtualization - Specifying a Different Golden Source System

Remain logged into the Asia Headquarters zone and goto the Gold and Silver Adaptor page again.

  • Select customer:1

  • Set the Asia Database - Postgres adaptor to BRONZE

  • Set the North America Database - Postgres as the GOLD adaptor

Set North America Adaptor as Gold

Return to the Data Access page and select the disk icon for CUSTOMERID 1AS7791993 again.

Since:

  1. Asia blocked the changes to North America

  2. The North America adaptor is the Gold adaptor

The firstname has the original firstname (Jewel) value:

View firstname is set back to Jewel

Compare Data Records Between Source Systems (e.g. Databases)

From the Data Access page, select the customer:1(FEDERATED) domain and select the record with CUSTOMERID``1AS7791993 (i.e. the Jewel or SUSAN McKereth record) then select COMPARE:

Select Metadata tab

The Compare tab is displayed.

Select the following adaptors:

  • Europe East Database - MySQL

  • Europe West Database - SQL Server

  • North America Database - Postgres

Select Metadata tab

Then select COMPARE:

The data records for each source system are displayed. Scroll to the bottom of the screen and select COMPARE.

The diffs are highlighted in red:

Select Metadata tab

Delete a Record - Right to be Forgotten

From the Data Access page, select the trash icon for the record where CUSTOMERID is 1AS7791993:

Select delete for CUSTOMERID 1AS7791993 tab

Then select Asia Database - Postgres and Delete At Adaptor(s) Only:

Select delete at Postegres adaptor

In the DB viewer, "refresh" the Postgres Asia database. The SUSAN MacKereth record (CUSTOMERID 1AS7791993) has been deleted from the database:

CUSTOMERID 1AS7791993 removed from table

Summary

That quick overview demonstrates most of the core YOUnite federated features:

  • Data Cataloging

  • Data Synchronization

  • Data Governance

  • Data Virtualization

  • Data Lineage

  • Global Delete

Notifications are another core feature covered in the Notification Service Guide.

Shutdown the Demo

Stop the Adaptors and Associated Databases

cd biz-demo
python3 scripts/stop.py -l

Shutdown the YOUnite Stack

Run the following from the working directory (where the docker-compose.yml was downloaded during the quick-start):

docker compose down -v

Alternatively, running the stop.cmd or stop.sh script will stop all running docker containers.

Troubleshooting

The following are some troubleshooting tips for checking that each service in the stack is running:

Troubleshooting Docker

One of the most common causes of failure of the YOUnite stack is lack of memory or disk space. Ensure that the Docker configuration is correct and that the hard drive with docker on it is not out of memory.

It’s not always apparent that this is the case, but things like containers stopping or restarting unexpectedly is one sign. Sometimes a docker container may also give a hint that it cannot allocate memory or disk space.

Checking Available Memory

To see how much memory Docker has available, run the following command and look for the line Total Memory:

docker system info

Freeing up Docker Disk Space

Docker may run out of disk space over time via normal usage. To free up space run the following commands, which will remove orphaned containers and volumes:

docker system prune
docker volume prune

Verify all docker containers are running

docker ps -a

The output should look like this and each service should have of a status of Up x minutes. If any have a status of Exited then there is a problem. Check the logs of the container to see why it exited.

CONTAINER ID   IMAGE                                                                                      COMMAND                  CREATED              STATUS                   PORTS                                                                                                                                                                             NAMES
5b51a593280a   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-db-adaptor:latest             "/bin/sh -c 'exec ja…"   About a minute ago   Up About a minute                                                                                                                                                                                          europe-east-adaptor
443428d8198b   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-db-adaptor:latest             "/bin/sh -c 'exec ja…"   About a minute ago   Up About a minute                                                                                                                                                                                          north-america-adaptor
fae4176e592c   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-db-adaptor:latest             "/bin/sh -c 'exec ja…"   About a minute ago   Up About a minute                                                                                                                                                                                          europe-west-adaptor
4b3721a078f1   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-db-adaptor:latest             "/bin/sh -c 'exec ja…"   About a minute ago   Up About a minute                                                                                                                                                                                          asia-adaptor
5b1c27d55f1b   mongo                                                                                      "docker-entrypoint.s…"   2 minutes ago        Up 2 minutes             0.0.0.0:27017->27017/tcp, :::27017->27017/tcp                                                                                                                                     mongo
7edd238ca9b0   mcr.microsoft.com/mssql/server:2017-latest                                                 "/opt/mssql/bin/nonr…"   2 minutes ago        Up 2 minutes             0.0.0.0:27019->1433/tcp, :::27019->1433/tcp                                                                                                                                       europe-west-sql-server-db
527b230c2aab   mysql:8                                                                                    "docker-entrypoint.s…"   2 minutes ago        Up 2 minutes             33060/tcp, 0.0.0.0:27020->3306/tcp, :::27020->3306/tcp                                                                                                                            europe-east-mysql-db
17f7624319a9   mongo-express                                                                              "tini -- /docker-ent…"   2 minutes ago        Up 2 minutes             0.0.0.0:8081->8081/tcp, :::8081->8081/tcp                                                                                                                                         mongo-express
5fc456ba60e9   postgres:12-alpine                      "docker-entrypoint.s…"   2 minutes ago        Up 2 minutes (healthy)   8080/tcp, 0.0.0.0:27021->1521/tcp, :::27021->1521/tcp                                                                                                                             asia-oracle-db
38b7e21e4e4c   postgres:12-alpine                                                                         "docker-entrypoint.s…"   2 minutes ago        Up 2 minutes             0.0.0.0:27018->5432/tcp, :::27018->5432/tcp                                                                                                                                       north-america-postgres-db
7f04abeec46a   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/youniteui:latest                      "/docker-entrypoint.…"   About an hour ago    Up About an hour         0.0.0.0:80->80/tcp, :::80->80/tcp                                                                                                                                                 younite-ui
e6e816be941a   postgres:12                                                                                "docker-entrypoint.s…"   About an hour ago    Up About an hour         0.0.0.0:5433->5432/tcp, :::5433->5432/tcp                                                                                                                                         younite-notification-service-db
e005f68ac724   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/activemq-artemis:2.18.0               "/docker-run.sh run"     About an hour ago    Up About an hour         1883/tcp, 5445/tcp, 5672/tcp, 0.0.0.0:8161->8161/tcp, :::8161->8161/tcp, 0.0.0.0:61613->61613/tcp, :::61613->61613/tcp, 9404/tcp, 0.0.0.0:61616->61616/tcp, :::61616->61616/tcp   younite-mb
ee99c91e67f4   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-api:latest                    "/bin/sh -c 'exec ja…"   About an hour ago    Up About an hour         0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                                                                                                                         younite-api
e91dc0f59d66   postgres:12                                                                                "docker-entrypoint.s…"   About an hour ago    Up About an hour         0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                                                                                                                         younite-db
8ff594114f35   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-notification-service:latest   "/bin/sh -c 'exec ja…"   About an hour ago    Up About an hour         0.0.0.0:8082->8080/tcp, :::8082->8080/tcp                                                                                                                                         younite-notification-service
32acedf65840   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-kibana:latest                 "/usr/local/bin/kiba…"   About an hour ago    Up 42 minutes            0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                                                                                                         younite-kibana
ee9ad58735c4   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-logstash:latest               "/usr/local/bin/dock…"   About an hour ago    Up About an hour         5044/tcp, 0.0.0.0:4560->4560/tcp, :::4560->4560/tcp, 9600/tcp                                                                                                                     younite-logstash
3c0620778383   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-elastic:latest                "/usr/local/bin/cust…"   About an hour ago    Up About an hour         9300/tcp, 9600/tcp, 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9650/tcp                                                                                                           younite-elastic
965c600636f2   160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/keycloak:14.0.0                       "/opt/jboss/tools/yo…"   About an hour ago    Up About an hour         8443/tcp, 0.0.0.0:8800->8080/tcp, :::8800->8080/tcp                                                                                                                               younite-keycloak

Verify the Databases are Healthy

See Accessing the Underlying Databases for info on how to connect to each database.

Verify the Adaptors are Healthy

The adaptors have no public endpoints, but there are a few things that can be done:

  1. Log in to the YOUnite UI and check that each adaptor has a status of STARTED.

  2. Check the logs of each adaptor, ie docker logs europe-east-adaptor

Troubleshooting the Flow of Data Events

If data is not flowing as expected, the following steps can be taken to diagnose where the problem may be occurring. Run the following steps and determine which validation steps failed:

  1. Load a test record into the source database, for example, this will load a customer record into the North America Adaptor:

    python3 scripts\load.py -a 3 -l -f test-data\customer_1a.sql
  2. Read the logs of the adaptor and look for an entry indicating that a batch was processed. docker logs north-america-adaptor:

    2021-10-27 | 21:54:35.386 | Thread-28            | INFO  | s.o.ScanningOrchestrationService | Added 1 keys to the queue for processing. Batch ID = 6179ca9b5850310c605b103c
    2021-10-27 | 21:54:36.582 | pool-1-thread-1      | INFO  | s.o.ScanningOrchestrationService | Processed batch 6179ca9b5850310c605b103c with size of 1 in 417ms. 1 modifications, 0 deletions, 0 discarded.
  3. Read the logs of the YOUnite Server. docker logs younite-api. The quick start turns on DEBUG logging so there will be a lot entries. What we’re looking for is an INFO entry that starts with Message Received: DrNotification and indicates a POST message was received (meaning a new record), for example:

    2021-10-27 21:54:36.596 [AR-Data-3] INFO  c.y.m.listeners.DataQueueListener - Message Received: DrNotification{drActions=[Action{action=POST, secondaryAction=null, dr=DR{uuid=null, domainVersionUuid=c9c58bb3-0a72-4e76-bc9c-0cbbe77ce68f}}], sourceAdaptor=3f031d28-cef5-4afa-884a-f31b2a9cf7c5, transactionUuid=null}
  4. Additionally, the DEBUG logs of the YOUnite Server should contain routing information. Look for entries that start with Sending JMS Message and include Destination = RA-Data-(destination adaptor uuid here). The RA-Data queues are what are read by the adaptors. For example:

    2021-10-27 21:54:38.404 [AR-Data-3] DEBUG c.y.messaging.AbstractMessageService - Sending JMS message. Sender = router, Destination = RA-Data-6ac05149-290c-4d78-b19b-15351472a472, source = DrNotification{drActions=[Action{action=POST, secondaryAction=null, dr=DR{uuid=6cf6a5fb-ddfd-4711-8b86-53cc7ee6bd62, domainVersionUuid=c9c58bb3-0a72-4e76-bc9c-0cbbe77ce68f}}], sourceAdaptor=3f031d28-cef5-4afa-884a-f31b2a9cf7c5, transactionUuid=f1a83ce8-6d75-40f6-bd23-57703007ff63}
  5. Finally, in the YOUnite Server logs, you should see that destination adaptors that successfully received the message will send back an INFO entry that starts with Message Received: DrNotification and indicates a LINK_DR message was received, for example:

    2021-10-27 21:54:39.603 [AR-Ops-1] INFO  c.y.m.l.OperationalQueueListener - Message Received: DrNotification{drActions=[Action{action=LINK_DR, secondaryAction=null, dr=DR{uuid=6cf6a5fb-ddfd-4711-8b86-53cc7ee6bd62, domainVersionUuid=c9c58bb3-0a72-4e76-bc9c-0cbbe77ce68f}}], sourceAdaptor=8864e3e1-67ac-47c8-b24e-14ded96399cc, transactionUuid=null}
  6. Next, check the destination adaptors logs to see if it received the data event. docker logs asia-adaptor:

    2021-10-27 | 21:54:39.252 | eListenerContainer-4 | INFO  | .y.a.s.s.AbstractJDBCDataService | Adding one record in customer:1

    If any of the following could not be validated

Step Likely Cause(s) Possible Solutions

1 - sql script failed

Database is down

1. Verify database connectivity

2. Stop both adaptor and database in docker. Restart the database first, once started, restart the adaptor.

2 - adaptor’s log entry not found

Adaptor is down / misconfigured or database does not have record

1. First verify existence of the record in the database.

2. Verify adaptor is in a STARTED status in YOUnite. If PAUSED, start the adaptor.

3. Check adaptor configuration (for the biz demo, the adaptors should already be configured correctly).

4. Check adaptor logs for error messages or stack traces.

5. Stop both adaptor and database in docker. Restart the database first, once started, restart the adaptor.

3 - POST message not received

Message bus not routing messages or YOUnite Server not receiving messages

1. Check the message bus logs for errors: docker logs younite-mb

2. Log in to Artemis Console - http://localhost:8161/console (user = artemis, password = artemis). Check the status, the broker should be running. Next, under queues find AR-Data. The column Message Count should be zero indicating there are no waiting messages. If the number is not zero, then the YOUnite Server is not receiving messages.

3. Check the logs of the YOUnite server for errors or stack traces: docker logs younite-api.

4. Stop the younite-api and younite-mb. Restart younite-mb first, then younite-api.

4 - data event not routed

Data Governance is blocking data events or DEBUG logging is not turned on

1. Check data governance configuration

2. If DEBUG logging is not enabled, this step cannot be validated, although the Artemis console can be checked to see if messages were sent to a specific adaptor.

5 - link messages not sent back

Destination adaptor is down / misconfigured

See step 6

6 - Destination adaptor log entry not found

Destination adaptor is down / misconfigured

1. Verify adaptor is in a STARTED status in YOUnite. If PAUSED, start the adaptor.

2. Check adaptor configuration (for the biz demo, the adaptors should already be configured correctly).

3. Check adaptor logs for error messages or stack traces.

4. Stop both adaptor and database in docker. Restart the database first, once started, restart the adaptor.

5. As in troubleshooting step 3 above, log in to the Artemis console. Find the data queue for the adaptor (it will be named RA-Data-{adaptor UUID}). If message count is not zero, that means there are pending messages that the adaptor has not received.