This guide describes the biz demo and how to use some of YOUnite’s core features.
Important
|
Follow the instructions in Local Deployment Quick Start to start the stack and the biz demo. |
Overview
The Biz
demo has four adaptors, in three zones:
-
Asia Zone
-
PostgreSQL
-
-
North America Zone
-
PostgreSQL
-
-
EU Zone
-
SQL Server
-
MySQL
-
Each adaptor supports nine data domains:
-
customer
-
employee
-
department
-
product
-
employee_department
-
manufacturer
-
promotion
-
purchase_order
-
sale
Prerequisites
Important
|
Python 3 (version 3.7+) is required to run these scripts. They are not compatible with Python 2, which has been deprecated. |
-
The YOUnite stack should be running with at least 12 GB of memory dedicated to Docker with 16 GB recommended (see the Local Deployment Quick Start guide).
-
Available Ports: The following ports must be free on your system:
Service | Port |
---|---|
Mongo Express |
8081 |
MongoDB |
27017 |
PostgreSQL database |
27018 |
SQL Server database |
27019 |
MySQL database |
27020 |
Postgres database |
27021 |
-
Optionally, you may want to load your favorite database GUI. If you don’t have one, DBeaver is quite popular and is used in this guide.
Starting the demo
-
Verify that the stack is up and running by logging in to the user interface as the
dgs
oradmin
user. The default password ispassword
:http://local.younite.org
There may be some python modules that need to be installed. If this is the case you will see an error such as:
ModuleNotFoundError: No module named 'requests'
Typically pip
is used to install missing Python modules. In this case:
python3 -m pip install requests
For more on Python and pip see Installing Python Modules
Access the Underlying Databases
The demo databases can be accessed in two ways:
-
Shell into the databases container and use one of the command line tools.
-
Using a database GUI.
Use the following to create connections to the databases:
Connection Information
Name | Database Type | Database | Container | Port (local) | Username | Password |
---|---|---|---|---|---|---|
Asia |
PostgreSQL |
biz |
asia-postgres-db |
27021 |
postgres |
mysecretpassword |
Europe West |
SQL Server |
biz |
europe-west-sql-server-db |
27019 |
sa |
Str0ngPwd! |
Europe East |
MySQL * |
biz |
europe-east-mysql-db |
27020 |
biz |
password |
North America |
PostgreSQL |
biz |
north-america-postgres-db |
27018 |
postgres |
mysecretpassword |
For example, using DBeaver:
Note
|
When connecting to MySQL with DBeaver, there may be a "Public Key Retrieval not allowed" error. If this occurs, bring up the connection settings popup. To bring up the popup: From the Database Navigator right click on the MySQL Database and select Edit Connection
|
-
Select the
Main
tab selectEdit Driver Settings
-
Select the
Connection properties
tab -
Match the following settings (to add a property just click into the
Name/Value
table), and selectOK
:
Mongo
In most instances, the YOUnite Off-the-Shelf Adaptor requires a "Processing" service that depends on MongoDB.
The processing service is automatically configured with the demo.
Mongo is deployed with mongo-express, a simple GUI to interact with it. Running locally, mongo-express can be accessed via http://localhost:8081.
For more on the Processing service and MongoDB see Processing Service Configuration in the "YOUnite Off-the-shelf Database Adaptor" guide.
Inspecting the Log Files With Docker
If there are issues with any of the containers, it is often helpful to view the Docker container’s log file:
Find the Docker container name:
docker ps
View the Docker container’s log:
docker logs <container-name>
Loading Data
After the start-biz-demo
script has been run, there will be a folder below this script named biz-demo
. It contains
a folder called test-data
with sql test files corresponding to the domains they have data for:
-
product.sql
-
department.sql
-
customer.sql
-
employee.sql
-
employee_department.sql
-
purchase_order.sql
-
sale.sql
To view them:
cd biz-demo
ls test-data
These contain either 1, 10, or 1,000 rows (or 100 rows in the case of department).
To load these files into a demo database, run the load.py
script, e.g.:
python3 scripts/load.py -l -a 2 -f test-data/customer_1a.sql
This example will load the customer_1a.sql
data into a locally running demo database number (#2 - Europe West - SQL Server).
The biz demo databases are numbered as follows:
-
Asia - Postgres
-
Europe - West SQL Server
-
North American - Postgres
-
Europe East - MySQL
Run load.py -h
to see more on how the script is used.
For this demo lets just load ten customer records into the Asia (Postgres) database:
python3 scripts/load.py -l -a 1 -f test-data/customer_10.sql
OPTIONAL: python3 scripts/load-all-local.py
is a quick script to load all the data:
OPTIONAL: The load.py
script includes a "post processing" option (-p) that will do the following if enabled:
-
Assign a
sales_rep_id
tocustomers
who don’t yet have one. This will choose a randomemployee_id
from theemployee
table. -
Create
employee_department
records for any employees who don’t have a record in this table. This will choose a randomdepartment_id
from thedepartment
table. -
Assign a
department_id
toproducts
who don’t yet have one. This will choose a randomdepartment_id
from theproducts
table.
Applying Data Governance and Synchronizing Data
From your browser, login into the local YOUnite UI:
http://local.younite.org/
Applying Data Governance
Once the data in the above load.sh
step was loaded into the Asia Postgres
database, the database’s adaptor detected the change and sent it as a data event to YOUnite. From there,
YOUnite synchronized the data event to the other database adaptors.
Data governance can restrict the flow in or out of zones and/or adaptors.
To try data governance out, login to the YOUnite UI as the dgs
(default password for the demo is password
) and select the Asia Zone
:
Select the Governance
tab on the landing page:
Select the Outbound ACL
tab and select ADD ACL
:
Make the following selections:
-
SOURCE ADAPTOR
selectALL
-
DOMAIN VERSION
selectALL
-
DESTINATION ZONE
selectNorth America Headquarters
-
DESTINATION ADAPTOR
selectALL
-
For
GET
,PUT
,POST
,DELETE
options:-
Select the
ALL
pulldown and chooseRestrict All
:
-
-
Select
ADD
button
This ACL will prevent all data events that originate in the Asia
zone from getting routed to the North America Headquarters
zone:
Your first ACL has been added:
If using your database GUI and the database connections have been made, open the Asia
(Postgres) database. For this example, we’ll be using DBeaver.
Note
|
With DBeaver, as with most other database viewers, you need to drill down in a Database Navigator to view the table data.
|
Using the connection information above, we have connected to the customer table in the four databases and created tabs for each:
It is necessary to refresh the table to get the latest contents:
Select the Asia Database and update Jewel MacKereth’s first name to Susan
and select Save
:
This change was made with the Postgres Database’s customer
table connected to the YOUnite database adaptor in the Asia
zone. The change was detected and sent as a data event to YOUnite where it was routed to only the two database adaptors in the Europe
zone:
However, since the Asia
zone had an ACL to block all data events originating in its zone to ALL adaptors in the North America
zone, the North America Postgres database did not receive the change so the first name is still Jewel
:
This is an example of how powerful data governance is with YOUnite.
Examine The Data Record’s Lineage
Understanding when a data record was created, updated and deleted is important for several reasons:
-
Managing data quality
-
Provides a tool to help reduce the costs associated with stale data
-
Data compliance - For customer related data, organizations will need to show when a data record was created and which systems it has been propagated to
There are two ways to get the lineage of a data record in YOUnite. The easiest way is:
Remain in the Asia Zone
:
Select Data Access
from the UI landing page:
Select the customer:1(FEDERATED)
data domain:
Then select LINEAGE
for any data record from the menu on the right:
The history of the data record is listed. To find the data record’s origin, look for ACTION
= POST
and (scrolling to the right) SEQUENCE
= ORIGINATED
:
The second method of retrieving a data record’s lineage is by viewing the actual logs. All requests to YOUnite and data events are logged in a central logger. More on logging and how to access and view the logs is is covered in the Logging guide.
Its important to understand that YOUnite does not log the actual data but just the events (this is important for data privacy policies).
However if logging the actual data is required see the YOUnite Temporal Adaptor guide.
Data Virtualization - Assemble Records in Real-Time
YOUnite does not store federated data records but assembles them in real-time from source systems.
Goto to the Gold and Silver Adaptor
page:
Then select customer:1
and set the Asia Database - Postgres
as the GOLD
adaptor:
Return to the Data Access
page and select the disk icon for CUSTOMERID 1AS7791993
(you may need to look on the second page of records to find it).
The updated record is displayed. Since the Asia Postgres adaptor is the golden source, the record contains the updated name "Susan":
Data Virtualization - Specifying a Different Golden Source System
Remain logged into the Asia Headquarters
zone and goto the Gold and Silver Adaptor
page again.
-
Select
customer:1
-
Set the
Asia Database - Postgres
adaptor toBRONZE
-
Set the
North America Database - Postgres
as theGOLD
adaptor
Return to the Data Access
page and select the disk icon for CUSTOMERID 1AS7791993
again.
Since:
-
Asia blocked the changes to North America
-
The North America adaptor is the Gold adaptor
The firstname has the original firstname (Jewel
) value:
Compare Data Records Between Source Systems (e.g. Databases)
From the Data Access
page, select the customer:1(FEDERATED)
domain and select the record with CUSTOMERID``1AS7791993
(i.e. the Jewel or SUSAN McKereth record) then select COMPARE
:
The Compare
tab is displayed.
Select the following adaptors:
-
Europe East Database - MySQL
-
Europe West Database - SQL Server
-
North America Database - Postgres
Then select COMPARE
:
The data records for each source system are displayed. Scroll to the bottom of the screen and select COMPARE
.
The diffs are highlighted in red:
Delete a Record - Right to be Forgotten
From the Data Access
page, select the trash icon for the record where CUSTOMERID is 1AS7791993
:
Then select Asia Database - Postgres
and Delete At Adaptor(s) Only
:
In the DB viewer, "refresh" the Postgres Asia database. The SUSAN MacKereth
record (CUSTOMERID 1AS7791993
) has been deleted from the database:
Summary
That quick overview demonstrates most of the core YOUnite federated features:
-
Data Cataloging
-
Data Synchronization
-
Data Governance
-
Data Virtualization
-
Data Lineage
-
Global Delete
Notifications are another core feature covered in the Notification Service Guide.
Shutdown the Demo
Stop the Adaptors and Associated Databases
cd biz-demo
python3 scripts/stop.py -l
Shutdown the YOUnite Stack
Run the following from the working directory (where the docker-compose.yml
was downloaded during the quick-start):
docker compose down -v
Alternatively, running the stop.cmd
or stop.sh
script will stop all running docker containers.
Troubleshooting
The following are some troubleshooting tips for checking that each service in the stack is running:
Troubleshooting Docker
One of the most common causes of failure of the YOUnite stack is lack of memory or disk space. Ensure that the Docker configuration is correct and that the hard drive with docker on it is not out of memory.
It’s not always apparent that this is the case, but things like containers stopping or restarting unexpectedly is one sign. Sometimes a docker container may also give a hint that it cannot allocate memory or disk space.
Checking Available Memory
To see how much memory Docker has available, run the following command and look for the line Total Memory
:
docker system info
Freeing up Docker Disk Space
Docker may run out of disk space over time via normal usage. To free up space run the following commands, which will remove orphaned containers and volumes:
docker system prune docker volume prune
Verify all docker containers are running
docker ps -a
The output should look like this and each service should have of a status of Up x minutes
. If any have a status of
Exited
then there is a problem. Check the logs of the container to see why it exited.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5b51a593280a 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-db-adaptor:latest "/bin/sh -c 'exec ja…" About a minute ago Up About a minute europe-east-adaptor
443428d8198b 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-db-adaptor:latest "/bin/sh -c 'exec ja…" About a minute ago Up About a minute north-america-adaptor
fae4176e592c 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-db-adaptor:latest "/bin/sh -c 'exec ja…" About a minute ago Up About a minute europe-west-adaptor
4b3721a078f1 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-db-adaptor:latest "/bin/sh -c 'exec ja…" About a minute ago Up About a minute asia-adaptor
5b1c27d55f1b mongo "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:27017->27017/tcp, :::27017->27017/tcp mongo
7edd238ca9b0 mcr.microsoft.com/mssql/server:2017-latest "/opt/mssql/bin/nonr…" 2 minutes ago Up 2 minutes 0.0.0.0:27019->1433/tcp, :::27019->1433/tcp europe-west-sql-server-db
527b230c2aab mysql:8 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 33060/tcp, 0.0.0.0:27020->3306/tcp, :::27020->3306/tcp europe-east-mysql-db
17f7624319a9 mongo-express "tini -- /docker-ent…" 2 minutes ago Up 2 minutes 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp mongo-express
5fc456ba60e9 postgres:12-alpine "docker-entrypoint.s…" 2 minutes ago Up 2 minutes (healthy) 8080/tcp, 0.0.0.0:27021->1521/tcp, :::27021->1521/tcp asia-oracle-db
38b7e21e4e4c postgres:12-alpine "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:27018->5432/tcp, :::27018->5432/tcp north-america-postgres-db
7f04abeec46a 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/youniteui:latest "/docker-entrypoint.…" About an hour ago Up About an hour 0.0.0.0:80->80/tcp, :::80->80/tcp younite-ui
e6e816be941a postgres:12 "docker-entrypoint.s…" About an hour ago Up About an hour 0.0.0.0:5433->5432/tcp, :::5433->5432/tcp younite-notification-service-db
e005f68ac724 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/activemq-artemis:2.18.0 "/docker-run.sh run" About an hour ago Up About an hour 1883/tcp, 5445/tcp, 5672/tcp, 0.0.0.0:8161->8161/tcp, :::8161->8161/tcp, 0.0.0.0:61613->61613/tcp, :::61613->61613/tcp, 9404/tcp, 0.0.0.0:61616->61616/tcp, :::61616->61616/tcp younite-mb
ee99c91e67f4 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-api:latest "/bin/sh -c 'exec ja…" About an hour ago Up About an hour 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp younite-api
e91dc0f59d66 postgres:12 "docker-entrypoint.s…" About an hour ago Up About an hour 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp younite-db
8ff594114f35 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-notification-service:latest "/bin/sh -c 'exec ja…" About an hour ago Up About an hour 0.0.0.0:8082->8080/tcp, :::8082->8080/tcp younite-notification-service
32acedf65840 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-kibana:latest "/usr/local/bin/kiba…" About an hour ago Up 42 minutes 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp younite-kibana
ee9ad58735c4 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-logstash:latest "/usr/local/bin/dock…" About an hour ago Up About an hour 5044/tcp, 0.0.0.0:4560->4560/tcp, :::4560->4560/tcp, 9600/tcp younite-logstash
3c0620778383 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/younite-elastic:latest "/usr/local/bin/cust…" About an hour ago Up About an hour 9300/tcp, 9600/tcp, 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9650/tcp younite-elastic
965c600636f2 160909222919.dkr.ecr.us-west-2.amazonaws.com/younite/keycloak:14.0.0 "/opt/jboss/tools/yo…" About an hour ago Up About an hour 8443/tcp, 0.0.0.0:8800->8080/tcp, :::8800->8080/tcp younite-keycloak
Verify the Databases are Healthy
See Accessing the Underlying Databases for info on how to connect to each database.
Verify the Adaptors are Healthy
The adaptors have no public endpoints, but there are a few things that can be done:
-
Log in to the YOUnite UI and check that each adaptor has a status of
STARTED
. -
Check the logs of each adaptor, ie
docker logs europe-east-adaptor
Troubleshooting the Flow of Data Events
If data is not flowing as expected, the following steps can be taken to diagnose where the problem may be occurring. Run the following steps and determine which validation steps failed:
-
Load a test record into the source database, for example, this will load a customer record into the North America Adaptor:
python3 scripts\load.py -a 3 -l -f test-data\customer_1a.sql
-
Read the logs of the adaptor and look for an entry indicating that a batch was processed.
docker logs north-america-adaptor
:2021-10-27 | 21:54:35.386 | Thread-28 | INFO | s.o.ScanningOrchestrationService | Added 1 keys to the queue for processing. Batch ID = 6179ca9b5850310c605b103c 2021-10-27 | 21:54:36.582 | pool-1-thread-1 | INFO | s.o.ScanningOrchestrationService | Processed batch 6179ca9b5850310c605b103c with size of 1 in 417ms. 1 modifications, 0 deletions, 0 discarded.
-
Read the logs of the YOUnite Server.
docker logs younite-api
. The quick start turns on DEBUG logging so there will be a lot entries. What we’re looking for is anINFO
entry that starts withMessage Received: DrNotification
and indicates aPOST
message was received (meaning a new record), for example:2021-10-27 21:54:36.596 [AR-Data-3] INFO c.y.m.listeners.DataQueueListener - Message Received: DrNotification{drActions=[Action{action=POST, secondaryAction=null, dr=DR{uuid=null, domainVersionUuid=c9c58bb3-0a72-4e76-bc9c-0cbbe77ce68f}}], sourceAdaptor=3f031d28-cef5-4afa-884a-f31b2a9cf7c5, transactionUuid=null}
-
Additionally, the
DEBUG
logs of theYOUnite Server
should contain routing information. Look for entries that start withSending JMS Message
and include Destination =RA-Data-(destination adaptor uuid here)
. TheRA-Data
queues are what are read by the adaptors. For example:2021-10-27 21:54:38.404 [AR-Data-3] DEBUG c.y.messaging.AbstractMessageService - Sending JMS message. Sender = router, Destination = RA-Data-6ac05149-290c-4d78-b19b-15351472a472, source = DrNotification{drActions=[Action{action=POST, secondaryAction=null, dr=DR{uuid=6cf6a5fb-ddfd-4711-8b86-53cc7ee6bd62, domainVersionUuid=c9c58bb3-0a72-4e76-bc9c-0cbbe77ce68f}}], sourceAdaptor=3f031d28-cef5-4afa-884a-f31b2a9cf7c5, transactionUuid=f1a83ce8-6d75-40f6-bd23-57703007ff63}
-
Finally, in the YOUnite Server logs, you should see that destination adaptors that successfully received the message will send back an
INFO
entry that starts withMessage Received: DrNotification
and indicates aLINK_DR
message was received, for example:2021-10-27 21:54:39.603 [AR-Ops-1] INFO c.y.m.l.OperationalQueueListener - Message Received: DrNotification{drActions=[Action{action=LINK_DR, secondaryAction=null, dr=DR{uuid=6cf6a5fb-ddfd-4711-8b86-53cc7ee6bd62, domainVersionUuid=c9c58bb3-0a72-4e76-bc9c-0cbbe77ce68f}}], sourceAdaptor=8864e3e1-67ac-47c8-b24e-14ded96399cc, transactionUuid=null}
-
Next, check the destination adaptors logs to see if it received the data event.
docker logs asia-adaptor
:2021-10-27 | 21:54:39.252 | eListenerContainer-4 | INFO | .y.a.s.s.AbstractJDBCDataService | Adding one record in customer:1
If any of the following could not be validated
Step | Likely Cause(s) | Possible Solutions |
---|---|---|
1 - sql script failed |
Database is down |
1. Verify database connectivity 2. Stop both adaptor and database in docker. Restart the database first, once started, restart the adaptor. |
2 - adaptor’s log entry not found |
Adaptor is down / misconfigured or database does not have record |
1. First verify existence of the record in the database. 2. Verify adaptor is in a 3. Check adaptor configuration (for the biz demo, the adaptors should already be configured correctly). 4. Check adaptor logs for error messages or stack traces. 5. Stop both adaptor and database in docker. Restart the database first, once started, restart the adaptor. |
3 - POST message not received |
Message bus not routing messages or YOUnite Server not receiving messages |
1. Check the message bus logs for errors: 2. Log in to Artemis Console - http://localhost:8161/console (user = artemis, password = artemis). Check the status,
the broker should be running. Next, under queues find 3. Check the logs of the YOUnite server for errors or stack traces: 4. Stop the |
4 - data event not routed |
Data Governance is blocking data events or DEBUG logging is not turned on |
1. Check data governance configuration 2. If DEBUG logging is not enabled, this step cannot be validated, although the Artemis console can be checked to see if messages were sent to a specific adaptor. |
5 - link messages not sent back |
Destination adaptor is down / misconfigured |
See step 6 |
6 - Destination adaptor log entry not found |
Destination adaptor is down / misconfigured |
1. Verify adaptor is in a 2. Check adaptor configuration (for the biz demo, the adaptors should already be configured correctly). 3. Check adaptor logs for error messages or stack traces. 4. Stop both adaptor and database in docker. Restart the database first, once started, restart the adaptor. 5. As in troubleshooting step 3 above, log in to the Artemis console. Find the data queue for the adaptor (it will be named RA-Data-{adaptor UUID}). If message count is not zero, that means there are pending messages that the adaptor has not received. |