Overview
For logging, YOUnite uses Elasticsearch, Logstash and Kibana which are open source products developed by Elastic (https://www.elastic.co/elastic-stack). There are several versions of this software that may be used, each with different options. The basic version of the software from Elastic includes only very rudimentary security options, therefore, by default YOUnite has chosen to use the OpenSearch fork of Elasticsearch (https://opensearch.org/docs/latest) to take advantage of its enhanced security options. This distribution is completely free, open source and community developed.
Logstash receives logging events from the YOUnite Server, routing them to Elasticsearch, which indexes and stores them. Kibana is used for dashboards, visualizations and an easy interface to search the logs.
All API requests and Message Bus traffic (most notably all data events) are sent to Elasticsearch.
This page explains events that are logged and the fields and data values used in logging. As a YOUnite user, you can perform searches on requests and data events across the entire YOUnite Data Fabric from a single dashboard.
Note
|
In a production environment, Elastic recommends a three-node, highly available cluster for Elasticsearch. |
Why Centralized Logging?
Having a centralized logging platform is critical in a Data Fabric because, from a single dashboard, it allows:
-
API consumers to debug and validate requests/responses
-
Data Governance Stewards, DBAs, and similar data-focused staff to trace data origins and updates
-
IT staff to track usage
What Gets Logged?
-
All API requests made to the YOUnite Server
-
Server logs for the YOUnite Server, YOUnite Notification Service and YOUnite Data Virtualization Service
-
All data events:
-
Data events generated by adaptors
-
Data events routed to adaptors
-
Data events generated by assembler requests
-
-
Adaptor logs:
-
Adaptor startup and initialization
-
Adaptor data errors
-
-
Notification Service Logs
-
Web hook registrations
-
Notifications routed to a web hook
-
For a complete listing of what gets logged, the fields used including the data values that are used for specific fields - see the Logging Entries and Fields section below
Note
|
Although YOUnite centrally logs all API requests and data events – for security reasons – it does NOT log the data payloads. |
Elasticsearch
If you aren’t familiar with Elastic, the reference for Elasticsearch is here https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html.
A few points worth noting:
-
An elastic log entry is stored as a document
-
A group of documents stored in a log (collection) is called an index. Indexes can be indexed which is often a point of confusion.
-
Kibana is the primary tool used to search the log indexes.
-
Two languages can be used to search an index from Kibana:
-
Lucene
-
Kibana Query Language (KQL)
-
Some examples of how to search using Lucene are provided below.
YOUnite creates several indexes for each zone as described in the Logging Entries and Fields below.
For more, see "Elasticsearch Indexes" below.
Kibana
To view the logs, users with the appropriate credentials can log directly into Kibana or, they can log into Kibana via the
YOUnite UI by selecting LOGS
from the main dashboard:
Use the hamburger menu in the upper left to navigate between features in Kibana (note that Elastic changes the Kibana interface regularly and navigation between versions may change).
The most common features used with Kibana and YOUnite are:
-
Dashboard: A collection of panels that are used to analyze log data. Dashboards are configurable. See https://www.elastic.co/guide/en/kibana/current/dashboard.html.
-
Discover: Search and filter Elastic Logs (indexes) to gain insights and relationships using Lucene or Kibana Query Language. See https://www.elastic.co/guide/en/kibana/current/discover.html.
-
Maps & Visualize: Modify or create visualizations and maps that can be added to dashboards as panels. See the "Create Panels" section at https://www.elastic.co/guide/en/kibana/current/dashboard.html.
These docs provide examples of how to get started with these features. For the complete reference see These docs provide examples of how to use Kibana see https://www.elastic.co/guide/en/kibana/current/index.html.
Kibana Discover Page
YOUnite events are logged to indexed on a per zone as described in the Elasticsearch Indexes section below.
Kibana uses Lucene’s query and KQL syntax in the search bar. To query an index:
-
Select the appropriate index template e.g.
younite-api-*
-
Set the desired Time Range and an auto-refresh rate (if desired) in the upper right corner
See https://www.elastic.co/guide/en/kibana/current/discover.html for more on how to use the Discover Page.
Lucene Query Examples
Queries Using the younite-data-* Index Template
Query | Lucene Search |
---|---|
|
action:GET AND source-adaptor.name:"Asia Database - Oracle" |
Trace Data Events for data record (DR) with UUID 621a5a25-e95a-475a-abbc-865da254522a. |
dr-uuid: 621a5a25-e95a-475a-abbc-865da254522 |
Find All POST data events that originated on adaptors in the |
zone.name:"Michigan HQ" AND action:POST AND NOT exists:destination-adaptor.uuid |
Same as above but restrict query to an adaptor |
source-adaptor.name:"Oracle DB" AND action:POST AND NOT exists:destination-adaptor.uuid |
Same as above but restrict query to an adaptor |
source-adaptor.name:"North America Database - Postgres" AND action:POST AND domain-version.domain-name:customer AND NOT exists:destination-adaptor.uuid |
Queries Using the younite-api-* Index Template
Query | Lucene Search |
---|---|
All POST /zones requests. |
request:"/zones" AND method:POST |
All API requests except |
request:\/domains* AND NOT method:"GET" |
To search the logs for data errors:
log-entry-type: "ADAPTOR_ERROR"
To limit results to a single adaptor use the adaptor’s UUID with the source-adaptor-uuid
property e.g.:
log-entry-type: "ADAPTOR_ERROR" AND source-adaptor-uuid: "8862fdc0-43c9-4ec6-b5b0-a14ec3bdd352"
Kibana Dashboard
YOUnite ships with several visualizations out of the box. Select Dashboard
from the Kibana menu to survey what is available in the current release of YOUnite.
Kibana Visualizations and Maps
You can create your own dashboards and create you own visualizations and maps and add them to the existing YOUnite dashboards. See:
-
Dashboards: https://www.elastic.co/guide/en/kibana/current/dashboard.html
-
Maps & Visualize: https://www.elastic.co/guide/en/kibana/current/dashboard.html
Elasticsearch Indexes
Elasticsearch stores data into "indexes" (similar to a table in a SQL database).
Rather than storing all similar types of data into a single index, YOUnite partitions some indexes by zone and/or date, for efficiency and ease of archiving.
The following index patterns are used by YOUnite:
Pattern | Partitioned by | Description |
---|---|---|
younite-data-(zone uuid) |
Zone UUID |
Data events (messages with data to or from an adaptor), partitioned by zone |
younite-api-(zone uuid) |
Zone UUID |
API requests, partitioned by zone. Any requests not for a specific zone are logged in the root zone. |
younite-adaptor-(zone uuid) |
Zone UUID |
Messages from adaptors, partitioned by zone |
serverlog-(date) |
Date |
Application logs for the YOUnite Server and YOUnite Notification Service partitioned by date |
notifications-data-(zone uuid) |
Zone UUID |
Notifications sent to a web hook for a data event |
notifications-metadata-(zone uuid) |
Zone UUID |
Notifications sent to a web hook for a metadata event (such as a new or updated zone, domain, etc) |
notifications-registration-(zone uuid) |
Zone UUID |
Registrations for web hook notifications |
When querying these indexes, a wildcard can be used to aggregate data from multiple indexes, e.g. searching younite-data-*
will get all message bus events, regardless of the zone they are associated with. Using a wildcard in conjunction with an index
is called an "Index Pattern".
Elasticsearch Security
YOUnite uses a custom security gateway to filter requests to Elasticsearch based on the permissions of the authenticated user.
Logging Entries and Fields
There are seven types of log entries, each of which have different fields. The seven types are:
-
Data events (messages with data to or from an adaptor)
-
API request logs
-
Adaptor messages (information and/or error messages)
-
YOUnite Server and YOUnite Notification Service logs
-
Routed notifications for data events
-
Routed notifications for metadata events
-
Notification registrations
Data events (messages with data to or from an adaptor)
Data events are the messages sent from an adaptor to the router and vice-versa. Data events usually indicate some sort of change to a data record (new, updated, deleted), but can also indicate other things, like the response to a request to assemble a record, or a confirmation to YOUnite that a data event was processed by an adaptor.
For a single data event, multiple log entries may be recorded. For example, in a typical workflow where a new data record is created at an adaptor:
-
The adaptor sends a data event to the router. This is logged as a
ORIGINATED
message with aPOST
orPUT
action. -
The router checks ACLs and sends
ROUTED
messages to all adaptors capable of receiving the data event. -
Adaptors that received a
ROUTED
messsage may respond with aLINK_DR
message, indicating they successfully processing the data event and that YOUnite should link the data record to the adaptor. This is only done for new records, not updated records.
Below are details of each field in a data event log entry, along with their possible values.
Fields in the log entry
name | description | type | required | example |
---|---|---|---|---|
timestamp |
Time that the data event was logged |
Timestamp |
yes |
|
hostname |
Host name of the system that generated the event (either the adaptor or the router) |
String |
yes |
|
network-address |
IP address of the system that generated the event (either the adaptor or the router) |
String |
yes |
|
zone |
Zone associated with the data event. For an incoming data event, the zone associated with the source adaptor. For an outgoing data event, the zone associated with the destination adaptor. |
Zone |
yes |
|
transaction-id |
Unique identifier of the data event, shared by all log entries for the data event |
UUID |
yes |
|
action |
Action code |
Action |
yes |
|
status |
Status of the data event (did it succeed) |
Status |
yes |
|
source |
Source of the data event |
Source |
yes |
|
sequence |
Sequence in the flow of the data event |
Sequence |
yes |
|
domain-version |
Domain Version Information |
Domain Version |
yes |
|
dr-uuid |
Unique identifier of the data record. Required, except in the case of matching exceptions where it has not yet been determined. |
UUID |
no |
|
source-adaptor |
Information about the adaptor that sent the data event to YOUnite |
Adaptor |
no |
|
destination-adaptor |
Information about the adaptor that will receive the data event from YOUnite |
Adaptor |
no |
|
message |
Message associated with the data event (usually if there is an error) |
String |
no |
|
sso-id |
Authenticated user that originated the request. Applicable for assembler requests only. |
String |
no |
|
match-status |
Status of the match for |
MatchStatus |
no |
|
match-message |
Error message to display when there’s a matching or other data event exception. |
String |
no |
Fields in Zone
name | description | type | required | example |
---|---|---|---|---|
uuid |
UUID of the zone |
UUID |
yes |
|
name |
Name of the zone |
String |
yes |
|
location |
Latitude and longitude of the zone |
Coordinates (lat and lon) |
no |
Fields in Domain Version
name | description | type | required | example |
---|---|---|---|---|
uuid |
UUID of the domain version |
UUID |
yes |
|
domainUuid |
UUID of the domain |
UUID |
yes |
|
name |
Name of the domain |
String |
yes |
|
versionNumber |
Version number |
Number |
yes |
|
Fields in Adaptor
name | description | type | required | example |
---|---|---|---|---|
uuid |
UUID of the adaptor |
UUID |
yes |
|
name |
Name of the adaptor |
String |
yes |
|
zone |
Adaptor’s zone information |
Zone |
yes |
Possible values for enumerated types
type | value | description |
---|---|---|
Action |
GET |
Request from the router or response from an adaptor to send a data record to YOUnite. Used for Data Record assembly. See Data Record Assembly |
POST |
Data event for a Data Record that was created at an adaptor (a new Data Record). Functionally |
|
PUT |
Data event for a Data Record that was modified at an adaptor. Functionally |
|
DELETE |
Data event for a Data Record that was deleted at an adaptor. |
|
LINK_DR |
Sent by an adaptor to the router indicating that a Data Record has been received and that the router should link the
Data Record to this adaptor. An adaptor sends this message after receiving a |
|
MATCH |
Sent by an adaptor to the router indicating it wants to receive a corresponding MATCH message in return that includes the DR UUID of the record in the payload. |
|
Status |
SUCCESS |
The data event was processed successfully |
ERROR |
The data event was NOT processed successfully. A corresponding message will indicate what the error was. |
|
EXCEPTION |
The data event resulted in a matching or other data event exception |
|
Source |
ADAPTOR |
The data event was sent from an adaptor to the router |
ROUTER |
The data event was sent from the router to an adaptor |
|
Sequence |
ORIGINATED |
The data event came from an adaptor |
ROUTED |
The data event was routed to an adaptor |
|
ROUTED_RETURNED |
A data event coming from an adaptor in response to an assembly request by the router. See Data Record Assembly |
|
ASSEMBLED |
A data event indicating the successful or unsuccessful assembling of a data record. See Data Record Assembly |
API Request Logs
YOUnite logs all requests made to its REST API, whether successful or not. The request’s URL is logged along with the resulting HTTP status and/or error message, however, the payload of the request is not.
Below are details of each field in an API request log entry, along with their possible values.
Fields in the log entry
name | description | type | required | example |
---|---|---|---|---|
timestamp |
Time that the api request was logged |
Timestamp |
yes |
|
hostname |
Reserved for future use |
String |
no |
|
network-address |
IP address of the device that made the API call |
String |
yes |
|
zone |
Zone associated with the request. API calls under |
Zone |
yes |
|
method |
HTTP method |
String |
yes |
|
request |
URL of the request |
String |
yes |
|
status |
Status of the request (did not succeed). Possible values are |
String |
yes |
|
resource-type |
Type of resource accessed. Corresponds to the first part of the path, ie for |
String |
yes |
|
sso-id |
Authenticated user that originated the request. Will be null if authentication failed, or if the endpoint is public. |
String |
no |
|
http-status |
HTTP status code returned |
Number |
yes |
|
message |
Error message, if the request was not successful |
String |
no |
Adaptor Messages (information and/or error messages)
Fields in the log entry
name | description | type | required | example |
---|---|---|---|---|
timestamp |
Time that the api request was logged |
Timestamp |
yes |
|
hostname |
Host name of the adaptor |
String |
yes |
|
network-address |
IP address of the adaptor |
String |
yes |
|
zone |
Zone of the adaptor. |
Zone |
yes |
|
adaptor |
Adaptor information |
Adaptor |
yes |
|
state |
State of the adaptor. Possible values are As an adaptor starts up, it goes through various states, moving from |
String |
no |
|
message |
Informational or error message |
String |
no |
YOUnite Server Logs
The YOUnite Server, YOUnite Notification Service and YOUnite Data Virtualization Service console log entries are normal log4j log entries. The format as they’re moved through logstash into elastic is noted here: https://github.com/logstash/logstash-logback-encoder#loggingevent-fields
A custom field service
is included in the log entries to indicate which service the log entry is for.
Routed notifications for data events and metadata events
When the YOUnite Notification Service routes a notification for a data or metadata event to a web hook, a log entry is created with information about the event and the web hook registration. The actual payload of the event is not included.
Fields in the log entry
name | description | type | required | example |
---|---|---|---|---|
timestamp |
Time that the notification was routed |
Timestamp |
yes |
|
hostname |
Host name of the notification service |
String |
yes |
|
network-address |
IP address of the notification service |
String |
yes |
|
zone |
Zone associated with the event (see notes below) |
Zone |
yes |
|
registration |
Notification registration information |
Registration |
yes |
|
notification |
Notification information |
Data / Metadata Notification |
yes |
The zone of the notification is:
-
For
ORIGINATED
data events the zone is that of the source adaptor. -
For
ROUTED
data events the zone is that of the destination adaptor. -
For metadata data events either the zone of the resource or the root zone if the resource is not part of a zone.
Registration fields
name | description | type | required | example |
---|---|---|---|---|
registration-name |
Name of the registration |
String |
yes |
|
sso-id |
SSO id of the user that created the registration |
String |
yes |
|
callback |
URL of the webhook |
String |
yes |
|
oauth2-registration-name |
Name of the OAuth2 configuration to use from the server’s application properties file |
String |
no |
|
http-headers |
Names (but not values) of the HTTP headers sent to the webhook |
String |
no |
|
callback-type |
Type of callback. Currently WEB_HOOK is the only supported type. |
String |
yes |
|
Data Notification fields
name | description | type | required | example |
---|---|---|---|---|
action |
Action that occurred (POST, PUT, PATCH, DELETE) |
String |
yes |
POST |
resource |
Resource string of the data event |
String |
yes |
|
dr-uuid |
Data Record’s UUID |
String |
yes |
|
domain-version-uuid |
Domain version’s UUID |
String |
yes |
|
source-adaptor-uuid |
Source adaptor’s UUID |
String |
no |
|
destination-adaptor-uuid |
Destination adaptor’s UUID |
String |
no |
|
transaction-uuid |
UUID of the transaction (shared by all ORIGINATED and ROUTED events) |
String |
yes |
|
event-type |
Data event type (ORIGINATED or ROUTED) |
String |
yes |
|
Metadata Notification fields
name | description | type | required | example |
---|---|---|---|---|
action |
Action that occurred (POST, PUT, PATCH, DELETE) |
String |
yes |
|
resource |
Resource that was added, updated or deleted |
String |
yes |
|
type |
Resource type |
String |
yes |
|
Notification registrations
When a user adds, updates or a deletes a registration for notifications, an entry is logged.
Fields in the log entry
name | description | type | required | example |
---|---|---|---|---|
timestamp |
Time that the registration was added, updated or deleted |
Timestamp |
yes |
|
hostname |
Host name of the notification service |
String |
yes |
|
network-address |
IP address of the notification service |
String |
yes |
|
zone |
Zone associated with the registration (currently always the root zone) |
Zone |
yes |
|
action |
Action performed on the registration (either PUT or DELETE) |
String |
yes |
'PUT' |
registration |
Notification registration information |
Registration |
yes |
Registration fields
name | description | type | required | example |
---|---|---|---|---|
registration-name |
Name of the registration |
String |
yes |
|
sso-id |
SSO id of the user that created the registration |
String |
yes |
|
callback |
URL of the webhook |
String |
yes |
|
oauth2-registration-name |
Name of the OAuth2 configuration to use from the server’s application properties file |
String |
no |
|
http-headers |
Names (but not values) of the HTTP headers sent to the webhook |
String |
no |
|
callback-type |
Type of callback. Currently WEB_HOOK is the only supported type. |
String |
yes |
|
events |
List of events to register for |
Array of Event |
yes |
Event fields
name | description | type | required | example |
---|---|---|---|---|
resource |
Resource string (may include wildcards) |
String |
yes |
|
dr-event-type |
For data event notifications, the type of data events to register for (ALL, ORIGINATED or ROUTED) |
String |
no |
|
actions |
Actions to register for |
Array of String |
yes |
|