Overview

log

For logging, YOUnite uses Elasticsearch, Logstash and Kibana which are open source products developed by Elastic (https://www.elastic.co/elastic-stack). There are several versions of this software that may be used, each with different options. The basic version of the software from Elastic includes only very rudimentary security options, therefore, by default YOUnite has chosen to use the OpenSearch fork of Elasticsearch (https://opensearch.org/docs/latest) to take advantage of its enhanced security options. This distribution is completely free, open source and community developed.

Logstash receives logging events from the YOUnite Server, routing them to Elasticsearch, which indexes and stores them. Kibana is used for dashboards, visualizations and an easy interface to search the logs.

All API requests and Message Bus traffic (most notably all data events) are sent to Elasticsearch.

This page explains events that are logged and the fields and data values used in logging. As a YOUnite user, you can perform searches on requests and data events across the entire YOUnite Data Fabric from a single dashboard.

Note
In a production environment, Elastic recommends a three-node, highly available cluster for Elasticsearch.

Why Centralized Logging?

Having a centralized logging platform is critical in a Data Fabric because, from a single dashboard, it allows:

  • API consumers to debug and validate requests/responses

  • Data Governance Stewards, DBAs, and similar data-focused staff to trace data origins and updates

  • IT staff to track usage

What Gets Logged?

  • All API requests made to the YOUnite Server

  • Server logs for the YOUnite Server, YOUnite Notification Service and YOUnite Data Virtualization Service

  • All data events:

    • Data events generated by adaptors

    • Data events routed to adaptors

    • Data events generated by assembler requests

  • Adaptor logs:

    • Adaptor startup and initialization

    • Adaptor data errors

  • Notification Service Logs

    • Web hook registrations

    • Notifications routed to a web hook

For a complete listing of what gets logged, the fields used including the data values that are used for specific fields - see the Logging Entries and Fields section below


Note
Although YOUnite centrally logs all API requests and data events – for security reasons – it does NOT log the data payloads.

Elasticsearch

If you aren’t familiar with Elastic, the reference for Elasticsearch is here https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html.

A few points worth noting:

  • An elastic log entry is stored as a document

  • A group of documents stored in a log (collection) is called an index. Indexes can be indexed which is often a point of confusion.

  • Kibana is the primary tool used to search the log indexes.

  • Two languages can be used to search an index from Kibana:

    • Lucene

    • Kibana Query Language (KQL)

Some examples of how to search using Lucene are provided below.

YOUnite creates several indexes for each zone as described in the Logging Entries and Fields below.

YOUnite UI zone selection pulldown

For more, see "Elasticsearch Indexes" below.

Kibana

To view the logs, users with the appropriate credentials can log directly into Kibana or, they can log into Kibana via the YOUnite UI by selecting LOGS from the main dashboard:

YOUnite UI Landing Page, Logs Icon

Use the hamburger menu in the upper left to navigate between features in Kibana (note that Elastic changes the Kibana interface regularly and navigation between versions may change).

Kibana Hamburger Main Menu Icon

The most common features used with Kibana and YOUnite are:

These docs provide examples of how to get started with these features. For the complete reference see These docs provide examples of how to use Kibana see https://www.elastic.co/guide/en/kibana/current/index.html.

Kibana Navigation Menu

Kibana Discover Page

YOUnite events are logged to indexed on a per zone as described in the Elasticsearch Indexes section below. 

Kibana Discover Nav

Kibana uses Lucene’s query and KQL syntax in the search bar. To query an index:

  • Select the appropriate index template e.g. younite-api-*

  • Set the desired Time Range and an auto-refresh rate (if desired) in the upper right corner

See https://www.elastic.co/guide/en/kibana/current/discover.html for more on how to use the Discover Page.   

Lucene Query Examples

Queries Using the younite-data-* Index Template

Query Lucene Search

GET data events on an adaptor named Asia Database - Oracle.

action:GET AND source-adaptor.name:"Asia Database - Oracle"

Trace Data Events for data record (DR) with UUID 621a5a25-e95a-475a-abbc-865da254522a.

dr-uuid: 621a5a25-e95a-475a-abbc-865da254522

Find All POST data events that originated on adaptors in the Michigan HQ Zone.

zone.name:"Michigan HQ" AND action:POST AND NOT exists:destination-adaptor.uuid

Same as above but restrict query to an adaptor Oracle DB.

source-adaptor.name:"Oracle DB" AND action:POST AND NOT exists:destination-adaptor.uuid

Same as above but restrict query to an adaptor Oracle DB and the domain customer.

source-adaptor.name:"North America Database - Postgres" AND action:POST AND domain-version.domain-name:customer AND NOT exists:destination-adaptor.uuid

Queries Using the younite-api-* Index Template

Query Lucene Search

All POST /zones requests.

request:"/zones" AND method:POST

All API requests except GET made to /domains/ and below.

request:\/domains* AND NOT method:"GET"

To search the logs for data errors:

log-entry-type: "ADAPTOR_ERROR"
adaptorErrorLogEntry

To limit results to a single adaptor use the adaptor’s UUID with the source-adaptor-uuid property e.g.:

log-entry-type: "ADAPTOR_ERROR" AND source-adaptor-uuid: "8862fdc0-43c9-4ec6-b5b0-a14ec3bdd352"

Kibana Dashboard

YOUnite ships with several visualizations out of the box. Select Dashboard from the Kibana menu to survey what is available in the current release of YOUnite.

Kibana Dashboard
Kibana Visualizations and Maps

You can create your own dashboards and create you own visualizations and maps and add them to the existing YOUnite dashboards. See:

Elasticsearch Indexes

Elasticsearch stores data into "indexes" (similar to a table in a SQL database).

Rather than storing all similar types of data into a single index, YOUnite partitions some indexes by zone and/or date, for efficiency and ease of archiving.

The following index patterns are used by YOUnite:

Pattern Partitioned by Description

younite-data-(zone uuid)

Zone UUID

Data events (messages with data to or from an adaptor), partitioned by zone

younite-api-(zone uuid)

Zone UUID

API requests, partitioned by zone. Any requests not for a specific zone are logged in the root zone.

younite-adaptor-(zone uuid)

Zone UUID

Messages from adaptors, partitioned by zone

serverlog-(date)

Date

Application logs for the YOUnite Server and YOUnite Notification Service partitioned by date

notifications-data-(zone uuid)

Zone UUID

Notifications sent to a web hook for a data event

notifications-metadata-(zone uuid)

Zone UUID

Notifications sent to a web hook for a metadata event (such as a new or updated zone, domain, etc)

notifications-registration-(zone uuid)

Zone UUID

Registrations for web hook notifications

When querying these indexes, a wildcard can be used to aggregate data from multiple indexes, e.g. searching younite-data-* will get all message bus events, regardless of the zone they are associated with. Using a wildcard in conjunction with an index is called an "Index Pattern".

Elasticsearch Security

YOUnite uses a custom security gateway to filter requests to Elasticsearch based on the permissions of the authenticated user.

Logging Entries and Fields

There are seven types of log entries, each of which have different fields. The seven types are:

  1. Data events (messages with data to or from an adaptor)

  2. API request logs

  3. Adaptor messages (information and/or error messages)

  4. YOUnite Server and YOUnite Notification Service logs

  5. Routed notifications for data events

  6. Routed notifications for metadata events

  7. Notification registrations

Data events (messages with data to or from an adaptor)

Data events are the messages sent from an adaptor to the router and vice-versa. Data events usually indicate some sort of change to a data record (new, updated, deleted), but can also indicate other things, like the response to a request to assemble a record, or a confirmation to YOUnite that a data event was processed by an adaptor.

For a single data event, multiple log entries may be recorded. For example, in a typical workflow where a new data record is created at an adaptor:

  1. The adaptor sends a data event to the router. This is logged as a ORIGINATED message with a POST or PUT action.

  2. The router checks ACLs and sends ROUTED messages to all adaptors capable of receiving the data event.

  3. Adaptors that received a ROUTED messsage may respond with a LINK_DR message, indicating they successfully processing the data event and that YOUnite should link the data record to the adaptor. This is only done for new records, not updated records.

Below are details of each field in a data event log entry, along with their possible values.

Fields in the log entry

name description type required example

timestamp

Time that the data event was logged

Timestamp

yes

Dec 3, 2020 @ 14:40:40.073

hostname

Host name of the system that generated the event (either the adaptor or the router)

String

yes

younite-api-0.younite-api.default.svc.cluster-domain.example

network-address

IP address of the system that generated the event (either the adaptor or the router)

String

yes

100.96.10.141

zone

Zone associated with the data event. For an incoming data event, the zone associated with the source adaptor. For an outgoing data event, the zone associated with the destination adaptor.

Zone

yes

transaction-id

Unique identifier of the data event, shared by all log entries for the data event

UUID

yes

495cf308-135c-4462-9aae-6cef03b20d2e

action

Action code

Action

yes

POST

status

Status of the data event (did it succeed)

Status

yes

SUCCESS

source

Source of the data event

Source

yes

ADAPTOR

sequence

Sequence in the flow of the data event

Sequence

yes

ORIGINATED

domain-version

Domain Version Information

Domain Version

yes

dr-uuid

Unique identifier of the data record. Required, except in the case of matching exceptions where it has not yet been determined.

UUID

no

495cf308-135c-4462-9aae-6cef03b20d2e

source-adaptor

Information about the adaptor that sent the data event to YOUnite

Adaptor

no

destination-adaptor

Information about the adaptor that will receive the data event from YOUnite

Adaptor

no

message

Message associated with the data event (usually if there is an error)

String

no

sso-id

Authenticated user that originated the request. Applicable for assembler requests only.

String

no

user@younite.us

match-status

Status of the match for ORIGINATED data events. RESOLVED unless there was a matching or other data event exception.

MatchStatus

no

RESOLVED

match-message

Error message to display when there’s a matching or other data event exception.

String

no

Fields in Zone

name description type required example

uuid

UUID of the zone

UUID

yes

b204b37e-9b63-4fc9-9c76-51a66259eb5f

name

Name of the zone

String

yes

North America Headquarters

location

Latitude and longitude of the zone

Coordinates (lat and lon)

no

Fields in Domain Version

name description type required example

uuid

UUID of the domain version

UUID

yes

8f3dfa68-0b16-4c79-b54b-e69c3441af94

domainUuid

UUID of the domain

UUID

yes

3a419554-bb6f-443d-bbab-b8fe52300ef6

name

Name of the domain

String

yes

customer

versionNumber

Version number

Number

yes

1

Fields in Adaptor

name description type required example

uuid

UUID of the adaptor

UUID

yes

4dd94b78-3855-4cc3-baf5-2f266a35198d

name

Name of the adaptor

String

yes

Customer Database Adaptor

zone

Adaptor’s zone information

Zone

yes

Possible values for enumerated types

type value description

Action

GET

Request from the router or response from an adaptor to send a data record to YOUnite. Used for Data Record assembly. See Data Record Assembly

POST

Data event for a Data Record that was created at an adaptor (a new Data Record). Functionally POST and PUT are treated the same and indicate a new or updated record.

PUT

Data event for a Data Record that was modified at an adaptor. Functionally POST and PUT are treated the same and indicate a new or updated record.

DELETE

Data event for a Data Record that was deleted at an adaptor.

LINK_DR

Sent by an adaptor to the router indicating that a Data Record has been received and that the router should link the Data Record to this adaptor. An adaptor sends this message after receiving a POST or PUT data event that results in creation or updating of a data record not previously linked in YOUnite. This allows YOUnite to keep a record of which data records are linked at which adaptors.

MATCH

Sent by an adaptor to the router indicating it wants to receive a corresponding MATCH message in return that includes the DR UUID of the record in the payload.

Status

SUCCESS

The data event was processed successfully

ERROR

The data event was NOT processed successfully. A corresponding message will indicate what the error was.

EXCEPTION

The data event resulted in a matching or other data event exception

Source

ADAPTOR

The data event was sent from an adaptor to the router

ROUTER

The data event was sent from the router to an adaptor

Sequence

ORIGINATED

The data event came from an adaptor

ROUTED

The data event was routed to an adaptor

ROUTED_RETURNED

A data event coming from an adaptor in response to an assembly request by the router. See Data Record Assembly

ASSEMBLED

A data event indicating the successful or unsuccessful assembling of a data record. See Data Record Assembly

API Request Logs

YOUnite logs all requests made to its REST API, whether successful or not. The request’s URL is logged along with the resulting HTTP status and/or error message, however, the payload of the request is not.

Below are details of each field in an API request log entry, along with their possible values.

Fields in the log entry

name description type required example

timestamp

Time that the api request was logged

Timestamp

yes

Dec 3, 2020 @ 14:40:40.073

hostname

Reserved for future use

String

no

network-address

IP address of the device that made the API call

String

yes

172.217.3.174

zone

Zone associated with the request. API calls under /zones are associated with that zone. API calls not under /zones are associated with the root zone.

Zone

yes

method

HTTP method

String

yes

GET

request

URL of the request

String

yes

/zones

status

Status of the request (did not succeed). Possible values are SUCCESS and FAILURE. HTTP statuses of 400 and greater are considered to be a FAILURE.

String

yes

SUCCESS

resource-type

Type of resource accessed. Corresponds to the first part of the path, ie for GET /zones the resource-type is ZONES.

String

yes

ZONES

sso-id

Authenticated user that originated the request. Will be null if authentication failed, or if the endpoint is public.

String

no

user@younite.us

http-status

HTTP status code returned

Number

yes

200

message

Error message, if the request was not successful

String

no

Adaptor Messages (information and/or error messages)

Fields in the log entry

name description type required example

timestamp

Time that the api request was logged

Timestamp

yes

Dec 3, 2020 @ 14:40:40.073

hostname

Host name of the adaptor

String

yes

customer-db-adaptor-85ff85d998-ldmqd

network-address

IP address of the adaptor

String

yes

100.96.10.142

zone

Zone of the adaptor.

Zone

yes

adaptor

Adaptor information

Adaptor

yes

state

State of the adaptor. Possible values are DOWN, DOWN_PAUSED, ERROR, STARTING, STARTED, PAUSING and PAUSED.

As an adaptor starts up, it goes through various states, moving from DOWN to STARTING to STARTED. If an error occurs during startup, it will switch to ERROR.

String

no

STARTED

message

Informational or error message

String

no

YOUnite Server Logs

The YOUnite Server, YOUnite Notification Service and YOUnite Data Virtualization Service console log entries are normal log4j log entries. The format as they’re moved through logstash into elastic is noted here: https://github.com/logstash/logstash-logback-encoder#loggingevent-fields

A custom field service is included in the log entries to indicate which service the log entry is for.

Routed notifications for data events and metadata events

When the YOUnite Notification Service routes a notification for a data or metadata event to a web hook, a log entry is created with information about the event and the web hook registration. The actual payload of the event is not included.

Fields in the log entry

name description type required example

timestamp

Time that the notification was routed

Timestamp

yes

Dec 3, 2020 @ 14:40:40.073

hostname

Host name of the notification service

String

yes

notification-service-0

network-address

IP address of the notification service

String

yes

100.96.10.143

zone

Zone associated with the event (see notes below)

Zone

yes

registration

Notification registration information

Registration

yes

notification

Notification information

Data / Metadata Notification

yes

The zone of the notification is:

  1. For ORIGINATED data events the zone is that of the source adaptor.

  2. For ROUTED data events the zone is that of the destination adaptor.

  3. For metadata data events either the zone of the resource or the root zone if the resource is not part of a zone.

Registration fields

name description type required example

registration-name

Name of the registration

String

yes

sso-id

SSO id of the user that created the registration

String

yes

user@younite.us

callback

URL of the webhook

String

yes

https://webhook.site/88f1af6d-b7e2-4769-bec5-18327b8bff8e

oauth2-registration-name

Name of the OAuth2 configuration to use from the server’s application properties file

String

no

http-headers

Names (but not values) of the HTTP headers sent to the webhook

String

no

callback-type

Type of callback. Currently WEB_HOOK is the only supported type.

String

yes

WEB_HOOK

Data Notification fields

name description type required example

action

Action that occurred (POST, PUT, PATCH, DELETE)

String

yes

POST

resource

Resource string of the data event

String

yes

dr-uuid

Data Record’s UUID

String

yes

54092e2e-693f-4390-94c3-e8b461888631

domain-version-uuid

Domain version’s UUID

String

yes

bbe34857-c326-453a-8cc5-fa868dac39c0

source-adaptor-uuid

Source adaptor’s UUID

String

no

e8e2fdcb-c30e-4555-b9cd-b33a04af4fc6

destination-adaptor-uuid

Destination adaptor’s UUID

String

no

938de407-c457-4a3c-bab0-cdce15566c47

transaction-uuid

UUID of the transaction (shared by all ORIGINATED and ROUTED events)

String

yes

cefcabca-b1d0-400d-a34b-c8734c3290a7

event-type

Data event type (ORIGINATED or ROUTED)

String

yes

ORIGINATED

Metadata Notification fields

name description type required example

action

Action that occurred (POST, PUT, PATCH, DELETE)

String

yes

POST

resource

Resource that was added, updated or deleted

String

yes

/zones/2d257c06-8937-46ae-a3f8-9a38c5778c34

type

Resource type

String

yes

Zone

Notification registrations

When a user adds, updates or a deletes a registration for notifications, an entry is logged.

Fields in the log entry

name description type required example

timestamp

Time that the registration was added, updated or deleted

Timestamp

yes

Dec 3, 2020 @ 14:40:40.073

hostname

Host name of the notification service

String

yes

notification-service-0

network-address

IP address of the notification service

String

yes

100.96.10.143

zone

Zone associated with the registration (currently always the root zone)

Zone

yes

action

Action performed on the registration (either PUT or DELETE)

String

yes

'PUT'

registration

Notification registration information

Registration

yes

Registration fields

name description type required example

registration-name

Name of the registration

String

yes

sso-id

SSO id of the user that created the registration

String

yes

user@younite.us

callback

URL of the webhook

String

yes

https://webhook.site/88f1af6d-b7e2-4769-bec5-18327b8bff8e

oauth2-registration-name

Name of the OAuth2 configuration to use from the server’s application properties file

String

no

http-headers

Names (but not values) of the HTTP headers sent to the webhook

String

no

callback-type

Type of callback. Currently WEB_HOOK is the only supported type.

String

yes

WEB_HOOK

events

List of events to register for

Array of Event

yes

Event fields

name description type required example

resource

Resource string (may include wildcards)

String

yes

/zones/*

dr-event-type

For data event notifications, the type of data events to register for (ALL, ORIGINATED or ROUTED)

String

no

ORIGINATED

actions

Actions to register for

Array of String

yes

[ POST, PUT ]