This document covers creating and managing adaptors. Other adaptor topics are covered in the following:

Note: Most tasks outlined in this document can be done in the YOUnite UI and some can only be done via the YOUnite Server API.

Adaptor Deployment Summary

Below is a summary of the steps needed to get an adaptor fully operational. Full descriptions are included below in the Adaptor Deployment Instructions below:

Adaptor

  1. Add an Adaptor Type using the YOUnite API POST /adaptor-types endpoint. This is an optional step that may make deploying groups of adaptors easier as it contains some default values that will be inherited by the adaptor.

  2. In the YOUnite UI Adaptor’s page (or YOUnite API POST /zones/<target-zone-uuid>/adaptors endpoint) add the adaptor to the target zone.

  3. The prior step will generate the necessary configuration for connecting to the YOUnite Message Bus: the Adaptor UUID and authentication credentials (clientId and clientSecret). These values will be used to configure the Adaptor.

  4. Set the adaptor Configuration Metadata and/or capabilities properties in the YOUnite UI’s adaptor page (or YOUnite Server API PUT /zones/<target-zone-uuid>/adaptors/<adaptor-uuid>). The Configuration Metadata property is typically used to specify configuration information for the Adaptor - see the the instructions provided with the Adaptor software for instructions on how to configure the adaptor with these properties.

  5. Follow the instructions provided with the Adaptor software to deploy the adaptor using the credentials generated in step #3. It must be on an instance that can connect to the YOUnite message Bus. Once it is running you can:

    1. Change an adaptor’s state:

      1. Pause: Instructs the adaptor to stop sending and receiving data events. Data events sent to the adaptor will be queued in the message bus until the adaptor resumes.

      2. Play: Instructs the adaptor to resume sending and receiving data events.

    2. Manage the adaptor:

      1. Delete an Adaptor

      2. Get new Credentials for an adaptor

Adaptor Software

The Adaptor software may be an off the shelf adaptor provided by YOUnite (see YOUnite DB Adaptor) or a custom adaptor from your development environment or a 3rd party product written with the YOUnite Adaptor SDK (see YOUnite Adaptor SDK Summary).

Adaptor Deployment Instructions

1. Creating an Adaptor Type in YOUnite

This is an optional step that may make deploying groups of adaptors easier as it allow you to define default capabilities and/or metadata that will be inherited by adaptors.

POST /adaptor-types

{
    "name": "My First Adaptor Type",
    "description": "An adaptor type with some default capabilities",
    "version": "1",
    "adaptorTypeLabel": "FIRST",
    "defaultCapabilities": [
      {
        "domainName": "customer",
        "versionNumber": 1,
        "action": "POST"
      }
    ],
    "defaultMetadata": {
      "meta": "data"
    }
}

2. Adding an Adaptor to YOUnite

Adding an adaptor merely informs the YOUnite Server that there is an intention to add an adaptor to a zone. Only users with POST /zones/<zone-uuid>/adaptors permissions can add an adaptor to a zone. By default only the Zone Data Steward (ZDS) has this permission.

An adaptor name is all that is required to POST an adaptor.

Adding an adaptor via the YOUnite UI:

image
Add Adaptor Popup

If the request succeeds, an adaptor UUID and security credentials are returned.

Note: When using the API to create the adaptor, you can set the capabilities and metadata in this step, or later in step #4.

POST /zones/<zone-uuid>/adaptors

{
    "name": "My First Adaptor",
    "description": "Our very first adaptor",
    "adaptorTypeUuid": "(optional - UUID of adaptor type created above)",
    "capabilities": [
      {
        "domainName": "customer",
        "versionNumber": 1,
        "action": "POST"
      }
    ],
    "metadata": {
      "meta": "data"
    }
}

3. Retrieve the Adaptor Credentials

The previous step which created the adaptor also created the unique UUID of the adaptor and credentials that are needed to connect to the Message Bus. These credentials will be needed when configuring the adaptor software.

In the YOUnite UI, the credentials can be retrieved by selecting the Adaptor Credentials button:

image

The credentials can also be retrieved after the adaptor is created by using the adaptor’s registration endpoint:

Example:

GET /zones/3c9000a9-3eb6-41fe-a11b-5a5859020c65/adaptors/3dfcc03d-e5d4-4d57-9e9b-5c5d2db32f9a/registration
{
    "uuid": "3dfcc03d-e5d4-4d57-9e9b-5c5d2db32f9a",
    "zoneUuid": "3c9000a9-3eb6-41fe-a11b-5a5859020c65",
    "clientId": "8c9167a6-bb83-4f77-bdfc-1947a946f77b",
    "clientSecret": "de02e3fa-4b23-46cb-aed6-5665a16e73d3",
    "messageBusUrl": "tcp://message-bus-url:61616"
}

See the API reference here for more information: YOUnite API Documentation

These credentials will be required to deploy the adaptor software in step #5. But before the adaptor can be started, it must have its capabilities and metadata configured.

4. Add Capabilities and Metadata to the Adaptor

If not included in the POST that created the adaptor, capabilities and metadata can be configured by updating the adaptor. In the YOUnite UI, these values can be modified by selecting the appropriate tab, editing the data and selecting save.

image

In the API, updating the adaptor will require a PUT or PATCH command, which, to prevent concurrent modifications, requires that the current state of the adaptor is first retrieved and the changeVersion is included in the request.

Warning
 PUT requires that the entire configuration for the adaptor is provided. Any properties omitted will overwrite the resource with a null value. Therefore, when updating metadata or capabilities only, the PATCH endpoint is recommended instead.

Retrieve Current Adaptor Configuration

GET /zones/<zone-uuid>/adaptors/<adaptor-uuid>

Notice the sample response below includes changeVersion. Copy this value and use in the PUT command in the next step.

{
    "uuid": "3dfcc03d-e5d4-4d57-9e9b-5c5d2db32f9a",
    "name": "My First Adaptor",
    "description": "Our very first adaptor",
    "adaptorType": {
        "uuid": "7f3cb295-45a5-4fca-bc5c-888e34adadc0",
        "name": "Default",
        "description": "Default Adaptor Type",
        "version": "1",
        "zoneUuid": "6c5a754b-6ce0-4871-8dec-d39e255eccc3",
        "dateCreated": 1586458238121,
        "lastUpdated": 1586458238121,
        "changeVersion": 1
    },
    "zoneUuid": "3c9000a9-3eb6-41fe-a11b-5a5859020c65",
    "state": "DOWN",
    "discoveryMode": false,
    "createdByUuid": "35c90d55-3b3f-4f00-9962-0ac57a4c4fda",
    "dateCreated": 1586463345429,
    "lastUpdated": 1586463386140,
    "changeVersion": 1605364736
}

Update Adaptor Configuration

Issue the following PATCH command with changeVersion retrieved in the previous step.

PATCH /zones/<zone-uuid>/adaptors/<adaptor-uuid>

{
    "changeVersion": 1605364736,
    "capabilities": [
      {
        "domainName": "customer",
        "versionNumber": 1,
        "action": "POST"
      }
    ],
    "metadata": {
      "meta": "data"
    }
}

5. Deploy and Start the Adaptor

Once the adaptor has been added and configured, the adaptor software can be configured with the credentials gathered in step #3 and launched.

It should successfully connect to YOUnite and move briefly into the STARTING state and then into the STARTED state unless otherwise configured. Once an adaptor is in the STARTED state, messages can be passed between the adaptor and the YOUnite Server.

Where to Next?

or, return to Adaptor Guides in the Knowledge Base.

Appendix A: The Infinite Update Issue

Adaptors are required to handle the changes they receive and apply from other adaptors. Failure to do so may result in detecting these changes as new updates and subsequently sending them back to YOUnite. This is called the Infinite update issue and more about it can be read here.

Appendix B: Adaptor Properties

On startup an adaptor may use the metadata and capabilities to configure itself, however, it does require some properties to get started so that it knows how to retrieve the metadata. The list below explains the minimum properties needed to start the Adaptor software.

Depending on how the Adaptor was written, these properties can come from a properties file (e.g. adaptor.properties) or environment variables. The YOUnite DB Adaptor accepts both methods.

Required properties:

Property Environment Variable Description Example Value

adaptor.uuid

ADAPTOR_UUID

UUID of the adaptor

3dfcc03d-e5d4-4d57-9e9b-5c5d2db32f9a

client.id

CLIENT_ID

Username used to connect to message bus

8c9167a6-bb83-4f77-bdfc-1947a946f77b

client.secret

CLIENT_SECRET

Password used to connect to message bus

de02e3fa-4b23-46cb-aed6-5665a16e73d3

message.bus.url

MESSAGE_BUS_URL

Message Bus URL

tcp://message-bus-uri:61617

auth.server.url

AUTH_SERVER_URL

OAuth Server to validate adaptor access credentials. The YOUnite Server runs an embedded OAuth server that your implementation may be using. By default it runs on port 8080 so, in this case the value would be http://ip-address-of-the-YOUnite-Server:8080

http://oauth-server-uri

Optional properties:

Property Environment Variable Description Default Value

hostname

HOSTNAME

Name of the host for the health check endpoint. Note that Kubernetes sets this environment variable by default.

localhost

port

PORT

HTTP Port to expose for health checks. The health check endpoint at /health, ie GET host:8001/health

8001

concurrency

CONCURRENCY

Concurrency for incoming data messages, ie, number of threads that will handle incoming messages. Note that this could lead to messages being processed out of order, though they are guaranteed to be processed in order based on their message grouping (the default is unique ID of the record, ie the DR UUID).

1

ops.concurrency

OPS_CONCURRENCY

Concurrency for incoming ops messages, ie, number of threads that will handle incoming messages. These are messages with adaptor status information and metadata, request for stats, shutdown requests. Expressed as a range, ie "1-5" or a single value indicating the maximum (with an implied minimum of 1).

1

assembly.concurrency

ASSEMBLY_CONCURRENCY

Concurrency for incoming assembly messages, ie, number of threads that will handle incoming messages. These are federated GET requests. Expressed as a range, ie "1-5" or a single value indicating the maximum (with an implied minimum of 1).

1

session.cache.size

SESSION_CACHE_SIZE

JMS session cache size.

10

enable.metrics

ENABLE_METRICS

Enable / disable displaying metrics at a given interval.

true

metrics.log.interval

METRIC_LOGS_INTERVAL

Interval in milliseconds to display metrics.

3600000 (one hour)

n/a

LOG_LEVEL

Log level for YOUnite logback messages. Applies to younite-db-adaptor implementations, but not the Adaptor SDK. Must come from an environment variable.

INFO

n/a

ROOT_LOG_LEVEL

Log level for non-YOUnite logback messages. Applies to younite-db-adaptor implementations, but not the Adaptor SDK. Must come from an environment variable.

INFO

Example:

# Configuration

# Adaptor UUID
adaptor.uuid = 3dfcc03d-e5d4-4d57-9e9b-5c5d2db32f9a

# ClientID and Secret to be used by JMS to connect to the message bus
client.id = 8c9167a6-bb83-4f77-bdfc-1947a946f77b
client.secret = de02e3fa-4b23-46cb-aed6-5665a16e73d3

# Message Bus URL
message.bus.url = tcp://192.2.200.25:61616

# Optional configuration
concurrency = 4
ops.concurrency = 1-4
assembly.concurrency = 2-2
session.cache.size = 10

Health Check Endpoint

The adaptor includes a HTTP endpoint to check the health of the adaptor at /health. The default port is 8001.

If the adaptor is healthy, it will return a 200 OK response when querying the endpoint. If it’s not healthy, it will return a 503 Service Unavailable. For example:

GET http://adaptor-host-name:8001/health

{
  "status": "UP"
}

In addition, it will return some information about the adaptor such as the status of each service and the uptime. For example, if the employee v1 domain handler is not working, the adaptor might return something like this:

GET http://adaptor-host-name:8001/health

{
  "status": "DOWN",
  "adaptorService": "UP",
  "upTimeMilliseconds": 100000,
  "customer:1": "UP",
  "employee:1": "DOWN"
}

Appendix C: Null Values vs Missing Values

Incoming data events from YOUnite include a JSON string with the data of the data record. For several reasons, this may or may not include all fields in the domain. Therefore, it should not be assumed that the value of a field is null if it is not present; nulls will be explicitly specific in the JSON string if that is their value.

Reasons why a field may be excluded:

  1. The adaptor that originated the data event did not include data for a field.

  2. Data Governance was applied and a field was excluded.

Appendix D: Adaptor States

Once an adaptor is added (step #2, above) it is in the DOWN state. Once the adaptor software has connected it will move briefly into the STARTING state, then STARTED or PAUSED.

If an error occurs it will move into an ERROR state and include a message indicating what the error is.

Summary of adaptor states:

adaptor state diagram

The adaptor moves between states as follows:

State Description

DOWN

Adaptor has been created but has not yet started.

DOWN_PAUSED

Adaptor has been created but has not yet started. When it does start, it is instructed to start in a PAUSED state.

STARTING

The adaptor has

  • Connected successfully to the YOUnite Server via the Message Bus and subscribed to the proper queues

  • Sent a message to the YOUnite Server indicating that it is ready to receive capabilities and metadata and start running.

STARTED

The adaptor has started and is able to send and receive data events.

PAUSING

The YOUnite Server has asked the adaptor to pause itself and is awaiting a response from the adaptor.

PAUSED

The adaptor has moved into a paused state and cannot send or receive data events.

An adaptor state can be changed using the YOUnite UI. The YOUnite UI uses the pause/play semantics used on common everyday devices and user audio/video interfaces.

If adaptor is in the STARTED state, the UI will display a pause button:

adaptor pause button

If adaptor is in the PAUSED state, the UI will display a play button:

adaptor play button

Or the adaptor state can be changed via the YOUnite API with the PATCH request:

PATCH /zones/<zone-uuid>/adaptors/<adaptor-uuid>

Request body to pause:
{
    "state": "PAUSE"
}

Request body to resume:
{
    "state": "PLAY"
}

Request body to ask the adaptor to send its current state to YOUnite:
{
    "state": "REFRESH_STATE"
}

Appendix E: Adaptor Management

See the YOUnite Server API documentation (https://younite.us/api) for more requests that can be made to the /zones/<zone-uuid>/adaptors endpoint.

A few common requests include:

Delete an Adaptor

This can be done in the YOUnite UI by selecting the vertical selection icon (dots) and selecting the trash can icon next to the adaptor:

delete adaptor trash can

Additionally, an adaptor can be deleted by making the following API request:

DELETE /zones/<zone-uuid>/adaptors/<adaptor-uuid> (soft delete, adaptor is set to inactive)
DELETE /zones/<zone-uuid>/adaptors/<adaptor-uuid>?force=true (hard delete, adaptor is permanently deleted)
Note
 When an adaptor is "soft deleted" (meaning the adaptor’s "active" setting will be set to false), all its information including linkage information is preserved. It may be re-activated by a PATCH request with { "active": true } in the body.

Health and Metrics API Requests

GET /zones/<zone-uuid>/adaptors/<adaptor-uuid>/health

{
    "adaptorUuid": "82ae9eec-7a8a-4ec2-9fde-2e836fea3b21",
    "requestId": "d5a1808d-262c-40f6-9208-67a282f55176",
    "status": "UP",
    "uptime": 73754094
}

GET /zones/<zone-uuid>/adaptors/<adaptor-uuid>/metrics

{
    "adaptorUuid": "cda06e25-7028-4495-a463-83ff1a3f940e",
    "requestId": "37c86788-dd47-4e02-84d4-03e8457ba7be",
    "adaptorMetricsDetail": [
        {
            "type": "SENT",
            "period": "ALL_TIME",
            "dateCount": [
                {
                    "date": 1649108968945,
                    "count": 1
                }
            ]
        },
        {
            "type": "RECEIVED",
            "period": "ALL_TIME",
            "dateCount": [
                {
                    "date": 1649108968945,
                    "count": 2010
                }
            ]
        },
        {
            "type": "SENT",
            "period": "DAY",
            "dateCount": [
                {
                    "date": 1649030400000,
                    "count": 1
                }
            ]
        },
        {
            "type": "SENT",
            "period": "HOUR",
            "dateCount": [
                {
                    "date": 1649106000000,
                    "count": 1
                }
            ]
        },
        {
            "type": "SENT",
            "period": "MINUTE",
            "dateCount": [
                {
                    "date": 1649108580000,
                    "count": 0
                },
                {
                    "date": 1649108640000,
                    "count": 0
                },
                {
                    "date": 1649108700000,
                    "count": 0
                },
                {
                    "date": 1649108760000,
                    "count": 0
                },
                {
                    "date": 1649108820000,
                    "count": 1
                },
                {
                    "date": 1649108880000,
                    "count": 0
                },
                {
                    "date": 1649108940000,
                    "count": 0
                }
            ]
        },
        {
            "type": "RECEIVED",
            "period": "DAY",
            "dateCount": [
                {
                    "date": 1649030400000,
                    "count": 2010
                }
            ]
        },
        {
            "type": "RECEIVED",
            "period": "HOUR",
            "dateCount": [
                {
                    "date": 1649106000000,
                    "count": 2010
                }
            ]
        },
        {
            "type": "RECEIVED",
            "period": "MINUTE",
            "dateCount": [
                {
                    "date": 1649108580000,
                    "count": 0
                },
                {
                    "date": 1649108640000,
                    "count": 0
                },
                {
                    "date": 1649108700000,
                    "count": 160
                },
                {
                    "date": 1649108760000,
                    "count": 441
                },
                {
                    "date": 1649108820000,
                    "count": 872
                },
                {
                    "date": 1649108880000,
                    "count": 537
                },
                {
                    "date": 1649108940000,
                    "count": 0
                }
            ]
        }
    ]
}

Shutting Down an Adaptor

An adaptor may be remotely shut down via a two step process:

GET /zones/<zone-uuid>/adaptors/<adaptor-uuid>/shutdown-code

{
    "adaptorUuid": "82ae9eec-7a8a-4ec2-9fde-2e836fea3b21",
    "code": "45b65591-96b8-49c9-afcc-9dcb22805d88"
}

PUT /zones/<zone-uuid>/adaptors/<adaptor-uuid>/shutdown?code=<code>

{
    "adaptorUuid": "82ae9eec-7a8a-4ec2-9fde-2e836fea3b21",
    "requestId": "a1d6c9c3-3349-4f6e-bf2a-f051debf6a94",
    "acknowledged": true
}

Appendix F: Finding Data Errors Logged by an Adaptor

Access the logs for the YOUnite data fabric. See Logging.

To search the logs for data errors:

log-entry-type: "ADAPTOR_ERROR"
adaptorErrorLogEntry

To limit results to a single adaptor use the adaptor’s UUID with the source-adaptor-uuid property e.g.:

log-entry-type: "ADAPTOR_ERROR" AND source-adaptor-uuid: "8862fdc0-43c9-4ec6-b5b0-a14ec3bdd352"