Adaptor

An Adaptor is software that is located within a system that is participating in the sharing of data through the YOUnite Data Fabric Platform and acts as the connection point between that system and the YOUnite Server which is the core service inside the YOUnite Data Fabric Platform. The adaptor focuses on ETL (Extract, Transform, and Load) functions, ensuring the outbound data from that system meets the format requirements of the data fabric and transforming the inbound data from the YOUnite Server into what any other system requires. An adaptor may have additional business logic such as filtering for specific data from the YOUnite Server.

Note
 This document gives background on adaptors including a section on the advanced topic of the Infinite Update issue which is intended for adaptor developers. That section can be skipped for those using out-of-the-box adaptors.

Introduction

Adaptors perform data operations (CRUD) on data records that conform to one or more data domains (for more information on data domains see the Data Domains page). The data operations an adaptor can perform are referred to as its "Capabilities" and are defined in an Adaptor Capabilities List.

The envelope in the diagram below represents a JSON data object being passed between the adaptors and the YOUnite Server. The data object could have originated in one of several ways. The most common are:

  • An adaptor detecting a change in a source system and sending it to the YOUnite Server

  • The YOUnite Server receiving a change from an adaptor and routing it out to other adaptors and API consumers that have subscribed to receive notifications for that specific change

  • The YOUnite Server querying the adaptor for information such as health, status, metrics, etc

  • The YOUnite Data Virtualization Service querying the adaptor for data to perform data record assembly (also known as Federated GET)

Adaptor Topology

Figure: Adaptor Topology

Adaptors and Federated Data Records

In order to provide data management, the YOUnite Server needs to know where the Federated data domain’s data and properties live. Federated Data Records (DRs) are stored in the source systems instead of centrally in the YOUnite Server.

The YOUnite Server contains a reference (links) to the federated domain data so it can retrieve and update the data. In order for this to happen the source entities (records) in the source systems need to be mapped to the Federated Data Domain Records in the YOUnite Server. This mapping can happen standard data linking or through data discovery creating a list of links to all source system for each federated data record. 

The mapping process includes creating YOUnite data records for each data domain entity. YOUnite will then retain the uniqueness rules for that data record.

See Linking YOUnite Data Records to Entities at the Source Systems for details.

Adaptors Can Be Either Uni-Directional or Bi-Directional

Uni-Directional - Inbound Only

image

An inbound uni-directional adaptor connects the YOUnite Data Fabric Platform to a source system in the following way:

  • Accepts data events from the YOUnite Server

  • Transforms data events from the data domain format to the source system’s physical format

  • Is capable of performing one or more CRUD event on source entities in the source system

Uni-Directional - Outbound Only

image

An outbound uni-directional adaptor connects the YOUnite Data Fabric Platform to a source system in the following way:

  • Detects change events to source entities in the source system

  • Transforms data change events from the source system’s physical format to the data domain format

  • Sends the change data event to the YOUnite Server for routing to other adaptors based on applicable governance ACLs.

Bi-Directional - Inbound and Outbound

image

A bi-directional adaptor performs all the functions of both an uni-directional inbound and outbound adaptor.

When developing a bi-directional adaptor, its imperative that the developer understand the "Infinite Update Issue" discussed below.

The Infinite Update Issue

The Importance of Strong Eventing in Source Systems and "Incoming Change Detection Cache" With Bi-Directional Adaptors

Infinite updates can occur when bi-directional adaptors are required for source systems that have less than optimal change event notification APIs (eventing).

A bi-directional adaptor can be thought of as having a right and left brain where each side isn’t always aware of all the goings-on in the other side.

With a bi-directional adaptor, one side of the adaptor accepts data events from the YOUnite Server and applies them to the source system (YOUnite side of the adaptor) while the other side waits to be notified of change events from the source system and sends the data events to the YOUnite Server (source system side). Optimally, the source system has robust APIs that allow the adaptor developer to listen and be notified of specific change events inside of the source system (i.e. robust eventing).

However, if the source system has weak eventing, then the adaptor developer needs to poll for changes instead of being notified. When the YOUnite side of the adaptor receives a data event from YOUnite and writes it to the source system, the source-system side detects the change but has no way of knowing if this was a change initiated by YOUnite or by the source system itself (e.g. a new user added to the customer DB via its normal business logic).

In the case of a source system with week eventing, as changes are received from YOUnite, they need to be stored in an Incoming Change Detection Cache. When a change is detected, the adaptor can look in the cache to see if it originated from YOUnite, and if it did, it can then be ignored. The entry is then removed from the change detection cache. Note that the YOUnite Off-the-Shelf DB Adaptor and other off the shelf adaptors have an Incoming Change Detection Cache built in and, but custom adaptors written using the SDK may need to implement their own.

When an adaptor propagates changes made by itself then an infinite update loop in YOUnite can occur and create "governance chaos" as shown in the following scenarios:

  • All adaptors are bi-directional

  • Governance is set so that:

    • adaptorA shares with adaptorB, but not adaptorC

    • adaptorB shares with both adaptorA and adaptorC

  • A source entity attached to adaptorA has a change to its data record, adaptorA detects it and sends it to the YOUnite server.

  • According to adaptorA’s governance rules, the change is routed to adaptorB, but not adaptorC

  • AdaptorB applies the change to the appropriate source entity in the source system attached to adaptorB

  • Because AdaptorB does not have the change detection logic it has no way of knowing that the adaptor itself wrote the change; consequently, adaptorB will push the change it just wrote up to YOUnite as a new data event

  • AdaptorB’s governance allows it to send changes to adaptorA and adaptorC

  • So even though the change originated at adaptorA, adaptorC received the change – a violation of the intended governance.

  • Worse, adaptorA receives its own change if it too doesn’t have the change detection logic and changes will continue infinitely.

Adaptor Design and Configuration Pitfalls

Having two uni-directional adaptors (one inbound only and the other outbound only) should be avoided unless the source system has strong eventing that allows the adaptor to filter out changed initiated by itself.  Otherwise it will be difficult to avoid the "Infinite Update Issue" without adding special caches to implement the change detection logic described above.

image

Selecting Enterprise Applications and Services That Can Meet Your Organization’s Governance Requirements

Developing adaptors can be the most difficult task when implementing YOUnite. While it is unlikely an organization will swap out an existing enterprise application to meet its governance needs (although, it may be required in some circumstances), an organization can require new applications and vendors to meet certain data event access, monitoring and data governance requirements.

To meet the needs of event driven discovery, cataloging, synchronization and governance, new applications and solutions must have robust APIs that allow:

  • Notification for source entity updates that map to the organization’s data domain definitions. (Read carefully "The Infinite Update Issue" above especially regarding the need for robust eventing.)  Adaptors should not have to scan for changes but should be able to register for change notifications for entities that map to the organization’s data domain definitions. Note that YOUnite provides off-the-shelf adaptors that can both listen for notifications as well as scan for changes.

  • Eventing and source entity CRUD operations should be at the data domain property level. This is a critical distinction because if the API only allows change detection and source entity updates at the source record level then governance can only be applied to the source entities in the system at the data record level. Ideally, an organization will want to detect changes and perform CRUD operations at the domain property level. This is the difference between allowing a customer record from being shared or restricted (record level) versus being able to restrict individual properties (property level) such as tax ID numbers, phone numbers, etc.

  • Source entity CRUD operations:

    • Create (POST)

    • Update (PUT)

    • Delete (DELETE)

    • Read (GET)

Return to Adaptor Guides in the Knowledge Base.