Overview

Data issues are issues that generally apply to adaptors and federated data that may or may not require resolution. Typically, whomever manages these issues will review them, resolve them (if necessary) and then close them. This can all be done in the YOUnite UI. This documentation describes the API calls and YOUnite UI pages to manage data issues.

Data event exceptions are automatically detected as data flows through YOUnite and filed as data issues. See Data Issues & Event Exceptions Overview.

Important
 Data exceptions are detected by YOUnite off-the-shelf adaptors, while third-party adaptors may not support the detection of data event exceptions.

Types of Data Issues

DATA_EVENT_EXCEPTION is the only type of data issue.

Code Description

DATA_EVENT_EXCEPTION

Issue relating to a data event. Automatically generated as data events flow through YOUnite. See Data Issues & Event Exceptions Overview.

Data Issues & Event Exceptions Overview

Before reading, see Matching Algorithms for information about matching algorithms.

An "exception" occurs when a data event cannot be processed. There are a few reasons this can happen:

  1. The data event could not be parsed according to the domain version’s schema requirements. These are "parsing" exceptions.

  2. The data event resulted in an "ambiguous" match status. These are "matching" exceptions.

  3. An adaptor reported an exception that prevented it from sending a data event to YOUnite. These are "adaptor" exceptions.

  4. A batch of updates or inserts have identical timestamps and had to be skipped (this is only for off-the-shelf database adaptors configured with the TIMESTAMP scanMethod).

  5. Some other service sends a message indicating there’s a data event exception. These are "other" exceptions.

When the exception is detected, an issue is created that will have one of the following Error types:

PARSING EXCEPTION

Reasons why a data event might not meet the schema’s requirements:

  1. A required property is missing

  2. A string property does not meet the minimum or maximum length requirement

  3. A numeric value is not within the allowable bounds

  4. A property is of the wrong data type (ie numeric was expected but found a string instead)

  5. etc…​

MATCHING EXCEPTION

Reasons why a data event might result in an "ambiguous" match status:

  1. The data event matched to more than one Data Record in the YOUnite Data Catalog.

  2. The data event matched to a Data Record with a score in the "possible" but not "definite" range.

  3. The data event matched to a Data Record that already exists for the adaptor.

ADAPTOR EXCEPTION

The off-the-shelf database adaptor reports transformation errors to YOUnite. These errors occur when transformation steps fail. Examples of why a transformation step might fail:

  1. A transformation step encountered an error or bug in the code

  2. A REST call to an external service failed (such as a data quality service)

  3. A transformation step purposefully set an error condition, for example, if the data failed to meet some data quality standard

ADAPTOR OVERFLOW EXCEPTION

The off-the-shelf database adaptor can be configured with timestamp change detection scanning. This scanning method is very fast however, it should only be used on databases that do traditional application insert and updates and not large batch operations.

The following are reasons why a data event might generate an adaptor overflow exception:

maxBatchSize was configured to low

  1. Timestamp scanning precision is set to high (e.g. SECONDS or MINUTES). Precision should be set to the database’s precision which is typically NANOSECONDS, MICROSECONDS or MILLISECONDS.

  2. A single bulk SQL COPY command with more rows than maxBatchSize.

  3. A single bulk insert of update with the timestamp specified and, the size of the update is larger than maxBatchSize.

  4. If a MERGE or UPSERT is larger than maxBatchSize. Some databases have a MERGE or UPSERT that will give all the affected records the same timestamp.

See Adaptor Overflow Exceptions and Recovery below for more details.

OTHER EXCEPTION

Other services could also report data event exceptions to YOUnite.

Handling Data Event Issues

When a data event exception occurs, an issue is generated and it must be resolved manually.

The first step to resolving the issue is to retrieve the details about the exception such as possible matches that were identified, parsing errors and/or other error messages.

Parsing Exceptions

Parsing exceptions occur when the data in the data event cannot be parsed either due to schema restrictions or data type problems.

For parsing exceptions, there are three likely outcomes:

  1. The schema needs to be adjusted. Adjust the schema, then re-process the data event.

  2. The source data needs to be adjusted. Adjusting the source data, will trigger a new data event.

  3. The exception can be ignored. Manually set the data issue status to CLOSED or optionally delete the data issue.

Matching/Ambiguous Exceptions

Matching exceptions occur when the data event leads to an ambiguous match.

For matching exceptions, there are three likely outcomes:

  1. One of the possible matches is correct. Accept it and re-process the data event.

  2. None of the possible matches are correct. Re-process the data event as a new Data Record.

  3. The data event can be matched to an existing Data Record even if it’s not in the list of possible matches.

  4. The exception should not be processed. Close the data issue.

Once an exception is "resolved", a message is sent to the originating adaptor asking it to re-perform the data event so that it can now be processed. Because the data event has been "resolved" it will not lead to a matching exception the second time around.

Note
 Matching Data issues cannot be deleted but must be resolved.

Adaptor Exceptions

Adaptor exceptions occur when an adaptor sends a message to YOUnite saying that it had a problem when attempting to send a data event. The likely outcomes are:

  1. The source data needs to be adjusted. Adjust the source data, then perform a "touch event" on the source data to trigger" a new data event.

  2. The adaptor’s configuration needs to be adjusted. Update the configuration, then re-process the data event, if necessary.

  3. Resolve the issue by closing or deleting the issue.

Adaptor Overflow Exceptions and Recovery

Database applications typically use the INSERT and UPDATE statements to create/update records in a database table and each INSERT and UPDATE typically gets a unique timestamp or at most a few INSERTs may have the same timestamp (but not more than a reasonably set maxBatchSize).

Adaptor overflow exceptions can occur with YOUnite DB adaptors that use the timestamp scan method when scanning for updates and/or new inserts into a data source.

An overflow occurs when a single batch of inserts or updates into the data source monitored by the YOUnite adaptor is larger than the maxBatchSize at the specified timestamp precision.

The timestamp scan method, maximum batch size and precision are specified in the adaptor configuration:

  .
  .
  "scanMethod": "TIMESTAMP",
  "scanOptions": {
    "maxBatchSize": 10000,
     .
     .
     .
  }
  "timestampSpec": {
    "type": "TIMESTAMP",
    "precision": "MICROSECONDS",
     .
     .
     .
  }

Using the adaptor configuration settings above, a single insert or update with the identical timestamp (to the microsecond) would have to larger than 10,000 rows to create an adaptor overflow exception.

If the above configuration settings are configured to reasonable values, an adaptor overflow is an extremely unlikely event especially for SQL databases.

The only known way of creating an adaptor overflow with an SQL DB configured with reasonable values:

  • A single SQL COPY command with a CSV file with more rows than maxBatchSize:

COPY  your_table(your_table_rows) FROM 'YOUR_LARGE_SQL_FILE' WITH (YOUR_CSV_FORMAT)

  • A bulk UPDATE or INSERT larger than maxBatchSize were an exact timestamp value is specified e.g.:

UPDATE your_table SET timestamp_column = CURRENT_TIMESTAMP WHERE condition;

One approach to recovering from a timestamp overflow is to break the larger batch into two or more smaller batches of less than maxBatchSize and executing a "touch operation" on each of the smaller batches solely to refresh the last updated timestamp. This only works if it is OK for these changes to be routed to other systems. This is not acceptable if these changes could potentially overwrite other changes made after the time of the adaptor overflow.

Contact your YOUnite SI or partner for additional methods to address the exception

Handling Data Issues in the YOUnite UI

If there are data issues for the zone and the zone user has permission to view issues, the data issue icon in the UI header will be badged with the data issue count:

Header w/exception icon and badge count set to 1

Navigate to the Data Issues page from the main Dashboard or by selecting the data issue icon.

Navigate to the data issue’s Resolve page by selecting the data issue from the list:

Selecting a data issue on Data Issues page

Details of the issue and possible resolutions are displayed.

ui dataIssues detials

More details on handling data issues is covered above.

Parsing and Adaptor Exceptions

On the exception’s page look in the Exception Error Message window. An example error message:

ui dataIssues parsing

Resolving the Issue

Get the Adaptor and Domain version

Using the above exception error message, the following shows that a row was added to the data source table or collection that had a missing total value. If this were an adaptor issue, the value would be invalid or mis-formatted. To find the database, table and column look at the adaptor’s configuration metadata:

ui dataIssues parsingDetails

For this example, it would be the in the configuration metadata for the Europe West Database - SQL Server adaptor’s purchaseOrder:1 definition for the off-the-shelf DB adaptor.

Find the Database, Table and Column Name

  1. Navigate to the Adaptors page

  2. Select the Adaptor

  3. Select the Configuration Metadata tab

  4. Search for the domain name (for example, purchaseOrder) until you locate the dataSource object, where the definition of tableName is located — then scroll down and look in the dataColumns for the propertyName (e.g. total). In this case the domain:version property name and the database column name (source) are both named total.

    ui dataIssues parsingAdaptorSearch

  5. To locate the database, search within the databases object for the value of dataSource.name (for example, database)

    ui dataIssues parsingAdaptorSearchDB

  6. Adaptor Issues Only: To find the source of the issue, consideration may also need to be given to the inboundTransformations and/or outboundTransformations within the adaptor’s configuration metadata for the dataSource:

    ui dataIssues adaptorTransformations

  7. To fix the issue, edit the database row/column with the issue by fixing the offending value. This will automatically trigger a new data event.

  8. To resolve the issue in YOUnite:

    1. Go to the Data Issues page and select the data issue

    2. From the Possible Resolutions pulldown, select Close Data Exception and enter a resolution message.

Matching Exceptions

A MATCHING error occurs when:

  1. a new record is detected for a given version of data domain

  2. The data domain version has custom rules or scoring based algorithms to match new records to existing records

  3. An ambiguous match occurs:

    1. For Rules Based matching, a match is considered ambiguous if the criteria matched to more than one Data Record.

    2. For Score Based matching, a match is considered ambiguous if a match was given a score in the "possible" range, or if there was a "definite" match two or more Data Records.

For more see:

Bottom Half of the Screen

The left side presents the DR key properties of the records, while the right side changes to display content corresponding to one of the three selected tabs.

Exception Error Message Tab

Shows the data exception error message.

Matching Algorithm Tab

Shows the matching algorithm for the domain:version.

Local Identifier Tab

This displays the data record’s local identifier at the data source if the off-the-shelf adaptor was configured with a local identifier, i.e. the id used by the data source to uniquely identify records.

Possible Resolutions

Select the Possible Resolutions pulldown to resolve the issue:

Resolve to Potential Match

The Potential match page displays the INCOMING DATA RECORD and potential matches with similar DR keys stored in other adaptors. The Score row shows the matching score. The potential matches with the highest scores are the candidates however, this feature allows the data steward/analyst to make the ultimate decision.

ui dataIssues resolvePotentialMatch

Selecting the appropriate MATCH button will match the incoming data record with the data record at the selected adaptor and resolve the issue.

Match to Other Record

The Match to Other Record page displays the Data Access page the domain version the incoming data record belongs to. From this page you can search for a data record that isn’t part of the Potential Match list and match the incoming record to it:

Match to Other Record’s Data Access Page

For more on search filters see the Lucene Query Examples on the Accessing Data Records page.

In the above example, the search for lastname starting with "A" and employeeId starting with 1E7177 returned one result. See TODO for search for data records.

The data can be retrieved by selecting the disk icon.

The record can be matched using the thumbs up icon.

This page doesn’t have all the features that the Data Access page, so it may be easier to visit the Data Access to search for the match. Once the match has been found, then copy its UUID (DR UUID) to the clipboard and revisit this page and search for the data record using the DR UUID. You can search for a data record by copying its DR UUID into the search bar.

Search bar with UUID and no search tag

To complete the match, select the "Thumbs up" icon and enter a resolution message.

Treat as New Data Record

Navigate to the Data Access page and search existing records for the domain:version for potential matches. If none are found, and this record is a new record then:

  1. Navigate to the Data Issues page and select the data issue

  2. Select Treat as a New Data Record from the Possible Resultion pulldown

  3. Enter a resolution message

Close Data Exception

Select Close Data Exception and enter a resolution message once the exception has been addressed.

Adaptor Overflow Exceptions

  1. Navigate to the issue’s Resolve page

  2. Check the Exception Error Message

  3. Note the adaptor where the exception has occurred

  4. See the following on Adaptor Overflow issues:

    1. Background on matching exceptions

    2. Adaptor overflow exceptions and recovery

  5. To resolve the issue in YOUnite:

    1. Go to the Data Issues page and select the data issue

    2. From the Possible Resolutions pulldown, select Close Data Exception and enter a resolution message.

Deleting Data Issues

Data issues can be deleted. This permanently deletes the data issue and it cannot be recovered. To delete an issue in the YOUnite UI, select its trash can icon:

ui dataIssues deleteIssue
Note
 Matching/Ambiguous Data issues cannot be deleted but must be resolved.

Data Issue APIs

For more information about the YOUnite APIs, see the API reference here: YOUnite API Documentation

For more information about additional APIs for Data Event Exceptions see the rest of this document.

Data Issue API

For more information about the YOUnite APIs, see the API reference here: YOUnite API Documentation

Retrieve a page of OPEN data issues for a zone. Sort by lastUpdated in descending order:

GET /zones/<zone-uuid>/issues?filters=type:DATA_EVENT_EXCEPTION,status:OPEN&sort=-lastUpdated&page=0&size=10
{
    "dataIssues": [
        {
            "uuid": "df3c8f08-5d8a-4801-bfed-98566792ccd1",
            "type": "DATA_EVENT_EXCEPTION",
            "title": "This is a title",
            "notes": "These are some notes",
            "zoneUuid": "0ee6432b-ba62-4443-aad6-827343ede23d",
            "issueStatus": "OPEN",
            "issuePriority": "HIGH",
            "dateCreated": 1632256731559,
            "lastUpdated": 1632256731559
        }
    ],
    "_links": {
        "first": "http://local.younite.org:8080/api/zones/0ee6432b-ba62-4443-aad6-827343ede23d/issues?page=0&size=10",
        "last": "http://local.younite.org:8080/api/zones/0ee6432b-ba62-4443-aad6-827343ede23d/issues?page=0&size=10",
        "self": "http://local.younite.org:8080/api/zones/0ee6432b-ba62-4443-aad6-827343ede23d/issues?page=0&size=10"
    },
    "_page": {
        "page": 0,
        "size": 10,
        "totalElements": 1,
        "totalPages": 1
    }
}
Note
 The type filter can be omitted since there is only the DATA_EVENT_EXCEPTION issue type.

Get a data issue:

GET /zones/<zone-uuid>/issues/<issue-uuid>
{
    "uuid": "df3c8f08-5d8a-4801-bfed-98566792ccd1",
    "type": "DATA_EVENT_EXCEPTION",
    "title": "This is a title",
    "notes": "These are some notes",
    "zoneUuid": "0ee6432b-ba62-4443-aad6-827343ede23d",
    "issueStatus": "OPEN",
    "issuePriority": "HIGH",
    "dateCreated": 1632256731559,
    "lastUpdated": 1632256731559
}

Change the priority of a data issue:

PATCH /zones/<zone-uuid>/issues/<issue-uuid>
{
    "priority": "enter new priority here"
}

Change the status of a data issue (OPEN or CLOSED):

PATCH /zones/<zone-uuid>/issues/<issue-uuid>
{
    "issueStatus": "OPEN or CLOSED",
    "resolutionMessage": "Whe closing an issue, enter the resolution here"
}

Retrieving a List of Exceptions

There are several ways to retrieve a list of exceptions, depending on your permissions:

GET /zones/<zone-uuid>/issues
GET /zones/<zone-uuid>/adaptors/<adaptor-uuid>/issues
GET /zones/<zone-uuid>/adaptors/<adaptor-uuid>/domains/<domain-uuid>/versions/<domain-version-uuid>/issues
Note
 If no filter is supplied, all exceptions are returned, even those that with a status of CLOSED. Therefore it is recommended to use one of the filters options to choose what types of records to retrieve. For example, GET /zones/<zone-uuid>/issues?filters=status:OPEN.

The following query parameters can be applied to these requests:

Query Parameter Description Example

page

Zero-based page number

GET …​/issues?page=0&size=10

size

Page size

GET …​/issues?page=0&size=10

filters

Filters (see below for options)

GET …​/issues?filters=type:DATA_EVENT_EXCEPTION,status:OPEN

GET …​/issues?filters=status:OPEN,exceptions:MATCHING

GET …​/issues?filters=status:OPEN,exceptions:PARSING

GET …​/issues?filters=status:OPEN,match-status:AMBIGUOUS:PRESUMED_NEW:PRESUMED_MATCH,domain-version:5a5bc628-ee6f-4df3-8aca-5510bfb8d70f,adaptor:c38b5025-52d6-47c3-9a95-9ad228dd4e6b

sort

Sort (see options below)

GET …​/issues?sort=adaptor.name,-lastUpdated

Filter Options

Note
 YOUnite UI users omit …​/issues?filters= when filtering data issues. For example, to filter all PARSING data issues that are CLOSED use exceptions:PARSING,status:CLOSED

Filters are in the format <option>:<value> and are separated by commas. For example: GET …​/issues?filters=status:OPEN,match-status:AMBIGUOUS,domain-version:5a5bc628-ee6f-4df3-8aca-5510bfb8d70f

Filter options that allow multiple values use a colon (:) to separate multiple values, ie GET …​/issues?filters=status:OPEN,match-status:AMBIGUOUS:PRESUMED_NEW:PRESUMED_MATCH.

Option Description Possible values Example

type

Can be omitted. Specifies the type of data issue wich is always of type DATA_EVENT_EXCEPTION.

DATA_EVENT_EXCEPTION

filters=type:DATA_EVENT_EXCEPTION

status

Status of the exception

OPEN or CLOSED

filters=status:OPEN

exceptions

Type(s) of exceptions to show

PARSING, MATCHING, ADAPTOR or OTHER

filters=exceptions:PARSING:MATCHING

match-status

Match Status(es) of the associated data record

RESOLVED, AMBIGUOUS, PRESUMED_NEW, and/or PRESUMED_MATCH

filters=match-status:AMBIGUOUS:PRESUMED_NEW:PRESUMED_MATCH

adaptor

Adaptor filter (not applicable when adaptor specified in URL)

Any adaptor UUID in the zone

filters=adaptor:c38b5025-52d6-47c3-9a95-9ad228dd4e6b

domain-version

Domain version filter (not applicable when domain version specified in URL)

Any domain version UUID

filters=domain-version:5a5bc628-ee6f-4df3-8aca-5510bfb8d70f

Sort options:

Any field in the response can be used to sort. Use the minus sign (-) before the field name to sort in descending order. Use dot notation to indicate nested fields. For example GET …​/issues?sort=adaptor.name,-lastUpdated would sort by the name of the adaptor followed by the last time updated in descending order.

Retrieving a Single Exception

If the UUID of the exception is already known, it can be retrieved in a number of ways, depending on your permissions:

GET /zones/<zone-uuid>/issues/<issue-uuid>
GET /zones/<zone-uuid>/adaptors/<adaptor-uuid>/issues/<issue-uuid>
GET /zones/<zone-uuid>/adaptors/<adaptor-uuid>/domains/<domain-uuid>/versions/<domain-version-uuid>/issues/<issue-uuid>

The following query parameters can be applied to these requests:

Option Description Possible values Example

rematch

Re-do matching before returning the result. The default value is false.

true / false

…​/issues/36484080-4698-4f00-899a-63b63c051d75?rematch=true

resolve

When re-matching, resolve if possible as long as the result is not ambiguous. The default value is false.

true / false

…​/issues/36484080-4698-4f00-899a-63b63c051d75?rematch=true&resolve=true

Re-matching A Data Event

When rematch=true in the above request, the previous matching information is cleared and matching is performed again.

This is useful in the rare instance that the domain version’s matching algorithm has been modified and re-matching will potentially resolve the ambiguity.

When rematching, there are different possible outcomes depending on the value of resolve.

If resolve=true, the event will get a status of either RESOLVED or AMBIGUOUS depending on whether a positive match can be made. If the exception is RESOLVED, the data event will automatically be re-processed.

If resolve=false (the default), the event will get a status of PRESUMED_NEW, PRESUMED_MATCH or AMBIGUOUS. PRESUMED_NEW indicates that based on the matching information, this record is presumed to be a new Data Record, but it will still need to be manually resolved. PRESUMED_MATCH indicates that based on the matching information, this record is presumed to have a positive match, but it will still need to be manually resolved.

Resolving a Matching Exception

Once the correct resolution has been identified, resolution can be performed in a number of ways, depending on your permissions:

POST /zones/<zone-uuid>/issues/<issue-uuid>/resolve
POST /zones/<zone-uuid>/adaptors/<adaptor-uuid>/issues/<issue-uuid>/resolve
POST /zones/<zone-uuid>/adaptors/<adaptor-uuid>/domains/<domain-uuid>/versions/<domain-version-uuid>/issues/<issue-uuid>/resolve

The following query parameters can be applied to these requests:

Option Description Possible values Example

dr-uuid

The UUID of the Data Record to resolve to.

A DR UUID

…​/issues/36484080-4698-4f00-899a-63b63c051d75/resolve?dr-uuid=c1e3146f-c1d7-449a-9860-eed71e1372da

new

True if resolving to a new Data Record. The default is false.

true / false

…​/issues/36484080-4698-4f00-899a-63b63c051d75/resolve?new=true

force

True to force resolving to a Data Record not in the list of possible matches. The default is false.

true / false

…​/issues/36484080-4698-4f00-899a-63b63c051d75/resolve?dr-uuid=c1e3146f-c1d7-449a-9860-eed71e1372da&force=true

Resolving a parsing exception

Parsing exceptions typically must be resolved at the source system. This may automatically trigger the adaptor to send a new data event. If this new data event succeeds the exception can be manually resolved by updating its status. See Updating the status of an exception.

Updating the status of a data event exception

The status of a data event may be set to OPEN or CLOSED with the following endpoints:

PATCH /zones/<zone-uuid>/issues/<issue-uuid>
PATCH /zones/<zone-uuid>/adaptors/<adaptor-uuid>/issues/<issue-uuid>
PATCH /zones/<zone-uuid>/adaptors/<adaptor-uuid>/domains/<domain-uuid>/versions/<domain-version-uuid>/issues/<issue-uuid>

The body of the request should include issueStatus and if the new status is CLOSED, resolutionMessage as well, like so:

{
    "issueStatus": "CLOSED",
    "resolutionMessage": "Manually resolved at source system"
}

Deleting an erroneous or unprocessable data event

If after reviewing the exception it is decided that the data event should be removed from YOUnite, it can be deleted via one of the following endpoints:

PATCH /zones/<zone-uuid>/issues/<issue-uuid>
PATCH /zones/<zone-uuid>/adaptors/<adaptor-uuid>/issues/<issue-uuid>
PATCH /zones/<zone-uuid>/adaptors/<adaptor-uuid>/domains/<domain-uuid>/versions/<domain-version-uuid>/issues/<issue-uuid>

The body of the request must include issueStatus = CLOSED, a resolutionMessage and deleteDataEvent = true like so:

{
    "issueStatus": "CLOSED",
    "resolutionMessage": "Data event was unprocessable, deleted from system",
    "deleteDataEvent": true
}
Note
 A data event cannot be deleted once it has been matched to data record.

Deleting a single exception

Typically, data event exceptions do not need to be deleted - instead, their status should be set to RESOLVED either manually or through one of the workflows above. This preserves the history of exceptions that occurred in the system.

Exceptions can be deleted with one of the following endpoints:

DELETE /zones/<zone-uuid>/issues/<issue-uuid>
DELETE /zones/<zone-uuid>/adaptors/<adaptor-uuid>/issues/<issue-uuid>
DELETE /zones/<zone-uuid>/adaptors/<adaptor-uuid>/domains/<domain-uuid>/versions/<domain-version-uuid>/issues/<issue-uuid>
Note
 Deletion is not possible if the data event has not been resolved (if there’s an associated data event). To delete an exception linked to an unresolved data event, the data event must first be deleted. See Deleting an erroneous or unprocessable data event.

Matching Exception Workflow Examples

Resolving to a Possible Match

This workflow resolves the ambiguity to one of the "possible" matches.

Step 1: Find the exception(s) you want to handle

Any of the following commands will retrieve the exception(s) we’re looking for:

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues?filters=status:OPEN

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/issues?filters=status:OPEN

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/domains/082887e0-809e-4ba0-a274-33f212d71ffa/versions/1ffb9de0-d64b-41f6-aaaf-e1176a682285/issues?filters=status:OPEN

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues?filters=exceptions:ALL,adaptor:50482f6f-2585-48e1-aad7-527384ed710f,domain-version:1ffb9de0-d64b-41f6-aaaf-e1176a682285

The response is as follows, notice there is one "possible" match, due to its score.

{
    "dataIssues": [
        {
            "uuid": "1888ed20-c004-4452-b19f-1a0a6a2a66d7",
            "adaptor": {
                "uuid": "50482f6f-2585-48e1-aad7-527384ed710f",
                "name": "North America Database - Postgres",
                "description": "North America Database - Postgres description.",
                "zoneUuid": "f1d53448-5e6f-46d6-8b17-c756e5cbf693"
            },
            "domainVersion": {
                "uuid": "1ffb9de0-d64b-41f6-aaaf-e1176a682285",
                "domain": {
                    "uuid": "082887e0-809e-4ba0-a274-33f212d71ffa",
                    "name": "employee"
                },
                "versionNumber": 1
            },
            "title": "Data Event Exception",
            "issueStatus": "OPEN",
			"issuePriority": "MEDIUM",
            "dateCreated": 1602545824177,
            "lastUpdated": 1602545824177,
            "dataEventException": {
                "drKey": {
                    "zip": "31704",
                    "lastName": "BALFRE",
                    "firstName": "Tamra",
                    "address": "94429 Roxbury Center",
                    "city": "Albany",
                    "employeeId": "DEF466",
                    "state": "GA",
                    "birthDate": "1963-03-21",
                    "email": "tamra@other"
                },
                "localIdentifier": {
                    "id": 7
                },
                "matchStatus": "AMBIGUOUS",
                "action": "POST",
                "exceptionMessages": [
                    {
                        "errorType": "MATCHING",
                        "errorMessage": "Match is ambiguous and requires resolution"
                    }
                ],
                "matchingAlgorithm": {
                    "description": "Employee matching algorithm",
                    "type": "SCORE_BASED",
                    "possibleMatchScore": 30,
                    "definiteMatchScore": 70
                },
                "matches": [
                    {
                        "drUuid": "e0dca122-9cb2-4622-81d4-33f378a4e2d8",
                        "adaptor": {
                            "uuid": "7b3b8567-a290-48ca-a76d-8517935b5dc8",
                            "name": "Asia Database - Oracle",
                            "description": "Asia Database - Oracle description.",
                            "zoneUuid": "ea1ff430-3e30-4017-bd44-06537a3b6a63"
                        },
                        "drKey": {
                            "zip": "31704",
                            "lastName": "Balfre",
                            "firstName": "Tamra",
                            "address": "94429 Roxbury Center",
                            "city": "Albany",
                            "employeeId": "1E67052TIY",
                            "state": "GA",
                            "birthDate": "1963-03-20",
                            "email": "tbalfrec@i2i.jp"
                        },
                        "score": 30,
                        "matchRating": "POSSIBLE_MATCH"
                    }
                ]
            }
        }
    ],
    "_links": {
        "_self": {
            "href": "http://localhost:8080/api/zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues?page=0&size=10"
        }
    }
}

Step 2: Resolve the exception

Based on the above response, we want to confirm that the "possible" match is correct. To do this we need to note the value of its drUuid: e0dca122-9cb2-4622-81d4-33f378a4e2d8. We also need the exception’s uuid: 1888ed20-c004-4452-b19f-1a0a6a2a66d7.

Any of the following will work:

POST /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues/1888ed20-c004-4452-b19f-1a0a6a2a66d7/resolve?dr-uuid=e0dca122-9cb2-4622-81d4-33f378a4e2d8

POST /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/issues/1888ed20-c004-4452-b19f-1a0a6a2a66d7/resolve?dr-uuid=e0dca122-9cb2-4622-81d4-33f378a4e2d8

POST /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/domains/082887e0-809e-4ba0-a274-33f212d71ffa/versions/1ffb9de0-d64b-41f6-aaaf-e1176a682285/issues/1888ed20-c004-4452-b19f-1a0a6a2a66d7/resolve?dr-uuid=e0dca122-9cb2-4622-81d4-33f378a4e2d8

This POST command should return a 204 No Content HTTP status (no body is returned).

Step 3: Verify the resolution

We can verify the resolution by retrieving the exception by UUID via one of the following commands:

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues/1888ed20-c004-4452-b19f-1a0a6a2a66d7

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/issues/1888ed20-c004-4452-b19f-1a0a6a2a66d7

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/domains/082887e0-809e-4ba0-a274-33f212d71ffa/versions/1ffb9de0-d64b-41f6-aaaf-e1176a682285/issues/1888ed20-c004-4452-b19f-1a0a6a2a66d7

The response now looks like this:

{
    "uuid": "1888ed20-c004-4452-b19f-1a0a6a2a66d7",
    "drUuid": "e0dca122-9cb2-4622-81d4-33f378a4e2d8",
    "adaptor": {
        "uuid": "50482f6f-2585-48e1-aad7-527384ed710f",
        "name": "North America Database - Postgres",
        "description": "North America Database - Postgres description.",
        "zoneUuid": "f1d53448-5e6f-46d6-8b17-c756e5cbf693"
    },
    "domainVersion": {
        "uuid": "1ffb9de0-d64b-41f6-aaaf-e1176a682285",
        "domain": {
            "uuid": "082887e0-809e-4ba0-a274-33f212d71ffa",
            "name": "employee"
        },
        "versionNumber": 1
    },
    "title": "Data Event Exception",
    "issueStatus": "CLOSED",
    "issuePriority": "MEDIUM",
    "dateCreated": 1602545824177,
    "lastUpdated": 1602546326946,
    "dataEventException": {
        "drKey": {
            "zip": "31704",
            "lastName": "BALFRE",
            "firstName": "Tamra",
            "address": "94429 Roxbury Center",
            "city": "Albany",
            "employeeId": "DEF466",
            "state": "GA",
            "birthDate": "1963-03-21",
            "email": "tamra@other"
        },
        "localIdentifier": {
            "id": 7
        },
        "matchStatus": "RESOLVED",
        "action": "POST",
        "exceptionMessages": [
            {
                "errorType": "MATCHING",
                "errorMessage": "Match is ambiguous and requires resolution"
            }
        ],
        "matchingAlgorithm": {
            "description": "Employee matching algorithm",
            "type": "SCORE_BASED",
            "possibleMatchScore": 30,
            "definiteMatchScore": 70
        },
        "matches": [
            {
                "drUuid": "e0dca122-9cb2-4622-81d4-33f378a4e2d8",
                "adaptor": {
                    "uuid": "7b3b8567-a290-48ca-a76d-8517935b5dc8",
                    "name": "Asia Database - Oracle",
                    "description": "Asia Database - Oracle description.",
                    "zoneUuid": "ea1ff430-3e30-4017-bd44-06537a3b6a63"
                },
                "drKey": {
                    "zip": "31704",
                    "lastName": "Balfre",
                    "firstName": "Tamra",
                    "address": "94429 Roxbury Center",
                    "city": "Albany",
                    "employeeId": "1E67052TIY",
                    "state": "GA",
                    "birthDate": "1963-03-20",
                    "email": "tbalfrec@i2i.jp"
                },
                "score": 30,
                "matchRating": "POSSIBLE_MATCH"
            }
        ]
    }
}

Notice the status has moved from AMBIGUOUS to RESOLVED and the drUuid is present.

Resolving an Exception as "new"

In this example, instead of matching the record to an existing one, we will resolve it as "new". We have looked at the possible matches and determined none of them are the same record.

Step 1: Find the exception(s) you want to handle

Any of the following commands will retrieve the exception(s) we’re looking for:

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues?filters=exceptions:ALL

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/issues?filters=exceptions:ALL

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/domains/082887e0-809e-4ba0-a274-33f212d71ffa/versions/1ffb9de0-d64b-41f6-aaaf-e1176a682285/issues?filters=exceptions:ALL

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues?filters=exceptions:ALL,adaptor:50482f6f-2585-48e1-aad7-527384ed710f,domain-version:1ffb9de0-d64b-41f6-aaaf-e1176a682285

The response is as follows, notice there is one "possible" match, due to its score.

{
    "dataIssues": [
        {
            "uuid": "08dde112-4af1-416c-a790-803522486dfd",
            "adaptor": {
                "uuid": "50482f6f-2585-48e1-aad7-527384ed710f",
                "name": "North America Database - Postgres",
                "description": "North America Database - Postgres description.",
                "zoneUuid": "f1d53448-5e6f-46d6-8b17-c756e5cbf693"
            },
            "domainVersion": {
                "uuid": "1ffb9de0-d64b-41f6-aaaf-e1176a682285",
                "domain": {
                    "uuid": "082887e0-809e-4ba0-a274-33f212d71ffa",
                    "name": "employee"
                },
                "versionNumber": 1
            },
            "title": "Data Event Exception",
			"issueStatus": "OPEN",
			"issuePriority": "MEDIUM",
            "dateCreated": 1602546972409,
            "lastUpdated": 1602546972409,
            "dataEventException": {
                "drKey": {
                    "zip": "77060",
                    "lastName": "Berlin",
                    "firstName": "Suzy",
                    "address": "477 Mandrake Pass",
                    "city": "Houston",
                    "employeeId": "ABC12345",
                    "state": "TX",
                    "birthDate": "1987-02-02",
                    "email": "sberlind@usa.gov"
                },
                "localIdentifier": {
                    "id": 7
                },
                "matchStatus": "AMBIGUOUS",
                "action": "POST",
                "exceptionMessages": [
                    {
                        "errorType": "MATCHING",
                        "errorMessage": "Match is ambiguous and requires resolution"
                    }
                ],
                "matchingAlgorithm": {
                    "description": "Employee matching algorithm",
                    "type": "SCORE_BASED",
                    "possibleMatchScore": 30,
                    "definiteMatchScore": 70
                },
                "matches": [
                    {
                        "drUuid": "7a9d64bf-8732-44d2-a569-f485867469b6",
                        "adaptor": {
                            "uuid": "7b3b8567-a290-48ca-a76d-8517935b5dc8",
                            "name": "Asia Database - Oracle",
                            "description": "Asia Database - Oracle description.",
                            "zoneUuid": "ea1ff430-3e30-4017-bd44-06537a3b6a63"
                        },
                        "drKey": {
                            "zip": "77060",
                            "lastName": "Berlin",
                            "firstName": "Tobey",
                            "address": "454 Mandrake Pass",
                            "city": "Houston",
                            "employeeId": "1E12052OYC",
                            "state": "TX",
                            "birthDate": "1987-02-02",
                            "email": "tberlind@usa.gov"
                        },
                        "score": 30,
                        "matchRating": "POSSIBLE_MATCH"
                    }
                ]
            }
        }
    ],
    "_links": {
        "_self": {
            "href": "http://localhost:8080/api/zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues?page=0&size=10"
        }
    }
}

Step 2: Resolve the exception as new

Based on the above response we decided these records are for two different employees. Make note of the exception’s uuid: 08dde112-4af1-416c-a790-803522486dfd.

Any of the following will work:

POST /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues/08dde112-4af1-416c-a790-803522486dfd/resolve?new=true

POST /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/issues/08dde112-4af1-416c-a790-803522486dfd/resolve?new=true

POST /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/domains/082887e0-809e-4ba0-a274-33f212d71ffa/versions/1ffb9de0-d64b-41f6-aaaf-e1176a682285/issues/08dde112-4af1-416c-a790-803522486dfd/resolve?new=true

This POST command should return a 204 No Content HTTP status (no body is returned).

Step 3: Verify the resolution

We can verify the resolution by retrieving the exception by UUID via one of the following commands:

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/issues/08dde112-4af1-416c-a790-803522486dfd

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/issues/08dde112-4af1-416c-a790-803522486dfd

GET /zones/f1d53448-5e6f-46d6-8b17-c756e5cbf693/adaptors/50482f6f-2585-48e1-aad7-527384ed710f/domains/082887e0-809e-4ba0-a274-33f212d71ffa/versions/1ffb9de0-d64b-41f6-aaaf-e1176a682285/issues/08dde112-4af1-416c-a790-803522486dfd

The response now looks like this. Notice that the status is now RESOLVED and the drUuid assigned is a new one ie it doesn’t match the drUuid of the possible match:

{
    "uuid": "08dde112-4af1-416c-a790-803522486dfd",
    "drUuid": "9a0926fc-3f1f-49a3-b009-e200c6596127",
    "adaptor": {
        "uuid": "50482f6f-2585-48e1-aad7-527384ed710f",
        "name": "North America Database - Postgres",
        "description": "North America Database - Postgres description.",
        "zoneUuid": "f1d53448-5e6f-46d6-8b17-c756e5cbf693"
    },
    "domainVersion": {
        "uuid": "1ffb9de0-d64b-41f6-aaaf-e1176a682285",
        "domain": {
            "uuid": "082887e0-809e-4ba0-a274-33f212d71ffa",
            "name": "employee"
        },
        "versionNumber": 1
    },
    "title": "Data Event Exception",
    "issueStatus": "CLOSED",
    "issuePriority": "MEDIUM",
    "dateCreated": 1602546972409,
    "lastUpdated": 1602547184650,
    "dataEventException": {
        "drKey": {
            "zip": "77060",
            "lastName": "Berlin",
            "firstName": "Suzy",
            "address": "477 Mandrake Pass",
            "city": "Houston",
            "employeeId": "ABC12345",
            "state": "TX",
            "birthDate": "1987-02-02",
            "email": "sberlind@usa.gov"
        },
        "localIdentifier": {
            "id": 7
        },
        "matchStatus": "RESOLVED",
        "action": "POST",
        "exceptionMessages": [
            {
                "errorType": "MATCHING",
                "errorMessage": "Match is ambiguous and requires resolution"
            }
        ],
        "matchingAlgorithm": {
            "description": "Employee matching algorithm",
            "type": "SCORE_BASED",
            "possibleMatchScore": 30,
            "definiteMatchScore": 70
        },
        "matches": [
            {
                "drUuid": "7a9d64bf-8732-44d2-a569-f485867469b6",
                "adaptor": {
                    "uuid": "7b3b8567-a290-48ca-a76d-8517935b5dc8",
                    "name": "Asia Database - Oracle",
                    "description": "Asia Database - Oracle description.",
                    "zoneUuid": "ea1ff430-3e30-4017-bd44-06537a3b6a63"
                },
                "drKey": {
                    "zip": "77060",
                    "lastName": "Berlin",
                    "firstName": "Tobey",
                    "address": "454 Mandrake Pass",
                    "city": "Houston",
                    "employeeId": "1E12052OYC",
                    "state": "TX",
                    "birthDate": "1987-02-02",
                    "email": "tberlind@usa.gov"
                },
                "score": 30,
                "matchRating": "POSSIBLE_MATCH"
            }
        ]
    }
}