The YOUnite Data Store data management refers to a model where domain data records are stored in the YOUnite Server, which serves as a database as well as a router that receives and accurately transmits the data to be shared to the proper source systems that have subscribed to data changes. In other words the YOUnite Server is the service of a YOUnite Data Fabric deployment that provides accurate and most recent versions of data to which all source systems subscribe to. In the case of YOUnite Data Store, the domain data records would be stored in the YOUnite Server. The YOUnite data store is designed for reference data such as a list of infrequently-changed data, such as states, countries, zip-codes, etc.

The advantage of the YOUnite Data Store model lies in its centralized data handling approach. Unlike federated systems, it eliminates the need to reference source systems for data requests, streamlining the data retrieval process.

The limitations of the YOUnite Data Store model include:

  1. The responsibility placed on YOUnite Servers to retrieve data from their local stores, which can be problematic if these servers are already under heavy use.

  2. The necessity for top-tier tools for data curation and management, as the YOUnite Data Store is exclusively managed via the YOUnite API.

  3. Potential misalignment with an organization’s data governance principles, as the YOUnite Data Fabric platform being used as a data repository may conflict with the norm of business units owning and storing reference data in their respective source systems.

YOUnite Data Store: Cross Domains

YOUnite Data Store records are created by POSTing to the /drs endpoint. Cross-referenced are established for YOUnite Data Store records when their data is POSTED.

A Quick YOUnite Data Store Example

In the following example, a student domain version makes a cross-reference to the country:v1 domain-version.

{
  "modelSchema": {
    "properties": {
      "studentID": {
        "type": "string",
        "required": true
      },
      "name": {
        "type": "string"
      },
      "homeCountry": {
        "$ref": "/domains/country:v1"
      },
      "birthday": {
        "type": "string"
      }
    }
  },
  "drKeyProperties": [
    "studentID"
  ]
}

When student data records are POSTed, the request body would include a reference to a specific country by its DR Key Properties. In this example country:v1 has one DR Key Property "abbreviation", which is the 3 letter code of the country:

{
    "studentID": "XYZ-123",
    "name": "Juana Masa",
    "homeCountry": "MEX",
    "birthday": "12/22/1988"
}

When the student data record is retrieved (i.e. via GET /drs/<dr-uuid>), the response will insert the country:v1 information:

{
    "studentID": "XYZ-123",
    "name": "Juana Masa",
    "homeCountry": {
        "name": "Mexico",
        "abbreviation": "MEX",
        "capitalCity": "Mexico City"
    },
    "birthday": "12/22/1988"
}

The cross-reference can be to an entire data record (e.g. all of the properties in the data record):

"$ref": "/domains/country:v1"

Or for a specific property in the data record (e.g. just the country name):

"$ref": "/domains/country:v1/name"

For the example above, where the reference is instead /domains/country:v1/name, the result would look like this:

{
    "studentID": "XYZ-123",
    "name": "Juana Masa",
    "homeCountry": "Mexico",
    "birthday": "12/22/1988"
}

Defining YOUnite Data Store Cross-References

Note
 YOUnite Data Store domains cannot cross-reference Federated domains, but Federated domains may cross-reference YOUnite Data Store domains.

YOUnite Data Store Cross-Reference Model Schema Syntax

Cross-reference uses the token $ref and a property location that defines the cross-reference (reference).

The syntax of the $ref location is /domains/{domain-name}[:v{version-number}][/{property-name}].

Example:

{
    "modelSchema": {
        "properties": {
            "country": {
                "$ref": "/domains/country:v1"
            },
            "countryCapital": {
                "$ref": "/domains/country:v1/capitalCity"
            }
        }
    }
}

Enum Cross-Reference Model Schema Syntax

A cross-reference may optionally include an enum to limit the allowable values (YOUnite Data Store records only).

{
    "modelSchema": {
        "properties": {
            "country": {
                "$ref": "/domains/country:v1",
                "enum": [
                    "MEX",
                    "JPN",
                    "USA"
                ]
            }
        }
    }
}

Location Syntax

The syntax of the location is /domains/{domain-name}[:v{version-number}][/{property-name}].

The values are in square brackets are optional. The following are all valid locations:

/domains/country
/domains/country:v1
/domains/country/capitalCity
/domains/country:v1/capitalCity

Reference to an Entire Data Record

An entire data record can be referenced with a location:

"$ref": "/domains/{domain-name}[:v{version}]"

For example:

{
    "modelSchema": {
        "properties": {
            "homeCountry": {
                "$ref": "/domains/country:v1"
            }
        }
    }
}

Or, if the version number is not specified, the latest domain version is used:

{
    "modelSchema": {
        "properties": {
            "homeCountry": {
                "$ref": "/domains/country"
            }
        }
    }
}
Posting cross-references in a YOUnite Data Store record

When POSTing a YOUnite Data Store record, use the Dr Key of the desired data record e.g.:

{
  "homeCountry": "MEX"
}

Sometimes the DrKey for a data record will contain multiple properties. For example if the DrKey for the domain version state:v1 contains two properties:

{
  "drKeyProperties": ["stateCode", "countryCode"]
}

When POSTing a data record it needs to include both property values, in JSON format:

{
  "state": "{\"stateCode\": \"CA\", \"countryCode\": \"USA\"}"
}

In the rare cased where a value contains a quote (") or backslash (\), they must be escaped twice, i.e. quote (") is represented as \\\" and backslash (\) is represented as \\\\.

For example, if path = legal\hr\"employment doc.pdf", the JSON would look like so:

"documentPaths": "{\"drive\": \"D\", \"path\": \"legal\\\\hr\\\\\\\"employment doc.pdf\\\"\"}"

Reference a Property in a Data Record

A single property of a data record can be referenced with a location with the following syntax:

"$ref": "/domains/{domain-name}[:v{version}]/{property-name}"

For example:

{
    "modelSchema": {
        "properties": {
            "homeCountry": {
                "$ref": "/domains/country:v1/capitalCity"
            }
        }
    }
}

A Cross Reference Example

The following demonstrates in more detail the example where properties from an existing country domain version are to be referenced by a new domain version students. Assume the country domain version contains many properties including the country name and the address for the immigration office. The student domain version will reference the countries domain version twice:

  1. In a property called homeCountry

  2. In a property called immigrationAddress

GET the address Property in the countries:v1 Model Schema
GET /domains/versions/<country-version-uuid>/properties

Response:

[
    {
            .
            .
        "propertyName": "immigrationAddress",
        "propertyType": "object",
            .
            .
    },
    {
            .
            .
        "propertyName": "name",
        "propertyType": "string"
            .
            .
    }
]
Create a New students Domain Version and Add the Cross-Reference To the First Version’s Model Schema
POST /domains/versions/<students-domain-uuid>

Response:

{
    "modelSchema": {
        "properties": {
            "homeCountry": {
                "$ref": "/domains/country:v1/name"
            },
            "immigrationAddress": {
                "$ref": "/domains/country:v1/immigrationAddress"
            }
        }
    }
}

Note that homeCountry is a primitive type while immigrationAddress is an object and contains several items such as street, city, province-state, and postal-code.

Again, the students domain version could just reference the country data record once, which would cause the entire country data record to be inserted.

To limit the list of countries to those just in North America (Canada, Mexico and United States) an enum should be used. For example to create the domain version north_american_countries:v1:

{
    "modelSchema": {
        "properties": {
            "countries": {
                "$ref": "/domains/country:v1/name",
                "enum": ["CAN", "MEX", "USA"]
            }
        }
    }
}

Rules About Default and Required Values With YOUnite Data Store properties

If a domain property has a "default" value defined in its "modelSchema", then any domain data records posted will use the default value if the property either does not include a value or is sent in with a null value. Note an empty or blank property is not the same as null.

If a domain property has "required" set to "true", then a valid non-null value must be posted for it (by default "required" is set to "false"). If the property is missing or is null, then an HTTP 400 (Bad Request) will be returned.

How to Establish Circular Cross-References

Consider the situation where a state domain version needs to reference country and country needs a reference back to multiple state data records.

This creates a "chicken-or-the-egg" problem since the schema for one will exist before the other. To remedy this problem, one of the domains versions can be updated after both are created.

For example:

POST the modelSchema for the state domain:

{
    "name": {
        "type": "string"
    }
}

POST the domain version for country with a model schema that includes a reference to state:

{
    "name": {
        "type": "string"
    },
    "state": {
        "$ref": "/domain/state:v1/name"
    }
}

PATCH the domain version for state with a model schema that includes the reference to country:

{
    "name": {
        "type": "string"
    },
    "country": {
        "$ref": "/domain/country:v1/name"
    }
}

Data can be posted only if all ref’s have a valid location.

Note
 Patching a domain version may add properties but may not modify or remove existing properties once the domain version is in use.

Posting Records to a YOUNITE_DATA_STORE Domain

Once the domain/version has been created, data records can be POSTed to it using the /drs endpoint: 

POST /drs

{
    "name": "states",
    "version": 1,
    "json": {
        "name" : "California",
        "abbreviation" : "CA"
    }
}

CSV import into YOUNITE_DATA_STORE

Another way to load data into an YOUNITE_DATA_STORE domain is from a CSV file. This is done via POST /drs/bulk?filters=name:<domain name>,version:<domain version>.

The CSV file must:

  1. Have a header row with column names that exactly match the names of the properties in the domain version.

  2. For nested properties, dot notation can be used, ie address.city.

  3. Loading data into arrays is not supported.

  4. By default, the maximum file size is 10 MB.

To load the data, a multipart/form-data request must be made, with the file in the data field.

CURL example:

curl --request POST 'https://<api-server>/api/drs/bulk?filters=name:person,version:1' \
--header 'Authorization: Bearer bc66b9a6-06a4-44c6-858c-4d6fde2e6e8c' \
--form 'data=@/C:/Data/person_data.csv'

Retrieving YOUnite Data Store Data Records

YOUNITE_DATA_STORE examples (includes the actual data of the data record):

GET /drs?filters=name:country
GET /drs?filters=name:country,version:4

Response:

{
    "drs": [
        {
            "uuid": "f0350e7d-9675-4890-9672-49abb4f673dc",
            "domainVersionUuid": "1de3051d-7a56-4c5c-996b-58461a0abd98",
            "createdByUuid": "da823503-20e3-4505-922a-ae2ca319ef82",
            "dateCreated": 1608768413771,
            "json": {
                "name": "United States of America",
                "countryCode": "USA",
                "capital": "Washington, DC",
                "population": 331928610
            },
            "drKey": {
                "countrycode": "USA"
            }
        },
        {
            "uuid": "c4f57e9d-8d96-4e19-831a-34c95f4b5fc2",
            "domainVersionUuid": "1de3051d-7a56-4c5c-996b-58461a0abd98",
            "createdByUuid": "da823503-20e3-4505-922a-ae2ca319ef82",
            "dateCreated": 1608768418615,
            "json": {
                "name": "Mexico",
                "countryCode": "MEX",
                "capital": "Mexico City",
                "population": 128932753
            },
            "drKey": {
                "countryCode": "MEX"
            }
        }
    ],
    "_links": {
        "first": "http://localhost:8080/api/drs?filters=name%3Acountry&page=0&size=10",
        "last": "http://localhost:8080/api/drs?filters=name%3Acountry&page=0&size=10",
        "self": "http://localhost:8080/api/drs?filters=name%3Acountry&page=0&size=10"
    },
    "_page": {
        "page": 0,
        "size": 10,
        "totalElements": 2,
        "totalPages": 1
    }
}

Querying YOUnite Data Store Records via the API

Note
 Federated Data Records cannot be queried this way. Refer to Accessing Data Records for more information on assembling federated data records.

Use the /drs endpoint and the appropriate domain and DrKey to GET an YOUnite Data Store data record:

GET /drs?filters=name:states,drKey:CA

{
    "name": "states",
    "version": 3,
    "json": {
        "name" : "California",
        "abbreviation" : "CA"
    }
}

The above GET request would be made for the default version (3) of states.

If there are multiple versions of a domain, and a domain other than the default is needed, the version number can be included in the URI. For example, assume there are three versions of the states domain and the current version is version 3. The consumer can retrieve the California version 1 data record by using the following:

GET /drs?filters=name:states,version:1,drKey:CA

{
    "name": "states",
    "version": 1,
    "json": {
        "name" : "California",
        "abbreviation" : "CA"
    }
}