Critical data elements (preview)

Not all data elements have the same importance or sensitivity. Dedicating resources to manage the quality of all data indiscriminately can be impractical and costly. Critical data elements are a logical grouping of important columns across tables in your data sources that you can use to focus your efforts.

These columns should be critical pieces of information that are necessary for decision making and so need to be governed with the highest care. For example, a "Customer ID" critical data element can map "CustID" from one table and "CID" from another table into the same logical container. Users can match this value across data assets to make connections, and when data producers create a new asset, they can use this element as a blueprint to provide quality information in the correct format.

Critical data elements (preview) in Microsoft Purview Unified Catalog are created within governance domains and can have policies set to manage these important pieces of information. When you create a critical data element, your organization can allocate resources strategically, focusing governance effort on areas that have the most significant effect on the business.

Prerequisites

• You must be using Microsoft Purview Unified Catalog.
• To create critical data elements and add columns to them, you must have data steward and data product owner permissions. For more information, see governance roles and permissions.
• To access data assets in the data map, you need at least data reader permissions on the collection where the asset is housed. For more information about data map permissions, see Data Map permissions.

Access critical data

Critical data is defined in governance domains. To see critical data in Unified Catalog:

1. In Unified Catalog, select Catalog management, then select Governance domains. You see a list of all the governance domains you have access to based on your permissions.
2. Select the governance domain where you want to see your critical data.
3. On the Details tab, find the Critical data elements card and select View all.
4. The Critical data elements (preview) page lists all of your critical data elements. Select a name to view its details.

You can also view a list of all critical data elements:

1. In Unified Catalog, under Discovery, select Enterprise glossary.
2. Select the Critical data elements tab.
3. Select the name of a critical data element to open its details page.

Filter your view by custom attribute

Use business concept attributes as a filter when exploring critical data elements.

1. On the Critical data elements (preview) page, select Add filter.
2. At Filter, select an attribute name from the dropdown list.
3. At Operator, select a condition, such as Equals or Starts with, which varies based on the kinds of values allowed by the attribute.
4. At Value, enter your value.
5. Select Apply.

Critical data element details

Select the name of a critical data element to view its details page. You can create and manage access policies for the critical data element by selecting Manage policies.

Overview

On the Overview tab, you see a description and:

• Columns: All data columns mapped from your assets to this critical data element. You can add columns.
• Associated data products: An automatically generated list of data products that have assets associated with your critical data element.

  Note

  The list of associated data products only refreshes when you add or remove columns.

Custom attributes

Under Custom attributes, you can see any custom attribute groups and attributes within groups, and the values set for each attribute. Only attributes with values are shown by default, but you can toggle Show attributes without a value on to view all attributes.

Create critical data

There are two ways you can create critical data elements:

1. Individually
2. In bulk, by importing a formatted CSV file (preview)

Create a single critical data element

1. In Unified Catalog, select Catalog management, then select Governance domains.
2. You see a list of all the governance domains you have access to based on your permissions. Select the governance domain where you want to add critical data.
3. On the Details tab, find the Critical data elements card and select View all.
4. Select New critical data element.
5. On the Basic details page, enter a name for your critical data. If you use a name that already exists, you'll see a warning during the creation process. While duplicating names isn't recommended, you won't be blocked from using a duplicate name.
6. Add a description, owner, and select an expected data type. Then select Next.
7. On the Custom attributes page, you can set values for all custom attributes defined by your admin in the Custom attributes section of the catalog. If the attribute is marked as required, you need to fill out all values on this page before you can complete the process.
8. When done, select Create.

You see your critical data in the list of critical data elements.

Bulk import critical data elements (preview)

You can import multiple critical data elements at once into a governance domain by uploading a formatted CSV file. During the bulk import process, you download a sample CSV file containing a header row. You fill out the file according to the instructions below, then reupload it. The critical data elements then appear in the governance domain in a draft state.

The bulk import process can't be used to edit or update critical data elements.

Follow these steps to import critical data elements in bulk to a governance domain:

1. In Unified Catalog, select Catalog management, then select Governance domains.

2. Select the name of the governance domain you want to add critical data elements to.

3. On the Details tab of the governance domain's details page, find the Critical data elements (preview) card and select View all.

4. On the Critical data elements (preview) page, select Import.

5. On the Import flyout pane, select Download a sample CSV file.

6. Open the downloaded file. Fill out the worksheet as described below to avoid upload errors.

   • Column A - name (required): Enter the critical data element name. Every name must be unique; there can't be multiple rows with the same name. Maximum length is 256 characters.
   • Column B - description (required): Enter a description for the critical data element. Maximum length is 10,000 characters.
   • Column C - owners (required): Enter a valid email address that belongs to the same tenant that you're importing from. You can add multiple email addresses separated with a semicolon; for example: owner1@contoso.com;owner2@econtoso.com. Groups without email aren't supported.
   • Column D - data type (required): Enter one of the possible values: number, text, date/time, Boolean.

7. Column E and onward: These are custom attributes, which differ by governance domain based on how they're defined and scoped. Note these formatting requirements for specific entry types: - Multiselect options: Enter the objects inside square brackets and in quotes; for example, ["a"], or ["a", "b"]. - Single select options: Enter the raw value; for example, a1, b2. - Booleans: Enter true or false; can be upper or lowercase. - String or rich text: Use regular text; don't add quotes unless you want quotes reflected in the attribute. - Numeric - short, int, long, double, float, bit, byte: Don't add quotes, just enter the number.

8. After formatting your CSV file, save it locally. Return to the Import flyout pane, select Browse, then find and select your CSV file to upload it.

9. On the Import flyout pane, select OK to start the import process.

If there's a problem with the CSV file, an error message appears on the Import pane explaining the error so you can fix the CSV file and reupload it.

If there are no problems with the file, the import process begins. Depending on the size of the CSV file, it might take some time to complete the import process. If the file is small, the import happens immediately and you see a success message. Larger files take longer for the import process to complete.

Check the status by returning to the governance domain details page. On the Details tab, select Monitoring to view the status of your bulk import jobs, including any errors, on the Monitoring page.

Add columns

After you create your critical data, you can add data columns to map critical columns from your assets to this logical concept.

1. Select your critical data element from the list of critical data elements in your governance domain.

2. Select the + Add column button.

3. Search for your data sources by using keywords and filters, and select a data source.

4. In that data source, select any relevant column.

5. Select Add.

6. You should now see all your columns mapped to your critical data element.

7. Repeat for any number of data sources and columns.

   Below the columns you add, you also see the list of data products that are associated with the assets you add to your critical data element.

Remove column

To remove a column from a critical data element, select the ellipsis ... button in the row where the column is listed, and select Remove.

If you can't find the column you want to remove, select the View all columns button to see the full list of columns.

Update and edit critical data

1. On the critical data's details page, select Edit.
2. On the Basic details page, you can edit the name, description, owner, and expected data type. If you change the name to one that already exists, you'll see a warning about the duplication but you can proceed to use the duplicate name.
3. On the Custom attributes page, you can set values for all custom attributes defined by your admin in the Custom attributes section of the catalog. If the attribute is marked as required, you need to fill out all values on this page before you can complete the process.
4. Select Save to save your changes.

To update the status of your critical data element, select the Status button to change between: Draft, Published, and Expired.

Manage related terms

You can link your critical data elements to related glossary terms.

1. Select the governance domain where your critical data element resides.
2. Select the Critical data elements button.
3. Search or browse for the critical data element and select it.
4. To link your critical data element to a glossary term, select the + Add term button.
5. Search for the term, or terms, you want to link and select them.
6. To add your selected terms, select Add.
7. To remove a related term, select the term, then select the ... ellipsis button, and then the Remove button.

Delete critical data

To delete a critical data element, you need to unpublish it and delete all columns within it, and any links to glossary terms. You can then select the Delete button and proceed with deleting the critical data element.

Remove data that's deleted from Data Map (preview)

If the parent data asset of a column in a critical data element is deleted in Data Map, a banner appears in the critical data element to inform you of the asset deletion. However, the column still appears in the critical data element until you remove it.

Open the critical data element to view the list of columns. Columns with a caution symbol on its row have been deleted from Data Map due to their parent data asset deletion. To remove the deleted column from your critical data element, select ellipsis (...) in the row, then select Remove.

Manage access policies

To manage access policies for critical data, you need data steward permissions.

1. On your governance domain's page, select your critical data elements.

2. Select a critical data element.

3. Select Manage policies.

4. From the policy configuration window, you're able to create and manage access policies for your critical data elements.

5. For more information about setting up access policies, see the documentation about managing access policies.

   These policies apply to all associated data products in your critical data element.