Process All Data

urn:js:virtue:aspire:principle:10.1

TL;DR

Where the appropriate metadata has been provided, all incoming data must be loaded, processed and made available for provisioning via the data access layer.

Rational

Where the appropriate metadata has been provided (and the data and business process analysis have also been completed), all incoming data must be loaded, processed and made available for provisioning via the data access layer. Subsets of data can be subsequently created as part of the provisioning process.

Note: When applying this principle due consideration must be given to the legal and regulatory as well as the supplier constraints (see Section 4.3.4).

The main driver is:

Business continuity – In most cases, if a feed contains a small number of non-critical quality issues and or errors, the majority of the data should still be processed and presented to the consumers. Whereas the data with the quality issues should be isolated and the relevant data steward notified.
Reduce requests for resubmission – Wherever possible and/or practical, data quality issues and errors should be managed within the strategic data store. This will reduce the need to request resubmissions from the data suppliers.
Reduce future effort – The effort needed to augment an existing dataset will be considerably less if the additional data elements are readily available and there is no need to fetch them from the underlying layers and/or source.

Implications

The potential implications are:

Data error and exception handling – This approach will require robust error and exception handling routines to manage data quality issues in a controlled manner.
Comprehensive error and exception reports – To ensure that all data quality issues are forwarded to the appropriate data stewards will require automated alerts along with a comprehensive suite of error and exception reports.
Data correction process – Wherever it is practical and/or feasible, data correction will be done within the strategic data store. Only as a last resort should a replacement feed be requested from the data supplier.
De-identification of sensitive data – Sensitive personal identifying information must be appropriately masked, pseudonymised or anonymised to suit the relevant usage scenario.