Read VARIANT through VIEW

urn:js:virtue:aspire:principle:14.1

TL;DR

Pipelines that source data held in VARIANT columns should do so through a view over the VARIANT column, not directly from the VARIANT column itself.

Rational

• Complexity of structure held in VARIANT can be abstracted away from the consumer
• Data can be appropriately typed and cast where necessary
• VARIANT schema changes can be handled in a single object, avoiding changes to potentially multiple pipelines
• Improved lineage
• VIEW can be documented in WhereScape and used to generate DML for data pipelines

Implications

• Existing pipelines should be reviewed and migrated