Reloading data from Staging

urn:js:virtue:aspire:pattern:.

TL;DR

This is a pattern for when staging data needs to be reloaded into the DV layers. It assumes staging data is being consumed via a Snowflake stream.

Instructions

1. Create a temporary table in the ADW_STAGE schema containing the data that needs reloading - this can be a CTAS off the staging table.
2. Using the temp data, soft delete the records in the staging table.
3. Manage the data in the DV and PL as appropriate - this may not be required if transforms are idempotent.
4. Consider pausing data loads into the staging table if necessary - again dependant on idempotence.
5. Run your load.

Appendix

Optional - Misc details related to what is being proposed.