S3 to Staging Pipeline

urn:js:virtue:aspire:principle:26.1

TL;DR

Data should flow from S3 into Snowflake staging directly using a Snowflake COPY INTO statement either as a bulk data load, or a continuous load using Snowpipe.

Rational

Using a common pattern will enable automation and drive consistency for data delivery, monitoring and security.

Implications

• Effort from squads to migrate to this pattern where pre-existing pipelines exist