Get answers to the following options and more, as to what is the cheapest and most maintainable solution for this kind of data pipeline.
- Do the data scraping ourselves OR use an established 3rd party aggregated data source
- Use temporary embedded DBs OR a cloud master DB server
- Use a standard ELT pipeline pattern OR do ETL within each pipeline step
- Implement purely via the pipeline DAGs and SQL OR create a separate Python package
The following areas will be covered.
How the Airflow data pipeline framework can be used.
The approach to the architecture and reasons for the rewrite.
Python code for scraping data, along with Soda testing to verify the steps data.
The client data consumption architecture to our Go cloud service
How does cloud pricing work?
It is complicated! For example, hard disk storage cost should be simple?
But it varies based on hardware type, size, throughput, iops, different price depending on zone (us-east2 vs ap-south1 etc.), charges for traffic between regions vs within regions Then there are the backup costs based on its retention size/time, schedule.
What data sources does our case study need?
Cloud price lists are not small. The full combined price lists for AWS, Google and Azure are 5 million prices.
Review possible data sources and related tools, such as Kubecost, Infracost.io and the cloud providers direct sources.
What is the cost of the different pipeline architectures
Often data scientists put together a PoC pipeline without considering cost, it becomes production and starts eating money!
Charges for cloud SaaS are more concealed than raw Cloud PaaS. But it must be assessed too, and they have some interesting pricing. For example if your data pipeline is open source and public, Github workflows are free! If it is private they charge twice as much for compute as using direct Azure.
Similarly cloud pipeline providers are big in this space, but how does Astronomy SaaS pricing stack up vs. running yourself on cloud PaaS?
