Which Azure storage technology for weather forecast data

manpreet Best Answer 2 years ago

I would like some advice/tips about the right technology to select in order to store some forecast data on Azure ref="https://forum.tuteehub.com/tag/technologies">technologies. My team and I are scraping some weather forecast data everyday from various sources and store it as is on a Azure File Storage. The files format is "grib2" which is a standard format of weather forecast data. We are able to extract the data from those "grib2" files using python script running on a Azure VM.

We now have several files that represent hundreds gigabytes of data to store and I'm struggling to find which data store from the Azure technologies suits the best our needs in term of praticity and cost.
We started using "Azure Table Storage" first because it's cheap solution, but I've read on many posts that it is a bit old and not very adapted to our solution as it for example does not allow more than 1,000 entites per query and no aggregation on data.

I considered using Azure SQL db but it seems that it can become very expensive very fast. 
I also considered the Azure Data Lake Storage Gen2 (and HDinsight) technologies but am not very at ease with those blob storages and am not really able to say if it can suit my needs in terms of praticity and if it is "easy to query". 

By now we just plan to achieve that :

1) Extract data from grib2 files thanks to a python script running on an Azure VM 
2) Insert the transformed data into [Azure storage] 
3) Query the [Azure storage] from Azure Machine Learning Service or a local R script (for example) 
4) Insert the computed data into [Azure storage]

where [Azure Storage] technology is to determine. 

Any help or advice would be much appreciated, thanks.