What technology I have to use for my project

manpreet Best Answer 2 years ago

I want help about what technology I need to use for writing my application. The requirement:

Data: A main table and can have millions of rows. The data won't change in this table(but rows get added every day) but there will be other tables(transaction tables) which are getting updated/inserted/deleted and these tables also will have millions of rows based on the processing.

Processing: Main table have a column of categories of rows. I need to pull a row randomly from each category and process it. The processing time will be very less, but random row selections will be very high, can be in millions. Means I need to pull millions of rows and process for every hour. And I need to store the processed details in other tables.

Is it ok to use hive table with spark? I'm new to the hadoop technologies. But my intention is to write a highly scalable application. Do I need to create a table for each category instead of single main table? Is it ok to maintain other transactional tables in hive? is there any alternative?

Thanks a million for your inputs!

Chintu

manpreet 2 years ago

If your data fits in the RDBMS there's no need to go to Hadoop/Spark and the lot. You would get a much more complicated and most probably slower solution (due to distribution, serialization etc.) vs. just using SQL

It is better to solve your problem now and rewrite if/when your problem would change in the future