Which technology ( SQL/NoSQL) to use for real-time data aggregation

manpreet Best Answer 2 years ago

I need to design a near real-time system where documents">documents ( with fields:id,keywords,timestamp ) are getting added to the system. The requirement is to get top-k keywords from the documents">documents added to the system in last x minutes. The typical document addition rate is around 100 documents">documents/sec, which may increase in the future ( hence technology should be horizontally scalable ).

I am thinking of using solr-facets ( with sharding ) to generate the top-k keywords, where I am a bit concerned about the high writes/sec for solr. Another option is to use Cassandra, but not sure how it will scale for range queries ( to compute aggregates ), as OrderPreservingPartitioner could make it difficult to distribute the load.