pymongo bulkwrite time consistently increasing

General Tech Bugs & Fixes 2 years ago

0 1 0 0 0 tuteeHUB earn credit +10 pts

5 Star Rating 1 Rating

Posted on 16 Aug 2022, this text provides information on Bugs & Fixes related to General Tech. Please note that while accuracy is prioritized, the data presented might not be entirely correct or up-to-date. This information is offered for general knowledge and informational purposes only, and should not be considered as a substitute for professional advice.

Take Quiz To Earn Credits!

Turn Your Knowledge into Earnings.

tuteehub_quiz

Answers (1)

Post Answer
profilepic.png
manpreet Tuteehub forum best answer Best Answer 2 years ago

I am working on a project where I need to process huge number of data.

As a part of processing I am going through following steps -

  1. Read CSV line
  2. Translate CSV row to dictionary
  3. Bulk Upsert 1000 rows in MongoDB
  4. Iterate till the end of CSV file

The logic is fairly simple and is working fine without issues for millions of records.

However, the bulk_write time is consistently increasing.

The first call takes 0.55 seconds and it starts increasing for next batches.

No matter what stage you're at in your education or career, TuteeHub will help you reach the next level that you're aiming for. Simply,Choose a subject/topic and get started in self-paced practice sessions to improve your knowledge and scores.