How to join two json files in python instead of nesting for loop

General Tech Bugs & Fixes 3 years ago

6.92K 2 0 0 0

User submissions are the sole responsibility of contributors, with TuteeHUB disclaiming liability for accuracy, copyrights, or consequences of use; content is for informational purposes only and not professional advice.

Answers (2)

Post Answer
profilepic.png
manpreet Tuteehub forum best answer Best Answer 3 years ago

 

Every time i will get 500 records form file1 to join file2 which contains more than 100000 records it costs two minutes !!

with open(file1,'r') as f1,open(file2,'r') as f2:
    a=json.load(f1)
    b=json.load(f2)
    list_a=[]
    for i in range(len(a)):
        for n in range(len(b)):
            if b[n]["id"]==a[i]["id"]:
                list_a.append(dict(b[n],**a[i]))
with open(result,'w') as f3:
    json.dump(list_a, f3,sort_keys=True, ensure_ascii=False)

File1:

[{ "id":"1", "name":"Tom" }, 
{ "id":"2", "name":"Jim" }, 
{ "id":"3", "name":"Bob" }, 
{ "id":"4", "name":"Jeny" },  
{ "id":"5", "name":"Lara" }, 
{ "id":"6", "name":"Lin" }, 
{ "id":"7", "name":"Kim" }, 
{ "id":"8", "name":"Jack" }, 
{ "id":"9", "name":"Tony" }]

File 2:

[ { "id":"1", "Details":[ { "label":"jcc", "hooby":"Swimming" }, { "label":"hkt", "hooby":"Basketball" }, ] }, 
{ "id":"2", "Details":[ { 
                                                
0 views
0 shares

profilepic.png
manpreet 3 years ago

 

I don't have the experience to know if this would speed it up. The solution below provided by Eugene Yarmash seems more reliable. I also don't have the big files to test speed, but you can try and see if using collections would speed up the iteration. I'd actually be curious myself if it would change anything:

File1 = [ { "id":"1", "name":"Tom" }, { "id":"2", "name":"Jim" }, { "id":"3", "name":"Bob" }, { "id":"4", "name":"Jeny" }, { "id":"5", "name":"Lara" }, { "id":"6", "name":"Lin" }, { "id":"7", "name":"Kim" }, { "id":"8", "name":"Jack" }, { "id":"9", "name":"Tony" } ]
File2 = [ { "id":"1", "Details":[ { "label":"jcc", "hooby":"Swimming" }, { "label":"hkt", "hooby":"Basketball" }, ] }, { "id":"2", "Details":[ { "label":"NTC", "hooby":"Games" } ] } ] 

from collections import defaultdict

d = defaultdict(dict)
for l in (File1, File2):
    for elem in l:
        d[elem['id']].update(elem)
Result = dict(d)

0 views   0 shares

No matter what stage you're at in your education or career, TuteeHUB will help you reach the next level that you're aiming for. Simply,Choose a subject/topic and get started in self-paced practice sessions to improve your knowledge and scores.

Similar Forum