Using encode UTF-8 on File.read()

manpreet Best Answer 3 years ago

I am trying to upload a csv file to a postgresql database and it is stuck with a error you see at the end of my question. The reason is there are unicode characters">characters in the file and it is encoded in windows-1252.

This is the line where I decode the file with UTF-8. However I would like to basically accept every encoding and decode it as UTF-8 or set the encoding to UTF-8 when reading the file and then decode with the line down below. I am not using open because I had problems with it, instead I am using InMemoryUploadedFile.read() (https://docs.djangoproject.com/en/2.2/ref/files/uploads/#django.core.files.uploadedfile.UploadedFile.read)

csv_file.seek(0)
file = csv_file.read().decode('utf-8').splitlines()
reader = csv.reader(file)

This is the error and it is because of this Character: d�mpe

'utf-8' codec can't decode byte 0xb3 in position 13969: invalid start byte

Any help would be appreciated.

manpreet 3 years ago

You can use the errors parameter to .decode() to ignore encoding errors or replace them with a replacement character.

csv_file.seek(0)
file = csv_file.read().decode('utf-8', errors='ignore').splitlines()
reader = csv.reader(file)

It would be better, of course, to fix the original file so that it is actual, correct UTF-8.