ValueError: dict contains fields not in fieldnames

Question

Can someone help me with this.

I have my Select query

selectAttendance = """SELECT * FROM table """

And I want the content of my select query and include a header when I download the csv file, So I did this query:

with open(os.path.join(current_app.config['UPLOAD_FOLDER'], 'csv.csv'), 'wb') as csvfile:
                writer = csv.DictWriter(csvfile,fieldnames  = ["Bio_Id","Last_Name","First_Name","late","undertime","total_minutes", "total_ot", "total_nsd", "total_absences"], delimiter = ';')
                writer.writeheader()
                writer.writerow(db.session.execute(selectAttendance))
            db.session.commit()

but it gives me this error

**ValueError: dict contains fields not in fieldnames**

I want to have like this output in my downloaded csv file:

Bio_Id Last_Name First_Name late undertime total_minutes total_ot total_nsd total_absences
1      Joe       Spark       1     1            2            1        1          1

Thank you in advance.

Answer 1

As the error message states, your dictionary contains keys that don't have a corresponding entry in your fieldnames parameter. Assuming that these are just extra fields, you can ignore them by using the extrasaction parameter during construction of your DictWriter object:

writer = csv.DictWriter(csvfile, fieldnames=["Bio_Id","Last_Name","First_Name","late","undertime","total_minutes", "total_ot", "total_nsd", "total_absences"], 
                        extrasaction='ignore', delimiter = ';')

Answer 2

As the error states: the dictionary that comes from the query contains more key than the field names you specified in the DictWriter constructor.

One solution would be to filter that in advance, something like this:

field_names = ["Bio_Id","Last_Name", ...]
writer = csv.DictWriter(csvfile,fieldnames=field_names , delimiter = ';')
writer.writeheader()
data = {key: value for key, value in db.session.execute(selectAttendance).items()
        if key in field_names}
writer.writerow(data)

Another solution could be to construct the query using only those fields:

query = 'SELECT %s FROM table' % ', '.join(field_names)

However, Tim Pietzcker's answer is the best.

Answer 3

To bring up to date the csv.DictWriter now has an option extrasaction='ignore' parameter.

Here is an example of how to use it:

csv_writer = csv.DictWriter(csvfile, fieldnames=['a','b','c'], extrasaction='ignore')
csv_writer.writerow({'a': 1, 'b': 2, 'c': 3, 'd': 4})

Answer 4

Here's a possible solution for the dumbass cut'n paste programmer squad (of which I am a charter member!)

If you define your titles like this:

customerTitles = '"ID","Gender","Type","Full Name","Territory","Customer Group","Tax ID","Customer Details"'

You will definitely see the Value Error the OP encountered.

However, using the extrasaction='ignore' flag and checking the resultant file you will see this beauty:

"""",I,D,"""",",","""",G,e,n,d,e,r,"""",",","""",T,y,p,e,"""",",","""",F,u,l,l, ,N,a,m,e,"""",",","""",T,e,r,r,i,t,o,r,y,"""",",","""",C,u,s,t,o,m,e,r, ,G,r,o,u,p,"""",",","""",T,a,x, ,I,D,"""",",","""",C,u,s,t,o,m,e,r, ,D,e,t,a,i,l,s,""""

Answer 5

Hi all thank you for answering. I finally found out how to make it. here is the code

    w = csv.writer(file(r'test.csv','wb'), delimiter=';')
    w.writerows([["Bio Id","Last Name",.....]])
    w.writerows(db.session.execute(selectAttendance))
    db.session.commit()