Append new data to existing dataframe more efficiently - python [duplicate]

Question

Is there a more efficient way to append new data to an existing dataframe? As per the following example, I'm importing an existing df (frame_orig). Then some more code is performed, which produces some new values. I then append these back to the original df and export out.

This works fine but if the original df becomes too large, the process can a long time.

Is it possible to append the new values as a list. Or can the type of the original df be manipulated?

import pandas as pd

#frame_orig = pd.read_csv('C:/path/to/file/frame_orig.csv')

frame_orig = pd.DataFrame({'Val1': ['1','2'],
               'Val2': ['3','4'],
                })

##some code

new_Val1 = '5000'
new_Val2 = '6000'

newvalues = []

newvalues.append([new_Val1, new_Val2])

df_values = pd.DataFrame(newvalues, columns = ['Val1','Val2'])

new_df = pd.concat([frame_orig, df_values], ignore_index=True)

Answer 1

IIUC use DataFrame.loc - new default index is count by length of original DataFrame:

new_Val1 = '5000'
new_Val2 = '6000'


frame_orig.loc[len(frame_orig)] = [new_Val1, new_Val2]
print (frame_orig)
   Val1  Val2
0     1     3
1     2     4
2  5000  6000

If need specify columns:

frame_orig.loc[len(frame_orig), ['Val1','Val2']] = [new_Val1, new_Val2]
print (frame_orig)
   Val1  Val2
0     1     3
1     2     4
2  5000  6000

Or:

frame_orig.loc[len(frame_orig)] = {'Val1': new_Val1, 'Val2': new_Val2}
print (frame_orig)
   Val1  Val2
0     1     3
1     2     4
2  5000  6000

Answer 2

Use ._append() with ignore_index=True: If you are appending a single row, you can use the ._append() method with ignore_index=True: Try like this

import pandas as pd


print(pd.__version__)
# Existing DataFrame
frame_orig = pd.DataFrame({'Val1': ['1', '2'],
                            'Val2': ['3', '4']})

# New values
new_Val1 = '5000'
new_Val2 = '6000'

# Append new values using pd.DataFrame.append
new_df = frame_orig._append(pd.DataFrame({'Val1': [new_Val1], 'Val2': [new_Val2]}), ignore_index=True)

print(new_df)

Output:

   Val1  Val2
0     1     3
1     2     4
2  5000  6000