How to load and append the data without duplicates using MYSQL LOAD DATA

Question

I have a machine that output the data into a text file.

This text file contains the raw data. Now I would like to insert the raw data from the text file by using the MYSQL LOAD DATA.

My condition is to append the new data to the existing table without duplicates. I made raw_data column as primary key.

Everytime i run the LOAD DATA command it terminates when duplicates encountered and will not continue to load the rest of none duplicate raw data.

Example:

LOAD DATA INFILE '/mnt/A3/rawdata.txt' INTO TABLE test(raw_data);

Error Msg:for key 'PRIMARY' Duplicate entry 'aabbcc'

My question is how can I load and append the raw data to existing table without duplicates?

Answer 1

Look at the documentation, keywords REPLACE or IGNORE