Comparing columns in pandas different data frames and fill in a new column

Question

I have two dataframes: One contains of company and its corresponding texts. The texts are in lists

**supplier_company_name   Main_Text**

JDA SOFTWARE          ['Supply chains','The answer is simple -RunJDA!']

PTC                    ['Hello', 'Solution']

The second dataframe is texts extracted from the company's website.

      Company            Text   
0   JDA SOFTWARE    About | JDA Software    
1   JDA SOFTWARE    833.JDA.4ROI
2   JDA SOFTWARE    Contact Us
3   JDA SOFTWARE    Customer Support    
4   PTC             Training    
5   PTC             Partner Advantage

I want to create the new column in second dataframe if the text extracted from the web matches with the any item inside the list in the Main_Text column of the first data frame, fill True else fill False.

Code:

target = []
for x in tqdm(range(len(df['supplier_company_name']))): #company name in df1
    #print(x)
    for y in range(len(samp['Company']): #company name in df2
        if samp['Company'][y] == df['supplier_company_name'][x]: #if the company name matches
            #check if the text matches
            if samp['Company'][y] in df['Main_Text'][x]:
                target.append(True)
            else:
                target.append(False)

How can I change my code to run efficiently?

Answer 1

I’ll take the hypothesis that your first dataframe (df) has unique company names. If so, you can easily reindex it by said company name and extract the (only one left) Main_Text Series to make it pretty much like a good old dict:

main_text = df.set_index('supplier_company_name')['Main_Text']

Now we just need to iterate over each line in samp, fetch the main text corresponding to the first column and generate a truthy value based on that and the second column. This is a job for apply:

target = samp.apply(lambda row: row[1] in main_text.loc[row[0]], axis=1)