How do I find an image contained within an image?

Question

I'm currently building what basically amounts to a cross between a search engine and a gallery for web comics that's focused on citing sources and giving authors credit.

I'm trying to figure out a way to search an image to find characters within it.

For example:

Assuming I have the red character and the green character saved as Red Man and Green Man how do I determine if an image contains one or the other.

This doesn't need to have 100% recognition or anything is this is more of an added feature I'd like to create, I'm just not sure where to start. I've done a lot of googling for image recognition but haven't found much helpful.

For what it's worth, I'd prefer to do this using Python.

Answer 1

As Moshe's answer only covers matching a template that is contained only once in the given picture. Here's how matching several at once:

import cv2
import numpy as np

img_rgb = cv2.imread('mario.png')
template = cv2.imread('mario_coin.png')
w, h = template.shape[:-1]

res = cv2.matchTemplate(img_rgb, template, cv2.TM_CCOEFF_NORMED)
threshold = .8
loc = np.where(res >= threshold)
for pt in zip(*loc[::-1]):  # Switch columns and rows
    cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0, 0, 255), 2)

cv2.imwrite('result.png', img_rgb)

(Note: I changed and fixed a few 'mistakes' that were in the original code)

Result:

Source: https://opencv24-python-tutorials.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_template_matching/py_template_matching.html#template-matching-with-multiple-objects

Answer 2

For anyone who stumbles across this in the future.

This can be done with template matching. To summarize (my understanding), template matching looks for an exact match of one image within another image.

Here's an example of how to do it within Python:

import cv2

method = cv2.TM_SQDIFF_NORMED

# Read the images from the file
small_image = cv2.imread('small_image.png')
large_image = cv2.imread('large_image.jpeg')

result = cv2.matchTemplate(small_image, large_image, method)

# We want the minimum squared difference
mn,_,mnLoc,_ = cv2.minMaxLoc(result)

# Draw the rectangle:
# Extract the coordinates of our best match
MPx,MPy = mnLoc

# Step 2: Get the size of the template. This is the same size as the match.
trows,tcols = small_image.shape[:2]

# Step 3: Draw the rectangle on large_image
cv2.rectangle(large_image, (MPx,MPy),(MPx+tcols,MPy+trows),(0,0,255),2)

# Display the original image with the rectangle around the match.
cv2.imshow('output',large_image)

# The image is only displayed if we call this
cv2.waitKey(0)

Answer 3

Important note: matchTemplate is even able to detect resized and rotated templates. Here's the code and outputs.

import matplotlib.pyplot as plt
import numpy as np
import cv2

image = cv2.imread('/content/picture.png')
template = cv2.imread('/content/penguin.png')
heat_map = cv2.matchTemplate(image, template, cv2.TM_CCOEFF_NORMED)

h, w, _ = template.shape
y, x = np.unravel_index(np.argmax(heat_map), heat_map.shape)
cv2.rectangle(image, (x,y), (x+w, y+h), (0,0,255), 5)

plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

Image: picture Template: penguin Result: detected

Detailed explanaition over here (my blog): simple-ai.net/find-and-replace-in-image

Answer 4

OpenCV has a Python interface that you could look at. If the characters, don't change too much you could try to use the matchTemplate function.

Here is their official tutorial on it (the tutorial is written using the C++ interface, but you should be able to get a good idea of how to use the function in Python from it).