Which Painting Do You Look Like? Comparing Faces Using Python and OpenCV

Many years ago, as I was wandering around the Louvre, I came across a painting which bore an uncanny resemblance to me!

Spooky, eh?

Yeah, yeah, it's not the greatest likeness ever, but people who know me seem to think I look like the chap on the left.

This got me thinking... Wouldn't it be great if when you entered an art gallery, a computer could tell you which painting you look most like?

Well, I think it would be great. This is my blog, so what I say goes!

Getting The Data

I'm using the Tate's Open Data Set to grab scans of all their artwork. ~60,000 images in total.

Finding Faces

Not all paintings are of people. Some artsy types like to paint landscapes, dogs, starry nights, etc.

Using Python and OpenCV, we can detect faces in paintings. Then crop out the face and save it. The complete code is on GitHub - but here are the essentials.

import sys, os
import cv2
import urllib
from urlparse import urlparse

def detect(path):
    img = cv2.imread(path)
    cascade = cv2.CascadeClassifier("haarcascade_frontalface_alt.xml")
    rects = cascade.detectMultiScale(img, 1.3, 4, cv2.cv.CV_HAAR_SCALE_IMAGE, (20,20))

    if len(rects) == 0:
        return [], img
    rects[:, 2:] += rects[:, :2]
    return rects, img

def box(rects, img, file_name):
    i = 0   #   Track how many faces found
    for x1, y1, x2, y2 in rects:
        print "Found " + str(i) + " face!"  #   Tell us what's going on
        cut = img[y1:y2, x1:x2] #   Defines the rectangle containing a face
        file_name = file_name.replace('.jpg','_')   #   Prepare the filename
        file_name = file_name + str(i) + '.jpg'
        file_name = file_name.replace('n','')
        print 'Writing ' + file_name
        cv2.imwrite('detected/' + str(file_name), cut)   #   Write the file
        i += 1  #   Increment the face counter

def main():
    #   all.txt contains a list of thumbnail URLs
    for line in open('all.txt'):
        file_name = urlparse(line).path.split('/')[-1]
        print "URL is " + line

        if (urllib.urlopen(line).getcode() == 200):
            #   Download to a temp file
            urllib.urlretrieve(line, "temp.jpg")
            #   Detect the face(s)
            rects, img = detect("temp.jpg")
            #   Cut and kepp
            box(rects, img, file_name)
        else:
            print '404 - ' + line

if __name__ == "__main__":
    main()

We now have a directory of files. Each file is a separate face. We assume that no two faces are of the same person - this is important for the next stage...

Building Eigenfaces

Imagine that a picture of your face could be represented by a series of properties. For example

• How far apart your eyes are.
• Distance from nose to mouth.
• Ratio of ear length to nose width.
• etc.

That is, in grossly simplified terms, what an Eigenface is.

If I have a database of Eigenfaces, I can take an image of your face and compare it with all the others and find the closest match.

We'll split this process into two parts.

Generate the EigenFaces

We need the arrange the images so that each unique face is in its own directory. If you know that you have more than one picture of each person, you can put those images in the same directory.

E.G.

   |-path
    -|-Alice
      | |-0.jpg
      | |-1.jpg
      |
      |-Bob
      | |-0.jpg
      |
      |-Carly
      ...

This code is adapted from Philipp Wagner's work.

It takes a directory of images, analyses them, and creates an XML file containing the Eigenfaces.

WARNING: This code will take a long time to run if you're using thousands of images. On a dataset of 400 images, the resulting file took up 700MB of disk space.

import os
import sys
import cv2
import numpy as np

def normalize(X, low, high, dtype=None):
    """Normalizes a given array in X to a value between low and high."""
    X = np.asarray(X)
    minX, maxX = np.min(X), np.max(X)
    # normalize to [0...1].
    X = X - float(minX)
    X = X / float((maxX - minX))
    # scale to [low...high].
    X = X * (high-low)
    X = X + low
    if dtype is None:
        return np.asarray(X)
    return np.asarray(X, dtype=dtype)

def read_images(path, sz=None):
    X,y = [], []
    count = 0
    for dirname, dirnames, filenames in os.walk(path):
        for subdirname in dirnames:
            subject_path = os.path.join(dirname, subdirname)
            for filename in os.listdir(subject_path):
                try:
                    im = cv2.imread(os.path.join(subject_path, filename), cv2.IMREAD_GRAYSCALE)
                    # resize to given size (if given)
                    if (sz is not None):
                        im = cv2.resize(im, sz)
                    X.append(np.asarray(im, dtype=np.uint8))
                    y.append(count)
                except IOError, (errno, strerror):
                    print "I/O error({0}): {1}".format(errno, strerror)
                except:
                    print "Unexpected error:", sys.exc_info()[0]
                    raise
            count = count+1
    return [X,y]

if __name__ == "__main__":
    if len(sys.argv) < 1:
        print "USAGE: eigensave.py "
        sys.exit()
    # Now read in the image data. This must be a valid path!
    [X,y] = read_images(sys.argv[1], (256,256))
    # Convert labels to 32bit integers. This is a workaround for 64bit machines,
    y = np.asarray(y, dtype=np.int32)

    # Create the Eigenfaces model.
    model = cv2.createEigenFaceRecognizer()
    # Learn the model. Remember our function returns Python lists,
    # so we use np.asarray to turn them into NumPy lists to make
    # the OpenCV wrapper happy:
    model.train(np.asarray(X), np.asarray(y))

    # Save the model for later use
    model.save("eigenModel.xml")

After that has run - assuming your computer hasn't melted - you should have a file called "eigenModel.xml"

Compare A Face

So, we have a file containing the Eigenfaces. Now we want to take a photograph and compare it to all the other faces in our model.

This is called by running:

python recognise.py /path/to/images photo.jpg 100000.0

The "100000.0" is a floating-point number which determines how close you want the match to be. A value of "100.0" would be identical. The larger the number, the less precise the match.

import os
import sys
import cv2
import numpy as np

if __name__ == "__main__":
    if len(sys.argv) < 4:
        print "USAGE: recognise.py  sampleImage.jpg threshold"
        print "threshold is an float. Choose 100.0 for an extremely close match.  Choose 100000.0 for a fuzzier match."
        print str(len(sys.argv))
        sys.exit()

    # Create an Eign Face recogniser
    t = float(sys.argv[3])
    model = cv2.createEigenFaceRecognizer(threshold=t)

    # Load the model
    model.load("eigenModel.xml")

    # Read the image we're looking for
    sampleImage = cv2.imread(sys.argv[2], cv2.IMREAD_GRAYSCALE)
    sampleImage = cv2.resize(sampleImage, (256,256))

    # Look through the model and find the face it matches
    [p_label, p_confidence] = model.predict(sampleImage)

    # Print the confidence levels
    print "Predicted label = %d (confidence=%.2f)" % (p_label, p_confidence)

    # If the model found something, print the file path
    if (p_label > -1):
        count = 0
        for dirname, dirnames, filenames in os.walk(sys.argv[1]):
            for subdirname in dirnames:
                subject_path = os.path.join(dirname, subdirname)
                if (count == p_label):
                    for filename in os.listdir(subject_path):
                        print subject_path

                count = count+1

That will spit out the path to the face that most resembles the photograph.

Who Am I?

Well, it turns out that my nearest artwork in the Tate's collection is...

Sir John Drake!

So, there you have it. My laptop isn't powerful enough to crunch through the ~3,000 faces found in The Tate's collection. I'd love to see how this works given a powerful enough machine with lots of free disk space. If you fancy running the code - you'll find it all on my GitHub page.