Find video file position using an image match

Question

What would be the best tool / method to automatically find a position from an video file, where the frame approximately matches* a given image?

Basically, a command to: "Find the position where [image.jpg] appears in the [video.mpg]"

Preferably with ffmpeg, or some other linux command-line tool.

^{* - or maybe the frame which best of all of the frames in the video, matches the image}

Answer 1

2023 way to do this

ffmpeg  -i "video.mp4" -r 1 -loop 1 -i image.png -an -filter_complex "blend=difference:shortest=1,blackframe=90:32" -f null -

Notice the 90 there, if set to 100 it would only match equally, with lower numbers you get to set a fuzzy search criteria

Answer 2

framepos.sh is a bash script to search images in a video file, using ffmpeg and findimagedupes.

with the threshold variable, you can specify the fuzziness of the search.

we use the script with success to remove intros and outros from tv show episodes.

for "audiodetect" we use dejavu, and maybe pyAudioAnalysis Segmentation can be used.

relevant pseudocode ....

most complexity in the code is to run "extract" and "compare" in parallel, to improve speed and efficiency:

function framepos() {

    # extract frames. run in background
    # bmp format is faster than png or jpg
    $ffmpeg_cmd \
        $ff_args -i "$V" \
        ${tmp_pre}frame-%04d.bmp \
        2>/dev/null &
    pid=$!

    # output function for findimagedupes
    script_include=$(cat <<-'EOF'
        VIEW () {
            for f in "$@"
            do
                echo -n "$f"
                echo -ne "\t\t\t\t"
            done
            echo
        }
EOF
    )

    n2=0
    while true
    do
        n=$(ls ${tmp_pre}frame-*.bmp 2>/dev/null | wc -l)

        (( $n == 0 )) && {
            kill -0 $pid 2>/dev/null || {
                # extract done
                echo debug found no match >&2
                break
            }

            kill -SIGCONT $pid
            sleep $step_duration
            continue
        }
        (( $n == 1 )) && {
            # only one frame extracted.
            # if ffmpeg is still extracting, this file is incomplete
            kill -0 $pid 2>/dev/null && {
                # extract running
                kill -SIGCONT $pid
                sleep $step_duration
                continue
            }
            n2=1
        }
        (( 1 < $n && $n <= $frames_bufsize )) && {
            # frame buffer not full
            # if extract is running, then wait before compare
            kill -0 $pid 2>/dev/null && {
                # extract running
                kill -SIGCONT $pid
                sleep $step_duration
                continue
            }
            n2=$(( $n - 1 ))
        }
        (( $n > $frames_bufsize )) && {         #echo found $n frames
            # pause frame extraction to save space
            # extract is faster than compare
            kill -SIGSTOP $pid
            n2=$(( $n - 1 ))
        }

        echo compare $n2 frames
        break_while=false
        for I_cur in "${I[@]}"
        do
            # we need the "real path" for findimagedupes
            pattern=$(readlink -f "$I_cur")

            # call findimagedupes
            # to find "visually similar images"
            res=$(
                ls ${tmp_pre}frame-*.bmp \
                | head -n $n2 \
                | xargs findimagedupes -t $threshold \
                    -i "$script_include" "$pattern" \
                | grep -a "$pattern"
            )

            if [ ! -z "$res" ]
            then
                res=$(
                    echo "$res" \
                    | sed 's/\t\t\t\t/\n/g' \
                    | grep -v '^$' \
                    | grep -v "$pattern" \
                    | sort \
                    | head -n 1 \
                    | sed -E 's/^.*frame-(.*?)\.bmp$/\1/'
                )

                # get frame time
                # note: frame numbers start with 1
                # frame minus one:
                t=$(
                    echo $T1 $res $fps \
                    | awk '{printf "%.4f\n", $1 + ( ( $2 - 2 ) / $3 ) }'
                )
                # matching frame:
                #   | awk '{printf "%.4f\n", $1 + ( ( $2 - 1 ) / $3 ) }'

                # return
                echo $t

                # stop extracting
                kill -9 $pid 2>/dev/null

                # remove all temp files
                rm ${tmp_pre}frame-*.bmp

                break_while=true
                break
            fi
        done

        $break_while && break

        # remove processed temp files
        (( $n2 > 0 )) \
        && ls ${tmp_pre}frame-*.bmp | head -n $n2 | xargs rm
    done
}

Answer 3

You can use the md5 and framemd5 muxers if the image and frame are exactly the same. Example:

1. Get MD5 sum of target image:

   $ ffmpeg -i frame.jpg -f md5 - 2>&1 | grep MD5
   MD5=b7fb5124a65108ebb067129d9d81ed57
   

2. Find exact image frame in video:

   $ ffmpeg -i video.mov -f framemd5 - 2>&1 | grep b7fb5124a65108ebb067129d9d81ed57
     0,        62,        62,         1,  1424400, b7fb5124a65108ebb067129d9d81ed57
   

The output numbers refer to: stream_index, packet_dts, packet_pts, packet_duration, packet_size, MD5.

However, this is probably not what you're looking for. Conceivably this will work if video.mov is mjpeg; such as if you make a video from jpg inputs and stream copy them to the output:

ffmpeg -pattern_type glob -i "*.jpg" -codec copy output.mkv