Have ffmpeg merge a matte key file over the normal video file removing the background

Question

I currently have two ffmpeg videos. Video one is a matte key file , that looks like so

Video two, would be the normal video without the background and just the person.

How would I go about merging these two videos in ffmpeg to have it look like this

I've tried the command

ffmpeg -i word.mp4 -i word.matte.mp4 -filter_complex "[0:v][1:v]alphamerge" -shortest -c:v qtrle -an output.mp4

but I get the following error

ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'word.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:06.24, start: 0.000000, bitrate: 741 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 738 kb/s, 25.79 fps, 25.79 tbr, 11040 tbn, 51.59 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'word.matte.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:06.19, start: 0.000000, bitrate: 493 kb/s
    Stream #1:0(und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 568x320 [SAR 1:1 DAR 71:40], 491 kb/s, 26 fps, 26 tbr, 13312 tbn, 26 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 (h264) -> alphamerge:main
  Stream #1:0 (mpeg4) -> alphamerge:alpha
  alphamerge -> Stream #0:0 (qtrle)
Press [q] to stop, [?] for help
[swscaler @ 0x5619d30b9900] No accelerated colorspace conversion found from yuv420p to argb.
[Parsed_alphamerge_0 @ 0x5619d2f57700] Input frame sizes do not match (1280x720 vs 568x320).
[Parsed_alphamerge_0 @ 0x5619d2f57700] Failed to configure output pad on Parsed_alphamerge_0
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #1:0

Any help would be appreciated. Here is a link to both video files, the matte file, and the normal video

Edit: Here is the example of this working. https://backgroundremover.app/video

Answer 1

Make a new mask to match the video size

Your mask is smaller than the video (568x320 vs 1280x720). Optimal solution is to make a new mask that matches the size and aspect ratio of the video. This will avoid the need to upscale the mask and will provide the best results. Avoid upscaling when possible.

If you have to use the original mask

You can use the scale2ref filter to upscale the mask to match the video:

ffmpeg -i video.mp4 -i mask.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge=shortest=1" -c:v qtrle output.mov

See the scale2ref filter documentation for more details.

Answer 2

I read through your error log and found this interesting line of text:
[Parsed_alphamerge_0 @ 0x5619d2f57700] Input frame sizes do not match (1280x720 vs 568x320).

In this error line, it says the sizes of both frames don't match, and thus, cannot be put together. By downscaling/upscaling one of your files to match the other's resolution, the videos should be put together with ease.