FFMpeg Video Overlay with Delayed Audio and Video in Background Video

Question

I am attempting to create the appearance of a picture in a picture video using multiple video files.

I've created a .mp4 file using FFMpeg. The video displays correctly in Windows "Movies & TV" app. It does not display correctly in Chrome, Firefox, VLC Media Player or any other app I have. I plan to display the video in a Browser, so cross browser support is important to me.

The video was created by taking two parts of a video call and doing the following with FFMpeg:

1. Put remote video (remote.mp4) in background.
2. The remote video should show the first image of the video and appear paused until about 2.501 seconds. I've done this using the itoffset.
3. The remote audio should also start playing after 2.501 seconds. Setting the itoffset did not seem to accomplish this so I added a "adelay" option to the filter_complex.
4. Placed local video (local.mp4) as an overlay in the bottom right corner.
5. The local video and audio should start playing at 0 seconds (immediately upon playing the video.)
6. Rotated the background picture 90 degrees (it was recorded sideways).
7. Add a audio delay to the remote video so it starts playing the audio stream 2.501 seconds after the start.
8. Merge the audio inputs.
9. Used aac codec (because it seems popular, I'm totally willing to switch if that would be useful.)
10. Use h264 video codec, again because it seems popular.

My FFMpeg command is as follows:

ffmpeg -itsoffset 2.501 -i remote.mp4 -i local.mp4 \
     -filter_complex \
     " [1:v]scale=iw/4:-1:flags=lanczos[loc0]; \
     [0:v]transpose=1[rotate1]; \
     [rotate1][loc0]overlay=main_w-overlay_w-10:main_h-overlay_h-10:eof_action=pass[rem0]; \
     [0:a]adelay=2501|2501[0adelay]; \
     [0adelay]apad[0a]; \
     [0a][1:a]amerge=inputs=2[a]" \
     -map "[rem0]" -map "[a]" \
     -ac 2 -vcodec libx264 \
     -ar 44100 -acodec aac \
     completed.mp4

Video Properties:

1. completed.mp4 (16 seconds, 480x640)
2. local.mp4 (16 seconds, 640x480)
3. remote.mp4 (14 seconds, 640x480, turned sideways)

When I look at the completed video in a browser, instead of being 16 seconds it says 18 seconds. When I use ffprobe on the completed video it says: Duration: 00:00:16.68. The local audio and video also waits 2.501 seconds to play in a browser.

Any help would be wonderful. I have no idea why the browser is rendering it differently than I intend. Thanks!

Answer 1

With timestamp offsets, ffmpeg will effect that through edit lists in the output MP4. Looks like browsers don't parse them. So, we need a workaround.

ffmpeg -i remote.mp4 -i local.mp4 \
     -filter_complex \
     " [1:v]scale=iw/4:-1:flags=lanczos[loc0]; \
     [0:v]transpose=1,setpts='if(eq(N,0),PTS,PTS+2.501/TB)',fps=30[rotate1]; \
     [rotate1][loc0]overlay=main_w-overlay_w-10:main_h-overlay_h-10:eof_action=pass[rem0]; \
     [0:a]adelay=2501|2501,apad[0a]; \
     [0a][1:a]amerge=inputs=2[a]" \
     -map "[rem0]" -map "[a]" \
     -ac 2 -vcodec libx264 \
     -ar 44100 -acodec aac \
     completed.mp4

The setpts shifts timestamps of all frames except the first. The fps filter then fills in that gap with duplicates of the first frame. I've assumed an input stream rate of 30.