Can ffmpeg automaticaly align audio and video tracks

Question

I have searched and found many similar questions, but not this one. I've experimented and can't come up with an answer.

I have a Standalone "Digital TV Converter Box". This receives broadcast TV, and can act as a DVR. Mine is a "viewtv", but it's available from dozens of different vendors. It records combined video and audio (and optionally closed captions / subtitles) into a BDAV format file. I want to watch the program on other platforms, and sometimes edit the file and separate programs, and the BDAV format is inconvenient (to say the least). ffmpeg can easily perform the first part of converting the file into something more useable:

ffmpeg -hide_banner -err_detect ignore_err -loglevel verbose -stats -i "channel09202022-0858.mts" -map 0 -c copy "channel-09202022-0858.m2ts"

Here is where my frustration starts. There is usually a delay between the video and audio file, and it's different every time. MediaInfo reports something this:

Audio
ID                                       : 4352 (0x1100)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : 129
Duration                                 : 3 h 7 min
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Delay relative to video                  : -1 s 501 ms
Stream size                              : 258 MiB (9%)
Language                                 : English
Service kind                             : Complete Main

This delay can confuse some editing programs. If I cut segments out with ffmpeg the audio and video don't quite start in the same place. I would really like to have ffmpeg re-synchronize the files, but have not been able to do it. I've tried -vsync with various options (1, 2, vfr) -async with various options including -async 1 which the documentation seems to indicate will synchronize the beginning, -fflags +genpts+igndts in various combinations, -af aresample=async=1 , in every combination I can think of. In every single case, ffmpeg reads and writes with no reported errors: but the new file still has a "Delay relative to video" carried through to the new file. I've spend quite a lot of time on this, and all I have to show for it is severe aggravation.

The only thing I've found so far that comes close is to convert the file once, get the value of the delay, and then do this:

ffmpeg -err_detect ignore_err -hide_banner -loglevel info -stats -async 1 -itsoffset -1.501 -i "Story-09202022-0858.m2ts" -strict normal -map 0 -c:v copy -c:s mov_text -c:a aac -b:a 192k -ar 48000 Story-09202022-0858.mkv

This usually (but not always) re-encodes the audio in a way that does not show any significant delay between video and audio. However, it requires manually entering the values every time, and it must be exactly as I've shown it. Since -async is going to be "deprecated" (for no good reason I can see), I tried the so-called replacement of -af aresample=async=1 in this example and it did not work.

Can someone please tell me if there is a way to get ffmpeg to look at the information in a file that says what the delay is and then "fix" it? It seems very odd to me that there isn't a way to fix this synchronization between video and audio when the value of the off is is already known and available within the file.

If not, is there at least a way to obtain the value of the offset so I can write a command procedure (I normally use Windows for this, and have everything in BAT files) that will automate the procedure?

As I've said, I've searched through many forums, and all of the examples I've found have to do with audio and video being different tempos, or different durations, or similar problems. I've never seen an example of fixing the initial offset other than with -itsoffset.

If it matters, the ultimate output will be AVC (MP4) video and AAC audio. Unfortunately, just copying the audio in AC3 isn't going to work as there are some systems (such as DLNA servers and Roku Media Player) that can't handle it properly.

Thanks.

To respond to the question about ffprobe: you can get the delay, but the best that will happen is that you can direct the output to a text file. Then what do you do with it?

I prefer MediaInfo, which in this case will do the same thing:

mediainfo --Inform=Audio;%Video_Delay% "InputFile.m2ts

-666

(No, I'm not being flippant, that's the real delay on that particular file.)

Something similar could be done with ffprobe, but finding the exact command to return just one value is a royal pain.

Regardless, what would you do with a text file that just has one line with

-666

in it. How would you turn it into something that could go into the command line in a batch file?

I still maintain that since this data is in the input file, there ought to be a reasonable way to get ffmpeg to read it and use it.

I've been researching some additional possible commands, including -copyts, -async 1, variations on ignpts and genpts, -start_at_zero, and various -vsync options, and I can't find anything that will tell ffmpeg to use the delay to synchronize the audio and video streams.

Answer 1

A linux bash script to get the delay and use it in a further ffmpeg command is quite simple:

myscript.bash

#!/bin/bash

inputfile = $1
outputfile = ${inputfile%.m2ts}.mkv

delay = $(mediainfo --Inform=Audio;%Video_Delay% "$inputfile")

ffmpeg -err_detect ignore_err -hide_banner -loglevel info -stats \
       -async 1 -itsoffset $delay -i "${inputfile}" -strict normal \
       -map 0 -c:v copy -c:s mov_text -c:a aac -b:a 192k -ar 48000 \
       "${outputfile}"

And you would run it like this: myscript.bash "Story-09202022-0858.m2ts"

Answer 2

In a way, I've sort of given up on my original goal. However, I have found a reasonable work-around. I use a Windows batch file to process multiple files at a time. I use a two step process to change the way the various streams are stored in the container file, which has the side effect of fixing (more-or-less) the timing issue.

FOR /F "tokens=*" %%G IN ('dir /b *.mts') DO ffmpeg -hide_banner -err_detect ignore_err -loglevel info -stats -benchmark ^
 -i "%%G" -map 0 -c copy "%%~nG.m2ts"

FOR /F "tokens=*" %%G IN ('dir /b *.mts') DO ffmpeg -hide_banner -err_detect ignore_err -loglevel info -stats -benchmark ^
 -avoid_negative_ts auto -i "%%G" -i "%%G" -strict very -map 0:v -map 1:a -map 0:s:? -c:s copy -c:v copy -c:a copy "%%~nG-Step1.m2ts"

You are probably wondering why the two-step process. The MTS input file has one BDAV stream, which mixes everything together. ffmpeg, and some other video processors have problems interpreting this: Avidemux, for example, can't read it. So the first step is to do a copy to an M2TS container, which separates the streams into individual streams.

The second step then does a copy again. It reads the input file twice: the first contains the data, and reading it the second time is used to pick up the timing information from the audio stream. This appears to put things back together in synchronization, because after this I can use ffmpeg or Avidemux or anything else I've tried and the video and audio appear to be in sync.

Since the two processes are "just" copying files and not re-encoding them, they run through very fast. So far it has addressed the problem for me.