Video Compression Basics
- 2. Agenda
Introduction to Image/Video Compression
Still Picture compression
RGB to YCrCb
DCT
Entropy Coding
Video Compression
Motion Compensation
Compression Standards : JPEG, MPEG-1, MPEG-
II, H.264, MPEG4, WMV
2
- 3. VOB/MPEG-2
MP4, MOV, H.264
MPEG2/MPEG4
Movies
PSP :
MPEG2/MPEG4/H
Supports
.264
MP4/H.264
3
- 4. Why Compress/Encode ?
Digital color image: 352x288
pixel
RGB representation: 24 bpp
(8bits for red,green,blue)
Total amount of bytes:
> 300K
JPEG: common image
compression standard,
< 20K, similar quality
4
- 5. Why Compress/Encode ?
Output Data Rate[Kbits/sec]
Video Source
Quarter VGA @20 frames/sec 36 864.00
CIF camera @30 frames/sec 72 990.72
VGA @30 frames/sec 221 184.00
Transmission Medium Data Rate [Kbits/sec]
Wireline modem 56
GPRS (estimated average rate) 30
3G/WCDMA (theoretical maximum) 384
5
- 7. Steps in Image Compression
1. If the color is represented in RGB mode, translate it to
YCrCb.
2. Divide the file into 8 X 8 blocks.
3. Transform the pixel information from the spatial domain
to the frequency domain with the Discrete Cosine
Transform.
4. Quantize the resulting values by dividing each coefficient
by an integer value and rounding off to the nearest
integer.
5. Look at the resulting coefficients in a zigzag order. Do a
run-length encoding of the coefficients ordered in this
manner. Follow by Huffman coding.
7
- 8. RGB and YCrCb Color space
Original Original
R Y
G Cr
B Cb
8
- 9. Step 1: Converting RGB to YCrCb-I
YCrCb color mode stores color in terms of its luminance
(brightness) and chrominance (hue).
The human eye is less sensitive to chrominance than luminance.
Compression can be achieved by storing more luma details then
chroma details. This is called chroma-sub sampling.
Chroma sub sampling formats : 4:4:4, 4:2:1, 4:2:2, 4:1:1, 4:2:0
4:4:4 4:2:2 4:2:0
Y
Cb
Cr
9
- 10. Step 1: Converting RGB to YUV-II
It is simple arithmetic to convert RGB to YUV. One
example is:
Y = 0.299 * R + 0.587 * G + 0.114 * B
U = -0.1687 * R – 0.3313* G + 0.5 * B + 128
V = 0.5 * R – 0.4187 * G – 0.813 * B + 128
When conversion is done from RGB to YCrCb, it
makes 4:4:4 format, but 2 color components are
discarded. Thus bandwidth is reduced by 50%
10
- 11. Step 2: Divide into 8 X 8 blocks
With YCrCb color, we have 16 pixels of
information in each block for the Y
component (though only 8 in each direction
for the Cr and Cb components).
If the file doesn’t divide evenly into 8 X 8
blocks, extra pixels are added to the end and
discarded after the compression.
11
- 12. Step 3 : Transform to Frequency Domain :
Discrete Cosine Transform
The DCT transforms the data from the spatial
domain to the frequency domain.
The spatial domain shows the amplitude of the color
as you move through space
The frequency domain shows how quickly the
amplitude of the color is changing from one pixel to
the next in an image file.
For the 8 X 8 matrix of color data, we’ll getting an 8
X 8 matrix of coefficients for the frequency
components.
12
- 13. Step 3 : Time(Space) Domain and Frequency
domain
Time
Domain
Frequency
Domain
13
- 14. Step 3: Discrete Cosine Transform
Discrete cosine transform (DCT) is a Fourier-related
transform similar to the discrete Fourier transform (DFT), but
using only real numbers.
The most common variant of discrete cosine transform is the
type-II DCT, which is often called simply quot;the DCTquot;; its
inverse, the type-III DCT, is correspondingly often called
simply quot;the inverse DCTquot; or quot;the IDCTquot;.
The frequency domain is a better representation for the data
because it makes it possible for you to separate out – and
throw away – information that isn’t very important to human
perception.
The human eye is not very sensitive to high frequency changes
– especially in photographic images, so the high frequency
data can, to some extent, be discarded.
14
- 15. Step 3: Preparation for DCT
An Example 8x8 block of
pixel data is shown here >
The next step is to
transform the 8x8 matrix
from a positive range to
one centered around zero.
Since 8 bits are used for
each value, each value is
subtracted by 128.
The resultant block is
shown here ->
15
- 16. Step 3: DCT Calculation
For an N X N pixel image puv ,0 u N ,0 v N
the DCT is an array of coefficients
DCTuv ,0 u N ,0 v N where
(2 x 1)u (2 y 1)v
1 N 1 N 1
Cu Cv x 0 y 0 p xy cos
DCTuv cos 2 N
2N
2N
where
1
Cu C v for u , v 0
2
Cu Cv 1 otherwise
16
- 17. Step 3: Result of DCT
Original 8x8 Pixel block
Corresponding DCT
coefficient block
17
- 18. Step 3: Result of DCT
The DCT is lossless in that the reverse DCT will
give you back exactly your initial information
(ignoring the rounding error that results from
using floating point numbers.)
The values from the DCT are initially floating-
point.
They are changed to integers by quantization.
Because of the heavy calculations involved in
DCT process, embedded systems implement it in
hardware.
18
- 19. Step 4: Quantization
Quantization involves dividing each coefficient by
an integer between 1 and 255 and rounding off.
The quantization table is chosen to reduce the
precision of each coefficient to no more than
necessary.
The quantization table is carried along with the
compressed file.
19
- 20. Step 4: Quantization
Quantization Table
Resultant 8x8 DCT
coefficients after each
coefficient is divided by
corresponding
Quantization factor
20
- 21. Step 5: Entropy Coding
This is done so that the coefficients are in
order of increasing frequency.
The higher frequency coefficients are more
likely to be 0 after quantization.
This improves the compression of run-
length encoding.
Do run-length encoding and Huffman
coding.
21
- 22. Step 5a: Arrange in “zigzag” order
Zig Zag scan
Pattern
Resultant
data
22
- 23. Step 5b: Run Length Encoding
The Output of the Zigzag scan is further encoded
using Run Length Encoding procedure.
In Run Length Encoding, consecutive pixels with
the same value are encoded using a run-length and
value pair. E.g.
0x000x000x000x000x000x000x000
x000x000x000 ->
0x0A 0x00
23
- 24. Step 5c: Huffman Encoding
The Output of the RLE data is further
encoded using Huffman encoding
Huffman coding refers to the use of a
variable-length code table for encoding a
source symbol (such as a character in a
file) where the variable-length code table
has been derived in a particular way
based on the estimated probability of
occurrence for each possible value of the
source symbol.
An EOB (End Of Block) marker is put at
the end and the data is written to a file
24
- 25. Summary of Entropy Coding
Weighted scalar quantization:
loss of precision, few non-
zero coefficients are left
Zig-zag scan: non-zero
coefficients tend to be
grouped together
Run-Level encoding: encode each
coefficient value as a (run,level)
pair
- run: number of zeros
preceding values
- lenght: non-zero value
non-
25
- 27. Video Compression Means
Reduce redundancy and irrelevancy
Sources of redundancy:
spatial: nearby pixels often correlated (as
in still images)
temporal: adjacent frames highly
correlated
Irrelevancy:
Perceptually unimportant information
27
- 28. Commonly used terms - 1
To encode a frame each operation is performed at
macroblock (MB) level (nxn block of pixel. n=16 in
MPEG2)
Intra coded frame (I): every MB of the frame is coded
using spatial redundancy
Inter coded frame (P): most of the MBs of the frame are
coded exploiting temporal redundancy (in the past)
Bi-predictive frame (B): most of the MBs of the frame are
coded exploiting temporal redundancy in the past and in
the future.
Group of Picture (GOP): sequence of pictures between two
I-frames.
28
- 29. Commonly used terms - 2
Main idea: predict current frame using previously coded
one (reference frame)
For each MB motion information is extracted from current
and reference frame Motion Estimation (ME)
Temporal predicted frame is obtained from reference frame
using motion information estimated Motion
Compensation (MC)
The residual (original-MC) and motion information are
coded using transform coding like for intra frame
The prediction can be improved using frames in the future
(bi-directional prediction)
29
- 30. Motion Estimation/Compensation
The frames are partitioned in blocks of pixels (e.g. A block
of 16x16 as in MPEG)
Each block is predicted from a block of equal size in the
reference frame.
The blocks are not transformed in any way apart from
being shifted to the position of the predicted block. This
shift is represented by a motion vector.
The motion vectors of two neibouring MBs are not
independent. These are encoded differentially means only
the difference in motion vectors is encoded
One main disadvantage of block motion compensation is
that it introduces discontinuities at the block borders
(blocking artifacts).
30
- 31. Types of Motion Compensation
Frame Based Motion Compensation
Fixed Size Block Motion
Compensation
Variable Size Block Motion
Compensation
Object based Motion Compensation
Fixed Size
Variable Size
31
- 32. Frame Based Motion Estimation
Frame N Frame N+1
This approach is
applicable in camera
motions such as dolly
(forward, backwards),
track (left, right), boom
(up, down), pan (left,
right), tilt (up, down) and
-
roll (along the view axis).
It works best for still
scenes without moving
objects.
32
- 33. Fixed Size Block based Motion
Estimation-1
Most common approach
in use today
The current frame is
divided in blocks of fixed
size (e.g. 16x16 in
MPEG)
the maximum
displacement might be +-
64 pixels from a block's
original position
Various search strategies
like full search, grid
search etc. possible
33
- 34. Fixed Size Block based Motion
Estimation-2
Find the MB in the reference frame which is the most similar to the
current MB
To find optimal motion vectors, one basically has to calculate the
block prediction error for each motion vector within a certain search
range and pick the one that has the best compromise between the
amount of error and the number of bits needed for motion vector data.
Current MB
Reference frame Current frame
Search
window
34
- 35. Variable Size Block Based
Motion Compensation
In this approach, the
estimation is first
started with large
size block which are
then repeatedly
divided into smaller
blocks.
More efficient, but
very complex to
implement
35
- 36. Motion Vector
The Motion Vector is
having two
components
Horizontal and
Vertical
There can be
maximum of 2
motion vector in
standards like
MPEG, but can be 16
MVs in H.264
36
- 37. Object Based Motion Compensation
The latest standards like MPEG4 are based
on media objects. Media Object can be:
Still images (e.g. as a fixed background);
Video objects (e.g. a talking person -
without the background;
Audio objects (e.g. the voice associated
with that person, background music)
37
- 39. Multiple Reference Frames
The encoder can select for motion compensation among a
larger number of pictures previosly decoded and stored
Useful when dealing with motion that is periodic or in
presence of camera switching between 2 scenes
39
- 40. Bidirectional Prediction
it is possible to temporally predict MBs from both previous and future
coded frames.
The size of a B-Picture may be reduced to 10% of the actual picture
size
Previous Current frame Future frame
frame (B- frame)
40
- 43. Deblocking filter
Block artifact is one of the main disavantageof block-based coding
Latest Video Codecs like MPEG4/H.264 deploy an adaptive
deblocking filter to smoothen block edges
The strength of the filter is controlled by the value of several syntax
element
No Deblock
deblock
43
- 46. MPEG-4 (MP4) - 1
MPEG4 is the latest standard from MPEG group. In 2002, Apple
adopted it in QuickTime and then in iPod Video, making it highly
popular video standard.
MPEG4 absorbs most good features from MPEG2 and other related
standards and adds these new features like:
VRML(Virtual Reality Modeling Language) based 3D rendering
support
Video Objects based Motion Compensation
Support for DRM
Support for various type of interactivity.
All most popular video codecs available today like DivX, Xvid, Nero
Digital and 3ivx and by Quicktime 6 are based on MPEG4 part 2
Other most recent ones are Nero Digital AVC, Quicktime 7 and next-
gen DVD formats like HD DVD and Blu-ray Disc use MPEG4 part 10.
All high-quality multimedia on wireless devices like mobile phones,
use 3GPP and 3GPP2, are based on MPEG-4
46
- 47. MPEG-4 (MP4) - 2
Most of the features in MPEG4 are optional, means these may or may
not be followed by the developers. This means that there are probably
no complete implementations of the entire MPEG-4 set of standards.
To deal with this, the standard includes the concept of quot;profilesquot; and
quot;levelsquot;, allowing a specific set of capabilities to be defined in a
manner appropriate for a subset of applications.
Profiles describe somehow the content of an MPEG-4 object. E.g. the
phone call would use the Speech Audio Profile, while the orchestra
would require the High Quality Audio Profile.
MPEG-4 is patented proprietary technology. This means that,
although the software to create and play back MPEG-4 videos may
be readily available, a license is needed to use it legally in countries
that acknowledge software patents.
Patents covering MPEG-4 are claimed by over two dozen companies.
47
- 48. MPEG-7
Media tagging format for doing searches on arbitrary media formats via feature
formats
extraction algorithms
Visual descriptors such as:
Basic Structures
Color
Texture
Shape
Localization of spatio-temporal objects
spatio-
Motion
Face Recognition
Audio descriptors such as :
Sound effects description
Musical Instrument Timbre Description
Spoken Content Description
Melodic Descriptors (search by tune)
Uniform Silence Segment
Example application: Play a few notes on a keyboard and have matched song retrieved.
matched
48
- 50. Video Container Formats
A container format is a file format that can contain various types of
data, compressed by means of standardized audio/video codecs.
The container file is used to identify and interleave (multiplex) the
different data types.
Advanced container formats can support multiple audio and video
streams, subtitles, chapter-information, and meta-data (tags) - along
with the synchronization information needed to play back the various
streams together.
Example :
AVI (Audio Video Interleaved) from Microsoft. Very popular in
movies
MP4 (standard audio and video container for the MPEG-4
multimedia portfolio) - used by iPod and most mobile phones
3gp (used by iPod and most mobile phones)
MOV (standard QuickTime video container from Apple Inc.)
VOB : DVD Video OBject is a container based on DVD media. It
internally contains MPEG-2 codec.
50
DAT : a container format for popular with VCDs.
- 51. Conclusion
Media compression is indispensable even as
storage and streaming capacities increase
Future goals oriented towards increasing ease
of access to media information (similar to
google for text based information)
51