The world has changed in many ways over the past two years. Technology, riding on the back of various events, has led to many surprising trends. One such trend is the tremendous popularity of video services, encompassing everything from Over the Top (OTT Services) to Video Conferencing. For instance, the number of subscribers of the OTT services has increased significantly – Netflix gained more than 50 million subscribers, while Prime Video gained more than 75 million subscribers as of 2021. And indeed, much of this is due to the frenzy of pandemic-induced quarantine.
The one thing without which this could not have been possible at all is the Video Codecs. A Video codec helps compress a video file, making it easier to manage the file (storing, transmitting, distributing, etc.). To put things into perspective, a 1-hour video file at 720p resolution without video compression would require 298 GB of space to store it, a bandwidth of 298.6 GB to transmit it, and a time of around 6.6 hrs (seems like a lot of work, right?)
As interesting as the technology, the history of video codecs is equally intriguing. Historically, video files were stored as an analog signal on a magnetic tape. Around the 1970s, the video transmission resolution was approx. 486*440@30fps, which started with a bitrate of 140 Mbits/s.However, the modern-day transmission of 854*480@30 fps is possible at 6 Mbits/s, which is a whopping 95% reduction in the bandwidth/storage requirements. That is the power of Codecs. In this series of Video Codec articles, we will uncover the numerous technologies that made the visions of countless people a reality.
This article focuses on video codec first principles. The objective of this article is to make you familiar with the basic terms of the domain so that you are comfortable reading any material related to the domain. Let’s get started with the basic terminologies.
Image
As we know it, an image is simply a visual representation of something. It could be a photograph of a scene as an example. However, technically a digital image is a collection of various small boxes (called pixels).
Digital images are made up of a layer of pixels. If we want to add colors to the picture, we can simply use layers of primary colors – Red, green & blue, and voila!
Pixel
As explained above, pixels are the building blocks of an image, and they represent the intensity (usually in numerical value) of a given color. The range of colors that a pixel can represent depends upon the size of each pixel (which is equivalent to the amount of information contained in the pixel). The various examples below explain this in detail.
Colour images generally contain pixels with values from 0 to 256 in all 3 channels (Red, Green, and Blue). For example, a green pixel means a maximum of green, 0 of blue, and 0 of red. The pink color pixel can be defined as a mixture of the three colors. If we define color intensity from 0 to 255, the pink pixel would be defined as Red=255, Green=192, and Blue=203.
Resolution
Resolution is a qualitative measure of an image described as the number of pixels in one dimension. It is generally defined as width × height.
Video
A Video is a sequence of images that are played at high speeds. This produces an optical illusion that the objects (or people) on the screen are moving.
Generally, these images are called frames in video coding context and are played such that we see 30, 60, 90, etc., frames in one second. This is called frame rate, usually expressed in fps (frames per second).
Also, the amount of data needed to show this video per second is called the bitrate. It is defined as bit rate = width * height * bit depth * frames per second. For example, a 30-fps movie with 24 bits per pixel and a resolution of 480×240 will likely need 82,944,000 (approx.) bits per second or 82.944 Mbps (30x480x240x24) if we don’t employ any compression.
The Big Question
After knowing the basics, the next big question that comes into our mind is why the hype about codecs and why do we need compression. A Video codec reduces redundancy and irrelevancy in the video frames (or images) and then stores the file. As explained earlier, the raw video files are massive, making it very difficult to manage them.
This brings us to the following question: where the redundancy is, and how it is removed. So, there are two major sources of redundancy in a video file, and they are classified as Spatial and Temporal redundancy and are briefly described below.
Spatial Redundancy: If we observe a frame from any video, we can see that groups of nearby pixels are often correlated. We can save some bitrate if we use this to our advantage (as is generally done in I frames). This is called redundancy in space (Spatial Redundancy).
Temporal Redundancy: Similar to Spatial redundancy, temporal redundancy is also about correlation, but in time. If we observe two or more successive frames from any video, we can see that some of the areas in the images stay the same for some number of frames. One can harvest major savings in bitrate by reducing this kind of redundancy (as is generally done in p frames).
Frame Types
Our journey to understand the basics of Video Coding is not quite complete if we don’t talk about the various types of frames present in a compressed video. To put things into perspective, when the individual frames of a video file are compressed by reducing the redundancy, these frames are called encoded frames. These encoded frames can be of three types, depending upon how the redundancy is removed. These frames are:
Intra-coded picture (I frames)
Predicted picture (P frames)
Bidirectional predicted picture (B frames)
Let’s say we have a 30fps movie, with the first four frames as below
To encode these first 4 frames. We will do as follows:
1. I frames: The first frame will be kept as a whole. This frame would contain all the information, and some compression can be obtained by removing the spatial redundancy.
Going back to the way we store data for a frame, imagine that a specific area of an image is pink (color intensities = 255 for the red channel, 192 for the green channel, and 203 for the blue channel). This would be stored in a raw frame as
There are 18 pixels represented by 432 bits of memory. One straightforward way to reduce this redundancy would be to keep the color information of 1 pixel and refer to that pixel for color information (This would increase processing needs but reduce storage requirements). In this case, let’s assume that we keep the color information of the first pixel, which lies in the first row of the first column (Location = 1,1)
*Note: The above representation is exemplary, and the actual implementation may vary.
Thus, effectively representing the same information with just 58 bits (achieving 86.5% compression for this region).
The frames encoded in such a way are called I frames. These frames are self-contained and do not rely on other frames to be decided/rendered. As you can imagine, an I-frame looks similar to a static photo.
2. P frames:
We will remove temporal redundancy to encode the second frame. By keeping the first frame whole, we have a starting point from where we can predict other frames without the need for much information on the second frame. Since we are referencing frames that are already decoded, this process is also referred to as referencing the past frames.
In this case, we will need the information about what has changed from the first frame to the second frame. For example, if only some specific portion of an image changes from frame 1 to frame 2, we can keep the portion that has changed in frame 2 and discard everything else (essentially subtracting the first frame from the second frame).
If we need the discarded portion, we can always refer to the I frame to get this information. The frames resulting from this “virtual” subtraction are called P frames.
A p frame consumes significantly less space than I frame and provides significant compression and bitrate savings. However, it always needs I frame to be rendered.
3. B frames: We now know that p frames refer to the past frames and provide excellent compression. What if we refer to future and past frames for better compression? This concept is used in B frames. To specify its content, it uses differences between the current frame (B-frame) and preceding and following I-frames
History of Patent Filing
The readers would be surprised to note that the patents related to Video Coding (IPC = H04N19) started being filed as early as 1982. The most noteworthy early birds are the US5936945A from Hitachi Ltd, EP538013B1 from SONY CORPORATION, and Tokyo EP558377B1 from ALCATEL BUSINESS SYSTEMS, etc.
The US5936945A from 1991 discloses a video teleconference module that consists of a camera and display and has video codecs built into the system for coding and decoding video signals according to the JPEG algorithms. This system is intended for video conferencing, which itself got popularised around 2004.
EP538013B1, also from 1991, claims a similar device to transmit digital images (moving or still) by primitive compression. The coding of these pictures happens in a series of steps that include sampling Analog videos at several different clock phases to produce pictures stored in memory and compressing them with an orthogonal transform circuit.
EP558377B1 from 1992 describes a Video Codec for similar video telephone applications that receive video signals coming from a camera and, after compression, transmit them to a remote video decoding unit through phone lines. The codec described in the claim consists of a DCT operator and quantizer followed by a loop-type digital low-pass filter. Astonishingly, these steps are still used in any video coder/decoder.
Bonus Section
Encoding properties of videos from various devices
Since we have learned so much about the basics of encoded videos, I think it will be fascinating for the users to look at the encoded videos generated from various devices.
Apple iPhone 13 pro: 1080p 60fps (Duration: 6.160 seconds)
Codec: H.265
Resolution: 1920*1080
Frame rate: 59.94 fps
Number of I frames: 6
Number of P frames: 72
Number of B frames: 230
Notice that the encoding process, in this case, follows a pattern where each 1-sec interval has 1 I frame and 3 B frames follow each p frame. The height of the bar represents the size of that frame.
Samsung S21 Ultra 5G: 1080p 60fps (Duration: 07:881)
Codec: H.265
Resolution: 1920*1080
Frame rate: 60.02 fps
Number of I frames:8
Number of P frames: 465
Number of B frames: NA
It is fascinating to note that there are no B frames in this video, but the p frames seem to be optimized to take significantly less space and improve the coding efficiency.
OnePlus 7T: 1080p 60fps (Duration: 6.263 seconds)
Codec: H.265
Resolution: 1920*1080
Frame rate: 60.03 fps
Number of I frames: 7
Number of P frames: 201
Number of B frames: 168
Notice that the encoding process, in this case, has produced a comparatively higher number of p frames (some of these are equal or bigger compared to I frames) as compared to B frames. Thus, contributing to an overall bigger encoded file.
This section concludes this article. I hope it was an amazing journey for you in the Video Compression world. If there is something that you wish me to include in the article, or if you want me to analyze videos from other devices, please send a mail to marketing@lumenci.com. I will soon publish another blog that will describe how video encoding works and the various steps involved, including;
Picture partitioning
Predictions
Transform
Quantization
Entropy Encoding
Till then, stay tuned and keep shooting and streaming!
*Disclaimer: This report is based on information that is publicly available and is considered to be reliable. However, Lumenci cannot be held responsible for the accuracy or reliability of this data.
Editorial Team at Lumenci
Through Lumenci blogs and reports, we share important highlights from the latest technological advancements and provide an in-depth understanding of their Intellectual Property (IP). Our goal is to showcase the significance of IP in the ever-evolving world of technology.
