Picture Quality Analysis System

PQA500

Features & Benefits

• Fast, Accurate, Repeatable, and Objective Picture Quality Measurement
• Predicts DMOS (Differential Mean Opinion Score) Measurement Based on Human Vision System Model.
• Picture Quality Measurements Can be Made on a Variety of HD Video Formats (1080i, 720p) and SD Video Formats (525 or 625)
• Makes Picture Quality Comparison across Different Resolutions from HD to SD, or HD/SD to CIF
• User-Configurable Viewing Condition and Display Models for Reference and Comparison
• Attention/Artifact Weighted Measurement
• Automatic Temporal and Spatial Alignment
• Easy Regression Testing and Automation with XML Scripting
• Multiple Results View Options
• Optional SD/HD SDI Interface for Generating / Capturing Video
• Pre-Installed Sample Reference and Test Sequences

Applications

• CODEC Design, Optimization and Verification
• Conformance Testing, Transmission Equipment and System Evaluation
• Digital Video Mastering
• Video Compression Services
• Digital Consumer Product Development and Manufacturing

Picture Quality Analysis System

The PQA500 is the latest-generation Picture Quality Analyzer built on Tektronix’ Emmy Award winning PQA200/300. Based on the concepts of the human vision system, the PQA500 provides a suite of repeatable, objective quality measurements that closely correspond with subjective human visual assessment. These measurements provide valuable information to engineers working to optimize video compression and recovery, and maintaining a level of common carrier and distribution transmission service to clients and viewers.

Compressed Video Requires New Test Methods

The true measure of any television system is viewer satisfaction. While the quality of analog and full-bandwidth digital video can be characterized indirectly by measuring the distortions of static test signals, compressed television systems pose a far more difficult challenge. Picture Quality in a compressed system can change dynamically based on a combination of data rate, picture complexity, and the encoding algorithm employed. The static nature of test signals does not provide true characterization of picture quality. A test scene with natural content and motion can be used, with human viewers reporting the results, but this method of evaluating the capabilities of a compressed video system is inefficient and not very objective. Subjective testing with human viewers is impractical for CODEC design and operational quality evaluation. The PQA500 provides a fast, practical, repeatable and objective measurement alternative to subjective evaluation of picture quality.

Human viewer testing has been traditionally conducted as described in ITU-R Rec. BT.500-11. A test scene with natural content and motion is displayed in a tightly controlled environment, with human viewers expressing their opinion of picture quality to create a Differential Mean Opinion Score, or DMOS. Extensive testing using this method can be refined to yield a consistent subjective rating. However, this method of evaluating the capabilities of a compressed video system can be inefficient, taking several weeks to months to perform the experiments. This test methodology can be extremely expensive to complete, and often the results are not repeatable. Thus, subjective DMOS testing with human viewers is impractical for the CODEC design phase, and inefficient for ongoing operational quality evaluation. The PQA500 provides a fast, practical, repeatable and objective measurement alternative to the subjective DMOS evaluation of picture quality.

System Evaluation

The PQA500 can be used for installation, verification and trouble-shooting of each block of the video system because it is video technology agnostic: any visible differences between video input and output from processing components in the system chain can be quantified and assessed for video quality degradation. Not only can CODEC technologies be assessed in a system, but any process that has potential for visible differences can also be assessed. For example, digital transmission errors, format conversion (i.e. 1080i to 480p in set-top-box conversions), 3-2 pull-down, analog transmission degradation, data errors, slow display response times, frame rate reduction (for mobile transmission and videophone teleconferencing), and more can all be evaluated, separately or in any combination.

How It Works

The PQA500 takes two video files as inputs: a reference video sequence and a compressed, impaired, or processed version of the reference. First, the PQA500 performs a spatial and temporal alignment between the two sequences, without the need for a calibration stripe embedded within the video sequence. Then the PQA500 analyzes the quality of the test video, using measurements based on the human vision system and attention models, and then outputs quality measurements that are highly correlated with subjective assessments. The results include overall quality summary metrics, frame-by-frame measurement metrics, and an impairment map for each frame. The PQA500 also provides traditional picture quality measures such as PSNR (peak signal-to-noise ratio) as an industry benchmark impairment diagnosis tool for measuring typical video impairments and detecting artifacts.

Each reference video sequence and test clip can have different resolutions and frame rates. The PQA500 can provide picture quality measurement between HD vs SD, SD vs CIF or any combination. This capability supports a variety of repurposing applications such as format conversion, DVD authoring, IP broadcasting, and semiconductor design. The PQA500 can also support measurement clips with unlimited sequence duration, allowing a full length movie to be quantified for picture quality through various conversion processes.

Prediction of Human Vision Perception

PQA500 measurements are developed from the human vision system model and additional algorithms have been added to improve upon the model used in the PQA200/300. This new extended technology allows legacy PQR measurements for SD while enabling predictions of subjective quality rating of video for a variety of video formats (HD, SD, CIF, etc.). It takes into consideration different display types used to view the video (for example, interlaced or progressive and CRT or LCD) and different viewing conditions (for example, room lighting and viewing distance).

A model of the human vision system has been developed to predict the macro-behavioral response to light stimulus with the following varying parameters:

• Contrast including supra-threshold
• Mean Luminance
• Spatial Frequency
• Temporal Frequency
• Angular Extent
• Temporal Extent
• Surround
• Eccentricity
• Orientation
• Adaptation effects

This model has been calibrated, over the appropriate combinations of ranges for these parameters, with reference stimulus-response data from vision science research. As a result of this calibration, the model provides a highly accurate prediction.

The following graphs are examples of scientific data regarding human vision characteristics used to calibrate human vision system modeling in the PQA500. Graph (A) shows modulation sensitivity vs. temporal frequency, and graph (B) shows modulation sensitivity vs. spatial frequency. The use of over 1400 calibration points supports high-accuracy measurement results.

The following picture (C) is a single frame from the reference sequence of a moving sequence, and picture (D) is the perceptual contrast map calculated by the PQA500. The perceptual contrast map shows how the viewer perceives the reference sequence. The blurring on the background is caused by temporal masking due to camera panning and the black area around the jogger shows the masking effect by the high contrast between the background and the jogger. The PQA500 creates the perceptual map for both reference and test sequences, then makes a perceptual difference map from them.

Comparison of Predicted DMOS with PSNR

In the example below, Reference (E) is a scene from one of the VClips library files. The image Test (F), has been passed through a compression system which has degraded the resultant image. In this case the background of the jogger in Test (F) is blurred compared to the Reference image (E). A PSNR measurement is made on the PQA500 of the difference between the Reference and Test clip and the highlighted white areas of PSNR Map (G) shows the areas of greatest difference between the original and degraded image. Another measurement is then made by the PQA500, this time using the Predicted DMOS algorithm and the resultant Perceptual Difference Map for DMOS (H) image is shown. It shows the greater perceptual difference with the highlighted white area. Using the human vision model of the PQA500 you can observe the areas of the image which the eye will observe as degraded. In this case the jogger in the image is not as noticeably degraded as the PSNR would have indicated.

Attention Model

The PQA500 also incorporates a new Attention Model to support the predicted human focus of attention. This model considers:

• The Motion of Objects
• Identifies People by Skin Detection
• Location
• Contrast
• Shape
• Size
• The Distraction of Noticeable Artifacts

These attention parameters can be customized to give greater or less importance to each function. This allows each measurement using the attention model to be user-configurable. The model is especially useful to evaluate the video process tuned to the specific application. For example, if the content is sports programming, the viewer is expected to have higher attention in limited regional areas of the scene. Highlighted areas within the attention image map will show the areas of the image drawing the attention of the eye.

Artifact Detection

Artifact Detection reports a variety of different changes to the edges of the image:

• Loss of Edges or Blurring
• Addition of Edges or Ringing/Mosquito Noise
• Rotation of Edges to Vertical and Horizontal or Edge Blockiness
• Loss of Edges Within an Image Block or DC Blockiness

They work as weighting parameters for subjective and objective measurements with any combination. The results of these different measurement combinations can help to improve picture quality through the system.

For example, artifact detection can help answer questions such as: “Will the DMOS be improved with more de-blocking filtering?” or, “Should less pre-filtering be used?”

If edge-blocking weighted DMOS is much greater than blurring-weighted DMOS, the edge-blocking is the dominant artifact, and perhaps more de-blocking filtering should be considered.

In some applications, it may be known that added edges, such as ringing and mosquito noise, are more objectionable than the other artifacts. These weightings can be customized by the user and configured for the application to reflect this viewer’s preference, thus improving DMOS prediction.

Likewise, PSNR can be measured with these artifact weightings to determine how much of the error contributing to the PSNR measurement comes from each artifact.

The Attention Model and Artifact Detection can also be used in conjunction with any combination of Predicted subjective or Objective measurements. This allows, for example, evaluation of how much of a particular noticeable artifact will be seen where a viewer is most likely to look.

Comprehensive Picture Quality Analysis

The PQA500 provides full-reference (FR) comparison between test and reference quality measurements and no-reference (NR) measurements on the luminance signal. Reduced reference (RR) measurements can be made manually from differences in no-reference measurements. The suite of measurements includes:

• Critical Viewing (Human Vision System Model-based, Full-reference) Picture Quality
• Casual Viewing (Attention Weighted, Full-reference or No-reference) Picture Quality
• Peak Signal to Noise Ratio (PSNR, Full-reference)
• Focus of Attention (Applied to Both Full-reference and No-reference Measurements)
• Artifact Detection (Full-reference, Except for DC Blockiness)
• DC Blockiness (Full-reference and No-reference)

The PQA500 supports these measurements through preset and user-defined combinations of display type, viewing conditions, human vision response (demographic), focus of attention and artifact detection, in addition to the default ITU BT-500 conditions. The user-configurable conditions capability helps to optimize CODEC parameters to be appropriate to the specific application and investigate what condition affects the picture quality measurement results by comparing the results under several measurement conditions. The user-defined measurement condition is set up by modifying pre-configured measurement sets or creating a new one, then saving and recalling the user-defined measurement set selected from the Configure Measure dialog menu.

Easy-to-Use Interface

The PQA500 has two modes: measurement and review. The measurement mode is used to execute the measurement selected in the Configure Dialog. During measurement execution, the summary data and map results are displayed on-screen and saved to the system hard disk. The review mode is used to view previously saved summary results and maps created either with the measurement mode or XML script execution. The user can choose multiple results in this mode and compare each result side by side using the synchronous display in Tile Mode. Comparing multiple results maps made with the different CODEC parameters and/or different measurement configurations enables easy investigation of the root cause of any difference.

Multiple Result Display

Resultant maps can be displayed synchronously with the reference and test video in a tiled or overlaid display. Individual videos can also be viewed at full resolution, one at a time, to accommodate resolutions greater than what the tiled display can accommodate. In Overlay Display, the user can control the mixing ratio with the fader bar, enabling co-location of difference map, reference and impairments in test video sequences.

Summary measures of standard parameters and perceptual summation metrics for each frame and over all frames are provided. Summary measure results are displayed as data lists, maps or graphs with a bar chart during video playback.

Error logging and alarms are available to help users efficiently track down the cause of video quality problems.

The logging parameters are:

• Registration information found in automatic temporal and spatial alignment: cropping, scale, shift in horizontal and vertical, Y gain and DC offset.
• Alignment confidence (cross-correlation coefficient): (1.0 is perfect match).
• Logs of when measurement values per frame exceed either warning or error levels (configurable by user via the summary node).

All results, data and graphs can be recalled to the display for critical examination.

Automatic Temporal / Spatial Alignment

The PQA500 supports automatic temporal and spatial alignment, as well as manual alignment.

The automatic spatial alignment can measure the cropping, scale and shift in each dimension, even across different resolutions (for example, by aligning SD to HD video). If extra blanking is present within the standard active region, it is measured as cropping when this function is enabled.

The automatic spatial and temporal alignment allows the picture quality measurement to be made among different resolutions and frame rates.

Automation Test with XML Scripting

In the CODEC debugging / optimizing process, the designer can repeat several measurement routines as CODEC parameters are revised. Automated regression testing with XML scripting can ease the restrictions of manual operation by allowing the user to write a series of measurement sequences within an XML script. Measurement results of the script operation can be viewed by using either the PQA500 user interface or any spreadsheet application that is able to read the created .csv file format as a summary. Up to four scripts can be executed simultaneously for faster measurement results.

Optional SD/HD SDI Interface

An optional SD/HD SDI interface enables both generation and capture of SDI video signals, including simultaneous generation and capture.

This allows the user to playout the reference video clips directly from the PQA500 into the device under test. The test output from the device can then be simultaneously captured by the PQA500. This saves the user from having to use an external video source to apply any required video input to the device under test. With this generation capability, files created by video editing software can be directly used as reference and test sequences for picture quality measurements.

The SD/HD SDI video option can generate SDI video from files in the following formats (all formats are 8–bit):

• .yuv (UYVY, YUY2)
• .rgb (BGR24)
• .avi (uncompressed, BGR24 / UYVY / YUY2)
• .vcap (created by PQA500 video capture)

	Supported Frame Geometry	Supported Format by SD/HD SDI Interface
SD-SDI	720x486, 720x576	525i/59.94, 625i/50
HD-SDI	1280x720, 1920x1080	720P50, 720P59.94, 720P60
		1080i/50, 1080i/59.94, 1080i/60
		1080p/23.98, 1080p/24, 1080p/25, 1080p/29.97, 1080p/30

Supported File Formats for measurement

All formats support 8–bit only:

• .yuv (UYVY, YUY2, YUV4:4:4, YUV4:2:0_planar)
• .rgb (BGR24, GBR24)
• .avi (uncompressed, BGR24 / UYVY / YUY2)
• ARIB ITE format (4:2:0 planar with 3 separate files (.yyy, .bbb, .rrr))
• .vcap (created by PQA500 video capture)

Pre-Installed Video Sequences

Vclips	1920x1088	YUV4:2:0 planar	V031202_Eigth_Ave, V031255_TimeSquare, V031251_Stripy_jogger
	1920x1080	UYVY	V031251_Stripy_jogger
	1280x720	UYVY, YUV4:2:0 planar	V031002_Eigth_Ave, V031055_TimeSquare, V031051_Stripy_jogger with 3/10/26 Mbps
	864x486	YUV4:2:0 planar	Converted V031051_Stripy_jogger with 2/4/7 Mbps
	320x180	YUV4:2:0 planar	Converted V031051_Stripy_jogger with 1000/1780/2850 kbps
PQA300 without Trigger	720x486	UYVY	Ferris, Flower, Tennis, Cheer with 2 Mbps_25 fps
	720x576	UYVY	Auto, BBC, Ski, Soccer
PQA300 with Trigger	720x486	UYVY	Mobile with 3/6/9 Mbps
	720x576	UYVY	Mobile with 3/6/9 Mbps

Related Products

Recent updates and more details are available in each data sheet.

MTS4EA Elementary Stream Analysis Software for VC-1, H.264/AVC, MPEG-2, MPEG-4, H.263 and 3GPP Standards

Features & Benefits

• Next Generation (VC-1, H.264/AVC, MPEG-4 & 3GPP) and Legacy (MPEG-2 & H.263) CODEC Support
• Frame-by-Frame and Block-by-Block Analysis to Allow Easy CODEC Comparison
• Easy-to-Interpret Detailed Graphical Displays (Requires User Installed Microsoft® Excel)
• Comprehensive Semantic Trace File Output to Determine Block-by-Block Encoder Decision Making
• AV Delay Measurement (option)
• Audio Decode and Analysis (option)
• Synchronized Audio and Video Analysis
• Real-time and Non-Real-Time Decoding and Analysis of Compressed Video Streams (Dependant on PC Performance)
• BitStream Editing
• Batch Mode to Allow Automated Testing
• YUV Decoded Video Output for Baseband Video Analysis and Picture Quality Analysis
• Extraction of Elementary Stream from Transport Stream
• Available as Single-User Local License for PC and Tektronix Instruments or Server Based Floating License.

Vclips - for Video Testing and Evaluation

Features & Benefits

Vclips are a diverse set of short video clips designed to test video encoders and decoders to the limits of their abilities.

Video Sizes	Test with many different video sizes; Sub-QCIF, QCIF, CIF, D1, HD (720p and 1080i)
Difficult Subjects	Test with fine detail, night time, areas of high contrast, sharp borders, uniform areas, bright and dull colors
Visual Objects	People, buildings, vehicles, trees, landscapes, clouds, water and synthetic objects
Movement	Fast, slow, uniform, random, multiple moving objects. Also pan, zoom and rotate
Test Card Sequences	Precisely defined motion, bright colors, dull colors, lines, patterns and grids. Also strobing and white noise