Multimedia Quality Assessment

19 Pages·2010·4.93 MB·English

by  Pengpeng Ni

Multimedia Quality Assessment
Pengpeng Ni
Context
Simula Research Laboratory AS
2010-04-29
Content
✴
What is multimedia “quality”?
✴
How to measure multimedia “quality”?
✴
What are the difficulties and challenges?
Quality of Service requirements in distributed
multimedia systems
Network Quality Link Quality
Capacity Bitrate
Coverage Frame/Bit/Packet loss
Handoff Delay
User Perceived Quality
Speech fidelity
Audio fidelity
Image fidelity
Video fidelity
Quality of service (QoS) is defined in [ITU-T E.800] as the collective
effect of performance which determines the degree of satisfaction
of a user of the service.
Quality of Experience (QoE) :
The overall acceptability of an application or service, as
perceived subjectively by the end-user.
✴
QoE includes the complete end-to-end
system effects.
✴ Context
Over acceptability may be influenced by
user expectations and context.
Quality of Service vs Quality of Experience
✴
QoS: Value of the average userʼs experience richness
estimated by a service/product/content provider
- Service provider-centric concept
✴
QoE: Value (estimated or actually measured ) of a specific
userʼs experience richness.
- User centric concept
✴
QoE is the dual and extended view of QoS problem
Human centered multimedia services need measure/
predict QoE accurately.
How is the quality of multimedia experience
measured today?
✴
Subjective evaluation
- Mean Opinion Score (MOS) rating test
✴
Objective quality metrics, estimating
subjective MOS.
- MOS predictors based on fidelity measures,
examples include PSNR, SSIM, etc.
Subjective evaluation tests
✴
International recommendation for subjective
test methodologies:
-Recommendation ITU-R BT. 500-11 “Methodology for the subjective
assessment of the quality of television pictures” (1974-2002).
-Recommendation ITU-T P. 910 “Subjective video quality assessment
methods for multimedia applications” (1999).
-Recommendation ITU-R BT. 1788 “Methodology for the subjective
assessment of video quality in multimedia applications” (2007).
✴
Viewing condition, Test material, Test
procedure, Measurement scales and basic
statistics for data analysis.
Subjective Test Methodology I
Double Stimulus (DS) method
Explicit reference
5
 Impercep)ble
4
 Percep)ble
 but
 not
 annoying
3
 Slightly
 annoying
2
 Annoying
1
 Very
 annoying
Hidden reference
A B
Excellent
A B
Bad
Categorical or Numerical Scale
Subjective Test Methodology II
Single Stimulus (SS) methods
Categorical scale
5
 Excellent 5
 Impercep)ble
4
 Good 4
 Percep)ble
 but
 not
 annoying
3
 Fair 3
 Slightly
 annoying
2
 Poor 2
 Annoying
1
 Bad 1
 Very
 annoying
A 11-grade numerical categorical scale
shows best results in SS method