When it comes to assessing an audio system’s sound quality through listening, the human auditory system makes things extremely difficult. The sensation of hearing is affected not only by the audio system, but also by many other factors. These include the quality of the sound source, the acoustic environment, the listening position and angle, the individual’s hearing abilities, preferences, expectations, short term / long term memory and all the other human senses; sight, taste, smell and touch.

As the definition of quality is ‘conformance to requirements’, both a measurement - in the case of a listening session, the sensation of hearing - and a requirement are always needed to arrive at a quality assessment. One problem is that, for hearing sensations, the requirements are the listener’s preferences and expectations, formed by previous listening experiences. Since these experiences are affected by all the factors listed above, the quality assessment is valid only if they are exactly the same during the listening session, as well during as the previous listening experience. Any deviation will affect the outcome and make the assessment invalid.

The solution is simple: always compare systems, or multiple states of the same system, by ruling out the long term ‘aural scene’ memory and focusing on using short term ‘aural activity image’ memory. This can be done by using short clips - preferably shorter than the aural activity image memory of about 20 seconds. The clips have to be exactly the same - switching between system ‘A’ and ‘B’ while playing a song will not work because a song changes over time. For this reason it’s also virtually impossible to do a valid listening session with a live performance, because the performance itself may be the most significant change. Using short clips also makes it easier to ensure all non-audio sensory factors are consistent; for example keeping the temperature of the listening environment, the light intensity, smell etc, constant during the session.

Ruling out long term memory necessitates hiding the sound system components from the listener, i.e. performing a ‘blind’ test. Opinions from previous experiences - very probably having occurred under different listening conditions - can distort the assessment. For example, if the shape of loudspeakers reveals their brand and type, knowing the cost of the product may affect the assessment, favouring the more expensive loudspeaker. Blind testing also rules out non-audio cues from the equipment under test. For example, if we associate a red colour with warmth and a white colour with cold, a red loudspeaker might sound ‘warmer’ then a white one.

When we want to assess the sound quality of a dedicated component in an audio system, such as a pre-amplifier or a loudspeaker, it is important to know that we can never listen to only that component. We always listen to the whole audio system, which is reproducing an audio source. So, for a valid (fair) assessment of the system, we have to keep all other components fixed: play the same sound clip, apply the same equalisation, loudspeaker position, etc. A factor often underestimated is the system gain; the human hearing system can detect a level difference of down to 0.5 dB. The problem is that in most cases, ‘louder’ is most commonly perceived as ‘better’, so it is crucial that level differences are kept below 0.5 dB.

A final major factor is that of anticipation. The mere anticipation of a result can cause test subjects to experience the result, even if there is none. Because of this placebo effect, all clinical trials in the medical field include two groups of test subjects, one receiving the drug under test, and the other receiving the placebo.

The placebo effect also plays a role in listening tests - when a change is anticipated, even if there is no change, a change might somehow be detected. This is not a shortcoming of the listener, anticipation is simply one of the many factors that affect a hearing sensation. In fact, anticipation can significantly amplify the pleasantness of the hearing sensation - a concept used in many music compositions and performances.

To assess the quality of a sound system, however, anticipation should be avoided to prevent it from affecting the quality assessment. For this purpose, ‘ABX’ testing is an accepted method - featuring blind listening to two situations A and B, then confronting the test subject with an unknown situation X, which can be either A or B. Performing an ABX test multiple times gives a statistically significant statement on whether the difference could be detected or not. If a difference is not detected in an ABX test, then it may not make sense to compare the systems further.