Audiophile HomePod Reviewer Turns Out to Not Know Much about Measuring Audio

The much touted review of the HomePod posted by an “audiophile” on Reddit last week – and gleefully tweeted by Apple’s Phil Schiller – turns out to be a long mess of uninformed and poorly made measurements.

This reply on Reddit highlights many of the problems, notably the fact that the HomePod wasn’t measured in an anechoic room, but mainly the fact that the “reviewer” fudged the display of his graphs, making them look better than they were.

Here’s one of the original graphs:

LmAel7t

The experimenter seems obsessed with that graph which they claim shows a very flat frequency response. They even say, further down the review, that it’s an “almost perfectly flat speaker”. Mmm. I opened that same measurement in REW and here’s what I get (with the same 1/12 octave smoothing as the above image):

3nHZimq

Doesn’t look as nice doesn’t it? That’s because of the scale, you see. It’s the ages-old trick of messing with the vertical scale to make things look flatter than they really are. In the screenshot that the experimenter posted, the interval between ticks is 10 dB. That’s enormous. Almost anything will look almost flat at that scale.

This is why it’s wrong to assume that some random guy who writes 5,000 words and includes a bunch of numbers and graphs knows what he’s doing. Another comment from the comment I linked to above:

I find it absolutely hilarious that the experimenter is specifying conditions like “Room temperature was 72ºF (22.2ºC) and the humidity outside was 97%. Air Pressure was 30.1 inHg (764.54 mmHg)”. It sounds like they’ve done very rigorous measurements in highly controlled conditions, but that’s rendered moot by the overwhelming influence of the specific room in which they made the measurements.

Finally:

Conclusion: no, these measurements don’t show that “The HomePod is 100% an Audiophile grade Speaker”, far from it. Because the measurements were made in a reverberant room without windowing, the data is mostly meaningless. The linearity, SPL and distortion measurements are usable to some extent, but these are not the most important criteria when assessing the audio quality of a loudspeaker (unless loud bass is really important for you). Many parts of the “review” are misleading, at times egregiously so, leaving the impression that the experimenter is interpreting the data through Apple-colored glasses.

I wonder if Phil Schiller had anyone from Apple’s audio team look at the original “review” before tweeting it. My guess is no; they would have spotted the incorrect measurements, and warned him not to share it. It makes Apple look bad, because of Schiller’s sharing it, now that it has turned out to be quite wrong.

10 thoughts on “Audiophile HomePod Reviewer Turns Out to Not Know Much about Measuring Audio

  1. I am not normally a fan of frequency response being used to prove a speaker’s worth, that would be a different and much longer discussion. However, with HomePod you have a design claim of self calibration for room placement. Measuring the device in a room is reasonable. A comparison of response for different room placements would be even better.

  2. Fast Company published a review of the HomePod today. They used all kinds of acoustics testing gear and software provided by NTi Audio AG, and tested the HomePod in a real-life natural habitat. (Consumer Reports DID NOT do this kind of scientific testing of the HomePod, nor did they listen to it in a “natural” environment, or set it in a space where the audio beamforming could be reflected, unimpeded).

    The entire article is worth reading, and includes testing of the HomePod’s audio consistency:
    “After measuring the the full frequency spectrum (6.3Hz through 20,000Hz) of the HomePod’s sound output from each location… The average variance for all audible frequency bands was less than a decibel–0.95 decibel.” [Any variance less than 1 decibel is undetectable by human hearing]

    The article ends with:
    “It’s true that the speakers and other components inside the HomePod are of very high quality. But it’s those algorithms–the intelligence of the speaker itself–that back up the claim that the HomePod is the best-sounding smart speaker on the market.”

  3. I don’t understand some of your criticism of that report. I get that “old” speakers do not compensate for the room they are in, so audiophiles may want to measure them in an Anechoic chamber (for repeatable results). But these speakers are very different. They listen to the room and make adjustments. Putting them in an anechoic room would be neutering them, and the result is not interesting.

    These speakers cry for a new testing methodology, that is done in real rooms. It makes it hard to replicate from place to place, but that is in fact the point. New testing is required for new paradigms.

    Now I am not defending the whole report, just that part of it.

    • No, you can’t compare things in “real” rooms, because of the effect of the room. You have to start in a totally neutral environment to see how the speaker performs.

      Also, as I’ve said several times, given the speaker’s DSP, it’s not clear if that DSP would be active on single tones (a sine wave sweep) so any such test is a bit specious. Since the music you play does have DSP applied, then a test of isolated tones won’t exhibit the same frequency response as actual music, which is more complex, and which requires that the DSP react.

  4. I have connected an almost full 8 TB WD external hard drive to a brand new iMac running High Sierra OS .The WD is formatted HFS+ and the iMac recognizes the drive.However,when I access the WD in ITunes or try to search the media and jpeg files content with Spotlight,the iTunes search and Spotlight features don’t work properly. If I reformat the WD to APFS do the WD’s files stay on the drive intact or get erased. Also is reformatting the solution to the glitches with itunes and Spotlight? Or to I have to reindex the WD 8 terabyte drive?

  5. “notably the fact that the HomePod wasn’t measure in an anechoic room, …”

    That’s not a shortcoming, that is the correct way to talk about HomePod. Consumer Report did it – which amounts to testing HomePod in an environment where no consumer will ever operate this device.

    It’s like testing a concert hall by first removing all the reflective devices installed to improve the acoustics.

    • No, it’s not. You have to have benchmark settings for how it performs in optimal conditions before you can start testing it in real world conditions; this is how speakers are always measured.

  6. I agree with Mark_i, and disagree with Kirk, on the question of testing criteria for audio products. All testing is reductionist, reducing the number of variables encountered in actual usage situations to a minimum number. The problem is that the reduction of variables too often removes elements that are critical to judging the performance of the product in its desired usage environment. It is not sufficient to say “we’ve always done it this way.” With increasing frequency, we need to increase the complexity of the tests we use, to match the increasing complexity of our technology and usage patterns.

    You are probably aware of reviews of digital cameras, in which standard tests may show two cameras as very similar, while our perceptions show a big difference. The same is true for audio equipment. In the USA, the standard test for home insulation is conducted in a carefully controlled thermal chamber at 75 degrees Fahrenheit. This test has been in use for decades, and is specified by various laws and regulations. Yet it is a poor test. No home needs insulation at this temperature, and many products perform in different and unexpected ways in cold (or hot) weather.

    Another concern is the ease of cheating on simple standard test protocols, which manufacturers use to their advantage. VW’s diesel engines performed very well on the standard tests, because it was easy to cheat the simplistic test standards. Similar “testing anomalies” show up in audio equipment tests. The common audio tests have no chance to evaluate meaningfully which aspects of Apple’s claimed HomePod performance are fake, and which are an advancement in sound reproduction technology. In the USA, there are thousands of test protocols specified by ASTM standards. If you ask an engineer what ASTM stands for, the reply may be the standard joke, “Another Stupid Test Method”.

    We need good, realistic testing standards, and the emphasis needs be on “realistic”, rather than the audio industry’s self-satisfied insistence that they are “standard”. The human ear/brain combination is both extremely subtle and very easily fooled. The development of better and more sophisticated tests in evaluating audio equipment is long overdue.

    • You have to start with basic benchmarks to know how things perform in optimal conditions. You can later do other benchmarks, but the only way you can compare devices is if you can test them in the same settings, and if anyone can test them in similar settings.

Leave a Comment