Headphone reviews: Why we test

Headphone reviews can be tough to navigate. Compared to other gadgets, this product category is probably the most rife with compromised ethics, bad testing methodology, and simply untrue statements — whether they’re simply mistakes or simple parroting of marketing copy. Though standardized testing has started to proliferate to more and more sites, clarity surrounding testing headphones hasn’t improved as much as it should have. Competing target curves, confusing terminology, and unclear assessments of the same products are enough to make anyone’s head spin.

We’ve been measuring products since 2018 to provide a more objective picture of what’s going on with the products you buy. But why? Why do we spend so much time to generate some charts and numbers for something that most may not even bother to read? In short, we do it to show our work, and not leave you to take our word for it.

Editor’s note: this article was updated on January 9, 2024, to ensure timeliness of the information within, and to add information surrounding Multi-Dimensional Audio Quality Scores.

Reviews often don’t tell you what you want to know

Much like any other study, approaching the problem of “which headphones do I buy” requires a certain philosophy to guide you. Because your life will dictate different needs than what others have, there is no one “best” set of headphones—only what meets your needs. Before you read a review, you should be looking to answer a few specific questions:

• Do the headphones have the features you need?
• What can you do without?
• What kind of sound do you like?
• Are there any unsavory surprises waiting for you?

While you can read about the first two just about anywhere, it’s that last one that most sites trip up on because it’s very difficult—and very expensive—to do it right. For this reason, a figurative monsoon of content that lacks credibility or substance clogs forums, Amazon pages, and blogs that review without any sort of verification of fact.

Reading these reviews is not only a chore, but often worse than useless. I say “worse” because many straight-up lie. They could be written by a marketer for all you know! This happens all the time.

On top of that, the way that many people describe sound is also misleading. Many of these reviews will contain overly-flowery prose rife with unquantifiable adjectives. The surest sign that a review is a waste of your time is if you’re reading lots of descriptions of sound that can’t be supported with numbers. Because audio is something we know about, it doesn’t help to explain in terms that aren’t math-based. While that can be pretty boring, it’s much more helpful to actually address the issue in a way that isn’t subjective.

Objective testing helps us as writers lay all the cards on the table so you’re not making any purchasing decisions based on faith. How does a product sound? Is there a flaw? What exactly does it block out in a noisy area? Nothing’s left to the imagination: you’ll have the information you need.

How to separate facts from fiction

Like any other online reviews, finding a source you can trust right away is very difficult. However, I’ll give you a few questions to ask yourself that works for any product category you could ever come across.

1. Who’s writing this review?

If we’re trying to establish what’s useful to know and what’s not, starting with strongly ethical writers dedicated to finding and communicating truth is essential. Amazon reviewers could be anyone, from just some guy to an intern at a marketing department trying to drive that product rating up. You want someone who not only knows a lot about your product category, but will also be honest with you—an expert without an ulterior motive.

For us here at SoundGuys, we wear our qualifications on our sleeves. Our team collectively has decades of experience in objectively testing audio products like headphones, and we’ve been around long enough to see where things might be headed. On top of that, we have a strict ethics policy that prevents any of our writers from giving any product preferential treatment, and we also look to hire trained journalists to test products. Should we get something wrong: we own it, and fix the issue rather than risk readers getting misinformation. Our singular goal is to arm you with the best information possible.

You can look for similarly-minded reviewers by reading the ethics pages of their sites. If they don’t clearly detail how they test products… you want to move on.

2. What information is conveyed? Is opinion minimized?

Though it can be tough to tell the difference between information and opinion, the litmus test most people use is simply “Does the reviewer show their work?” For headphone critics, that can be anything from showing charts and graphs to simply describing tests they performed and what to listen for yourself. Without understanding how something is assessed, it’s almost impossible to translate the reviewer’s experience to your own. This is especially true for comparisons; if you’re not familiar with the headphones the reviewer is comparing your unit to, what have you learned?

While numbers and technical explanations are about as interesting as watching paint dry, they offer an extremely valuable insight into how well a product actually works, especially when you have an expert to contextualize the results. They’ll tell you what kind of sound headphones have, if you need an amp, or if a certain feature is a gimmick or is actually useful.

3. Separate fact from feelings

Finally, if a review is confusing, that may be by design. Remember those unquantifiable adjectives I mentioned earlier? People often search for words to act as filler when they can’t demonstrate a concept to a reader. Marketers also sometimes do this, but in a different way. If you read copy that contains a lot of flowery, nice-sounding words that sound like they came out of a wine review, you should pay close attention to how each of those words like “lush,” “sweet,” or “rich” make you feel. These words aren’t there to convey information, but to make you have emotion about a product.

Needless to say, most people shouldn’t be dealing with strong emotions when deciding what products to buy. It’s a pair of headphones, not a wedding venue. If you want the best information possible, you need to deal in facts and figures instead of emotions. It’s easy to feel strongly about something, but it’s critical to explore issues as they are—not how we might feel they should be.

There’s a lot to unpack when discussing the philosophy behind online headphone reviews, but for the most part all you need to know is that it follows a straightforward disagreement: subjectivists vs. objectivists. Objectivists believe you can learn everything you need to know from scientifically collected data, and subjectivists believe you should trust your ears. Because the extremes of each camp disregard each others’ arguments completely, scores of heated arguments rage online.

Obviously, SoundGuys falls somewhere in between these two camps, because we currently provide hard measurements accompanied by a results of a machine learning model that projects what a large group of listeners would rate headphones. Since it’s currently impossible to allow all of our readers to demo the headphones we review for themselves, we trudge on with the best available methods available to us. While there are a lot of purely subjective reviews out there that use feeling more than fact, we caution you not to put stock into those descriptions. They don’t tell you as much as you think they might be telling you.

Holding reviewers accountable

You may still run into situations where your experience differs from the reviewer’s. That’s why we have a Frequently Asked Questions section below every review and best list, as it keeps an open line of communication between you and the authors. Additionally, you can reach out to us on Twitter or email. While you may not receive a response, putting your questions out is important. How else will publishers know what to address if they’ve missed something? Always speak up if you can—you never know what kind of change you may spark!

A disciplined approach and a healthy curiosity will bring you a deep appreciation for any subject—and that’s why we test the products we review. We want to discover and report the truth about everything we cover.