Note: I originally posted a version of this for members, this is a revised and expanded version of that post.
I have a lot of problems with how review sites write their reviews. They developed non-sensical scoring systems, loosely follow their stated use of those systems, and over time end up rating everything as 9.8 out of 10 — or whatever the equivalent is. Review sites, if you haven’t noticed, very rarely review a product where they end up calling it utter shit.
Instead, review sites focus on their rating or ranking system. They focus on votes, as if that can’t be gamed or holds any value whatsoever over Amazon’s review/ratings. Review sites wear the amount of items they ‘reviewed’ to determine something being the best, as the end all and be all for reviewers and some sort of merit badge of quality. These sites never quite explain how testing 150 backpacks for 5 minutes each means they actually learned anything at all, but rather you should be impressed they tested 150, and pushed all 150 off a table to see if the 1,000lbs test strength nylon would somehow break from a 34” fall to the ground.
Or these sites focus on how many words they can force you to read, before they tell you that, coincidentally, the items they picked is one where they have the highest affiliate link earning percentage. Odd, but convenient for them! “But you should trust us because we really like rain jackets and we’ve been in rain before.”
I loathe review sites. I yearn for the old days, the days where you’d see a random person tell you why the backpack they’ve been using for the last 4 years is just fucking great. Maybe that happens on some social media site I am unaware of, but I am obviously unaware if it does. Those reviews were never planned, it was never some person waiting 4 years to get epic traffic and no money for talking about it. Just someone with some gear they love — pure.
This is why I push myself to constantly test, and review more things — at least as many as my time permits. Because I don’t feel there’s many other sites doing an even acceptable job at it. And, while I cannot test everything, at the very least I can fact check most of the bullshit out there.
The explanation for the current state of these sites is easy to explain though: it’s hard to make money by writing good and honest reviews. You either need to write a lot, or well you need to write a lot. That’s all there is to it.
This all churned up again with Charlie Warzel writing on the fall of The Wirecutter:
While scrolling through product reviews for microphones, his area of expertise, Casabona started to feel that Wirecutter’s recommendations were off—“too obvious,” he told me. Then, after a string of underwhelming purchases based on the site’s recommendations, he got fed up, tweeting in January 2022 that the site’s recommendations had gotten worse.
His post has been everywhere, but the article answers nothing — much the same way The Wirecutter is today, Warzel’s article is essentially useless for determining what actually is going on. But that’s ok, becuase what happened to The Wirecutter is pretty simple to understand if you are not worried about pissing some people off.
The first problem the Wirecutter has always had is the base premise of the site: the best thing for most people. But there is no such thing as most people thus there is no actual target audience. It would be better and more useful to state what’s best without such a contrived caveat.
But by trapping themselves with the concept of ‘for most people’ The Wirecutter all of a sudden had no real people they were writing for and thus no way to determine if what they wrote was useful. The writers are not most people, nor is their mom or dad, nor are their friends. Most people is not a real person, so how can you write to them? You cannot.
The most common advice: write to someone. They were not writing to anyone because they picked a flawed ideology of a person. And this led to the most common response I heard from The Wirecutter’s staff when I would tell them they got a pick wrong: “well this is for most people, so it makes sense you would disagree”. Oh, fuck off with that excuse. Not only did The Wirecutter no longer have an article written for anyone, they now had a catch all excuse to coverup the uselessness of the advice, an excuse they would trot out as if they reached some level of exceptional gear enlightenment while all the experts in those categories were laughing at their shit picks.
The next problem created by The Wirecutter is the idea that you can get a lot of things ‘at once’ and test them against each other in a reasonably short period of time, thus reaching some definitive conclusion about what is the best. In doing this, you root out the bad, but you don’t/can’t hone in on the good. Why? Becuase the difference between fine, good, and best is in the nuance and you only experience nuance with extended use. You gotta live with it.
That type of testing doesn’t lend itself to a website trying to make money from affiliate links as that monetization strategy is about quantity of the output, not quality of it. Taking a year to find the best of anything, is unlikely to recoup the testing costs, and the costs of the reviewer’s time, and so forth. That’s not even to tell you that you need a year to do this, but it’s to tell you that you cannot test the amount of gear The Wirecutter (and sites like it) test, in the time they test it, and know anything other than which ones are utter shit. You can barely discern which ones might be good in that time period for that quantity of items.
If I bought 47 backpacks tomorrow — a category I know a whole hell of a lot about — I could tell you maybe in a year which is the actual best, but it would only take 2 days to tell you which are awful. I try to have an item for 4-6 weeks before I review it. I need it for at least 7 days before I know if it is shit. And at least 14 days to know if it is potentially great. Anything before 14 days, claiming something is the best, and I am talking out my ass.
All The Wirecutter has ever been good at is telling you what is awful, and what may not be awful. In the same way you can’t trust that an Amazon product with 20,000 reviews and a 4.8 average rating is actually a great product. But you sure as shit can trust that an Amazon product with an average 2.5 star rating is utter trash, no matter how many reviews it has.
Lastly, the Wirecutter has always placed too much emphasis on keeping the price of the product low, as that means more affiliate sales, so the raw cost of the item is always a factor of the rating. (I pushed them on this, and they refuse to say that’s why they focused on item cost, but rather that “most people” want a lower cost item. Mmmkay) The moment you take price into account, you start to lose perspective of what might be the best.
I try to take value into account, but I largely try to leave price out of it. Value is what you get for what you paid, not taking into account the raw cost of the actual payment. And that’s not to say that spending more always is the correct path to finding the best things, but either you are trying to find the best, or you are trying to find the best you can afford. But “for most people” is not a category where you can define a budget either, as you have no clue how much most people would be willing to spend on something.
Many people, are not happy to spend more than $100 on a backpack — let me tell you. But I have no idea if that is how most people feel. I have no idea what most people spend on a backpack — neither does The Wirecutter.
In other words, if The Wirecutter were reviewing the best way to fly from point A to point B for most billionaires, they might end up saying a jet-share is the best. Because first class is pretty expensive, but low cost entry, and buying your own jet with your own pilots and such — well that’s really expensive with an extreme initial price tag, so that’s clearly out.
But it’s all rather meaningless right? If you can afford to own your own jet and staff, well that’s what you fucking do and don’t even bother lying to me that you wouldn’t if you could. But, The Wirecutter would end up recommending that Bezos use a jet-share, with First Class being the budget option. And the upgrade pick would be partnering with Zuck to share a single jet and crew with him. Because that’s what happens when you ignore value, when you ignore a true target individual, and try to find some generic defensible thing where price can be a factor. You’ll end up recommending that two of the world’s richest people go halve-sies on a jet and crew.
That’s how fucking stupid The Wirecutter recommendations have always been. It just took us a while to see it. That said, if they say something is trash, it probably is trash.
And now that we have a half a dozen Wirecutter clones, you should take notice that they are all just as bad as the next. None of that is to say that my reviews are some magic pill either, my reviews are written for me and only me. They are the best things for me, because I know really well what I like and what I think is the best. If your tastes and life aligns with mine, they’ll work out pretty well. And at the very least, my reviews should give you some semblance of an idea as to whether the item is any good, or not, for you.
So, what now? It’s not feasible for the methodology used by sites like The Wirecutter to turn out any amount of useful content given their business models. You simply cannot make money from that structure, while also doing a really good job of picking items. There are some sites that do an admirable job of having experts test shit: Outdoor Magazine seems to be one of the last, but they are limited to ‘outdoor’ life.
YouTube, sadly for those who love the written word, offers the best person to person review content out there.
Before I let you go, let me highlight the discrepancy here with a category I know nothing about: Women’s Rain Jackets.
The Wirecutter says: Patagonia Torrentshell 3L (for the outdoorsy pick, which is the easiest pick to compare across sites).
Outdoor GearLab: Outdoor Research Aspire II (as tested by multiple people).
Outside Magazine: Norrona Senja Gore-Tex Active Jacket (the best all-arounder).
I couldn’t tell you which is the best of those, but I can tell you that it’s completely unhelpful if you are trying to buy just one good rain jacket that they are all different picks.