Box Wine Bias: A Case Study
Many people look down their noses at box wines. No reason. Many box wines are quite good. For an example of box wine bias, I recently suggested to a restaurateur that he include box wines as inexpensive house wines on his list. My proposal:
As you know, the Lenox Wine Club has now held 6 tastings. The results of our most recent one can be found here. The Bota Box got the best score against wines costing $50+. For Lenox Wine Club members, the results were not surprising. Box wines have either won or come in second in the 6 tastings we have had to date. And in all of those tastings, we included even more expensive wines with the highest Wine Spectator ratings we could get.
So I have a proposal: why don’t you start offering a line of house wines costing $3 a glass with a bottle price of maybe $10. Here are the details: Assuming you can get 5 glasses out of a regular 750ML wine bottle, you could get 20 glasses out of a 3L container. I can buy any 3L Bota Box for $15 at Price Chopper. You would have to buy from M.S. Walker at $16.67. That means each glass would cost you 83 cents ($16.67/20 glasses)! To give the wines a bit more elegance, you might serve them from decanters. And when people order a bottle, you could serve them in a 750ML decanter.
The Bota Box product line includes the following: Pinot Grigio, Chardonnay, Merlot, Cabernet Sauvignon, Shiraz, Malbec, Zinfandel and a heavy red blend including Zinfandel, Merlot, Cabernet Sauvignon, Syrah, and Petite Sirah.
The restaurateur responded as follows:
I am giving it some thought although I have to say I am not a big fan of the Bota Box wines. I am not sure I have faith in the tasting results. Certainly Bota Box scored well but then again it did only ok in the middle of the pack. I am not sure that the results would be the same if there were more knowledgeable or experienced tasters on the panel. The Bota Box wines are made to be easily accessible and maybe that is why they score so well in the blind tastings. Also the way some of the other wines are handled might be a factor. Some of the bigger wines would benefit with decanting and some time allowed for breathing etc. I have only tasted one of the Bota Box wines so in all fairness I need to give it a chance and see what I think again. I don’t mean to discredit your work on this and perhaps I need to spend more time to really understand it but I am skeptical that you would get these same results if we had wine shop proprietors, the sommeliers from high-end hotels, other restaurateurs and any of my wine salesmen in on the tasting? I would have a hard time offering a wine by the glass that I did not enjoy myself. My goal for the list is not to put the least expensive wines in place but to offer wines that represent good values and have unique qualities.
Thanks for your thoughts. I offer a few introductory remarks and then deal specifically with your comments.
1. From “Good” and “Bad’ Wines to “Almost All Good”
When I was growing up, a clear distinction could be drawn between “good” and “bad” wines. The bad (inexpensive) wines included Mateus, Lancer’s Sparkling Rose, Thunderbird, Ripple, and some of the Chiantis packed in straw. The good wines were the French Bordeaux, Burgundies, and Champagnes with a few others from Italy, Spain, and Germany.
Since then, the wine world has changed dramatically. Excellent wines are now being produced all over the world: Argentina, Australia, Chile, New Zealand, South Africa, and the US. And good wines now are available at almost all price levels.
Gradual recognition of this started back in 1973 at the Judgment of Paris where for the first time, Californian wines did a bit better than French wines. This was followed by similar tastings with similar results. What has happened really hit home with me when a year ago, even some New Jersey wines kept up with vintage French wines (Judgment of Princeton). Perhaps the most extreme example of this took place in Princeton last year when New Jersey wines did almost as well as the top of the line of French wines. And more fundamental changes are occurring in the wine industry: for example, vineyards must now compete with producers who buy all their grapes in bulk and mix them.
2. The Results of Recent Blind Tastings: Price No Longer Matters
Robin Goldstein has analyzed data from 6,000 blind tastings. I quote from his findings: Individuals who are unaware of the price do not derive more enjoyment from more expensive wine. …we find that the correlation between price and overall rating is small and negative, suggesting that individuals on average enjoy more expensive wines slightly less….
Lecocq and Visser analyzed data from three blind tasting data sets totaling 1,387 observations. They report similar findings: When non-experts blind-taste cheap and expensive wines they typically tend to prefer the cheaper ones.
Economists agree: more expensive wines do not taste better. In the most recent major blind tasting at Stellenbosch, the issue of whether wine ratings correlated with price did not even come up. And they did not.
Another important point comes from the analysis of data from blind tastings: there is very little congruence among the tasters on wine rankings. And this lack of congruence holds for “experienced” as well as new wine drinkers. The best statistic for measuring agreement among tasters is the Kendall’s Tau: a higher number indicates greater uniformity among tasters. The Tau for our six tastings ranged from a low of .013 for the “heavy whites” tasting to .222 for the Latin American tasting. At Stellenbosch, the Tau value for reds was very low – .070 – suggesting no agreement between tasters on wine scores. The Tau for whites was fair or average – .460 – suggesting only modest agreement among the judges.
So how can this lack of correlation among tasters’ ratings be explained? My answer:
- Nearly all wines are now good.
- People have different taste preferences.
- These different taste preferences now dominate blind tastings and are the reason for the lack of correlation among tasters’ scores.
With this as a backdrop, I now respond specifically to quotes from the restaurateur. In what follows, I refer to him as FR. I am EM.
FR: I am not a big fan of the Bota Box wines…I would have a hard time offering a wine by the glass that I did not enjoy myself….
EM: I often do not enjoy wines that others enjoy. But if there is a group of people in our wine club, who want to taste wines I do not like, we taste them and I always learn something.
FR: I am not sure I have faith in the tasting results….
EM: Our tasting results and methodologies are reported in detail in the following reports:
1. “Heavy Whites”;
2. “Heavy Reds”;
3. “Light Whites”;
5. “Light Whites”; and most recently,
I urge you to take a look and let me know the source of your lack of “faith”.
FR: Certainly Bota Boxes scored well but then again they were only OK in the middle of the pack.
EM: The results of how the box wines did is given in the following table along with the most expensive wines we tasted. The box wines were not just “ok in the middle of the pack”. They came in either first or second in all tastings.
FR: The goal for my list is not to put the least expensive wines in place but to offer wines that represent good values and have unique qualities.
EM: I am not suggesting you to put the least expensive wines in place. Throughout our wine tastings, the Box wines got the highest scores, without regard to price. That is the reason for my suggestion.
FR: The Bota Box wines are made to be easily accessible and maybe that is why they score so well in these blind tastings.
EM: Not sure what you mean by easily accessible. In a blind tasting, all wines are accessible.
FR: … the way some of the other wines are handled might be a factor. Some of the bigger wines would benefit with decanting and some time allowed for breathing etc.
EM: At the restaurant where most of our tastings took place last year, I managed the timing of when the bottles were opened and poured. The bigger wines were opened well in advance of the tastings.
FR: I am not sure that the results would be the same if there were more knowledgeable or experienced tasters on the panel…. I don’t mean to discredit your work on this…but I am skeptical that you would get these same results if we had wine store proprietors, sommeliers, and any of my wine salesmen doing the tastings.
EM: When it comes to wine, what exactly does more “knowledgeable” and “experienced” mean? Members of the Lenox Wine Club have all been drinking wine for 30+ years. They know what they like by now. Are you suggesting that the wine likes and dislikes of the Wine Club members are somehow inferior to wine professionals? This is a very slippery slope for you to go down. If you saying your professional wine friends should be the arbiters for good and bad wine, I disagree. And keep in mind that your friends have commercial interests that might be reflected in their wine ratings.
In my view, the only legitimate reason for questioning a wine taster’s judgment is whether or not s/he fail “Hodgson’s test”. I elaborate: Robert Hodgson has his own winery and has been troubled by the erratic ratings his wines were receiving from judges at tastings. Consequently, he developed some tests for judges and he has been using them to analyze judge performance at the California State Fair for over a decade.
The key to his method? Have the candidates do blind tastings that include more than one glass of the same wine in each tasting. If the candidates do not rank/score glasses of the same wine nearly the same, they are not competent to judge wines. Hodgson’s suggested overall scheme is quite rigorous: candidates must do four blind tastings of ten glasses each. At each tasting, there are three glasses from the same bottle. And for a candidate to qualify as a judge, the scores given on the glasses on the same wines must be “close”.
The key result is that at the California tastings, only about 10% of the judges are consistent in their ratings, and this 10% are not the same judges year to year. A further note: he finds no relationship between “experience” and judging consistency. This probably means that your professional wine friends would not get a better “Hodgson score” on judging wines than members of the Lenox Wine Club.
In our wine tastings, we employ a modified version of the Hodgson test: we have two glasses poured from the same bottle. In our most recent tasting, we found that people with higher Hodgson’s scores gave higher ratings to the Bota Box than the overall group.
There is a bias against box wines and words (see above) will do little to remove it. But maybe I can get MR and his friends to do a blind tasting that includes a box wine and a Hodgson test!
 This is a sad story. I live in Massachusetts where all restaurants and liquor stores are required to buy their wines from wholesalers. The state has been told by the courts to change this law because it violates the Interstate Commerce Clause of the U.S. Constitution. But the distributors are very influential and nothing has been done. For more on this, see my article on the subject. The upshot is that even though the wine is cheaper at the supermarket, the restaurants are not allowed to buy from them and must buy from the distributors.
 For a great book on the subject, see George M. Taber, Judgment of Paris, Scribner, 2005.
 The Wine Trials 2011, edited by Robin Goldstein et al, Fearless Critic Media.
 Sébastien Lecocq, and Michael Visser, “What Determines Wine Prices: Objective vs. Sensory Characteristics”, Journal of Wine Economics, vol. 1, no. 1.