Monday, July 17, 2006

THE GREAT WINE SCORING WASHUP

Once upon a time, in a galaxy far, far away ……………there was a band of Jedi that came upon the “Knights Who Say ‘NI’” (forget mixed metaphors – I am mixing my movies). The Jedi were a peace loving group that wanted order in the cosmos whilst all the Knights Who Say ‘Ni’ wanted was ….. a shrubbery!!! (hope you all like Monty Python – but I can’t go any further with the mix because I am already lost).

Thanks to all who contributed to the discussion that took place when I posed the simple questions – Why a Wine Scoring System? And Which One? The response was amazing! Being just an amateur wine enthusiast I was totally unaware, not only of the variety of systems of scoring but also, of the depth of feeling in relation to various systems. I feel I have benefited greatly from the discussion and the different points of view.

I did not pose the original questions on a whim. It was very much a considered response to where I am at with my tasting and enjoyment of wine. I think I covered my reasons sufficiently in the original post so I won’t go over them again but, I will attempt to summarise the points raised by various people in the discussion and attempt to address them all in stating how I am going to approach giving any wine a score.

NUMERICAL & NON-NUMERICAL SYSTEMS
It appears there are really only two streams of scoring systems – numerical and non-numerical. I would include any scale of one star (glass etc), two stars (glasses etc) as numerical scales. Non-numerical scales would include word descriptor scales (such as “Cat piss” through to “Ultimate”) or pictorial scales such as Wine Girl’s various faces. Non-numerical systems attempt to rate a wine on an overall basis whilst numerical system users are broken into those who attempt to give an overall score and, those who attempt to score the individual attributes of a wine to come up with a total score.

Most disagreement in the discussion began around the value of numerical versus non-numerical scoring systems and ultimately refined to being about the objectivity or subjectivity of a system and thus its corresponding value to others who may read and/or use the score.

OBJECTIVITY & SUBJECTIVITY
Proponents of non-numerical scoring systems appear to believe that as subjectivity is inherent in all tasting, attempting to be objective can be misleading and shouldn’t be attempted. Proponents of numerical systems appear to believe that despite there being subjectivity in scoring (whether or not an attempt is made to be objective), a numerical score is not only the most widely used standard but that there is really no difference (except possibly in immediate understandability) between a numerical and a non-numerical system, as they both rate wines along a gradient – the only real difference being the blurring or ‘rubberiness’ of the edges of each category in a non-numerical system which isn’t really available if you give a single numerical score.

I can see a real honesty and intelligence in Torb’s argument that he accepts wine scoring has a subjective element to it (and in cases may even dominate) and therefore he only, and openly, provides a purely subjective non-numerical score to wines. Because he knows more about wine than many people (myself included) his subjective score is a very good indication of how good a wine is.

The real problem for non-numerical systems (as I see it) comes if and when someone wants to differentiate between wines within the one grouping. For example there may be a wine that just scrapes into “Highly recommended” and one that just misses out on “Excellent”. Are these wines essentially the same standard? Mind you, this desire to differentiate would most likely only arise with professionals or enthusiastic amateurs. Other ‘punters’ just couldn’t be bothered. To these people, non-numerical systems have the greatest value.

Numerical scorers can also score on a subjective basis, especially if they score on (but not limited to) an overall viewpoint rather than score the individual attributes. This does not mean that the intention is to be subjective. It is possible for a numerical scorer, even whilst trying to be totally objective, to be influenced by personal taste (amongst other things). This would be especially true for amateurs with little tasting and scoring experience (such as myself). Although a professional taster (eg a show judge), whilst they may still influenced by personal taste and other subjective elements, if they are a true professional would attempt to put aside those influences and score as objectively as humanly possible. Is absolute objectivity ever really possible? Maybe not! Maybe Torb has a point but, I think the attempt to be absolutely objective is something that should be made.

However, I believe that scoring individual attributes is the most difficult to do (especially for the amateur who lacks knowledge and experience) but lends itself inherently to the most objectivity. Subjectivity can still have an effect.

MY PERSONAL VIEW (MY CHOICE)
I really don’t think either the numerical system or non-numerical system is right or wrong – they are just, well, different. (It’s sort of like toilet paper usage – you are either a ‘folder’ or a ‘scruncher’ – just different).

However, I want to try and look at the issue from a different angle. I have been looking at the issue in terms of outcomes or goals. Why am I doing this? What am I looking to achieve? What am I aiming at? How will I get value from scoring the wines I am drinking? In other words, the old (1) where am I now? (2) where do I want to be? (3) what is the best way of getting there?, goal oriented progression.

I want a system that I can use to score wines for my benefit (all that first person – what a selfish bugger). If others can get value from it, fine. If they think it is bullsh*t, fine. I want to able to score a wine so that I have some reference point to use either for purchasing or discussion in the future. In other words I am doing it primarily for my own use. If the score is subjective, so be it.

I agree with Cam and Torb that the most important part of any review is the tasting note, and I really need to work on mine. Irregardless how important the tasting note is, it still does not allow me to attempt to compare accurately a wine with another or the same wine over time or other vintages. Torb might argue that any numerical score, because of its subjectivity, may not allow me to do that accurately anyway.

What I am really looking to achieve is…. the holy grail!!!....objectivity!!! Look at it this way – if you assume (for a second at least) that there is one ‘true’ absolute score for any wine at a given point in time. Good wine tasters, who know what they are doing, who are able to put personal preferences aside (and yes, I believe this is possible – professionals in all walks of life do it all the time), should be able to consistently get close to that score. This is called – objectivity. That is what I am striving for. If I get it wrong then I have made a mistake. If I am being subjective in my scoring then I am no worse off than most other scorers.

Suggestions by GW and Cam suggesting that a numerical system is a more widely accepted standard appears to be true, but is that the reason I should use a numerical scoring system? I don’t think so.

I have decided to go with a numerical system for two main reasons (1) differentiation and (2) clarity. For me at least, there is a desire to differentiate between two wines that might rate a ‘Highly Recommended’ in a non-numerical system. One may have scraped in and the other may have just missed out on the next level up. (Maybe this is because I am an accountant?!)

Secondly, and this is the clincher for me AND the reason numerical systems are so widely accepted and used, I firmly believe that the use of a numerical system is more intuitive. When faced with 2 wines, one rating 85 and one 89, which is the better wine? Intuitively, the one scored as 89. Now, in deference to Torb, this may not be true because of errors and subjectivity problems, but in most cases it should be true and the difference should be noticeable. In a non-numerical system these two wines could potentially be scored the same, by being placed in the same category.

Even if someone else disagrees with either the scores and/or the positioning of the wines relative to each other, it really doesn’t matter. This is because when I see a score by Halliday, Parker, Robinson etc or even Ed or GW, I know that it is their score – I don’t have to agree. However, I believe that if a person is honestly and consistently attempting to objective, then their scoring will reflect it and be of value.


I note in some places Robert Parker is being criticized as being captured by the same system he railed against when he started out. He criticized establish wine critics and their associations with wine makers and accused them of lacking objectivity especially when it came to newer, less established wines. Parker championed some of the ‘new comers’ and is now being criticized for lacking objectivity in relation to other wines because he allegedly has become too close to some wine makers he championed. Are his scores still of value? Yes! But remember they are his scores. You or I may not agree (but even I would take his score over mine). Then why are his ratings so popular? Because he knows what he is doing and has proven himself to be a good judge and attempts to be objective and is consistently accurate.

I have decided, at least for the foreseeable future, to score numerically using two systems until I can find which one I find the best. I am going to give an overall score out of 100 based on my tasting notes and then once this score has been done go back and try to score each of the components using the score sheet from
Torb's site to end up with a score out of 20. I may even score on a small range out of 100 – eg. 85-87/100. This is not a ‘doff of the hat’ to the non-numerical systems but is purely a confidence thing on my part and I hope it will disappear over time.


The best thing about being an amateur is that I can be wrong. I don’t mind being wrong and making mistakes, however I do hate not learning from them. I am sure I will make plenty of mistakes scoring and talking about wine – that’s a given. However I will always be trying my best to get things right and be as objective as possible.

Now, GW – where’s that friggin’ light sabre????? May the force be with you!!





21 Comments:

Blogger GollyGumDrops said...

At the risk of throwing a spanner in the works, how about a third category for price. Is a 84 point wine at 6.99 or an 89 point wine at 32.99 the better buy? If objectivity is the goal then there must be an absolute right market price for a wine!

11:06 PM  
Blogger Edward said...

Mal,

Now that you are out of the closet and giving scores, you REALLY DO have a wine problem!
Welcome to the madness. . .

1:30 AM  
Blogger Nia said...

What I'd like to know is the relativity of 'scruncher' or 'folder' to numerical or non-numberical scoring systems.....this will make it much easier to select the appropriate scoring system for me....so....I eagerly await your responses....or perhaps al ittle more research Mal.

7:17 AM  
Blogger Michael Pollard said...

Mal,

Even though you have a scoring system there still seems to be a question (in some minds) about whether its really needed.

We won't name names, but below are four views on the same wine - one is mine. Its not a guessing game, it just shows how different tasting notes can be. So my question to the folks who gave their comment on what scoring systems you should use is what value is a tasting note in and of itself? And if any one says its just a matter of calibrating your palate to one expert or another I want chapter and verse on how they actually did that - if they ever have! Don't be too subjective with your answers now.

2002 Penfolds Shiraz Magill Estate
1) It possesses sweet aromas of vanillin, white chocolate, cassis, and flowers. Medium to full-bodied, expansive, pure, and nicely textured, this complex beauty is somewhat European in its restraint and elegance.

2) The pure fruit driving the bouquet screams class. The finest grained tannins imaginable are perfectly balanced to a gob full of lip-smacking, pure, persistent fruit; structure and build are sensational. The flavour starts off verrrry slowly: with slightly sweet blackberry, progressing into savoury red berries, chocolate and meat; it finishes to clean liquorice with a long, persistent finish. Showing great harmony, this refined and elegant package blows away the rest of the line up.

3) It’s musky and sweet and custardy, with pure, plummy, stylish fruit resting beneath, but once the tight, focussed finish takes hold it’s oak that does the bulk of the talking. The effect: it appears simple. Once it’s had a good amount of time in the glass though the fruit rises and the oak fades: given time to settle and evolve, this could easily drink outstandingly.

4) Complex aromatics that include prominent toasted coconut and blueberry followed by cherries and pepper with an underlay of dark caramel and earth. Full bodied with a soft and supple entry. Great structure with excellent depth and complexity across the palate. Super soft tannins backed up by juicy acidity and a persistent finish.


There is no doubt that its classy wine as the scoring below indicates, but when you read those four notes you could not be faulted for thinking four diferent wines were being described. Sometimes a score is the most universally understood indicator we have.

1) 90
2) Excellent with *** for value
3) 92
4) 94 (I'm a generous scorer!!)

Mike

10:14 AM  
Anonymous Anonymous said...

Mal,

Your decision has been based on a well thought out logical look at the facts and what you think will suit you best – well done!

Mike,

As soon as I saw that note, I knew there was going to be some difference in the notes and conclusions. As soon as I saw JO’s note and Campbell’s, I emailed both of them about the differences and suggested both try the wine again, and if possible after it had a long decant. The differences in TN’s could be bottle variation, or decant time. (FWIW, Brian – Red Bigot – drank the bottle with me about 10 hours after it was open and agreed with my assessment (TN.)

As far as your second question is concerned, in years gone by I calibrated my palate to Halliday. The way I did it was that every time I opened a bottle of wine, I wrote a TN and then looked up Halliday’s note and looked to see where we differed and agreed. Over time, I came to know that if he described a wine in certain terms, I knew I would like it, but if he used certain other words, I knew it would not be my sort of wine.

As our palate differences diverged further over the years, I started looking at Oliver’s notes closely and did the same thing. Over time I learned to spot the same sorts of things. Now my palate is reasonably aligned with Oliver but there are still differences – the trick is to know what to look for and how to read between the lines. In many cases, it’s being able to interpret what is between the lines.

6:30 PM  
Blogger Mal said...

Gollgumdrops

No spanner - I actually considered this but didn't write about it. The problem with an objective scale of 'value' is the intial callibration and it is encapsulated in the example you gave.

If you accept that value is a function of two things (quality and price) then an objective value score will be one that is found by applying some formula to the quality score and the price. What formula? - and how do you calculate it?

If you accepted you would pay $1000 (purely for the sake of the argument, as may wines cost more than this)for a hundred point wine (perfect) wine and you wouldn't pay anything for any wine rating 75/100 (again a purely arbitrary selection) or less, you have to come up with some formula to get to a value score. You can't just say there are 25 point to go to 100 and therefore each point is worth $40 because, no-one would pay $40 for a 76 point wine.

It all gets too hard after that - I think you would really need a serious mathematician to look at the problem.

In the mean-time I think all we can do is to give the score and the price and let people make up their own mind on value.

It is a great thought though and I would like to see something like that!

Thanks for the comment

7:29 PM  
Blogger Mal said...

Edward

Thanks for the support! After writing that piece I found myself very thirsty!

It is a madness and I don't think there is a cure!

7:30 PM  
Blogger Mal said...

Tannia

Well done! All that study is really paying off!

I knew I could count on you to zero in and focus on the really crux of the article. The rest of the piece was pure window dressing - I was really hoping to get the whole folder v scruncher debate happening.

I have a theory (or generalisation) or just a vague notion that should not be used or really ever spoken of again. Could it be that non-numerical system people are scrunchers and numericals are folders?????

7:33 PM  
Blogger Mal said...

Mike

Thanks for the comment! It has really been an interesting debate thus far and I feel I am learning a lot.

I have no answer about the tasting notes, in fact I rasied some of the same issues with the tasting notes on wine labels and websites in a recent post (http://wineamateur.blogspot.com/2006/07/chapel-hill-shiraz-2002-mclaren-vale_05.html)
I actually feel that I may have gone too far and made it look too much like it was the wine makers' problem when it could just have been either mine in not being sufficiently experienced enough to find the things they were saying, or (2) a problem with either the cellaring or drinking conditions.

Anyway thanks again for the comment!

7:43 PM  
Blogger Nia said...

I guess if the consumer likes it enough to pay for it then it's the right price ;)

Mal I live in awe of your research and analysis skills....how much will you charge to do my data and stats analysis assignments?

I think it will be some time before I am confident enough to give scores...some years...much research...and many hundreds of bottles...

8:12 PM  
Blogger Mal said...

Torb

Thanks! I really do appreciate the comment - especially so knowing you disagree!

8:14 PM  
Blogger Mal said...

Tannia

I like your style! Once again you have focussed on the important (and the most fun) part - research.

I think it fair to say that all who have commented are extremely keen 'researchers'!! I think we are all committed to the hundreds and hundreds of bottles research.

If you keep up with the Winorama site, they are posting tasting notes at a few per day - I have a long way to catch up - damn that Yoda!

8:22 PM  
Blogger GollyGumDrops said...

I'm a scruncher and a non-scorer. I suspect folders score!

10:06 PM  
Anonymous Anonymous said...

An excellent summary and an outcome that follows the path of wisdom. The force is strong as I guessed.

GW

PS. I'm holding the Winorama fort on my own at the moment..but I have a backlog of wines...so I need to write up at least 3 or 4 wines a day to catch up...for at least few weeks.

10:09 PM  
Blogger Michael Pollard said...

Mal, TORB et al,

OK, I don’t want to belabor the point because this is Mal’s blog and I’ve addressed the fallacy of palate calibration on my own blog. But if you are really serious about calibrating your palate to that of another individual then I think you will find that it is next to impossible to do. (And I’m somewhat surprised that Ric was the one to offer up his history because he has argued in the previous post on scoring on this site that much of his analysis of a wine is subjective –a judgment based on his previous experience which is not identical to anyone else.)

Let me start by saying that I know of no one who has done an objective analysis of wine critics abilities in order to identify ways in which one might use such knowledge to calibrate palates. Why would that be important? Well do all critics know what blackberry actually smells like, or the taste of juicy acidity? What does the term “well structured” mean to them? In order to calibrate your palate to some critic you would need to first calibrate said critics to ensure that they know what they are talking about (impossibility No.1 unless you are really close friends with said critics and sensory perception research scientists). Remember, just like Ric, or Robert Parker, or me, or Mal, everyone’s impressions of a wine are subjective. Second you need to calibrate yourself against the critic. Is your (subjective) ability to detect blackberry, acid, sugar, alcohol, balance, structure etc, etc equivalent to the critic (impossibility No.2, unless again you are really good friends with… see No 1 above). Now I’m sure there are people reading this who are stamping their feet and huffing and puffing and saying that all this is not necessary. Fine, I have no problem with that, mainly because I see no reason to calibrate my palate to any critic; even though in the past I have tried this until the light bulb was finally switched on and I came out of the darkness.

If you want to see how futile palate calibration is just make your way to any local tasting event where they taste single blind and provide the tasting notes of one or another critic for the wines. I attend this sort of event almost every month and the reviews are by (mostly) Parker, Tanzer and Wine Spectator. In terms of palate calibration there are two instructive parts to these tastings. First, during the tasting there will be people trying to identify the wines based on the tasting notes – I’ve never seen anyone identify the wines this way. Second, once the wines are revealed people will compare their notes with those of the reviewers, again there is very little concordance. Now these aren’t Halliday or Mattinson, or JO tasting notes but in the US if your palate does not agree with Parker, Tanzer or WS then the alternatives are slim pickin’s indeed for those who seek the Holy Grail of palate calibration.


Mike

3:48 AM  
Blogger Michael Pollard said...

Mal

Unfortunately I hit the wrong button with your latest post on my blog and deleted the post on "French Lose the World Cup but Win the Wine Cup". Can you repost?

And do you have an email link?

Mike

7:48 AM  
Anonymous Anonymous said...

I've learned that knowing that I like a wine is not enough -- I want to know why I prefer one wine over another. That's where scoring a wine can help.

I had a lot of help from my wine-tasting friends developing a 20-point scoring sheet that I use quite frequently. We've found that a 20-point system is definitely more manageable than a 100-point system, which is difficult to attempt by even the most vainglorious of tasters. A 5-star system, I've found, is just too simple, and doesn't offer any real educational value.

This wine scoring sheet, linked to below, is broken into seven criteria with numeric values assigned to each; sample descriptive adjectives are offered within each criterion. It also is two-sided, allowing input for individual wine scores for seven wines, as well as space for tasting notes and group scoring on the second page to aggregate a group's individual ratings. A third page includes instructions for how to use it.

http://www.californiawinehikes.com/wine_scoring_sheet.pdf

While it can argued as to whether "taste/flavor" should be 4 points and "finish" only 2, this system can be a big help for people who want to learn to taste more thoughtfully by breaking a rating down into more manageable chunks.

Most of my guests are new tasters who want to learn why they like a wine (or why they don't); many return for follow-up tastings. That's a vote of confidence, indicating that they derive value from this scoring system. Perhaps you will too! Please let me know what you think.

4:48 AM  
Blogger Mal said...

Winehiker

Thanks for the comment! It sounds as you were at the same place as I was - needing some objective score to measure different wines against.

I have downloaded you scoring sheet and will give it a whirl. To date I have been using one I found in this handy manual at

http://www.torbwine.com/rr/WINE%20TASTING%20manual.htm

It has been working well but I am open to new ideas.

Thanks again.

1:04 PM  
Anonymous Anonymous said...

You're welcome, Mal, and I hope you'll find value in that scoring sheet. Better yet, gather your friends together and try to "break it"!

5:21 AM  
Blogger Mal said...

Shel

Thanks for the comment! You are right, scoring and the systems used tend more towards subjectivity than objectivity. It simply says we are all different. You are not good with numbers - I am (although sometimes I wish I wasn't). We are just different. The really important thing is that we do as much research into wines as possible.

9:16 AM  
Blogger meshmarketer said...

I hope you don't mind that I am going to share with you my own wine scoring sheet. I developed this because I was offering in-home winetastings and I wanted my audience to be able to remember what they liked and why. Enjoy! Free Winetasting Score Sheet

12:18 PM  

Post a Comment

<< Home

Site Feed
Enter your Email and receive the latest posts immediately


Powered by FeedBlitz