For those of us who are interested, the following site has over 300+ 35mm and MF lenses tested with detailed MTF curves and scores.

http://www.photodo.com/prod/

-- Mani Sitaraman (bindumani@pacific.net.sg), November 26, 1998

Answers

I will point out that the data in the Contax SLR body section of photodo.com is laughably inaccurate. This is a real shame since Contax provides Dead Trees marketing collateral as well as a corporate web site that has specifications. It is clear that whoever authored the Contax SLR section of this web site A.) was very careless about data entry, and B.) has no familiarity whatsoever with the Contax SLR product line.

That said, I question the accuracy of these 300+ lens tests.

-- Sean Yamamoto (seany@altavista.net), November 28, 1998.

There are certain Zeiss primes especially the wide angle side have inferior MTF curves than the third party equivalences in the test. There are also some Zeiss lens with Macro capability are labelled as not. I personally won`t trust anything from this web site.

-- Eric Ung (ung@chevalier.net), November 28, 1998.

I was interested in both Sean's and Eric's remarks.

First, the MTF graphs were clearly generated by a different person than the individual(s) who transcribed the information Sean questions. It isn't necessary to distrust the MTF data simply because of errors that have nothing to do with the data.

Second, the data were generated by a single identified individual working at one machine in one site for all the measurements. That this is not the same individual, the same machine, or the same site that were involved in generating the MTF data that Eric cites is not at all surprising. For this and other reasons it is also not surprising that the data are not in complete agreement with the data presented by the manufacturer.

I have no idea how the manufacturer generates the MTF data. I suppose that there are several possibilities, including: (1) test a number of lenses, and present the data for the best individual lens (2) test a number of lenses, and then present the data for a lens that is truly representative of the sample (3) test a number of lenses, and use the data to generate average curves for the sample. I strongly suspect, though, that the manufacturer does NOT do what the photodo site workers apparently do: generate the MTF curve from a single, random sample of a lens purchased at retail.

What you trust or mistrust may say more about you than about the data. (I might be more critical of the information available at photodo.com if the MTF data presented for my favorite Mamiya 7 lenses didn't compare so favorably to, say, some of the Zeiss lenses...)

-- Brian Walsh (mche@ucdavis.edu), November 29, 1998.

>>> generate the MTF curve from a single, random sample of a lens purchased at retail.

i can't tell if you think is this a good thing or a bad thing. i will tell you think this it's a bad thing. a horrible thing in fact.

back in my college statistics classes we learned how big a sample was needed to produce reasonably accurate results. i don't remember the exact formulas, (they were pretty involved) but i do remember that ONE was not a good sample size for anything.

i recently bought a brand new EOS-5 body. the shutter failed on the very first frame i fired it on. (so i sent it back and got a new one) if i worked at photodo.com i guess i'd say that the average shutter lifespan for EOS-5 cameras was 1 firing. nonsense!

any lens test that reports data based on a sample size of ONE is beyond worthless.

-- Sean Hester (seanh@ncfweb.net), December 01, 1998.

Most magazines test a sample of one, for economic reasons. I once talked with someone involved in testing who said that if the results looked "odd" (very good for a very cheap lens, or very bad for an expensive lens), they'd test another one. Hardly scientific I agree.

The excuse behind testing one sample is thet modern QC techniques are good enough that there is very little spread in performance. Of course that's not much good if they happen to test one of the outliers, and by testing just one they don't know when that happens. You can probably say that most tests are fine, the trouble then being that you don't know which one's aren't!

You could write a test of any modern consumer zoom without even testing it. "Slightly soft at maximum and minumum apertures, but sharp and contrasty at intermediate apertures. Excellent performance at the short end with a slight drop when zoomed out fully. Light fall off gone by 2 stops down. Slight to moderate barrel distortion at the short end changing to slight to moderate pincushion distortion at the long end. Flare well controlled at the short end but showing some ghosting in strong backlight at the long end."

-- Bob Atkins (bobatkins@hotmail.com), December 01, 1998.

Once again Bob Atkins has hit the nail on the head. What he wrote sounds like almost every Popular Photography test I have ever read. It means almost nothing in the real world, but sounds scientific enough to fool most of the people most of the time. Lens tests are good for getting information on specifications, construction, and handling of a lens, but really not much else.

The Canon Rebel G ads are right. Image IS everything. The image you put on film that is. Even though it tests badly the Canon 75-300 'IS' lens has allowed me to get a few shots that I wouldn't have gotten with any other lens that was available when I bought it. That is a much more important test of any lens.

-- Brad (reloader@webtv.net), December 01, 1998.

Sean quoted me as suggesting that the testers at photodo might "...generate the MTF curve from a single, random sample of a lens purchased at retail."

Then Sean said: "i can't tell if you think is this a good thing or a bad thing. i will tell you [I] think this it's a bad thing. a horrible thing in fact..."

I believe the folks at photodo merely provide information that can be thrown into the mix and weighted as you wish. Testing a necessarily limited number of lenses, even a single lens, is similar to Consumers Report technicians testing one--and not hundreds--of Chevy Suburbans before presenting data on, say, how long it takes the beast to lumber from 0-60 mph; if CR says that the single vehicle tested required 0.2 seconds longer than the figure published in Chevrolet's product literature, is that a horrible thing?

I don't think so. Instead, while I'd probably ascribe the difference to the (very) limited sample size, I'd also be glad that apparently honest work allowed me to consider an unbiased independent assessment, no matter it's limitations.

Of course, when I recently purchase a pricey lens, I did exactly what I think the photodo workers do: I used a sample size of one. Is that a horrible thing? As Bob suggested, modern quality control techniques should protect me from purchasing an outlier, but every 43mm lens Mamiya made cannot be precisely like another. Would the MTF data generated by the folks at photodo for a single sample of that lens exactly duplicate the curves from the manufacturer? I doubt it, but I'll never know: unlike Carl Zeiss, or even Chevrolet, the manufacture of my new baby doesn't publish test data, so there's nothing to get excited about.

-- Brian Walsh (mche@ucdavis.edu), December 03, 1998.

i'd like to see them test 2 (or maybe 3) lenses. hundreds would not be necessary. if the results of the 2 sample were similar that would mean 100 times more (even though there were only 2 tested) then the results of only one. if the results were wildly different then the data would obviously be more suspect. they could either test a third (probably similar to one of the first two) or just leave it with "take your chances"

as it stands i have to read several different lens tests to gain any useful data. if i look at 4 places and 3 say "great lens" and one says "horrible lens" i go with "great lens". and thank myself i didn't only read the "horrible lens" test and draw the wrong conclusion. photodo does provide a useful data point. but i don't place all my trust in them.

as for your own sample size of one. when i buy a lens, i do a limited test of it. if it performs worse then what i expected, i exchange it for another. i have only done this once, but it was worth it. my 2nd one was much better then the first.

-- Sean Hester (seanh@ncfweb.net), December 03, 1998.