The following article was published in "Virus News International", November 1993, pp. 40-41, 48. It is Copyright © 1993 VNI and is reproduced here with their kind permission.
I’m not claiming that all anti-virus product reviews conform to the guidelines below but I can tell you I’ve seen every one of these tricks used in magazine reviews. In some cases, a ‘master reviewer’ has shown such adroitness that he or she has been able to employ several of the tricks in the same review. In many (if not most) cases, the reviewer was unaware that he was using these tricks but in some cases, it looks as if they have been used deliberately.
The main weapons at your disposal are the choice of what features to review and what to ignore and the weights given to the features you do cover. By a careful use of this, even GrottyScan can be the Editor’s Choice.
By the way, GrottyScan and WonderScan are entirely fictitious products and is not meant to stand in for any of the products on the market today. And Grotty Inc. and Wonder Inc. are fictitious companies.
1. Put a lot of weight on User Interface. Then, you can legitimately claim that you liked GrottyScan’s user interface better than the others. User Interface is a matter of personal preference. Some people like a command line, others a full screen. Some people like lots of knobs and buttons, others like a clean interface (i.e., no options). If GrottyScan is optionless, give the most points for ‘a clean, uncluttered, user interface’. If GrottyScan is chock-full of bells and whistles, do a tick chart and give the most points for quantity of features.
2. If GrottyScan doesn’t have a TSR, then don’t test TSRs. You can either just ignore the whole issue or else claim that no-one should use a TSR, perhaps on the grounds of TSR conflict, or on grounds of security, or on any other grounds you choose. In extreme cases, you might say that any vendor offering a TSR is a scoundrel.
3. If GrottyScan doesn’t offer file repair, then don’t give any points for repair. You could claim that repair is insecure and everyone should delete-and-replace. Or you could explain that some products don’t do it very well, so nobody should use it (even though other products may do it extremely well).
4. If GrottyScan does repair but not very well, then give lots of points for the fact that it does repair but don’t actually test it.
5. You’re going to have to do a run against a load of viruses. If GrottyScan is really bad at detection, then use just 11 viruses - that way, it doesn’t look any worse than the others.
6. If GrottyScan is slow, you can mask that nicely with several deft touches:
Scan a floppy disk. That means that the speed is governed by diskette reading speed, not by the product speed.
Scan a hard disk without much on it, on a fast machine. That way, all the products take just a few seconds and there isn’t much in it. If GrottyScan is ten times slower, that doesn’t really look bad if its run time is 10 seconds.
Do your timing test on a disk full of viruses. That way, WonderScan will be slowed down by the screen display and other things it has to do when it finds a virus, whereas GrottyScan won’t be slowed down, as it won’t have found many viruses.
7. If GrottyScan uses its own naming scheme, award half the points for detection and the other half for correctly naming the virus (correct, of course, means using GrottyScan names). Yes, I really have seen this done.
8. If GrottyScan is poor at polymorphic viruses, then use just one specimen of each, this giving it a 100 per cent score. The NCSA standard testing protocol uses this trick.
9. If GrottyScan can’t deal with Stealth viruses in memory, then don’t test with a stealth virus in memory (again, the NCSA protocol does this).
10. If GrottyScan has options to run fast and options to detect most viruses, then choose the Fast option in the timing test and Secure in the detection test. Naturally, you won’t report this.
11. If GrottyScan has a heuristic analyser, then make sure you don’t run it on a clean machine but only on an infected machine. That way, you don’t have to report any the false alarms, you can wax lyrical about the way it can detect new viruses, however.
12. If GrottyScan has a behaviour blocker, emphasise the fact that it can stop viruses. Don’t install the thing and try to use it in daily use, or you’ll have to report that all the false alarms it gives makes it unusable. I’ve seen a journalist rate such an unusable product as the best anti-virus product on the market.
13. If the documentation tells you to install WonderScan in a certain way, then install it differently, then give lots of details about how it didn’t work when it was wrongly installed.
14. If GrottyScan has a five-page manual, drone on about conciseness and how this is much preferable to the wrist-breaking tomes that come with other products. If GrottyScan has a large manual, emphasis the importance of full documentation.
15. If when you phone Grotty Inc. for technical support, you get put on hold for fifteen minutes and then get given dangerous advice, don’t review tech support. On the other hand, if Grotty Inc. gives prompt and accurate support, do a table on how good their technical support is.
16. Take several viruses and patch them; write nulls over part of the virus code. Then, see which scanners still detect the viruses. Patch different places until GrottyScan detects the viruses and the other products don’t - even better, get Grotty Inc. to do it for you. After all, they know what part of the virus to patch.
17. You’ll need a test suite. Ideally, you should get it from Grotty Inc. You might find that Grotty Inc. don’t have a virus library, in which case, you should find a collection of files that contains viruses and also lots of corrupted and innocent files. That way, if half the files you use are not viruses, the GrottyScan score of 30 per cent doesn’t look too bad compared with the 40 per cent that the best product will get.
18. Give a copy of the exact test files you will be using, to Grotty Inc., three months before the test (this happened in an American review).
19. If GrottyScan finds false alarms in some of your files, count this as a plus, rather than a minus.
20. If GrottyScan doesn’t do a self-test to see if it is infected before running, don’t test to see if other products do check their own integrity.
21. Use the ‘faint praise’ technique. If you need to say something good about WonderScan, say things like: "suitable for home computer users", or: "the packaging was attractive".
22. Use the magnification technique. If you find some minor, unimportant problem with WonderScan, say that "unfortunately, WonderScan is flawed by ...." People will read that as ‘very bad’ but you can justify the statement by using the dictionary definition of ‘flaw’, meaning very minor defect.
23. If you find some major problem in GrottyScan that you are forced to report, call the vendor and you’ll be able to say, "by the time you read this, this problem will have been fixed". Indeed, since that is true, why bother to tell the reader about the problem!
24. If Wonder Inc. complain and challenge you to produce the ‘virus’ that you claim they cannot find, take refuge behind a non-disclosure agreement that says that you cannot send out the specimen.
25. Don’t use viruses at all. Use simulated viruses. Assume that the simulation is perfect and that therefore all products should detect them.
26. Make a mistake in the summary table, accidentally giving WonderScan two stars when you meant four. When they complain, correct this in the next issue, in a little box that no-one will read. You can safely make the opposite mistake with GrottyScan; it is unlikely that they will complain at being given four stars.
Mistakes caused by these techniques are exploited by the marketing departments of all companies in the anti-virus market. At the end of the day, it is you the user who is being exploited.