"Marketing Pornography on the Information Superhighway"
July 2, 1995 (version 1.00)
Donna L. Hoffman & Thomas P. Novak
Associate Professors of Management
Co-Directors, Project 2000
Owen Graduate School of Management
In this critique, we provide a detailed analysis of the recent article "Marketing Pornography on the Information Superhighway..." (Rimm 1995, Georgetown Law Journal, Volume 83, June, pp 1849-1934) that was also the subject of a recent Time cover story (Elmer- DeWitt, July 3, 1995). For a detailed critique of the Time article, see Hoffman & Novak (July 1, 1995, version 1.01). First, we offer general comments about the study. We criticize the study on conceptual, logical, and process grounds, including: 1) misrepresentation, 2) manipulation, 3) lack of objectivity, and 4) methodological flaws. Second, we provide a series of detailed examples that support our general conclusions of these four major difficulties. For ease of exposition, these specific comments follow the order of the article. Our objective with this note is to begin a constructive and open critique process. Thus, our note is not meant to be an exhaustive cataloging of the lapses, discrepancies, inconsistencies, and errors in this article, only a summary of those we consider to be among the most severe. We do not debate the existence of pornography in "cyberspace." Indeed, pornography exists and is transmitted through many media, including cable television, books and magazines, video tapes, private "adult" bulletin boards, the postal mail, computer networks, interactive multimedia like CDROM, fax, and telephone, to name a few. What we dispute are the findings presented in this study concerning its extent and consumption on what Rimm calls the "information superhighway." The critically important national debate over first amendment rights and restrictions on the Internet and other emerging media requires facts and informed opinion, not hysteria. The critique is also important because the Time cover story has given the Rimm study a credibility it does not deserve. If it is difficult for a professional journalist to evaluate the validity of such research, it is reasonable to assume that many others will have difficulties, as well. The general and specific comments that follow represent our professional opinion of the critical flaws and errors in the Rimm study. Our critique has benefitted from the impassioned discussions in the WELL Media conference (topic 1029), of this study and the larger debates of media responsibility and first amendment rights. ----------------------------------------------------------------- General comments * Misrepresentation * The study is positioned as "marketing pornography on the information superhighway." Yet, it deals neither with marketing nor the information superhighway, and displays a considerable lack of understanding of both areas. Out of 148 footnotes in Rimm's manuscript, only one (footnote 22) cites a reference to a marketing journal. For a study purporting to deal with "marketing pornography on the information superhighway," this demonstrates a blatant disregard and ignorance of the marketing literature. As "marketing" appears as the first word in the title of this manuscript, and the word "marketing" appears frequently throughout the manuscript, it is particularly disturbing that Rimm does not support any of his "marketing" insights with references to the marketing literature. * This is a study of descriptions of pornographic images on selected adult BBS in the United States. The author finds, not surprisingly, that adult BBSs contain pornography. While the author attempts to generalize beyond this domain to the "Information Superhighway," no generalization is possible, and the results of this study should not be used for this purpose. Unfortunately, the juxtaposition of unrelated analyses of adult BBSs and Usenet newsgroups may create in the casual reader's mind the impression that what is stated about adult BBSs is also true of the "Information Highway" as a whole. We caution, in the strongest possible terms, against such misinterpretation on the part of the casual reader. Rimm concludes his study by saying (p.1915) "These and other findings may assist policymakers and others concerned with the future of Cyberspace to make informed decisions, with reliable data, about the evolving Information Superhighway." Unfortunately, this paper provides no actionable insights for policymakers about the future of Cyberspace, as the results, at the maximum, can only be generalized to all adult BBSs in the United States. * The study is positioned as the product of a "research team" at Carnegie Mellon University. It is described throughout as the "Carnegie Mellon study" and it is frequently mentioned that the "research team" estimated this percentage or counted that set of items. Yet, nowhere is it mentioned that Rimm was an undergraduate electrical engineering student at CMU at the time the study was performed. Instead, Rimm is listed as a "Researcher" and "Principal Investigator." (Note that the four funding sources are identified only as coming from Carnegie, but the type and kind of grants these are is not revealed.) This positioning capitalizes upon the reputation of Carnegie Mellon University, lends an air of authority and credibility to the paper, and increases Rimm's own authority by association. The article is sole-authored. Not a single member of the extensive research team shared in the credit for the authorship of this paper. Given established standards of authorship as ownership of intellectual property in the academic and scientific community, we can only infer from this that no one on the "research team" felt their contributions merited the significance of shared authorship. * Manipulation * The study was not subjected to peer-review. The manuscript was deliberately "embargoed" for at least six months prior to publication, and was not made available to interested researchers. This is highly unusual. The paper was submitted to a law journal which is not peer- reviewed, despite the fact that it probably would be more appropriate in a behavioral science or public policy journal (most of which are peer-reviewed). Since law journals have no one on the board to evaluate the merits of the methodology and likely not even the distinctions among BBSs, Usenet news groups, the Web, and the Internet, we offer the following hypothesis: did Rimm place his article somewhere where it would appear credible and go unchallenged? At some point, an agreement was negotiated in which Time magazine obtained an advance copy of the manuscript in exchange for an "exclusive." This was used in preparation of the July 3, 1995 Time cover story written by Philip Elmer-DeWitt. Given the vast array of conceptual, logical, and methodological flaws in this study, documented thoroughly below, Time magazine behaved irresponsibly in accepting the statements made by Rimm in his manuscript at face value. Time had a responsibility to its readers to do its own peer reviewing, despite the embargo. Indeed, Time reporters were made aware that the study appeared to have serious conceptual, logical, and methodological flaws that Time needed to investigate prior to reporting its story. If Time was not able to evaluate the manuscript on its own, Time should have held the story until the manuscript was publicly available, so that expert opinion could have been solicited, or sought its own panel of objective experts for a "private" peer review. In this way, Time would likely have recognized the study for what it was and not what it purported to be and prepared a balanced, critical report on the subject of digital pornography. Instead, Time presented, around lurid and sensationalistic art, an uncritical and unquestioning report on "cyberporn" based on Rimm's flawed study. This has had the extremely unfortunate effect of giving the study an instant credibility that is not warranted nor deserved and fueling the growing movement toward first amendment restrictions and censorship. * The study appears to be driven by an underlying political agenda. It is difficult to read the paper in its entirety and not come away with the conclusion that it is written in a manner which provides policymakers with the ammunition they need to obtain support for legislation that would censor certain types of information on the Internet and other emerging media. * Lack of Objectivity * Rimm makes numerous unsubstantiated causal statements. These causal statements are not supported by the data. In many cases, the causal statements are inflammatory and outrageous. Sometimes they are ridiculous. Additionally, data are often interpreted in a biased and selective manner. * Methodological Flaws * The article is rife with methodological flaws, several of them extremely serious. The origins of many numbers presented in the article are difficult, if not impossible, to determine. Much greater attention is paid to sensationalistic and inflammatory descriptions of image files, for example, than accurate descriptions of survey methodology. In fact, in many cases important aspects of the methodology are simply not described at all. Methodological details are either omitted entirely or presented in such sparse detail that it is impossible for other researchers to 1) determine what Rimm actually did and 2) replicate the results. * The study contains numerous discrepancies that cannot be resolved and raises a series of fundamental procedural, analytic, and implementation questions that can only be addressed outside of the article itself. * Operational definitions of "pornography" are ad-hoc, inconsistent, and misleading. * Much of the data presented is consistently misinterpreted, particularly the Usenet data. * The paper describes the results in a confusing manner which makes it very difficult to determine what Rimm actually did. The manuscript is way too long and rambles. It is organized in such a manner as to obscure the methodological issues. This makes it difficult for the casual reader to draw his or her own conclusions about the merits of particular results from the study. For example, discussions of Usenet readership at a single university are interwoven with worldwide Usenet readership statistics. This is confusing and makes it easier to misinterpret his results, thinking that he might be talking about Usenet in general when in fact he is only talking about readership at a single university. Definitions of online media are similarly presented in such a way that the reader is likely to draw the conclusion that BBSs, Usenet news groups, the World Wide Web, and the Information Superhighway are all one and the same, and what applies in one domain, is relevant to all. * Rimm makes numerous unsubstantiated leaps of faith in his logical arguments. * The research methodology is not up to the rigorous standards of a peer-reviewed journal. * The study procedure raises a number of troubling ethical questions. ------------------------------------------------------------------ Specific Comments Title and acknowledgement (p. 1849) * The article's title states it concerns the marketing of pornography on the so-called "information superhighway,". yet it appears in a law journal that is, by custom, not rigorously peer-reviewed. The acknowledgement indicates that organizations and experts in pornography were consulted (but not listed), but no organizations or experts conversant in marketing research, survey methodology, and marketing on the Internet and related online markets appear to have been consulted. I. Overview (pp. 1849-1864) * Rimm: "The ... study adopts the "definition" utilized in current everyday practice by computer pornographers. Accordingly, "pornography" is defined here to include the depiction of actual sexual contact [hereinafter "hard-core"] and depiction of mere nudity or lascivious exhibition [hereinafter "soft-core"]...Accordingly, data was (sic) collected for this article only from bulletin board systems (BBS) which clearly marketed their image portfolios as "adult" rather than "artistic." Any BBS or World Wide Web site which made even a modest attempt to promote itself as "artistic" or "informational" was excluded." (fn. 1) Rimm's definition of pornography is central to his study. It is therefore reasonable to expect a detailed analysis of what pornography is, along with arguments for how it may be defined and measured. Such discussion would include the advantages and disadvantages of each measurement approach and lead to a reasoned position of the operational definition employed in the study. Because the results of his study depend on his definition and measurement scheme, it is surprising that the definition he proposes is so weakly supported, and fluid besides. For example, in analysis to follow, Usenet newsgroups appear to be classified as "pornographic" if they contain the word "sex" in the title (except for alt.safe.sex), or if he judges them to be so. Further, the footnote is misleading, because it implies that Rimm studied the Web with the same energy that he applied to adult BBSs, when in fact, he only searched the Web in order to locate and provide a simple count of sites judged to be "sex-related." (Appendix C, p. 1923 ff). * Rimm: It is essential to note that Usenet and the World Wide Web are merely different protocols." (p. 1869) This statement is erroneous and suggests a disturbing misunderstanding of the nature of online media, particularly as they relate to consumers and providers. For definitions and discussion, see Hoffman and Novak's paper on Marketing in Computer-Mediated Environments (http://www2000.ogsm.vanderbilt.edu/cme.conceptual.foundations.html) * Rimm: "...this article discusses only the content and consumption patterns of sexual imagery currently available on the Internet and "adult" BBS..." (fn. 2) This statement is misleading because, in fact, the article discusses the content analysis of descriptive listings of images obtained from adult BBSs and the readership data from selected Usenet newsgroups. Usenet readership data can only tell that a Usenet group was accessed, but does not tell if any text files were read or any images were downloaded. * Rimm: "Every time consumers log on, their transactions assist pornographers in compiling databases of information about their buying habits and sexual tastes. The more sophisticated computer pornographers are using these databases to develop mathematical models to determine which images they should try to market aggressively." (p. 1850-51) Every time consumers log on to what? In the final analysis the article provides very little evidence, other than anecdotal or case study, to support the idea that pornographers are engaging in such activities. * Rimm: "Computer pornographers are also moving from a market saturation policy to a market segmentation, or even individualized, marketing phase." (p. 1851) The statement is misleading because it implies that pornographers have a strategic policy which is now shifting. However, the article supplies no evidence of the original policy, let alone the shift to a market segmentation strategy. * Rimm: "It is clear that pornography is being vigorously marketed in increasingly sophisticated ways and has now found a receptive audience in a wide variety of computer environments." (p. 1852) The article supplies no evidence that pornography is being "vigorously marketed," nor does it define marketing. The study does not investigate audience receptivity in a "wide variety of computer environments." Instead, it studies download records from selected adult BBSs in the United States and Usenet postings (but not Usenet downloads). Thus, this conclusion cannot be supported from the research presented in this paper. * Rimm: "'Information Superhighway' and 'Cyberspace' are used to refer to any of the following: Internet, Usenet, World Wide Web, BBS, other multimedia telephone, computer, and cable networks." (fn. 7) These two definitions are misleading and do not conform to commonly understood meanings of the terms by researchers and experts in the field. * Rimm: In the top paragraph on page 1853, Rimm argues that his study is the first to systematically examine "pornography on the Information Superhighway," and that it is now possible to obtain "vast amounts of information about the distribution and consumption" of pornography on a much larger scale than previously possible. Are there any previous studies of pornography on the "Information Superhighway," even if unsystematic? In what ways does this study have to do with the "Information Superhighway?" A framework should be developed for adult BBSs - the focus of this study - in the context of the "information superhighway." For example, what percent of traffic do adult BBSs represent of the total "highway?" What percent of users of the "highway" use adult BBSs? What do the distributions look like nationally and internationally? And so on. * Rimm: "...it maybe be difficult for researchers to repeat this study, as much valuable data is no longer publicly available." (fn. 9) This is an astonishing and intellectually suspect statement, almost transparent in its effort to set up a case that this study cannot be falsified. If subsequent research shows disagreement with the results of this study, Rimm can discount such results by saying that it could not be repeated anyway. Instead, good scientific practice demands Rimm work to show how the study can be replicated by subsequent researchers. However, as analysis below argues, even the analyses here cannot be replicated because Rimm provides no details of methodology which would enable that to happen. * Rimm: The first full paragraph on page 1853 discusses the 917,410 "pornographic" items downloaded 8.5 million times that form the bulk of the study.) Subsequent sections of the paper show that this paragraph is misleading in the extreme, as is the article title. The title of the article suggests the research will concern a "survey of 917,410 images, descriptions, short stories and animations downloaded 8.5 million times." (Note that Rimm does not perform "survey research" in this study, as no one is surveyed.) On page 1853, the 917,410 items are broken down as: * 450,620 items downloaded 6.4 million times from 68 adult BBSs * 75,000 items with an unspecified number of downloads from 6 adult BBSs * 391,790 items with no download information from 7 adult BBSs These items include images, animations, and text files. Rimm says that 10,000 "actual images" were "randomly downloaded" from adult BBSs, the Usenet or CD-ROM and used to verify the accuracy of the descriptive listings. Rimm does not, however, 1) report the methodology used to randomly select the images, 2) provide frequency distributions of the images across the media they were obtained from, 3) specify the exact media used to obtain the listings (e.g. which CDROMS?), nor 4) indicate how the accuracy verification procedure was performed. In footnote 10 on page 1853, Rimm says the original number of downloads was counted at 6.4 million and that "a total of 5.5 million downloads are analyzed here." (emphasis ours). He explains that the other "0.9 million concern animations, text, and other miscellaneous files" which he presumably excluded from analysis. He continues that "an additional 2.1 million downloads was later obtained from...Amateur Action BBS. In this way, the total number of downloads tabulated is 8.5 million." We note that this tabulation of 8.5 million downloads is misleading for two reasons: 1) Rimm did not specify the period of time in which the 8.5 million downloads accumulated. Was it one month? One year? Five years? Ten years? 2) Rimm did not actually analyze 8.5 million unique downloads, as at least some were apparently excluded from analysis. While 8.5 million exposures to pornographic images may sound like a large number, let us put it into perspective. Suppose a pornographic newsstand magazine had a circulation of 500,000, including subscriptions, newsstand sales and pass-along readership. If there were 10 pornographic photographs in a single issue of this magazine, there would be 5 million "exposures" in this single issue alone. Thus, 8.5 million must be set in a context which specifies the time period, and the equivalent exposures in "competing" media during this time period. * Rimm: "A total of 292,114 image descriptions remained and are discussed here. At least 36% of the images studied were identified as having been distributed by two or more "adult" BBS." (p. 1854) Apparently, Rimm analyzed 292,114 descriptive listings of images only, presumably representing 5.5 million downloads. No indication is given of how duplicates were identified as such, nor distributed across the listings, either by individual adult BBS or by, for example, geographic region. In footnote 11 on page 1854, Rimm suggests that whatever method was used to identify duplicates had its validity confirmed by randomly sampling 100 "suspected duplicates," and presumably examining them. Yet, he does not indicate how he "suspected" them in the first place, how they were sampled, and how the validity was "confirmed," as no details or statistics are provided to support the statements.) * Rimm: Part II of this article addresses three issues concerning pornography on the Usenet: (1) the percentage of all images available on Usenet that are pornographic; (2) the popularity of pornographic boards in comparison to non- pornographic boards, at both a university studied and worldwide; and (3) the origins of pornographic imagery on the Usenet. (p. 1854) Our critique of Part II will show that due to serious methodological flaws, the study does not, in fact, provide accurate data on these issues. * Rimm: All BBS data was (sic) collected in May and June 1994, unless otherwise noted. (p. 1855) In footnote 15 on page 1856, Rimm states that the study "tracks image repertoires over a fourteen-year period." No clarification is provided here or subsequently to reconcile the discrepancy between the two-month data collection period and the 14 years or to illuminate on how the study follows "image repertoires" longitudinally. * Rimm: "...this study focuses entirely upon what people actually consume, not what they say they consume; it thus provides a more accurate measure of actual consumption." (p. 1855) Rimm analyzes aggregate download counts of descriptive listings of images available on adult BBSs. Although download patterns would be expected to correlate with actual consumption (i.e. viewing), we do not know the extent to which individuals actually looked at the images (or, indeed, whether they looked at all). These limitations are not addressed in the study and no thoughtful discussion of the consumption experience is ever provided. Further, absolutely no download behavior on Usenet news groups was ever examined by Rimm. * Rimm: Because the data is (sic) in many respects exhaustive, statistical techniques and assumptions that are commonly invoked to impute general consumer behavior are not necessary for this dataset. Thus the research team considers the inferences drawn highly robust." (p. 1856) Are the data really exhaustive of adult BBS? Rimm does not provide evidence that the listings obtained from the BBSs represent a census. Further, the statement that statistics are not necessary for these data is astonishing. No evidence, statistical or otherwise, is ever provided in the article that the inferences drawn from these data are, indeed, "robust." * Rimm: "The ... study examine 917,410 images, image descriptions, short stories, and short films..." (fn. 15) Yet earlier on page 1854 and footnote 10 on page 1853, Rimm suggests he deleted all but the images from the database under consideration and retained 292,114 for "discussion." Thus, how many items did the study actually examine? * Rimm: "The study results suggest a tremendous rift between the sexual activities in which Americans claim to engage, as reported most recently by the study Sex in America, and the sexually explicit activities presented in images that many Americans consume." (p. 1857) This statement is misleading, because Rimm did not study individuals, but aggregate download counts of descriptive listings of images available on adult BBSs. The Sex in America study surveyed the general population, and did not examine individuals' consumption behavior as measured in downloads on adult BBSs in the United States. In other words, the two studies examine two completely different populations. Thus, there is no basis for the conclusion that a "tremendous rift" exists and the statement represents an "apples and oranges" comparison. * Rimm: "Among the ultimate findings of this study are that digitized pornographic images are widely circulated in all areas of the country and that due to market forces, digitized pornographic images treat themes...which are not otherwise widely available." (p.1857) The conclusion is not supported by the data because Rimm examined only downloads of pornography on adult BBSs and readership statistics of selected Usenet newsgroups. He did not examine the distribution or consumption of pornography, by category or otherwise, in other media, nor does he provide evidence from others' examination. Thus, there is no basis for the comparison. * Rimm: "One of the more intriguing questions raised by this study is whether the general population will demand the same types of imagery currently in high demand among computer users." (p. 1857) This statement is misleading. All computer users? Some computer users? How many are "demanding" it now? What types? Indeed, why would the general population be expected to exhibit the same types of preferences as subscribers to adult BBSs, which is the only group of "computer users" for which Rimm studied imagery? * Rimm: "The widespread availability of pornography on computer networks may have a profound effect on those who wish to utilize the emerging National Information Infrastructure for non- pornographic purposes." (p. 1858) This statement is blatantly biased. Rimm did not examine the extent of pornography on "computer networks" such as the Internet or online services, and provides no discussion, nor references to balanced discussion of these issues. * Rimm: "While it may not be possible in the next decade for such technology to automatically classify images with the same precision as the Carnegie Mellon linguistic parsing software..." (fn. 21) The precision of the noted software is never established, let alone described in any detail. * Rimm: "More than two dozen faculty, staff, graduate and undergraduate students at Carnegie Mellon University contributed in some manner to this study." (p. 1861) Yet in fact, the article is a sole-authored study, performed when the author was an undergraduate student in Electrical Engineering, that was not subjected to the usual rigors of peer-review and revision that are common for this type of research. No one, other than Rimm, has accepted responsibility for the intellectual property in this study. Further, the individuals listed on page 1849 represent an acknowledgement by Rimm, rather than an endorsement by all of them of the manuscript. * Rimm: "After a year of exploring the Internet, Usenet, World Wide Web, and computer Bulletin Board Systems (BBS), the research team discovered that one of the largest (if not the largest) recreational applications of users of computer networks was the distribution and consumption of sexually explicit imagery." (p. 1861) As we continue to note, Rimm's study concerns download patterns on selected adult BBSs and readership statistics on selected Usenet newsgroups. Rimm may have explored these systems, but provides no evidence for the conclusion stated above. Further, Rimm's statement is misleading, as it implies that the largest recreational application is not just in downloads (i.e. "consumption"), but also in uploads (i.e. "distribution"). Rimm's study does not examine uploads. * Rimm: An unusual amount of data was (sic) freely available from commercial "adult" BBS, primarily as a consequence of the evolution of the online industry. Large commercial BBS such as American Online, Compuserve, and Prodigy do not carry hard-core pornographic imagery, either for legal or policy reasons. As a consequence, several thousand comparatively small "adult" BBS have sprung up across the country." (p. 1861) The statements are misleading because no evidence is provided to support the conclusion of a causal link between activities on commercial online services and adult BBSs. * Rimm: "In many instances, the research team was able to persuade the owners of these BBS to provide information about subscriber consumption habits." (p. 1862) This is a troubling statement. How was Rimm able to obtain such consent? Was it "informed consent?" Did Rimm provide full disclosure to these operators about the nature and objectives of his study? Did Rimm "debrief" them afterwards? Did he get the permission of the subscribers of these BBSs to examine information about their consumption habits? Did Rimm submit a proposal of his methodology for such "persuasion" to the University Human Subjects Committee? Did they approve the research and the methodology? II. Usenet (pp. 1865-1876) * Rimm: "This article will first discuss the methodology and results of the study of Usenet images and will then explain the methodology and results of the study of BBS images." (p. 1865) Footnote 28 (p. 1865) appended to this sentence refers the reader to footnotes 25-27 "for discussion of the distinction between the Usenet and commercial BBS." Such distinctions are critical for correct interpretation of Rimm's results and do not belong in footnotes. Nevertheless, examination of the footnotes reveals the following: footnote 25 "assumes that the reader has a basic understanding of Usenet and BBS." (p. 1862), and refers to reader to several books and a magazine; footnote 26 cites a 20-year old FCC document on "MTS and Wats," (p. 1863) and footnote 27 cites a brochure "on file with the Georgetown Law Journal" (p. 1864). * Pornographic vs nonpornographic imagery in the alt.binaries groups Rimm: Rimm states that he examined "[a]ll of the Usenet newsgroups with the prefix 'alt.binaries'" from September 21-September 27, 1994 and goes on to say that "[t]he number of new images posted each day was tabulated for both pornographic and non-pornographic newsgroups." (p. 1865) No rationale for excluding audio and text is provided other than they were "not the subject of this study." Does it make sense to look at all types of pornography on the Usenet and compare that to all other types of information? Rimm does not indicate how he determined which alt.binaries groups were pornographic and which were not. In what manner did Rimm control for duplicates, resent, or non-pornographic images? Did Rimm counts posts or a complete image? (Note that a single image could have up to 10 more files to make it complete.) In effect, what was the unit of analysis: a post or an image? On Saturday, 7/1/95, a colleague counted the number of posts on alt.binaries.pictures.erotica and found 1650 posts. One image was 41 posts long and represented 2.5% of the message volume alone. The article is moot on these important methodological details. * Popularity of Pornographic vs NonPornographic Usenet Newsgroups Rimm: "The research team was also able to examine the online habits of 4227 users at a mid- sized, private university in the northeast." (p. 1865) This raises troubling issues. How was Rimm able to conduct such examination? Did he obtain "informed consent" from each student? Did Rimm provide full disclosure to these students about the nature and objectives of his study? Did Rimm "debrief" them afterwards? Did the University Human Subjects Committee approve this examination? It is curious that Rimm argues in numerous places about the possible public policy implications of his work, but does not raise the ethical implications of conducting such research (only the implications of reporting it). See, for example, footnote 40 on page 1869, where he discusses his decision not to report "detailed demographics of the university population of computer pornography consumers" but makes no mention of whether it is appropriate to gather the data in the first place. * Rimm: In footnote 30 on page 1865, Rimm argues that the 11% of computer users at the private university "block" site statisticians from monitoring in order to "avoid detection" of their online activities. After discussing a behavioral analysis of child molesters, he proposes that "it is possible that some Internet users who block their accounts prefer sexual images of children and wish to avoid detection." This argument is one of the more outrageous in the paper and represents an invalid causal link. In the first place, there is no evidence that the 11% who "block" their activities are child molesters, and in the second place, there is no evidence that the 11% are representative of the broader population of Internet users. Thus, there is no basis for the proposal that Internet users who do not wish their activities monitored prefer to look at "sexual images of children." * Percent of pornographic imagery in Usenet binaries groups Rimm: "Among the pornographic newsgroups, 4206 image posts were counted, or 83.5% of the total posts." (p. 1867) The interpretation is incorrect and the number is grossly inflated. It is based upon 17 alt.binaries groups that Rimm considered "pornographic" and 15 alt.binaries groups that Rimm considered "non-pornographic." However, Rimm does not provide a listing of the names of these groups, no distributions of posts in these groups, and no methodological discussion of how he counted and determined posts were either pornographic or not, so there is no objective evidence of whether these groups are, in fact, "pornographic." Also, no information is provided on the degree to which these 32 groups comprise the complete universe of Usenet imagery. Further, as the methodology for counting the number of images is not specified, it is likely that even given Rimm's definitions and selection of 32 groups, the percentage is inflated due to the inclusion of non-pornographic next comments and multi-part images in the counts. What are the distributions of posts, by type of post (imagery, text, audio) in each of these newsgroups? What were the total numbers of posts to each group and to each set of groups and to Usenet overall during the period? How did Rimm determine that the 4206 image posts to the 17 supposed pornographic alt.binaries groups are, in fact, pornographic? A more accurate interpretation is that of 83.5% of the images posted to 32 alt.binaries newsgroups came from 17 groups that Rimm determined were pornographic. To make matters worse, Rimm grossly overgeneralizes his results in footnote 36 (p. 1868) and his summary (p 1914): "83.5% of all images posted on the Usenet are pornographic." This is a particularly misleading misinterpretation of his narrow result. * Misleading interpretation of "popularity" of types of Usenet newsgroups. Rimm: (p 1849) "'Pornography' is defined here to include the depiction of actual sexual contact...and depiction of mere nudity or lascivious exhibition." Rimm uses bold text to identify "newsgroups identified as having pornographic content" in Table 1 and Table 2. Included among pornographic newsgroups are "alt.sex" and "alt.binaries.pictures.supermodels." This is not consistent with Rimm's stated definition of pornography, as there is little of what would be considered pornographic content in these groups. It is a biased and inflammatory characterization of these Usenet groups. The column headings in Table 1 are not explained. Is the user base 4227 from page 1865 or some other number? This particular site receives only 3600 (p. 1870) of the 14,000 Usenet newsgroups (p. 1862) or only 25.7% of all groups. This seems like a small percentage of total groups. Is it? What do the percentages at other institutions look like? Without knowing this, it is difficult to generalize beyond this site to the entire Usenet domain. What would happen if we included data from the other 10,000+ sites? It is truly astonishing that there are no .comp or .news groups in the Top 40 Usenet news group at the university studied. Indeed, if the university is Carnegie Mellon, this is simply unbelievable. By this chart, only 99 readers are required in order to put it at number 40. Additionally, the Top 40 newsgroups in Table 1 differ dramatically from the Top 40 overall, according to the arbitron statistics. Rimm: In footnote 30 on page 1865, Rimm argues that "there is no reason to believe consumption at the university study differs from that of other universities from which pornographic Usenet newsgroups can be accessed." But, in fact, there are reasons to believe otherwise. A study conducted at Vanderbilt University (Varki 1995) as part of the requirements for a doctoral seminar on "Marketing in Computer-Mediated Environments" showed that the top Usenet news groups in terms of number of postings differs markedly and in important ways from the Top 40 list presented in Table 2 (p. 1872). Since this is a worldwide listing, intuition alone would suggest the likely presence of regional differences, at the least. In any event, no evidence is presented to support his reasoning in footnote 30. All of these problems suggest that the university in question may actually be fairly atypical in its use of Usenet newsgroups, which limits its generalizability. Rimm: "In broad terms, the research indicated that pornographic newsgroups are accessed more frequently during the school year than during summer recess. This suggests that, in comparison to teachers, faculty, and staff, a disproportionately large number of students access Usenet pornography." (p. 1969-1870) In fact, the conclusion does not follow since Rimm does not present evidence (e.g. counts, frequencies, and proportions) indicating how many students access pornography relative to the other groups. Rimm does not provide a version of Table 1 for the academic year, so that readers may draw their own conclusions. Rimm: "The fact that alt.sex.stories is currently more popular than alt.sex.pictures.binaries.erotica has been often misinterpreted as an indication that stories are more popular than images." (p. 1871) In fact, Rimm presents no evidence that such "misinterpretation" exists, although we can assume the interpretation exists. His alternative explanation is interesting, but no data are offered on how many users are discouraged by the level of technical sophistication required to access these groups. Indeed, a rival hypothesis is that these groups are accessed by a singularly technically sophisticated user, not the reverse. * Percentage of sites containing "pornographic" Usenet newsgroups. Rimm: (p 1871) "The worldwide statistics suggest that Usenet hosts appear less willing to offer their readers access to pornographic newsgroups than other types of newsgroups. 81.2% of the sites offer access to non- pornographic newsgroups, whereas only 55.8% of the sites offered their readers access to the pornographic newsgroups." In our opinion, Rimm has clearly misinterpreted the data. An examination of Tables 2 and 3 will immediately reveal that the important distinction is not between "pornographic" and "non-pornographic" groups, but between "alt" and "non-alt" hierarchies. All Rimm's "pornographic" groups are from the "alt" hierarchy. No alt group in Tables 2 and 3 is carried by more than 66% of sites. While alt.binaries.pictures.erotic (one of Rimm's "pornographic" groups) is carried by 53% of sites, alt.binaries.pictures (a "non-pornographic" group) is carried by only 49%, and alt.binaries.sounds.tv is carried by only 34% of sites. * Misleading portrayal of newsgroup readership. Rimm: (p 1873) "The newsgroups are ranked in Table 2 by the estimated total number of readers worldwide." Rimm identifies Brian Reid's "arbitron" script as the source of the data in Table 2. However, Rimm does not provide Reid's caveat on exactly what "readership" really means. Reid (Usenet Readership Summary Report for May 95) is careful to note that: "To 'read' a newsgroup means to have been presented with the opportunity to look at least one message in it." ... "Assuming that 'reading a group' is roughly the same as 'thumbing through a magazine', in that you don't necessarily have to read anything, but you have to browse through it and see what is there." This is a critical point. There is absolutely no information from Table 2 on how many of the 260,000 "readers" of alt.binaries.pictures.erotica actually downloaded and uudecoded a binary image file. The arbitron data is not tracking downloads. In fact, it would be completely consistent with Reid's definition of readership if none of the "readers" of alt.binaries.pictures.erotica ever saw a pornographic image. Thus, the results shown in Table 2 simply cannot be used to establish the exposure of "readers" to pornographic imagery. A reasonable hypothesis is that "readers" are simply curious about what is in these groups, and browse the titles to get some idea. As Rimm notes, decoding Usenet binaries requires a non-trivial degree of technical skill. We should further note that if one takes the estimate of individuals with Internet access as 20 million, then at most we are speaking of about .1% of Internet users accessing the alt.binaries.pictures.erotica newsgroup, and almost surely, the percentage actually downloading and uudecoding pornographic images is much lower than even this very low percentage. Rimm: In footnote 31 on page 1866, Rimm suggests that the (presumably total) number of readers of alt.binaries.pictures.erotica on Usenet is 260,000 per month. Rimm provides no discussion of the methodological details necessary to understand this estimate. How is this number estimated? How are multi-part image files counted? How are robot extractions handled? Are these 260,000 people unique? Or, could they possibly represent, for example, the same 9000 individuals per day for 30 days? How does the "arbitron" script keep track of individual users? In other words, are reach and frequency confounded? Does Rimm know? * Amount of pornography in Usenet groups. Rimm: "Of this 11.5%, approximately 3% [of messages on the Usenet] is associated with Usenet newsgroups containing pornographic imagery." (p. 1869) Rimm fails to take these traffic percentages to their logical conclusion, which is that less than 1/2 of 1% (3% of 11.5%) of the messages on the Internet are associated with newsgroups that contain pornographic imagery. Further, of this half percent, an unknown but even smaller percentage of messages in newsgroups that are "associated with pornographic imagery" actually contain pornographic material. Much of the material that is in these newsgroups is simply text files containing comments by Usenet readers. Rimm: (p 1873) "Moreover, 20,644 of the 101,211 monthly Usenet posts in the top forty newsgroups, or 20.4%, are pornographic. This figure is inflated and incorrect. Rimm is assuming that 100% of the content of the so-called "pornographic" newsgroups in Table 2 is pornography. But, this is obviously incorrect. A large number, if not the majority, of messages in these groups are simply text representing discussion and comments - not pornographic. In addition, large images are typically broken into multiple parts, so that one large .gif file might actually consist of ten or more physical files. Further, even single file images often have a separate descriptive header (which should be considered non-pornographic). While it is impossible to determine from the results Rimm has presented what proportion of monthly Usenet posts are "pornographic," we can safely conclude that the percentage is far below what Rimm states. * Origins of pornographic imagery on the Usenet. Rimm: (p 1874) "71%, or 1671 of the 2534 pornographic images downloaded from the five Usenet newsgroups studied over a four month period, originated from "adult" BBS." This is a critical percentage, yet we question its validity. Virtually no support is given for this percentage other than Rimm's in the text statement that 1671 images originated from adult BBSs. We cannot determine how Rimm arrived at this number from our reading of the manuscript. Is it an estimate? A count? How was it estimated or counted? Rimm lists the five Usenet newsgroups on which he says "[t]he largest selection of sexual imagery was discovered" at the northeastern university (p. 1866) and notes in footnote 32 (p. 1867) that these sites were the largest available at the "research site." He further says that between "April and July of 1994,...all available images (3254) [were downloaded] from these five newsgroups." (p. 1867) There must be a typographical error, because earlier Rimm stated that the alt.binaries groups were not examined until September of that year, so it cannot be possible that months earlier he was able to determine the groups with the largest selection of sexual imagery. The appearance of alt.sex.fetish.watersports is also confusing since it is not an alt.binaries group. It is possible that it all makes sense, but it is very difficult to sort out from the confusing exposition. We also wonder if group size and availability are confounded with amount of imagery. The main issue is that convincing evidence has not been presented that these five groups contain the largest selection of sexual imagery. Where did this list come from? These groups are at a single university site. Was a systematic analysis of all Usenet groups performed to generate this list? Did Rimm control for duplicates, resent, or non- pornographic images? Did Rimm counts posts or a complete image? What was the unit of analysis: a post or an image? Rimm states that the images from the five Usenet groups were classified into three categories (p. 1867): 1) images originating from adult BBSs ("the name, logo, and telephone number of the BBS appeared next to or within the image."; 2) pornographic images which did not originate from BBSs; 3) "PG/R" images ("no sexual contact or lascivious exhibition."). Curiously, there is no category for images were are not pornographic! Was every single image on these groups pornographic? Rimm does not indicate whether these categories are mutually exclusive; for example, how were "PG/R" images with a BBS logo counted? In any event, Rimm states (on page 1867) that there were "a total of 2830 images for analysis," but does not report the frequency of images in each of the three categories. (He states that 13% of the images could not be downloaded, which makes us wonder whether other figures presented need to be similarly adjusted to account for technical difficulties which must ultimately lower consumption rates.) However, seven pages later, the total number of pornographic images downloaded from the five groups shrinks to 2354 images, with no explanation! If we accept the 1671 as indicative of the number of images in those five groups that Rimm determined came from adult BBSs, then the percent of images originating is 59% (1671/2830) if we use his first number and 71% (1671/2354), if we use his second. Rimm: (p 1874-5) "For those who consider pornography to include the additional 476 'PG' or 'R' rated images defined in the methodology section, 59% of all Usenet images originate from 'adult' BBS." Since adding counts to the numerator of a fraction must increase the resulting percentage, 71% followed by 59% cannot be correct. Of course, perhaps this is a typographic error or an error of confusion. Was the 59% meant for the first percentage reported using the first (and larger) denominator? In that case, is the 71% also a typographical error or error of confusion? In any event, this additional calculation makes no sense unless all images in the third "PG/R" category were exclusively from adult BBSs. Thus, the percentage of 59%, which should be higher than 71% but must be a typo), is misleading until clarification can be provided! Note that the percentages are suspect (Which denominator is "right?" The larger? The smaller? Neither? Are 1671 and 476 even the correct quantities?), not only because of the confusing manner in which the information is presented, but also because of the more serious methodological criticisms made earlier about the selection of the five newsgroups on which these numbers are apparently based. III. Pornographic "Adult" Commercial BBS (pp. 1876-1905) * Number of adult BBSs examined Rimm: "...[T]he team either subscribed to, or logged on as a new user or guest, to a number of representative pornographic BBS and collected descriptive lists of the files offered by each." (p. 1876) Rimm reports that Boardwatch estimates that 5% of BBSs in the country are "adult," (fn. 35, p. 1867) but does not report a figure on the total number of BBSs, only that 5000 BBSs of any type were identified (p. 1877) and that 500 "active" adult boards were located for further study. Since this represents 10% of his list, we can assume that Rimm's list of BBSs was not complete. Rimm indicates that "most" of these 500 adult boards were "chat" boards, and still others were "transient." He gives no figures on how many comprised each category. Rimm: "To the best of the research team's knowledge, the BBS included in this study comprise most of the medium- and large-sized "adult" BBS in the country that existed at the time of the research." (p. 1877) Rimm does not indicate how many boards this represents, how they were sampled to be "representative" (p. 1876), whether the list of adult BBSs Rimm sampled from was exhaustive, or whether Rimm used his "judgment" in selecting BBSs or in generating the list of BBSs to sample from. * Number of descriptive lists examined Rimm: "This portion of the study analyzes a total of 450,620 files that are classified..." (p. 1876) Previously, Rimm indicated that 292,114 descriptive listings were retained for analysis. How many listings were actually collected? How many pornographic images do these listings represent? How many were movies? How many were text files? How many images were selected from the BBSs and how were they selected? Rimm indicates that both "descriptive lists" of pornographic images as well as a "representative sampling of the images themselves" were collected from the BBSs (p. 1877). Rimm does not say how many images were sampled, how they were sampled to be "representative," or what they were supposed to be representative of. * Ethical lapse? Rimm: "Members of the research team did not, as a rule, identify themselves as researchers." (p. 1878) As before, this is troubling. Why didn't Rimm identify himself and his research objectives to the operators? Did Rimm obtain permission to "collect" the information from the BBSs? Did Rimm provide full disclosure to these operators about the nature and objectives of his study? Did Rimm "debrief" them afterwards? Did he get the permission of the subscribers of these BBSs to examine information about their consumption habits and report the cities they lived in (see Appendix D: pages 1926-1934)? Did Rimm submit a proposal of his methodology for such to the University Human Subjects Committee? Did they approve the research and the methodology? Does Carnegie Mellon approve of publishing the cities that consumers of adult BBSs live in? How did Rimm obtain the demographic information on adult BBS subscribers? (as noted on p. 1895) * Results of the linguistic classification scheme Despite twelve pages of largely anecdotal discussion of the content analysis of the descriptive listings, the methodology is never once described formally, either in terms of the algorithm, or the software used to implement the algorithm. In the scholarly literature it is not only customary to offer the software to those who wish to replicate your results, for some journals it is mandatory (as is making the data available). Nowhere does Rimm indicate that the data or the software that categorized the listings are "available from the author." Validity and reliability are not established. This despite the fact that standard statistical procedures are available for determining reliability and validity. The few numbers that are presented in this section are either poorly defined or not defined at all. Other quantities are mentioned as being "high," but not reported (e.g. see footnote 70 in which Rimm asserts that "[t]he presentation of kappa values...was considered unnecessary because of the high level of reliability." Yet this "high level" is never reported). Elsewhere, Rimm suggests that "validity was high," (p. 1888), but it too is not reported statistically; or, Rimm states that he performed "a statistical analysis" on the data (p. 1894), but the type of analysis nor its results are not reported. Such examples, which render the statements they are intended to support, meaningless, are too numerous to catalog here. Relatedly, numbers or data are not reported that would help the reader understand the analysis, and numbers that are reported are pursued for additional insights. This section is ad hoc and weak; no reliable and valid conclusions can be drawn from the analysis as presented. Moreover, this is a standard content analysis problem. Content analysis has a large and rich literature, yet there is not a single citation to the either that vast literature or the related areas of AI software, and classification and categorization. As Rimm presents it, it is not possible to replicate the categorization he performed, let alone determine how he performed it. Thus, the methodology and this entire section, are impossible to evaluate. Numerous questions must be raised: What time period or periods are represented in the listings? Did Rimm control for time in his analyses? Were the data adjusted to account for differing lengths of time of each listing? For example, adjusting for date first posted on the BBS? Rimm's procedure implicitly assumes that all downloads are a function of consumer demand and no other variables. What about availability of certain kinds of images? The cost of the images? Their size? Consumer demographics? Rimm states that "[m]any BBS either hide [the listing] information from their customers or do not provide it." (pp. 1879-1880). But on page 1878, Rimm states that listings have a typical record structure which he diagrams in Figure 3. If the information is hidden by "many" operators (how many hid it in his study?), or many do not provide it (how many did not provide it in his study?), how did Rimm get it? Rimm suggests that operators were "persuaded" to provide the information "privately." (p. 1881) What does this mean? How were they persuaded? What is meant by "privately?" How valid are the sixty-three basic categories? Were the categories validated by judges? Did human beings ever look at any descriptions to validate the classification scheme? If so, how many? What exactly was the procedure the judges went through as part of the classification process? Rimm notes that "judges...were not avid consumers of pornography and thus did not recognize names of particular pornographic 'stars." (p. 1886) Did this lack of experience on the part of the judges affect or bias the classification procedure? Typically, judged are chosen for their expertise. How was the "final precedence scheme" in Figure 6 arrived at? Given that Appendix A describes dozens of categories, why are percentages not reported for individual categories within the major groupings. These percentages are important to know because some of the individual categories may be considered less extreme than others. Without knowing the distribution of categories of images within each broad group, it is difficult to know what the group actually represents. Rimm: Thus, analyzing the data in these four classes presents a highly reliable means of exploring the explosive growth of pornography on the Information Superhighway." (p. 1886) Rimm never shows that his method of analysis is "highly reliable" and Rimm never shows that the growth in pornography is "explosive," on the "information superhighway" or anywhere else. Page 1889 adds nothing to the reader's understanding of the methodology. What is the point of including this discussion? On page 1890, Rimm notes that there were 35 adult BBSs. How is this figure reconciled with the 68, 6 and 27 adult BBSs discussed on page 1853? What is the point of including the Amateur Action BBS Case Study (pp. 1896-1905)? In general, the conclusions Rimm makes are not supported by his analysis. Because the content analysis and classification scheme are "black boxes," because no reliability and validity results are presented, because no statistical testing of the differences both within and among categories for different types of listings has not been performed, and because not a single hypothesis has been tested, formally or otherwise, no conclusions should be drawn until the issues raised in this critique are resolved. IV. Conclusions Rimm: "[A]ttempts at regulation may be ineffective; "the net interprets censorship as damage and routes around it" is a well known expression among Usenet enthusiasts. The current structure of the Usenet requires that individual sites choose between an "All things not expressly permitted are prohibited policy," or conversely, an "All things not expressly prohibited are permitted." A middle ground does not appear viable. Curiously, Rimm does not consider the alternative of user-imposed, rather than state-imposed controls.