Picking a good book

This story began with a headline that read Scientists find secret to writing a best-selling novel. You can read the whole of the story here and you can read the actual paper here. It was in the Proceedings of the 2013 Conference in Empirical Methods in Natural Language Processing.

I will just quote one paragraph from that paper: it says a great deal if you read it aloud, so you can savour the clumsiness in the construction.

For our experiments, we procure novels from project Gutenberg. Project Gutenberg houses over 40,000 books available for free download in electronic format and provides a catalog containing brief descriptions (title, author, genre, language, download count, etc.) of these books. We experiment with genres in Table 1, which have sufficient number of books allowing us to construct reasonably sized datasets.

I think it makes me wonder: do these people know how to write coherent English? To put it another way, I would not be asking them to review or revise one of my manuscripts.  They are junior academics, and I am sure they are trying hard, but they needed more sage advice before they rushed into print.


True, they had some quantitative measures such as counting the connectives (good) and mention of body parts (ludicrous). They also concluded that prepositions were good and verbs were bad, though apparently not as bad as foreign words, symbols and interjections!

Basically, Vikas Ganjigunte Ashok, Song Feng and Yejin Choi, a trio of computer scientists at Stony Brook University in New York, had grabbed a whole bunch of classic books from the Project Gutenberg collection.  Then they analysed the  texts and compared their measures with historical information on the success of the work.

They ran their study over a number of books in different genres measuring things like “interestingness”, novelty, the style of writing, and how engaging the storyline was. Yeah, right, no chance of any halo effect there, eh?  No chance of somebody saying "This is a book about Harry Potter, so it mustn't be very interesting, right?


Wrong. That's the first flaw: the analysis is far from objective.

What followed reminded me of a dodgy attempt by the Control Data Corporation in 1980 or 1981 to foist an outdated and kludgey system called PLATO on four government instrumentalities in New South Wales: they wanted to sell the BSLS (Basic Skills Learning System) to Correctional Services, the Apprenticeship Directorate, the Education Department and TAFE.

I am fairly certain in retrospect that somebody had been bribed, because it was getting an and uncritical easy run, with a total of $200,000 to be shelled out--a lot of money back then, when a deputy principal got less than $20,000 a year. Subtext: it was a lot of money!

My boss thought that there was a rodent-like odour to the proposal, and asked me to have a play with the evaluation reports they had given us, some 400 words of raw (and cooked) data.  He and I had done the same Master's course in evaluation and he knew my weird penchant for dissecting out buried dross and detecting things that do not compute. It didn't take long before my antennae began to twitch.

No wonder: there was dross everywhere.  From memory (the original papers are two floors down, in an unmarked box), they showed us an evaluation where the control group size was zero (but the control group still had scores). There were other cases where the control group was scoring at the ceiling level of a test, but the experimental group had the grade levels of their equivalent scores extrapolated.

In plain language, the control groups were fitted with hobbles and a ball and chain, the experimental group (the ones using PLATO) were issued with JATO packs, and then the groups were sent out to run a race.

The perceptive reader may be wondering why I don't mention the Hawthorne effect here.  Yes, that was alive and running as well, but that was minor compared with the procedural fudging that was going on.

These people weren't even good cheats. In the batch of research papers they sent (apparently hoping we would weigh them rather than read them), they accidentally included a memo where a distraught underling told the shiny gung-ho boss at Control Data that the whole thing was a fiasco, and I was able to look at the dates on which the shiny boss had published his latest glowing reports. I could show that they came after the date received that he had stamped on the deadly memo.

I had this bloke chapter and verse, because like him, I used to submit stuff to the ERIC clearing-house. I knew when one of my submissions was received at ERIC, and I had a lower accession number than one of his glowing reports, but the date stamp on the memo was even earlier than my submission.  It followed that after he got the heads-up, he kept on flogging the dead horse.

In short, these people were frauds, but they were hopelessly incompetent frauds because they gave me the evidence on a plate, but the funniest part was one study where they had used regression analysis to present an equation which claimed that the students using BSLS would gain one whole grade BEFORE they started the course. That's what happens when you apply any sort of statistical analysis without knowing what you are doing.

That was what reminded me of the book-quality study complained of above.  You see, figures don't lie, but liars can figure.  Proof is here.

I was told to bury my report, which made me wonder if somebody higher-up either suspected or knew that bribes had gone out.  I know from the attitudes of the people I spoke to that they weren't on the take — either that, or they were better actors than I would expect.

The purchase was blocked, so I did as instructed, leaving my report in a drawer for several years. Then I learned that a known muppet and dodgy character (he back-stabbed me once) was spruiking PLATO at AARE (the Australian Association for Research in Education).

So, being full of charity and having itchy shoulder blades, I gave a paper the following year, outlining the damning faults and showing what a rort the whole PLATO thing had been.  I guess it had the desired effect, as the muppet left the country.

I kept all of the evidence, just in case one of those I foiled ever reads this and decides to try a lawsuit.  It won't fly, my friend. It's past history, but if you want to be silly, I eat people like you.

Note in September 2014: I have now told the whole PLATO fraud story, in complete and technical detail. Note the comment about detail: the new piece isn't for the faint-hearted.


Coming up to modern dodginess, the funny thing was: before I even got through the abstract of the book-quality paper, the same antennae were twitching.

All I can say of this paper is that I hope no publisher ever hears about this idea, because this appears to be a great example of what we old computerists call GIGO.  They tried...

Or was this one of the 120 papers of computer-generated gibberish that Cyril Labbe of Joseph Fourier University detected recently?

That's about the only logical explanation.