ESV and inadequacy of reading level tests

My previous post linked to today's ESV Bible blog post listing Flesch-Kincaid (F-K) readability levels determined by running ESV text through the grammar checker of Microsoft Word. I pointed out that the F-K test is inadequate for checking readability of texts which are not written in contemporary English. I previously posted on some inadequacies of F-K, as did commenters on that post. Others have written more technically about deficiencies of the F-K and similar reading tests. For instance, click here to download a technical article and start reading in the middle of page 29. Also click here to download another technical article on this subject. Here I wish to illustrate some of the problems by running the F-K test on portions of one of the plays of William Shakespeare, "All's Well That Ends Well," written in 1603 A.D. before the KJV was published in 1611.

Here is the first soliloquy by the Countess:

Be thou blest, Bertram, and succeed thy father
In manners, as in shape! Thy blood and virtue
Contend for empire in thee, and thy goodness
Share with thy birthright! Love all, trust a few,
Do wrong to none; be able for thine enemy
Rather in power than use, and keep thy friend
Under thy own life's key; be check'd for silence,
But never tax'd for speech. What heaven more will,
That thee may furnish, and my prayers pluck down,
Fall on thy head! Farewell. My lord,
'Tis an unseason'd courtier; good my lord,
Advise him.

F-K says that this small text has Flesch Reading Ease of 81.2 and F-K Grade Level of 4.5. But I know of no readers of current English of a Grade 4.5 level who could understand a large percentage of what the Countess said. F-K simply is not designed to pick up on the older vocabulary, such as "unseason'd" and "courtier." I don't know if current elementary students understand the older second person pronoun "thy." Many of the syntactic phrasings of this short text would not be understood by current English readers of Grade 4.5 level.

Let's see how lines spoken by Helena in the same play are analyzed by F-K. Here is Helena's first long soliloquy:

Here is O, were that all! I think not on my father;
And these great tears grace his remembrance more
Than those I shed for him. What was he like?
I have forgot him; my imagination
Carries no favour in't but Bertram's.
I am undone; there is no living, none,
If Bertram be away. 'Twere all one
That I should love a bright particular star
And think to wed it, he is so above me.
In his bright radiance and collateral light
Must I be comforted, not in his sphere.
Th' ambition in my love thus plagues itself:
The hind that would be mated by the lion
Must die for love. 'Twas pretty, though a plague,
To see him every hour; to sit and draw
His arched brows, his hawking eye, his curls,
In our heart's table-heart too capable
Of every line and trick of his sweet favour.
But now he's gone, and my idolatrous fancy
Must sanctify his relics. Who comes here?

The F-K results are identical to those of the soliloquy by the Countess. Yet common sense would tell us that current readers of Grade 4.5 would not comprehend this text, either. Words which might not be understand by such readers would be: remembrance, collateral, ambition, plagues, hawking, sweet (as used here by Shakespeare), idolatrous, sanctify. Again, the older syntax used would make it difficult for Grade 4.5 readers to understand.

F-K is giving us a Grade 4.5 results, which is essentially the same as results for English Bible versions such as the CEV (for which a F-K level of 4.5 would be accurate because the CEV is written in contemporary English vocabulary and syntax) and the NCV, because the average length of words is relatively short (this is not a technical scientific document which would have much longer words) as is the length of the sentences.

I think most current speakers of English who are not familiar with Shakespearean English would have difficulty understanding these lines from Shakespeare as easily as they would the CEV or NCV Bible versions which have essentially the same F-K ratings. For current speakers of English to understand these soliloquys as easily as they could read something truly written in contemporary English at Grade 4.5 level would require someone teaching the older syntax and vocabulary to them. But that, then, would defeat the purpose of F-K results, which are supposed to indicate what Grade 4.5 readers currently can read and comprehend.

I continue to suggest that a more accurate readability level for the ESV, with its frequently outdated syntax, or even non-English syntax, and obsolete and higher register vocabulary would be something like Grade 10. We need better readability instruments to give us accurate measurements of documents which are not written in contemporary English and/or which are written in higher registers than those assumed by the default levels of the Flesch-Kincaid tests. Scientifically accurate use of readability tests can only be obtained if we compare the proberbial apples with apples and oranges with oranges. F-K and similar tests were designed to give reading levels for current readers of contemporary English. Using such tests on the non-standard English of the ESV is mixing apples with oranges, or, more appropriately, trying to squeeze a square peg into a round hole. We would come up with more accurate results if we did scientific field testing of ESV tests with statistically reliable representative groups of subjects from each reading grade level.

Categories: ESV, Flesch-Kincaid