{"id":81963,"date":"2021-08-16T14:38:14","date_gmt":"2021-08-16T19:38:14","guid":{"rendered":"https:\/\/www.johndcook.com\/blog\/?p=81963"},"modified":"2023-02-06T08:27:49","modified_gmt":"2023-02-06T14:27:49","slug":"initial-letter-frequency","status":"publish","type":"post","link":"https:\/\/www.johndcook.com\/blog\/2021\/08\/16\/initial-letter-frequency\/","title":{"rendered":"Initial letter frequency"},"content":{"rendered":"

I needed to know the frequencies of letters at the beginning of words for a project. The overall frequency of letters, wherever they appear in a word, is well known. Initial frequencies are not so common, so I did a little experiment.<\/p>\n

I downloaded the Canterbury Corpus<\/a> and looked at the frequency of initial letters in a couple of the files in the corpus. I first tried a different approach, then realized a shell one-liner [1] would be simpler and less-error prone.<\/p>\n

cat alice29.txt | lc | grep -o '\\b[a-z]' | sort | uniq -c | sort -rn<\/pre>\n
This shows that the letters in descending order of frequency at the beginning of a word are t<\/em>, a<\/em>, s<\/em>, \u2026, j<\/em>, x<\/em>, z<\/em>.<\/p>\n
The file alice29.txt<\/code> is the text of Alice’s Adventures in Wonderland<\/em>. Then for comparison I ran the same script on another file, lcet10.txt<\/code>. a lengthy report from a workshop on electronic texts.<\/p>\n
This technical report’s initial letter frequencies order the alphabet t<\/em>, a<\/em>, o<\/em>, \u2026, y<\/em>, z<\/em>, x<\/em>. So starting with the third letter, the two files have different initial letter frequencies.<\/p>\n
I made the following plot to visualize how the frequencies differ. The horizontal axis is sorted by overall letter frequency (based on the Google corpus summarized here<\/a>).<\/p>\n
<\/p>\n
I expected the initial letter frequencies to differ from overall letter frequencies, but I did not expect the two corpora to differ.<\/p>\n
Apparently initial letter frequencies vary more across corpora than overall letter frequencies. The following plot shows the overall letter frequencies for both corpora, with the horizontal axis again sorted by the frequency in the Google corpus.<\/p>\n
<\/p>\n
Here the two corpora essentially agree with each other and with the Google corpus. The tech report ranks letters in essentially the same order as the Google corpus because the orange dashed line is mostly decreasing, though there is a curious kink in the graph at c<\/em>.<\/p>\n
Related posts<\/h2>\n