Sunday, May 16, 2010

Grammarly review: Most useful for advanced English learners, but has tough competition from free sites

Grammarly bills itself as "the best grammar checker and proofreader" for English-language documents. They've provided me with a free premium account (which I'll be giving away to a reader this upcoming week; if you want to be that reader, click here) to review.

To summarize, Grammarly is only going to be of use to more advanced English learners, due to the fact that (1) its engine aims at correcting native speakers' mistakes, not those of non-native speakers, and (2) all of its explanations are in English. So if your mistakes are somewhere in the ballpark of a native speaker's (and many advanced learners' mistakes are), and you can understand their explanations, then you're in good shape to make use of Grammarly. However, given the free corrections you can get on sites like Lang-8, it's probably not necessary for most language learners to invest in a premium account on Grammarly.

Before I get into the review, let me first explain the difference between their free and premium accounts, which has been a source of confusion in the past. At this time, all you can get for free is a report quantifying the kinds of corrections you need; you cannot see the specific text that needs to be corrected. While this might be useful for native speakers (who can go through the document and fairly easily find the issues flagged in the report), it's of much less use to non-native speakers (who aren't going to be able to find their mistakes as easily). For that reason, this review focuses only on the premium account, which shows specifically where changes are suggested and thus is of most use to language learners. The standard prices for Grammarly's premium subscriptions are $19.95 for a monthly subscription, $39.95 for a quarterly subscription ($13.32/month), or $94.95 for an annual subscription ($7.91/month).

To conduct the review, I looked at how Grammarly handled three documents:

A three-page selection from an article that I had published in a legal journal, which I and the journal's editors edited heavily before publication. The idea here was to explore the outer bounds of what Grammarly is capable of.

A page-and-a-half English-language summary of a dissertation that an Italian-speaking English learner posted on Lang-8 for corrections. The text could be fairly easily understood, even if there were numerous grammatical problems. Here I wanted to see how Grammarly would work for an advanced but clearly non-native speaker.

A short entry from another Lang-8 user whose English had many more problems. It was fairly difficult to figure out exactly what the author was trying to say in each sentence, and is what I would say is typical of a learner who still has a long way to go. Here I wanted to see how Grammarly would work for a beginner.

Starting with the excerpt of my own article, I wanted to see if Grammarly would find anything that the editors and I had missed. It didn't. Over the three-page selection, it flagged nine issues, all of which were false positives (i.e., they were flagged but didn't need to actually be changed). The only ones for which I considered making a change were a few sentences flagged as wordy, but after reading those sentences they seemed fine to me.

Even though the issues flagged were all false positives, Grammarly offers up concise but clear explanations of each of the points of grammar for which it thinks there might be an issue. These explanations are clear and informative, and provide good information to explain whether you should act on a flagged issue or not. However, whether or not you should act is often something that would require an advanced English level to decide. This is both because you need first to understand the suggestions and then to be able to determine whether to make a change. While any non-native speaker can probably eventually figure out Grammarly's suggestions with the help of a dictionary, determining whether to make a change based on a given suggestion is the much harder step for non-native speakers.

Turning to the Italian speaker's English text, Grammarly was a huge help in reducing the errors, but predictably missed a few key errors that are common among non-native speakers but uncommon among native speakers.

When I first ran the text through Grammarly, it flagged 41 errors. Of those 41, only three were false positives, two of which were false positives found in text quoted directly from Shakespeare. I fixed all the other mistakes it pointed out on the first run-through and then ran it through the system again. This time, it flagged four new errors. On the next run-through, there was one new error and after that none. All of the suggested changes were clear and relatively easy to make, and making them clearly improved the quality of the document greatly.

The situation was more problematic for the short beginner's text. Grammarly flagged 15 suggestions for change, most of which consisted of simply reminding that a space is needed after terminal periods and question marks, but none of which were false hits. However, even after making those changes, the text was still difficult to understand and needed a lot more work to make it anywhere near a native's speech.

In neither non-native text was Grammarly enough to get the text to full-fledged native language (something they don't claim to do, in any case), although it was much more useful with the advanced learner's text. For both, once I let Grammarly do all it could do, I went through the document and made all of the rest of the necessary edits. This revealed certain kinds of mistakes commonly made by non-native speakers that Grammarly does not seem to cover well:

The use of the articles "a" and "the". Although it did catch one mistake where "a" was used before a plural noun, it missed all the rest of the mistakes: lack of a necessary "a" or "the"; mixing up "a", "the", or the omission of both; etc. As the use of articles varies greatly from language to language, and many languages don't even have articles at all, these mistakes are very common among non-native speakers but uncommon among native speakers.

The placement of adverbs. Grammarly doesn't seem to check for awkwardly placed adverbs (like "I went to also the store" as opposed to "I also went to the store"), and this is another kind of common mistake made by non-native speakers.

Punctuation. Grammarly does cover a number of punctuation issues, but its seems to miss some of the errors made by non-native speakers. For instance, the author of the dissertation summary put a space after each opening quotation mark, and Grammarly did not catch that error.

These results of course are not particularly surprising, given Grammarly's primary focus thus far has been as a tool for native speakers.

Indeed, Max Lytvyn, one of the founders of Grammarly, anticipated these kinds of issues from a language learners' perspective when he wrote to me with the review account:

[L]et me give you some background on Grammarly usage we are seeing. The biggest user segment right now is students who are native English speakers. The second largest segment is professionals who may or may not be native speakers (this segment is likely to outgrow the first one soon). English language learners are not very active users of Grammarly. There are several factors contributing to lower popularity of Grammarly among language learners: 1) explanations are in English, and thus may be difficult to understand for some English learners; 2) Grammarly, being a computer algorithm, cannot be 100% accurate in its detection of mistakes; for native speakers such inaccuracies (false positives) are easy to see and do not take away much from the learning value of the product, but for language learners with lower levels of English proficiency it can be an issue. We do have examples of successful applications of Grammarly in a number of international educational institutions (e.g., ICU Japan, IQRA University in Pakistan, and dozens of private schools in Korea are licensing Grammarly for all of their students), but these institutions have rather high levels of English proficiency, and understand that Grammarly is an additional resource and not the ultimate authority. …

In general, individual users and educational institutions use our products to help writers 1) be more productive during the proofreading stage; 2) gain better understanding of different points of grammar and avoid repeatedly making the same mistakes; 3) identify areas of weakness of a particular writer to prepare for a meeting with a tutor/advisor or to plan a study session. Also, some tutors and educational institutions use Grammarly as a tool to help grade student papers and provide more expanded feedback in a time-efficient way. This is a legitimate usage scenario as long as users understand limitations of the product and use it as an additional resource, without expecting it to deliver the final grade for a paper.

Finally, let me quickly run through a catch-all list of my other comments that don't seem to fit in elsewhere:

When I copied and pasted the selection from a Word document into their checker, new paragraphs were ignored, so, during the check, I got an error saying that I needed a space where those paragraphs were ignored. (I excluded these errors from the error counts above.) This didn't happen when I pasted directly from Lang-8.

Their error explanation at times partially obscured the text it was referring to, as shown in the image below. You can move it around manually to see what's under it, but it'd be better if you didn't have to bother.

I found their synonym suggestions somewhat arbitrary. I wasn't sure why certain words were picked to be replaced. Were the words to be replaced repetitive? It's not clear. The suggested replacements also seemed arbitrary. For instance, it suggested replacing the "late" in the "late 1990s" with "past", which just wouldn't make any sense. These kinds of suggestions could be especially troubling for language learners, so I would probably recommend that all but the most advanced language learners using Grammarly ignore the synonym suggestions completely, especially as I've seen language learners get in trouble before through haphazard use of thesauruses.

In one of the articles I ran through the system, it missed the misspelled word "continuosly". It also didn't have a thing to say about a number of quotes in foreign languages, so I'm guessing that the spellcheck only looks for common mistakes rather than truly doing a thorough spellcheck. That said, it is, after all, a grammar-checking system as opposed to a spell-checking system.

So, if you're learning English, should you invest in a Grammarly account? It's tough to say yes to that because, if you put a little effort into it, it's pretty easy to get free, fast, high-quality corrections from native speakers on sites like Lang-8. And the native speakers on these sites will also be able to make corrections beyond the limitations of Grammarly, such as fixing awkward phrasing that is technically grammatically correct. (Indeed, I'd imagine native speakers of English could get other native speakers to make corrections on sites like those as well, but I've never seen that happen before.)

The English learners who could make the most use of Grammarly are those who produce a lot of text in English, such as those whose jobs require them to liaise frequently with English speakers. In such cases, Grammarly could be a better choice than the free sites because of Grammarly's almost immediate turn-around time and because it can handle larger texts and larger volumes of texts more easily, even though Grammarly's corrections aren't necessarily going to result in native speech.

If Grammarly came out with a product for Japanese, I would be very interested in it. I write a good deal in Japanese, and getting my own errors systematically flagged down for me would be very helpful in improving my writing, just as Grammarly is used by native-English speakers. So while I think Grammarly is of limited use to language learners because of sites like Lang-8, I still hope that we might see them apply their technology to other languages so that language learners other than English speakers might be able to take advantage of what Grammarly can offer.

Update: On May 18, Max wrote to me with the following comments about the review:

Thanks for a good review. I think it is an accurate assessment of the product. Grammarly is designed for a specific population and specific purposes, and those who are starting to learn English are not the best target. Grammarly will benefit those who use language for work or education at a higher level of proficiency, regardless of them being native speakers or not. Still, our Korean clients are very successful in using Grammarly to help their students prepare for English essay tests, so there are scenarios where Grammarly may help English learners as well.

Few comments:

The first text you checked had no mistakes, which explains that 100% of suggestions were false positives. It is a statistical phenomenon that many people do not immediately understand. For example, if you test 10,000 completely healthy people for a rare disease, and the accuracy of the test is 99%, the test will flag 100 people as sick, and will be wrong in 100% of the cases, while still being 99% accurate. It is a simplification, but in general should explain why testing texts with no mistakes produces non-representative results. We have to explain this phenomenon rather often, as people frequently test Grammarly by copying professionally edited texts from the Web.

We are working on expanding the support for mistakes typical to non-native speakers. Support for articles, for example, is one of the most active areas of improvement (there is a limit there – in many cases it is impossible to determine if definite or indefinite article is required without actually understanding the meaning of the text). We also will look into adverb placement mistakes.

Versions of Grammarly with cards in other languages are possible. We are considering a version with cards in Korean.

Versions of Grammarly for languages other than English are not yet planned, but are a possibility.

15 comments:

In terms of foreign language grammar checkers, I found bonpatron.com an excellent web-based one for French. It specifically targets non-natives and I found it very handy for improving my writing. It doesn't correct errors, only points out where the errors are and provide an explanation why.

For more advanced users, I _highly_ recommend Antidote from Druide.com . When I first bought it, I was blown away by how effective it was, catching things that people on lang-8 missed (even after multiple native speakers had checked the text.) The integrated dictionary and corpus search were very useful too. I won't list all of its features to avoid sounding like ad copy, but anybody doing French writing, native or non, should at least check it out.

Thanks a lot for this review! I was quite intrigued when I read about the G. in your previous posts, mainly because I started writing PhD thesis and was wondering if investing in G. membership over the next few months would be a good idea. However, it seems that it wouldn't be very helpful for me, since the main purely grammar errors in my writings are the redundant or missing articles... Anyway, if anyone knows any decent grammar checkers for Spanish or Russian, please share :)Milena

This is a terrible, unreliable program. It has no feel for tone or proper sentence structure. If you are not a strong writer, then you do not know the difference between good and bad. This is a scam. Go with a human editing service.

Grammarly is alright but if you are trying to edit a large text it is difficult to use. Not very useful for the average author. Also, the program was advertised as having a 14 day free trial however my credit card was charged immediately and for the last 48 hours has been down for maintence. I'm not very impressed.

I have been having a problem to download the Grammarly software for the last two weeks. After having my article edited I was required to fill in my credit card number or PayPal account.I do not have a credit card but only a debit card and a PayPal account.Having entered my debit card account I got a message that I was not authorized to use my debit card.My only choice was providing my PayPal account.Having entered my account a message popped requesting my credit card account.It was indeed like a merry go round.I sent an e mail to Mr.Greg Carpets without any response from him until today.With this kind of attitude how is the Grammarly team going to compete selling software worlwide . I felt as though I was dealing with robots.I hope the team wakes up soon.

I subscribed to Grammarly on June 15th for 3 months at 39.95. After couple of log-in in that week, I was not allowed to log-in and it kept showing me as a non-subscriber. I sent several emails but no response. Then I notice this month's their renewal and a charge of 39.95 again. I tried to see if I have a subscriber status but was denied again. I think Grammarly is either the worst service or a scam. In any event, I want my full refund for both term subscriptions.

Based on how Grammarly markets its product and how its forums are flooded with positive reviews that use stock profile photos, I'd say this thing is a scam. It reminds me of when I was writing fake reviews for products like perpetual motion machines and other impossibilities.

Also, is the maker of Grammarly claiming that the false positives were a normal statistical anomaly? That getting the positives wrong were simply a matter of chance, and this reviewer simply happened to be the 1%? Wow, those are some odds. Vincent, you need to play the lottery with improbable luck like that.

I am getting the similar kind of problem.Till now i have never used like this application so might be getting this problem.I usually refer http://www.youtube.com/user/twominenglish to learn and practice English.

Grammarly thieves stole $239.80 from me. Unfortunately, I was sidetracked with semester deadlines, a sick father that passed shortly after the semester, and a mother who went to pieces. I had contacted them when I was having difficulties with their download right after a 3/31/2014 purchase and never received a response. It was a time crunch, and impulsive purchase. I was having a difficult time keeping focus and had to move on and get help with my paper. Having contacted my university writing center for free, was beautiful and I'm kicking myself for wanting to work from home instead. Regardless if I feel like an idiot, shame on them!