------------------------------------------------------
------------------------------------------------------
....THE REVIEWS....
------------------------------------------------------
------------------------------------------------------
Reviewer A:
CLARITY: For the reasonably well-prepared reader, is it clear what was done and why? Is the paper well-written and well-structured?:
5. Very clear.
ORIGINALITY/INNOVATIVENESS: How original is the approach? Does this paper break new ground in topic, methodology, or content? How exciting and innovative is the research it describes?
Note that a paper could score high for originality even if the results do not show a convincing benefit.
:
3. Respectable: A nice research contribution that represents a notable extension of prior approaches or methodologies.
SOUNDNESS/CORRECTNESS: First, is the technical approach sound and well-chosen? Second, can one trust the claims of the paper -- are they supported by properexperiments and are the results of the experiments correctlyinterpreted?:
5. The approach is very apt, and the claims are convincingly supported.
MEANINGFUL COMPARISON: Does the author make clear where the presented system sits with respect to existing literature? Are the references adequate?:
5. Precise and complete comparison with related work. Benefits and limitations are fully described and supported.
SUBSTANCE: Does this paper have enough substance (in terms of the amount of work), or would it benefit from more ideas or analysis?:
4. Represents an appropriate amount of work for a publication in this journal. (most submissions)
IMPACT OF IDEAS OR RESULTS: How significant is the work described? If the ideas are novel, will they also be useful or inspirational? If the results are sound, are they also important? Does the paper bring new insights into the nature of the problem?:
3. Interesting but not too influential. The work will be cited, but mainly for comparison or as a source of minor contributions.
REPLICABILITY: Will members of the ACL community be able to reproduce or verify the results in this paper?:
4. could mostly reproduce the results, but there may be some
variation because of sample variance or minor variations in their interpretation of the protocol or method.
IMPACT OF PROMISED SOFTWARE: If the authors state (in anonymous fashion) that their software will be available, what is the expected impact of the software package?:
1. No usable software released.
IMPACT OF PROMISED DATASET(S): If the authors state (in anonymous fashion) that datasets will be released, how valuable will they be to others?:
1. No usable datasets submitted.
TACL-WORTHY AS IS? In answering, think over all your scores above. If a paper has some weaknesses, but you really got a lot out of it, feel free to recommend it. If a paper is solid but you could live without it, let us know that you're ambivalent.
Note: after you submit this review form, you'll need to answer a related but different question via a pull-down menu: how long would it take for the authors to revise the submission to be TACL-worthy?
:
5. Strong: I'd like to see it accepted; it will be one of the better papers in TACL.
Detailed Comments for the Authors:
This paper provides a method to compute the complexity of inflectional morphology of a language. The presented method is a language-independent method and can be used whenever a the paradigms of inflected words forms are provided for the given language.
This is a very well-written paper, and everything has been clearly explained. Overall, I will be happy to have this paper accepted to TACL without any changes.
The comparison (theoretical) to related work is outstanding. I specially like how the authors go extra mile to describe the drawbacks of Ackerman and Malouf (2013) and how they overcame those in their own model. (Actually if they hadn't done this, it would not be that much worth a submission, because these details are not immediately clear to a reader who is not aware of the prior work in this area. In fact, as soon as a I read equation 2, I wanted to know how the presented model is different from Ackerman's model -- maybe the authors should add a note there saying that they will explain it in section 5)
I think it would be nice to see how the complexity incduced in the model presented by the authors compare to that of Ackerman's model. even if th enumber are not directly comparable, it would be nice to see how the order of languages (according to morphological complexity) differ among the two models. Maybe just borrow some values presented from the original Ackerman paper?
I like how the authors were able to compute the 'p' value for the pareto graph they obtained to be having a smaller area than a randomly selected graph. I would actually even suggest to draw one of these random graphs in Figure 2 for better understanding of what is being compared against.
I do not see any Indian language in Table 2, is there a particular reason for that?
Typo-logical errors (pun intended):
Equation 1: m_gerun should be in the third place, and m_past should be in the last place to be consistent with examples in the following paragraph.
Section 4.1, first paragraph: Remove "For example, to" from the end of the paragraph.
REVIEWER CONFIDENCE:
5. Positive that my evaluation is correct. I read the paper very carefully and am familiar with related work.
------------------------------------------------------
------------------------------------------------------
Reviewer B:
CLARITY: For the reasonably well-prepared reader, is it clear what was done and why? Is the paper well-written and well-structured?:
5. Very clear.
ORIGINALITY/INNOVATIVENESS: How original is the approach? Does this paper break new ground in topic, methodology, or content? How exciting and innovative is the research it describes?
Note that a paper could score high for originality even if the results do not show a convincing benefit.
:
5. Seminal: Significant new problem, technique, methodology, or insight -- no prior research has attempted something similar
SOUNDNESS/CORRECTNESS: First, is the technical approach sound and well-chosen? Second, can one trust the claims of the paper -- are they supported by properexperiments and are the results of the experiments correctlyinterpreted?:
5. The approach is very apt, and the claims are convincingly supported.
MEANINGFUL COMPARISON: Does the author make clear where the presented system sits with respect to existing literature? Are the references adequate?:
5. Precise and complete comparison with related work. Benefits and limitations are fully described and supported.
SUBSTANCE: Does this paper have enough substance (in terms of the amount of work), or would it benefit from more ideas or analysis?:
5. Contains more ideas or analysis than most publications in this journal; goes the extra mile.
IMPACT OF IDEAS OR RESULTS: How significant is the work described? If the ideas are novel, will they also be useful or inspirational? If the results are sound, are they also important? Does the paper bring new insights into the nature of the problem?:
5. Will affect the field by altering other people's choice of research topics or basic approach.
REPLICABILITY: Will members of the ACL community be able to reproduce or verify the results in this paper?:
4. could mostly reproduce the results, but there may be some
variation because of sample variance or minor variations in their interpretation of the protocol or method.
IMPACT OF PROMISED SOFTWARE: If the authors state (in anonymous fashion) that their software will be available, what is the expected impact of the software package?:
1. No usable software released.
IMPACT OF PROMISED DATASET(S): If the authors state (in anonymous fashion) that datasets will be released, how valuable will they be to others?:
1. No usable datasets submitted.
TACL-WORTHY AS IS? In answering, think over all your scores above. If a paper has some weaknesses, but you really got a lot out of it, feel free to recommend it. If a paper is solid but you could live without it, let us know that you're ambivalent.
Note: after you submit this review form, you'll need to answer a related but different question via a pull-down menu: how long would it take for the authors to revise the submission to be TACL-worthy?
:
6. Exciting: I'd fight to get it accepted; probably would be one of the best papers in TACL this year.
Detailed Comments for the Authors:
There's not too much to say about this paper. It's a tour de force, and I hope it has the impact that it deserves. I have a feeling that the clever variational approach taken may help to start to get a handle on many other typological problems. The one complaint I have is that simple permutation tests, while a reasonable thing to do, somehow don't seem like the right baseline. It seems to me that a better test would be to re-run on artificial paradigms of some sort and look at the results. I have, at a minimum, a nagging intuition that somehow it might be mathematically hard for any set of objects to get into the upper right corner; or maybe extremely easy. At any rate, even for "weird" pseudo-paradigms, perhaps some surprising properties of their i-complexity would emerge. However, this intuition is extremely vague, and I don't have a clear sense for what the right baseline would be or precisely what nuisance factors introduced by the simple fact of *having* a paradigm of a given size would be. It's just that permuting y is surely not going to give the same result. I don't expect a correction or a comment on this, but it needs to be thought about.
Two small things: unfinished sentence at the end of the first paragraph of 4.1. And, "isolating languages like Chinese and Japanese" - Japanese? Surely Japanese is not an isolating language by any reasonable definition?
REVIEWER CONFIDENCE:
2. Willing to defend my evaluation, but it is fairly likely that I missed some details, didn't understand some central points, or can't be sure about the novelty of the work.
------------------------------------------------------
________________________________________________________________________
Transactions of the Association for Computational Linguistics
https://www.transacl.org/ojs/index.php/tacl/