Discussion Tag Cloud

Welcome to Math2.0This website is an archive of the publishing.mathforge.org discussion forum. The forum is no longer active but much discussion took place on these pages so an archive has been preserved.

Author: John BaezFormat: MarkdownItexI'm glad to see this forum!
Here's my recent thinking:
People seem to get frozen in indecision when contemplating big serious changes to the current journal system. The consequences are too huge. So, I'm thinking we could hasten the evolution of good ideas by doing something *quick, easy and fun*. Something that's not too painful to set up. And something that we use because we enjoy it, not because it's good for us.
I'm imagining something like this:
A system linked to the arXiv, which looks a lot like the arXiv: one page per paper. If you click to download the paper you get sent to the arXiv. But you can also comment on the paper, and/or give a simple 'rating', on a scale from 1 to 5. When you first join the system, you could say whether you want your ratings to be visible to everyone, or only contribute to an average rating.
Next to every paper you could see how many people have clicked to download it, its average rating, and a link to the discussion thread.
In a further refinement, this could all be combined with a reputation system, so that people could gradually develop fancier and fancier algorithms to decide which papers are 'good'.
*BUT*: the short-term goal is not to provide a system for ranking job candidates or carrying out official peer review - merely to provide a fun place to talk about papers and help decide what we might like to read.
I think this "lowered ambition" will make it easier for people, especially youngsters, to explore new ways of doing things.
Unfortunately I don't have the skill or desire to do the programming. But I'm imagining that perhaps it's not too hard to start a system vaguely like the one I describe.
Ideally what would happen is that people would enjoy the system but dislike certain aspects of it - and then people would improve it, or create competitors. After a certain point, people would start taking one of these systems seriously enough to base hiring or tenure decisions on it, in the same way that they use journals now. But not right away.

I’m glad to see this forum!

Here’s my recent thinking:

People seem to get frozen in indecision when contemplating big serious changes to the current journal system. The consequences are too huge. So, I’m thinking we could hasten the evolution of good ideas by doing something quick, easy and fun. Something that’s not too painful to set up. And something that we use because we enjoy it, not because it’s good for us.

I’m imagining something like this:

A system linked to the arXiv, which looks a lot like the arXiv: one page per paper. If you click to download the paper you get sent to the arXiv. But you can also comment on the paper, and/or give a simple ’rating’, on a scale from 1 to 5. When you first join the system, you could say whether you want your ratings to be visible to everyone, or only contribute to an average rating.

Next to every paper you could see how many people have clicked to download it, its average rating, and a link to the discussion thread.

In a further refinement, this could all be combined with a reputation system, so that people could gradually develop fancier and fancier algorithms to decide which papers are ’good’.

BUT: the short-term goal is not to provide a system for ranking job candidates or carrying out official peer review - merely to provide a fun place to talk about papers and help decide what we might like to read.

I think this “lowered ambition” will make it easier for people, especially youngsters, to explore new ways of doing things.

Unfortunately I don’t have the skill or desire to do the programming. But I’m imagining that perhaps it’s not too hard to start a system vaguely like the one I describe.

Ideally what would happen is that people would enjoy the system but dislike certain aspects of it - and then people would improve it, or create competitors. After a certain point, people would start taking one of these systems seriously enough to base hiring or tenure decisions on it, in the same way that they use journals now. But not right away.

Author: CschaffnerFormat: Textwe are currently trying to revive it: https://github.com/cschaffner/scirate
Any help is appreciated (please email me know if you're interested in helping). The basic scirate.com allowed users to "vote" for arxiv-papers which were then displayed in descending order of votes. One could also make comments, but I don't think that functionality was heavily used. I loved and appreciated Dave Bacon's idea of crowd-sourcing the daily routine of looking through the long list of new arxiv-papers, it saved me quite some time every day.
I completely agree with John that what he describes is a good system to "have out there", and it does not seem too hard to actually implement. I hope we'll be there soon.

we are currently trying to revive it: https://github.com/cschaffner/scirateAny help is appreciated (please email me know if you're interested in helping). The basic scirate.com allowed users to "vote" for arxiv-papers which were then displayed in descending order of votes. One could also make comments, but I don't think that functionality was heavily used. I loved and appreciated Dave Bacon's idea of crowd-sourcing the daily routine of looking through the long list of new arxiv-papers, it saved me quite some time every day.

I completely agree with John that what he describes is a good system to "have out there", and it does not seem too hard to actually implement. I hope we'll be there soon.

Author: Charles RezkFormat: TextI don't really care for or see the need of a numerical rating system. But having a default place to host a discussion for each and every individual paper could be incredibly useful. (And if having a rating system makes it "fun", and so encourages people to use it, I can live with it.)

I don't really care for or see the need of a numerical rating system. But having a default place to host a discussion for each and every individual paper could be incredibly useful. (And if having a rating system makes it "fun", and so encourages people to use it, I can live with it.)

Author: Charles RezkFormat: MarkdownItexIncidentally, the quickest and easiest thing to do is to encourage every mathematician to *post all their new papers to the arxiv*. Even though there are about 100 new math papers a day being put on arXiv, that's still probably less than half of all published papers.

Incidentally, the quickest and easiest thing to do is to encourage every mathematician to post all their new papers to the arxiv. Even though there are about 100 new math papers a day being put on arXiv, that’s still probably less than half of all published papers.

Author: Yiftach BarneaFormat: TextIf you build it, they will come. By they I mean people who need numbers to rank people. These numbers will be used in hiring, promoting, etc. But the system you suggest is so open to abuse that I think it would be a really bad idea to implement it.

If you build it, they will come. By they I mean people who need numbers to rank people. These numbers will be used in hiring, promoting, etc. But the system you suggest is so open to abuse that I think it would be a really bad idea to implement it.

Author: Peter KrautzbergerFormat: MarkdownItex* For commenting and "voting" there is also [papercritic.com](http://www.papercritic.com). This is a young and active project coming out of programming competition from Mendely and PLoS. They might be inclined to add a paper-view. Also, Mendeley has data on how many papers have been read, annotated, shared etc. This data is available through an API.
* Generally speaking, encourage people to make their reviews public! After all, we all do it when we read papers.
* Tell people to sign up at wordpress.com, with a pseudonym if they like, and start *writing*.
* Tell them to sign up at [researchblogging.org](http://www.researchblogging.org) and cross-post reviews of arXiv papers.
I would like to politely disagree with the [If you build it, they will come](http://boolesrings.org/krautzberger/2012/02/06/if-you-build-it-will-they-come/)-attitude. Without a culture of making multiple reviews public, I don't think a platform will work (e.g. even Nature failed at that).

For commenting and “voting” there is also papercritic.com. This is a young and active project coming out of programming competition from Mendely and PLoS. They might be inclined to add a paper-view. Also, Mendeley has data on how many papers have been read, annotated, shared etc. This data is available through an API.

Generally speaking, encourage people to make their reviews public! After all, we all do it when we read papers.

Tell people to sign up at wordpress.com, with a pseudonym if they like, and start writing.

I would like to politely disagree with the If you build it, they will come-attitude. Without a culture of making multiple reviews public, I don’t think a platform will work (e.g. even Nature failed at that).

Author: gowersFormat: TextI'm not terribly keen on numerical ratings -- I worry that there would be a certain amount of non-fun mixed in with the fun. However, comments, reviews etc. (which might implicitly rate papers, but in a polite way that wouldn't make an author think, "Hey, my paper's at least a 4, so why did that jerk give it a 3?") seem OK. I also worry that people who are somehow "popular" would get their papers more highly rated -- a phenomenon that applies to questions/answers on Mathoverflow.
So my view is that to keep it quick, easy and fun, numerical ratings should not be part of it.

I'm not terribly keen on numerical ratings -- I worry that there would be a certain amount of non-fun mixed in with the fun. However, comments, reviews etc. (which might implicitly rate papers, but in a polite way that wouldn't make an author think, "Hey, my paper's at least a 4, so why did that jerk give it a 3?") seem OK. I also worry that people who are somehow "popular" would get their papers more highly rated -- a phenomenon that applies to questions/answers on Mathoverflow.

So my view is that to keep it quick, easy and fun, numerical ratings should not be part of it.

Author: Chris SchommerPriesFormat: TextThere are lots of problems with most rating systems. For example on Mathoverflow questions which are in areas where there are more mathematicians using MO get higher ratings, just because there is a larger base of viewers and not for any intrinsic reason. Giving a 1-5 rating is better, but there are all sorts of consistency problems that could emerge.
So maybe John's easy and fun idea is not easy enough? Maybe we should just have a site which can host a discussion for individual papers posted on the arxiv? This should be even easier to set up, right? I want to second Rezk's statement that this alone could be incredibly useful, and as gower's suggests this might implicitly rate papers afterall (but in a more subtle fashion).

There are lots of problems with most rating systems. For example on Mathoverflow questions which are in areas where there are more mathematicians using MO get higher ratings, just because there is a larger base of viewers and not for any intrinsic reason. Giving a 1-5 rating is better, but there are all sorts of consistency problems that could emerge.

So maybe John's easy and fun idea is not easy enough? Maybe we should just have a site which can host a discussion for individual papers posted on the arxiv? This should be even easier to set up, right? I want to second Rezk's statement that this alone could be incredibly useful, and as gower's suggests this might implicitly rate papers afterall (but in a more subtle fashion).

Author: xbadFormat: TextI have a concern about pubic rating systems. In the current refereeing system, there is no communication among reviewers, which has the consequence that the referee reports are independent assessments. This feature is worth trying to preserve. My fear is that, in a public system, the the early reviews will have undue influence on later reviews. A related concern is that early critiques will lead to rapid-fire revisions of manuscripts. It can take a real investment of time to do a serious review. This work will become all the more difficult once a manuscript becomes a moving target.

I have a concern about pubic rating systems. In the current refereeing system, there is no communication among reviewers, which has the consequence that the referee reports are independent assessments. This feature is worth trying to preserve. My fear is that, in a public system, the the early reviews will have undue influence on later reviews. A related concern is that early critiques will lead to rapid-fire revisions of manuscripts. It can take a real investment of time to do a serious review. This work will become all the more difficult once a manuscript becomes a moving target.

Author: Yiftach BarneaFormat: TextThe anonymity of the referee has advantages and disadvantages. But the main strength of the peer-review system is its independence. The author has very little influence on the choice of referee. Thus, we can trust the process to some extent. In the system that is suggested the authors could influence the ranking. It is obvious that once there is such a system some people will use it to rank other people (that is exactly what we do by looking where people published). If you believe that no abuse will take place you are incredibly naive.

The anonymity of the referee has advantages and disadvantages. But the main strength of the peer-review system is its independence. The author has very little influence on the choice of referee. Thus, we can trust the process to some extent. In the system that is suggested the authors could influence the ranking. It is obvious that once there is such a system some people will use it to rank other people (that is exactly what we do by looking where people published). If you believe that no abuse will take place you are incredibly naive.

Author: John BaezFormat: MarkdownItexIf most people don't like the idea of numerical ratings, we should skip that, or modify it. All I care about is that lots of people start using an arXiv overlay website to discuss papers.
The first version will inevitably be imperfect; people will complain about it, but if a lot of people use, it they'll keep trying to improve it, or set up better systems.
I mentioned the idea of ratings merely because most forums that become extremely popular have some feature like this:
* Facebook lets you 'like' something
* Google+ lets you '+1' something
* Amazon lets you rate items from 1 to 5
* Netflix lets you give movies from 1 to 5 stars
* MathOverflow lets you click 'up' or 'down' on questions and answers to those.
and so on. In some of these systems you can see who liked or disliked a given item; in others that information is hidden. Perhaps, for obvious reasons, it's hidden in all systems where people can say they disliked something (instead of merely not saying they liked it).
People love to complain about these systems, and of course they're open to abuse, but they're still useful, and they seem to really engage people. That's why I suggested introducing one.
Of course, none of these systems are attempting to be a substitute for 'peer review',... and that's the exactly the point! _I'm not proposing a system that's supposed to substitute for peer review. It's only supposed to help you find papers you might like, and see what other people have to say about those papers._
This lowered goal will make it easier for people to start such a website without becoming paralyzed with worry. However, if people feel inclined to become paralyzed with worry over a particular issue, even despite this lowered goal, we should sidestep that issue.
It's been pointed out that _[an arXiv overlay website for discussing papers already existed for papers on quant-ph](http://dabacon.org/pontiff/?p=3782)_. It was developed by David Bacon. People in the quantum information community enjoyed it a lot: they would go there instead of the arXiv to see the new papers and see which ones people liked. (I believe this system merely allowed you to 'like' a paper, a bit like Google+ or Facebook.) Unfortunately David Bacon got a job at Google and became too busy to maintain the system, so it no longer exists. However, as noted here, people are now trying to revive it - thanks to the Elsevier boycott! And I know there are people with ideas on how to expand it to the whole arXiv.

If most people don’t like the idea of numerical ratings, we should skip that, or modify it. All I care about is that lots of people start using an arXiv overlay website to discuss papers.

The first version will inevitably be imperfect; people will complain about it, but if a lot of people use, it they’ll keep trying to improve it, or set up better systems.

I mentioned the idea of ratings merely because most forums that become extremely popular have some feature like this:

Facebook lets you ’like’ something

Google+ lets you ’+1’ something

Amazon lets you rate items from 1 to 5

Netflix lets you give movies from 1 to 5 stars

MathOverflow lets you click ’up’ or ’down’ on questions and answers to those.

and so on. In some of these systems you can see who liked or disliked a given item; in others that information is hidden. Perhaps, for obvious reasons, it’s hidden in all systems where people can say they disliked something (instead of merely not saying they liked it).

People love to complain about these systems, and of course they’re open to abuse, but they’re still useful, and they seem to really engage people. That’s why I suggested introducing one.

Of course, none of these systems are attempting to be a substitute for ’peer review’,… and that’s the exactly the point! I’m not proposing a system that’s supposed to substitute for peer review. It’s only supposed to help you find papers you might like, and see what other people have to say about those papers.

This lowered goal will make it easier for people to start such a website without becoming paralyzed with worry. However, if people feel inclined to become paralyzed with worry over a particular issue, even despite this lowered goal, we should sidestep that issue.

It’s been pointed out that an arXiv overlay website for discussing papers already existed for papers on quant-ph. It was developed by David Bacon. People in the quantum information community enjoyed it a lot: they would go there instead of the arXiv to see the new papers and see which ones people liked. (I believe this system merely allowed you to ’like’ a paper, a bit like Google+ or Facebook.) Unfortunately David Bacon got a job at Google and became too busy to maintain the system, so it no longer exists. However, as noted here, people are now trying to revive it - thanks to the Elsevier boycott! And I know there are people with ideas on how to expand it to the whole arXiv.

Author: Scott MorrisonFormat: MarkdownItexI fear that "build it and they will come" is wildly naive! Instead, projects like this must be [useful at the point of use](http://www.math.ntnu.no/~stacey/Mathforge/Math2.0/comments.php?DiscussionID=6).
Happily, I think there is definitely a way to achieve this in the current context. Most mathematicians receive a daily email from the arXiv listing new papers in chosen subject areas. If you're at all like me, you want to subscribe to more than one area, but many (most? nearly all?) of the individual papers aren't of much interest.
My life would be noticeably better if some pre-filtering or sorting could be done! I'd love to have that email try to cluster papers by tighter topic areas than just the arxiv subject areas, and then only show me my preferred clusters. I'd love to have my attention automatically drawn to papers my friends liked! Indeed, I'd be happy just to have a list of every paper Noah glances at, just to be reminded of what a terrible reader-of-papers I am. :-)
It sounds like David Bacon's site achieved some of these things. Can we do more?

Happily, I think there is definitely a way to achieve this in the current context. Most mathematicians receive a daily email from the arXiv listing new papers in chosen subject areas. If you’re at all like me, you want to subscribe to more than one area, but many (most? nearly all?) of the individual papers aren’t of much interest.

My life would be noticeably better if some pre-filtering or sorting could be done! I’d love to have that email try to cluster papers by tighter topic areas than just the arxiv subject areas, and then only show me my preferred clusters. I’d love to have my attention automatically drawn to papers my friends liked! Indeed, I’d be happy just to have a list of every paper Noah glances at, just to be reminded of what a terrible reader-of-papers I am. :-)

It sounds like David Bacon’s site achieved some of these things. Can we do more?

Author: Yiftach BarneaFormat: TextI have been trying to think how to address the worries I have. Let me suggest some ideas. I would suggest that the authors of a paper would be able to decide that they do not want to receive any comments on their paper. More importantly any comment will first be available to the authors and only if they approve it will be available to everyone. This way the system cannot be abused to heart people, also its reliability as a tool to rank people will be so low that no one will use it this way. On the other hand, most people don’t want to know which papers not to read but which ones to read, so negative comments are less helpful to them. However, the authors will still be able to see the negative comments and react to them. If we would like to make things a bit clearer, we could make it visible how many comments were not approved by the authors.

I have been trying to think how to address the worries I have. Let me suggest some ideas. I would suggest that the authors of a paper would be able to decide that they do not want to receive any comments on their paper. More importantly any comment will first be available to the authors and only if they approve it will be available to everyone. This way the system cannot be abused to heart people, also its reliability as a tool to rank people will be so low that no one will use it this way. On the other hand, most people don’t want to know which papers not to read but which ones to read, so negative comments are less helpful to them. However, the authors will still be able to see the negative comments and react to them. If we would like to make things a bit clearer, we could make it visible how many comments were not approved by the authors.

Author: gowersFormat: TextOne additional thought is that while I'm still not keen on giving a rating to papers, there might be a place for something like Amazon's option to judge whether a comment/review is helpful/unhelpful. So if somebody gives a particularly insightful assessment of a paper and many people find it helpful, that assessment would be trusted more (in general, though one might have to be careful how to interpret the data).

One additional thought is that while I'm still not keen on giving a rating to papers, there might be a place for something like Amazon's option to judge whether a comment/review is helpful/unhelpful. So if somebody gives a particularly insightful assessment of a paper and many people find it helpful, that assessment would be trusted more (in general, though one might have to be careful how to interpret the data).

Author: darij grinbergFormat: TextI think that, rather than a 1-5 overall rating for the paper, there should be separate ratings about importance, readability, correctness and self-containment. I wouldn't be comfortable with downrating a reasonable paper just because it uses some nonstandard notations and doesn't care to introduce them; but I would give it a 3/5 in self-containment. But these all are technical details of a feature of little importance. The really important part, I believe, is the commenting, not the rating, **even if** evaluation is the purpose. Rating is just too easily manipulated, and even if it is not, people will think it is. You can create 20 user accounts which rate each of your papers a 5,5,5,5, but you can't have 20 sockpuppets give you positive reviews that don't look fake at the first sight.

I think that, rather than a 1-5 overall rating for the paper, there should be separate ratings about importance, readability, correctness and self-containment. I wouldn't be comfortable with downrating a reasonable paper just because it uses some nonstandard notations and doesn't care to introduce them; but I would give it a 3/5 in self-containment. But these all are technical details of a feature of little importance. The really important part, I believe, is the commenting, not the rating, **even if** evaluation is the purpose. Rating is just too easily manipulated, and even if it is not, people will think it is. You can create 20 user accounts which rate each of your papers a 5,5,5,5, but you can't have 20 sockpuppets give you positive reviews that don't look fake at the first sight.

Author: zskodaFormat: MarkdownItexThe fact that __popular__ sites Facebook, google+, amazon, netflix, MO have like buttons does not mean that their popularity mechanism makes them better in inducing the true development of personality etc. Popularity is not a measure of development and __achievement__ itself. A Serbian kid growing up in California came to his parents place in Belgrade and was surprised that he was accepted to any children group to play with on the street, not preceded by marked belonging to a clique, clan or band, unlike what he learned in modern California, driven by TC shows, popularity culture, hi school prom craze. In fact I NEVER heard in my childhood in Croatia that the phrase "be popular" applies to kids. Word "popular" was about movie stars and so on, not about something relevant to our ordinary lives. This was few decades ago. Now it is probably changing, everybody needs to have a good picture to put on social site, and people kill when get defriended. Finally, the democracy in US is ridiculuous -- the PR makes people choose whoever, all is lobbyied and advertised; voting is not discovery of the truth.
I fully agree with more sensitive 11: especially this -- for peer reviewing it is beneficial to be independent, that one referee does not know what the other one read. Also ASSIGNING a referee makes people discover things which are not easy to appreciate without through reading. It may be a boring looking paper, but the arguments while on the surface written by the same kind of letters and equation symbols are in fact a deep breakthrough. A casual audience will miss this, while true reader will be forced to understand. Once understood it is more likely to give a more balanced mark of the achievement.
On the other hand, I do second that one should have pages of discussion for individual articles. Not collections of opinions, but more a wiki style. Look at wikipedia. If somebody writes something without arguments and references, he is likely to be overwritten. It is also possible to write remarks about the paper. If you observe some interesting relation to other areas, or that the result follows from something else it stays in the wiki. If you repeat the bullshitting which somebody else wrote before, your sentence will get into the history and erased from the new assesment, or labelled as irrelevant.
The danger is also that the author himself will make his work more appreciated by answering to the craze. Instead, as Arnold used to say, one should work in old Russian style of sitting in the library and proving hard theorems. It is not that one should not value or finance education and dissemination of knowledge. But it should be DIFFERENT job and not interfering with appreciation/evaluation of primary results. If somebody is going to get a journalism prize let it not be called a theorem of the year, but a self presentation of the year or whatever. Mathematicians are sometimes subtle in distinguishing nuiances, and should do that even more emphatically here, where people starting mixing apples and pears.

The fact that popular sites Facebook, google+, amazon, netflix, MO have like buttons does not mean that their popularity mechanism makes them better in inducing the true development of personality etc. Popularity is not a measure of development and achievement itself. A Serbian kid growing up in California came to his parents place in Belgrade and was surprised that he was accepted to any children group to play with on the street, not preceded by marked belonging to a clique, clan or band, unlike what he learned in modern California, driven by TC shows, popularity culture, hi school prom craze. In fact I NEVER heard in my childhood in Croatia that the phrase “be popular” applies to kids. Word “popular” was about movie stars and so on, not about something relevant to our ordinary lives. This was few decades ago. Now it is probably changing, everybody needs to have a good picture to put on social site, and people kill when get defriended. Finally, the democracy in US is ridiculuous – the PR makes people choose whoever, all is lobbyied and advertised; voting is not discovery of the truth.

I fully agree with more sensitive 11: especially this – for peer reviewing it is beneficial to be independent, that one referee does not know what the other one read. Also ASSIGNING a referee makes people discover things which are not easy to appreciate without through reading. It may be a boring looking paper, but the arguments while on the surface written by the same kind of letters and equation symbols are in fact a deep breakthrough. A casual audience will miss this, while true reader will be forced to understand. Once understood it is more likely to give a more balanced mark of the achievement.

On the other hand, I do second that one should have pages of discussion for individual articles. Not collections of opinions, but more a wiki style. Look at wikipedia. If somebody writes something without arguments and references, he is likely to be overwritten. It is also possible to write remarks about the paper. If you observe some interesting relation to other areas, or that the result follows from something else it stays in the wiki. If you repeat the bullshitting which somebody else wrote before, your sentence will get into the history and erased from the new assesment, or labelled as irrelevant.

The danger is also that the author himself will make his work more appreciated by answering to the craze. Instead, as Arnold used to say, one should work in old Russian style of sitting in the library and proving hard theorems. It is not that one should not value or finance education and dissemination of knowledge. But it should be DIFFERENT job and not interfering with appreciation/evaluation of primary results. If somebody is going to get a journalism prize let it not be called a theorem of the year, but a self presentation of the year or whatever. Mathematicians are sometimes subtle in distinguishing nuiances, and should do that even more emphatically here, where people starting mixing apples and pears.

Author: kwalkerFormat: TextI like this idea, and I agree with most of what John Baez and Scott Morrison wrote.
Because of the complexity of the issues involved, and because of the many different directions such a comment site could take, the discussion here is very wide-ranging. Perhaps it would be useful to have a summary of of the issues that have been raised, just to keep the discussion organized and moving forward (as opposed to swirling around). I was about to attempt such a summary myself, but I suspect there are others here who would do a better job than I would, so I'll refrain for the moment.

I like this idea, and I agree with most of what John Baez and Scott Morrison wrote.

Because of the complexity of the issues involved, and because of the many different directions such a comment site could take, the discussion here is very wide-ranging. Perhaps it would be useful to have a summary of of the issues that have been raised, just to keep the discussion organized and moving forward (as opposed to swirling around). I was about to attempt such a summary myself, but I suspect there are others here who would do a better job than I would, so I'll refrain for the moment.

Author: zskodaFormat: MarkdownItex> and help decide what we might like to read
One top mathematician whom I pretty much appreciate told me that he almost never goes to the library nowdays and reads only on the internet. He says, in my memory:
> I do not care what is published what is not, I google for the name of the author whom I trust or the title I suspect to be interesting, and find it on the arxiv or the homepage of the author. __I do not need others__ (i.e. journal boards) __to decide what I will read, I want myself to decide what I will read.__
This is not a view of a youngster but of an active mathematicians closer to the retirement than to the PhD. And of course he does appreciate peer advice on what he will read, but only from the colleagues he directly talks to and trusts, of those whose good lecture he might have heard on the topic and not from the "societies", networks, pyramid schemes, points systems and alike.
This is not to discourage building sites for discussions, wikis etc. -- __we need them, especially to diseminate the information from the rare centers of hi concentration of expertise__ in some areas of knowledge __to the rest of the intellectual world__ -- but rather to cautiously raise the awareness of the importance of the preservation of healthy basic old practices in refereeing, reading, teaching and research practices; those which the popularity chase and craze and related psychological and social rearrangements, new orders and disorders, eases and diseases do not properly replace. But some new technology will add them. There is so much more new in software and on the internet on the market, than the liking/popularity/points systems.

and help decide what we might like to read

One top mathematician whom I pretty much appreciate told me that he almost never goes to the library nowdays and reads only on the internet. He says, in my memory:

I do not care what is published what is not, I google for the name of the author whom I trust or the title I suspect to be interesting, and find it on the arxiv or the homepage of the author. I do not need others (i.e. journal boards) to decide what I will read, I want myself to decide what I will read.

This is not a view of a youngster but of an active mathematicians closer to the retirement than to the PhD. And of course he does appreciate peer advice on what he will read, but only from the colleagues he directly talks to and trusts, of those whose good lecture he might have heard on the topic and not from the “societies”, networks, pyramid schemes, points systems and alike.

This is not to discourage building sites for discussions, wikis etc. – we need them, especially to diseminate the information from the rare centers of hi concentration of expertise in some areas of knowledge to the rest of the intellectual world – but rather to cautiously raise the awareness of the importance of the preservation of healthy basic old practices in refereeing, reading, teaching and research practices; those which the popularity chase and craze and related psychological and social rearrangements, new orders and disorders, eases and diseases do not properly replace. But some new technology will add them. There is so much more new in software and on the internet on the market, than the liking/popularity/points systems.

Author: tlawsonFormat: MarkdownItexI'm not in support of having any kind of "point" system papers on the arXiv. There are already a number of mechanisms in place that reward scholarship, and already more than enough incentives to produce widely popular work.
I don't think that it's necessarily a bad idea to have some kind of "reward" system in place, but I think that people underestimate how carefully thought-out these point systems are. These don't make the site inherently engaging or psychologically addictive on their own. They specifically provide some form of inexpensive reward for contributing to the site, and a reason to check back regularly, until using the site becomes a part of the user's routine.
So if you're trying to organize large-scale adoption of a site and you want users to
* collect public lists of literature that interests them,
* submit short synopses or reviews of papers,
* ask questions about papers,
* respond to questions on their own papers,
* make sure that even "lower-profile" papers get reviews, or
* maintain professional community standards,
then I feel like these are the things that you should be providing rewards for.

I’m not in support of having any kind of “point” system papers on the arXiv. There are already a number of mechanisms in place that reward scholarship, and already more than enough incentives to produce widely popular work.

I don’t think that it’s necessarily a bad idea to have some kind of “reward” system in place, but I think that people underestimate how carefully thought-out these point systems are. These don’t make the site inherently engaging or psychologically addictive on their own. They specifically provide some form of inexpensive reward for contributing to the site, and a reason to check back regularly, until using the site becomes a part of the user’s routine.

So if you’re trying to organize large-scale adoption of a site and you want users to

collect public lists of literature that interests them,

submit short synopses or reviews of papers,

ask questions about papers,

respond to questions on their own papers,

make sure that even “lower-profile” papers get reviews, or

maintain professional community standards,

then I feel like these are the things that you should be providing rewards for.

Author: Andrew StaceyFormat: MarkdownItex(Kevin: remember that you can start a new discussion on any particular idea that strikes you. Just link between the two to keep them connected.)
With all the ideas I read here, I try to examine them under the spotlight of "How would this make a difference to my daily work?". A review site sounds fun, but I don't know that I would use it, and I don't think that I would be all that happy having my papers there. I'm worried that mine would fall into that category where the "number of people who read it is a number between 0 and 1". The time that I want to talk about my work is when I'm doing it, not when I've written the paper. I like the quote from Dror-Bar Natan [on MathOverflow](http://mathoverflow.net/questions/83092/what-is-the-main-goal-of-a-paper-really/83093#83093) (see also [Fran&ccedila;ois Dorias' website](http://dorais.org/archives/630)):
> Papers are written so that their author(s) can forget their content and move on to other things. Therefore when you write you should be very careful to put in enough of the big picture and enough of the details so you’d be able to reconstruct your thoughts 10 years later if you’ll need to, assuming you’ll forget everything but retain some familiarity with some basic principles of mathematics.
So when I've written the paper and submitted it to a journal, I'm ready to move on to the next thing. I'm quite happy to talk about that paper, but I'm not so excited about it as I was. Indeed, often by the time I've written the paper I get a strange "flat" feeling about it as though I'll be glad to see the back of it - at least for a short period.
The time when I'd really like to talk about it is when I'm writing the paper. Call this the "alpha" stage. That's when I'm sure of the basic ideas, but am aware that I might radically change how I think of them as time goes on. Then when I've written the paper, I don't really want to be told of a whole new way of thinking about it, but I don't mind being told that my notation is rubbish and I should have written it all in the passive voice. Call this the "beta" stage. Finally, I submit the article to a journal and hope that that's dealt with.
That's as an author.
As a reader, I have considerable sympathy with Zoran's last post and with Scott's further up. I don't need to be told whether or not I should be interested in a paper. If it contains the result I want, then I'm interested. If it doesn't, then I'm not. At least, that is, when I'm doing my Main Research - of course, sometimes I read articles just to find out what's going on, but that's a bit different. So I don't want to be told that I should read a paper because it has 5 stars! But I would find it useful to know if there are any corrections or clarifications attached to the paper - this is subtly different to a review.

(Kevin: remember that you can start a new discussion on any particular idea that strikes you. Just link between the two to keep them connected.)

With all the ideas I read here, I try to examine them under the spotlight of “How would this make a difference to my daily work?”. A review site sounds fun, but I don’t know that I would use it, and I don’t think that I would be all that happy having my papers there. I’m worried that mine would fall into that category where the “number of people who read it is a number between 0 and 1”. The time that I want to talk about my work is when I’m doing it, not when I’ve written the paper. I like the quote from Dror-Bar Natan on MathOverflow (see also Fran&ccedila;ois Dorias’ website):

Papers are written so that their author(s) can forget their content and move on to other things. Therefore when you write you should be very careful to put in enough of the big picture and enough of the details so you’d be able to reconstruct your thoughts 10 years later if you’ll need to, assuming you’ll forget everything but retain some familiarity with some basic principles of mathematics.

So when I’ve written the paper and submitted it to a journal, I’m ready to move on to the next thing. I’m quite happy to talk about that paper, but I’m not so excited about it as I was. Indeed, often by the time I’ve written the paper I get a strange “flat” feeling about it as though I’ll be glad to see the back of it - at least for a short period.

The time when I’d really like to talk about it is when I’m writing the paper. Call this the “alpha” stage. That’s when I’m sure of the basic ideas, but am aware that I might radically change how I think of them as time goes on. Then when I’ve written the paper, I don’t really want to be told of a whole new way of thinking about it, but I don’t mind being told that my notation is rubbish and I should have written it all in the passive voice. Call this the “beta” stage. Finally, I submit the article to a journal and hope that that’s dealt with.

That’s as an author.

As a reader, I have considerable sympathy with Zoran’s last post and with Scott’s further up. I don’t need to be told whether or not I should be interested in a paper. If it contains the result I want, then I’m interested. If it doesn’t, then I’m not. At least, that is, when I’m doing my Main Research - of course, sometimes I read articles just to find out what’s going on, but that’s a bit different. So I don’t want to be told that I should read a paper because it has 5 stars! But I would find it useful to know if there are any corrections or clarifications attached to the paper - this is subtly different to a review.

Author: yfarjounFormat: TextARGH!!! I wrote a long comment and then it got erased since somehow I got logged out by the time I clicked submit. At this point I'm too annoyed to write it fully again, so here's the main points:
I think that there are flaws in any point-based system like "+1" "like" etc. I suggest a similar but different approach based on Twitter.
The idea is that each user can "read" any number of papers on the arxiv and also to "follow" any number of users (and thus see what papers they are "read"ing). from the resulting graph, many different scores can be formulated, and the users can each select to view next to each paper a subset of these scores. Good score formulas would render sockpuppeteering useless.
To jump-start the system, one could use the download information from the arXiv (using hashed IP as the user) if they give it.

ARGH!!! I wrote a long comment and then it got erased since somehow I got logged out by the time I clicked submit. At this point I'm too annoyed to write it fully again, so here's the main points:

I think that there are flaws in any point-based system like "+1" "like" etc. I suggest a similar but different approach based on Twitter.

The idea is that each user can "read" any number of papers on the arxiv and also to "follow" any number of users (and thus see what papers they are "read"ing). from the resulting graph, many different scores can be formulated, and the users can each select to view next to each paper a subset of these scores. Good score formulas would render sockpuppeteering useless.

To jump-start the system, one could use the download information from the arXiv (using hashed IP as the user) if they give it.

Author: Yiftach BarneaFormat: TextIt seems that quite a lot of people would like to see some kind of ranking. If indeed such ranking will exist, that will be a negative incentive for me to put my papers in such system. I am sure there are many other people like me. The same is true for being exposed to negative comment. I am fine with receiving negative comments, but I would not like everyone to see them. So if you would like everyone (more or less) to take part in such a system, then you have to accommodate to such concerns. The fact that you are not worry about it is not relevant. The question is whether substantial amount of people are worried about it.

It seems that quite a lot of people would like to see some kind of ranking. If indeed such ranking will exist, that will be a negative incentive for me to put my papers in such system. I am sure there are many other people like me. The same is true for being exposed to negative comment. I am fine with receiving negative comments, but I would not like everyone to see them. So if you would like everyone (more or less) to take part in such a system, then you have to accommodate to such concerns. The fact that you are not worry about it is not relevant. The question is whether substantial amount of people are worried about it.

Author: John BaezFormat: MarkdownItexAndrew wrote:
> I don't need to be told whether or not I should be interested in a paper.
I don't 'need to be told', either. That sounds pathetic. I _*want to hear*_ what papers people are interested in - particularly, specific people I know and trust. And I'm not alone in this: lots of people want to know what their friends and respected colleagues are reading.... and what they think about those papers. They gossip about this endlessly at conferences.
This is why I like the idea of a system where people are able to opt to let others see which papers they like. If someone like Witten or Atiyah or Deligne did this, you can darn well bet that hundreds of people would turn on their computer each morning to see what papers those guys liked.
On a lesser scale, _This Week's Finds in Mathematical Physics_ was a way for me to tell people what papers I liked, and also say a bit about them. People seemed to read it, even though I started as a nobody. Most people don't have the energy to write a column like that, but maybe they'd have the energy click 'like' on a paper now and then, and sometimes even write a paragraph saying something about it. The system I'm envisioning would make all that easy. Of course people could include links to blogs, too.

Andrew wrote:

I don’t need to be told whether or not I should be interested in a paper.

I don’t ’need to be told’, either. That sounds pathetic. I want to hear what papers people are interested in - particularly, specific people I know and trust. And I’m not alone in this: lots of people want to know what their friends and respected colleagues are reading…. and what they think about those papers. They gossip about this endlessly at conferences.

This is why I like the idea of a system where people are able to opt to let others see which papers they like. If someone like Witten or Atiyah or Deligne did this, you can darn well bet that hundreds of people would turn on their computer each morning to see what papers those guys liked.

On a lesser scale, This Week’s Finds in Mathematical Physics was a way for me to tell people what papers I liked, and also say a bit about them. People seemed to read it, even though I started as a nobody. Most people don’t have the energy to write a column like that, but maybe they’d have the energy click ’like’ on a paper now and then, and sometimes even write a paragraph saying something about it. The system I’m envisioning would make all that easy. Of course people could include links to blogs, too.

Author: John BaezFormat: MarkdownItexKevin wrote:
> Because of the complexity of the issues involved, and because of the many different directions such a comment site could take, the discussion here is very wide-ranging. Perhaps it would be useful to have a summary of of the issues that have been raised, just to keep the discussion organized and moving forward (as opposed to swirling around). I was about to attempt such a summary myself, but I suspect there are others here who would do a better job than I would, so I'll refrain for the moment.
No, do it! - unless you have some power to make those better people do it.

Kevin wrote:

Because of the complexity of the issues involved, and because of the many different directions such a comment site could take, the discussion here is very wide-ranging. Perhaps it would be useful to have a summary of of the issues that have been raised, just to keep the discussion organized and moving forward (as opposed to swirling around). I was about to attempt such a summary myself, but I suspect there are others here who would do a better job than I would, so I’ll refrain for the moment.

No, do it! - unless you have some power to make those better people do it.

Author: plmFormat: TextI definitely agree with John. Please someone (Andrew?) make a comment site linked to the Cornell library arxiv mirror.
Hopefully this would soon be officially integrated to arxiv, with the possibility to open one's article to comments or not, for privacy. I think some would be deterred from posting on the arxiv if it were like opening a thread in a forum, but it is not clear how well the (default) option to disable comments would remedy this; though I guess the decision to post on the arxiv already requires enough openness from the poster to accept possibly (rightly or wrongly) annoying comments.
Someone who posted as Ralph on Tim's blog also mentioned this:
http://gowers.wordpress.com/2012/01/29/whats-wrong-with-electronic-journals/#comment-15399

I definitely agree with John. Please someone (Andrew?) make a comment site linked to the Cornell library arxiv mirror.

Hopefully this would soon be officially integrated to arxiv, with the possibility to open one's article to comments or not, for privacy. I think some would be deterred from posting on the arxiv if it were like opening a thread in a forum, but it is not clear how well the (default) option to disable comments would remedy this; though I guess the decision to post on the arxiv already requires enough openness from the poster to accept possibly (rightly or wrongly) annoying comments.

Someone who posted as Ralph on Tim's blog also mentioned this:http://gowers.wordpress.com/2012/01/29/whats-wrong-with-electronic-journals/#comment-15399

Author: Charles RezkFormat: MarkdownItexI think John's got it right: the draw is not the ability to "rate papers" on some absolute scale. The draw is to be able to find out what your colleagues think is interesting, where "your colleagues" is a function of "you".

I think John’s got it right: the draw is not the ability to “rate papers” on some absolute scale. The draw is to be able to find out what your colleagues think is interesting, where “your colleagues” is a function of “you”.

Author: zskodaFormat: MarkdownItex28 I agreed above that the discussion sites are welcome, the disagreement was about a point driven system.
21 tlawson says that if one is going to compile bibliography lists and so on, this should be driven by reward system. I have created hundred of nLab entries and put there many hundred references and do that on daily basis for a couple of years. If anybody would pressure me by having the points of my contribution publicly seen (daily followed) at semaphore I WOULD STOP CONTRIBUTING. (Edit: the fact that I am much less active in MathOverflow is partly because I dislike the atmosphere of the reward system there). Many people like me in mathematics find beauty and peace and do not believe that one creative involvement can be linearly compared to any other involvement. If one is creating, one is expecting to see something nice building up. I do not need point like rewards, I need to see that something is created and built, and that some things are solved my own way which I like in the process. I like that I can come back to my notes when I need them again. If I made something neat that this neatness helps me in future, and helps my students and so on. Of course, I like others contributing where I start. But why to mix social needs, cooperative works and so on, with the popularity and linearizing craze ? I know that
US media dominate the world and that US culture is obsessed with point systems from kindergarten, but why would adult creative people be bogged with that ?
P.S. I proposed accompanying arXiv with discussion features and hep-th-comment section for short remarks and replies instead of papers still in 2000 in a discussion with Vafa at a conference on orbifolds in Madison. But not in the faintest dream I thought of a point system and rewards, rather than posting arguments, comments, in other words stimulative CONTENT and not pure likes and dislikes.

28 I agreed above that the discussion sites are welcome, the disagreement was about a point driven system.

21 tlawson says that if one is going to compile bibliography lists and so on, this should be driven by reward system. I have created hundred of nLab entries and put there many hundred references and do that on daily basis for a couple of years. If anybody would pressure me by having the points of my contribution publicly seen (daily followed) at semaphore I WOULD STOP CONTRIBUTING. (Edit: the fact that I am much less active in MathOverflow is partly because I dislike the atmosphere of the reward system there). Many people like me in mathematics find beauty and peace and do not believe that one creative involvement can be linearly compared to any other involvement. If one is creating, one is expecting to see something nice building up. I do not need point like rewards, I need to see that something is created and built, and that some things are solved my own way which I like in the process. I like that I can come back to my notes when I need them again. If I made something neat that this neatness helps me in future, and helps my students and so on. Of course, I like others contributing where I start. But why to mix social needs, cooperative works and so on, with the popularity and linearizing craze ? I know that
US media dominate the world and that US culture is obsessed with point systems from kindergarten, but why would adult creative people be bogged with that ?

P.S. I proposed accompanying arXiv with discussion features and hep-th-comment section for short remarks and replies instead of papers still in 2000 in a discussion with Vafa at a conference on orbifolds in Madison. But not in the faintest dream I thought of a point system and rewards, rather than posting arguments, comments, in other words stimulative CONTENT and not pure likes and dislikes.

Author: Yiftach BarneaFormat: Text29 +1 :-) (Sorry, I couldn’t resist.)
I agree with Zoran, in some sense coming exactly from the other direction. I tend to be competitive. I don’t care about the point system in MathOverflow, because it has no real life affect, so it is harmless fun. But I really don’t want a point system on anything important like research. Would you have a point system at home? You changed the baby diaper, you get 10 points. Your partner took the kids to a birthday party, 20 points. Now, you are behind, what would you do?

29 +1 :-) (Sorry, I couldn’t resist.)I agree with Zoran, in some sense coming exactly from the other direction. I tend to be competitive. I don’t care about the point system in MathOverflow, because it has no real life affect, so it is harmless fun. But I really don’t want a point system on anything important like research. Would you have a point system at home? You changed the baby diaper, you get 10 points. Your partner took the kids to a birthday party, 20 points. Now, you are behind, what would you do?

Author: kwalkerFormat: MarkdownItexAs requested, here's an attempt to summarize the discussion. Reading through the comments a second time they seem less diverse and unfocused than they did to me on the first reading, so perhaps this is unnecessary. "#n" means comment number n. I've probably overlooked some stuff; bring it to my attention and I'll edit this post accordingly.
*Main idea* (#1). A site where one can discuss Arxiv papers, possibly also with some sort of rating system or some way of finding papers which are popular in general or popular with particular people (such as the hugely influential Noah Snyder; see #14).
*Existing systems*. Similar things have been tried before, so we should try to learn from these earlier attempts (scirate.com (#2, #3), papercritic.com (#7), David Bacon quant-ph thing (#12) [is this the same as scirate.com?]).
*Ratings*. They could take various forms:
* Actual direct ratings of papers, on a scale of 1 to 5, say.
* Separate ratings for importance, readability, etc. (#17).
* Simple "+1" or "favorite" type system.
* Each person could maintain a list of papers that they like, but these lists would not necessarily be aggregated (#14, #23).
* Perhaps (only?) comments should be rated (like on amazon.com) (#16, #21).
Arguments in favor of ratings:
* Many successful sites use a rating system, and perhaps the system is partly responsible for their success (#12).
* It is important to be able to quickly and easily find your way to the most useful papers, and some sort of rating system would help with this (#14, #25, #28).
Arguments against ratings:
* Too many to list fully here. See #4, #6, #8, #9, #10, #18, #20, #24, #29, #30 (probably I missed a few).
* Ratings are too arbitrary, which is maybe OK on mathoverflow when not much is at stake, but would be bad for a site that might influence hiring decisions, etc.
* If the ratings are important, they will likely be gamed.
* Ratings could make the site less fun.

As requested, here’s an attempt to summarize the discussion. Reading through the comments a second time they seem less diverse and unfocused than they did to me on the first reading, so perhaps this is unnecessary. “#n” means comment number n. I’ve probably overlooked some stuff; bring it to my attention and I’ll edit this post accordingly.

Main idea (#1). A site where one can discuss Arxiv papers, possibly also with some sort of rating system or some way of finding papers which are popular in general or popular with particular people (such as the hugely influential Noah Snyder; see #14).

Existing systems. Similar things have been tried before, so we should try to learn from these earlier attempts (scirate.com (#2, #3), papercritic.com (#7), David Bacon quant-ph thing (#12) [is this the same as scirate.com?]).

Ratings. They could take various forms:

Actual direct ratings of papers, on a scale of 1 to 5, say.

Separate ratings for importance, readability, etc. (#17).

Simple “+1” or “favorite” type system.

Each person could maintain a list of papers that they like, but these lists would not necessarily be aggregated (#14, #23).

Author: John BaezFormat: MarkdownItexRezk wrote:
> I think John's got it right: the draw is not the ability to "rate papers" on some absolute scale. The draw is to be able to find out what your colleagues think is interesting, where "your colleagues" is a function of "you".
Right. I should have made that more clear - and the website that does this should also make it very clear!
pln wrote:
> I definitely agree with John. Please someone (Andrew?) make a comment site linked to the Cornell library arxiv mirror.
My friend [Chris Lee](http://faculty.chemistry.ucla.edu/institution/personnel?personnel_id=45756), head of bioinformatics at UCLA, is interested in helping set up such a site. He's good at programming and he's written an interesting paper:
* [Open peer review by a selected papers network](http://thinking.bioinformatics.ucla.edu/2011/07/02/open-peer-review-by-a-selected-papers-network/).
I hope some people here can join with him in rapidly setting up some sort of 'quick, easy and fun' website, while leaving open the ability to expand and improve it. I'll keep talking to him and all of you to help coordinate things. I've tried to get him to post comments here himself, too.

Rezk wrote:

I think John’s got it right: the draw is not the ability to “rate papers” on some absolute scale. The draw is to be able to find out what your colleagues think is interesting, where “your colleagues” is a function of “you”.

Right. I should have made that more clear - and the website that does this should also make it very clear!

pln wrote:

I definitely agree with John. Please someone (Andrew?) make a comment site linked to the Cornell library arxiv mirror.

My friend Chris Lee, head of bioinformatics at UCLA, is interested in helping set up such a site. He’s good at programming and he’s written an interesting paper:

I hope some people here can join with him in rapidly setting up some sort of ’quick, easy and fun’ website, while leaving open the ability to expand and improve it. I’ll keep talking to him and all of you to help coordinate things. I’ve tried to get him to post comments here himself, too.

Author: John BaezFormat: MarkdownItexBy the way: one reason I hope this 'quick, easy and fun' website gets started soon is that it will pressure people who dislike it to come up with something better. I'm not sure what the best solutions to our problems are. But I don't think we can find out until someone tries something.

By the way: one reason I hope this ’quick, easy and fun’ website gets started soon is that it will pressure people who dislike it to come up with something better. I’m not sure what the best solutions to our problems are. But I don’t think we can find out until someone tries something.

Author: zskodaFormat: MarkdownItexTim Gowers in the article [a-more-modest-proposal](http://gowers.wordpress.com/2011/11/03/a-more-modest-proposal) proposed a site which would be dedicated to mutual commenting of people (with point-pay for comments from author to the commenter) on their preprints, helping so to improve the papers which are not yet in the final stage. While credits are given, they do not feed the vanity in that case, but the direct need to have the points earned (for the job or pre-reviewer) in order to pay for your own papers being pre-reviewed and hence helped in their finalization and improvement. I like his proposal. The comments there are also very interesting.

Tim Gowers in the article a-more-modest-proposal proposed a site which would be dedicated to mutual commenting of people (with point-pay for comments from author to the commenter) on their preprints, helping so to improve the papers which are not yet in the final stage. While credits are given, they do not feed the vanity in that case, but the direct need to have the points earned (for the job or pre-reviewer) in order to pay for your own papers being pre-reviewed and hence helped in their finalization and improvement. I like his proposal. The comments there are also very interesting.

Author: Peter KrautzbergerFormat: MarkdownItexI'm wondering if it is ok to open a related thread (or better, a wiki page somewhere) to collect existing technologies, i.e., isolate that aspect of Kevin Walker's summary #31 (thanks Kevin!).
This could be both about technology (scirate, papercritic, altmetrics.org, +1 etc) but also anything to put these tools in perspective (for example [this study on tweets predicting citations](http://blogs.lse.ac.uk/impactofsocialsciences/2012/02/09/more-tweets-more-citations/)).
While this thread shows that there's need to discuss how we value these tools, I think it might be good to have a neutral thread/wiki page to simply collect facts and usage.
Just like John's #33 said, the sooner we start, the sooner we'll know what works. (I would add: there won't be one tool that works for everybody, but there will be equally useful tools so that everybody can find one that suits them.)

I’m wondering if it is ok to open a related thread (or better, a wiki page somewhere) to collect existing technologies, i.e., isolate that aspect of Kevin Walker’s summary #31 (thanks Kevin!).

While this thread shows that there’s need to discuss how we value these tools, I think it might be good to have a neutral thread/wiki page to simply collect facts and usage.

Just like John’s #33 said, the sooner we start, the sooner we’ll know what works. (I would add: there won’t be one tool that works for everybody, but there will be equally useful tools so that everybody can find one that suits them.)

Author: Andrew StaceyFormat: MarkdownItexWhy not start a discussion here first (make it clear that it's just for gathering information, not discussing it) and then when the amount of information gets unwieldy, we can look for a more permanent home for it. Wikis are easy to set up, and if needed then we could set up a wiki linked to this forum to gather information - if there isn't already somewhere suitable. But I don't want to scatter information too much too early.

Why not start a discussion here first (make it clear that it’s just for gathering information, not discussing it) and then when the amount of information gets unwieldy, we can look for a more permanent home for it. Wikis are easy to set up, and if needed then we could set up a wiki linked to this forum to gather information - if there isn’t already somewhere suitable. But I don’t want to scatter information too much too early.

Author: Andrew StaceyFormat: MarkdownItexJohn said in #25:
> If someone like Witten or Atiyah or Deligne did this, you can darn well bet that hundreds of people would turn on their computer each morning to see what papers those guys liked.
> On a lesser scale, This Week's Finds in Mathematical Physics was a way for me to tell people what papers I liked, and also say a bit about them. People seemed to read it, even though I started as a nobody.
There is a huge difference between these two situations, and I like the second but dislike the first.
I remember what led me to the idea of review panels. It was when I learnt about the [Journal of Stuff I Like](http://scienceblogs.com/principles/2008/09/science21_the_journal_of_stuff.php). Initially, I thought this was great. But as I thought more, I decided that it wasn't such a great idea after all. The danger with it is that it leads to the *Cult of the Personality*. John is absolutely right - everyone would tune in to see what Witten or Atiyah or Deligne were reading (or Terry Tao or Tim Gowers or John Baez), but no-one would be all that interested in what Andrew Stacey was reading. So we'd be shifting the pressure from trying to be published in Annals to being mentioned by one of the Big Wigs. This feels wrong on *so* many levels.
However, *This Week's Finds* is a wholly different beast. I admit that I've only dipped in here and there, so I may have the wrong impression. That impression is that you (John) are more interested in telling us about a particular *topic* than about a particular *paper*. There's always "Where to find out more" lists at the end, and it may be a paper that sparked the interest in the topic, but the central theme is of "Here's something cool to know about" rather than "Here's something cool to read". Now *that* is far more useful. Whilst I feel that I don't really need someone to tell me whether or not to read a particular article - once I have that article in front of me - I often feel that I need someone to say "If you want to learn about *X* then the papers to start with are *A*, *B*, and *C*".
So I see a big difference between setting up a system for adding reviews to the arXiv, and a system for making it easier to find papers by topic. The former doesn't seem to add much usefulness, just a load of froth(!), but the latter makes it easier for me to do my job.

John said in #25:

If someone like Witten or Atiyah or Deligne did this, you can darn well bet that hundreds of people would turn on their computer each morning to see what papers those guys liked.

On a lesser scale, This Week’s Finds in Mathematical Physics was a way for me to tell people what papers I liked, and also say a bit about them. People seemed to read it, even though I started as a nobody.

There is a huge difference between these two situations, and I like the second but dislike the first.

I remember what led me to the idea of review panels. It was when I learnt about the Journal of Stuff I Like. Initially, I thought this was great. But as I thought more, I decided that it wasn’t such a great idea after all. The danger with it is that it leads to the Cult of the Personality. John is absolutely right - everyone would tune in to see what Witten or Atiyah or Deligne were reading (or Terry Tao or Tim Gowers or John Baez), but no-one would be all that interested in what Andrew Stacey was reading. So we’d be shifting the pressure from trying to be published in Annals to being mentioned by one of the Big Wigs. This feels wrong on so many levels.

However, This Week’s Finds is a wholly different beast. I admit that I’ve only dipped in here and there, so I may have the wrong impression. That impression is that you (John) are more interested in telling us about a particular topic than about a particular paper. There’s always “Where to find out more” lists at the end, and it may be a paper that sparked the interest in the topic, but the central theme is of “Here’s something cool to know about” rather than “Here’s something cool to read”. Now that is far more useful. Whilst I feel that I don’t really need someone to tell me whether or not to read a particular article - once I have that article in front of me - I often feel that I need someone to say “If you want to learn about X then the papers to start with are A, B, and C”.

So I see a big difference between setting up a system for adding reviews to the arXiv, and a system for making it easier to find papers by topic. The former doesn’t seem to add much usefulness, just a load of froth(!), but the latter makes it easier for me to do my job.

Author: Alexander WooFormat: TextI am a little apprehensive about this idea, and I'm more worried about it working too well than about it not working at all.
The problem is that this service does nothing for those papers no one reads and those mathematicians whose papers no one reads.
If this service works too well, and the research community abandons the journal system for a system where papers are tracked by how many mathematicians (or, worse yet, which well-known mathematicians) read and recommend them, then we lose the ability to give a minimal certifying stamp of 'this is valid and most likely correct and somewhat novel mathematics research' to uninteresting papers. The danger is that the 'research elite' (which I'm defining broadly enough to include I think everyone whom I've seen post on this forum so far) create a system that works for us and abandons everyone else.

I am a little apprehensive about this idea, and I'm more worried about it working too well than about it not working at all.

The problem is that this service does nothing for those papers no one reads and those mathematicians whose papers no one reads.

If this service works too well, and the research community abandons the journal system for a system where papers are tracked by how many mathematicians (or, worse yet, which well-known mathematicians) read and recommend them, then we lose the ability to give a minimal certifying stamp of 'this is valid and most likely correct and somewhat novel mathematics research' to uninteresting papers. The danger is that the 'research elite' (which I'm defining broadly enough to include I think everyone whom I've seen post on this forum so far) create a system that works for us and abandons everyone else.

Author: Scott MorrisonFormat: MarkdownItex@Alexander, I remember reading a great comment you made somewhere, where you talked about what the purpose of peer review is, for all the researchers outside of that 'research elite' you mention. Now I can't find it, even with the help of google. Could you point me to it if you remember the one I mean, or otherwise write somewhere (here, a blog, etc? :-) about this again? I agree people need to appreciate this point further, but I know I can't articulate it quite right yet!

@Alexander, I remember reading a great comment you made somewhere, where you talked about what the purpose of peer review is, for all the researchers outside of that ’research elite’ you mention. Now I can’t find it, even with the help of google. Could you point me to it if you remember the one I mean, or otherwise write somewhere (here, a blog, etc? :-) about this again? I agree people need to appreciate this point further, but I know I can’t articulate it quite right yet!

Author: Alexander ChervovFormat: TextJohn Baez wrote: "A system linked to the arXiv, which looks a lot like the arXiv: one page per paper .... But you can also comment on the paper ... "
Imho, this is very good and necessary idea and actually it is almost-done - arXiv has system of trackbacks.
So I think the pretty simple step to start right today - is the following:
When one submits a paper to arXiv , he by himself creates a) say wordpress page (e.g. on his blog or some joint blog)
which is devoted solely to this paper b) trackback to it.
Everyone can leave comments/questions/ remarks on this wordpress page.
That's it. We do not need new soft or whatever.
The only thing I guess is may be worth to add is to create some mailing-list or better site, where those people who wants to get comments
can show up themselves: "Hey guys comments to my paper are welcome". It is not necessary for those people who look all arXiv papers everyday
because they will see the trackback with "comment on me" when paper appears,
however it would be helpful for those who are looking arXiv irregullary... (E.g. I am looking QA, almost everyday, but alg-geom not so often).

John Baez wrote: "A system linked to the arXiv, which looks a lot like the arXiv: one page per paper .... But you can also comment on the paper ... "

Imho, this is very good and necessary idea and actually it is almost-done - arXiv has system of trackbacks. So I think the pretty simple step to start right today - is the following:

When one submits a paper to arXiv , he by himself creates a) say wordpress page (e.g. on his blog or some joint blog)which is devoted solely to this paper b) trackback to it.Everyone can leave comments/questions/ remarks on this wordpress page.

That's it. We do not need new soft or whatever.

The only thing I guess is may be worth to add is to create some mailing-list or better site, where those people who wants to get commentscan show up themselves: "Hey guys comments to my paper are welcome". It is not necessary for those people who look all arXiv papers everydaybecause they will see the trackback with "comment on me" when paper appears,however it would be helpful for those who are looking arXiv irregullary... (E.g. I am looking QA, almost everyday, but alg-geom not so often).

Author: zskodaFormat: MarkdownItexWell, the email addresses are included in the papers and often there is a mark that the comments are welcome. This does not make many people commenting by email anyway. Web makes it sometimes easier, unfortunately many commenting sites will have their own resistrations which make a turn off to many. So it is good that it is not just trackbakc link but indeed one unique system for the whole arxiv. Gowers' [idea](http://gowers.wordpress.com/2011/11/03/a-more-modest-proposal) that we could have a system of credits for "reviewing" preprints, granted by the authors, who have some credits to give up, is a good incentive for certain kind of activity. Having possibly new arXiv categories of new micro-"comment" articles which could be stably cited is another. We need to discuss more mechanisms and forms.

Well, the email addresses are included in the papers and often there is a mark that the comments are welcome. This does not make many people commenting by email anyway. Web makes it sometimes easier, unfortunately many commenting sites will have their own resistrations which make a turn off to many. So it is good that it is not just trackbakc link but indeed one unique system for the whole arxiv. Gowers’ idea that we could have a system of credits for “reviewing” preprints, granted by the authors, who have some credits to give up, is a good incentive for certain kind of activity. Having possibly new arXiv categories of new micro-“comment” articles which could be stably cited is another. We need to discuss more mechanisms and forms.

Author: Alexander ChervovFormat: Text@Zoran, I agree that Gowers proposal is much bigger, just that it is first step can be done today within existing technologies and build up community doing this.
Concerning private comments to authors by e-mails. I see the big differences, do not You ?

@Zoran, I agree that Gowers proposal is much bigger, just that it is first step can be done today within existing technologies and build up community doing this.

Concerning private comments to authors by e-mails. I see the big differences, do not You ?

Author: foobaronFormat: TextHi, I'm new here. I hear some people voicing various assumptions that open selection systems can never work as well as closed selection systems (i.e. journals). Before you assume that, we would love it if you would look at the specifics of what we're actually proposing. Sorry to be annoying, but it seems like everyone just ignored the link John posted above to our detailed proposal and analysis of the challenges of open selection networks (written by me, but based on ideas from many, many people; see the Acknowledgments, linked from John's post #32).

Hi, I'm new here. I hear some people voicing various assumptions that open selection systems can never work as well as closed selection systems (i.e. journals). Before you assume that, we would love it if you would look at the specifics of what we're actually proposing. Sorry to be annoying, but it seems like everyone just ignored the link John posted above to our detailed proposal and analysis of the challenges of open selection networks (written by me, but based on ideas from many, many people; see the Acknowledgments, linked from John's post #32).

Author: Peter KrautzbergerFormat: MarkdownItex@Andrew #37 I understand and share the worry. But I think it's equally likely that we could end up with Mathematical Critics (much like, say, literature critics), i.e., people who are renowned for there knowledge and experience at identifying good research. Add to that the study [I linked to in #35](http://blogs.lse.ac.uk/impactofsocialsciences/2012/02/09/more-tweets-more-citations/) -- tweets may be able to predict citations. That's not "Tim Gowers's tweets predict citations", it's the "wisdom of crowds". For a different experiment check [Crowdometer](http://crowdometer.org/).
@Alexander Chernov #40 second that! (In fact, see #7) -- wordpress can solve a lot of these problems out of the box.

@Andrew #37 I understand and share the worry. But I think it’s equally likely that we could end up with Mathematical Critics (much like, say, literature critics), i.e., people who are renowned for there knowledge and experience at identifying good research. Add to that the study I linked to in #35 – tweets may be able to predict citations. That’s not “Tim Gowers’s tweets predict citations”, it’s the “wisdom of crowds”. For a different experiment check Crowdometer.

@Alexander Chernov #40 second that! (In fact, see #7) – wordpress can solve a lot of these problems out of the box.

Author: zskodaFormat: MarkdownItex44 With many proposals involving wordpress -- what is the gurantee that the wordpress will stay free ? This is uninformed but serious question: I know there is a commercial owner of the wordpress web site, there are free user accounts, and there is some software there and some related software (and add-ons) in circulation. I do not know which portion of the software is transferrable to other sites retaining full functionality, how much of it is open source and what are ultimately the licences involved. I am sure some people here know -- please let us know.

44 With many proposals involving wordpress – what is the gurantee that the wordpress will stay free ? This is uninformed but serious question: I know there is a commercial owner of the wordpress web site, there are free user accounts, and there is some software there and some related software (and add-ons) in circulation. I do not know which portion of the software is transferrable to other sites retaining full functionality, how much of it is open source and what are ultimately the licences involved. I am sure some people here know – please let us know.

Author: Yemon ChoiFormat: MarkdownItexPeter at 44: isn't the danger that "good research" becomes increasingly defined as "what those Big Name good critics like?" There are also people doing mathematics in systems or societies where the kind of openness taken for granted by this kind of model *is not so easy*, and the loudest voices may not be the wisest.
I also share Alexander's concerns at 38.

Peter at 44: isn’t the danger that “good research” becomes increasingly defined as “what those Big Name good critics like?” There are also people doing mathematics in systems or societies where the kind of openness taken for granted by this kind of model is not so easy, and the loudest voices may not be the wisest.

Author: Peter KrautzbergerFormat: MarkdownItex@Yemon yes, there's a danger. But how is that different from today? Editorial boards are precisely "Big Name critics". Yet journals are so narrow in their design, that they will never say "hey, I just read this fantastic expository paper about this obscure proof that nobody understood; there's nothing new, but it's the best paper I read all month".
As I tried to argue with the study about tweets predicting citations, the "wisdom of crowds" approach is likely the most fruitful (and most democratic) and specifically eliminates this problem of "Prof BigWig said so".
@szkoda wordpress is open source. So yes, it will always be free. The distinction is between wordpress.com, a free hosting service, and wordpress.org the underlying software. But wordpress.com also lets you export anything from your site there and import into your own installation. I know a little bit about this since we've started [boolesrings.org](http://boolesrings.org) last summer, an experiment towards a wordpress hosting service specifically for mathematicians. We are working on a new site that would be open to everybody, but above all we focus on a decentralized approach, i.e., identify the open source tools that mathematicians might need for the purpose of their academic homepages (from citations management to course management). If you want to look at some of its potential, you might want to check out [Joel David Hamkin's site](http://boolesrings.org/hamkins)

@Yemon yes, there’s a danger. But how is that different from today? Editorial boards are precisely “Big Name critics”. Yet journals are so narrow in their design, that they will never say “hey, I just read this fantastic expository paper about this obscure proof that nobody understood; there’s nothing new, but it’s the best paper I read all month”.

As I tried to argue with the study about tweets predicting citations, the “wisdom of crowds” approach is likely the most fruitful (and most democratic) and specifically eliminates this problem of “Prof BigWig said so”.

@szkoda wordpress is open source. So yes, it will always be free. The distinction is between wordpress.com, a free hosting service, and wordpress.org the underlying software. But wordpress.com also lets you export anything from your site there and import into your own installation. I know a little bit about this since we’ve started boolesrings.org last summer, an experiment towards a wordpress hosting service specifically for mathematicians. We are working on a new site that would be open to everybody, but above all we focus on a decentralized approach, i.e., identify the open source tools that mathematicians might need for the purpose of their academic homepages (from citations management to course management). If you want to look at some of its potential, you might want to check out Joel David Hamkin’s site

Author: zskodaFormat: MarkdownItexTo 46, let me give just a small unimportant comment: even Grothendieck was at times unhappy with such a great mathematician as Weil surpressing the unconsent voices within such an open and small circle like Bourbaki was (according to his memoirs). In current refereeing process editors control that even the strong opinions of experts will be taken into account mostly if they are concrete and well pronounced. It is not always good that the same expert makes an argument (recommendation) and the decision: the two step process with different subjects involved makes the personal opinions having less weight.
47: thanks

To 46, let me give just a small unimportant comment: even Grothendieck was at times unhappy with such a great mathematician as Weil surpressing the unconsent voices within such an open and small circle like Bourbaki was (according to his memoirs). In current refereeing process editors control that even the strong opinions of experts will be taken into account mostly if they are concrete and well pronounced. It is not always good that the same expert makes an argument (recommendation) and the decision: the two step process with different subjects involved makes the personal opinions having less weight.

Author: Yemon ChoiFormat: MarkdownItexPeter: which crowds? I really think there are contingent factors to do with location and politics that are not always accounted for, at least in the corners of maths where I send my working time. You want to know how big a crowd care about the cohomology of Banach algebras *and* know what they are talking about? I could probably have them over for dinner in one go...
And actually I trust journal boards or handling editors *more* than the small world of online big name bloggers. I don't want to have to compete for the *attention* of e.g. Gowers, Tao, Kalai, Baez, Cameron, Ghoussoub, Hamkins, Laba, Krautzberger... ;) I just could do with the grudging *approval* of e.g. Blecher, Davidson, Elliott, Haagerup, Helemskii, Lau, Meyer, Rochberg, Smith, Toms, Voiculescu, or whoever, that my current effort is OK even if not of great "noncommutative spacetime!" interest.

Peter: which crowds? I really think there are contingent factors to do with location and politics that are not always accounted for, at least in the corners of maths where I send my working time. You want to know how big a crowd care about the cohomology of Banach algebras and know what they are talking about? I could probably have them over for dinner in one go…

And actually I trust journal boards or handling editors more than the small world of online big name bloggers. I don’t want to have to compete for the attention of e.g. Gowers, Tao, Kalai, Baez, Cameron, Ghoussoub, Hamkins, Laba, Krautzberger… ;) I just could do with the grudging approval of e.g. Blecher, Davidson, Elliott, Haagerup, Helemskii, Lau, Meyer, Rochberg, Smith, Toms, Voiculescu, or whoever, that my current effort is OK even if not of great “noncommutative spacetime!” interest.