When to believe new physics results

Here’s a brief summary giving my understanding of how physics results are determined in collaborations of hundreds or thousands of physicists such as the experiments at the LHC and when to believe a new physics effect has been seen.

Someone within the collaboration from an institute (university, lab, etc.) has an idea for an analysis.

A few people within the institute do some preliminary studies on existing experimental and/or simulated data to see if the analysis is feasible.

If they decide it is, they make a presentation to a larger working group which manages analyses relevant to the physics of interest. At ATLAS these groups include the Higgs group, the SUSY group and the Standard Model group among others.

In very large collaborations chances are someone else at another institute wants to study the same analysis, in which case these institutes join together to form a sub-working group for the analysis.

This is where the bulk of the analysis work is done. The analysis group must convince themselves that their work is robust. They must regularly liaise with the working group and exerts in the reconstruction of the physics objects they are interested in, for example photons.

All of the analysis work must be fully documented, this becomes the official internal documentation for the analysis and can stretch to hundreds of pages. If anyone requests a cross-check of any kind it must be performed and added to the documentation.

Once the sub-working group can convince the working group of the quality of their analysis, ie. that it is in a complete form, that relevant cross-checks have been made and that it is fully documented then it can considered for publication.

A group of around 3 senior people within the collaboration are assigned to the analysis as a publication reviewers. They read the documentation in full. Any further cross-checks required by these reviewers must be performed and documented.

Together with the reviewers the analysts draft a document for publication.

The publication draft is circulated to the whole collaboration. A review period of about a week is opened during which time anyone active within the collaboration with access to the internal documentation can request further cross-checks, clarifications, etc. regarding the analysis.

The publication reviewers work with the analysts in responding to all concerns raised during the collaboration review period. If significant modifications need to be made then the collaboration review repeats as necessary.

A final draft of the analysis publication is read by another senior member of the collaboration as a final reading. Any remaining concerns must be addressed.

Once all the above is satisfied the publication paper is submitted to a journal and uploaded on the arXiv server. It has completely satisfied the internal review of the collaboration. Any collaborators not satisfied with the analysis can withdraw their names from the paper.

The journal assigns the paper to 2-3 external referees who decide whether the analysis is appropriate for the journal and who may require further cross-checks and clarifications to be provided. Generally the referees will not have access to the internal documentation.

If the referees are satisfied the journal accepts the analysis paper for publication.

This is not an exact recipe and may vary by collaboration but the key message is the same: there are many levels of review which must be satisfied before an analysis represents a final result. It is also worth noting that a result published in a journal which claims a statistical significance below the magic 5 sigma level should be treated with caution until the same collaboration can provide more convincing results with more data or until a rival experiment/collaboration can check their own independent data and confirm the same result. For me only this final point, ie. analyses from independent collaborations which have passed through the above review process BOTH seeing the same new physics effect will convince me that it is there. I emphasise this as the CDF di-jet resonance currently causing excitement does not satisfy this requirement.

As a final comment, the rumours concerning a Higgs signal at ATLAS which have emerged in the last few days represent the content of a note which was written at stage 2 of the above list. It is highly unfortunate that someone within the collaboration thought it necessary to leak the details of an analysis at such an early stage when it is clearly very incomplete.

A full list of submitted publications from the ATLAS experiment is here. I have made an unofficial twitter stream here showing the same information which I update when I remember!

9 Responses

Good point, in experimental particle physics the collaboration press release usually comes with the arXiv submission and not with journal acceptance. This leaves room for red faces if the journal then rejects the paper. I forgot to mention that the CDF di-jet paper has only been submitted and not accepted for publication by PRL.

Your other point about journal papers being worng is also important. It’s why I think to be really convincing a new physics effect needs to be seen by independent analysts on independent data with an independent review process!

What I really wanted to show here was that the excitement about ATLAS Higgs data yesterday was from an analysis not even reviewed at any level within the collaboration let alone externally. It was extremely irresponsible of someone to leak it.

[…] Here's a brief summary giving my understanding of how physics results are determined in collaborations of hundreds or thousands of physicists such as the experiments at the LHC and when to believe a new physics effect has been seen. Someone within the collaboration from an institute (university, lab, etc.) has an idea for an analysis. A few people within the institute do some preliminary studies on existing experimental and/or simulated data to … Read More […]

[…] “Here’s a brief summary giving my understanding of how physics results are determined in collaborations of hundreds or thousands of physicists such as the experiments at the LHC and when to believe a new physics effect has been seen …” (more) […]

[…] or about 15 million GB per year. It takes a very long time to thoroughly analyse this data, with many rounds of review. Even then, because we’re talking about random events, you still have the chance of […]

The sad thing is that we outsiders have no way of knowing what stage in the process you were… which means that we have to rely on following decent physics blogs to make sure that we’re not being duped. It’s generally a good process, but seems rather heavy on bureaucracy.

That’s essentially the motivation for confidentiality before journal submission or a conference talk. It means that public results are a statement unambiguously endorsed by the whole collaboration. Those who blog rumours claim it’s good to generate publicity, but I don’t really agree because that publicity creates confusion about the reliability of results as we’ve just seen.