Collisions (1900 Bunches)

Preamble: If you (after consulting with the shift leader) find these instructions are unclear, ambiguous, or contradictory, please ring the HLT doc to clarify. The HLT doc writes them and is responsible for ensuring that these instructions are correct and clear.

The Level-1 rate before deadtime ("FINOR" rate) should never exceed 100 kHz. Before switching into another column, check in the "Rate Preview" panel that its FINOR rate will not be too high. During Lumi Levelling, rates may vary considerably, so check the Rate Preview over a few lumisections. By default, the Rate Preview panel shows the next column (n+1) compared to the one you are running in (n). You can select another Preview column but remember that this will affect what everybody is seeing (this switching occurs in the hardware).

When in flat top, squeeze, and adjust, stay in column 0, which will give a ~few Hz each of stream physics, express and DQM. When in stable beams, move to the appropriate column based on the table below. Currently we expect this column to be 2. Note, at the start of the fill the luminosity can be higher than expected for a lumi section or two as the LHC finalises the leveling. So if the L1 rates are very high, dont panic, wait two lumi sections and see if the LHC has brought us down to the expected lumi of ~1.5E34. If after two lumi sections (~1min) the rate is still high, take appropriate action as below in "Chosing a column while leveling". If you have concerns over which column to use, please ring the HLT doc who will clarify.

During emittance scan only: it is not necessary to change columns to try to match the luminosity. Stay in the same column (appropriate for the lumi at the beginning of the scan) until the end of the scan.

Express is expected to be 60-100Hz during collisions.

How to check if a prescale is really the right one? (after switching based on lumi)

L1 rates of 95KHz are fine, in fact we do not care about the L1 rate, we care about deadtime. High L1 rates can cause high deadtime though, thats why we are so concerned about L1 rate.

If you see high deadtime and the pre-deadtime L1 rate <90KHz, there is an unrelated problem and the shift crew should understand its origin (and hopefully get it fixed).

As the fill progresses, please resist the urge to change to a higher column because you think the L1 and/or HLT rate is too low. The luminosity-based switch points of the prescale columns have been designed to target a certain average HLT output rate over the course of the fill, so that high rates at the beginning of the fill are balanced out by low rates at the end of the fill. Please stick to the table below unless there is a problem.

We expect 0.7 to 1.3KHz for the HLT. We aim to average roughly 1KHz over a fill.

If the rate has significantly jumped, check if its all due to one trigger or not. There should be no trigger with rate >300Hz which starts with "HLT_" (so not DST_ or Alca_)

Important Note: If the deadtime spikes to >50% for multiple (2+) lumisections this should be investigated immediately. First check if any individual L1 triggers have a pre-deadtime rate >50 kHz, next determine if these high rate triggers are associated with a specific sub-system (e.g. L1_ETM* / L1_SingleEG* --> ECAL, L1_SingleJet* --> HCAL). If the high rates are clearly related to a specific sub-system you should immediately contact the corresponding sub-system expert and notify them of the high deadtime as well as which triggers have increased rates. [In case of ECAL subsystem (—> high EG rates), if the DOC is not reachable, please RED RECYCLE immediately ECAL]
If there are high L1 / HLT rates and it is still unclear what is causing the high deadtime, please call either the L1 or HLT Docs.

Note: when we are running in columns 11 - 20, an additional 1kHz of rate will be added to the normal physics rate. So a streamPhysics rate of ~2kHz (0.7kHz-1.3kHz goes to 1.7kHz-2.3kHz)

Choosing a Column While Leveling:
We expect that you will be able to start in column 2, the 1.5E34 at the lumi leveling period and can select the columns as normal. If there is high deadtime (>5% and L1 pre-deadtime rate < 95kHz) its probably not trigger but you should identify and have the source fixed. If there is high deadtime (>5%) and L1 pre-time 95kHz - 120kHz, move to column 1 and assess why there is increased L1 rates (hot calo cell, muon issues etc), moving back to 2 once it is fixed. If you can not see a reason why there is increased L1 rates, please ring the HLT doc. If there is high deadtime (>5%) and > 120kHz L1 rate, its probably a hot tower or similar hardware issue, identify it, if you are unclear what is causing it, ring the L1 doc.

Currently there is another dimension to choosing the column while leveling. The MET PD is very large due to ECAL laser corrections introduced in Fill6012. This is slowly returning to normal as we continue to take data. See this plot : http://cern.ch/go/gKk7 which is a MET trigger rate for the first runs of Fills 6011,6012,6013 and 6014, it shows the rate decreasing with each fill. If the MET PD is greater > ~350 Hz, this presents a problem. As such after switching prescale columns please make sure that the MET PD rate remains < 350Hz. If the MET PD rate is above 350Hz, please ring the HLT doc immediately (day or night). You can see check the MET PD date from WBM with this link: https://cmswbm.cern.ch/cmsdb/servlet/DatasetSummary?RUN=305341&NAME=MET&OPTION=EVENTRATE, replace 305341 in the url with the current runnr.

In general the shift leader should verify with the L1 shifter that the preview pre-dead time trigger rate for the target column is below 95 kHz when switching columns.

Cosmics

The cosmics mode has prescale columns to fit your needs in case you need to do a test (e.g. with large random rate), or a problem crops up (e.g. noisy triggers). The rate of the Express stream should not exceed ~100-120 Hz during cosmics. Note the only triggers in express for cosmics are HLT_L1SingleMuOpen, HLT_L1SingleMuOpen_DT, HLT_L1SingleMuCosmics, HLT_Random. So any high express rate has to be either due to muons or randoms. In the case of high express rate, the first thing you should do is ask the DAQ shifter what rate of random triggers they have selected. If it is >600Hz, this is likely the reason for your high express rate and you should immediately switch column 1 or 2 depending on the random rate.

we have a known issue with the L1_SingleEG5, it is currently 2kHz often during cosmics due to some noise. Given this is no issue, it will not be fixed in the near future. So if in cosmics/circulating the L1_SingleEG5 rate is 2kHz, no need to call the ECAL Doc

Circulating

The circulating menu is essentially the same as the Cosmics menu (with the addition of the BptxOR path). The prescale columns are the same as described above for Cosmics. For nominal running use column 0.

The prescale columns are the same as described above for Cosmics. For nominal running use column 0. Please note that the rate of HLT_Random is expected to be high in this menu, which is a few hundred Hz.

High Rate Test

The high rate test key will not write output to deliver to Tier 0, however as an extra safety mechanism it's recommended to ask DAQ shifter to turn off transfer to Tier0. Even though there are no events written, the HLT will still produce files to monitor rates, hence you will get information on HLT rates in the control room. This also means that our rate monitoring script will notice the high rate and start to complain. Currently this warning can be ignored only in the high rates mode.

1300 Bunch Menu (Ramp-up after TS2)

To be used in case of emergency Call the HLT DOC if you feel the need to go to this column during stable beams

1

2.0E34

Lumi >1.9e34, Lumi<2.1e34

2.0e34 physics column Call the HLT DOC if you feel the need to go to this column during stable beams

2

1.8e34

Lumi >1.7e34, Lumi<1.9e34

1.8e34 physics column

3

1.6e34

Lumi >1.5e34, Lumi<1.7e34

1.6e34 physics column

4

1.4e34

Lumi >1.25e34, Lumi<1.5e34

1.4e34 physics column

5

1.2e34

Lumi >1.1e34, Lumi<1.25e34

1.2e34 physics column

6

1.0e34

Lumi >0.8e34, Lumi<1.1e34

1.0e34 physics column

7

7.5e33

Lumi >0.6e34, Lumi<0.8e34

7.5e33 physics column

8

5.0e33

Lumi >0.4e34, Lumi<0.6e34

5.0e33 physics column

9

2.5e33

Lumi <0.4e34

2.5e33 physics column

10

1.8e34 +ZB

Lumi >1.7e34, Lumi<1.9e34

Adds 1kHz ZeroBias

11

1.6e34 +ZB

Lumi >1.5e34, Lumi<1.7e34

Adds 1kHz ZeroBias

12

1.4e34 +ZB

Lumi >1.25e34, Lumi<1.5e34

Adds 1kHz ZeroBias

13

1.2e34 +ZB

Lumi >1.1e34, Lumi<1.25e34

Adds 1kHz ZeroBias

14

1.0e34 +ZB

Lumi >0.8e34, Lumi<1.1e34

Adds 1kHz ZeroBias

15

7.5e33 +ZB

Lumi >0.6e34, Lumi<0.8e34

Adds 1kHz ZeroBias

16

5.0e33 +ZB

Lumi >0.4e34, Lumi<0.6e34

Adds 1kHz ZeroBias

17

2.5e33 +ZB

Lumi <0.4e34

Adds 1kHz ZeroBias

600 Bunch Fill (Ramp-up after TS2)

The plan is to first have ~1 hr of head-on collisions during the beginning of this fill, during which time you should be using the normal collisions run mode (collisions2017). The prescale columns for this are the following:

To be used in case of emergency Call the HLT DOC if you feel the need to go to this column during stable beams

1

2.0E34

Lumi >1.9e34, Lumi<2.1e34

2.0e34 physics column Call the HLT DOC if you feel the need to go to this column during stable beams

2

1.8e34

Lumi >1.7e34, Lumi<1.9e34

1.8e34 physics column

3

1.6e34

Lumi >1.5e34, Lumi<1.7e34

1.6e34 physics column

4

1.4e34

Lumi >1.25e34, Lumi<1.5e34

1.4e34 physics column

5

1.2e34

Lumi >1.1e34, Lumi<1.25e34

1.2e34 physics column

6

1.0e34

Lumi >0.8e34, Lumi<1.1e34

1.0e34 physics column

7

7.5e33

Lumi >0.6e34, Lumi<0.8e34

7.5e33 physics column

8

5.0e33

Lumi >0.4e34, Lumi<0.6e34

5.0e33 physics column

9

2.5e33

Lumi <0.4e34

2.5e33 physics column

10

1.8e34 +ZB

Lumi >1.7e34, Lumi<1.9e34

Adds 3kHz ZeroBias

11

1.6e34 +ZB

Lumi >1.5e34, Lumi<1.7e34

Adds 3kHz ZeroBias

12

1.4e34 +ZB

Lumi >1.25e34, Lumi<1.5e34

Adds 3kHz ZeroBias

13

1.2e34 +ZB

Lumi >1.1e34, Lumi<1.25e34

Adds 3kHz ZeroBias

14

1.0e34 +ZB

Lumi >0.8e34, Lumi<1.1e34

Adds 3kHz ZeroBias

15

7.5e33 +ZB

Lumi >0.6e34, Lumi<0.8e34

Adds 3kHz ZeroBias

16

5.0e33 +ZB

Lumi >0.4e34, Lumi<0.6e34

Adds 3kHz ZeroBias

17

2.5e33 +ZB

Lumi <0.4e34

Adds 3kHz ZeroBias

Then, during the low-PU part of the same fill (after the beams have been separated), please switch to the "lowPU2017" run mode. The inst lumi for this part of the fill is supposed to be kept at/around 6e31, but may fluctuate. Please start in the 6e31 column, and move to a different one if the lumi is significantly different from 6e31. The prescale columns are: