Msg loss is weighed average over all servers, excluding those where sequence numbers are broken (weighed = taking into account message volume)Other causes of data loss than UDP msg loss are not covered by this report (e.g. total outage for several hours)

UDP message loss: average gap between sequence numbers per squid (should be close to 1000 in 1:1000 sampled log)

For calculation of message loss percentage from average gap between sequence numbers see bottom of page

api

api

mobile

mobile

mobile

ssl

ssl

ssl

text

text

text

text

text

upload

upload

upload

upload

upload

upload

upload

varnish

role?

RIP1

RIP2

RIP3

RIP5

eqiad

pmpta

eqiad

esams

pmpta

eqiad

esams

pmpta

eqiad

esams

esams

knams

pmpta

eqiad

eqiad

esams

esams

esams

knams

pmpta

eqiad

loc

pmpta

knams

pmpta

loc

month

msg loss

cp1001-1005

sq33-36

cp1041-1060

cp3011-3014

cp1041-1044

ssl1001-1009

ssl3001-3004

ssl1-4

cp1006-1020

amssq31-46

knsq23-29

knsq23-29

sq37-78

cp1021-1064

dysprosium

amssq47-62

cp3003-3010

knsq16-22

knsq16-22

sq41-86

cp1037-1068

?.cp4001-4020

sq31-147

knsq1-29

mobile1-35

?.maerlant

2010-04

18.1%

1222

1226

1221

1225

1220

1220

1220

2010-05

23.4%

1290

1323

1302

1294

1322

1301

1292

1288

1314

2010-06

22.6%

1292

2234

2253

2216

2226

2246

2222

2223

2216

2010-07

24.4%

1455

6166

1320

6336

4062

2307

2507

3792

2119

2010-08

1.4%

1004

1007

1008

1005

1017

1006

1005

1024

1320

2010-09

1.3%

1013

1016

1016

1016

1017

1006

1005

1027

1058

2010-10

0.8%

1011

1007

1007

1006

1014

1006

1006

1016

1044

2010-11

2.3%

1039

1022

1021

1023

1032

1022

1021

1033

1049

1004

2010-12

0.7%

1018

1004

1004

1006

1012

1004

1004

1016

1039

1008

2011-01

0.9%

996

1005

1005

1011

1017

1006

1004

1017

1016

1009

2011-02

0.2%

976

1001

999

1000

1007

1001

1004

1000

1002

1006

2011-03

0.3%

970

1002

999

1001

1009

1002

1002

997

1002

1009

2011-04

0.3%

968

1003

1000

1003

1004

1003

1004

995

1003

1008

2011-05

0.5%

964

1003

1000

1006

1002

1003

1009

993

1003

1008

2011-06

18.8%

1278

1235

1269

1226

1246

1253

1214

1240

1231

1240

2011-07

0.6%

971

1004

1000

1006

1004

1005

1010

996

1005

1009

237096

2011-08

6.3%

1052

1063

1062

1070

1063

1063

1071

1054

1065

1080

302258

2011-09

17.7%

1328

1209

1207

1221

1208

1209

1219

1202

1210

1260

406349

2011-10

6.2%

1342

1209

21958

28563

1061

1063

1069

1062

1062

1066

1062

1060

2011-11

0.7%

1220

1019

29847

20005

23643

1005

1005

1010

1004

1004

1009

1003

1004

2011-12

0.4%

1200

26634063

23631

21049

19346

1002

1001

1009

1001

1002

1005

1000

1002

3544

2012-01

0.7%

1094

1184

1008

18341

10596

16153

1003

1004

1011

1003

1003

1010

1002

1002

8291

2012-02

1.3%

1020

1270

1021

12863

11167

11723

1009

1008

1008

1038

1008

1008

1011

1042

1157

16466

2012-03

1.0%

1020

1257

1019

11114

8310

6850

1016

1005

1006

1044

1196

1005

1005

1011

1066

22101

2012-04

4.9%

1002

1185

1246

28518

14158

10331

1002

1004

1003

1041

1516

1002

1002

1009

24366

2012-05

1.5%

1003

591

1001

54716

23612

13090

1001

1001

1001

1042

1168

1001

1002

1016

25579

2012-06

0.4%

1001

629

1014

37010

26109

14885

1001

1002

1001

1053

1196

1002

1001

1008

25497

2012-07

14.7%

1201

592

1010

1012

65623

57994

33362

1140

1091

1038

1054

1228

1156

1091

1244

25547

2012-08

3.2%

1194

563

1020

1019

95813

84078

53305

1212

1007

1007

1056

1251

1006

1007

1008

2012-09

0.4%

1001

613

1013

1015

126317

113675

70828

1000

1001

1001

1058

1040

1001

1001

1006

2012-10

3.7%

1000

560

1016

1016

125048

152557

71502

1001

1001

1001

1049

1152

1001

90

1001

1085

2012-11

0.4%

1000

497

1018

1018

27267

102465

2041

1001

1001

1001

1059

1011

1000

1000

928

1227

2012-12

0.4%

1001

492

1020

255664806

12302

238049

4131

1000

1001

1001

1061

1012

1001

472

1001

1239

2013-01

0.6%

1004

492

1006

1004

7656

425056

2397

1005

1005

1005

1065

1005

1005

845

1019

1239

2013-02

0.0%

1001

562

1001

13602

8757

228542

2441

1001

1001

1002

1003

1057

1000

1001

978

1023

1022

1230

2013-03

0.3%

1003

547

1002

1003

11223

355509

2526

1002

1004

1005

1050

1003

1004

999

1027

1239

626

2013-04

1.2%

1001

456

1023

1001

1001

1001

1002

1051

1002

1000

1082

1002

1095

1227

678

2013-05

0.5%

1001

478

1001

1240

1380

103758

1133

1001

1001

1096

1056

1001

1000

1197

1001

1177

1218

826

2013-06

0.1%

1001

426

1010

868

1001

2594

124849

1069

1001

1001

1134

1057

1000

1000

1173

1000

1200

1202

344

2013-07

4.5%

1000

334

1047

1105

1211

1000

1000

1046

1156

1205

1189

1000

1232

988

835

2013-08

3.8%

1000

401

1000

1001

1000

1001

1032

1146

1223

1180

1000

1209

1135

2013-09

3.8%

1001

1002

1002

1001

1001

1041

1151

1330

1178

1001

1227

1173

2013-10

3.6%

1001

1001

1001

1001

1001

1051

1125

1216

1195

1000

1215

993

1031

2013-11

2.0%

1071

1002

998

1079

1058

1082

1001

1049

1002

1218

994

1136

2013-12

0.4%

1122

1001

989

1116

1082

1105

1000

1014

1000

1210

991

1224

2014-01

1.0%

1011

987

1012

1012

1002

1004

1242

2014-02

15.5%

1001

988

1003

1008

914

1390

1018

1002

994

6974

2014-03

0.5%

1001

981

1002

1006

853

1001

1015

1001

992

1008

Traffic volume: average events per squid per active hour in 1:1000 sampled log

api

api

mobile

mobile

mobile

ssl

ssl

ssl

text

text

text

text

text

upload

upload

upload

upload

upload

upload

upload

varnish

role?

RIP1

RIP2

RIP3

RIP5

eqiad

pmpta

eqiad

esams

pmpta

eqiad

esams

pmpta

eqiad

esams

esams

knams

pmpta

eqiad

eqiad

esams

esams

esams

knams

pmpta

eqiad

loc

pmpta

knams

pmpta

loc

month

cp1001-1005

sq33-36

cp1041-1060

cp3011-3014

cp1041-1044

ssl1001-1009

ssl3001-3004

ssl1-4

cp1006-1020

amssq31-46

knsq23-29

knsq23-29

sq37-78

cp1021-1064

dysprosium

amssq47-62

cp3003-3010

knsq16-22

knsq16-22

sq41-86

cp1037-1068

?.cp4001-4020

sq31-147

knsq1-29

mobile1-35

?.maerlant

2010-04

1064

2791

1568

2380

2202

1215

2377

2010-05

1079

2102

1364

1686

1198

1729

2142

1241

1327

2010-06

1224

1946

640

1648

1065

1011

1897

1233

660

2010-07

115

2318

763

3547

1198

1122

2276

1233

669

2010-08

1459

1171

1092

2140

1444

1488

2461

1343

785

2010-09

1464

1311

1313

2341

1548

1617

2458

1425

985

2010-10

1376

1679

1676

2567

1463

1536

2499

1542

1100

2010-11

1552

1878

1887

2781

1612

1693

2538

1649

1238

292

2010-12

1487

1877

1894

2642

1559

1645

2255

1544

1215

305

2011-01

1579

2113

2110

2697

1825

1915

2330

1654

1571

365

2011-02

1343

1844

1842

2589

1796

1885

2533

1701

1610

375

2011-03

1519

1451

1443

2349

1874

1968

2667

1766

1620

343

2011-04

1383

1253

1247

2141

1782

1788

2581

1654

1460

397

2011-05

1431

1334

1326

2129

1820

1821

2532

1700

1492

411

2011-06

1219

1274

1263

2118

1807

1806

2376

1583

1471

307

2011-07

1391

1206

1189

2060

1640

1638

2311

1574

1346

345

8

2011-08

1285

1115

1110

2213

1503

1495

2407

1701

1225

342

68

2011-09

7

1138

1131

2142

1632

1630

2382

1669

1361

269

80

2011-10

4

1059

22

54

1399

1394

2494

2002

2000

2938

1923

1668

2011-11

4

1379

196

103

105

1579

1576

2670

2384

2378

3275

2084

1984

2011-12

4

1689

78

97

95

1505

1489

2448

2369

2362

2935

1946

1972

5

2012-01

9

4

1052

27

47

55

1547

1493

2431

2632

2622

3133

1853

2184

9

2012-02

2014

4

900

47

50

46

2183

1431

1405

734

2849

2766

3469

581

413

12

2012-03

2279

4

968

77

67

48

2284

1427

1313

27

164

2860

2695

3432

22

11

2012-04

2268

4

1139

167

98

65

2278

1355

1455

48

2087

3026

2847

2592

6

2012-05

2675

1

1544

187

156

86

2696

1665

1657

28

1458

3092

3029

2976

5

2012-06

2293

1

1559

156

158

96

2305

1367

1369

21

2

2855

2867

3438

5

2012-07

2236

1

1236

1257

131

163

77

2249

1326

1323

20

2

2525

2515

3307

4

2012-08

2244

1

1277

1281

125

182

80

2249

1272

1270

114

3

2562

2551

3620

2012-09

2488

1

1349

1339

149

225

75

2494

1519

1517

20

749

3082

3040

3686

2012-10

2633

1

1369

1367

180

262

68

2639

1636

1632

41

2197

3338

1

3224

2878

2012-11

2719

1

1263

1267

407

627

4

2718

1805

1804

19

6433

4203

4836

2833

4

2012-12

2449

1

1425

1711

517

1190

4

2450

1677

1678

18

5785

3487

1

3213

3

2013-01

2797

1

1501

1506

749

1487

7

2796

1998

1999

24

6609

4056

7871

3174

10

2013-02

2894

1

1903

2065

859

1287

5

2895

1908

1904

1906

19

6766

2909

10526

2612

2447

3

2013-03

2822

1

2155

2153

1065

1608

6

2822

1842

1844

23

7152

2998

11174

2548

3

1

2013-04

2994

1

2074

2400

2992

1871

1798

21

4983

37262

1186

9779

1027

2

1

2013-05

2828

1

1726

1724

4958

4235

8

2828

2180

517

18

4621

37135

2

10539

2

2

1076

2013-06

2782

1

1358

3

1535

5426

4314

9

2784

2222

8

19

4381

35820

2

9407

2

2

1

2013-07

3295

1

2363

1221

26

3295

2357

25

4727

454

2

8309

2

5

1

2013-08

3547

1

2433

2361

3545

2706

36

4493

2

2

8141

2

695

2013-09

3291

2498

2685

3290

3224

25

4663

2

2

10888

2

672

2013-10

2485

2335

2701

2485

3533

28

5964

2

2

11081

2

3571

428

2013-11

655

2134

2741

655

772

16

12391

3517

11526

2

6869

1028

2013-12

9

2430

2861

9

13

10

12063

3510

10734

2

7262

3

2014-01

2746

3292

12473

3706

11994

7073

3

2014-02

1840

3177

523

2406

1

10521

3676

12222

5247

1631

2014-03

1674

3195

515

2504

1

9380

3584

11987

4700

1778

Calculation of msg loss percentage:

Assume total volume of messages per hour per server is vWe expect average gap between messages per server is 1000
We see avarage gap is g