A scalable online monitoring system based on elasticsearch for distributed data acquisition in CMS /

Jean-Marc Andre; Ulf Behrens; James Branson; Philipp Brummer; Olivier Chaze; Sergio Cittolin; Diego Da Silva Gomes; Georgiana-Lavinia Darlea; Christian Deldicque; Zeynep Demiragli; Marc Dobson; Nicolas Doualot; Samim Erhan; Jonathan Richard Fulcher; Dominique Gigi; Maciej Gladki; Frank Glege; Guillelmo Gomez-Ceballos; Jeroen Hegeman; Andre Holzner; Mindaugas Janulis; Michael Lettrich; Audrius Mečionis; Frans Meijers; Emilio Meschi; Remigius K Mommsen; Srecko Morovic; Vivian O'Dell; Luciano Orsini; Ioannis Papakrivopoulos; Christoph Paus; Petia Petrova; Andrea Petrucci; Marco Pieri; Dinyar Rabady; Attila Racz; Valdas Rapševičius; Thomas Reis; Hannes Sakulin; Christoph Schwick; Dainius Šimelevičius; Mantas Stankevičius; Cristina Vazquez Velez; Michail Vougioukas; Christian Wernet; Petr Zejdl

doi:10.1051/epjconf/201921401048

Title	A scalable online monitoring system based on elasticsearch for distributed data acquisition in CMS /
Authors	Andre, Jean-Marc ; Behrens, Ulf ; Branson, James ; Brummer, Philipp ; Chaze, Olivier ; Cittolin, Sergio ; Gomes, Diego Da Silva ; Darlea, Georgiana-Lavinia ; Deldicque, Christian ; Demiragli, Zeynep ; Dobson, Marc ; Doualot, Nicolas ; Erhan, Samim ; Fulcher, Jonathan Richard ; Gigi, Dominique ; Gladki, Maciej ; Glege, Frank ; Gomez-Ceballos, Guillelmo ; Hegeman, Jeroen ; Holzner, Andre ; Janulis, Mindaugas ; Lettrich, Michael ; Mečionis, Audrius ; Meijers, Frans ; Meschi, Emilio ; Mommsen, Remigius K ; Morovic, Srecko ; O'Dell, Vivian ; Orsini, Luciano ; Papakrivopoulos, Ioannis ; Paus, Christoph ; Petrova, Petia ; Petrucci, Andrea ; Pieri, Marco ; Rabady, Dinyar ; Racz, Attila ; Rapševičius, Valdas ; Reis, Thomas ; Sakulin, Hannes ; Schwick, Christoph ; Šimelevičius, Dainius ; Stankevičius, Mantas ; Velez, Cristina Vazquez ; Vougioukas, Michail ; Wernet, Christian ; Zejdl, Petr
DOI	10.1051/epjconf/201921401048
Full Text
Is Part of	23rd International Conference on Computing in High Energy and Nuclear Physics (CHEP 2018), Sofia, Bulgaria, July 09-13, 2018.. Les Ulis : EDP Sciences. 2019, art. no. 01048, p. [1-8]
Abstract [eng]	The part of the CMS Data Acquisition (DAQ) system responsible for data readout and event building is a complex network of interdependent distributed applications. To ensure successful data taking, these programs have to be constantly monitored in order to facilitate the timeliness of necessary corrections in case of any deviation from specified behaviour. A large number of diverse monitoring data samples are periodically collected from multiple sources across the network. Monitoring data are kept in memory for online operations and optionally stored on disk for post-mortem analysis. We present a generic, reusable solution based on an open source NoSQL database, Elasticsearch, which is fully compatible and non-intrusive with respect to the existing system. The motivation is to benefit from an offthe-shelf software to facilitate the development, maintenance and support efforts. Elasticsearch provides failover and data redundancy capabilities as well as a programming language independent JSON-over-HTTP interface. The possibility of horizontal scaling matches the requirements of a DAQ monitoring system. The data load from all sources is balanced by redistribution over an Elasticsearch cluster that can be hosted on a computer cloud. In order to achieve the necessary robustness and to validate the scalability of the approach the above monitoring solution currently runs in parallel with an existing in-house developed DAQ monitoring system.
Published	Les Ulis : EDP Sciences
Type	Conference paper
Language	English
Publication date	2019
CC license

„A scalable online monitoring system based on elasticsearch for distributed data acquisition in CMS /“