Presented by

  • Rafael Martinez Guerrero

    Rafael Martinez Guerrero
    @rafaelma_
    https://e-mc2.net/

    Rafael got his first computer in the 80', an Amstrad CPC 6128 with a Z80 processor and 128Kb of ram. Since then he has used and tried multiple different systems and vendors but his main operative system for the past 23 years has always been Linux. He has been a Linux user since 1996 and has had the opportunity and privilege to see and experience the evolution of Linux, its community and the open source movement during all these years. Rafael specializes in Linux system administration, PostgreSQL database administration, monitoring and capacity planning, automation, performance tuning, security hardening, high availability and disaster recovery. Rafael is currently working as a Chief Engineer for the Center for Information Technology at The University of Oslo. During the past few years he has been a member of the Department for Data Collection and Automation, working with system monitoring, data analytics, automation and trending. He is also the author of PgBackMan, a PostgreSQL backup manager, and Zabbix-CLI, both released as open source projects.

Abstract

Behind every security measure you take, you should have an information management system helping you take decisions. If you work with security, you need a way to collect, process, save and analyze huge amounts of data that should be used to control how your systems are behaving, find anomalies and evaluate the results of your actions. Have you ever wondered how to manage billions of logs and metrics from thousands of devices in your infrastructure? If you need high-availability and a resilient and stable system to process your data this is the tutorial for you. Based on the experience obtained in the past 4 years at the University of Oslo processing billions of logs a day from more than 15000 devices, this tutorial will give some inside information and many tips about how to achieve this with Linux and open source software. You will learn how to put together HAProxy, agents, Logstash, Elasticsearch and RabbitMQ to work at scale. You will also hear about the problems and pitfalls we have experienced during these years and what we learned from them.