In this episode, Chris chats to Anurag Gupta, co-founder at Calyptia about the requirements for system observability, specifically using Fluentd and Fluent Bit. Fluentd is a data collection tool, written in Ruby and C and used to collect data from multiple system types. Fluent Bit is a log collection tool written in C with an emphasis on efficiently collecting data from embedded Linux systems, including the IoT market. In this discussion Anurag explains the background to observability versus monitoring and how modern SaaS applications also need data to be collected and analysed. Data collection needs to be secure, filtered and obfuscated where necessary, to ensure PII or other sensitive data isn’t leaked inadvertently. Calyptia manages Fluentd and Fluent Bit deployments, providing a SaaS portal to establish rules and policies on data sources, filtering and storage.

For more information, check out https://calyptia.com/ and https://www.fluentd.org/.

• 00:00:00 – Intros
• 00:02:40 – Microsoft loves Linux! (Now)
• 00:04:13 – What is Fluentd and Fluent Bit?
• 00:06:23 – Modern app complexity requires a greater depth – observability
• 00:08:30 – A filter system is required to highlight important metrics
• 00:09:16 – What data are customers typically collecting?
• 00:11:30 – SaaS applications also have observability requirements
• 00:13:58 – Why Fluentd and Fluent Bit?
• 00:16:08 – How much overhead does observability add?
• 00:19:12 – Fluentd/Fluent Bit are light touch solutions
• 00:19:40 – How are policies applied to data collection?
• 00:22:00 – Calyptia adds on enterprise-class management functionality
• 00:25:15 – Log data needs to be treated securely and safely, with “zero trust”
• 00:28:45 – Are there any gaps in log data collection?
• 00:32:15 – Calyptia supports Lua rules
• 00:33:30 – How is collected data visualised?
• 00:39:00 – Wrap Up

Copyright (c) 2016-2023 Unpacked Network. No reproduction or re-use without permission. Podcast episode #hg5t.