Newslurp

<< Stories

Build Systems, Not Heroes 🦸, NextGen AI Workloads 🪄, Reducing Log Sizes 📃

TLDR DevOps <dan@tldrnewsletter.com>

October 16, 11:07 am

TLDR DevOps
Scientists have achieved groundbreaking data transfer speeds of 301 Tbps using fiber-optic technology, vastly surpassing conventional broadband speeds ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌  ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 

TLDR

Together With Chronosphere

TLDR DevOps 2024-10-16

Video: Telemetry pipelines and log transformation in practice (Sponsor)

Log data has surged by 250% in just the past year. With sources multiplying, formats diversifying, and the number of destinations increasing, managing this data is becoming frustratingly expensive and complex.

In this recorded session, Chronosphere's experts show how to use a Telemetry Pipeline to make log data management more manageable — including:

  • Common processing rules and how to apply them to your log data.
  • Best practices for excluding noisy logs from your observability and SIEM tools.
  • How to pre-process data in different ways based on where it's being routed.
  • Converting noisy logs to metrics for faster alerting

Watch the demo on-demand

📱

News & Trends

Fiber-optic data transfer speeds hit a rapid 301 Tbps — 1.2 million times faster than your home broadband connection (4 minute read)

Scientists have achieved groundbreaking data transfer speeds of 301 Tbps using fiber-optic technology, vastly surpassing conventional broadband speeds, by utilizing the previously untapped "E-band" in the electromagnetic spectrum with custom-built devices.
Amazon EC2 now supports Optimize CPUs post instance launch (1 minute read)

Amazon EC2 now allows customers to modify an instance's CPU options after launch. You can now modify the number of vCPUs and/or disable the hyperthreading of a stopped EC2 instance to save on vCPU-based licensing costs.
🚀

Opinions & Tutorials

The Istio Service Mesh for People Who Have Stuff to Do (7 minute read)

This developer shares his experience contributing to Istio, an open-source service mesh that simplifies communication, security, and observability in microservices. He highlights Istio's key features, including traffic management, mutual TLS for secure communication, and built-in observability tools, emphasizing its role in enhancing the resilience and management of complex distributed systems.
How Uber Reduced Their Log Size By 99% (5 minute read)

Uber successfully reduced its log storage size by 99% using a custom tool called Compressed Log Processing, which compresses unstructured logs significantly while enabling efficient searches. By implementing a two-phase compression strategy, Uber increased its log retention from three days to one month and managed to compress 5.38PB of logs down to 31.4TB.
Airflow on Kubernetes: A Platform Engineer's Journey (3 minute read)

This blog post shares insights from setting up Apache Airflow on Kubernetes, focusing on high-scale automation, overcoming integration challenges, and leveraging robust infrastructure for scalable data pipeline management.
🧑‍💻

Resources & Tools

What if your runbooks were actually runnable? (Sponsor)

Runme replaces docs and scripts with interactive Notebooks (think Jupyter) and TUIs — so you can interact with cloud-native tools and cloud providers/infra inside your workflows, test your Notebooks in CI/CD, and add code samples that actually work to your documentation. Get started at runme.dev (free and open source)
Omniparse (GitHub Repo)

OmniParse is a powerful platform designed to transform any unstructured data into structured, actionable information optimized for GenAI applications. It supports a wide range of file types and formats while providing tools for table and image extraction, transcription, and web page crawling, all easily deployable via Docker and Colab.
exo (GitHub Repo)

Run your own AI cluster at home using everyday devices with exo, an innovative and experimental software by exo labs that allows any device, from iPhones to Linux laptops, to function as a powerful GPU for AI models.
🎁

Miscellaneous

Powering the Next Generation of AI Workloads on Amazon EKS with Anyscale (2 minute read)

Anyscale has launched RayTurbo, an optimized version of the Ray framework designed to enhance AI workloads on Amazon EKS with faster data processing and reduced node requirements. This integration allows customers to efficiently manage AI applications on AWS and leverage the benefits of Kubernetes for scalability and performance.
Deconstructing Security Monitoring Antipatterns (7 minute read)

This article discusses common security monitoring antipatterns, or practices that lead to inefficiencies or mistakes. It highlights issues such as relying solely on data collection without detection, using too many tools unnecessarily, and the misconception that purchasing more tools inherently improves security.
⚡

Quick Links

DevOps Implementation Guide [Plan, Strategy, & Steps] (14 minute read)

The guide outlines the key steps, tools, and best practices for implementing a successful DevOps strategy, starting with clear planning and stakeholder consultation, and includes a roadmap for measuring success.
Should you use uv's managed Python in production? (5 minute read)

The UV tool streamlines Python setup by installing versions as needed without requiring Python pre-installed, but its use in production raises security and compatibility concerns that require careful evaluation.
Build Systems, Not Heroes (5 minute read)

Enterprise programming emphasizes system-based approaches over individual skills to manage complexity and ensure scalability, reliability, and timely delivery.

Love TLDR? Tell your friends and get rewards!

Share your referral link below with friends to get free TLDR swag!
https://refer.tldr.tech/fc7198a6/10
Track your referrals here.

Want to advertise in TLDR? 📰

If your company is interested in reaching an audience of devops professionals and decision makers, you may want to advertise with us.

If you have any comments or feedback, just respond to this email!

Thanks for reading,
Kunal Desai & Martin Hauskrecht


If you don't want to receive future editions of TLDR DevOps, please unsubscribe from TLDR DevOps or manage all of your TLDR newsletter subscriptions.