[Webinar] Build Your GenAI Stack with Confluent and AWS | Register Now
Technology
Data Products, Data Contracts, and Change Data Capture
Change data capture is a popular method to connect database tables to data streams, but it comes with drawbacks. The next evolution of the CDC pattern, first-class data products, provide resilient pipelines that support both real-time and batch processing while isolating upstream systems...
Introducing Confluent Cloud Freight Clusters
Learn how the latest innovations in Kora enable us to introduce new Confluent Cloud Freight clusters, which can save you up to 90% at GBps+ scale. Confluent Cloud Freight clusters are now available in Early Access.
Contributing to Apache Kafka®: How to Write a KIP
Learn how to contribute to open source Apache Kafka by writing Kafka Improvement Proposals (KIPs) that solve problems and add features! Read on for real examples.
Yes, Virginia, You Really Do Need a Schema Registry
I ran into the schema-management problem while working with my second Hadoop customer. Until then, there was one true database and the database was responsible for managing schemas and pretty […]
Apache Kafka Hits 1.1 Trillion Messages Per Day – Joins the 4 Comma Club
I am very excited that LinkedIn’s deployment of Apache Kafka has surpassed 1.1 trillion (yes, trillion with a “t”, and 4 commas) messages per day. This is the largest deployment of Apache […]
Log Compaction | Highlights in the Kafka and Stream Processing Community | September 2015
September is the start of the fall conference season. Between Strata + Hadoop World New York and ApacheCon: Big Data Europe, there is plenty to keep us busy learning.
Distributed Consensus Reloaded: Apache ZooKeeper and Replication in Apache Kafka
This post was jointly written by Neha Narkhede, original co-creator of Apache Kafka, and Flavio Junqueira, co-creator of Apache ZooKeeper. Many distributed systems that we build and use currently rely on dependencies like […]
Log Compaction | Highlights in the Kafka and Stream Processing Community | August 2015
Welcome to the first edition of Log Compaction, a monthly digest of highlights in the Apache Kafka and stream processing community. Today’s edition are the highlights from July and early […]
Apache Kafka, Samza, and the Unix Philosophy of Distributed Data
One of the things I realised while doing research for my book is that contemporary software engineering still has a lot to learn from the 1970s. As we’re in such […]
Compression in Apache Kafka is now 34% faster
Apache Kafka is widely used to enable a number of data intensive operations from collecting log data for analysis to acting as a storage layer for large scale real-time stream […]
Getting started with Kafka in node.js with the Confluent REST Proxy
Previously, I posted about the Kafka REST Proxy from Confluent, which provides easy access to a Kafka cluster from any language. That post focused on the motivation, low-level examples, and […]
Making Apache Kafka Elastic With Apache Mesos
This post has been written in collaboration with Derrick Harris from Mesosphere and Joe Stein, a Kafka committer. For an updated version of this article, please see Apache Mesos, Apache Kafka and […]
Hands-Free Kafka Replication: A Lesson in Operational Simplicity
Building operational simplicity into distributed systems, especially for nuanced behaviors, is somewhat of an art and often best achieved after gathering production experience. Apache Kafka‘s popularity can be attributed in […]
The Value of Apache Kafka in Big Data Ecosystem
This is a repost of a recent article that I wrote for ODBMS. In the last few years, there has been significant growth in the adoption of Apache Kafka. Current […]
Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)
This is an edited transcript of a talk I gave at the Craft Conference 2015. The video and slides are also available.
How I Learned to Stop Worrying and Love the Schema
The rise in schema-free and document-oriented databases has led some to question the value and necessity of schemas. Schemas, in particular those following the relational model, can seem too restrictive, […]
Compatibility Testing For Apache Kafka
Testing is one of the hardest parts of building reliable distributed systems. Kafka has long had a set of system tests that cover distributed operation but this is an area […]