A question people often ask about Kafka is whether it is okay to use it for longer term storage. The short answer is "yes". | Continue reading
With Apache Kafka and event streaming tools, developers and data engineers can access real-time data and perform streaming ETL using KSQL and Kafka Connect. | Continue reading
Kafka Connect, MQTT Proxy, and REST Proxy enable event streaming at scale through real-time IoT data integration and data processing. | Continue reading
In this tutorial, learn to set up Spring Boot in Pivotal Application Service (PAS), connect Kafka to PAS applications, and configure your PAS and PKS configurations so they work with Kafka. | Continue reading
In this tutorial, set up a Kafka cluster on Enterprise Pivotal Container Service with Confluent Operator to deploy and run Confluent Platform on Kubernetes. | Continue reading
Scrapinghub’s AutoExtract uses AI to deliver next-generation web scraping capabilities. Learn how Confluent Cloud let them deploy and run Kafka at scale. | Continue reading
Confluent Cloud, our fully managed Apache Kafka service, is now serverless on GCP, Azure, and AWS. Get started with Confluent Cloud for free. | Continue reading
With Confluent Platform 5.4-preview, operators now have a centralized location to enforce data format correctness and Schema Validation on write. | Continue reading
Use SQL to connect Rockset and Apache Kafka for ingesting data streams, joining datasets, and creating a real-time dashboard for streaming analytics. | Continue reading
Businesses aren't just using more software—they are increasingly defined in software. Apache Kafka and the event streaming platform will be at the heart of the modern software-defined company. | Continue reading
Learn how load balancing works, the challenges of existing load balancing protocols, and how incremental cooperative rebalancing allows large-scale deployment of Kafka clients. | Continue reading
Kafka Summit San Francisco is almost here! Here are some tips on making the most of this event, choosing tracks, learning about Apache Kafka and stream processing, and networking with the Kafka community. | Continue reading
Unlocking the benefits of event streaming can be painful due to infrastructure challenges. We’ll cover what to look for in a complete, fully managed Apache Kafka as a service for multi-cloud or serverless data processing. | Continue reading
Tim Berglund, senior director of developer experience at Confluent, discusses real-time ETL processes, data pipelines, and the need for an event streaming platform. | Continue reading
Static membership enhances the rebalance protocol, reducing any downtime caused by rebalances for Kafka client implementations, such as the Kafka consumer, Kafka Connect, and Kafka Streams. | Continue reading
As Confluent turns five, Senior Director of Developer Experience Tim Berglund reflects on our journey and what’s most exciting about being in the Kafka community. | Continue reading
Learn about traditional event sourcing vs. derivative event sourcing, in which the latter derives events from something that has been observed. This differs from the more common practice of emitting events directly from the service where the event took place. | Continue reading
Follow this tutorial to enable Schema Registry and Avro serialization format in Spring Boot applications both on-premises and in Confluent Cloud. | Continue reading
Visualize Kafka Streams with Neo4j by taking any data, turning it into a graph, leveraging graph processing, and piping the results back to Apache Kafka, adding visualizations to your event streaming applications. | Continue reading
We are excited to announce the release of Confluent Cloud Schema Registry in general availability, available in Confluent Cloud, our fully managed event streaming service based on Apache Kafka. | Continue reading
This article revisits the guarantees provided by transactions in relational database systems and how to achieve them using Kafka as a datastore. Learn how such an architecture can be extended, complete with an example. | Continue reading
The revised Confluent Verified Integrations Program has simplified verification requirements, streamlined the verification process, and updated partner-facing documentation, making it easier and faster to build connectors. | Continue reading
With the release of Apache Kafka 2.3 and Confluent Platform 5.3 came several advancements to Kafka Connect—particularly the introduction of Incremental Cooperative Rebalancing and changes in logging, including REST improvements, the ability to set `client.id`, and connector-level … | Continue reading
Get started with Secret Protection, end-to-end security, and encryption—now available in Confluent Platform, extending the security capabilities for Kafka brokers and Kafka Connect. | Continue reading
Here are 10 reasons why you can’t miss Kafka Summit San Francisco. Yes, the other definition of event sourcing. | Continue reading
We’re excited to announce Kafka Tutorials! Learn about Apache Kafka, real-time data stream processing, KSQL, and more with complete code examples and guides. | Continue reading
This tutorial helps you quickly bootstrap your KSQL UDF and UDAF projects by building your own custom KSQL functions using the Maven archetype available in Confluent Platform 5.3.0. | Continue reading
One aspect of the Kafka Streams framework that is less talked about is its ability to store local state derived from stream processing. We at Imperva took advantage of this ability to build shared state microservices that serve as fault-tolerant, highly available single sources o … | Continue reading
Confluent Platform 5.3 delivers enhancements that will help you automate Apache Kafka operations for production environments using Confluent Operator and Ansible playbooks. It also allows you to simplify management across user interfaces, and secure the platform end to end using … | Continue reading
How is data flowing through my distributed system? What if Jaeger goes down? Jaeger does a fantastic job of tracing data as it flows through a distributed system, but adding a layer of Apache Kafka in front of it also gives you fault tolerance, storage, and replayability. | Continue reading
Our flagship Kafka Summit 2019 event is almost here! Gwen Shapira shares all about her top picks on the agenda and how to get a 30% discount. | Continue reading
Build your own burglar alert system to alert you before the actual event takes place by using network-connected cameras and analyzing the camera images with Apache Kafka, Kafka Streams, and TensorFlow. You can use Kafka Connect to load and transform your images before sending the … | Continue reading
Here in part 3, we'll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. | Continue reading
From a data perspective, the World Cup represents an interesting source of information. The idea in this blog post is to mix information coming from two distinct channels: the RSS feeds of sport-related newspapers and Twitter feeds of the FIFA Women’s World Cup. The goal will be … | Continue reading
You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise, they’ll try to connect to the internal host address—and if that’s not reachable, then probl … | Continue reading
The Confluent clients for Apache Kafka have passed a major milestone—the release of version 1.0. Leading up to the 1.0 release, we've put a lot of effort into the .NET client in particular. The API we've arrived at contains a bunch of new features and major improvements. | Continue reading
Kai Waehner discusses why Apache Kafka became the de facto standard and backbone for microservice architectures—not just replacing other traditional middleware but also building the microservices themselves using domain-driven design and Kafka-native APIs like Kafka Streams, KSQL … | Continue reading
It’s official: Apache Kafka® 2.3 has been released! Here is a selection of some of the most interesting and important features we added in the new release, covering Core Kafka, Kafka Connect, and Kafka Streams KIPs. | Continue reading
It’s easy to use Apache Kafka and Kafka Connect to scale your search infrastructure by connecting different source applications, databases, and your search engine. This solution uses a single technology stack to create one uniform approach that helps your project integrate differ … | Continue reading
Each night, large-scale astronomical telescope surveys detect millions of changing objects in the sky and need to stream results to scientists for time-sensitive, complementary follow-up observations. This data pipeline is a great example of a use case for Apache Kafka. Kafka has … | Continue reading
Using Kubernetes, Helm and Confluent Cloud together is a powerful combination for quickly getting up and running with Apache Kafka. This blog post focuses on the specific Helm Chart configuration needed to connect the Confluent Platform components to your Kafka cluster running in … | Continue reading
TokenAnalyst uses Apache Kafka for ingestion of blockchain data—which is directly pushed from their cluster of Bitcoin and Ethereum nodes—to different streams of transformation and loading processes. | Continue reading
Part 3 of the Spring for Apache Kafka Deep Dive blog series explains how Spring Cloud Data Flow helps you gain developer productivity and manage Apache-Kafka-based event streaming application development. | Continue reading
Red Pill Analytics shares the challenges they faced in developing, building, and deploying the KSQL portion of their application, and how they used Gradle to address them. | Continue reading
Common types of operational mistakes made in self-managing one's own Schema Registry are bad designs, inconsistent configurations, and operational faux pas. Here's how to avoid them. | Continue reading
Kafka Summit London 2019 session videos are available! If you were there, you know what a great time it was, and you know that you had to make sometimes-agonizing decisions about which sessions to attend and which to miss. Well, now you can make all those tradeoffs right by watch … | Continue reading
We’ll share an emerging machine-to-machine (M2M) architecture pattern in which MQTT, Apache Kafka® and Scylla all work together to provide an end-to-end IoT solution. We’ll also provide demo code so you can try it out for yourself. | Continue reading