Posts

As Kubernetes has become the standard for deploying and managing containerized applications, testing has also adapted to this environment. Ensuring the reliability and performance of your application’s endpoints is crucial, especially in cloud-native deployments. A powerful combination to achieve this is using K6 for load and performance testing and Helm with its helm test hooks to integrate testing into the Kubernetes deployment lifecycle. In this blog post, we’ll explore how you can use K6 for endpoint testing within Kubernetes using Helm’s test hooks. This setup allows you to test your application right after it has been deployed in your Kubernetes cluster, ensuring everything is running as expected before marking the deployment as successful. ...

In a stateful streaming application the state of an application is one of the most important part. In Apache Flink we have the possibility to kind of backup the state with a so called savepointing mechanism. From these savepoints you as a developer or operations manager are able to stop-and-resume, fork, or update your Flink jobs (more to read about savepoints in the docs). So it’s a great possibility to try out different implementations of a fork for a Flink application. ...

Recently I got the chance to help a customer moving from a Helm based deployment of Apache Kafka in Kubernetes to the Strimzi, which manages the deployment of Kafka in Kubernetes with an operator. In the final phase, I had to transfer the data to the newly deployed Apache Kafka. Setting the scope The customer is using compacted topics (cleanup.policy=compact and log.rentention.ms=-1) as the default configuration for all topics in Apache Kafka, so they’re using it somehow like a database where messages are produced with an unique ID and are nulled afterwards when it is required to delete a message. ...

Coming from Kafka-Streams continuous delivery (CD) is quite an easy task, and almost no effort has to be done compared to Apache Flink. Because the state of a Kafka-Streams application is stored in Kafka, and it can build up the state after a redeployment from so-called changelog topics, therefore Kafka-Streams is also bounded to have source and sink to Apache Kafka. Apache Flink on the other hand has the freedom to choose from a variety of source systems, e.g. Kafka, Pulsar, RabbitMQ and many other sources to consume and build the streaming application. Therefore it could not rely on changelog topics as Kafka-Streams does. Apache Flink offers out of the box following State Backends for stateful streaming processing: ...

Building pipelines aka Continuous Integration and Continuous Delivery (CI/CD) are not really new buzzwords in the tech industry or as sexy as bitcoin and friends, but I’m still quite excited about the recent release of the GitLab CE 11.7 Version which was released on 22nd January 2019. In this post we will have a look to newly added feature in GitLab CI where we will MAKE THE GITLAB-CI.YML DRY AGAIN TL;DR As of GitLab CE version 11.7, you can use include commands to include your tasks from other .yml files. This is a big deal, because you don‘t have to repeat your self in each of your projects any more. Especially in projects that consist of a large number of microservices sharing (parts) of the pipeline definition is major win. ...

Originally published at codecentrics blog “ETL with Kafka” is a catchy phrase that I purposely chose for this post instead of a more precise title like “Building a data pipeline with Kafka Connect”. TLDR You don’t need to write any code for pushing data into Kafka, instead just choose your connector and start the job with your necessary configurations. And it’s absolutely Open Source! Kafka Connect Kafka Before getting into the Kafka Connect framework, let us briefly sum up what Apache Kafka is in couple of lines. Apache Kafka was built at LinkedIn to meet the requirements that message brokers already existing in the market did not meet – requirements such as scalable, distributed, resilient with low latency and high throughput. Currently, i.e. 2018, LinkedIn is processing about 1.8 petabytes of data per day through Kafka. Kafka offers a programmable interface (API) for a lot of languages to produce and consume data. ...