Amazon Managed Streaming for Apache Kafka (MSK) – Now Generally Available


I am always amazed at how our customers are using streaming data. For example, Thomson Reuters, one of the world’s most trusted news organizations for businesses and professionals, built a solution to capture, analyze, and visualize analytics data to help product teams continuously improve the user experience. Supercell, the social game company providing games such as Hay Day, Clash of Clans, and Boom Beach, is delivering in-game data in real-time, handling 45 billion events per day.
Since we launched Amazon Kinesis at re:Invent 2013, we have continually expanded the ways in in which customers work with streaming data on AWS. Some of the available tools are:
Kinesis Data Streams, to capture, store, and process data streams with your own applications.
Kinesis Data Firehose, to transform and collect data into destinations such as Amazon S3, Amazon Elasticsearch Service, and Amazon Redshift.
Kinesis Data Analytics, to continuously analyze data using SQL or Java (via Apache Flink applications), for example to detect anomalies or for time series aggregation.
Kinesis Video Streams, to simplify processing of media streams.
At re:Invent 2018, we introduced in open preview Amazon Managed Streaming for Apache Kafka (MSK), a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data.
I am excited to announce that Amazon MSK is generally available today!
How it works
Apache Kafka (Kafka) is an open-source platform that enables customers to capture streaming data like click stream events, transactions, IoT events, application and machine logs, and have applications that perform real-time analytics, run continuous transformations, and distribute this data to data lakes and databases in real time. You can use Kafka as a streaming data store to decouple applications producing streaming data (producers) from those consuming streaming data (consumers).
While Kafka is a popular enterprise data streaming and messaging framework, it can be difficult to setup, scale, and manage in production. Amazon MSK takes care of these managing tasks and makes it easy to set up, configure, and run Kafka, along with Apache ZooKeeper, in an environment following best practices for high availability and security.
Your MSK clusters always run within an Amazon VPC managed by the MSK service. Your MSK resources are made available to your own VPC, subnet, and security group through elastic network interfaces (ENIs) which will appear in your account.
Customers can create a cluster in minutes, use AWS Identity and Access Management (IAM) to control cluster actions, authorize clients using TLS private certificate authorities fully managed by AWS Certificate Manager (ACM), encrypt data in-transit using TLS, and encrypt data at rest using AWS Key Management Service (KMS) encryption keys.
Amazon MSK continuously monitors server health and automatically replaces servers when they fail, automates server patching, and operates highly available ZooKeeper nodes as a part of the service at no additional cost. Key Kafka performance metrics are published in the console and in Amazon CloudWatch. Amazon MSK is fully compatible with Kafka versions 1.1.1 and 2.1.0, so that you can continue to run your applications, use Kafka’s admin tools, and and use Kafka compatible tools and frameworks without having to change your code.
Based on our customer feedback during the open preview, Amazon MSK added may features such as:
Encryption in-transit via TLS between clients and brokers, and between brokers
Mutual TLS authentication using ACM private certificate authorities
Support for Kafka version 2.1.0
99.9% availability SLA
HIPAA eligible
Cluster-wide storage scale up
Integration with AWS CloudTrail for MSK API logging
Cluster tagging and tag-based IAM policy application
Defining custom, cluster-wide configurations for topics and brokers
Pricing and availability
Pricing is per Kafka broker-hour and per provisioned storage-hour. There is no cost for the Zookeeper nodes used by your clusters. AWS data transfer rates apply for data transfer in and out of MSK. You will not be charged for data transfer within the cluster in a region, including data transfer between brokers and data transfer between brokers and ZooKeeper nodes.
You can migrate your existing Kafka cluster to MSK using tools like MirrorMaker (that comes with open source Kafka) to replicate data from your clusters into a MSK cluster.
Upstream compatibility is a core tenet of Amazon MSK. Our code changes to the Kafka platform are released back to open source.
Amazon MSK is available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Tokyo), Asia Pacific (Singapore), Asia Pacific (Sydney), EU (Frankfurt), EU (Ireland), EU (Paris), and EU (London).
I look forward to see how are you going to use Amazon MSK to simplify building and migrating streaming applications to the cloud![Source]-https://aws.amazon.com/blogs/aws/amazon-managed-streaming-for-apache-kafka-msk-now-generally-available/
AWS Training Institute Courses in Mumbai. 30 hours practical training program on all avenues of Amazon Web Services. Learn under AWS Expert.


Comments

Post a Comment

Popular posts from this blog

Why, when and how to return Stream from your Java API instead of a collection

Introduction Collections are basic and commonly used data structures. Programmers from the beginning of their career learn how to use them to receive, process and return data. Getting more advanced in Java programming, they find stream() method to convert a collection into a stream and learn how to process data using some of the stream’s useful methods like map, flatMap or reduce. They could also notice that other APIs in Java return a stream too e.g. String.lines(), Matcher.results(), Files.find(), Random.ints(). If you have experience with consuming streams but haven’t produced them yet, this article is for you. I’m going to show you some scenarios where streams can be very convenient and examples on how to use them. Additionally, I shortly mention error handling and resource management. The article is based on standard Java library java.util.stream. It’s related neither to reactive streams nor to other implementation of streams like e.g. Vavr. Also, I’m not going to cover a...
Read more

What is Kubernetes?

Kubernetes (also known as k8s or “kube”) is an open source container orchestration platform that automates many of the manual processes involved in deploying, managing, and scaling containerized applications. In other words, you can cluster together groups of hosts running Linux containers, and Kubernetes helps you easily and efficiently manage those clusters. Kubernetes clusters can span hosts across on-premise, public, private, or hybrid clouds. For this reason, Kubernetes is an ideal platform for hosting cloud-native applications that require rapid scaling, like real-time data streaming through Apache Kafka. Kubernetes was originally developed and designed by engineers at Google. Google was one of the early contributors to Linux container technology and has talked publicly about how everything at Google runs in containers. (This is the technology behind Google’s cloud services.) Google generates more than 2 billion container deployments a week, all powered by its ...
Read more

Best way to learn Java programming

Having said that I am writing this post dedicated to all my young (or I should say beginner) fellows who want to attain a certain level of proficiency in java technology and somewhere would like to take my advice on this. Keep in mind that if you do not like the way to learn java, I am proposing in this post, then just ignore me. Period. OR better, suggest me what you think is the better way to learn java fast or easily. Here I am assuming the people reading this post will be, who are very new to language, so I will start by listing first thing first. Make sure you have prepared your Java development environment ready i.e. You have installed JDK/JRE and you have an IDE like Eclipse. 1) Learn the language basics This is the first step for very obvious reason. If you don’t know the basics then you will never know either what to do next or what you are doing wrong. Initially, I do not expect from you to become the master of all java basic stuffs like keywords, core concepts o...
Read more