Kafka Summit: Big Data In Real-Time Gains Momentum

...

There is a renewed focus on risk data aggregation and reporting (RDAR) solutions, as financial ins

Confluent hosted the first Kafka Summit focused on use-cases for the open source real-time streaming and messaging technology for big data -- featuring support and integration announcements from partners. Here's why this technology is so hot right now.

One of the frequently cited use-cases for real-time data comes from Uber, which has harnessed the power of data and real-time analytics to actually predict when your driver will arrive to pick you up. Uber created this service with a technology called Apache Kafka (among other technologies). It is the power and momentum of what real-time data can do -- as in that Uber application -- that brought more than 500 users to the Kafka Summit in San Francisco this week.

Hosted by the messaging and streaming technology's distributor Confluent, the inaugural event featured customer use-cases. It served as the backdrop for a handful of partners to unveil their own announcements around the technology, as well.

Apache Kafka is a technology that was created inside LinkedIn to manage that social network's massive data messaging flows. The technology was then spun out as an Apache Software Foundation big data project and a separate company called Confluent.

"The whole Hadoop community is pivoting to data in motion," Gartner VP of research Merv Adrian told InformationWeek in an interview. "Most of the conversation about Hadoop had been about data at rest, but increasingly the whole community is thinking about real-time or near real-time."

Are you prepared for a new world of enterprise mobility? Attend the Wireless & Mobility Track at Interop Las Vegas, May 2-6. Register now!

Apache Kafka is one of the open source projects today that fulfills that need.

"Kafka is one of the emerging stars here," Adrian said. The technology is supported by four out of the five Hadoop distributors (Cloudera, Hortonworks, IBM, and Amazon), according to Adrian. The one Hadoop distribution company that doesn't officially support Kafka as part of its stack, MapR, announced a new training program at the Kafka Summit this week that is designed to help developers connect its platform with Kafka.

MapR announced free stream processing on-demand training for real-time analytics and IoT applications. The Hadoop distributor said that the new training will enable Apache Kafka developers to extend their real-time analytics and their IoT applications. The training covers MapR streams that provide Kafka compatibility, according to a statement released by MapR.

Confluent itself announced the results of a survey of more than 100 Kafka users around the world. Twenty-nine percent of the respondents work for organizations with more than $1 billion in annual sales. The survey, conducted by Researchscape International and sponsored by Confluent, revealed that 88% of the respondents said Kafka would be a mission-critical part of their data and application infrastructure by 2017.

A total of 72% of respondents use Kafka for stream processing to enable all incoming data to flow in a continuous stream, and 68% said they plan to incorporate more stream processing over the next six to twelve months.

Organizations are using the technology for a wide variety of applications, including application monitoring, recommendation and decision engines, security and fraud detection, IoT applications, and dynamic pricing applications.

"We see more and more organizations embracing real-time data and stream processing, and Kafka is at the heart of that shift," said Jay Kreps, one of Kafka’s co-creators and the CEO and cofounder of Confluent, in a prepared statement.

End-to-end streaming integration and intelligence platform company Striim also announced a new partnership with Confluent in conjunction with the Kafka Summit. Striim said the partnership would bring real-time change data capture to the Kafka ecosystem.

"At Striim, we help companies free data from enterprise databases," said Sami Akbay, founder and EVP of Striim, in a prepared statement. "Our partnership with Confluent makes it easy for our joint customers to release their most valuable data and make it available in real time to the organization in a way that is orchestrated, secure and reliable."

The news out of the Summit this week built on Confluent's announcement from last week that it was adding more than a dozen new connectors, including HDFS, JDBC, Cassandra, and S3 through Kafka Connect. These new connecters enable new real-time data streams for Kafka.

Adrian told InformationWeek that Kafka and other technologies like it are examples of how the software development and delivery model has changed. Over the last decade, technology has been developed, proved its general usefulness, and then moved out into the broader community. Hadoop, Java, MySQL, Linux, and others have fit this pattern.

"It's a new way for products to emerge compared to the way they used to come out of R&D labs," Adrian said. "This is different. We are in our second decade of this happening. The pace and delivery of software has been completely transformed."

Categories
APPLICATIONS
0 Comment

Leave a Reply

Captcha image


RELATED BY

  • 5300c769af79e

    GE Puts Amazon's Alexa Inside a Table Lamp

    You'll find Amazon's Alexa voice assistant ready to take your commands from an Echo speaker, an Amazon Fire TV remote, and now, a table lamp.The lamp can be used to trigger Alexa's core functions, like ordering from Amazon, as well as Alexa "skills" that give the assistant power to control third-party devices and services, like your Google calendar or your lawn sprinklers.
  • 5300c769af79e

    Got $20K? This Start-Up Has a Secure Smartphone for You

    One start-up is pinning its hopes on the answer being "a lot.Based on Google's Android operating system, the phone will reportedly run other, unspecified technology that won't be available to mass-market consumers for another two to three years.
  • 5300c769af79e

    VMware's 2Q Results Show Upticks In NSX, Mobile Management Adoption

    What you l VMware reported solid results for its second quarter, with upticks in licenses, virtualized workspaces, mobility management, and NSX networking.69 billion in the second financial quarter of 2016, an increase of 11% over revenues for the same quarter last year.
  • 5300c769af79e

    6 Reasons Bimodal IT Is Wrong For You

    A couple of years ago, the good folks at Gartner unleashed a new label on the world of enterprise IT --  Bimodal IT.Some companies eagerly embraced bimodal IT as a way of "having their cake and eating it, too.