InfoQ Homepage Big Data Content on InfoQ
-
Beyond the Database, and beyond the Stream Processor: What's the Next Step for Data Management?
Databases have been around forever with the same shape: you make a request to your data and then you receive an answer. Now, stream processors came along with a different approach: data isn’t locked up, it is in motion. Understand how stream processors and databases relate and why there is an emerging new category of databases that focus on data that stays in place as well as data that moves.
-
The End of the Privacy Shield Agreement Could Lead to Disaster for Hyperscale Cloud Providers
The recent ending of the Privacy Shield agreement by the European Court of Justice (ECJ) might impact cloud adoption. This article looks at the demise of this agreement, and possible solutions.
-
COVID-19 and Mining Social Media - Enabling Machine Learning Workloads with Big Data
In this article, author Adi Pollock discusses how to enable machine learning workloads with big data to query and analyze COVID-19 tweets to understand social sentiment towards COVID-19.
-
From Cloud to Cloudlets: a New Approach to Data Processing?
The growing popularity of small, distributed clouds, or “cloudlets” is an implicit recognition of the limitations of the “traditional” cloud model, and could signal a major shift in the way that data is collected, stored, and processed.
-
Combining DataOps and DevOps: Scale at Speed
DataOps is an extension of DevOps standards and processes into the data analytics world. It's about streamlining the processes involved in processing, analyzing and deriving value from big data.
-
Data Leadership Book Review and Interview
Data Leadership book, authored by Anthony Algmin, covers the data leadership topic and how data leaders should manage and govern the data management programs in their organizations. Data Leadership is how organizations choose to apply their energy and resources toward creating data capabilities to influence their business.
-
Apache Arrow and Java: Lightning Speed Big Data Transfer
Apache Arrow puts forward a cross-language, cross-platform, columnar in-memory data format for data. It is designed to eliminate the need for data serialization and reduce the overhead of copying.
-
Data Analytics in the World of Agility
Is it all about customer-centric business, or is there any data left? Can we integrate data analytics and customer empathy? This article explores how we can move towards a more customer-centric business and what information we require in order to understand the most valuable thing we have: our customer.
-
Stream Processing Anomaly Detection Using Yurita Framework
In this article, author Guy Gerson discusses the stream processing anomaly detection framework they developed by PayPal, called Yurita. The framework is based on Spark Structured Streaming.
-
Real-Time Data Processing Using Redis Streams and Apache Spark Structured Streaming
Structured Streaming, introduced with Apache Spark 2.0, delivers a SQL-like interface for streaming data. Redis Streams enables Redis to consume, hold and distribute streaming data between multiple producers and consumers. In this article, author Roshan Kumar walks us through how to process streaming data in real time using Redis and Apache Spark Streaming technologies.
-
Conquering the Challenges of Data Preparation for Predictive Maintenance
Predictive maintenance (PdM) applications aim to apply machine learning (ML) on IIoT datasets in order to reduce occupational hazards, machine downtime, and other costs. In this article, the author addresses some of the data preparation challenges faced by the industrial practitioners of ML and the solutions for data ingest and feature engineering related to PdM.
-
Analytics Zoo: Unified Analytics + AI Platform for Distributed Tensorflow, and BigDL on Apache Spark
In this article we described how Analytics Zoo can help real-world users to build end-to-end deep learning pipelines for big data, including unified pipelines for distributed TensorFlow and Keras on Apache Spark, easy-to-use abstractions such as transfer learning and Spark ML pipeline support, built-in deep learning models and reference use cases, etc.