InfoQ Homepage Data Pipelines Content on InfoQ
-
Real-Time Performance Analysis of Data-Processing Pipelines with Spring Cloud Data Flow, Micrometer
Christian Tzolov and Sabby Anandan demonstrate how to consistently produce structured metrics to Prometheus, and how to visualize them with dashboards such as Grafana.
/presentations/spring-pipeline-prometheus-grafana/en/mediumimage/12big-1583366438136.jpg)
-
Future of Data Engineering
Chris Riccomini talks about the current state-of-the-art in data pipelines & data warehousing, and shares some of the solutions to current problems dealing with data streaming & warehousing.
/presentations/data-engineering-pipelines-warehouses/en/mediumimage/Chisbig-1575474871986.JPG)
-
Building and Operating a Serverless Data Pipeline
Will Norman discusses the motivations of switching to a serverless infrastructure, and lessons learned while building and operating such a system at scale.
/presentations/serverless-data-pipeline/en/mediumimage/Will-Norman-m-1571264580667.jpg)
-
Announcing Broadway
José Valim discusses how Broadway connects multiple stages and producers, how it leverages GenStage to provide back-pressure, and other features such as batching, rate-limiting, partitioning and more.
/presentations/broadway/en/mediumimage/Jobig-1568591143171.JPG)
-
Cloud-Native Data Pipelines with Apache Kafka
Gwen Shapira discusses how data engineering requirements changed in a cloud-native world, and how the solutions change with them.
/presentations/cloud-native-data-pipeline-kafka/en/mediumimage/Gwebig-1564871370264.JPG)
-
Event-Driven Architectures with Apache Geode and Spring Integration
Charlie Black deploys Spring Integration pipelines to react to changes of the data stored in Apache Geode.
/presentations/apache-geode-spring-integration-pipelines/en/mediumimage/Chabig-1553140295208.JPG)
-
Designing Automated Pipelines for Unseen Custom Data
Kevin Moore discusses some challenges in designing automated machine learning pipelines that can deal with custom user data that it has never seen before, as well as some of Salesforce’s solutions.
/presentations/design-automatic-pipelines-ml/en/mediumimage/Kevin-Moore-m-1547037948084.jpeg)
-
Zero to Multi-Cloud
Marcin Grzejszczak, Jon Schneider discuss using Spring Cloud Pipelines and Spinnaker together.
/presentations/spring-cloud-pipelines-spinnaker/en/mediumimage/Jobig-1544244214507.JPG)
-
ML Data Pipelines for Real-Time Fraud Prevention @PayPal
Mikhail Kourjanski focuses on the architectural approach towards PayPal’s real-time service platform that leverages ML models, delivers performance and quality of decisions.
/presentations/paypal-ml-fraud-prevention-2018/en/mediumimage/Mikbig-1534991790754.JPG)
-
Simplifying ML Workflows with Apache Beam
Tyler Akidau discusses how Apache Beam is simplifying pre- and post-processing for ML pipelines.
/presentations/ml-workflow-apache-beam/en/mediumimage/Tybig-1529542935186.JPG)
-
Orchestrating Data Microservices with Spring Cloud Data Flow
Mark Pollack discusses how to create data integration and real-time data processing pipelines using Spring Cloud Data Flow and deploy them to multiple platforms – Cloud Foundry, Kubernetes, and YARN.
/presentations/microservices-spring-cloud-data-flow/en/mediumimage/Marbig-1528670572828.JPG)
-
Data Pipelines for Real-Time Fraud Prevention at Scale
Mikhail Kourjanski discusses the architecture of PayPal’s data service which combines a Big Data approach with providing data in real time for decision making in fraud detection.
/presentations/paypal-data-service-fraud/en/mediumimage/Mikbig-1527121529188.JPG)