InfoQ Homepage Data Pipelines Content on InfoQ
-
Simplifying Real-Time ML Pipelines with Quix Streams
Tomáš Neubauer discusses Quix Streams, an open-source Python library that helps data scientists and ML engineers to build real-time ML pipelines.
/presentations/quix/en/mediumimage/TomášNeubauer-medium-1705999549605.jpg)
-
Streaming from Apache Iceberg - Building Low-Latency and Cost-Effective Data Pipelines
Steven Wu discusses the design of the Flink Iceberg, comparing the Kafka and Iceberg sources for streaming and how the Iceberg streaming source can power many common stream processing use cases.
/presentations/apache-iceberg-streaming/en/mediumimage/Steven-Wu-medium-1689934544144.jpg)
-
Orchestrating Hybrid Workflows with Apache Airflow
Ricardo Sueiras discusses how to leverage Apache Airflow to orchestrate a workflow using data sources inside and outside the cloud.
/presentations/aws-apache-airflow/en/mediumimage/Ricardo Sueiras-m-1680500699940.jpg)
-
Modern Data Pipelines in AdTech—Life in the Trenches
Roksolana Diachuk discusses how to use modern data pipelines for reporting and analytics as well as the case of historical data reprocessing in AdTech.
/presentations/data-pipelines-adtech/en/mediumimage/roksolana-m-1668076313804.jpg)
-
Building & Operating High-Fidelity Data Streams
Sid Anand discusses building high-fidelity nearline data streams as a service within a lean team.
/presentations/building-high-fidelity-data-streams/en/mediumimage/Sid-Anand-m-1648803206604.jpg)
-
Data Pipelines & Data Mesh: Where We Are and What the Future Looks Like
Zhamak Dehghani, Tareq Abedrabbo and Jacek Laskowski discuss the current challenges for building Modern Data Pipelines and applying Data Mesh in the real world, what the future looks like, and tools.
/presentations/data-pipelines-mesh/en/mediumimage/data-pipelines-mesh-m-1648200228421.jpg)
-
Cloud Native Continuous Delivery on Kubernetes with Tekton
Jerop Kipruto introduces the building blocks of Tekton and shows how they fit with Kubernetes. Then she demonstrates how Tekton works and how to use it in an end-to-end continuous delivery process.
/presentations/tekton-cloud-native/en/mediumimage/Jerop Kipruto-m-1643729034500.jpg)
-
Designing IoT Data Pipelines for Deep Observability
Shrijeet Paliwal discusses how Tesla deals with large data ingestion and processing, the challenges with IoT data collecting and processing, and how to deal with them.
/presentations/data-pipeline-tesla/en/mediumimage/shrijeet-paliwal-m-1624363169856.jpg)
-
Getting Started with Azure Event Hubs
Chad Green shows how to create an event-processing pipeline in Azure.
/presentations/azure-event-pipeline/en/mediumimage/Chad-Green-m-1588252194610.jpg)
-
Securing Your CI/CD Pipeline
Jeroen Willemsen shares his experience from various security automation implementation projects, showing how to secure a pipeline.
/presentations/ci-cd-security-pipeline/en/mediumimage/Jerobig-1584279444225.JPG)
-
Real-Time Performance Analysis of Data-Processing Pipelines with Spring Cloud Data Flow, Micrometer
Christian Tzolov and Sabby Anandan demonstrate how to consistently produce structured metrics to Prometheus, and how to visualize them with dashboards such as Grafana.
/presentations/spring-pipeline-prometheus-grafana/en/mediumimage/12big-1583366438136.jpg)
-
Future of Data Engineering
Chris Riccomini talks about the current state-of-the-art in data pipelines & data warehousing, and shares some of the solutions to current problems dealing with data streaming & warehousing.
/presentations/data-engineering-pipelines-warehouses/en/mediumimage/Chisbig-1575474871986.JPG)