InfoQ Homepage Big Data Content on InfoQ
-
/articles/time-series-storage-design/en/smallimage/Time-Series-Storage-Design-Choices-That-Shape-Cost-and-Performance-thumb-1778155792101.jpg)
Time-Series Storage: Design Choices That Shape Cost and Performance
Every time-series database makes a set of storage design decisions: how to lay out rows, when to compress, what to partition on. These decisions determine cost and query performance more than the choice of database itself. This article works through those fundamentals from first principles, using widely available tools like PostgreSQL and Apache Parquet to make each trade-off measurable.
-
/articles/micro-batch-streaming-lessons-learned/en/smallimage/micro-batch-streaming-lessons-learned-thumbnail-1777381781538.jpg)
From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline
This article describes how a production delta-index pipeline migrated from scheduled batch to micro-batch Spark Structured Streaming. It covers why record-level streaming was rejected, how partition-based watermarks replaced fragile S3 completion markers, overlap-window correctness, and restart-as-design strategies for better predictability in object-store–based ingestion systems.
-
/articles/agent-reinforcement-learning-apache-spark/en/smallimage/thumbnail-1769431334097.jpg)
Autonomous Big Data Optimization: Multi-Agent Reinforcement Learning to Achieve Self-Tuning Apache Spark
This article introduces a reinforcement learning (RL) approach grounded in Apache Spark that enables distributed computing systems to learn optimal configurations autonomously, much like an apprentice engineer who learns by doing. The author also implements a lightweight agent as a driver-side component that uses RL to choose configuration settings before a job runs.
-
/articles/secure-multi-party-computation/en/smallimage/secure-multi-party-computation-thumbnail-1743067702920.jpg)
How to Compute without Looking: a Sneak Peek into Secure Multi-Party Computation
This article shows how you can compute a function across multiple parties that do not trust each other without forcing them to share their individual inputs. This technique can be used to split secrets among parties, perform logical operations, or count votes in a way that ensures data privacy is preserved.
-
/articles/zero-knowledge-proofs-intro/en/smallimage/zero-knowledge-proofs-1710429219949.jpg)
Zero-Knowledge Proofs for the Layman
This article will introduce you to zero-knowledge proofs, a kind of cryptography you can use to provide the proof you know a secret, such as a private key or the solution to a problem, without ever sharing it to an interested party. While many articles exist on the topic, this will not require any high math knowledge.
-
/articles/impact-machine-learning-climate/en/smallimage/minimising-the-impact-of-machine-learning-on-our-climate-small-1685091723283.jpg)
Minimising the Impact of Machine Learning on our Climate
This article introduces the field of green software engineering, showing the Green Software Foundation’s Software Carbon Intensity Specification, which is used to estimate the carbon footprint of software, and discusses ideas on how to make machine learning greener. It aims to give you the tools to take an active part in the climate solution.
-
/articles/data-protection-federal/en/smallimage/logo-1673874259012.jpg)
Data Protection Methods for Federal Organizations and beyond
The Federal Data Strategy describes a plan to “accelerate the use of data to deliver on mission, serve the public, and steward resources while protecting security, privacy, and confidentiality." This article covers what it is and how it can be applied to any organization.
-
/articles/anatomy-code-obfuscation/en/smallimage/logo-1667849003893.jpg)
Who Moved My Code? An Anatomy of Code Obfuscation
In this article, we introduce the topic of code obfuscation, with emphasis on string obfuscation. Obfuscation is an important practice to protect source code by making it unintelligible. Obfuscation is often mistaken with encryption, but they are different concepts. In the article we will present a number of techniques and approaches used to obfuscate data in a program.
-
/articles/transatlantic-data-privacy-2022-virtual-panel/en/smallimage/virtual-panel-the-new-US-EU-data-privacy-framework-small-image-1663768696087.jpg)
Virtual Panel: the New US-EU Data Privacy Framework
Recent rulings by several European courts have set important precedents for restricting personal data transmission from the EU to the US. As a consequence, the US and EU have started working on a new agreement. In this virtual panel, three knowledgeable experts discuss where the existing agreements fall short, and whether a new privacy agreement could improve the current situation.
-
/articles/cloud-native-apache-dolphinscheduler/en/smallimage/logo-1655898721435.jpg)
Embracing Cloud-Native for Apache DolphinScheduler with Kubernetes: a Case Study
This article shares how Apache DolphinScheduler was updated to use a more modern, cloud-native architecture. This includes moving to Kubernetes and integrating with Argo CD and Prometheus. This improves substantially the user experience of deploying, operating, and monitoring DolphinScheduler.
-
/articles/deep-learning-institutional-incremental-learning/en/smallimage/institutional-incremental-learning-small-image-v2-1640874831613.jpg)
Developing Deep Learning Systems Using Institutional Incremental Learning
Institutional incremental learning promises to achieve collaborative learning. This form of learning can address data sharing and security issues, without bringing in the complexities of federated learning. This article talks about practical approaches which help in building an object detection system.
-
/articles/deep-learning-apache-spark-nvidia-gpu/en/smallimage/spark-logo-1623166971929.jpg)
Accelerating Deep Learning on the JVM with Apache Spark and NVIDIA GPUs
In this article, authors discuss how to use the combination of Deep Java Learning (DJL), Apache Spark v3, and NVIDIA GPU computing to simplify deep learning pipelines while improving performance and reducing costs. They also show the performance comparison of this solution with GPU vs CPU hardware, using Amazon EMR and NVIDIA RAPIDS Accelerator.