InfoQ Homepage Database Content on InfoQ

Articles

RSS Feed

Newer Older

AI, ML & Data Engineering

AI, ML, and Data Engineering InfoQ Trends Report - September 2023

In this annual report, the InfoQ editors discuss the current state of AI, ML, and data engineering and what emerging trends you as a software engineer, architect, or data scientist should watch. We curate our discussions into a technology adoption curve with supporting commentary to help you understand how things are evolving.

Roland Meertens Srini Penchikala Sherin Thomas Daniel Dominguez Anthony Alford
on Sep 06, 2023
Java

Leveraging Eclipse JNoSQL 1.0.0: Quarkus Integration and Building a Pet-Friendly REST API

Eclipse JNoSQL 1.0.0 modernizes NoSQL integration with advanced features, standardized specs (Jakarta NoSQL & Jakarta Data), enhanced queries, schema migration, and Quarkus framework compatibility. It simplifies NoSQL use, boosts performance, scalability, and integrates seamlessly, empowering developers with tools to streamline data management in modern apps.

Otavio Santana
on Aug 23, 2023
Architecture & Design

Designing the Jit Analytics Architecture for Scale and Reuse

As a SaaS provider, analytical data at Jit needs to be useful to both their customers and to internal stakeholders. AWS services including EventBridge, Kinesis Data Firehose, and Timestream handle data ingestion and UI platforms from Mixpanel and Segment provide data visualization.

Ariel Beck Hen Kling Jonathan Rosenboim
on Jun 29, 2023
AI, ML & Data Engineering

In-Process Analytical Data Management with DuckDB

DuckDB is an open-source OLAP database for analytical data management that operates as an in-process database, avoiding data transfer overhead. Leveraging vectorized query processing and Morsel-Driven parallelism, the database optimizes performances and multi-core utilization for analytical data processing.

Hannes Mühleisen
on Jun 12, 2023
Culture & Methods

Debugging outside Your Comfort Zone: Diving beneath a Trusted Abstraction

This article takes a deep dive through a complex outage in the main database cluster of a payments company. We’ll focus on the aftermath of the incident - the process of understanding what went wrong, recreating the outage in a test cluster, and coming up with a way to stop it from happening again, and dive deep into the internals of Postgres, and learn about how it stores data on disk.

Chris Sinjakli
on Jun 07, 2023
Culture & Methods

Minimising the Impact of Machine Learning on our Climate

This article introduces the field of green software engineering, showing the Green Software Foundation’s Software Carbon Intensity Specification, which is used to estimate the carbon footprint of software, and discusses ideas on how to make machine learning greener. It aims to give you the tools to take an active part in the climate solution.

Sara Bergman
on May 30, 2023
Cloud

Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud.

Facundo Agriel
on May 15, 2023
Architecture & Design

Banking on Thousands of Microservices

Lessons learned building a banking platform, starting from technological choices like using Cassandra and Kubernetes in the early days to maintain the speed of execution through platform engineering and developer experience. With some mistakes and incidents along the way.

Suhail Patel
on May 08, 2023 4
AI, ML & Data Engineering

Understanding and Applying Correspondence Analysis

Customer segments, personality profiles, social classes, and age generations are examples of effective references to larger groups of people sharing similar characteristics. Correspondence analysis (CA) is a multivariate analysis technique that projects categorical data into a numeric feature space which captures most of the variability in the data by fewer dimensions.

Maarit Widmann Alfredo Roccato
on Feb 23, 2023
DevOps

Data Protection Methods for Federal Organizations and beyond

The Federal Data Strategy describes a plan to “accelerate the use of data to deliver on mission, serve the public, and steward resources while protecting security, privacy, and confidentiality." This article covers what it is and how it can be applied to any organization.

Alex Tray
on Jan 18, 2023
Development

InfoQ Software Trends Report: Major Trends in 2022 and What to Watch for in 2023

2022 was another year of significant technological innovations and trends in the software industry and communities. The InfoQ podcast co-hosts met last month to discuss the major trends from 2022, and what to watch for in 2023. This article is a summary of the 2022 software trends podcast.

Daniel Bryant Wesley Reisz Thomas Betts Shane Hastie Srini Penchikala
on Jan 13, 2023
Java

Securing a Kafka Cluster in Kubernetes Using Strimzi

Deploying an Apache Kafka cluster to Kubernetes is easy if you use Strimzi, but that’s only the first step; you need to secure the communication between Kafka and the consumers and producers, provide RBAC to access topics, spread the secrets correctly to Kafka Connect components and all using a Kubernetes GitOps way.

Alex Soto
on Dec 30, 2022