InfoQ Homepage Batch Processing Content on InfoQ
-
Karrot Improves Conversion Rates by 70% with New Scalable Feature Platform on AWS
Karrot replaced its legacy recommendation system with a scalable architecture that leverages various AWS services. The company sought to address challenges related to tight coupling, limited scalability, and poor reliability in its previous solution, opting instead for a distributed, event-driven architecture built on top of scalable cloud services.
-
Uber Completes Massive Kubernetes Migration for Microservices and Large-Scale Compute Workloads
Uber has successfully completed a large Kubernetes migration, transitioning its entire compute platform from Apache Mesos to Kubernetes across multiple data centers and cloud environments.
-
Scaling Uber’s Batch Data Platform: a Journey to the Cloud with Data Mesh Principles
Some months ago, Uber started the migration to the cloud, on Google Cloud Platform (GCP), of its batch data analytics and machine learning platform. In a recent post on its engineering blog, Uber provided additional information regarding its batch data cloud migration that incorporated crucial data mesh principles.
-
JobRunr Introduces Version 7.0 with Built-in Support for Virtual Threads
JobRunr v7 now defaults to virtual threads for applications using JDK 21, optimizing concurrency for I/O-bound tasks and allowing more jobs to run simultaneously. This update maintains compatibility with Java 8 and supports GraalVM native mode. RedisStorageProvider and ElasticSearchStorageProvider are planned to be dropped in future releases. MongoDB driver is also upgraded.
-
AWS Batch Introduces Multi-Container Jobs for Large-Scale Simulations
Recently, AWS announced the support of multi-container jobs in AWS Batch through the management console. This new feature simplifies the process of running simulations, particularly for testing complex systems such as those used in autonomous vehicles and robotics.
-
Cadence 1.0: Uber Releases Its Scalable Workflow Orchestration Platform
Uber released a major version of its workflow orchestration platform named Cadence after six years in development. Uber and other companies use Cadence to build stateful services at scale using native programming languages.
-
Pfizer Uses Serverless Architecture on AWS to Scale Processing of Digital Biomarkers
Pfizer upgraded the serverless architecture for processing digital biomarker data at scale to make it more flexible and configurable. They created a framework that uses a file processing pipeline built with AWS Step Functions and other serverless services, as well as a custom Python package for data ingestion and processing.
-
Cloudflare Previews Globally Distributed Queues without Egress Fees
Cloudflare recently announced the private beta of Cloudflare Queues, a message queuing service that allows applications to send and receive messages using Cloudflare Workers. The new service provides at-least once message delivery, supports batching of messages, and does not charge bandwidth egress fees.
-
Google Cloud Introduces Batch, a Service for Scheduling Batch Jobs
Google Cloud recently announced the preview of Batch, a managed service to run batch jobs at scale. The new service supports the latest T2A Arm-based instances and Spot VMs for large batch jobs utilizing task parallelization.
-
AWS Introduces Batch Support for AWS Fargate
During the first week of the annual re:invent, AWS introduced the ability to specify AWS Fargate as a computing resource for AWS Batch jobs. With the AWS Batch support for AWS Fargate, customers will have a way to run jobs on serverless compute resources, fully-managed from job submission to completion.
-
Dynein – an Asynchronous Background Job Service from Airbnb
At Airbnb, they move time consuming, resource intensive tasks over to asynchronous background jobs to improve scalability. The job scheduling system has become a very important component and they have therefore built Dynein, a distributed delayed job queueing service and scheduler. In a blog post, Andy Fang from Airbnb describes the background and challenges in designing and building the service.
-
Google Cloud Scheduler is Now Generally Available
In a recent blog, Google announced that customers can now securely invoke HTTP targets on a schedule using Cloud Scheduler – a fully managed cron job service that allows any application to invoke batch, big data, and cloud infrastructure operations.
-
Google Announces a Managed Cron Service: Google Cloud Scheduler
Google announced a new Service on the Google Cloud Platform (GCP) - Cloud Scheduler, a fully managed cron job service that allows any application to invoke batch, big data and cloud infrastructure operations. The service is currently available in beta.
-
Amazon Introduces AWS Batch Preview
At the recent AWS Re:Invent event, Amazon announced a new preview service, called AWS Batch. AWS Batch allows organizations to optimize their scheduling and workload execution across a cloud-based landscape. Amazon has built this service in response to many AWS customers building their own batch platforms using EC2 instances, containers and CloudWatch.
-
AnyPresence Soups up Enterprise MBaaS Platform- Part 1 of 2
Mobile Backend as a Service provider AnyPresence continues to hone their chops. Launching the fifth update to their self-titled platform geared for the enterprise. Co-founder Rich Mendis provides some insights for InfoQ readers…