InfoQ Homepage Local Inference Content on InfoQ
News
RSS Feed-
Google ML Drift Accelerates On-Device AI Inference across Diverse GPUs
Google has released ML Drift, an open-source, cross-platform GPU compute engine designed for on-device AI and machine learning inference. Licensed under Apache 2.0, it abstracts the complexities of different GPU hardware and low-level APIs, supporting OpenGL ES, OpenCL, Metal, and WebGPU.
-
FreeToken Unlocks Frontier MoE Inference on Consumer Hardware via Dynamic Co-Execution
Researchers from UC Berkeley and MIT have developed FreeToken, an open-source inference engine that enhances the utility of Mixture-of-Experts models on consumer hardware. By implementing a dynamic scheduling policy and optimising weight management, FreeToken improves decoding speeds and execution efficiency in edge AI applications, fostering self-hosted reasoning systems.
-
Meta Open-Sources Muse Glimmer: a 30B Local Agentic Model Optimised for On-Device Execution
Meta AI Research has introduced Muse Glimmer, a 30-billion-parameter open-weight model under the Apache 2.0 license, designed for local workflows. It enables autonomous agents and complex task execution on consumer GPUs without relying on cloud APIs. The model employs a multi-stage training approach for efficient performance and supports multimodal inputs, enhancing coding and automation tasks.