InfoQ Homepage Neural Networks Content on InfoQ
-
Perceiver: One Neural-Network Model for Multiple Input Data Types
Google’s DeepMind company has recently released a state-of-the-art deep-learning model called Perceiver that receives and processes multiple input data ranging from audio to images, similarly to how the human brain perceives multimodal data. Perceiver is able to receive and classify input multiple data types, namely point cloud, audio and images.
-
Microsoft Releases AI Training Library ZeRO-3 Offload
Microsoft recently open-sourced ZeRO-3 Offload, an extension of their DeepSpeed AI training library that improves memory efficiency while training very large deep-learning models. ZeRO-3 Offload allows users to train models with up to 40 billion parameters on a single GPU and over 2 trillion parameters on 512 GPUs.
-
Alibaba Announces 10 Billion Parameter Multi-Modal AI M6
Alibaba has created an AI model called Multi-Modality to Multi-Modality Multitask Mega-transformer (M6). The model contains 10 billion parameters and is pretrained on a dataset consisting of 1.9TB of images and 292GB of Chinese-language text. M6 can be fine-tuned for several downstream tasks, including text-guided image generation, visual question answering, and image-text matching.
-
Google's Apollo AI for Chip Design Improves Deep Learning Performance by 25%
Scientists at Google Research have announced APOLLO, a framework for optimizing AI accelerator chip designs. APOLLO uses evolutionary algorithms to select chip parameters that minimize deep-learning inference latency while also minimizing chip area. Using APOLLO, researchers found designs that achieved 24.6% speedup over those chosen by a baseline algorithm.
-
Google DeepMind’s NFNets Offers Deep Learning Efficiency
Google’s DeepMind AI company recently released NFNets, a normalizer-free ResNet image classification model that achieved a training performance of 8.7x faster than current state-of-the-art EfficientNet. In addition, it helps neural networks to generalize better.
-
PyTorch 1.8 Release Includes Distributed Training Updates and AMD ROCm Support
PyTorch, Facebook's open-source deep-learning framework, announced the release of version 1.8 which includes updated APIs, improvements for distributed training, and support for the ROCm platform for AMD's GPU accelerators. New versions of domain-specific libraries TorchVision, TorchAudio, and TorchText were also released.
-
Google Open-Sources AutoML Algorithm Model Search
A team from Google Research has open-sourced Model Search, an automated machine learning (AutoML) platform for designing deep-learning models. Experimental results show that the system produces models that outperform the best human-designed models, with fewer training iterations and model parameters.
-
Microsoft Satin Audio Codec Uses AI to Outperform Skype Silk
Microsoft announced Satin, a new audio codec that leverages AI techniques to outperform Skype's Silk codec over ultra-low bandwidth and highly constrained network conditions.
-
Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
Researchers at Google Brain have open-sourced the Switch Transformer, a natural-language processing (NLP) AI model. The model scales up to 1.6T parameters and improves training time up to 7x compared to the T5 NLP model, with comparable accuracy.
-
OpenAI Announces GPT-3 Model for Image Generation
OpenAI has trained a 12B-parameter AI model based on GPT-3 that can generate images from textual description. The description can specify many independent attributes, including the position of objects as well as image perspective, and can also synthesize combinations of objects that do not exist in the real world.
-
Facebook Open-Sources Multilingual Speech Recognition Deep-Learning Model
Facebook AI Research (FAIR) open-sourced Cross-Lingual Speech Recognition (XSLR), a multilingual speech recognition AI model. XSLR is trained on 53 languages and outperforms existing systems when evaluated on common benchmarks.
-
TensorFlow 2.4 Release Includes CUDA 11 Support and API Updates
The TensorFlow project announced the release of version 2.4.0 of the deep-learning framework, featuring support for CUDA 11, cuDNN 8, and NVIDIA's Ampere GPU architecture, as well as new strategies and profiling tools for distributed training. Other API updates include mixed-precision in Keras and a NumPy frontend.
-
AI Models from Google and Microsoft Exceed Human Performance on Language Understanding Benchmark
Research teams from Google and Microsoft have recently developed natural language processing (NLP) AI models which have scored higher than the human baseline score on the SuperGLUE benchmark. SuperGLUE measures a model's score on several natural language understanding (NLU) tasks, including question answering and reading comprehension.
-
DeepMind's AlphaFold2 AI Solves 50-Year-Old Biology Challenge
The Protein Structure Prediction Center announced that AlphaFold2, an AI system developed by DeepMind, has solved its Protein Structure Prediction challenge. AlphaFold2 achieved a median score of 92.4 on the Global Distance Test (GDT) metric, above the threshold considered competitive with traditional methods.
-
Caltech Open-Sources AI for Solving Partial Differential Equations
Researchers from Caltech's DOLCIT group have open-sourced Fourier Neural Operator (FNO), a deep-learning method for solving partial differential equations (PDEs). FNO outperforms other existing deep-learning techniques for solving PDEs and is three orders of magnitude faster than traditional solvers.