InfoQ Homepage Natural Language Processing Content on InfoQ
-
Tel-Aviv University Releases Long-Text NLP Benchmark SCROLLS
Researchers with Tel-Aviv University, Meta AI, IBM Research, and Allen Institute for AI have released Standardized CompaRison Over Long Language Sequences (SCROLLS), a set of natural language processing (NLP) benchmark tasks operating on long text sequences drawn from many domains. Experiments on baseline NLP models show that current models have significant room for improvement.
-
OpenAI Introduces InstructGPT Language Model to Follow Human Instructions
OpenAI overhauled the GPT-3 language model and introduced a new default tool called InstructGPT to address complaints about toxic language and misinformation.
-
OpenAI Announces Question-Answering AI WebGPT
OpenAI has developed WebGPT, an AI model for long-form question-answering based on GPT-3. WebGPT can use web search queries to collect supporting references for its response, and on Reddit questions its answers were preferred by human judges over the highest-voted answer 69% of the time.
-
AI Listens by Seeing as Well
Meta AI released a self-supervised speech recognition model that also uses video and achieves 75% better accuracy for some amount of data than current state-of-the-art models. This new model, Audio-Visual Hidden BERT (AV-HuBERT), uses audiovisual features for improving models based only on hearing speech. Visual features used are based on lip-reading, similar to what humans do.
-
Facebook Open-Sources Two Billion Parameter Multilingual Speech Recognition Model XLS-R
Facebook AI Research (FAIR) open-sourced XLS-R, a cross-lingual speech recognition (SR) AI model. XSLR is trained on 436K hours of speech audio from 128 languages, an order of magnitude more than the largest previous models, and outperforms the current state-of-the-art on several downstream SR and translation tasks.
-
Google Trains 280 Billion Parameter AI Language Model Gopher
Google subsidiary DeepMind announced Gopher, a 280-billion-parameter AI natural language processing (NLP) model. Based on the Transformer architecture and trained on a 10.5TB corpus called MassiveText, Gopher outperformed the current state-of-the-art on 100 of 124 evaluation tasks.
-
Facebook Develops New AI Model That Can Anticipate Future Actions
Facebook unveiled its latest machine-learning process called Anticipative Video Transformer (AVT), which is able to predict future actions by using visual interpretation. AVT works as an end-to-end attention-based model for action anticipation in videos.
-
BigScience Research Workshop Releases AI Language Model T0
BigScience Research Workshop released T0, a series of natural language processing (NLP) AI models specifically trained for researching zero-shot multitask learning. T0 can often outperform models 6x larger on the BIG-bench benchmark, and can outperform the 16x larger GPT-3 on several other NLP benchmarks.
-
Baidu Announces 11 Billion Parameter Chatbot AI PLATO-XL
Baidu recently announced PLATO-XL, an AI model for dialog generation, which was trained on over a billion samples collected from social media conversations in both English and Chinese. PLATO-XL achieves state-of-the-art performance on several conversational benchmarks, outperforming currently available commercial chatbots.
-
Baidu's ERNIE 3.0 AI Model Exceeds Human Performance on Language Understanding Benchmark
A research team from Baidu published a paper on the 3.0 version of Enhanced Language RepresentatioN with Informative Entities (ERNIE), a natural language processing (NLP) deep-learning model. The model contains 10B parameters and achieved a new state-of-the-art score on the SuperGLUE benchmark, outperforming the human baseline score.
-
Google Announces 800M Parameter Vision-Language AI Model ALIGN
Google Research announced the development of A Large-scale ImaGe and Noisy-Text Embedding (ALIGN), an 800M-parameter pre-trained deep-learning model trained on a noisy dataset of 1.8B image-text pairs. The model can be used on several downstream tasks and achieves state-of-the-art accuracy on several image-text retrieval benchmarks.
-
EleutherAI Open-Sources Six Billion Parameter GPT-3 Clone GPT-J
A team of researchers from EleutherAI have open-sourced GPT-J, a six-billion parameter natural language processing (NLP) AI model based on GPT-3. The model was trained on an 800GB open-source text dataset and has performance comparable to a GPT-3 model of similar size.
-
Google Open-Sources Token-Free Language Model ByT5
Google Research has open-sourced ByT5, a natural language processing (NLP) AI model that operates on raw bytes instead of abstract tokens. Compared to baseline models, ByT5 is more accurate on several benchmark tasks and is more robust to misspellings and noise.
-
NLP Library spaCy 3.0 Features Transformer-Based Models and Distributed Training
AI software makers Explosion announced version 3.0 of spaCy, their open-source natural-language processing (NLP) library. The new release includes state-of-the-art Transformer-based pipelines and pre-trained models for 17 languages.
-
Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
Researchers at Google Brain have open-sourced the Switch Transformer, a natural-language processing (NLP) AI model. The model scales up to 1.6T parameters and improves training time up to 7x compared to the T5 NLP model, with comparable accuracy.