License: 
Creative Commons Attribution 4.0 International license (CC BY 4.0)
When quoting this document, please refer to the following
DOI: 10.4230/DagRep.12.6.14
URN: urn:nbn:de:0030-drops-174549
URL: http://dagstuhl.sunsite.rwth-aachen.de/volltexte/2023/17454/
Dodge, Jesse ;
Gurevych, Iryna ;
Schwartz, Roy ;
Strubell, Emma ;
van Aken, Betty
Weitere Beteiligte (Hrsg. etc.): Jesse Dodge and Iryna Gurevych and Roy Schwartz and Emma Strubell and Betty van Aken
Efficient and Equitable Natural Language Processing in the Age of Deep Learning (Dagstuhl Seminar 22232)
Abstract
This report documents the program and the outcomes of Dagstuhl Seminar 22232 "Efficient and Equitable Natural Language Processing in the Age of Deep Learning". Since 2012, the field of artificial intelligence (AI) has reported remarkable progress on a broad range of capabilities including object recognition, game playing, speech recognition, and machine translation. Much of this progress has been achieved by increasingly large and computationally intensive deep learning models: training costs for state-of-the-art deep learning models have increased 300,000 times between 2012 and 2018 [1]. Perhaps the epitome of this trend is the subfield of natural language processing (NLP) that over the past three years has experienced even sharper growth in model size and corresponding computational requirements in the word embedding approaches (e.g. ELMo, BERT, openGPT-2, Megatron-LM, T5, and GPT-3, one of the largest models ever trained with 175B dense parameters) that are now the basic building blocks of nearly all NLP models. Recent studies indicate that this trend is both environmentally unfriendly and prohibitively expensive, raising barriers to participation in NLP research [2,3]. The goal of this seminar was to mitigate these concerns and promote equity of access in NLP.
References.
[1] D. Amodei and D. Hernandez. 2018. AI and Compute. https://openai.com/blog/ai-and-compute
[2] R. Schwartz, D. Dodge, N. A. Smith, and O. Etzioni. 2020. Green AI. Communications of the ACM (CACM)
[3] E. Strubell, A. Ganesh, and A. McCallum. 2019. Energy and Policy Considerations for Deep Learning in NLP. In Proc. of ACL.
BibTeX - Entry
@Article{dodge_et_al:DagRep.12.6.14,
author = {Dodge, Jesse and Gurevych, Iryna and Schwartz, Roy and Strubell, Emma and van Aken, Betty},
title = {{Efficient and Equitable Natural Language Processing in the Age of Deep Learning (Dagstuhl Seminar 22232)}},
pages = {14--27},
journal = {Dagstuhl Reports},
ISSN = {2192-5283},
year = {2023},
volume = {12},
number = {6},
editor = {Dodge, Jesse and Gurevych, Iryna and Schwartz, Roy and Strubell, Emma and van Aken, Betty},
publisher = {Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
address = {Dagstuhl, Germany},
URL = {https://drops.dagstuhl.de/opus/volltexte/2023/17454},
URN = {urn:nbn:de:0030-drops-174549},
doi = {10.4230/DagRep.12.6.14},
annote = {Keywords: deep learning, efficiency, equity, natural language processing (nlp)}
}
|
Keywords: |
|
deep learning, efficiency, equity, natural language processing (nlp) |
|
Collection: |
|
DagRep, Volume 12, Issue 6 |
|
Issue Date: |
|
2023 |
|
Date of publication: |
|
19.01.2023 |
