Representation Learning for Natural Language Processing.

• Author: Liu, Zhiyuan.
• TeilnehmendeR: Lin, Yankai.; Sun, Maosong.
• Place / Publishing House: Singapore : : Springer,, 2023.; Ã2023.
• Year of Publication: 2023
• Edition: 2nd ed.
• Language: English
• Physical Description: 1 online resource (535 pages)

Table of Contents

Intro -- Preface -- Book Organization -- Book Cover -- Note for the Second Edition -- Prerequisites -- Contact Information -- Acknowledgments -- Acknowledgments for the Second Edition -- Acknowledgments for the First Edition -- Contents -- Contributors -- Acronyms -- Symbols and Notations -- 1 Representation Learning and NLP -- 1.1 Motivation -- 1.2 Why Representation Learning Is Important for NLP -- 1.2.1 Multiple Granularities -- 1.2.2 Multiple Knowledge -- 1.2.3 Multiple Tasks -- 1.2.4 Multiple Domains -- 1.3 Development of Representation Learning for NLP -- 1.3.1 Symbolic Representation and Statistical Learning -- 1.3.2 Distributed Representation and Deep Learning -- 1.3.3 Going Deeper and Larger with Pre-training on Big Data -- 1.4 Intellectual Origins of Distributed Representation -- 1.4.1 Representation Debates in Cognitive Neuroscience -- 1.4.2 Knowledge Representation in AI -- 1.4.3 Feature Engineering in Machine Learning -- 1.4.4 Linguistics -- 1.5 Representation Learning Approaches in NLP -- 1.5.1 Feature Engineering -- 1.5.2 Supervised Representation Learning -- 1.5.3 Self-supervised Representation Learning -- 1.6 How to Apply Representation Learning to NLP -- 1.6.1 Input Augmentation -- 1.6.2 Architecture Reformulation -- 1.6.3 Objective Regularization -- 1.6.4 Parameter Transfer -- 1.7 Advantages of Distributed Representation Learning -- 1.8 The Organization of This Book -- References -- 2 Word Representation Learning -- 2.1 Introduction -- 2.2 Symbolic Word Representation -- 2.2.1 One-Hot Word Representation -- 2.2.2 Linguistic KB-based Word Representation -- 2.2.3 Corpus-based Word Representation -- 2.3 Distributed Word Representation -- 2.3.1 Preliminary: Interpreting the Representation -- 2.3.2 Matrix Factorization-based Word Representation -- 2.3.3 Word2vec and GloVe -- 2.3.4 Contextualized Word Representation.

2.4 Advanced Topics -- 2.4.1 Informative Word Representation -- 2.4.2 Interpretable Word Representation -- 2.5 Applications -- 2.5.1 NLP -- 2.5.2 Cognitive Psychology -- 2.5.3 History and Social Science -- 2.6 Summary and Further Readings -- References -- 3 Representation Learning for Compositional Semantics -- 3.1 Introduction -- 3.2 Binary Composition -- 3.2.1 Additive Model -- 3.2.2 Multiplicative Model -- 3.3 N-ary Composition -- 3.4 Summary and Further Readings -- References -- 4 Sentence and Document Representation Learning -- 4.1 Introduction -- 4.2 Symbolic Sentence Representation -- 4.2.1 Bag-of-Words Model -- 4.2.2 Probabilistic Language Model -- 4.3 Neural Language Models -- 4.3.1 Feed-Forward Neural Network -- 4.3.2 Convolutional Neural Network -- 4.3.3 Recurrent Neural Network -- 4.3.4 Transformer -- 4.3.5 Enhancing Neural Language Models -- 4.4 From Sentence to Document Representation -- 4.4.1 Memory-Based Document Representation -- 4.4.2 Hierarchical Document Representation -- 4.5 Applications -- 4.5.1 Text Classification -- 4.5.2 Information Retrieval -- 4.5.3 Reading Comprehension -- 4.5.4 Open-Domain Question Answering -- 4.5.5 Sequence Labeling -- 4.5.6 Sequence-to-Sequence Generation -- 4.6 Summary and Further Readings -- References -- 5 Pre-trained Models for Representation Learning -- 5.1 Introduction -- 5.2 Pre-training Tasks -- 5.2.1 Word-Level Pre-training -- 5.2.2 Sentence-Level Pre-training -- 5.3 Model Adaptation -- 5.3.1 Full-Parameter Fine-Tuning -- 5.3.2 Delta Tuning -- 5.3.3 Prompt Learning -- 5.4 Advanced Topics -- 5.4.1 Better Model Architecture -- 5.4.2 Multilingual Representation -- 5.4.3 Multi-Task Representation -- 5.4.4 Efficient Representation -- 5.4.5 Chain-of-Thought Reasoning -- 5.5 Summary and Further Readings -- References -- 6 Graph Representation Learning -- 6.1 Introduction.

6.2 Symbolic Graph Representation -- 6.3 Shallow Node Representation Learning -- 6.3.1 Spectral Clustering -- 6.3.2 Shallow Neural Networks -- 6.3.3 Matrix Factorization -- 6.4 Deep Node Representation Learning -- 6.4.1 Autoencoder-Based Methods -- 6.4.2 Graph Convolutional Networks -- 6.4.3 Graph Attention Networks -- 6.4.4 Graph Recurrent Networks -- 6.4.5 Graph Transformers -- 6.4.6 Extensions -- 6.5 From Node Representation to Graph Representation -- 6.5.1 Flat Pooling -- 6.5.2 Hierarchical Pooling -- 6.6 Self-Supervised Graph Representation Learning -- 6.7 Applications -- 6.8 Summary and Further Readings -- References -- 7 Cross-Modal Representation Learning -- 7.1 Introduction -- 7.2 Cross-Modal Capabilities -- 7.3 Shallow Cross-Modal Representation Learning -- 7.4 Deep Cross-Modal Representation Learning -- 7.4.1 Cross-Modal Understanding -- 7.4.2 Cross-Modal Retrieval -- 7.4.3 Cross-Modal Generation -- 7.5 Deep Cross-Modal Pre-training -- 7.5.1 Input Representations -- 7.5.2 Model Architectures -- 7.5.3 Pre-training Tasks -- 7.5.4 Adaptation Approaches -- 7.6 Applications -- 7.7 Summary and Further Readings -- References -- 8 Robust Representation Learning -- 8.1 Introduction -- 8.2 Backdoor Robustness -- 8.2.1 Backdoor Attack on Supervised Representation Learning -- 8.2.2 Backdoor Attack on Self-Supervised Representation Learning -- 8.2.3 Backdoor Defense -- 8.2.4 Toolkits -- 8.3 Adversarial Robustness -- 8.3.1 Adversarial Attack -- 8.3.2 Adversarial Defense -- 8.3.3 Toolkits -- 8.4 Out-of-Distribution Robustness -- 8.4.1 Spurious Correlation -- 8.4.2 Domain Shift -- 8.4.3 Subpopulation Shift -- 8.5 Interpretability -- 8.5.1 Understanding Model Functionality -- 8.5.2 Explaining Model Mechanism -- 8.6 Summary and Further Readings -- References -- 9 Knowledge Representation Learning and Knowledge-Guided NLP -- 9.1 Introduction.

9.2 Symbolic Knowledge and Model Knowledge -- 9.2.1 Symbolic Knowledge -- 9.2.2 Model Knowledge -- 9.2.3 Integrating Symbolic Knowledge and Model Knowledge -- 9.3 Knowledge Representation Learning -- 9.3.1 Linear Representation -- 9.3.2 Translation Representation -- 9.3.3 Neural Representation -- 9.3.4 Manifold Representation -- 9.3.5 Contextualized Representation -- 9.3.6 Summary -- 9.4 Knowledge-Guided NLP -- 9.4.1 Knowledge Augmentation -- 9.4.2 Knowledge Reformulation -- 9.4.3 Knowledge Regularization -- 9.4.4 Knowledge Transfer -- 9.4.5 Summary -- 9.5 Knowledge Acquisition -- 9.5.1 Sentence-Level Relation Extraction -- 9.5.2 Bag-Level Relation Extraction -- 9.5.3 Document-Level Relation Extraction -- 9.5.4 Few-Shot Relation Extraction -- 9.5.5 Open-Domain Relation Extraction -- 9.5.6 Contextualized Relation Extraction -- 9.5.7 Summary -- 9.6 Summary and Further Readings -- References -- 10 Sememe-Based Lexical Knowledge Representation Learning -- 10.1 Introduction -- 10.2 Linguistic and Commonsense Knowledge Bases -- 10.2.1 WordNet and ConceptNet -- 10.2.2 HowNet -- 10.2.3 HowNet and Deep Learning -- 10.3 Sememe Knowledge Representation -- 10.3.1 Sememe-Encoded Word Representation -- 10.3.2 Sememe-Regularized Word Representation -- 10.4 Sememe-Guided Natural Language Processing -- 10.4.1 Sememe-Guided Semantic Compositionality Modeling -- 10.4.2 Sememe-Guided Language Modeling -- 10.4.3 Sememe-Guided Recurrent Neural Networks -- 10.5 Automatic Sememe Knowledge Acquisition -- 10.5.1 Embedding-Based Sememe Prediction -- 10.5.2 Sememe Prediction with Internal Information -- 10.5.3 Cross-lingual Sememe Prediction -- 10.5.4 Connecting HowNet with BabelNet -- 10.5.5 Summary and Discussion -- 10.6 Applications -- 10.6.1 Chinese LIWC Lexicon Expansion -- 10.6.2 Reverse Dictionary -- 10.7 Summary and Further Readings -- References.

11 Legal Knowledge Representation Learning -- 11.1 Introduction -- 11.2 Typical Tasks and Real-World Applications -- 11.3 Legal Knowledge Representation and Acquisition -- 11.3.1 Legal Textual Knowledge -- 11.3.2 Legal Structured Knowledge -- 11.3.3 Discussion -- 11.4 Knowledge-Guided Legal NLP -- 11.4.1 Input Augmentation -- 11.4.2 Architecture Reformulation -- 11.4.3 Objective Regularization -- 11.4.4 Parameter Transfer -- 11.5 Outlook -- 11.6 Ethical Consideration -- 11.7 Open Competitions and Benchmarks -- 11.8 Summary and Further Readings -- References -- 12 Biomedical Knowledge Representation Learning -- 12.1 Introduction -- 12.1.1 Perspectives for Biomedical NLP -- 12.1.2 Role of Knowledge in Biomedical NLP -- 12.2 Biomedical Knowledge Representation and Acquisition -- 12.2.1 Biomedical Knowledge from Natural Language -- 12.2.2 Biomedical Knowledge from Biomedical Language Materials -- 12.3 Knowledge-Guided Biomedical NLP -- 12.3.1 Input Augmentation -- 12.3.2 Architecture Reformulation -- 12.3.3 Objective Regularization -- 12.3.4 Parameter Transfer -- 12.4 Typical Applications -- 12.4.1 Literature Processing -- 12.4.2 Retrosynthetic Prediction -- 12.4.3 Diagnosis Assistance -- 12.5 Advanced Topics -- 12.6 Summary and Further Readings -- References -- 13 OpenBMB: Big Model Systems for Large-Scale Representation Learning -- 13.1 Introduction -- 13.2 BMTrain: Efficient Training Toolkit for Big Models -- 13.2.1 Data Parallelism -- 13.2.2 ZeRO Optimization -- 13.2.3 Quickstart of BMTrain -- 13.3 OpenPrompt and OpenDelta: Efficient Tuning Toolkit for Big Models -- 13.3.1 Serving Multiple Tasks with a Unified Big Model -- 13.3.2 Quickstart of OpenPrompt -- 13.3.3 QuickStart of OpenDelta -- 13.4 BMCook: Efficient Compression Toolkit for Big Models -- 13.4.1 Model Quantization -- 13.4.2 Model Distillation -- 13.4.3 Model Pruning.

13.4.4 Model MoEfication.