136041 SE Topics in Deep Learning and Natural Language Processing (2023S)

[A] Word vectors:

[A.1] Mikolov et al. "Distributed representations of words and phrases and their compositionality."
[A.2] Bojanowski et al. "Enriching word vectors with subword information."

[B] Attention:

[B.1] Hermann et al. "Teaching machines to read and comprehend."
[B.2] Bahdanau et al. "Neural machine translation by jointly learning to align and translate."
[B.3] Vaswani et al. "Attention is all you need."

[C] Pre-Trained language models:

[C.1] Devlin et al. "Bert: Pre-training of deep bidirectional transformers for language understanding."
[C.2] Brown et al. "Language Models are Few-Shot Learners."
[C.3] Clark et al. "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators"
[C.4] Ouyang et al. "Training language models to follow instructions with human feedback"

[D] Data Sets and Evaluation:

[D.1] Kwiatkowski et al. "Natural questions: a benchmark for question answering research."
[D.2] Wang et al. "Glue: A multi-task benchmark and analysis platform for natural language understanding."

[E] Probing of pre-trained language models, explainability:

[E.1] Jawahar et al. "What does BERT learn about the structure of language?"
[E.2] Petroni et al. "Language models as knowledge bases?"
[E.3] Tenney et al. "BERT Rediscovers the Classical NLP Pipeline"
[E.4] Ribeiro et al. ""Why should I trust you?" Explaining the predictions of any classifier."
[E.5] Ribeiro et al. "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList."
[E.6] Bender, Koller: "Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data"

[F] Language bias and learned representations:

[F.1] Bolukbasi et al. "Man is to computer programmer as woman is to homemaker? debiasing word embeddings."
[F.2] Garg et al. "Word embeddings quantify 100 years of gender and ethnic stereotypes."
[F.3] Sap et al. "Social bias frames: Reasoning about social and power implications of language."
[F.4] Zhao et al. "Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods"

[G] Relation Extraction and Distant supervision:

[G.1] Riedel et al. "Relation extraction with matrix factorization and universal schemas."
[G.2] Verga et al. "Multilingual relation extraction using compositional universal schema."

[H] Weak Supervision:

[H.1] Karamanolakis et al. “Self-Training with Weak Supervision”
[H.2] Fu et al. “Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods”
[H.3] Zhang et al. “WRENCH: A Comprehensive Benchmark for Weak Supervision”
[H.4] Li et al. “Prefix-Tuning: Optimizing Continuous Prompts for Generation.“
[H.5] Stephan et al. "SepLL: Separating Latent Class Labels from Weak Supervision Noise"

[I] Ethical and security aspects of Language Models

[I.1] Weidinger et al. "Taxonomy of Risks posed by Language Models"
[I.2] Gary Marcus and Ernest Davis. "GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about"
[I.3] Luz de Araujo and Roth "Checking HateCheck: a cross-functional analysis of behaviour-aware learning for hate speech detection"