September 11th | Data Science meets Political Science (AS.190.360)

Dr. Kenton Murray will talk about the fundamentals of modern AI systems. The readings are particularly technical and it is not expected that students without an engineering or math background will understand everything. However, it is expected that you will have at least tried to read through it and come to class with questions about what you did not understand.

We will start by discussing the Transformer paper (Vaswani et al., 2017). This describes a Machine Learning Model (specifically, it is a neural network architecture). It is the “T” in “GPT”.
This paper was very important, but also dense, so many people learn the details of it through The Annotated Transformer blog post (Rush, 2018).
Finally, we will read the GPT-1 paper (which was actually never pubished aside from the researchers’ peronal websited (Radford et al., 2018).

This class and its readings are also mandatory for one of your four relections. It is due before class in two weeks (no class next week - enjoy!).

References

2018

The annotated transformer

Alexander M Rush

In Proceedings of workshop for NLP open source software (NLP-OSS), 2018

Bib

@inproceedings{rush2018annotated,
  title = {The annotated transformer},
  author = {Rush, Alexander M},
  booktitle = {Proceedings of workshop for NLP open source software (NLP-OSS)},
  pages = {52--60},
  year = {2018}
}

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, and 2 more authors

2018

Bib

@article{radford2018improving,
  title = {Improving language understanding by generative pre-training},
  author = {Radford, Alec and Narasimhan, Karthik and Salimans, Tim and Sutskever, Ilya and others},
  year = {2018},
  publisher = {San Francisco, CA, USA}
}

2017

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, and 5 more authors

Advances in neural information processing systems, 2017

Bib

@article{vaswani2017attention,
  title = {Attention is all you need},
  author = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {\L}ukasz and Polosukhin, Illia},
  journal = {Advances in neural information processing systems},
  volume = {30},
  year = {2017}
}