机器之心|从word2vec开始，说下GPT庞大的家族系谱( 十 )

[4] Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin (2017). Attention Is All You NeedCoRR, abs/1706.03762.
[5] Zihang Dai and Zhilin Yang and Yiming Yang and Jaime G. Carbonell and Quoc V. Le and Ruslan Salakhutdinov (2019). Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextCoRR, abs/1901.02860.
[6] P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer. Generating wikipedia by summarizing long sequences. ICLR, 2018.
[7] Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training.
[8] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9.
[9] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, & Dario Amodei. (2020). Language Models are Few-Shot Learners.
[10]Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingCoRR, abs/1810.04805.
[11]Zhilin Yang and Zihang Dai and Yiming Yang and Jaime G. Carbonell and Ruslan Salakhutdinov and Quoc V. Le (2019). XLNet: Generalized Autoregressive Pretraining for Language UnderstandingCoRR, abs/1906.08237.
[12] attention 机制及 self-attention(transformer). Accessed at: https://blog.csdn.net/Enjoy_endless/article/details/88679989
[13] Attention 机制详解（一）——Seq2Seq 中的 Attention. Accessed at: https://zhuanlan.zhihu.com/p/47063917
[14]一文看懂 Attention（本质原理 + 3 大优点 + 5 大类型.Accessed at:https://medium.com/@pkqiang49/%E4%B8%80%E6%96%87%E7%9C%8B%E6%87%82-attention-%E6%9C%AC%E8%B4%A8%E5%8E%9F%E7%90%86-3%E5%A4%A7%E4%BC%98%E7%82%B9-5%E5%A4%A7%E7%B1%BB%E5%9E%8B-e4fbe4b6d030
[15]The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning).Accessed at:http://jalammar.github.io/illustrated-bert/
[16] Yang, Zhilin, et al. "Xlnet: Generalized autoregressive pretraining for language understanding." Advances in neural information processing systems. 2019.
[17] Dai, Zihang, et al. "Transformer-xl: Attentive language models beyond a fixed-length context." arXiv preprint arXiv:1901.02860 (2019).
[18] NLP——GPT 对比 GPT-2. Accessed at: https://zhuanlan.zhihu.com/p/96791725
[19] 深度学习：前沿技术 - GPT 1 & 2. Accessed at: http://www.bdpt.net/cn/2019/10/08/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%EF%BC%9A%E5%89%8D%E6%B2%BF%E6%8A%80%E6%9C%AF-gpt-1-2/

机器之心|从word2vec开始，说下GPT庞大的家族系谱( 十 )

推荐阅读

王者荣耀怎么隐身登陆，王者荣耀怎么设置隐身登陆

『网闻』闺蜜笑话我，我说了一句话让她立马安静了……，网易云热评：一直没人追

「斯利廷」顶级三星火速跳水！骁龙865＋120Hz，直降一千能成为真香机吗？

「清平乐」《清平乐》禾儿忠诚、曹后理性、张妣晗感性，你最欣赏哪一个？

聪明的人财|里昂：十分相信恒大的降负债能力，众专业投行力挺恒大

红茶菌功效,莲子心的功效与作用

大盘@操盘丨目前60-90分钟顶背离依然有效，“五一”节之前A股会怎么走？

温碧霞 |网友迪士尼偶遇54岁温碧霞，戴可爱发箍吹泡泡，自带专业摄影师

央视：包括退休后重回岗位者，英国19名抗疫医务人员去世

老人补钙别过量，否则适得其反

嗓子不舒服不要乱吃抗菌消炎药

银行“金饭碗”怎么突然不香了

末位评论@网友：类似高考查分，希望渺茫，热议孙杨上诉！舆论已彻底转向

教师不得通过微信QQ布置作业各地规定相继出台

金山翠芽价格金山翠芽多少钱一斤？

数说科技数码5G视频手机充电哪家强？数码大V实测在线揭秘

黑科技篮子 3大趋势不可逆，1药网创始人于刚：医药健康将是中国第一大产业

青年|王者荣耀：难怪你打不过对面，射手之间的克制链，你了解多少

穿衣搭配|金建希就任后罕见底气不足！穿衬衫陪总统老公像秘书，气质上大变

生肖|生肖鼠：近期5天，将有“痛哭流涕”的大事件，佛说一切都是命运