Created by: golovenkov
Description
Add several ranking matching models for retrieval-based multi-turn response selection:
-
Sequential Matching Network (SMN) [Wu et al., 2017]
-
Deep Attention Matching Network (DAM) [Zhou et al., 2018]
-
Deep Attention Matching Network with Universal Sentence Encoder (DAM-USE-T). This is our new proposed neural ranking architecture based on [Zhou et al., 2018] and [Cer D. et al., 2018]. The model DAM-USE-T outperforms the current state-of-the-art models for the Ubuntu Dialogue Corpus v1 [Lowe et al., 2015] with the test metrics:
0.7957/0.8932/0.9734The DAM-USE-T has high test metrics for the Ubuntu Dialogue Corpus v2 [Lowe et al., 2017] as well:0.7446/0.8677/0.9738.
Scores for Ubuntu Dialogue Corpus v1 (test):
| Model | R@1 | R@2 | R@5 |
|---|---|---|---|
| SMN last [Wu et al., 2017] | 0.723 | 0.842 | 0.956 |
| SMN last (DeepPavlov) | 0.759 | 0.871 | 0.968 |
| DAM [Zhou et al., 2018] | 0.767 | 0.874 | 0.969 |
| DAM (DeepPavlov) | 0.779 | 0.880 | 0.970 |
| MRFN-FLS [Tao et al., 2019] | 0.786 | 0.886 | 0.976 |
| IMN [Gu et al, 2019] | 0.777 | 0.880 | 0.974 |
| IMN Ensemble [Gu et al, 2019] | 0.794 | 0.893 | 0.978 |
| DAM-USE-T (DeepPavlov) | 0.7957 | 0.8932 | 0.9734 |
Scores for Ubuntu Dialogue Corpus v2 (test):
| Model | R@1 | R@2 | R@5 |
|---|---|---|---|
| SMN last [Wu et al., 2017] | - | - | - |
| SMN last (DeepPavlov) | 0.6791 | 0.8149 | 0.9563 |
| DAM [Zhou et al., 2018] | - | - | - |
| DAM (DeepPavlov) | 0.7154 | 0.8366 | 0.9633 |
| MRFN-FLS [Tao et al., 2019] | - | - | - |
| IMN [Gu et al, 2019] | 0.771 | 0.886 | 0.979 |
| IMN Ensemble [Gu et al, 2019] | 0.791 | 0.899 | 0.982 |
| DAM-USE-T (DeepPavlov) | 0.7446 | 0.8677 | 0.9738 |
UPD: added pre-trained models (SMN, DAM, DAM-USE-T) for both Ubuntu Corpus v1 and v2 datasets. There are 6 models in total, see ranking configs, for example, pre-trained model DAM-USE-T for Ubuntu-v1 is here
References
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909.
Ryan Thomas Lowe, Nissan Pow, Iulian Vlad Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau. 2017. Training end-to-end dialogue systems with the ubuntu dialogue corpus. Dialogue and Discourse 8(1).
Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu and Hua Wu. 2018. Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1118-1127, ACL. http://aclweb.org/anthology/P18-1103
Yu Wu, Wei Wu, Ming Zhou, and Zhoujun Li. 2017. Sequential match network: A new architecture for multi-turn response selection in retrieval-based chatbots. In ACL, pages 372–381.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, Ray Kurzweil. 2018a. Universal Sentence Encoder for English. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Pages 169–174. http://aclweb.org/anthology/D18-2029
Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. Multi-Representation Fusion Network for Multi-turn Response Selection in Retrieval-based Chatbots. In WSDM'19. https://dl.acm.org/citation.cfm?id=3290985
Gu, Jia-Chen & Ling, Zhen-Hua & Liu, Quan. (2019). Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. https://arxiv.org/abs/1901.01824