Entity-Guided Contrastive Learning for Multimodal Abstractive Sentence Summarization
Keywords:
multimodal summarization, abstractive sentence summarization, entity-guided learning, contrastive learning, cross-modal alignmentAbstract
Multimodal abstractive sentence summarization condenses a news sentence into a headline with the help of its accompanying photograph. Vision-guided models inject a global visual representation into the encoder, yet they treat the entities named in the source sentence as ordinary tokens, so the decoder receives little fine-grained grounding and the visual signal is easily swamped by the stronger linguistic signal. This paper presents an entity-guided contrastive framework built on BART. Knowledge-graph entities recognised in the source sentence are represented with pretrained TransE embeddings and encoded jointly with the text and with the image by two encoders that share parameters. A gated fusion module combines the two encoder outputs into a single memory for the decoder, and contrastive objectives align the resulting representations with each other and with the target-summary representation. On the 2,000 instances of the MMSS test set, and against a text-only variant trained from the same pretrained weights under the same recipe, the framework improves ROUGE-1 from 55.13 to 56.38, ROUGE-2 from 31.60 to 32.89 and ROUGE-L from 51.38 to 52.47, and a paired bootstrap places the ROUGE-1 gain outside the zero-difference interval. A diagnostic section locates the remaining headroom in the decoder rather than in the alignment: the contrastive representations do separate the two branches, but the decoder grants the visual block under two per cent of its cross-attention mass, which bounds what the alignment can contribute to the headline.
References
[1]Li, H., Zhu, J., Liu, T., Zhang, J., & Zong, C. (2018). Multi-modal sentence summarization with modality attention and image filtering. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (pp. 4152-4158).
[2]Jangra, A., Mukherjee, S., Jatowt, A., Saha, S., & Hasanuzzaman, M. (2023). A survey on multi-modal summarization. ACM Computing Surveys, 55(13s), Article 296, 1-36.
[3]Rush, A. M., Chopra, S., & Weston, J. (2015). A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 379-389).
[4]See, A., Liu, P. J., & Manning, C. D. (2017). Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (pp. 1073-1083).
[5]Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 7871-7880).
[6]Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2020). PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning (pp. 11328-11339).
[7]Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., & Zhou, M. (2020). ProphetNet: Predicting future n-gram for sequence-to-sequence pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 2401-2410).
[8]Yu, T., Dai, W., Liu, Z., & Fung, P. (2021). Vision guided generative pre-trained language models for multimodal abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 3995-4007).
[9]Xiao, M., Zhu, J., Lin, H., Zhou, Y., & Zong, C. (2023). CFSum coarse-to-fine contribution network for multimodal summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8538-8553).
[10]Liu, N., Sun, X., Yu, H., Zhang, W., & Xu, G. (2020). Multistage fusion with forget gate for multimodal summarization in open-domain videos. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (pp. 1834-1845).
[11]Liang, Y., Meng, F., Xu, J., Wang, J., Chen, Y., & Zhou, J. (2023). Summary-oriented vision modeling for multimodal abstractive summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (pp. 2934-2951).
[12]Zhu, J., Li, H., Liu, T., Zhou, Y., Zhang, J., & Zong, C. (2018). MSMO: Multimodal summarization with multimodal output. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 4154-4164).
[13]Palaskar, S., Libovicky, J., Gella, S., & Metze, F. (2019). Multimodal abstractive summarization for How2 videos. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 6587-6596).
[14]Zhang, Y., Liu, Y., Wu, S., Zhang, K., Liu, X., Liu, Q., & Chen, E. (2024). Leveraging entity information for cross-modality correlation learning: The entity-guided multimodal summarization. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 9851-9862).
[15]Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (pp. 8748-8763).
[16]Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. C. H. (2021). Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems 34 (pp. 9694-9705).
[17]Li, J., Li, D., Xiong, C., & Hoi, S. (2022). BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning (pp. 12888-12900).
[18]Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (pp. 1597-1607).
[19]He, B., Wang, J., Qiu, J., Bui, T., Shrivastava, A., & Wang, Z. (2023). Align and attend: Multimodal summarization with dual contrastive losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14867-14878).
[20]Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., & Yakhnenko, O. (2013). Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26 (pp. 2787-2795).
[21]Erkan, G., & Radev, D. R. (2004). LexRank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22, 457-479.
[22]Nallapati, R., Zhai, F., & Zhou, B. (2017). SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (pp. 3075-3081).
[23]Clarke, J., & Lapata, M. (2008). Global inference for sentence compression: An integer linear programming approach. Journal of Artificial Intelligence Research, 31, 399-429.
[24]Zhou, Q., Yang, N., Wei, F., & Zhou, M. (2017). Selective encoding for abstractive sentence summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (pp. 1095-1104).
[25]UzZaman, N., Bigham, J. P., & Allen, J. F. (2011). Multimodal summarization of complex sentences. In Proceedings of the 16th International Conference on Intelligent User Interfaces (pp. 43-52).
[26]Speer, R., Chin, J., & Havasi, C. (2017). ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (pp. 4444-4451).
[27]Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop (pp. 74-81).
[28]Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311-318).
[29] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating text generation with BERT. In Proceedings of the Eighth International Conference on Learning Representations.
