Comparative Analysis of Transformer-Based Models for Patent Summarization
DOI:
https://doi.org/10.53032/tvcr/2025.v7n2.43Keywords:
Patent Summarization, Abstractive Summarization, Transformer Models, BigBirdPegasus, PEGASUS, BART, Longformer (LED), Hugging Face Transformers, ROUGE Evaluation Metrics, Fine-Tuning Pretrained ModelsAbstract
Patent documents contain critical technical and legal information but are often lengthy and complex, making it difficult for researchers and businesses to extract key information efficiently. This research paper compares the performance of four transformer-based models - PEGASUS, BART, LED, and BigBirdPegasus for abstractive summarization of patents using the BIGPATENT dataset (subset "d" - Textiles; Paper patents), explores the architectural differences among these models, discusses the training strategies used and examines their implications for improving automated patent summarization. Each model was fine-tuned under identical training conditions, using 10% of the training dataset as per the hardware constraints and their performance was evaluated using ROUGE metrics. The results provide insights into which model is best suited for summarizing patent documents efficiently.
References
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. (2020). PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. Proceedings of the 37th International Conference on Machine Learning (ICML), 113, 11328-11339.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 7871-7880.
Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., & Ahmed, A. (2020). Big Bird: Transformers for Longer Sequences. Advances in Neural Information Processing Systems (NeurIPS), 33, 17283-17297.
Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv preprint arXiv:2004.05150.
Sharma, A., Li, Y., Wang, W. Y., & Mitamura, T. (2019). BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization of Patent Documents. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2204-2213.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, vol. 30, pp. 5998–6008.
Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
Valli, P. S., Satwika, N. U., Swaroop, S. V., Prasad, V. V. L., & Lakshmi, B. V. (2023). BigBirdPegasus-based Abstractive Multi-Document Summarization.
Eduard Hovy and Chin-Yew Lin. 1998. Automated Text Summarization and the Summarist System. In TIPSTER TEXT PROGRAM PHASE III: Proceedings of a Workshop held at Baltimore, Maryland, October 13-15, 1998, pages 197–214, Baltimore, Maryland, USA. Association for Computational Linguistics.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 The Voice of Creative Research
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
