Exploring Prompt Patterns for Effective Vulnerability Repair in Real-World Code by Large Language Models

Published

June 5, 2025

Author(s)

Yining Luo, Baobao Li, Anoop Singhal, Pei-Yu Tseng, Lan Zhang, Qingtian Zou, Xiaoyan Sun, Peng Liu

Abstract

Large Language Models (LLMs) have shown promise in automating code vulnerability repair, but their effectiveness in handling real-world code remains limited. This paper investigates the capability of LLMs, in repairing vulnerabilities and proposes a systematic approach to enhance their performance through specialized prompt engineering. Through extensive evaluation of 5826 code samples, we found that while LLMs successfully repair vulnerabilities in simple cases, they struggle with complex real-world code that involves intricate dependencies, contextual requirements, and multi-file interactions. To address these limitations, we first incorporated Control Flow Graphs (CFGs) as supplementary prompts, achieving a 14.4% success rate in fixing previously unresolvable vulnerabilities. Through analysis of repair failures, we identified three primary challenge categories and developed corresponding prompt patterns incorporating techniques such as granular contextual information provision and progressive code simplification. Evaluation on real-world projects demonstrated that our approach significantly improved LLMs' repair capabilities, achieving over 85% success rates across all identified challenge categories. Our findings suggest that while LLMs have inherent limitations in handling complex vulnerabilities independently, they can become effective tools for automated vulnerability repair when guided by carefully crafted prompts.

Proceedings Title

IWSPA '25: Proceedings of the 2025 ACM International Workshop on Security and Privacy Analytics

Conference Dates

June 4-6, 2025

Conference Location

Pittsburgh, PA, US

Conference Title

2025 ACM International Workshop on Security and Privacy Analytics (IWSPA '25)

Pub Type

Conferences

Download Paper

https://doi.org/10.1145/3716815.3729010

Local Download

Keywords

software vulnerabilities, Large Language Models, C++ Programming, prompt engineering

Citation

Luo, Y. , Li, B. , Singhal, A. , Tseng, P. , Zhang, L. , Zou, Q. , Sun, X. and Liu, P. (2025), Exploring Prompt Patterns for Effective Vulnerability Repair in Real-World Code by Large Language Models, IWSPA '25: Proceedings of the 2025 ACM International Workshop on Security and Privacy Analytics, Pittsburgh, PA, US, [online], https://doi.org/10.1145/3716815.3729010, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=959492 (Accessed June 19, 2025)