Literature Survey: Llm Code Generation

Domain: Modern Software Engineering
Topic Search: llm code generation
Timeframe: 2023 - 2026

This is a curated survey of recent publications focusing on llm code generation. Results are filtered for top-tier journals and prominent conferences.

📚 Curated Peer-Reviewed Publications

1. LLM-based and Retrieval-Augmented Control Code Generation

Venue: LLM4CODE@ICSE | Year: 2024 | Citations: 92 Authors: Heiko Koziolek, Sten Grüner, Rhaban Hark, Virendra Ashiwal, Sofia Linsbauer et al.

Control code is designed and implemented for industrial automation applications that manage power plants, petrochemical processes, or steel production. Popular large language models (LLM) can synthesize low-level control code in the Structured Text programming notation according to the standard IEC 61131-3, but are not aware of proprietary control code function block libraries, which are often used in practice. To automate control logic implementation tasks, we proposed a retrieval-augmented control code generation method that can integrate such function blocks into the generated code. With this method control engineers can benefit from the code generation capabilities of LLMs, re-use proprietary and well-tested function blocks, and speed up typical programming tasks significantly. We have evaluated the method using a prototypical implementation based on GPT-4, LangChain, Open-PLC, and the open-source OSCAT function block library. In several spot sample tests, we successfully generated IEC 61131-3 ST code that integrated the desired function blocks, could be compiled, and validated through simulations.CCS CONCEPTS• Software and its engineering Automatic programming; Command and control languages; • Applied computing → Computer-aided design; • Computing methodologies → Natural language processing.

2. CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation

Venue: LLM4Code@ICSE | Year: 2025 | Citations: 90 Authors: Jinjun Peng, Leyi Cui 0001, Kele Huang, Junfeng Yang, Baishakhi Ray et al.

Large Language Models (LLMs) have significantly aided developers by generating or assisting in code writing, enhancing productivity across various tasks. While identifying incorrect code is often straightforward, detecting vulnerabilities in code is more challenging, especially for developers with limited security knowledge, which poses considerable security risks of using LLM-generated code and underscores the need for robust evaluation benchmarks that assess both functional correctness and security. Current security evaluation benchmarks like CyberSecEval and SecurityEval are hindered by unclear and impractical specifications, failing to assess both functionality and security accurately. To tackle these deficiencies, we introduce CWEval, a novel outcome-driven evaluation framework designed to enhance the evaluation of secure code generation by LLMs. This framework not only assesses code functionality but also its security simultaneously with high-quality task specifications and outcome-driven test oracles which provides high accuracy. Coupled with CWEval-bench, a multilingual, security-critical coding benchmark, CWEval provides a rigorous empirical security evaluation on LLM-generated code, overcoming the shortcomings of previous benchmarks. Through our evaluations, CWEval reveals a notable portion of functional but insecure code generated by LLMs, and shows a serious inaccuracy of previous evaluations, ultimately contributing significantly to the field of secure code generation. We open-source our artifact at: https://github.com/Co1lin/CWEval.

3. LLM-based Control Code Generation using Image Recognition

Venue: LLM4CODE@ICSE | Year: 2024 | Citations: 37 Authors: Heiko Koziolek, Anne Koziolek

LLM-based code generation could save significant manual efforts in industrial automation, where control engineers manually produce control logic for sophisticated production processes. Previous attempts in control logic code generation lacked methods to interpret schematic drawings from process engineers. Recent LLMs now combine image recognition, trained domain knowledge, and coding skills. We propose a novel LLM-based code generation method that generates IEC 61131-3 Structure Text control logic source code from Piping-and-Instrumentation Diagrams (P&IDs) using image recognition. We have evaluated the method in three case study with industrial P&IDs and provide first evidence on the feasibility of such a code generation besides experiences on image recognition glitches.CCS CONCEPTS• Software and its engineering→Automatic programming; Command and control languages; • Applied computing → Computer-aided design; • Computing methodologies →Natural language processing.

4. Learn to Code Sustainably: An Empirical Study on Green Code Generation

Venue: LLM4CODE@ICSE | Year: 2024 | Citations: 24 Authors: Tina Vartziotis, Ippolyti Dellatolas, George Dasoulas, Maximilian Schmidt, Florian Schneider et al.

The increasing use of information technology has led to a significant share of energy consumption and carbon emissions from data centers. These contributions are expected to rise with the growing demand for big data analytics, increasing digitization, and the development of large artificial intelligence (AI) models. The need to address the environmental impact of software development has led to increased interest in green (sustainable) coding and claims that the use of AI models can lead to energy efficiency gains. Here, we provide an empirical study on green code and an overview of green coding practices, as well as metrics used to quantify the sustainability awareness of AI models. In this framework, we evaluate the sustainability of auto-generated code. The auto-generated code considered in this study is produced by generative commercial AI language models, GitHub Copilot, OpenAI ChatGPT-3, and Amazon CodeWhisperer. Within our methodology, in order to quantify the sustainability awareness of these AI models, we propose a definition of the code’s “green capacity”, based on certain sustainability metrics. We compare the performance and green capacity of human-generated code and code generated by the three AI language models in response to easy-to-hard problem statements. Our findings shed light on the current capacity of AI models to contribute to sustainable software development.

5. Code Roulette: How Prompt Variability Affects LLM Code Generation

Venue: LLM4CODE@ICSE | Year: 2026 | Citations: 10 Authors: Andrei Paleyes, Diana Robinson, Radzim Sendyka, Christian Cabrera 0001, Neil D. Lawrence et al.

Code generation is one of the most active areas of application of Large Language Models (LLMs). While LLMs lower barriers to writing code and accelerate development process, the overall quality of generated programs depends on the quality of given prompts. Specifically, functionality and quality of generated code can be sensitive to user’s background and familiarity with software development. It is therefore important to quantify LLM’s sensitivity to variations in the input. To this end we propose an evaluation pipeline for LLM code generation with a focus on measuring sensitivity to prompt augmentations, completely agnostic to a specific programming tasks and LLMs, and thus widely applicable. We provide extensive experimental evidence illustrating utility of our method and share our code for the benefit of the community.

6. Semantically Aligned Question and Code Generation for Automated Insight Generation

Venue: LLM4CODE@ICSE | Year: 2024 | Citations: 7 Authors: Ananya Singha, Bhavya Chopra, Anirudh Khatry, Sumit Gulwani, Austin Z. Henley et al.

Automated insight generation is a common tactic for helping knowledge workers, such as data scientists, to quickly understand the potential value of new and unfamiliar data. Unfortunately, automated insights produced by large-language models can generate code that does not correctly correspond (or align) to the insight. In this paper, we leverage the semantic knowledge of large language models to generate targeted and insightful questions about data and the corresponding code to answer those questions. Then through an empirical study on data from Open-WikiTable, we show that embeddings can be effectively used for filtering out semantically unaligned pairs of question and code. Additionally, we found that generating questions and code together yields more diverse questions.

7. From Theory to Practice: Code Generation Using LLMs for CAPEC and CWE Frameworks

Venue: LLM4Code@ICSE | Year: 2025 | Citations: 0 Authors: Murtuza Shahzad, Joseph Wilson, Ibrahim Al Azher, Hamed Alhoori, Mona Rahimi et al.

No abstract available.

⚡ Latest Pre-Prints

1. Context Engineering for Multi-Agent LLM Code Assistants Using Elicit, NotebookLM, ChatGPT, and Claude Code

Published: 2025-08-09 Authors: Muhammad Haseeb

Large Language Models (LLMs) have shown promise in automating code generation and software engineering tasks, yet they often struggle with complex, multi-file projects due to context limitations and knowledge gaps. We propose a novel context engineering workflow that combines multiple AI components: an Intent Translator (GPT-5) for clarifying user requirements, an Elicit-powered semantic literature retrieval for injecting domain knowledge, NotebookLM-based document synthesis for contextual understanding, and a Claude Code multi-agent system for code generation and validation. Our integrated approach leverages intent clarification, retrieval-augmented generation, and specialized sub-agents orchestrated via Claude’s agent framework. We demonstrate that this method significantly improves the accuracy and reliability of code assistants in real-world repositories, yielding higher single-shot success rates and better adherence to project context than baseline single-agent approaches. Qualitative results on a large Next.js codebase show the multi-agent system effectively plans, edits, and tests complex features with minimal human intervention. We compare our system with recent frameworks like CodePlan, MASAI, and HyperAgent, highlighting how targeted context injection and agent role decomposition lead to state-of-the-art performance. Finally, we discuss the implications for deploying LLM-based coding assistants in production, along with lessons learned on context management and future research directions.

2. Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation

Published: 2024-07-29 Authors: Manish Bhattarai, Javier E. Santos, Shawn Jones, Ayan Biswas, Boian Alexandrov, Daniel O’Malley

The advent of large language models (LLMs) has significantly advanced the field of code translation, enabling automated translation between programming languages. However, these models often struggle with complex translation tasks due to inadequate contextual understanding. This paper introduces a novel approach that enhances code translation through Few-Shot Learning, augmented with retrieval-based techniques. By leveraging a repository of existing code translations, we dynamically retrieve the most relevant examples to guide the model in translating new code segments. Our method, based on Retrieval-Augmented Generation (RAG), substantially improves translation quality by providing contextual examples from which the model can learn in real-time. We selected RAG over traditional fine-tuning methods due to its ability to utilize existing codebases or a locally stored corpus of code, which allows for dynamic adaptation to diverse translation tasks without extensive retraining. Extensive experiments on diverse datasets with open LLM models such as Starcoder, Llama3-70B Instruct, CodeLlama-34B Instruct, Granite-34B Code Instruct, and Mixtral-8x22B, as well as commercial LLM models like GPT-3.5 Turbo and GPT-4o, demonstrate our approach’s superiority over traditional zero-shot methods, especially in translating between Fortran and CPP. We also explored varying numbers of shots i.e. examples provided during inference, specifically 1, 2, and 3 shots and different embedding models for RAG, including Nomic-Embed, Starencoder, and CodeBERT, to assess the robustness and effectiveness of our approach.

3. Evaluating LLM-Based Regression Test Generation

Published: 2025-01-19 Authors: Jing Liu, Seongmin Lee, Eleonora Losiouk, Marcel Böhme

Large Language Models (LLMs) have shown tremendous promise in automated software engineering. In this paper, we investigate LLMs for just-in-time regression test generation for programs, like parsers, interpreters, or compilers, that take highly structured, human-readable inputs. When a bug fix or code change is committed, the repository (as part of CI/CD) runs an LLM for a few minutes to generate regression tests that exercise the changed code and potentially trigger bugs. We frame LLM-based regression test generation as a machine translation task that takes the developer-provided commit message, the code change, and the input format name (e.g., XML), and produces regression tests for the described change in that format. Testing 72 commits to Mujs, Libxml2, Poppler, JerryScript, Z3, PHP, JQ, and MicroPython, our feedback-directed, zero-shot prototype Cleverest performed well, even without the code change. In under 2 minutes on average, Cleverest found as many bugs as the state-of-the-art directed greybox fuzzer WAFLGo did in 24 hours, even though WAFLGo started with a commit-reaching seed corpus in most cases. Using the Cleverest-generated tests as a seed corpus in coverage-guided greybox fuzzing doubles the number of bugs found; we call this integration ClevFuzz. We also find that some commit messages are more expressive than others, thus we ask how it impacts Cleverest’s effectiveness. Cleverest picks up on the change intention: e.g., given a commit message that a patch changes how floating point variables are treated in the Mujs JavaScript interpreter, Cleverest generates JavaScript programs that contain floating point variables. Minimally changing commit messages to reduce or increase their information substantially impacts effectiveness: adding 17 words on average (max. 43) to make ineffective commit messages more expressive significantly increased the number of bugs found.

4. An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code

Published: 2026-04-25 Authors: Jelena Ilić Vulićević

Large language models (LLMs) have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysis. However, most prior work relies on cloud-based models or specialized hardware, limiting practical applicability in privacy-sensitive or resource-constrained environments. In this paper, we present a systematic empirical evaluation of two locally deployed LLMs, LLaMA 3.2 and Mistral, for real-world Python bug detection using the BugsInPy benchmark. We evaluate 349 bugs across 17 projects using a zero-shot prompting approach at the function level and an automated keyword-based evaluation framework. Our results show that locally executed models achieve accuracy between 43% and 45%, while producing a large proportion of partially correct responses that identify problematic code regions without pinpointing the exact fix. Performance varies significantly across projects, highlighting the importance of codebase characteristics. The results demonstrate that local models can identify a meaningful share of bugs, though precise localization remains difficult for locally executed LLMs, particularly when handling complex and context dependent bugs in realistic development scenarios.

5. VerilogReader: LLM-Aided Hardware Test Generation

Published: 2024-06-03 Authors: Ruiyang Ma, Yuxin Yang, Ziqian Liu, Jiaxi Zhang, Min Li, Junhua Huang, Guojie Luo

Test generation has been a critical and labor-intensive process in hardware design verification. Recently, the emergence of Large Language Model (LLM) with their advanced understanding and inference capabilities, has introduced a novel approach. In this work, we investigate the integration of LLM into the Coverage Directed Test Generation (CDG) process, where the LLM functions as a Verilog Reader. It accurately grasps the code logic, thereby generating stimuli that can reach unexplored code branches. We compare our framework with random testing, using our self-designed Verilog benchmark suite. Experiments demonstrate that our framework outperforms random testing on designs within the LLM’s comprehension scope. Our work also proposes prompt engineering optimizations to augment LLM’s understanding scope and accuracy.

6. BanglaForge: LLM Collaboration with Self-Refinement for Bangla Code Generation

Published: 2025-12-22 Authors: Mahir Labib Dihan, Sadif Ahmed, Md Nafiu Rahman

Bangla is a low-resource language for code generation, lacking large-scale annotated datasets and tools to transform natural language specifications into executable programs. This makes Bangla-to-code generation a challenging task requiring innovative solutions. To address this, we introduce BanglaForge, a novel framework for generating code from Bangla function descriptions. BanglaForge leverages a retrieval-augmented dual-model collaboration paradigm with self-refinement, combining in-context learning, llm-based translation, systematic prompt engineering, and iterative self-refinement based on execution feedback, where a coder generates initial solutions and a reviewer enhances them for robustness. On the BLP-2025 Bangla Code Generation benchmark, BanglaForge achieves a competitive Pass@1 accuracy of 84.00%, demonstrating the effectiveness of retrieval, model collaboration, and self-refinement for low-resource Bangla code generation.

7. Security Degradation in Iterative AI Code Generation – A Systematic Analysis of the Paradox

Published: 2025-05-19 Authors: Shivani Shukla, Himanshu Joshi, Romilla Syed

The rapid adoption of Large Language Models(LLMs) for code generation has transformed software development, yet little attention has been given to how security vulnerabilities evolve through iterative LLM feedback. This paper analyzes security degradation in AI-generated code through a controlled experiment with 400 code samples across 40 rounds of “improvements” using four distinct prompting strategies. Our findings show a 37.6% increase in critical vulnerabilities after just five iterations, with distinct vulnerability patterns emerging across different prompting approaches. This evidence challenges the assumption that iterative LLM refinement improves code security and highlights the essential role of human expertise in the loop. We propose practical guidelines for developers to mitigate these risks, emphasizing the need for robust human validation between LLM iterations to prevent the paradoxical introduction of new security issues during supposedly beneficial code “improvements”.

8. Anka: A Domain-Specific Language for Reliable LLM Code Generation

Published: 2025-12-29 Authors: Saif Khalfan Saif Al Mazrouei

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet they exhibit systematic errors on complex, multi-step programming tasks. We hypothesize that these errors stem from the flexibility of general-purpose languages, which permits multiple valid approaches and requires implicit state management. To test this hypothesis, we introduce Anka, a domain-specific language (DSL) for data transformation pipelines designed with explicit, constrained syntax that reduces ambiguity in code generation. Despite having zero prior training exposure to Anka, Claude 3.5 Haiku achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems. Critically, Anka demonstrates a 40 percentage point accuracy advantage over Python on multi-step pipeline tasks (100% vs. 60%), where Python’s flexible syntax leads to frequent errors in operation sequencing and variable management. Cross-model validation with GPT-4o-mini confirms this advantage (+26.7 percentage points on multi-step tasks). Our results demonstrate that: (1) LLMs can learn novel DSLs entirely from in-context prompts, achieving near-native accuracy; (2) constrained syntax significantly reduces errors on complex tasks; and (3) domain-specific languages purposefully designed for LLM generation can outperform general-purpose languages on which the LLM has extensive training. We release the complete language implementation, benchmark suite, and evaluation framework to facilitate further research.

9. Beyond Code Generation: Assessing Code LLM Maturity with Postconditions

Published: 2024-07-19 Authors: Fusen He, Juan Zhai, Minxue Pan

Most existing code Large Language Model (LLM) benchmarks, e.g., EvalPlus, focus on the code generation tasks. Namely, they contain a natural language description of a problem and ask the LLM to write code to solve the problem. We argue that they do not capture all capabilities needed to assess the quality of a code LLM. In this paper, we propose a code LLM maturity model, based on the postcondition generation problem, to access a more complete set of code LLM capabilities. We choose the postcondition generation problem as it requires the code LLM to understand the code including semantics, natural language, and also have the capability to generate unambiguous postconditions in programming languages (i.e., the generation capablity). Moreover, postconditions have various types, requiring different levels of these capabilities, making it suitable to evaluate the maturity of the code LLM. Based on our designed maturity model, we augment the EvalPlus dataset to a postcondition testing benchmark, and evaluated several open-sourced models. Our results highlight the necessary improvements needed for better LLMs for code. Code: https://github.com/MatureModel/PostcondGen

10. AUTOGENICS: Automated Generation of Context-Aware Inline Comments for Code Snippets on Programming Q&A Sites Using LLM

Published: 2024-08-27 Authors: Suborno Deb Bappon, Saikat Mondal, Banani Roy

Inline comments in the source code facilitate easy comprehension, reusability, and enhanced readability. However, code snippets in answers on Q&A sites like Stack Overflow (SO) often lack comments because answerers volunteer their time and often skip comments or explanations due to time constraints. Existing studies show that these online code examples are difficult to read and understand, making it difficult for developers (especially novices) to use them correctly and leading to misuse. Given these challenges, we introduced AUTOGENICS, a tool designed to integrate with SO to generate effective inline comments for code snippets in SO answers exploiting large language models (LLMs). Our contributions are threefold. First, we randomly select 400 answer code snippets from SO and generate inline comments for them using LLMs. We then manually evaluate these comments’ effectiveness using four key metrics: accuracy, adequacy, conciseness, and usefulness. Overall, LLMs demonstrate promising effectiveness in generating inline comments for SO answer code snippets. Second, we surveyed 14 active SO users to perceive the effectiveness of these inline comments. The survey results are consistent with our previous manual evaluation. However, according to our evaluation, LLMs-generated comments are less effective for shorter code snippets and sometimes produce noisy comments. Third, to address the gaps, we introduced AUTOGENICS, which extracts additional context from question texts and generates context-aware inline comments. It also optimizes comments by removing noise (e.g., comments in import statements and variable declarations). We evaluate the effectiveness of AUTOGENICS-generated comments using the same four metrics that outperform those of standard LLMs. AUTOGENICS might (a) enhance code comprehension, (b) save time, and improve developers’ ability to learn and reuse code more accurately.

🧠 Architectural & Methodological Insights

Trajectory: The subfield is converging on knowledge‑rich, context‑aware pipelines that couple powerful LLMs with external artefacts (retrieval corpora, visual inputs, DSL constraints) while expanding evaluation beyond raw correctness to security, sustainability, and robustness. Multi‑agent orchestration and prompt engineering are becoming integral system components, indicating a move from monolithic “LLM‑as‑coder” to LLM‑as‑coordinator architectures.

🚀 Critical Research Gaps

1. Unified Benchmark for Multi‑modal, Retrieval‑augmented, and Secure Code Generation

   • Motivation: Existing benchmarks focus on a single dimension (e.g., functional correctness in CWEval or prompt sensitivity in Code Roulette). No comprehensive suite evaluates a system that simultaneously consumes visual schematics, retrieves proprietary APIs, and is judged on security and sustainability.
   • Gap: Lack of a holistic, reproducible benchmark that stresses all these axes together, making it hard to compare RAG‑based pipelines against pure LLM baselines.

2. Theoretical Understanding of Iterative Refinement’s Security Degradation

   • Motivation: Security Degradation in Iterative AI Code Generation empirically shows vulnerability growth across refinement cycles, but provides no formal model of why or how LLMs introduce new bugs.
   • Gap: Absence of a principled framework (e.g., probabilistic or information‑theoretic) that predicts security risk accumulation during multi‑turn generation, hindering the design of provably safe refinement strategies.

3. Prompt‑Robustness Guarantees for Domain‑Specific Languages

   • Motivation: Anka demonstrates that a DSL can dramatically improve accuracy, yet the evaluation is limited to zero‑shot prompts. It remains unclear how sensitive Anka‑oriented pipelines are to prompt variations or ambiguous specifications.
   • Gap: No systematic study of prompt‑induced failure modes for DSL‑guided generation, nor methods to automatically normalize or verify prompts before feeding them to the model.

4. Energy‑aware Generation Algorithms and Their Trade‑offs

   • Motivation: Learn to Code Sustainably quantifies “green capacity” post‑hoc but does not integrate energy considerations into the generation process itself.
   • Gap: Missing algorithmic techniques (e.g., token‑budgeted decoding, model‑size selection) that explicitly optimize for carbon emissions while preserving functional and security quality.

💡 High‑Impact Open Problems

1. Design and Release of a Multi‑Modal, Retrieval‑Augmented, Secure Code Generation Benchmark (MARS‑Bench)

   • Problem Statement: Construct a benchmark suite that includes (a) textual problem statements, (b) associated engineering diagrams (e.g., P&IDs), (c) a curated retrieval corpus of proprietary function‑block libraries, and (d) security/sustainability annotations (CWE IDs, energy‑impact scores). Provide evaluation metrics that jointly measure compile‑success, functional correctness, vulnerability count, and estimated carbon cost.
   • Research Contributions: (i) Dataset curation pipeline; (ii) Metric formulation for composite scoring; (iii) Baseline implementations (pure LLM, RAG, multi‑agent) and open‑source evaluation toolkit.

2. Formal Model of Security Risk Propagation in Iterative LLM Code Refinement

   • Problem Statement: Develop a probabilistic model (e.g., Bayesian network) that captures how each refinement step updates the distribution over latent vulnerability classes given the prompt, prior code, and model temperature. Validate the model against the empirical data from Security Degradation in Iterative AI Code Generation.
   • Research Contributions: (i) Theoretical analysis of risk accumulation; (ii) Derivation of safe‑refinement policies (e.g., bounded‑iteration, adversarial verification checkpoints); (iii) Implementation of a verification‑in‑the‑loop system that halts refinement when predicted risk exceeds a threshold.

3. Prompt Normalization and Verification Layer for DSL‑Guided Code Generation

   • Problem Statement: Create a preprocessing module that (a) parses natural‑language specifications into a canonical intermediate representation (IR) aligned with the DSL’s grammar (e.g., Anka), (b) detects ambiguities or underspecifications, and (c) either resolves them via clarification queries or rejects the prompt. Evaluate the impact on generation accuracy and robustness across a suite of prompt perturbations.
   • Research Contributions: (i) IR design bridging NL and DSL; (ii) Ambiguity detection algorithm (e.g., constraint‑solving over the DSL grammar); (iii) Empirical study showing reduction in prompt‑sensitivity variance compared to raw prompting.

4. Energy‑Constrained Decoding Strategies for Green Code Synthesis

   • Problem Statement: Extend standard decoding (e.g., nucleus, beam) with a carbon budget constraint that dynamically selects model size, temperature, and number of retrieval examples to stay within a predefined energy envelope while meeting a target functional‑security score.
   • Research Contributions: (i) Cost model linking token generation to estimated energy usage (leveraging hardware‑level profiling); (ii) Multi‑objective optimization algorithm (e.g., Pareto‑front search) for decoding; (iii) Empirical evaluation on the Learn to Code Sustainably dataset showing trade‑offs between green capacity and correctness/security.

These problems are deliberately scoped to be tractable for a PhD‑level effort yet promise to close pivotal gaps identified in the current literature, pushing the field toward more reliable, secure, and sustainable LLM‑driven code generation.