LLM-Based Code Explanation Quality on Legacy Codebases

LLM-based code explanation can accelerate legacy comprehension, but readable prose is not proof that a model has recovered a system’s intent. The boundary to keep in mind is between spotting familiar code patterns and explaining the business rules built into unusual constructs. Trust depends on knowing where that boundary is and treating generated explanations as drafts that people who know the domain must check.

Why legacy codebases are structurally hard to explain, even before AI enters the picture

The engineer trying to explain a legacy system often inherits code without the people who knew why it was written that way. Much of the intent was never recorded. It survives implicitly in tightly coupled modules, obsolete frameworks, and conventions that once made sense to their authors. Reverse engineering therefore requires more than describing what individual statements do. It requires recovering the relationship between those statements and the institutional decisions they implement.

Inaccurate records create a different burden from missing ones, because they offer an account of the system that may contradict its behavior. An absent explanation leaves an obvious gap. A stale explanation can conceal it, particularly when the reader lacks the background needed to recognize the discrepancy. Code comprehension then becomes a task of reconciling competing evidence.

MITRE’s research team frames legacy systems in MUMPS and mainframe assembly as presenting simultaneous challenges in efficiency, maintenance, staffing, and security. Teams need explanations to maintain and migrate systems, but few people have the knowledge needed to write them. This is what makes automated documentation attractive. Its value begins with making otherwise inaccessible code easier to examine.

What LLMs can genuinely do for legacy comprehension

LLMs can generate readable intent summaries from code that has no other documentation, making function-level explanation feasible at a scale that manual work often cannot match. Agentic assistants including Claude Code, Cursor, Aider, Devin, and OpenHands also extend beyond completion into multi-step migration workflows. They can read a repository, generate documentation, translate modules, write tests, and iterate until tests pass.

Forte Group deployed fine-tuned LLM agents for function-level intent analysis on a legacy insurance-policy management platform. The agents generated natural-language summaries for thousands of functions, indexed with the source code in a semantic browser. Product owners and new engineers could explore functionality through those descriptions. The format was central to the benefit. A summary attached to a specific function makes the explanation easier to locate, compare with the implementation, and review.

The same framework identified and consolidated 86 redundant implementations of date-range filtering in a monolithic.NET application. LLMs suggested a shared utility layer and wrote refactored calls along with unit tests for the team to consider. Forte also used an LLM-driven translation pipeline to convert core business logic in a Delphi-based logistics engine into Go and TypeScript services, reducing a nine-month migration plan to five months. These cases demonstrate practical uses beyond generating prose. They do not, however, establish that every inferred business rule was correct or that migration speed measures explanation quality.

Earlier workflows often required teams to submit code piecemeal, losing relationships between modules as they divided the repository. Very large windows can preserve more of that context after irrelevant files are filtered out. This expands what the model can inspect together. It reduces one structural reason an otherwise useful explanation may be incomplete.

Where pattern-matching breaks down — the three failure modes that count in practice

The first failure mode is underrepresented legacy languages. A model trained predominantly on modern, high-resource languages has less basis for interpreting constructs it has rarely encountered. The output can follow the familiar shape of a technical explanation while assigning the wrong meaning to the code. This is a limitation of pattern continuation rather than a problem that clearer wording alone can solve.

Diggs et al. identify substantial uncertainty around LLM comprehension of MUMPS and IBM mainframe Assembly Language Code, or ALC, particularly in government applications. Their study finds ALC more challenging than MUMPS, showing that even low-resource languages do not form a uniform category. Some government programs also exceed commercially available models’ context windows. Representation and context are distinct constraints here. A model may struggle because it has little grounding in the language, because it cannot inspect the relevant code together, or because both conditions apply.

The second failure mode is entangled and implicit business logic. A model tends to explain code through recognizable patterns, but an inconsistent codebase may make accidental conventions look deliberate. Institutional decisions, regulatory interpretations, and vendor behaviors may also be absent from training data, whether they arose after the training cutoff or were never recorded publicly. Unless that context is supplied, the model’s account cannot reliably recover it from familiarity with syntax.

In the migration failure described in the brief, a model misread COBOL’s REDEFINES clause and treated a packed decimal, COMP-3, field as a standard integer. The Java application that came out of this wrote corrupted binary data into database columns and caused referential integrity failures.

The third failure mode is documentation quality itself. Diggs et al. identify the circularity in asking models to explain systems whose documentation is missing or inaccurate, because accurate comments improve comprehension while inaccurate comments degrade it. More context is not automatically better context. Stale comments can steer the model toward a wrong account more strongly than no comments at all. Before subject-matter expert review, there is no reliable way to determine which inherited comments deserve trust, so simply supplying more of them does not resolve the problem.

Why bad explanations are hard to catch — the broken evaluation problem

Diggs et al. find that generated comments for MUMPS and ALC are generally hallucination-free, complete, readable, and useful, yet they also find that automated metrics cannot reliably establish comment quality. That is the central evaluation paradox. Favorable assessments of the output do not provide an automated method for recognizing equally good output in the next system. Neither code complexity measures nor reference-based metrics strongly correlate with quality in the study.

This gap prevents teams from treating an automated score as a production acceptance test. No widely accepted metrics exist for measuring the quality of code documentation through comments. Readability is only one part of the task. An explanation must also describe the relevant behavior correctly, and legacy systems often lack the trustworthy reference material needed to assess that correctness. Where documentation is absent or wrong, comparison against existing text cannot by itself settle the question.

Coding benchmark performance does not close this gap. Success on a well-defined translation task answers a narrower question than whether a model can explain an undocumented system’s intent. The distinction is especially sharp when the language is underrepresented and there is no established ground truth. A high score on clean COBOL migration problems does not establish reliable explanations of open-ended legacy behavior. Consequently, a model’s demonstrated coding ability cannot substitute for evaluation of the explanations a modernization program will actually use.

GitClear’s 2026 analysis of hundreds of millions of code changes from 2023–2026 found that cross-file function calls, a signal of reuse and structural coherence, declined by 35% as AI authorship rose. That finding concerns future maintainability. Its relevance is nevertheless clear. AI-assisted development can reduce occasions for consolidating, documenting, or retiring old code, leaving tomorrow’s readers with more of the structural problems that complicate legacy comprehension today.

How the gap looks different depending on the language and domain

COBOL and mainframe environments have purpose-built tooling that can provide more context than a general-purpose model receives from an isolated code fragment. IBM watsonx Code Assistant for Z, built on IBM’s Granite models, offers explanations of COBOL, PL/I, JCL, and REXX, with Assembler in preview. Its Z Understand capabilities include application analysis, business rules discovery, and dependency mapping. These capabilities broaden the evidence available for explanation. Copybook libraries, job control metadata, and VSAM data definitions can compensate for some missing context, although specialized tools still have blind spots on the oldest constructs.

Anthropic positioned Claude Code for COBOL modernization in February 2026, targeting exploration and analysis. IBM also describes a global logistics company that got much more work done through COBOL-to-Java transformation. Such results support the case for assisted modernization. They do not remove the need to distinguish productivity gains from evidence that a particular explanation is semantically correct.

The study examined an electronic health records system in MUMPS and found generated comments generally usable, but the evaluation gap prevents automated measures alone from establishing that usability. Scale adds another limit. Government codebases in MUMPS and assembly can exceed context windows even after filtering, so larger windows do not eliminate the need to divide the code. Mainframe assembly was also harder than MUMPS in the study, so even within this group, teams need to assess each language on its own.

Delphi and older.NET environments occupy a more favorable position because their constructs are more likely to match patterns represented in training data. Entangled business logic remains difficult regardless of the implementation language, and mixed stacks can combine familiar modules with less familiar legacy constructs. Trust therefore depends on the particular language, domain context, and available evidence.

What responsible use of LLM explanations actually requires in practice

The practical response is to treat generated explanations as reviewable drafts. Teams should keep people in charge of checking edge cases, domain-specific business logic, and security-critical code, and of validation. A clear account of a function is useful only if the relevant expert can check its claims against the implementation and the business rule. The model accelerates the preparation of that account without assuming the engineer’s responsibility for accepting it.

To make verification tractable, teams should structure explanations around specific, inspectable claims. Forte’s function-level summaries indexed in a semantic browser illustrate this principle. Product owners and subject-matter experts can review particular descriptions against the associated code instead of confronting an undifferentiated account of the repository. The unit of review stays concrete. This structure makes disagreement and missing context easier to address within the workflow.

Modernization programs should consequently remain AI-augmented. Agentic code tools and specialized migration platforms can be combined with machine-learning lifecycle evaluation and human review at each meaningful gate. Automated evaluation still has a role. Its limits must remain explicit, however, because neither successful translation nor passing tests establishes that every generated statement about intent is correct. Domain-informed review is especially important where the explanation will guide changes to implicit business rules.

For the lowest-resource, highest-risk systems, the appropriate label is hypothesis generation. Government MUMPS, pre-1985 COBOL with ALTER statements, and mainframe assembly without surviving subject-matter experts fall into this category. Generated accounts can narrow the search space. They cannot close it merely by sounding complete, and the absence of an expert does not increase the evidential strength of the model’s output.

Diggs et al. call for better evaluation metrics for LLM-generated documentation in legacy systems, and teams should treat their absence as an open risk. This is a reason to preserve the distinction between an explanation that helps an engineer investigate and one that has enough supporting evidence to guide a production decision.