Loading...
How efficient is your clone? Evaluating LLM capabilities for runtime and memory-aware code selection
Date
2026-11-01
Abstract
Context: Selecting the most execution-efficient implementation from a bounded set of functionally equivalent code submissions is valuable, yet running code at scale is costly and environment-sensitive. Objective: We assess whether large language models (LLMs) can select the most efficient implementation without executing code, and how this ability varies by prompt design, language, model state, and efficiency aspect (runtime vs. memory). Method: We treat an LLM as a pairwise comparator within a deterministic, cost-aware evaluation protocol that recovers Top-1 and Top-3 using subquadratic comparisons. Six models (DeepSeek Coder, CodeLlama, StarCoder, CodeT5, CodeBERT, Claude) are evaluated on Project CodeNet and the Computer Language Benchmark Game (CLBG) across Python, Java, and C. Metrics are Top-1, Top-3, Mean Reciprocal Rank (MRR), and efficiency gap to optimal (eGap); inference is execution-free and deterministic. Results: On CodeNet, upgrading from a code-only prompt (𝑃1) to a code+features prompt (𝑃2) consistently improves decoder models: Top-1 rises from 38.39% to 48.95% for DeepSeek Coder, from 36.47% to 49.47% for CodeLlama, and from 35.14% to 50.32% for StarCoder, while eGap drops by 28%, 6%, and 16%, respectively. CodeT5 and CodeBERT change little. On CLBG, the same direction is visible but is moderated by frequent Top- 3 ceiling effects; DeepSeek Coder and StarCoder remain strongest under 𝑃2, whereas CodeLlama and CodeT5 show limited gains. Language effects are small and rarely significant on CodeNet but become more aspect dependent on CLBG. Lightweight LoRA fine-tuning yields selective improvements for decoder families and is unstable for the encoder baseline. Runtime typically improves Top-1 and MRR, whereas memory often reduces eGap. Conclusion: Findings suggest that prompt design — and, secondarily, lightweight fine-tuning — can measurably improve benchmark-grounded, execution-free runtime-/memory-aware selection for modern decoder LLMs, while language and aspect effects remain context-dependent. We distill actionable guidelines for prompting, metrics, aspect choice, model state, and protocol design, and we release prompts, seeds, code, and preprocessing scripts to support replication.
Supervisor
Description
Publisher
Elsevier
Citation
The Journal of Systems and Software 241, 112973
Collections
Files
ULRR Identifiers
Funding code
Funding Information
Sustainable Development Goals
External Link
License
Attribution-NonCommercial-ShareAlike 4.0 International