OpenAI expands tests of code behavior in low-resource programming languages
OpenAI has published evaluations for programming languages with fewer examples in training data. The work covers reproducing SQLite behavior across languages and migrating COBOL software. Testing whether matching inputs produce matching outputs goes beyond checking that a program compiles. The expanded evaluations address code generation and behavioral equivalence while keeping those results separate from the safe migration of an entire production application.
Artificial Intelligence··Morning
Less-represented languages get broader tests
OpenAI, the company behind ChatGPT, published new evaluations on October 19 for programming languages that have fewer examples in model training data.[1], [2]
The work covers software-engineering tasks, SQLite and migration from COBOL. Here, the shortage concerns examples of code in a particular language. It describes the material available for learning that language, rather than the memory or processing power of a computer running the software. The publication broadens the kinds of development tasks used to assess models across languages, with particular attention to whether generated programs actually carry out the intended operation.[1]
SQLite provides an output to compare
MultiLangBench, the cross-language evaluation used in the work, asks programs to reproduce the behavior of SQLite, a database engine. A database file and a query written in Structured Query Language, or SQL, provide inputs for the operation. The result can then be compared with the reference implementation. This makes the output of a program part of the assessment, alongside the source code it produces. Different programming languages have different syntax, but the operation being tested supplies a common point of comparison. The task tests matching behavior across those implementations without certifying that every requirement of a real application has been covered.[1]
COBOL migration must preserve numerical meaning
The migration work examines COBOL, a language used in legacy business software. A rewrite can compile successfully while changing what an existing program does. OpenAI therefore emphasizes semantic equivalence: corresponding inputs should produce corresponding outputs after the move to another language. Fixed-point arithmetic supplies a concrete example. Truncating a value and rounding it can produce different numerical results even when the rewritten code looks similar. Tests expose those differences between implementations. Code generation, passing an evaluation and replacing an organization’s production system remain separate stages. The announcement concerns development and assessment, with comparative results supplied by the company; it introduces no native execution of arbitrary programming languages inside ChatGPT.[1]
