Where the answer key comes from
In Property Finder’s property-image system, the answer key for evaluating a replacement came from the model being replaced. According to the technical account published with AWS on October 27, 96.8 percent of evaluation labels were inherited directly from the incumbent’s predictions. An alternative could therefore lose points for disagreeing even when its answer was correct. The methodological problem extends beyond a few mislabeled examples: the criterion of correctness and one candidate share an origin. The remaining 3.2 percent of human-labeled samples also contained categories outside the original task specification. With both label provenance and category definitions unsettled, a low score becomes difficult to interpret. This finding explains why the old comparison was unreliable before it establishes which replacement performs better. Agreement with an incumbent is an observable property, but treating it as ground truth makes a separate claim about correctness that needs independent support.[1]
The team tried to break that circularity by having a domain expert annotate 356 UAE images selected from production listings, without consulting previous model outputs. Ambiguous cases were clarified with the AWS team. The system evaluates 11 attributes. A corridor photograph with a bedroom visible through a doorway illustrates how requiring one room label can manufacture an error. Moving to multiple labels changes the definition of a correct answer for that case. Rules distinguishing decorative Arabic writing from watermarks are similarly specific to the task and region. Attributing the resulting score improvement entirely to a model change would hide contributions from annotation and task definition. The rebuilt labels make comparison more meaningful, but a process with several simultaneous interventions does not isolate the effect of each intervention. This is a practical repair of an evaluation workflow rather than a controlled experiment assigning one cause to every improvement.[1]
Two kinds of independence
The independence of the final comparison remains limited. The first 256 annotated images were used for prompt development, and the final evaluation used all 356 images, including 100 added later. Examples already seen during development consequently contribute to the reported evaluation. The authors explicitly describe the scores as optimistic for this reason. A separate 230-image Egyptian set provided a regional check, but its results are not reported in this account. Labels can therefore be independent of the incumbent while the final score is not independent of the development process. Label independence and test independence are distinct conditions: one asks where the answer key originated, while the other asks which examples informed model and prompt selection. Keeping those conditions separate prevents an improvement in annotation quality from becoming an overstated claim about generalization. The unpublished regional results also limit what can be inferred about performance beyond the reported UAE images.[1]
Tuning effort was also unequal. The incumbent received one prompt-tuning round, while the selected Llama 4 Maverick 17B Instruct received several rounds and additional Property Finder edge-case rules. The final evaluation reports micro F1 values of 0.56 and 0.80 respectively. That result compares two systems under the published setup; it does not isolate their underlying model capabilities. One alternative explanation is that extra optimization and task-specific rules account for part of the difference. Reusing development images can also make the reported score more favorable than performance on fresh examples. These limitations do not negate the fact that Property Finder selected the replacement for production. They distinguish a deployment decision from a claim of model superiority under equal conditions. The narrower supported conclusion is that the chosen configuration scored higher with these data and prompts. Equalizing tuning effort and retaining untouched examples would address different sources of uncertainty rather than merely repeating the same comparison.[1]
The scope of scores and costs
Metric choice is part of that narrowing. Micro F1 pools label decisions, allowing frequent categories to dominate. The team therefore considered macro F1 and per-label precision and recall as well. Per-sample intersection over union assessed overlap between predicted labels and expert labels, while exact-match accuracy was used for the nine single-label attributes. These are not interchangeable names for one question; they expose different distributions of error. An overall score can rise while a rare attribute remains weak. Conversely, weighting every class equally may not reflect the distribution of the actual business workload. Explaining which errors matter to which decision therefore provides more information than selecting one aggregate number. Presenting several metrics makes that distinction inspectable. Their usefulness depends on the task definition and sample composition, however, so the presence of multiple scores alone cannot repair overlap between development and evaluation data or make a small rare-class sample representative.[1]
The cost comparison has a separate boundary. The published estimate of an 86 percent reduction rests on token and pricing assumptions for Ohio in December 2025. The prompt used for the accuracy comparison differs from the baseline prompt in the cost calculation. The percentage therefore cannot be carried forward as a current price advantage for every customer. The account is valuable because it exposes label provenance, development-set overlap and unequal tuning effort to examination. Independent labels, untouched test examples, comparable development effort and explicit cost assumptions together make a deployment decision easier to assess. Missing tests matter because they identify which question the available result answers, rather than requiring the result to be discarded. Selecting a cheaper configuration is a concrete business decision. Generalizing that decision across models and new data distributions requires additional evidence. A careful reading preserves the operational choice while placing its accuracy and cost claims within the conditions actually described.[1]