OpenVINO Throughput: Count Timely Decisions Instead of Completed Requests
OpenVINO performance hints express latency or throughput intent while allowing device-specific configuration based on the model. Throughput mode may increase loading time and memory demand. Choose using the application objective and measured behaviour rather than the name of a configuration option.
Define useful throughput
State whether the count covers submitted requests, completed inferences or valid workpiece results. Record repeated, expired and unassociated outputs separately. More computation is not automatically more useful production capacity when the resulting decisions cannot be applied to the intended inputs.
For this review, identify the delivered device, software and configuration revisions first. Record unsupported, disabled and untested capabilities separately. Evidence from another model is useful context, but cannot establish what the delivered combination actually provides. Keep the original input beside the settings used to interpret it.
Compare hints on equal inputs
Use the same model, input set and correctness requirement for latency and throughput hints. Record concurrency and the actual device. Changes to precision or preprocessing belong in a separate comparison; otherwise the test cannot explain which change produced the observed result.
Retain the original input and the reason for the decision, rather than a successful screenshot alone. Change only the variables agreed for the comparison, and keep the before-and-after configuration. List conditions that were not exercised explicitly; they should not silently become part of the accepted operating range.
Track result age as well as count
Measure input-to-result time per request and inspect queue growth. A higher completion rate can coexist with older outstanding results. For a moving target, establish whether those decisions remain inside the application validity window rather than treating all completed work as equally useful.
Use the three illustrated observations as starting questions, rather than as diagnoses of an installation that has not been inspected. Check record completeness, units and identity before selecting the next observation. An unresolved result is more informative than a confident explanation assembled from missing fields or unrelated test runs.
Review resource and startup cost
Record loading time, memory and competing workloads during sustained operation and restart. Completing more requests in a short benchmark does not establish that a resource-limited device can preserve the same behaviour continuously or recover within the required startup deadline.
Let the site owner choose an approved isolated way to exercise relevant exceptions. Record normal operation, the exceptional condition and recovery separately. The purpose is to establish how the application recognises and explains untrustworthy input, without requiring production equipment to act on deliberately invalid results.
Explain the configuration choice
Deliver a comparison stating whether the objective is batch capacity, continuous inspection or single-part response. Identify the measured workload range and review triggers after resource or model changes. A performance hint should remain an explained choice, rather than becoming an unquestioned permanent default.
Deliver the configuration, representative samples, result table and unresolved questions together. Agree who reviews subsequent changes. The receiving team should be able to select one conclusion and trace it back to its input and decision path without relying on the original author to reconstruct the missing steps.
Original documentation and further reading
Continue with related articles in this section, or use selection guides to prepare the next review. Apply the device-specific documentation and agreed operating conditions when implementing a test.