Android App Performance Testing: Build a Repeatable Device Matrix

Define the user task and metric

Choose a task such as cold launch, scrolling a long list, loading a large PDF or completing a game level. Specify the beginning, end and success condition before collecting measurements.

The profiling guide distinguishes low-overhead profileable release testing from deeper debug-only inspection.

Build a practical matrix

Not every combination needs equal frequency. Prioritize the population and failure modes that matter to the product.

Automate stable journeys

Use Macrobenchmark for supported end-to-end performance journeys. Keep compilation mode, app state and iteration setup explicit so results are comparable.

Do not compare debug and optimized release timings as though only one code change differed.

Measure distributions, not a lucky run

Repeat the workload, inspect variability and retain traces for unusually slow runs. Temperature, background activity and cache state can affect results.

Set a regression threshold appropriate to normal measurement noise. A tiny improvement smaller than the run-to-run variation is not convincing evidence.

Close the loop with production

Android vitals helps reveal device-specific and field failures that a laboratory matrix missed. Use those cohorts to improve the next test cycle.

APKLint's performance and release checkers can organize acceptance criteria and package inspection. They do not execute benchmarks, so a release decision should include actual measurements, the tested matrix and known untested conditions rather than a static checklist score alone.