# We Let an AI Agent Do QA on Real Hardware > An AI agent tests our apps on a physical phone instead of an emulator. What it catches, what it costs, and where the approach stops being worth it. By Alexandr Bykov, OxiSoft. Published 15 September 2026. Canonical: https://oxisoft.io/blog/ai-agent-android-qa/ There is a phone on a desk here, plugged into a USB port. Nobody touches it. An AI agent installs our apps on it, drives the screens, takes screenshots, looks at them, and says what is broken. This started as a shortcut for producing store screenshots and turned into something more useful. Here is what works, what it found, and the part that went wrong. ## Why real hardware and not an emulator An emulator would have been easier. We tried it and moved away from it, for three reasons. Emulators are slow unless you give them hardware acceleration you may not have spare. A real phone on the end of a cable answers immediately, and when the loop is capture, look, decide, tap, repeated a few hundred times, latency is the whole budget. More importantly, the phone runs the manufacturer's real operating system. Phone makers modify Android, and those modifications are exactly the kind of thing that breaks an install or a permission flow in the field. One of our test devices refuses a normal automated install and puts up its own confirmation dialog first. That is a genuine condition for a large share of real users, and no emulator would ever have shown it to us. And an emulator cannot tell you that an app looks wrong. It renders pixels. The valuable part of our loop is a model looking at a screenshot and saying "the progress bar is squashed," which needs a screenshot worth looking at. ## The loop It is unglamorous. Three steps, repeated. **Capture** the screen as an image. **Read** the interface: on most apps the system can hand back a machine-readable list of every visible element with its label and position. **Act** by tapping a coordinate, then start again. That middle step is where apps differ, and the difference matters more than it sounds. A normal app built from standard interface components describes itself fully, so navigation is exact. A game does not. Anything that paints to a single drawing surface exposes no structure at all, so the agent has nothing to read and must work purely from the picture, judging coordinates by eye. Two completely different approaches, and you want to know which one you are in before planning a test run. Around that core we ended up with a handful of small scripts: unlock the device, install a build through the manufacturer's dialog, find a settings row by its label, list the state of every toggle on a screen, and enter a credential without it ever appearing in a log. Each one exists because the agent kept needing it and kept doing it slightly differently. ## What it actually caught Two bugs that matter, both of which survived ordinary testing. **A proportion bug invisible on phones.** One of our apps draws a row of small shapes as a progress indicator. The row sized itself as a percentage of screen width, but with a fixed height in code. On a phone the two numbers happen to produce the right shape. On a tablet-sized screen the row grows wider while the height stays put, and the shapes stretch into flattened ovals. Nobody had noticed, because nobody had looked at the app on a large screen. The agent was capturing screenshots at tablet size, looked at one, and said the shapes were wrong. The fix was to derive the height from the width. One commit. **A failure that only exists in release builds.** A debug build started fine. The release build hung on launch, because the build step that strips unused resources removed an image that was only referenced from code it could not see, so initialisation waited for something that was no longer there. This class of bug cannot be found in a debug build, by definition. It is also exactly the class that reaches customers, because customers get the release build. That produced a standing rule: test the release artifact whenever the project can produce one. ## Screenshots for hardware you do not own App stores want screenshots at tablet sizes. We do not own tablets. You do not need them. Android lets you override the resolution and pixel density a device reports to apps. The app then lays itself out as though it were running on a larger screen, and the captured image comes out at the size the store asks for. Only the physical panel is unchanged. Two things we got wrong before it worked. Density is not a cosmetic setting: pick the wrong value and the app reports a screen so large that a list page renders half empty, so the numbers need a sweep with something judging each result. And the app has to be fully restarted after the override, because a running app does not re-lay-out. Skip that and you capture the old layout scaled up, which looks convincing and is wrong. Reset the override when finished. It survives reboots, and every later step that depends on screen coordinates will quietly misbehave until you do. ## Guardrails matter more than you expect An agent that can tap a real screen needs limits that are enforced, not assumed. Ours check the state of the device before every action: the right app has to be in the foreground before anything is captured, and returning to a known screen means closing and reopening the app rather than guessing at navigation. That sounds fussy for a QA tool. It is the whole difference between automation you can leave running and automation you have to supervise. When you give an agent a tool that acts on the real world, the dangerous moments are not the ones where a command fails and something obviously breaks. They are the ones where every command succeeds and the context quietly stopped being what you assumed. ## Is it worth it For interface regressions and layout bugs on real hardware, yes. It found two genuine bugs that manual testing missed, and it produces every store screenshot we ship, including sizes for hardware we do not own. It is not a replacement for a test suite. It is slow, every action is a round trip, and giving it something repetitive wastes time badly. We learned not to have it play a game to generate score data, for instance. A person does in ten seconds what costs the agent several minutes. Treat it as an extra pair of eyes that never gets bored of looking at the same screen on the tenth run, which is the same argument we make for [AI-assisted development generally](/blog/ai-driven-development-with-claude-code/). It works for the same reason: not because the machine is cleverer, but because it does the boring check every single time. The apps it tests are on the [apps page](/apps/), and they are all built the same way: [no tracking, no ads, nothing leaving the device](/blog/why-we-dont-track-users/).