← All posts

What happens when you test an AI agent on an app that does not exist yet?

Context. This work was done during my time at Google DeepMind in Bangalore, working with Sagar Gubbi, Aravindan Raghuveer, Deepanway Ghosal and Hiroki Furuta, as part of my student researcher role. The stint was 24 weeks, from June to December 2025.

Consider a request to an AI assistant to book a train ticket from Bangalore to Mysore for the following Tuesday. Fulfilling it requires the assistant to open a booking interface, check availability, select a seat, enter passenger details and complete payment. Each step depends on the one before it, and an error at any point causes the system to reject the transaction.

Work of this kind is increasingly expected of AI agents, yet measuring how well they perform it remains difficult. This article describes VISTA-Bench, a project that began with a specific limitation in existing evaluation methods.

The Login Wall

The applications for which agents are most valuable are those with real consequences: banking, government portals, healthcare and travel. These are also the applications that cannot be used to test an agent directly, because they sit behind authentication. A research agent cannot be given access to a real bank account in order to observe its behavior. We refer to this barrier as the Login Wall. The two common ways around it each have drawbacks:

• Static traces (datasets such as Mind2Web and UGIF) are cached snapshots of web pages. They are safe to use but unresponsive. If an agent selects the wrong control, nothing changes, so it is not possible to observe an agent fail a login or encounter an error, and recovery behavior cannot be studied.
• Live environments (WebArena, OSWorld) run real or simulated browsers. They are realistic, but they are slow to reset, costly to maintain and largely restricted to open-source software and public websites. Applications that sit behind closed accounts remain out of reach.

A third approach was therefore needed. It had to satisfy four requirements. It had to be safe, using no real accounts and no real money. It had to be reactive, so that an incorrect action produces an incorrect result which the agent must handle. It had to be inexpensive to reset, so that a scenario can be repeated many times. And it had to be producible in volume, since a benchmark of five applications offers little evidence. Building a realistic banking or government portal by hand for every case did not satisfy the last requirement, which led us to look for an existing source of workflows.

The approach should also be distinguished from tool-use benchmarks such as ToolBench and Gorilla, which test whether a model can call APIs correctly. In many of those benchmarks the calls are independent: retrieving a weather forecast has no bearing on whether a table can subsequently be booked. The applications in VISTA-Bench are designed differently. Each action depends on the state left by the preceding one, a property we call stateful dependency. Selecting a seat fails unless a flight has already been selected, because the internal flag on which the action depends has not been set. The agent must therefore maintain a model of the application's hidden state across a long sequence of calls, which is the capability that Login Wall applications require.

Generating applications from video tutorials

The internet contains a large number of tutorials that demonstrate, step by step, the tasks of interest, such as applying for a passport in India or booking a driving licence test. Each tutorial already records the workflow: the order of screens, the fields to complete and the rules that govern progress, for example that a user cannot continue without completing a captcha.

VISTA-Bench uses these videos as specifications. A multimodal model processes the audio and the frames of a tutorial and extracts a workflow graph composed of three elements: the entities that must be tracked (an Aadhaar number, an OTP, a PNR), the states of the application (login screen, dashboard, payment gateway) and the constraints that govern transitions between states. The model then writes a self-contained Python class that behaves like the application.

The result is a small state machine and not a website. The agent receives a toolbox of functions, such as selecting a service centre or verifying a mobile number, together with a plain-language instruction. Every call is processed by real application logic. If the agent attempts to withdraw more than the available balance, it receives a structured error. If it attempts to select a seat before choosing a flight, the call fails, because the flag on which it depends was never set.

This form has a clear advantage for evaluation. A state machine has no pixels to misread, no page that loads slowly and no pop-up that appears intermittently. When an agent fails inside one, the cause is almost always in its reasoning: it called the wrong tool, called tools in the wrong order, supplied the wrong input or stopped too early. Failures are therefore easier to interpret and to count. Each application is also a few hundred lines of code that can be tested, versioned and reset immediately, which makes running several hundred episodes practical.

The instruction given to the agent is written as a user might phrase it, in plain language and in the language of the source video. The agent is not given the workflow graph. It receives only the tools and the request and must determine the correct path itself, as a person would on an unfamiliar site.

Structure of a generated application

Every application inherits from the same base class, so the agent always encounters the same three components:

• The toolbox. A method that returns the list of functions the agent may call, for example selecting a service centre or verifying a mobile number.
• The instruction. A method that returns the system prompt, written as a short manual of the steps the agent should follow.
• The success check. A method that returns the secret codeword, and only when the end conditions are satisfied.

The generated code follows a strict schema. Every method returns a dictionary containing a status and a message, and all internal state is held in a single state dictionary on the application. Returning a dictionary in place of raw HTML is deliberate: it removes parsing noise, so that failures reflect reasoning and not difficulty reading a page.

Each trial begins from a clean state. The runner retrieves the application from a registry, instantiates it and forces a reset of its state, so that nothing carries over from a previous run. The application then supplies the agent with its tools and instruction. The interaction that follows is a loop: the agent reasons and requests a tool call, the application validates the arguments, updates its flags and returns a structured result, and the cycle repeats. The loop does not end after a single call. It ends only when the agent decides that it can answer and writes a final text response in place of a tool call. At that point the call log is compiled and the codeword check is run.

Formally, the interaction is modelled as a partially observable decision process. The state is a dictionary of variables defined by the generated code; for a bank application it might contain whether the user is logged in, the balance and whether an OTP has been verified. The actions are calls to the functions the application exposes. Ordinary Python exceptions act as transition signals, so an invalid action, such as withdrawing more than the balance, returns a structured error and does not advance the application. Transitions are deterministic and the reward is sparse: one if the agent retrieves the codeword and zero otherwise. The agent never observes the state directly and must infer it from the responses of the tools.

A trial from start to finish

The process is easiest to follow through an example. Consider an application built from a tutorial on booking an Aadhaar appointment. The runner creates the application, resets its state and gives the agent a request to book an appointment, together with the application's toolbox, which includes a function that selects a service centre and one that verifies a mobile number, and a short manual of steps.

The agent begins by choosing a centre. The application checks that the city supplied is valid, sets a flag recording that a location has been selected and returns a status and a message. The agent reads the response and proceeds to the next step, where the application again validates the input and updates its flags. If the agent supplies an input in the wrong format, the application returns an error and no progress is made until the agent corrects it. If the agent calls a step too early, the call fails, because the flag on which that step depends has not yet been set.

When the agent concludes that the task is complete, it stops calling tools and writes its answer. Only at this point does the codeword become relevant. The application releases it only if all end conditions hold, for example a location has been selected and a payment has been verified. An agent that skipped the payment step may write a confident closing message, but it has no codeword to include, and the trial is recorded as a failure. The principle of the design is that what the agent states is not evidence of completion; the state of the application is.

Verifying task completion

The success check is a central element of the design. Asking a model whether it completed a task is unreliable, because models readily claim that they did. Each application therefore holds a secret codeword. The application reveals it only if the agent has driven the internal state to a valid end condition, for example a location selected and a payment verified. A trial counts as a success only if the agent's final answer contains that exact, dynamically generated string. An agent that fabricates a success message obtains nothing.

Because the codeword is generated dynamically, an agent cannot memorize it, and because it is exposed only when the underlying state is valid, it cannot be obtained by assertion. This matters because the most common way for a language model to appear capable is to sound confident. A check based on the final answer alone rewards confidence, whereas a check based on the state of the application rewards completing the task.

Each run is also logged as a function call trace recording which tools were called, with which arguments, and what each call returned. The trace allows analysis of why an agent failed, and not only of whether it did.

Quality control of generated applications

Code written by a model can be incorrect in ways that are not apparent. Every generated application therefore passed through a rubric pipeline before it was admitted to the benchmark. The application had to match the workflow in the source video, pass its generated unit tests (approximately eight per application), avoid hard-coded passwords or success flags, track its state correctly and use the same language as the video. A tutorial in Hindi must produce an application in Hindi.

The correction loop is deliberately limited. When an application fails the rubrics, the model writes unit tests for it, runs them, reads the error logs and rewrites the code. Three attempts are permitted. If the application still fails, the video is discarded. The rubrics check for meaningful names, state that is updated after every transaction, strict types for parameters and proper handling of invalid input. A second group of checks covers fidelity to the video, absence of hard-coded data, a matching language and useful documentation for the agent.

The source videos were also selected with care. Candidates were filtered for community validation, such as high view counts, favorable like ratios and positive comments, as a rough signal that a tutorial is accurate and useful. The videos range in length from 30 seconds to 38 minutes.

The attrition figures indicate how strict the process was. The pipeline began with 240 candidate videos. Of these, 218 yielded a usable workflow, and 150 applications passed the rubrics. After manual removal of duplicates, 125 applications remained, approximately 52% of the starting set. The main reason for rejection was not broken code but state logic that was disconnected from what the video showed.

A human audit was also conducted on 30 applications, each checked against its source video. In 28 of 30 the workflow matched the video (93.3%), and in 27 of 30 the constraints derived from the video and not from the model's general knowledge (90.0%), for an overall fidelity of 91.6%. The errors were minor, such as a missed confirmation pop-up or "Pin Code" being rendered as "Zip Code", and were corrected.

The purpose of the audit is as follows. Automated rubrics are effective at detecting code that is broken or that cheats, but a person must judge whether the application genuinely mirrors the video. A reviewer therefore watched each of the 30 sampled videos and walked through the matching application, asking two questions: whether the order of steps matches, and whether the rules the application enforces come from the video or from assumptions made by the model. Rules derived from general knowledge are a quiet source of leakage, because they make the application easier for an agent that already holds the same knowledge.

Circularity and the symmetric protocol

Synthetic benchmarks face a specific risk. If a single model writes all of the environments, that model is likely to perform well on them, because the applications reflect its own habits. We refer to this as circularity, and to its effect as home-field advantage.

Our response was a symmetric protocol. The applications were divided almost evenly among three models: 42 were written by Gemini 3 Pro, 42 by GPT-5.1 and 41 by Claude Opus 4.5. Every model was then evaluated on all three sets. This makes it possible to separate a home score (an agent on applications written by its own model) from a held-out score (the same agent on applications written by the other models). The held-out score is the more reliable measure of general reasoning. Gemini 2.5 Pro was included as an older baseline. It was too weak at writing complex stateful code to serve as an author, and therefore acted only as an agent.

The benchmark covers applications from several regions and languages, which proved important for the findings reported below. The tutorials were not limited to English. Hindi, Kannada, Tamil, Urdu, Arabic and other languages appear, and each generated application retains the language of its source video, so the agent must also work in that language.

Benchmark composition

The 125 applications yield 250 prompts. The domains are government services (40%), banking and finance (30%), travel (20%) and utilities (10%), a mix chosen to make the logic and state tracking as demanding as possible. An average application contains about 310 lines of code, between 4 and 14 functions and about eight tests. The technical report describes applications in 29 languages from 41 countries, with a deliberate focus on regions that benchmarks usually omit.

Agent evaluation

The agent runs a ReAct-style loop of reasoning, tool calling and observation, with a limit of 15 turns and a temperature of 0.1. We ran 625 episodes, five trials for each of the 125 applications. The applications have an average interaction depth of 6.8 steps (between 4 and 14) and expose about 5.2 tools at each turn, so the agent must distinguish valid from invalid moves throughout.

A limit of 15 turns is generous for applications that average fewer than seven steps, so an agent that is proceeding in the right direction has room to recover from an occasional mistake. Running five trials per application, and not one, allows us to report Pass@5, the share of applications for which at least one of five attempts succeeds, and it reduces the influence of chance in any single sample at low temperature.

The cross-evaluation shows the home-field advantage in every row:

Every frontier model scored highest on the applications it had written itself. Claude Opus 4.5 had a home score of 86.0 but declined by 18.7 points on applications from other authors. Gemini 3 Pro showed the smallest gap and the highest held-out score at 75.3. Had a single model written the entire benchmark, that model's ranking would have been inflated, which is the reason for the symmetric design.

Three observations follow from the table. First, the older baseline, Gemini 2.5 Pro, is clearly behind on all three sets, which confirms that the benchmark distinguishes stronger agents from weaker ones. Second, the three frontier agents are close enough on foreign applications that their ordering would have been unclear without the held-out column. Third, the size of the decline from home to held-out is itself informative. A small decline suggests an agent that reasons about the task, while a large decline suggests an agent that partly relies on the style of the application it is examining.

None of this constitutes a criticism of any model. It is what would be expected of systems trained on text that includes the conventions of their own model family, such as naming conventions, error phrasing and the usual layout of a flow. The purpose of the design is that the size of the effect does not have to be guessed. It can be read directly from the table.

Failure analysis

Coding errors were expected to dominate, but they did not. Among the failures classified, the largest category was reasoning failures at 36.4%, in which the agent did not determine the correct order of calls. One example was a rail booking in which the agent chose UPI as the payment method and then ended the session without confirming the payment, and so never obtained the PNR.

This failure is instructive about agent behavior. The agent made no typographical error and did not call a missing tool. It formed a plan that appeared reasonable, executed most of it and then wrote a closing message as though the plan were complete. The missing step was a verification, and nothing in the agent's own context indicated that verification was required before the end.

• Input validation (18.2%): using "555-123" after the application clearly requested a strict 10-digit number.
• Constraint violations (18.2%): a user instructed that the PNR be given "only once full amount is cleared," and the agent provided it after a partial payment.
• Privacy violations (9.1%): printing a full simulated card number in the final answer in place of a masked one.
• State logic traps (9.1%): requesting an address again after it had already been submitted.
• Terminal state failure (9.1%): stating that the task was complete without retrieving the codeword.

Two stress tests were also conducted. In the first, an application modelled on UAE Pass requires an introspection check after login. The agent logged in and omitted the check entirely, giving priority to the user's goal over the security protocol. This is the most significant finding of the report. Safety training has focused largely on harmful content, and it does not clearly teach an agent to follow a strict business procedure when doing so feels unhelpful. We describe this as a gap in procedural safety.

In the second test, conducted on non-English applications such as France Connect, a pattern emerged that we named terminal state recognition failure. In 7 non-English applications the agent completed the task but did not recognize that it had reached the end state. Responses and flags that depart from the usual English form made its reasoning less reliable.

Regional results

The regional breakdown produced an unexpected result. English is normally where models perform best, but here the pattern was close to the opposite:

Our hypothesis, which the report presents only as a hypothesis, is a "parametric memory trap". In English, and for well-known procedures such as standard international banking, the model holds a great deal of pre-trained knowledge. It sometimes trusts that knowledge over the tools it was actually given, and so shortcuts steps or invents parameters. In lower-resource settings there is less competing memory, and the model follows the instructions in the prompt more closely. For strict procedural tasks, extensive world knowledge can be a distraction.

The report depicts this as a curve of success against how much the model already knows about a domain. At the low-knowledge end the model has little to fall back on, so it follows the tool definitions and performs well. At the high-knowledge end its own memory competes with the instructions it was given, and protocol adherence declines. In English the model sometimes shortcuts a process or invents parameters from general knowledge, while in Urdu and Arabic it adheres to the provided context. This remains a lead to be tested and not a settled result.

Caution is needed in reading the table. It does not show that models are better in Arabic than in English. The regions differ in which tutorials were found, which applications resulted and how demanding their workflows are, so the comparison is not controlled. What the table does suggest is that the usual assumption, that greater familiarity always helps, does not hold when the task is to follow a specific procedure exactly. Familiarity can lead an agent to fill in a step from memory in place of reading what the tool requested.

The European and North American figures are the lowest, and these are also the settings where the procedures, such as card payments and standard banking, are the best known. If the memory trap explanation is correct, this is the pattern that would be expected. If it is incorrect, the pattern is still worth testing independently, and the benchmark makes such testing straightforward.

Limitations

VISTA-Bench removes the untidy parts of the real web: rendering, overlays and latency. This is deliberate. It isolates whether an agent can follow business logic and track state across many steps, and an agent that cannot do so will not do so on the real web either. It is a necessary test and not a sufficient one. The pipeline is also sensitive to how the source video is described, and the agent sometimes reverted to English for internal names and errors even when the task was in Hindi or Tamil.

Agent failures can be divided into two kinds. Perceptual failures concern the inability to find the correct element on a screen. Procedural failures concern the inability to follow the underlying business logic. VISTA-Bench isolates the second kind. The argument is that an agent that cannot follow the logic when the interface is clean will not follow it when the interface is cluttered, which is why a clean test is informative even though it is incomplete.

A further limitation should be stated. Because the applications are written by language models, the benchmark inherits some of their blind spots. A flow that no model finds natural to write may be under-represented. The audit and the rubrics reduce this effect without removing it, which is another reason to treat the scores as a comparison among agents on equal terms and not as a prediction of how an agent would score on a particular real service.

Most of the analysis was also run with a single agent, Gemini 3 Pro, as a proof of concept. The report is concerned with the method for generating environments more than with ranking models.

Conclusions

Agents are fluent, and that fluency conceals how unreliable they can be with strict systems. The failures that matter are not exotic. They are skipped steps, premature declarations of completion and disregard of a rule the user stated clearly. If agents are to handle banking and government services, training and evaluation must reward procedural adherence and state tracking, and environments are needed in which agents can fail safely first. Converting existing tutorials into such environments is a practical way to begin.

The central point is that evaluation should be built around what the agent must get right, and in what order, with the surface of an application being secondary. A model that is polite, fluent and confident can still omit the one check that mattered. A useful next step is to train agents against environments of this kind, where skipping a check has a visible cost, and then to test them again on held-out applications from other authors to determine whether the improvement carries over.

This article summarizes the VISTA-Bench technical report by Shereen Anand, Sagar Gubbi, Deepanway Ghosal, Hiroki Furuta and Aravindan Raghuveer, Google DeepMind.