Data Agents and Business Dashboards: When an AI Tool Is Enough and When You Need a Custom Application

A dashboard shows what happened. A data agent can continue with the question that usually consumes an analyst's time: why did it happen? If revenue falls during a week when traffic grows, a fixed report can reveal the difference. An agent connected to data through controlled tools can find the relevant tables, check metric definitions, test hypotheses and expose the queries behind its answer.

That flexibility is valuable, but it creates a commercial misconception: if people can question company data in natural language, do they still need dashboards, reports and custom applications? The short answer is yes, but not for every question. A data agent is strong at investigation. A dashboard is strong at repeatable monitoring. A custom application becomes necessary when analysis must connect to roles, alerts, approvals and real actions.

The essential idea: a data agent should not become the new official source of company figures. It is an investigation layer over the organisation's data, definitions and permissions. The source of truth remains in controlled systems, and important decisions must remain verifiable.

What OpenAI's Data agent actually is

OpenAI has published a demonstration project and a step-by-step guide for building a data agent with the Agents API. The example is not a universal product enabled by one button, and it does not promise automatic access to every business system. It is a reference architecture that a technical team can adapt to its own data warehouse.

In the documented configuration, the application connects to a PostgreSQL-compatible warehouse through a read-only account. The agent receives four narrow tools: table discovery, business-context search, read-only SQL execution and the explicit saving of a correction to memory. Persistent sessions keep the original question and its follow-ups together, so the investigation does not restart with every message.

Application ownership is an important detail. The agent never receives the database credentials. Company code executes the queries, blocks write statements, applies a timeout and limits the result. An answer can include the tables consulted, assumptions and executed SQL. This traceability separates a plausible explanation from one an analyst can actually verify.

The example documentation requires a PostgreSQL-compatible warehouse. BigQuery, Snowflake, MongoDB, SharePoint, Google Drive or other sources can be integrated through custom connectors and services, but they should not be described as capabilities guaranteed by the Data agent demo. Authentication, permissions, result schemas, limits and retention policies must be designed for every source.

A data agent, a BI dashboard and a custom application solve different problems

In practice, these solutions complement one another. A data agent may discover that a conversion decline came from mobile traffic after a technical release. An analyst verifies the explanation. If the problem needs daily monitoring, the metric moves into a dashboard. If crossing a threshold must create a ticket, notify a team and request approval, that rule belongs in a controlled application or automation.

How the agent moves from a question to a verifiable answer

A mature workflow does not simply send a question and an entire database to a model. It narrows access and builds the answer in stages:

1. Identify the user and role. The application establishes which company, department, region and level of detail the person is permitted to access.
2. Discover relevant tables. A tool returns real columns, ownership and freshness instead of asking the model to guess the schema.
3. Load definitions. “Revenue”, “active customer” or “paid conversion” receives the organisation's approved meaning, including exclusions and time windows.
4. Reuse reviewed analysis. Previously validated queries provide reliable patterns for joins, filters and recurring calculations.
5. Build read-only SQL. The application validates the statement, applies a timeout and limits result volume.
6. Compare and explain. The agent tests hypotheses, separates observation from interpretation and names data limitations.
7. Expose evidence. The user can inspect the sources, assumptions and queries supporting the answer.
8. Remember only explicit corrections. A rule becomes personal or team memory only when an authorised user asks for it to be saved.

Memory should not turn every conversation into a permanent rule. An observation valid for one campaign or one month may be false in another context. Shared corrections need an owner, date, scope and a way to withdraw them, just like definitions in a data catalogue.

Example: traffic rises while revenue falls

A manager notices that sessions increased by 18% while revenue declined. The question “why?” sounds simple, but it can cross advertising, analytics, commerce, payment and return data. A correct answer does not begin with a story. It begins by validating the metrics.

The agent should check whether the two periods contain the same number of days, whether they use one time zone and whether revenue means gross, net or net of returns. It can then segment traffic by source, device, landing page, product category and customer type. If it finds a mobile decline, it can inspect add-to-cart rate, payment errors and technical releases during the interval.

A useful answer would say that the decline does not affect every channel, but is concentrated in one campaign's mobile traffic; checkout completion fell after a Tuesday release; and the last 24 hours are still incomplete. Queries, tables and exclusions must accompany that explanation. Without them, it remains a well-written hypothesis.

After verification, the company may decide that mobile payment completion is an operational signal. At that point, it is inefficient for a manager to question the agent every day. The metric should be defined in a business dashboard with automated reporting, refreshed through a stable rule and accompanied by an alert threshold.

Good data matters more than the wording of the question

A capable model cannot repair a warehouse in which one concept has five definitions. The most serious errors often do not appear because the agent cannot write SQL. They appear because the data does not describe the business consistently.

• Incompatible definitions. Marketing calls every form submission a lead, while sales counts only validated contacts.
• Fragmented identifiers. The same customer has different IDs in the store, CRM and billing system.
• Late-arriving data. Returns, payments and assisted conversions arrive after a report closes.
• Duplicate events. One order sent by both browser and server is counted twice.
• Currencies and time zones. Two sources compare values that look identical but do not share the same basis.
• Fields without owners. Nobody can say who approves a definition or when it changed.

Before integration, it is worth building a concise dictionary for important metrics: formula, source, update frequency, exclusions, owner and examples. The agent can use this information as business context. Without it, the model may produce a technically reasonable interpretation that differs from the one used in official reporting.

When a data agent can be enough

A data agent can be the right solution when questions change frequently, users are analysts or managers able to evaluate an answer, and the output informs an investigation rather than immediately executing an irreversible action.

• exploring an anomaly first noticed in a report;
• comparing segments, periods or cohorts;
• preparing for a meeting using information spread across several tables;
• discovering questions that deserve to become formal metrics later;
• finding a reviewed analysis and adapting it to a new filter;
• prototyping a hypothesis before investing in a permanent report.

In these situations, conversation reduces the time between a question and the first verifiable explanation. The gain does not come from eliminating analysts. It comes from reducing the work of catalogue search, schema discovery and rewriting familiar queries.

When you need a dashboard or a custom application

An exploratory result should not become an operational screen merely because it looks convincing. A company needs a stable dashboard when the same metric is monitored by several people, must be compared over time and must retain the same definition every time it is opened.

A custom application with AI integration is justified when the user needs more than an answer. They may need to select a customer, verify documents, request approval, create a task, track its resolution and retain an audit trail. The model can interpret a request or explain the data, while application code enforces business rules and process state.

A hybrid architecture is usually the healthiest choice

Instead of replacing the reporting system, a data agent can sit alongside it. The dashboard displays approved metrics. An investigation control opens a conversation in the exact context of the chart, period and user's role. The agent consults only permitted sources, and its answer retains a path to the analysis and SQL that produced it.

If a finding recurs and becomes important, it should not remain buried inside conversations. An analyst validates the formula, the data team moves it into the semantic layer, and the dashboard displays it consistently. If the result must trigger an action, business automation with AI integration can prepare the task while confirmation and audit remain in the application.

This separation avoids two extremes: a dashboard with hundreds of charts for every imaginable question, and a chat in which nobody knows whether two answers used the same formula.

Permissions must be enforced before the model

An agent that can formulate SQL must not decide on its own which data it may see. Authorisation belongs in the database and application services: read-only roles, access limited to required schemas, filtering by company or region, masking of sensitive columns, and time and volume limits.

Query validation needs to go beyond searching for the word SELECT. Multiple statements should be blocked, risky functions restricted, and timeouts, row limits and logging applied. Text sources create another risk because notes and documents can contain malicious instructions. Retrieved content is data; it does not gain authority over the agent.

Shared memory has its own risk. If every user can save a “correction”, one false definition can contaminate future investigations. Saving should be explicit and attributed to an author. Team rules should require approval. People must be able to inspect and delete memory.

Transmitted data, retention and company responsibility

According to the OpenAI documentation reviewed for this article, data sent through the API is not used to train models unless the organisation explicitly chooses to share it. However, “not used for training” does not mean “never retained”. Abuse-monitoring logs may exist by default, and some capabilities retain application state.

Current documentation lists Agents API objects with application state retained until deletion and without standard Zero Data Retention eligibility. Policies can change and differ by endpoint, project and approved controls. Before financial, commercial or personal data enters the workflow, the team must verify the exact configuration, retention period, region, third-party services and deletion process.

Minimisation remains the strongest defence. The agent does not need every column merely because the database contains it. An aggregated view without names, email addresses or direct identifiers may be safer. An audit log can record who asked, which tool was called and which query ran without unnecessarily copying the entire result.

The real cost is more than model usage

One investigation can require several calls: search the schema, load definitions, run an initial query, test a segment and compose the explanation. Total cost includes the model, agent infrastructure, database or warehouse processing, session storage, observability and connector maintenance.

In a warehouse billed by bytes scanned, a valid query can still be too expensive. The application should constrain default date ranges, use aggregated tables, estimate query cost where the platform permits and stop excessive investigations. Caching can help with metadata and definitions, but it should not serve stale figures as current.

Useful operational metrics include cost and duration per completed investigation, the percentage of answers accepted without correction, blocked-query count, data freshness and analyst time saved. Conversation volume alone does not show whether the agent created value.

How to test before granting access to real data

A carefully selected demonstration is not an evaluation. A pilot should begin with real questions and reference answers prepared by people who know the data. OpenAI's evaluation guidance recommends representative test data and explicit criteria compared with a known correct result.

1. Select 20 to 30 questions. Include simple requests, multi-step investigations, ambiguous terms and cases where the available data is insufficient.
2. Establish ground truth. An analyst records the accepted result, tables, filters and exclusions.
3. Test discovery. The agent should choose the correct source, not merely produce executable SQL.
4. Measure the figures and explanation separately. A correct value can receive a false causal story, while a good explanation can start with incomplete data.
5. Verify permissions. Deliberately inappropriate questions should be refused or filtered without leaking information through error messages.
6. Run variations. Change the wording, order of follow-ups, period and level of aggregation.
7. Record cost and latency. A correct answer that is too slow or expensive is not yet a useful product.
8. Retain human review. During the first stage, answers should not automatically trigger decisions or actions.

A realistic 30-day implementation plan

1. Week 1: problem and data. Choose one domain, inventory its sources and approve five to ten important definitions.
2. Week 2: controlled access. Build discovery and query tools, the read-only role, limits and logging.
3. Week 3: context and evaluation. Add reviewed queries, test questions and an interface that exposes evidence.
4. Week 4: restricted pilot. A small group compares the agent with the current process, reports corrections and identifies recurring questions.

The final decision is not merely whether the agent “answers well”. The team decides what remains conversational investigation, which metric deserves promotion into a dashboard, and which workflow justifies custom application development. That is the valuable output of a pilot: a clear map of decisions, not an impressive chat without an owner.

Frequently asked questions

Does a data agent replace a business dashboard?

No. The agent suits variable questions and investigations. A dashboard provides the same approved metrics in the same form for monitoring and comparison. They create more value together than when either is treated as a replacement for the other.

Can the agent modify data in the company database?

OpenAI's example is designed for read-only SQL. That is a healthy boundary for analysis. If a later workflow needs an action, it should be exposed through a narrow, validated and confirmed function, not through broad write access to the warehouse.

Is connecting the database and starting the agent enough?

No. It needs a table catalogue, business definitions, permissions, cost limits, logging, tests and an interface in which users can inspect sources. Without context, the same column can be interpreted correctly at a technical level and incorrectly at a commercial level.

Can it work with BigQuery, Snowflake or other sources?

Integrations can be built, but the documented demonstration project uses a PostgreSQL-compatible warehouse. Every other source needs its own connector, authentication, rules and tests. Compatibility should not be assumed merely because an agent can call tools.

How do we reduce fabricated answers?

The agent should answer from executed queries and approved context, expose its sources and assumptions, and state when the evidence is insufficient. Important outputs should be compared with ground-truth results and remain subject to human review.

Is API data used for model training?

OpenAI states that API data is not used for training unless a customer explicitly opts in. Retention and application state are separate questions. Endpoint configuration, project controls and third-party policies must be reviewed before company data is connected.

The right decision starts with the rhythm of the question

If a question is new, ambiguous and requires exploration, a data agent can shorten the route to a verifiable explanation. If the question repeats, the metric should be stabilised in a dashboard. If the answer needs to change the state of a process, an application must control roles, approvals and actions.

The most mature implementation does not try to turn every report into a conversation or every conversation into automation. It keeps the agent where flexibility creates value, moves repeatable truth into the data layer, and leaves authority in code and with the people responsible.

Verified official sources

• OpenAI Developers – Data agent project and implementation guide
• OpenAI Developers – Agents API overview
• OpenAI Developers – evaluations with test sets and ground-truth results
• OpenAI Developers – adversarial testing and human oversight
• OpenAI Developers – data use, retention and available controls

Technical information was verified on 13 September 2026. The referenced Data agent is a demonstration project and implementation guide, not a promise that every data source or feature is available automatically. Capabilities, costs and data policies should be checked again for the concrete architecture.