Imagine you’re a finance operation analyst, and you want to know all the overdue invoices from a specific region along with few other details like the customer’s email, account number, and payment method.
Typically, you would either wait for someone from the engineering or Business intelligence team to pull this data for you, or you would spend time in navigating through multiple pages of your application to manually get the data.
But wouldn’t it be great if you could simply ask:
“Show me all the overdue invoices for clients in Europe, along with the customer email, account details and their payment methods.”
And the system instantly understands your request, fetches the right data, and gives you exactly what you asked for, maybe as an Excel report or a visual chart or just as a clean table in your app.
Having an agent that lets you to fetch data using natural language doesn’t just save time and effort, it makes you self-sufficient giving you faster access to insights without depending on technical teams.
Building an Ai agent to fetch data using natural language is a fascinating problem. Once an agent is able access data, it can do any number of things, from retrieving insights to generating charts or Excel reports. But this kind of natural language interface comes with its own set of challenges.
I wanted to come up with a unified, plug and play solution that could address most—if not all—of these issues.
Core challenges:
Let’s break down the core challenges that stand in the way of building such an agent.
Let us walk through the challenges one by one.
One size doesn’t fit all — SQL vs NoSQL
One of the first architectural challenges is supporting any kind of data source. Regardless of SQL or NoSQL.
Both significantly differ on how they store and access data.
- SQL relies on structured schemas, tables and relationships whereas NoSQL is more flexible and document- or key-value-based.
- In SQL, querying involves joins, filters and strict typing whereas in NoSQL we deal mostly on navigating through nested documents or flattening unstructured data into something useful.
To make natural language querying truly flexible, I needed a solution that could work with any type of database—whether it’s a relational system like PostgreSQL or SQL Server or a schema-less store like MongoDB or Cosmos DB. This required a database-agnostic approach to query generation.
Where’s the data? — microservices vs monolith
If you are dealing with a monolith, things are relatively straight forward: the data reside in a single codebase and database making it easy to query data as everything is centrally available.
But in a microservices architecture, data is distributed across multiple services, each owning its own schema, logic, and possibly even its own database. This creates a major challenge:
“How to come up with a single query from natural language to fetch data that lives in multiple, isolated services?”
To make this work, we need a
- Unified schema
- Strategies to handle data duplication and network latency
- And a query plan capable of fetching data from multiple services, Ideally within a single agent call.
Not all data should be exposed — privacy and security
Data privacy and security becomes a major concern when building a system that queries it. Just because someone can ask a question doesn’t mean they should have access to all the data.
We may not want to expose certain tables/columns or specific records based on permissions, roles or data sensitivity.
Organizations dealing with sensitive or regulated data typically cannot give AI agents direct access to databases, as this raises serious concerns about data exposure, compliance, and uncontrolled query execution.
Missing context, missing meaning — legacy databases
A major problem when dealing with legacy databases is that they may not be designed with utmost clarity or modern standards in mind.
You can often find tables with columns like code1, code2 or similar cryptic identifiers. You may also encounter inappropriate or misleading data types, like storing dates as integers. Sometimes skipping relationships and foreign keys.
In some cases, the table names can have non-English and native language names making things even harder to interpret their purpose.
These design shortcuts would have helped at the time of development, but they introduce a major ambiguity.
All these things make it hard for humans to understand the system, let alone an AI agent trying to create a query without proper context.
Bring it all together: What we're really missing is semantic meaning
We have seen a range of challenges:
- Distributed microservices with fragmented models
- Legacy databases with cryptic names and missing relationships
- Poorly typed fields and inconsistent schemas
- Security concerns that limit schema exposure
As we look across these problems a clear pattern emerges:
“The real underlying issue isn’t just structural. It’s semantic.”
In other words, it’s not that we don’t have the data. The problem is the user, agent and data source lack a shared understanding of what the data means.
For a developer, missing relationships or unclear names can be worked around by going through the code, asking teammates, or reading user stories/documentations. But an AI agent doesn’t have that luxury.
Ultimately the agent doesn’t just need structure. It needs context, definitions, and relationships that reflect real-world business meaning. It needs semantics.
“This missing layer of semantic meaning is what makes natural language querying so difficult.”
How GraphQL helps us
We have discussed the challenges involved in developing a system that fetches data using natural language. Now we’ll see how GraphQL helps address these problems and walk through the approach we have taken to design the solution.
Using GraphQL gives us the below listed benefits:
Single point of entry
GrpahQl acts as a single point of entry for all fetch requests. It doesn’t matter whether you’re working with a monolith or distributed microservices system, the agent queries through one consistent API, regardless of where or how the data is stored, abstracting away the underlying complexity.
With the GraphQL interface the agent has to make only a single call to fetch the required data even if the data is split across multiple services. The GraphQL gateway takes care of resolving and making the individual service calls behind the scenes. This allows the agent to focus purely on what to ask and not on how to fetch it.
Backend-agnostic interface
The agent interacts only with the GraphQL API and not the underlying system. So, behind the scenes, GraphQL can resolve data from SQL databases, NoSQL stores or even a rest API. While using REST under GraphQL isn’t optimal, it can be a good option during the transition or for integrating third party or legacy systems.
Another advantage is that the agent need not worry about crafting efficient and optimized queries with proper joins. Most modern GraphQL frameworks (like Hot Chocolate, Hasura) offer sorting, filtering and pagination capabilities out of the box. These are exposed via the schema.
This means that:
- You need to expose one GraphQL API per table or entity.
- Agent can construct powerful queries using arguments like filter, orderBy, and first, without needing any knowledge of the underlying database syntax or optimization techniques.
Fine-grained access control
The GraphQL schema decides what is available for the agent to query. This allows you to hide sensitive data that you do not want to expose.
If we don’t want to expose a table, we can simply exclude it from the schema. The same applies to individual columns as well. Removing a sensitive field from the schema ensures it won’t be exposed through the GraphQL API. This gives us fine-grained control over what data is accessible.
We can also implement authentication and authorization to control access based on the user operating the agent. This allows us to restrict query access depending on the user’s role, adding an extra layer of security.
Built-in semantic structure
GraphQL schema by default acts as a lightweight knowledge graph. It contains information about the domain in terms of entities, fields and relationships. This makes it easier for the agent to reason about the available data and map natural language queries to valid GraphQL queries.
Even if the legacy database tables or columns have unclear or inconsistent names, the GraphQL API allows us to expose clean, well-named types and fields that reflect real business concepts.
The nature of GraphQL encourages you to model your schema with proper relationships between entities. Since GraphQL is inherently graph-based, it pushes you to define how nodes connect, making relationships explicit, navigable, and semantically meaningful.
This structure not only benefits developers but also gives AI agents the context that it needs to understand and query the system effectively.
Our approach to query generation and execution
Let’s now look at the approach to querying data using natural language through GraphQL. You can find the source code of the implementation here.
The initial thought process was to directly pass the user input and let the LLM generate the GraphQL query. However, it struggled to generated syntactically and logically correct queries.
When we broke it down to undersatnd why, a few findings emerged:
- The LLM had to do too many things at once, understand the user’s intent, extract the entities, fields, filters and sort condition.
- It had to comply with the provided schema while making sure to fulfil the user’s request.
- The input to the LLM, with respect to query generation wasn’t ideal. It lacked structure, making it difficult for the model to reliably generate correct queries.
So, in-order to tackle this, we decided to separate the concerns. This allows us to focus on individual issues. Not only solving them but also making it easier to improve and enhance them independently.
This resulted in the following steps:
Step 1: Extract the key components from the user request into a structured format.
In this step, the LLM focuses only on extracting key information like theentity to query, fields to fetch, filters/sorting to apply, and any related entities to include and convert it into a structured output. The GraphQL schema is also passed in to help the model understand the structure of the available data.
By narrowing the LLM’s responsibility to just this task, it was easy for us to understand how the LLM interpreted each request. This insight helped us to improve its performance by providing better examples and guidance. The structure output will serve as a consistent input for the next step for generating the GraphQL query.
DSPy and Pydantic frameworks helps us to enforce structure and receive output as Python objects.
Example request: Generate a chart to visualize how bill amount varies from month to month for the commercial account of John.
Output:
ReportRequest(
main_entity=’Bill’,
fields_to_fetch_from_main_entity=[‘month’, ‘amount’],
or_conditions=None,
and_conditions=[
AndCondition(entity=’account’, field=’type’, operation=’eq’, value=’COMMERCIAL’),
AndCondition(entity=’customer’, field=’name’, operation=’eq’, value=’John’)
],
related_entity_fields=[
RelatedEntity(entity=’account’, fields=[‘type’, ‘customer’])
],
sort_field_order=None,
include_count=False
)
Step 2: Generate GraphQL query
The structured output from step 1 will act as consistent and reliable input which makes it easier for the LLM to generate correct queries. In this step, we pass the structured object, along with the schema to the model and it returns the query ready for execution.
Output
{
bills(where: {
account: { type: { eq: COMMERCIAL } },
customer: { name: { eq: “John” } }
}) {
nodes {
month
amount
account {
type
customer {
name
}
}
}
}
}
Step 3: Validate the generated GraphQL query
It’s important to validate the generated query for syntactical errors because even a small mistake will cause the execution to fail. This task cannot be delegated to an LLM, since in AI generated output there is always variation involved and lack strict determinism.
we run a validation step using GraphQL parser (via libraries like graphql-core) to ensure the query is reliable. This checks the query for any syntax or structural errors before execution. This deterministic check guarantees that only well-formed queries are sent forward for execution.
Step 4: Resolve errors
In case of any errors, Another LLM call is made including the original query, validation error and the GraphQL schema. Based on the validation error the model returns the correct query. Returned query is validated again to make sure that there are no syntax or structural errors. This makes sure that only query without any validation errors is executed.
This flow control is achieved using Pydantic-graph (an async graph and state machine library for Python). We try to resolve the error for finite number of times. The process is stopped, if the LLM is unable to produce a valid query within the defined threshold.
Step 4: Resolve errors
In case of any errors, Another LLM call is made including the original query, validation error and the GraphQL schema. Based on the validation error the model returns the correct query. Returned query is validated again to make sure that there are no syntax or structural errors. This makes sure that only query without any validation errors is executed.
This flow control is achieved using Pydantic-graph (an async graph and state machine library for Python). We try to resolve the error for finite number of times. The process is stopped, if the LLM is unable to produce a valid query within the defined threshold.
Conclusion
We started with a simple, relatable problem: “What if you could just ask for the data you need?” and explored on how empower AI agents to fetch data using natural language.
But as we saw, bridging the gap between user intent and database execution is no small task. Challenges like database diversity, isolated services, privacy concerns, and missing semantic context all stand in the way.
By introducing GraphQL as a consistent, secure, and semantically meaningful interface, we were able to:
- Abstract away backend complexities,
- Enforce access control at the schema level,
- Normalize inconsistent structures,
- And provide the AI agent with a unified way to understand and interact with the data world.
This isn’t just a technical pattern but a blueprint for building intelligent, explainable, and scalable data access systems.
Source code
Reference
All the images used in this article are generated by the author using OpenAI’s Sora.
DSPy: https://dspy.ai/learn/
Pydantic: https://docs.pydantic.dev/latest/
Pydantic-ai: https://ai.pydantic.dev/
No comment yet, add your voice below!