Why this matters
An Astra AI Agent is only as effective as the knowledge and instructions it uses. Incomplete knowledge, unclear instructions, or insufficient testing can lead to responses that are inaccurate, incomplete, or outside the intended scope.
A well-configured agent should provide useful answers from your approved business knowledge, recognize when it cannot answer a question, and hand off conversations when human assistance is needed.
Regular testing and evaluation also help you identify gaps before customers encounter them.
Who this is for
This applies to you if:
You are configuring or managing an Astra AI Agent.
Your agent uses multiple knowledge sources such as websites, documents, Q&A, or WhatsApp conversations.
You want to improve the quality and consistency of your agent's responses.
You want to reduce the risk of unsupported or out-of-scope responses.
This probably doesn't apply if:
You are looking for instructions to create an Astra AI Agent from scratch.
You are troubleshooting a specific agent issue, such as an agent not responding or not using a particular knowledge source.
Terms to know
Knowledge sources
Information that Astra can use when responding to customers. You can add sources such as websites, documents, Q&A, and WhatsApp conversations.
Instructions
Rules and guidance that define how your AI Agent should behave, including what it should answer, when it should refuse a request, and when it should hand off a conversation to a human.
Focused Search
Astra capability that helps an AI Agent retrieve information from the most relevant knowledge source when your business has multiple knowledge domains.
Evaluation
A way to assess your AI Agent's behavior before deployment so you can identify areas that need improvement.
The tips
Prefer a visual walkthrough? Watch our video guide on Astra AI agent best practices:
Tip 1: Build a comprehensive knowledge base
Why it matters
Your AI Agent needs relevant and current business information to provide useful answers. Missing or outdated information can limit the agent's ability to answer customer questions correctly.
How to do it
Add the knowledge sources that are relevant to your use case, including websites, documents, Q&A, and WhatsApp conversations. Organize information into meaningful business domains where appropriate.
If your business covers multiple products, services, or knowledge areas, you do not necessarily need a separate agent for each one. Use Focused Search to help Astra retrieve information from the most relevant knowledge source.
Outcome
Your agent has access to the information it needs and can retrieve relevant knowledge across different areas of your business.
Tip 2: Make your instructions explicit about boundaries
Why it matters
An AI Agent should not have to infer when it should answer, refuse, or ask a human for help. Ambiguous instructions can result in the agent attempting to answer questions outside its intended scope.
How to do it
Clearly define:
What the agent should answer.
Which knowledge it should use.
What it should do when information is unavailable.
Which requests are outside its scope.
When it should refuse to answer.
When it should hand off the conversation to a human.
For example, instruct the agent to answer using available knowledge and not invent or assume information that is not present in its knowledge sources.
Outcome
The agent has clearer behavioral boundaries and is more likely to handle unsupported or out-of-scope questions appropriately.
Tip 3: Design the agent for its channel
Why it matters
The customer experience and configuration requirements can differ depending on where the agent is used.
How to do it
Consider the intended channel when creating and configuring your agent. Astra currently requires separate AI Agents for WhatsApp and web chat widgets.
If you use agents on both channels, align their relevant knowledge, instructions, and response guidelines so customers receive a consistent experience.
Outcome
Your agent is configured with the intended customer experience and channel requirements in mind.
Tip 4: Test and evaluate before deploying
Why it matters
An agent can appear correctly configured while still producing unexpected responses. Testing helps you identify issues before customers interact with the deployed agent.
How to do it
Test realistic scenarios, including:
Questions the agent should answer.
Questions that require information from different knowledge sources.
Questions that are outside the agent's scope.
Requests that should be refused.
Situations that should trigger a human handoff.
Always run an evaluation before deploying the agent. If the agent's behaviour is not up to the mark, review and update its knowledge or instructions, then test and evaluate it again.
Outcome
You identify and address response-quality issues before deploying the agent to customers.
Tip 5: Publish only after your changes have been evaluated
Why it matters
Testing and evaluation only improve the version you have configured. Customers need the tested version to be published for those changes to take effect in the deployed agent.
How to do it
After updating your knowledge, instructions, or configuration:
Test the changes.
Run the evaluation.
Review the results.
Make further improvements if needed.
Publish the changes once you are satisfied with the agent's behavior.
Outcome
The version you deploy reflects the changes you have tested and evaluated.
Tip 6: Continuously review and improve agent behaviour
Why it matters
Your business knowledge and customer questions change over time. An agent that performs well today may need updates as your products, policies, or processes change.
How to do it
Regularly review agent conversations and identify:
Questions the agent could not answer.
Incorrect or unsupported responses.
Gaps in your knowledge sources.
Outdated information.
Unnecessary refusals or human handoffs.
Instructions the agent did not follow as expected.
Use these findings to update your knowledge sources or instructions. Test and evaluate the changes again before publishing.
Outcome
Your AI Agent stays aligned with your current business information and customer needs.
How to know it's working
Review your agent's conversations and evaluation results regularly. Look for improvements in areas such as:
The agent answers more of the questions it is expected to handle.
Responses are grounded in your available business knowledge.
Out-of-scope questions are handled according to your instructions.
Human handoffs happen when they are appropriate.
Fewer responses require correction or additional customer clarification.
Evaluation results improve after you update your knowledge or instructions.
There is no single accuracy metric that guarantees an agent will perform correctly in every customer scenario. Use evaluation results and conversation reviews together to identify areas for improvement.
Common mistakes to avoid
Adding a large amount of information without organizing it into useful knowledge sources.
Assuming the agent will automatically know when it should refuse or hand off a conversation.
Using the same agent configuration without considering whether the agent is intended for WhatsApp or a web chat widget.
Deploying an agent without testing realistic customer scenarios.
Skipping evaluation before deployment.
Making changes but forgetting to publish them.
Treating the initial setup as complete and never reviewing agent conversations.
Updating instructions without checking whether the underlying knowledge source also needs to be updated.
🚀 What's next
Now that you have configured and evaluated your agent, regularly review its conversations to identify knowledge gaps and unexpected behavior. Use those findings to continuously improve your knowledge sources and instructions before publishing future changes.