Assistant Agent

Overview

Assistant Agent is a personal AI chatbot assistant built as part of my broader Agent Platform project. The goal of the project is not just to build another chatbot, but to create a hands-on environment for experimenting with what makes an AI agent useful in practice.

The assistant currently supports tool calling, semantic memory, persistent conversation history, reasoning visibility, feedback collection, and a web-based chat interface. It uses a FastAPI backend, a Next.js frontend built on assistant-ui, PostgreSQL for persistence, and pgvector-backed semantic memory.

This project originally started as an attempt to build an autonomous coding agent. I later decided to first build a strong chatbot assistant agent, because it provides a more flexible way to test the core design problems that appear across many agentic systems: how tools should be defined, when memory should be retrieved, how context should be managed, when prompts should be preprocessed, and when more complex orchestration patterns are actually useful.

Since I am the primary user, I can use my own interactions as feedback and continuously tune the assistant based on what is genuinely useful in practice.

Demo

The assistant workflow starts with signing in, then moves into a persistent chat interface where the assistant can reason, call tools, retrieve memory, and stream responses back to the user.

1) Sign in

The user starts by logging into the assistant web app. The app supports email/password authentication, but the preferred flow is signing in with a Google account through Google OAuth.

After authentication, the frontend stores the session token and uses it to authorize subsequent requests to the backend.

2) Open or create a thread

After signing in, the user enters the main assistant interface. Existing conversation threads are shown in the sidebar, and the user can either continue a previous thread or start a new one.

This matters because the assistant is designed around persistent conversations rather than one-off prompts.

3) Ask a question or start a task

The user interacts with the assistant like a normal chatbot. For example, the screenshot shows a travel-related query asking about popular travel destinations for Singaporeans.

Although the interface looks like a standard chat window, the backend runs an agent loop that can decide whether to answer directly or use tools before responding.

4) Inspect reasoning and tool usage

When the assistant uses tools, the UI surfaces this inside the expandable Thinking section. This gives visibility into the assistant’s intermediate process without requiring a separate debugging screen.

In the example, the assistant:

• reasons about what information is needed,
• uses web search,
• checks memory through assistant_recall,
• fetches relevant pages,
• and then summarizes the result into a final answer.

This makes tool usage and reasoning inspectable while keeping the main experience close to a normal chat interface.

5) Continue the conversation

The response is streamed back into the chat interface, and the thread is saved so the user can return to it later. Over time, the assistant can use stored memories and previous context to better support the user’s personal workflows.

The goal is not to expose every backend mechanism as a separate UI screen, but to make the assistant usable as a normal chatbot while still providing enough visibility to debug and improve agent behavior.

What I built

• Assistant web app: Built a chatbot-style assistant interface using Next.js and assistant-ui, with custom UI components for reasoning display, tool calls, feedback, and thread management.
• FastAPI backend: Implemented the backend API for authentication, threads, chat streaming, feedback collection, and assistant execution.
• ReAct-style assistant loop: Built an assistant loop that sends messages to the LLM, handles tool calls, executes tools, stores tool results, and continues until a final response is produced.
• Custom tool framework: Created a lightweight tool system where tools define their name, description, JSON schema parameters, usage guidance, and async execution logic.
• Semantic memory system: Implemented memory storage and recall using PostgreSQL, pgvector, bge-m3 embeddings, and cosine similarity search.
• Persistent conversation history: Stored user messages, assistant responses, reasoning content, tool calls, and tool results in PostgreSQL.
• LLM observability: Logged LLM call metadata such as token usage, cached tokens, latency, finish reason, reasoning output, and tool-call behavior.
• Feedback collection: Added a feedback flow to capture response-level feedback and conversation snapshots for later debugging.
• Shared core library: Structured reusable infrastructure into a shared core package for LLM calls, tools, configuration, database access, and embeddings.
• Context management infrastructure: Started building adaptive context compaction for long conversations, including checkpointing and summarization infrastructure.

Key decisions

• Start with the assistant before the coding agent: I originally wanted to build an autonomous coding agent, but decided that a general assistant was a better foundation. It lets me test memory, tools, context management, and UI feedback loops across many types of tasks before specializing into coding workflows.

• Use a custom tool framework instead of a large agent framework: I wanted direct control over tool definitions, prompt construction, and execution flow. This makes it easier to experiment with how small changes in tool descriptions or usage guidance affect model behavior.

• Use assistant-ui to avoid overbuilding the frontend: I used assistant-ui as the frontend base so I could avoid spending most of the project rebuilding standard chat primitives such as message rendering, composers, threads, and chat layout. This let me focus more on the core learning goal of the project: building and evaluating the assistant agent itself, including tool design, memory behavior, context management, reasoning visibility, feedback collection, and backend integration.

• Expose memory through tools instead of always injecting it: Automatically inserting memories into every prompt can bloat context and introduce irrelevant information. Tool-based memory retrieval lets the assistant decide when memory is actually needed, though it also means the model may sometimes fail to recall relevant context.

• Use PostgreSQL with pgvector instead of a separate vector database: Keeping relational data and vector search in one database reduces operational complexity for a personal project while still supporting semantic memory retrieval.

• Treat this as a learning project, not just a product: Many agent design choices are hard to evaluate theoretically. This project is intentionally built so I can test design patterns through real use, observe failure modes, and iterate.

Constraints & limitations

• Design choices are still being validated: Many implementation decisions in this project are based on my current understanding of agent design and my own judgement about what should work well in practice. This includes choices around tool definitions, memory retrieval, prompt structure, context compaction, and how much control should sit with the model versus the surrounding system. These decisions are still being tested and refined through continued usage.

• Performance is not fully tuned yet: The assistant is functional, but it should not be treated as a polished production system. Further testing, debugging, and prompt/tool tuning are still ongoing, so response quality, tool-use accuracy, memory behavior, and latency may vary across tasks.

• Evaluation is focused on real usage first: I plan to evaluate the assistant mainly through direct usage and feedback from myself, family, and friends. While formal benchmarks can be useful, I believe the most important measure for this type of personal assistant is whether it works well for the actual workflows and users it is meant to support.

• Memory behavior is still experimental: Tool-based memory retrieval helps reduce unnecessary context bloat, but it depends on the model choosing to recall memory at the right time. I am still experimenting with how memory should be stored, retrieved, deduplicated, and surfaced.

• Context management is still evolving: Adaptive context compaction is part of the project direction, but the exact strategy still needs more testing. The challenge is preserving useful long-term context without adding too much noise or losing important details.

• Broader agent architecture is future-facing: The current working product is the assistant agent. The repository is structured to support future experiments with coding agents, orchestrator-led workflows, and multi-agent patterns, but those parts are not the main completed system yet.

Setbacks / lessons learned

• Agent quality depends heavily on tool design: Small changes in tool descriptions, parameters, and usage guidance can significantly affect whether the model chooses the right tool and uses it correctly.

• Memory should not be treated as “always more is better”: Passing too much memory can make the assistant less focused. Retrieval needs to be selective, relevant, and tuned around actual user needs.

• A simple ReAct loop is a good baseline, but not always enough: A single-agent tool loop works well for many tasks, but more complex workflows may benefit from orchestration, planning, or specialized sub-agents. The challenge is knowing when the added complexity is worth it.

• Observability is important for agent development: Logging token usage, latency, cached tokens, reasoning output, and tool-call behavior makes it much easier to understand why the assistant behaved a certain way.

• Frontend design affects agent usability: Reasoning display, tool-call visibility, and feedback collection are not just UI features. They make it easier to inspect, debug, and improve the assistant over time.

Future considerations

• Improve memory deduplication, retrieval quality, and staleness handling.
• Continue refining prompt structure and context compaction.
• Add more personal assistant tools.
• Add skill support so the assistant can use reusable workflows and task-specific guidance beyond raw tool calls.
• Build practical skills for common workflows such as research, document analysis, code inspection, planning, and memory usage to improve the assistant’s reliability and task performance.
• Explore orchestrator-led ReAct loops where a central orchestrator manages task state and delegates subtasks to specialized sub-agents.
• Experiment with parallel tool or agent calls for independent subtasks.
• Add stronger evaluation through usage logs, feedback review, and targeted test cases.
• Expand the coding-agent prototype once the assistant foundation is more stable.
• Explore MCP or another integration protocol for future tools and agents.
• Add more production hardening, including tests, rate limiting, deployment setup, and monitoring.
• Explore agent self-extension, where the assistant can identify repeated workflows or missing capabilities, propose new tools or skills, test them in a controlled environment, and add them to a reusable tool/skill library after review. This would allow the agent to gradually improve its practical usefulness without relying only on manually predefined tools.