What a year of agent projects taught us to ask before building anything

Pretty much exactly a year ago Des Train AI launched with it's objective of providing custom AI agents for business processes. In that time, the single most consistent request we've got is "we want an AI agent". However, it tells us almost nothing about what to build. When solving a custom business process problem, an agent is not a product you pick off a shelf. It's an architecture you design around a specific problem, a specific user, and a specific set of systems. The questions below are the ones we've learned to ask before touching a keyboard.

One thing worth clarifying upfront, because it shapes everything else: AI agents are not chatbots. A chatbot responds. An agent acts — it plans, calls tools, handles dependencies, and completes tasks that often take minutes or even hours to resolve. That's long by AI standards, but still a fraction of what a human would need to do the same work. Keep that in mind as you read through the questions. Some of them only make sense once you understand what you're actually asking the agent to do. Read more from our previous post What Is an AI Agent? The Definition

1. Where does it live?

Will this agent run on someone's local laptop, or will it be deployed as a reusable service? A local agent is much more simpler and faster to prototype but it scales poorly and might be hard to maintain.  A server-side deployment opens up multi-user access, better observability, and integration with enterprise systems, but it requires thinking seriously about reliability, cost, and security from day one. The answer here shapes almost every architectural decision that follows. We ask this first because we've seen projects get months in before discovering the deployment model doesn't match the use case. Ultimately boils down to basic software processes, organizational guidelines and architecture, not so much on the agent itself.

2. How many users will it serve?

An agent built for one internal analyst is a very different system from one built for ten thousand customers. With proper archtecture we can extend simplicity flexibly and cheap to few persons, even a small team, but as soon as the amount of users soar it requires different level of scalability. Scale affects not just compute and cost, but how you handle rate limits on third-party tools, how you queue and prioritize tasks, and whether you need to isolate user contexts from each other. The gap between "a few internal users" and "customer-facing at scale" is larger than most clients expect when they first describe their requirements.

3. How does the user authenticate and how does the agent authenticate to its tools?

This is where more agent projects run into trouble than anywhere else. Authentication is not just about the end user logging in. It's about the agent itself having the right credentials to act on the user's behalf across multiple tools and systems. If the agent connects to a CRM, a calendar, a document store, and an internal API, each of those connections may have its own auth requirements. MCP (Model Context Protocol) is increasingly the standard way to handle these tool connections cleanly, but even with MCP in place, someone has to define who the agent can act as, under what permissions, and with what limits. This question has legal and security implications, not just technical ones and it rarely gets the attention it deserves until something goes wrong.

4. What tools does it have and which ones read versus write?

The tools are the agent's capabilities, that is the entire answer to what it can actually do. Which systems does it connect to, which information sources does it draw from, what problem is it solving with them? This is where MCP earns its place: well-documented, standardized tool definitions the model can call reliably. But listing capabilities is only half the question. The read/write distinction is where risk enters: reading from a database is low-stakes, while writing to production systems, sending emails, or updating customer records can cause real damage if the agent is manipulated or simply wrong. And prompt injection through untrusted content is a genuine attack surface. That's why guardrails belong at the tool level: tight permission scoping, confirmation steps before irreversible actions, hard limits the tools enforce regardless of what the model decides. Guardrails at the model level help; guardrails at the tool level hold. You always need to map out what is the catastrophic failure maximum impact and be content with that.

5. What does the user interface look like?

Not all agents live behind a chat window. Some are invoked from a command line. Some listen for voice. Some are triggered by an event like an incoming email, a form submission, a calendar alert without direct user interaction at all. The interface shapes the entire interaction model: how the user provides context, how the agent communicates back, and how the user intervenes mid-task. It's also a significant workload for the project to create the user interface which again is more of a traditional software development issue.

6. What does it need to remember?

Memory in agents exists on a spectrum. At the simplest end, the agent remembers only the current session. More capable agents remember previous conversations. More capable still, they maintain persistent knowledge about the user like preferences, past decisions, ongoing work leading effectively to a long-term operational context. At the far end, some use cases require returning to a specific checkpoint in a past conversation and branching from there. Each step up the memory ladder adds complexity, storage requirements, and privacy considerations. We've found that clients almost always want more memory than they actually need. Defining what the agent must remember to do its job usually results in a simpler and more reliable system. Minimum for all agents is to at least have "write to guidelines" tooling. But then the next question immediately becomes: Should those guidelines be shared between users? That again depends on the case.

7. How do you measure whether it's working?

An agent without measurement is a black box. Quality in agents is harder to define than in traditional software. It always answers something so you're often evaluating judgment, not just correctness. Does the agent complete the task? Does it complete it accurately? Does it know when to stop and ask for help rather than proceeding with incomplete information? Before deploying, you need to define what "good" looks like: task completion rate, error rate, time-to-completion, user satisfaction, or some combination. Without this, you don't have a way to detect for example whether a model update upgrades or degrades performance. Something we've seen catch teams off guard more than once.

8. How do you gather feedback and improve based on it?

Measurement tells you how the agent is performing. Feedback tells you why it's failing and what to do about it. Feedback loops can be explicit like users rating outputs, flagging errors, correcting the agent's work or implicit, derived from behavioral signals like task abandonment or repeated retries. Both matter. A well-designed feedback loop turns every failure into signal. Over time it becomes one of the most valuable assets the system produces, and one of the clearest competitive advantages for teams that build it early. First iteration of an agent never is very good, but with a fast feedback loop the 5th to 10th iteration usually is!

9. How does the agent improve over time?

This is the question most teams skip when they're focused on shipping and the one we always make time to answer before a project closes. Agents can improve through periodic fine-tuning on accumulated feedback, through better tool definitions and documentation, through refined system prompts based on observed failure patterns, or through architectural changes as the problem becomes better understood. Some improvements happen automatically; most require deliberate work. An agent that isn't designed to improve is one that gets relatively worse as the business around it evolves.

Finally

These nine questions come from a year of building agents across different industries, team sizes, and integration environments. None of them have universal answers and that's the point. What they do is replace the vague ambition of "we want an AI agent" with a concrete design brief. The agent you end up building should match the problem you actually have, not a generic approximation of one. And that's only the agent itself. Re-designing the business process around it is a separate checklist entirely. More on that another time.

And to close where we started: an agent operates on a fundamentally different timescale than a chatbot. It may take ten minutes to complete a task that would take a human two hours. That gap is where the business value lives but capturing it requires building something genuinely designed to act, not just to respond.

---

Jonas Pomoell is the founder and lead AI business consultant at Des Train AI Oy, specializing in practical AI agent implementation for business. If you're working through these questions for your own use case, we'd be glad to help.

Illustration Photo by Laura Chouette on Unsplash.