Designing Permissioned AI Agents That Can Run Offline

AI agents are moving from chat interfaces into workflows that read files, call tools, modify state, and coordinate multiple services. In cloud-hosted designs, this often creates a fragile trust boundary: private data leaves the device, tools are authorized through broad API keys, and the user has limited visibility into what the agent can do. This talk presents an open-source, local-first approach to permissioned AI agents that can run offline on Linux edge or personal infrastructure, built on one assumption: every tool is hostile, so limits must be enforced in the tool, not in the agent. We will break the boundary down into tool invocation, deny-by-default capability scoping, keeping the grant out of the agent's reach, human approval for destructive changes, sandboxing at the process, container, microVM and Kubernetes level, and per-agent GPU slices for local model serving. The session is not about a specific agent framework; it is about the system boundary around agents. Attendees will learn how to separate reasoning from action, how to map tools to least-privilege permissions, how to defend around MCP servers they did not write, and how to keep sensitive context and compute local.