Interpret and decide.
Interpret instructions, reason about the task, pick a tool, generate a structured action, and evaluate whatever comes back.
Layer / AI & Development
Ayron Jins AI systems.
Turn models into systems.
AI and Development technical content
Building systems where AI can reason, use tools, interact with software and actually do digital tasks, sitting on top of the infrastructure that makes any of it possible.
04 / 05
FLAGSHIP PROJECT / CONTROLLED COMPUTER INTERACTION
AI with direct computer access.
Traditional assistants mostly just produce text. This project asks what changes once a language model gets connected to controlled tools and a real computer environment, the model stops just suggesting what to do and starts participating in a verified action loop.
The important idea isn't "an AI that clicks." It's a surrounding system that gives the model structured capabilities, watches the resulting state, handles failures, and keeps the user in control the whole time.
FROM CHATUser goal → model → text response.
TO AGENT SYSTEMUser goal → reasoning → tool selection → action → observation → next decision.
Explain the difference
Text generation versus controlled action.
CHATBOT / RESPONSE
USERPROMPTLLMTEXT RESPONSE
AI AGENT / CONTROL LOOP
USER GOALLLM / REASONINGPLANTOOL SELECTIONCOMPUTER ACTIONOBSERVATIONREASON AGAINTASK COMPLETE
The agent loop
Observe. Decide. Act. Verify.
The loop is understandable without animation: each stage returns information to the next decision.
OBSERVE
Receive information about the current task or environment.
Receive information about the current task or environment.
VERIFY OUTCOMES↺ INCOMPLETE / FAILED → OBSERVECOMPLETE → RETURN RESULT
INPUT REQUIRED → ASK USER
INPUT REQUIRED → ASK USER
Core architecture
A model inside a controlled software system.
CONTROLLED AGENT ARCHITECTUREUSER → AGENT CONTROLLER
CONTROL FLOW · SELECT A LAYER
Model layer
The LLM reasons; the surrounding system grants capability.
Reach the machine automatically.
Access files, click interfaces, run commands, inspect a computer. None of that happens without an agent/tool layer explicitly handing it those capabilities.
Tool layer
Tools turn intent into action.
Tools are controlled bridges between a model and an external system. The capability list should stay explicit and easy to update, rather than quietly implying unrestricted access.
FILESRead or create files where permitted.
COMMANDSExecute approved system operations.
WEBNavigate supported web resources.
DOCUMENTSCreate structured outputs.
APPLICATIONSInteract with supported software.
SYSTEMInspect available state.
APISCommunicate with external services.
PERMISSIONSKeep action boundaries visible.
Computer control
Action is only useful when state is verified.
Computer-control execution layer
One possible execution layer lets an agent interact with a computer environment. The real engineering problem is understanding the current state, picking the right action, running it, checking the result, recovering when something changes unexpectedly, and still getting back to the original goal.
AI DECISION↓ACTION REQUEST↓CONTROL LAYER↓KEYBOARD / MOUSE / API↓NEW STATE
Create a project report.
01UNDERSTAND
Interpret the request.
Interpret the request.
02INSPECT
Find available files.
Find available files.
03EXTRACT
Gather relevant information.
Gather relevant information.
04CREATE
Generate the document.
Generate the document.
05VERIFY
Confirm the output exists.
Confirm the output exists.
Human in the loop
Control is part of the architecture.
Controlled access is just stronger engineering than unrestricted access. Action levels need to stay visible and configurable.
LOW RISK / READRead or analyze information where allowed.
MEDIUM RISK / CREATECreate or modify output; confirmation may be appropriate.
HIGH RISK / CHANGEDelete, send or change systems only with explicit approval.
The hard part is not the chat window
Agent systems are systems problems.
STATEThe environment changes after every action.
TOOL SELECTIONThe system must choose an appropriate capability.
ERROR RECOVERYTools can fail or return unexpected results.
CONTEXTThe original goal must survive intermediate steps.
VERIFICATIONThe agent must know whether the action worked.
LOOPS / LATENCYRepeated interactions need boundaries and patience.
Action → result → decision.
↓
↓
↓
An agent should inspect the result before assuming success.
Keep the next decision grounded.
The useful context boils down to the user's goal, what's already been tried, what the tools returned, and the current state of the computer. This is describing task context, not claiming the system has persistent long-term memory.
USER GOALPREVIOUS ACTIONSTOOL RESULTSCURRENT STATENEXT MODEL DECISION
Agent command center / simulation
See the runtime as an event stream.
STATIC INTERFACE EXAMPLE / NOT LIVE TELEMETRY
Agent Command Center
This static interaction example shows how a controlled agent can surface its task, chosen tool, result, and verification state, without leaking prompts, credentials, or private system details.
TASKPortfolio quality audit
STATUSCOMPLETE
SELECTED VIEWSource structure
MODECONTROLLED
TOOLSFILES · DOM
TOOL files.inspect
INPUT index.html
RESULT structure mapped
INPUT index.html
RESULT structure mapped
[01] Source opened
[02] Sections indexed
[03] CSS rules mapped
[04] Interaction handlers mapped
[05] Sensitive-data scan passed
[06] Structure verified
[07] Complete
[02] Sections indexed
[03] CSS rules mapped
[04] Interaction handlers mapped
[05] Sensitive-data scan passed
[06] Structure verified
[07] Complete
AI infrastructure
Local compute makes the experiment tangible.
HOMELAB↓LOCAL COMPUTE↓OLLAMA↓LOCAL LLM↓OPENWEBUI / INTERFACE↓TOOLS / SERVICES
Ollama handles the local model runtime for experimentation. Open WebUI gives an interface for actually talking to those local AI services. Depending on the implementation, agent components may connect through different interfaces rather than all funneling through one path.
Control, learn, experiment.
Local models can support model choice, infrastructure learning, privacy-aware workloads, experimentation, and cost control once the model is actually available, though none of that is automatic in every scenario.
Use engineering judgment.
Local models offer ownership and the possibility of offline operation, but they need hardware and upkeep. API models offer hosted capability and easier scaling, but bring network, cost, and data-handling dependencies along with them.
Knowledge systems
RAG and fine-tuning solve different problems.
Bring relevant context to the request.
RAG pulls in external knowledge at inference time, instead of relying purely on what's baked into the model weights.
QUESTION↓SEARCH↓DOCUMENTS↓CONTEXT↓LLM
FINE-TUNING
Specialize behavior with examples.
Fine-tuning changes how a model behaves through extra training examples, a different thing entirely from treating a database as a source of facts.
BASE MODEL↓TRAINING EXAMPLES↓FINE-TUNING↓SPECIALIZED MODEL
PROMPTING
How should the model respond?
RAG
What external information can it access?
FINE-TUNING
How should behavior be specialized?
TOOLS
What actions can the model perform?
AGENT
How can reasoning and actions combine across steps?
EVALUATION
Did the system remain useful, safe and verifiable?
Development stack
Build around the model.
LANGUAGEPython / automation
BACKENDAPIs / FastAPI where used
DEPLOYMENTDocker / Linux
VERSIONINGGit / GitHub
LOCAL AIOllama / OpenWebUI
INTERFACESWeb apps / dashboards
AI is more than prompting
Useful systems have layers.
MODEL↓PROMPT / INSTRUCTIONS↓CONTEXT↓TOOLS↓APPLICATION LOGIC↓INTERFACE↓INFRASTRUCTURE
A good model alone doesn't make a reliable product. Reliability comes from everything built around it.
Evaluate the system, not just the answer.
TASK COMPLETION · intended goal achieved
TOOL USE · appropriate capability selected
RECOVERY · continues after failure
SAFETY · stays within boundaries
EFFICIENCY · avoids unnecessary actions
CONSISTENCY · behaves reasonably across tasks
Design for what goes wrong.
WRONG TOOLChoose another capability.
STALE STATERe-observe the environment.
INVALID OUTPUTValidate returned data.
LOOPSet a stop boundary.
DENIEDAsk for approval or stop.
AMBIGUOUS GOALRequest clarification.
PRIVACY & SECURITY
Public material must not expose API keys, secret system prompts, local IPs, tokens, private URLs, authentication information, credentials, or unrestricted computer-control endpoints. Real screenshots get sanitized before they go up.
References / Verified sources
AI & development references.
Official documentation for the tools and runtimes discussed in this technical work.
SOURCES / 08