AIMEC / Engineering notes / Build logs
What we tested.
What actually happened.
A running record of experiments with small language models, browser agents, local inference, orchestration and reliability. We include things that worked, failure modes and findings that remain provisional.
Research notes assembled from recorded experiments and working sessions through November 2026. Individual prototypes may be independent explorations, not AIMEC products or client deployments.
How to read this page
Evidence before claims.
A planning benchmark, a tool-call test and a completed real-world task measure different things. We report them separately. If a workflow failed, that failure belongs in the record.
Documented research threads and experiments, not 18 finished products
Model size used for one internal semantic coverage comparison
Correct cases out of 12 on that limited benchmark
Measured example / Semantic coverage
Improving the surrounding system without increasing model size
83.3% correct
91.7% correct
One additional correct result (+8.3 percentage points) in a 12-case test. Unsupported-case handling was 100% in the tested iterative configuration. These numbers are not statistically robust proof of general superiority.
Track 01 / Browser agents
Turning browser tools into reliable actions.
A small model can often choose a plausible action. The harder question is whether it targets the right element, fills valid arguments, recognises a failed interaction and verifies the goal.
Browser tools with Playwright and MCP
- Question / method
- Tested browser navigation, page snapshots, click and extraction tools against real websites, including an automation test storefront.
- Observation
- Live navigation and selected QA steps worked. More complex multi-step runs exposed wrong target selection, repeated actions, premature completion and unverifiable claims.
- Engineering lesson
- Keep action traces, restrict high-risk tools and verify success against actual page state rather than trusting an agent’s final message.
Compact browser state and simplified tool schemas
- Question / method
- Experimented with replacing large browser snapshots and tool schemas with smaller page representations and a constrained action vocabulary.
- Observation
- This reduced the amount of information a small model needed to interpret, but did not by itself establish reliable end-to-end browser task completion.
- Engineering lesson
- Separate observation, decision, argument filling, execution and post-action verification. Treat each as an independently testable component.
Deterministic product search and evidence checking
- Question / method
- Investigated a search helper for product-page tasks after a browser agent claimed that a search result had a missing image without adequate verification.
- Observation
- The agent’s defect report required independent validation; the wording alone was not trustworthy evidence of a broken image.
- Engineering lesson
- Find concrete DOM evidence for product name, price and image, and distinguish a failed image load from an incorrect model interpretation.
Argument-fill and cart workflow failure
- Question / method
- Traced a task to add a product, verify quantity and price, remove it and confirm the cart was empty.
- Observation
- The captured run lasted approximately 49 seconds and ended with an argument-fill error. It did not complete the requested cart verification.
- Engineering lesson
- Apply typed argument contracts and deterministic repair before tool calls; a successful navigation is not the same as completing a user goal.
Track 02 / Small model architecture
More capability from the harness.
These tests separate classification, extraction, retrieval, execution and verification, so that every improvement has a measurable target.
Fixed DAG planning with small decision and extraction models
- Question / method
- Split task planning, workflow routing, argument extraction and execution across components including a Strands decision model and MiniCPM-based extraction.
- Observation
- In one fixed-DAG evaluation, the combined setup produced complete plans for 23 of 25 cases (92%). A separate holdout evaluation reached 11 of 15 (73.3%).
- Engineering lesson
- Planning success is not browser success. Execution needs grounded elements, valid arguments, state updates and evidence of completion.
Routing benchmarks: Strands versus Qwen
- Question / method
- Compared routing decisions and latency between the Strands decision component and a Qwen alternative under the same historical evaluation.
- Observation
- Recorded routing results were 105/125 (84%) versus 95/125 (76%). Corresponding mean decision latencies were approximately 1.98 seconds versus 3.97 seconds.
- Engineering lesson
- A specialised router may improve a narrow decision step, but that result cannot be generalised to complete autonomous tasks.
Iterative semantic coverage with a 9B model
- Question / method
- Compared single-pass retrieval against evidence accumulation and deterministic requirement coverage using the same small planning model.
- Observation
- The recorded 12-case benchmark improved from 10/12 to 11/12 correct cases. Unsupported-case handling reached 100% in the tested iterative configuration.
- Engineering lesson
- Make requirements explicit, check which have evidence, and recover missing pieces without escalating the model size first.
Goal interpretation and lightweight verification
- Question / method
- Explored small-model interpretation of a natural-language request into success criteria, then yes/no checks following each browser action.
- Observation
- A useful design direction emerged, but available records do not demonstrate general end-to-end reliability for this configuration.
- Engineering lesson
- Keep success criteria stable across the run and compare them with observed evidence rather than allowing the agent to rewrite its goal.
Track 03 / Local inference
Engineering within a hardware budget.
Running a model is only one part of operating an AI system. Memory, embedding availability, concurrency and recovery affect whether a deployment is usable.
Memory and GPU constraints for multi-model agents
- Question / method
- Tested local decision and extraction components on consumer hardware with limited GPU memory.
- Observation
- A Strands configuration could not fit in roughly 4 GB of NVIDIA VRAM. In CPU testing, a process approached 7 GiB and was terminated under system memory pressure on a 14 GiB machine.
- Engineering lesson
- Model choice, concurrency, quantisation, and process isolation must be designed around real memory budgets.
Qwen model sizes and local agent performance
- Question / method
- Investigated Qwen-family models across small parameter sizes for planning, extraction and tool use; more recent 4B experiments extend this line of work.
- Observation
- Results vary by model and task. A single validated cross-model performance or cost ranking has not yet been established.
- Engineering lesson
- Report model version, quantisation, device, prompt/context size and exact evaluation task alongside any headline score.
Private inference and embedding services
- Question / method
- Worked with local inference infrastructure, including llama.cpp-based embeddings and a document-indexing workflow.
- Observation
- One indexing pipeline initially failed when its embedding service was absent. After the service was configured, health checks succeeded; missed indexing events still needed recovery.
- Engineering lesson
- A private AI deployment needs service monitoring, retry-safe ingestion and observability, not merely a running language model.
Track 04 / Orchestration and agent systems
From component success to end-to-end outcomes.
An agent can delegate, use tools and call a model successfully while still failing to deliver the requested result. These experiments expose the seams.
Multi-agent delegation and return-path failure
- Question / method
- Evaluated a private agent-network workflow in which work was delegated to other nodes and returned for synthesis.
- Observation
- In an acceptance run, delegation and task processing completed, but the final locally synthesised response timed out after the test window.
- Engineering lesson
- Track each transition from delegation to synthesis; a completed worker task is not a completed user request.
Agent tooling, approvals and security boundaries
- Question / method
- Tested tool discovery, safe tool access, approval checks and agent execution through MCP-connected systems.
- Observation
- A live browser step succeeded in one investigation, while an agent selected an unsafe browser execution tool and had to be paused for approval. Missing capabilities also prompted inappropriate workaround attempts.
- Engineering lesson
- Expose only needed tools, constrain permissions and distinguish a recoverable tool failure from authorisation to try another route.
Docker-based AI capability integration
- Question / method
- Tested a local stack joining agent-computer, world-model, forecasting, platform and inference components.
- Observation
- Integration checks found both functioning components and failures, including container-to-host Ollama connectivity and incompatible contracts. Later repairs verified several exposed capabilities.
- Engineering lesson
- Integration readiness depends on network addressing, service contracts and resource limits, not the count of running containers.
Persistent agents, memory and scheduled development
- Question / method
- Explored local agent harnesses with memory, schedulers, development tasks, approvals and inspectable status.
- Observation
- Some component tests and dashboard-reported milestones were recorded; runtime memory pressure also deferred further development. These do not establish unattended autonomy in production.
- Engineering lesson
- Preserve explicit approvals, reproducible traces, clear resource thresholds and a recoverable state between runs.
Track 05 / Ongoing explorations
Questions still open.
Some work has produced reproducible outcomes; other research remains at the prototype or evaluation-design stage. These entries state the difference.
Desktop automation and computer-use harnesses
- Question / method
- Investigated Vercel Agent Desktop and other computer-use patterns for testing compact models beyond browser-only actions.
- Observation
- Early work focused on controlled desktop tasks and evaluation setup; the available evidence does not support a broad desktop reliability claim.
- Engineering lesson
- Extend from constrained fixtures to harder workflows only when execution and verification traces can be inspected.
Model-weight interventions and weight-space research
- Question / method
- Explored weight-space programming research and an experimental study of changing model behaviour without traditional prompt-only control.
- Observation
- An independent replay/control evaluation produced recorded outputs, but live frontier-model comparisons were incomplete due to missing credentials and environment constraints.
- Engineering lesson
- Treat scripted controls, checkpoint replay and actual model trials as different evidence classes; never report one as the other.
AI workflow and agent-readiness checks
- Question / method
- Examined site- and workflow-level readiness measures, including what agent crawlers can observe and whether checks apply to the page type being assessed.
- Observation
- A third-party scan encountered a Cloudflare 403 despite the API request itself succeeding. Some generic checks were not appropriate for every page type.
- Engineering lesson
- Separate unreachable-source failures from site defects, and evaluate homepages, product lists and product details against different criteria.
Design principles / Emerging patterns
What the experiments keep teaching us.
Give the model a small, grounded choice set and keep tool contracts explicit.
A confident final answer is not evidence that a browser, API or agent accomplished the task.
Accuracy, latency, context, memory, retries and hand-off overhead all count.
Argument validation, search matching, state transitions and safety checks are often better handled in code.
Small benchmarks are directional. Failed integrations and incomplete tests help define the next experiment.
Private AI must include governance, network boundaries, permissions and observability.
AIMEC / Applied AI engineering
Need a system that works beyond the demo?
We help assess the task, choose an appropriate architecture and build around real operating constraints. For ongoing research, explore AIMEC Labs and the technical articles.
