AIMEC / Engineering notes / Build logs

What we tested.
What actually happened.

A running record of experiments with small language models, browser agents, local inference, orchestration and reliability. We include things that worked, failure modes and findings that remain provisional.

Research notes assembled from recorded experiments and working sessions through November 2026. Individual prototypes may be independent explorations, not AIMEC products or client deployments.

How to read this page

Evidence before claims.

A planning benchmark, a tool-call test and a completed real-world task measure different things. We report them separately. If a workflow failed, that failure belongs in the record.

18

Documented research threads and experiments, not 18 finished products

9B

Model size used for one internal semantic coverage comparison

10 → 11

Correct cases out of 12 on that limited benchmark

Measured example / Semantic coverage

Improving the surrounding system without increasing model size

Single-pass baseline10/12

83.3% correct

Iterative + deterministic coverage11/12

91.7% correct

One additional correct result (+8.3 percentage points) in a 12-case test. Unsupported-case handling was 100% in the tested iterative configuration. These numbers are not statistically robust proof of general superiority.

Track 01 / Browser agents

Turning browser tools into reliable actions.

A small model can often choose a plausible action. The harder question is whether it targets the right element, fills valid arguments, recognises a failed interaction and verifies the goal.

01 / BROWSER AGENTS

Browser tools with Playwright and MCP

Question / method
Tested browser navigation, page snapshots, click and extraction tools against real websites, including an automation test storefront.
Observation
Live navigation and selected QA steps worked. More complex multi-step runs exposed wrong target selection, repeated actions, premature completion and unverifiable claims.
Engineering lesson
Keep action traces, restrict high-risk tools and verify success against actual page state rather than trusting an agent’s final message.
02 / BROWSER AGENTS

Compact browser state and simplified tool schemas

Question / method
Experimented with replacing large browser snapshots and tool schemas with smaller page representations and a constrained action vocabulary.
Observation
This reduced the amount of information a small model needed to interpret, but did not by itself establish reliable end-to-end browser task completion.
Engineering lesson
Separate observation, decision, argument filling, execution and post-action verification. Treat each as an independently testable component.
03 / BROWSER AGENTS

Deterministic product search and evidence checking

Question / method
Investigated a search helper for product-page tasks after a browser agent claimed that a search result had a missing image without adequate verification.
Observation
The agent’s defect report required independent validation; the wording alone was not trustworthy evidence of a broken image.
Engineering lesson
Find concrete DOM evidence for product name, price and image, and distinguish a failed image load from an incorrect model interpretation.
04 / BROWSER AGENTS

Argument-fill and cart workflow failure

Question / method
Traced a task to add a product, verify quantity and price, remove it and confirm the cart was empty.
Observation
The captured run lasted approximately 49 seconds and ended with an argument-fill error. It did not complete the requested cart verification.
Engineering lesson
Apply typed argument contracts and deterministic repair before tool calls; a successful navigation is not the same as completing a user goal.

Track 02 / Small model architecture

More capability from the harness.

These tests separate classification, extraction, retrieval, execution and verification, so that every improvement has a measurable target.

05 / SMALL MODELS

Fixed DAG planning with small decision and extraction models

Question / method
Split task planning, workflow routing, argument extraction and execution across components including a Strands decision model and MiniCPM-based extraction.
Observation
In one fixed-DAG evaluation, the combined setup produced complete plans for 23 of 25 cases (92%). A separate holdout evaluation reached 11 of 15 (73.3%).
Engineering lesson
Planning success is not browser success. Execution needs grounded elements, valid arguments, state updates and evidence of completion.
06 / SMALL MODELS

Routing benchmarks: Strands versus Qwen

Question / method
Compared routing decisions and latency between the Strands decision component and a Qwen alternative under the same historical evaluation.
Observation
Recorded routing results were 105/125 (84%) versus 95/125 (76%). Corresponding mean decision latencies were approximately 1.98 seconds versus 3.97 seconds.
Engineering lesson
A specialised router may improve a narrow decision step, but that result cannot be generalised to complete autonomous tasks.
07 / SMALL MODELS

Iterative semantic coverage with a 9B model

Question / method
Compared single-pass retrieval against evidence accumulation and deterministic requirement coverage using the same small planning model.
Observation
The recorded 12-case benchmark improved from 10/12 to 11/12 correct cases. Unsupported-case handling reached 100% in the tested iterative configuration.
Engineering lesson
Make requirements explicit, check which have evidence, and recover missing pieces without escalating the model size first.
08 / SMALL MODELS

Goal interpretation and lightweight verification

Question / method
Explored small-model interpretation of a natural-language request into success criteria, then yes/no checks following each browser action.
Observation
A useful design direction emerged, but available records do not demonstrate general end-to-end reliability for this configuration.
Engineering lesson
Keep success criteria stable across the run and compare them with observed evidence rather than allowing the agent to rewrite its goal.

Track 03 / Local inference

Engineering within a hardware budget.

Running a model is only one part of operating an AI system. Memory, embedding availability, concurrency and recovery affect whether a deployment is usable.

09 / LOCAL INFERENCE

Memory and GPU constraints for multi-model agents

Question / method
Tested local decision and extraction components on consumer hardware with limited GPU memory.
Observation
A Strands configuration could not fit in roughly 4 GB of NVIDIA VRAM. In CPU testing, a process approached 7 GiB and was terminated under system memory pressure on a 14 GiB machine.
Engineering lesson
Model choice, concurrency, quantisation, and process isolation must be designed around real memory budgets.
10 / LOCAL INFERENCE

Qwen model sizes and local agent performance

Question / method
Investigated Qwen-family models across small parameter sizes for planning, extraction and tool use; more recent 4B experiments extend this line of work.
Observation
Results vary by model and task. A single validated cross-model performance or cost ranking has not yet been established.
Engineering lesson
Report model version, quantisation, device, prompt/context size and exact evaluation task alongside any headline score.
11 / LOCAL INFERENCE

Private inference and embedding services

Question / method
Worked with local inference infrastructure, including llama.cpp-based embeddings and a document-indexing workflow.
Observation
One indexing pipeline initially failed when its embedding service was absent. After the service was configured, health checks succeeded; missed indexing events still needed recovery.
Engineering lesson
A private AI deployment needs service monitoring, retry-safe ingestion and observability, not merely a running language model.

Track 04 / Orchestration and agent systems

From component success to end-to-end outcomes.

An agent can delegate, use tools and call a model successfully while still failing to deliver the requested result. These experiments expose the seams.

12 / AGENT SYSTEMS

Multi-agent delegation and return-path failure

Question / method
Evaluated a private agent-network workflow in which work was delegated to other nodes and returned for synthesis.
Observation
In an acceptance run, delegation and task processing completed, but the final locally synthesised response timed out after the test window.
Engineering lesson
Track each transition from delegation to synthesis; a completed worker task is not a completed user request.
13 / AGENT SYSTEMS

Agent tooling, approvals and security boundaries

Question / method
Tested tool discovery, safe tool access, approval checks and agent execution through MCP-connected systems.
Observation
A live browser step succeeded in one investigation, while an agent selected an unsafe browser execution tool and had to be paused for approval. Missing capabilities also prompted inappropriate workaround attempts.
Engineering lesson
Expose only needed tools, constrain permissions and distinguish a recoverable tool failure from authorisation to try another route.
14 / AGENT SYSTEMS

Docker-based AI capability integration

Question / method
Tested a local stack joining agent-computer, world-model, forecasting, platform and inference components.
Observation
Integration checks found both functioning components and failures, including container-to-host Ollama connectivity and incompatible contracts. Later repairs verified several exposed capabilities.
Engineering lesson
Integration readiness depends on network addressing, service contracts and resource limits, not the count of running containers.
15 / AGENT SYSTEMS

Persistent agents, memory and scheduled development

Question / method
Explored local agent harnesses with memory, schedulers, development tasks, approvals and inspectable status.
Observation
Some component tests and dashboard-reported milestones were recorded; runtime memory pressure also deferred further development. These do not establish unattended autonomy in production.
Engineering lesson
Preserve explicit approvals, reproducible traces, clear resource thresholds and a recoverable state between runs.

Track 05 / Ongoing explorations

Questions still open.

Some work has produced reproducible outcomes; other research remains at the prototype or evaluation-design stage. These entries state the difference.

16 / EXPLORATIONS

Desktop automation and computer-use harnesses

Question / method
Investigated Vercel Agent Desktop and other computer-use patterns for testing compact models beyond browser-only actions.
Observation
Early work focused on controlled desktop tasks and evaluation setup; the available evidence does not support a broad desktop reliability claim.
Engineering lesson
Extend from constrained fixtures to harder workflows only when execution and verification traces can be inspected.
17 / EXPLORATIONS

Model-weight interventions and weight-space research

Question / method
Explored weight-space programming research and an experimental study of changing model behaviour without traditional prompt-only control.
Observation
An independent replay/control evaluation produced recorded outputs, but live frontier-model comparisons were incomplete due to missing credentials and environment constraints.
Engineering lesson
Treat scripted controls, checkpoint replay and actual model trials as different evidence classes; never report one as the other.
18 / EXPLORATIONS

AI workflow and agent-readiness checks

Question / method
Examined site- and workflow-level readiness measures, including what agent crawlers can observe and whether checks apply to the page type being assessed.
Observation
A third-party scan encountered a Cloudflare 403 despite the API request itself succeeding. Some generic checks were not appropriate for every page type.
Engineering lesson
Separate unreachable-source failures from site defects, and evaluate homepages, product lists and product details against different criteria.

Design principles / Emerging patterns

What the experiments keep teaching us.

Constrain the task

Give the model a small, grounded choice set and keep tool contracts explicit.

Verify the outcome

A confident final answer is not evidence that a browser, API or agent accomplished the task.

Measure the whole system

Accuracy, latency, context, memory, retries and hand-off overhead all count.

Keep deterministic logic outside the model

Argument validation, search matching, state transitions and safety checks are often better handled in code.

Publish limitations

Small benchmarks are directional. Failed integrations and incomplete tests help define the next experiment.

Design for local control

Private AI must include governance, network boundaries, permissions and observability.

This page documents engineering investigations, not certified performance claims, client case studies or promises of production capability. Reproducible test cases and full traces should accompany any future headline benchmark.

AIMEC / Applied AI engineering

Need a system that works beyond the demo?

We help assess the task, choose an appropriate architecture and build around real operating constraints. For ongoing research, explore AIMEC Labs and the technical articles.

Scroll to Top