Putting a large language model into an iPhone raises two of the most common questions: what can an on-device LLM actually do, and even if it runs, how could it possibly be good enough? These two questions matter more than parameter count alone. The newly released Bonsai 27B does not prove that a phone can replace a cloud agent, but it offers a more concrete reference point for discussing both questions.
Phones concentrate a vast amount of personal privacy: photo albums, screenshots, chat histories, location, health data, bank notifications, and work files, often gathered on a single screen. Image input in particular carries both privacy concerns when sent to the cloud and the practical costs of large file sizes, network fluctuation, and round-trip waiting time. When a user snaps a receipt, captures a payment page, or opens a medical report, the ideal is for the device to process it locally on the spot, eliminating the need to upload data remotely and wait for a response.
This is where on-device models become useful. Rather than having a chatbot write poetry while offline on the subway, it’s better to keep the short, private, and immediately-needed comprehension tasks local: reading the screen, recognizing receipts, organizing photos, extracting key fields from local materials, or compressing sensitive content into a summary that the user can then decide whether to send. Data never leaves the phone, and the first interaction saves a network round-trip.
Looking further ahead, ambient agents open up new application space for phones. Google demoed this kind of exploration at I/O: a system that understands the user’s current screen and situational context, offers suggestions at the right moment, or completes actions on the user’s behalf with explicit authorization. It doesn’t interrupt frequently—more like a quiet collaborator. The logic of ambient agents runs along the same lines: for an agent to understand the screen, images, and real-time context, the phone is the closest point to all that raw data. On-device vision models draw a more reasonable boundary for privacy and latency in this direction, though they still haven’t solved the critical engineering problems of permission control, failure recovery, background scheduling, and accountability.
On-device model inference is constrained primarily in four areas.
The first is memory. Just because a model file fits into phone storage doesn’t mean it can comfortably fit into memory at runtime. The actual runtime memory footprint—the working set—includes not only the model itself but also the context KV cache, activations, runtime buffers, and the vision encoder. Even a model that appears to be “only 4 GB” on paper can still hit iOS process memory limits once a large image is read in or a conversation grows long.
The second is power consumption and heat. Every token generated requires repeatedly reading model weights; the larger the model, the sooner memory bandwidth and thermal limits become bottlenecks. Running a 20-second answer once isn’t hard. Generating tokens continuously for 30 minutes, maintaining stable speed while plugging and unplugging the charger, and resuming normally after switching back from the background—those are the real product thresholds.
The third is visual prefill. For multimodal tasks on a phone, the bulk of latency is concentrated in the initial image processing stage, not the later token-by-token generation. Images must first be decoded, resized, tiled, and then converted through the vision encoder into visual embeddings that the language model can read. A model that responds quickly in plain text chat doesn’t necessarily mean it can rapidly parse a dense table.
The fourth is reliability. Whether an agent task succeeds depends on the entire interaction chain, not merely on the model outputting a fluent sentence. It also needs to accurately read the screen, decide whether to invoke a tool, obtain user authorization, handle network or system interruptions, and preserve a recoverable task state along the way. What an on-device model addresses is only one step in this entire chain.
The prevailing approach has been to first reduce model size, then carefully optimize architecture and quantization strategy to save on per-inference overhead.
MiniCPM-V 4.6 is a textbook example. It combines a SigLIP2-400M vision encoder with a Qwen3.5-0.8B language model, for a total of about 1.3B parameters. Its strength lies in processing images, screenshots, documents, OCR, video, and UI interfaces on a phone, rather than general-purpose long-text reasoning. The model offers two tiers of visual token compression: 4x and 16x. The 16x setting runs faster, while 4x retains more detail. This also explains the intuitive experience many users report: the model is slowest when an image first enters for processing; once visual prefill finishes, the subsequent text generation feels strikingly fast. But when reading fine-grained OCR material, speed mode is no substitute for detail mode.
In data published by OpenBMB, MiniCPM-V 4.6’s scores under the 4x setting are: OCRBench v2 English 40.6, Chinese 47.3, OmniDocBench 84.6, DocVQA 89.4; it received an Intelligence Index score of 13 in Artificial Analysis’s launch test, ranking among the top open-weight models under 2B. Though these benchmarks can’t replace real-world testing against actual receipts or financial forms, the direction is clear: build the “eyes” that a phone needs first.
Google’s Gemma series is moving in the same direction. The latest Gemma 4 E2B/E4B targets mobile and edge scenarios, supporting text, image, audio, and video input; the E4B variant has roughly 4.5B effective parameters and around 8B total parameters. According to Artificial Analysis and Google’s official data, E4B scores 69.4 on MMLU Pro, 42.5 on AIME 2026, 52.0 on LiveCodeBench v6, and 42.2 on Tau2. Its strategy avoids forcibly and aggressively compressing an oversized dense model, instead choosing to preserve multimodal perception and tool-calling capability within a memory working set that a phone can accommodate.
Microsoft’s Phi series leans more towards plain text. Phi-4 Mini is a roughly 3.8B text model suited for lightweight local text reasoning; for image or voice tasks, you’d need to switch to Phi’s multimodal variant or another model. This also illustrates that “small and strong” cannot be directly equated with “a visual agent on a phone”: a model’s modality support, runtime optimization, and application integration are all indispensable.
MiniCPM5-1B offers yet another pairing approach. It doesn’t handle visual input but instead focuses on how well a 1B-class text-only model can perform local summarization, task planning, code writing, and lightweight tool calling. Viewing it alongside MiniCPM-V 4.6 reveals a division of labor that is more practical than simply comparing parameter counts: the vision model handles “seeing and extracting,” while the text model handles “understanding text, planning, and deciding the next action.”
Bonsai 27B chose a different path. Instead of retraining a 1B or 4B
model, it took Qwen3.6-27B as the base and compressed the language
network’s weights into binary {-1,+1} format, with every
128 weights sharing a single FP16 scale factor, yielding an effective
rate of approximately 1.125 bits per weight. After this treatment, the
27.3B-parameter language model has a native runtime working set of about
3.9 GB; with the optional vision tower of roughly 0.63 GB, the publicly
released MLX weight package totals approximately 5.13 GB.
The insight this design offers lies in breaking the conventional assumptions around on-device model selection, not in the “27B” label itself. Past inertia held that a model must first be shrunk in base parameter count to fit on a phone. Bonsai’s hypothesis is: as long as quantization is aggressive enough, and compute kernels can directly and efficiently handle packed binary weights, there is a chance to squeeze a large model’s inherent reasoning and coding ability into a phone’s memory window.
According to data published by PrismML, the 1-bit Bonsai achieves an average score of 76.11 across 15 thinking-mode benchmarks, retaining roughly 89.5% of the Qwen3.6-27B FP16 base model’s 85.07. Breaking it down by category, it scores 91.66 in math and 81.88 in code, but drops to 66.03 on tool calling and only 59.57 on vision—indicating that extreme compression is not without cost. On an iPhone 17 Pro Max, PrismML measured its TG128 decode speed at 11 tok/s, with continuous generation at approximately 10.8 tok/s; however, these figures come from the vendor’s own testing, and the demo notes the use of cached/prefilled image context, which should not be taken as cold-start OCR end-to-end speed for uncached images.
This is the core question that Bonsai truly poses. The two approaches target different use cases, and neither route wins across the board in practical selection.
For general text reasoning, math, coding, and well-bounded tool tasks, the advantage of “large model with extremely low bit-width” is quite pronounced. Bonsai, with its 27B base, retains knowledge and complex reasoning headroom that small models struggle to recover; the vendor’s own tests also show that its retention of math and code capability under 1-bit is markedly better than tool calling and image understanding. If the primary use is offline reading, summarization, drafting, or assisted programming, and the phone has enough memory to hold the model with short context, it genuinely touches a capability ceiling previously unseen on phones.
But when it comes to reading images, parsing documents, and operating phone interfaces, “small model with higher precision” isn’t necessarily behind. The bottleneck in visual tasks often lies in image encoding and preprocessing, not just the language model’s decode stage. MiniCPM-V 4.6’s approach uses a smaller language model, a specially optimized vision encoding pathway, and adjustable compression rates to bring the initial image-reading overhead within a practical range; Gemma’s edge variants add multimodal and audio capabilities on top. Faced with a concrete need like “quickly parse a high-resolution receipt and accurately extract fields,” a smaller but vision-optimized model may be more reliable, more power-efficient, and easier to deploy in engineering terms than an extremely low-bit large model.
Public benchmark data measures different dimensions of emphasis. Bonsai’s 76.11 is a 15-benchmark thinking-mode average measured by PrismML on H100 GPUs using EvalScope and vLLM, weighted towards pure text logic and code capability, but it doesn’t reflect thermal throttling on a phone or uncached visual latency; MiniCPM’s 13 is a composite Intelligence Index from Artificial Analysis that can’t directly gauge extraction accuracy for complex financial documents; Gemma’s figures come from its specific multimodal testing configuration. Even on the same OCRBench v2, Bonsai’s reported 58.65 and MiniCPM-V 4.6’s 40.6 (English) / 47.3 (Chinese) are not directly comparable due to differences in image preprocessing, prompts, visual token compression ratios, and scoring conditions. For product builders, these metrics reveal each model’s strengths under its respective setup, but text reasoning, visual recognition, and real on-device execution are three independent selection decisions—they cannot be hand-waved into a single score.
A more useful conclusion should be framed conditionally:
When the task is primarily text reasoning, code, and short-context synthesis, and the model can stably fit on the phone, prioritize evaluating the “large base with extremely low bit-width” route.
When the task is primarily images, screenshots, tables, receipts, and video, visual encoding efficiency, image token count, OCR detail retention, and first-round latency matter far more than language model parameter count; in this case, prioritize evaluating purpose-built on-device multimodal solutions like MiniCPM-V or Gemma.
To build a reliable multi-step agent, never look only at model benchmarks. The accuracy of tool calling, permission design, error recovery, thermal stability, and background runtime constraints are what determine whether it can actually complete tasks on the user’s behalf.
The emergence of Bonsai shows that a 27B-class model is no longer necessarily barred from phones by storage and memory thresholds. MiniCPM-V 4.6 shows that a model doesn’t need to be large to make on-device image comprehension smooth. These two paths don’t represent a simple substitution relationship; they represent two entirely different approaches to on-device system design.
What truly needs to be compared head-to-head next is not the tok/s speed on a spec sheet. It’s measuring, on the same iPhone, against the same set of real-world tasks—receipts, tables, screenshots, and screen interactions—cold-start latency, uncached image prefill delay, time-to-first-token, decode speed, field-level OCR accuracy, peak memory, per-token energy consumption, and thermal degradation after 30 minutes of continuous generation. Only with this set of real-world measurements can we give an evidence-based answer to the question “what is a large model on a phone good for”: whether an on-device model is better suited as a private local copilot, a pair of eyes that understand the screen, or an ambient agent that reliably executes operations once authorized.