When spending our days in front of a computer juggling countless web pages, documents, and code, many details slip our minds almost immediately. Ask an AI assistant offhand what you worked on yesterday or where a specific document was saved, and it usually has no idea. To give AI screen memory, a wave of products emerged attempting to record screens 24/7. Microsoft introduced full-screen snapshot feature Recall on Windows, only to switch it off by default after privacy backlash and restrict it to new PCs equipped with dedicated chips. On Mac, screen-recording memory tool Rewind shut down its screen recording feature altogether, with the team pivoting to build the Limitless audio pendant. The route of storing every single pixel has run into headwinds on both privacy and cost.
In contrast to heavyweight screen-recording approaches, a lightweight route captures purely text. Open-sourced under the MIT license on August 25, Ambient Context offers a minimalist example. The author noted on LinkedIn that over two days, he built a macOS menu-bar utility that reads text from the active foreground window every 5 seconds by default and appends it daily to a local Markdown file—taking zero screenshots and making zero network requests throughout. We read through the project’s roughly 2,000 lines of Rust and 188 lines of Swift core code, initially to verify its local privacy claims. After confirming that the implementation matched the claims completely, we noticed that the data format written to local files was rather peculiar. Below is an actual entry written to a local file by the tool:
## 10:03–10:41 · Keynote · Q3 调研
file: /Users/grapeot/slides/Q3调研.pptx
（当时屏幕上可见的零散文字，稀疏，不保证全）Why is the path placed on the very first line? The header records the time range, application name, and window title. The first line of body text below the header is a file path starting with file:, followed by the captured scattered text. While reading window text, macOS Accessibility APIs can directly provide the file path or URL opened behind the window. As the author wrote in the code comments, a path is worth more than any amount of scraped text, placing references above scraped text.
What is missing when a model reads this record, and what does the path fill in? If you ask an AI coding assistant on Wednesday what you worked on Tuesday morning, the model reads this Markdown log and sees a record block at 10:00 titled “Q3 Research”. Looking only at the few lines of scattered text captured in the block, the model gets only fragmented snippets and cannot answer in-depth questions. But the file: path on the first line completes the picture: by following the path to directly open that local presentation, the model retrieves the verbatim, exact original full text. This entry resembles a library catalog card—the path is the call number, the captured text is a rough note jotted on the card, and the book itself always stays on the shelf. The pointers mentioned later refer to local or network addresses pointing to original files. This approach of separating pointers from summaries addresses the exact dilemma that screen-recording tools repeatedly run into.
The first layer of this dilemma comes from privacy and hardware barriers. In 2024, Microsoft launched Recall, which captured full-screen snapshots every few dozen seconds and saved them locally to support search and playback. The privacy backlash following its release forced Microsoft to pull it back and make revisions. When it relaunched, the feature was changed to opt-in by default; snapshots still remain on the local machine, but users are required to run it on new Copilot+ PCs equipped with dedicated chips. Although the core functionality was retained, users must first pay a trust tax in the form of new hardware.
The second layer goes to the other extreme: lightweight recording leaves an informational void that cannot support retrieval or execution. Time-tracking tools like ActivityWatch and RescueTime record only application names, window titles, and dwell time, without touching a single word of on-screen content. While this approach sidesteps privacy risks, it leaves behind nothing more than an activity log. A user can see that Keynote was opened at 10:00 AM on Tuesday, but has no way of knowing what specific proposal was being worked on.
The third layer occurs at the more subtle consumption stage, where a disconnect arises between knowledge production and actual consumption. In my article from last week, I documented a developer’s experiment: he built an agent to automatically capture knowledge for AI programming, extracting technical details from daily conversations into an 18-page wiki and a 1,418-line index. Yet when his primary agent started working, it never proactively queried this accumulated knowledge, still requiring humans to provide context every time. Knowledge was collected in full, but the loop broke at the consumption stage, rendering storage itself meaningless.
Faced with these three dilemmas, Ambient Context’s design offers corresponding answers. To address the Recall-style privacy burden, it abandons pixel capture and reads only structured text from system APIs. Privacy protection is pushed back to the point of origin rather than relying on retroactive redaction; the system API simply skips password fields during the read stage. To address the content void of pure event logs, it captures window text; by rough estimation, same-day deduplication keeps a full day’s volume down to tens of thousands of tokens, which can fit directly into a model’s context window. To address the consumption disconnect exemplified by the 18-page wiki, it gives up on pre-distilling knowledge, treating the log directly as an index and including a reading guide for AI, AGENTS.md, in the data directory to steer models to first scan the timeline and then follow paths to open the original files.
According to the author, this open-source project represents only two days of development work. Its self-reported deduplication and denoising performance has not yet been verified by third-party data, and there is no systematic survey of how well various applications support Accessibility APIs. For instance, GPU-rendered terminals cannot have their text read, and identifying incognito windows via English keywords may fail on Chinese systems. We view this codebase as a design signal rather than a mature, off-the-shelf product. Placing this design signal back into the broader market: where does it actually stand, and how does it compare to the existing players?
To answer this positioning question, we first need to establish two evaluation criteria. The first criterion is granularity: pixels are closest to the original screen, followed by video, audio, and OCR/extracted text, with the coarsest being event streams that only record applications and titles—this reflects fidelity. The second criterion is the triggering moment: some tools run continuously 24/7, some capture only the active window, meeting software records during meetings, and AI coding tools record conversations only at the point of user queries—this reflects gating. With these two criteria established, major products on the market fall into place:
Pioneers of storing full pixels have generally faced contraction in the market—Rewind abandoned screen recording, and Recall requires specific chip hardware to enable. The text-recording route, on the other hand, shows remarkably high development efficiency; as the author described, a single person built a usable prototype in just two days. This divergence stems from the fact that the primary reader of memory systems has become the language model. Models consume text by tokens; if pixels must be handled in between, they must first pass through visual encoding or OCR transformation. Given that models consume text by tokens, will having models directly watch screen recordings in the future actually work, and does the math really add up?
Let’s crunch the specific numbers before answering this question. Multimodal models do not process images through standard text tokenization pipelines; before each frame enters the model, it must pass through a vision encoder to be converted into vision tokens. Text-dense screens suffer the most under this pricing model: vision encoders bill by frame count, so a frame with just ten words costs exactly the same as one with a thousand words.
The Gemini 3 official pricing provides concrete benchmarks. Video incurs 70 to 280 tokens per frame depending on the resolution tier; high-resolution frames exceeding specific dimensions are sliced into 768-by-768 tiles, with each tile consuming 258 tokens. To clearly read text on a 2560-by-1440 display, it roughly needs to be sliced into 12 tiles, consuming nearly 3,000 tokens per frame. At a sampling rate of 1 frame per second under mid-resolution pricing, running continuously for one hour consumes 250,000 tokens, and maintaining verbatim recognition throughout an entire day reaches into the millions. Large multimodal models typically achieve a document text transcription accuracy between 95% and 97%, with even more errors on small fonts and complex tables. By contrast, recording the same day’s screen activity purely as text is estimated at tens of thousands of tokens after local deduplication—and it preserves the raw strings provided by the operating system, ensuring character-level accuracy even for complex URLs.
Different types of content define the boundaries of where each approach applies. If the content is inherently textual and requires verbatim accuracy at consumption time—such as code editing, document writing, and web browsing—reading text directly offers advantages in both cost and accuracy. If the content inherently relies on visual imagery—such as layout design, data charts, illustration layouts, and video conference feeds—text extraction loses critical information the moment it is captured; in these cases, pixel snapshots must be preserved, deferring parsing to be triggered on-demand at query time.
Behind both paths lies an invariant design core: when perception takes place. The pixel approach preserves raw signals prior to perception, requiring the perception cost to be repaid on every single query. The text approach shifts perception forward to the capture stage, leveraging Accessibility APIs to directly receive semantic results already processed by the system, at the expense of giving up visual screen details. Compression ratios in video codecs cannot solve the model-side consumption problem: while H.265 can shrink the file size of static screens significantly using inter-frame prediction, video decoders are designed for human eyes, whereas a model’s context window requires parsing results at the semantic level.
Real-world engineering decisions corroborate this: video-preserving tools like Rewind and Windrecorder both rely on OCR text to build search indexes, and Recall’s local search similarly extracts text via OCR first. Currently, no practical system feeds raw, continuous pixels directly into a model. Since perception can either be shifted forward to capture or deferred to query time, what rule should engineering follow to allocate across different data types?
In answering this allocation question, prior experience points to a clear rule: store pointers for navigable targets, store raw bytes for non-navigable targets, and use low-fidelity text to determine where to navigate. If the original file still resides on a local disk or remote server, recording the access path is sufficient. If it is ephemeral oral communication, a video conference, or a paper draft that cannot be recaptured after the fact, storage space must be dedicated to keeping the raw bytes. The overall system thus converges into a three-tier architecture: high-fidelity preservation of original artifacts, low-fidelity construction of indexes, and on-demand parsing at the query stage.
This line of thinking aligns directly with the three-tier caching mechanism I laid out in my March piece on context management. When preparing context for AI, AGENTS.md acts as L1 cache, the skill index serves as L2 cache guiding the model to find needed resources, and specific skill files are L3 content loaded on demand. The L2 index is essentially pointer mechanics applied on the consumption side: rather than dumping all content into the context at once, only addresses are passed in and read upon actual invocation. Now, the same logic extends to the data capture end: not only is context loading on-demand, but source recording is also captured on-demand, linked together via pointers.
From this, we can draw a conclusion: the lower the parsing cost of multimodal models becomes, the more economical the pointer approach is. As parsing capabilities grow stronger and invocation costs drop, re-fetching original files by following addresses becomes more efficient, and the scope of content requiring routine byte-level storage will narrow further—primarily covering special, visually native scenarios. The approach that will truly lose its viability is indiscriminately dumping continuous streams of pixels straight into a model’s context window.
If you are building a memory module for your AI assistant, there are two audits you can start right away. First, check whether your existing records still require another round of perceptual parsing during retrieval—such as hoarding massive piles of screenshots without generating corresponding text indexes. Second, spot-check the pointers in your records to verify whether the original files they point to still exist. A browser history entry may remain intact while the web page behind it has long since become inaccessible. Pointers can break, and that is precisely where low-fidelity text proves its value as a fallback, as well as the core reason for treating original artifacts as first-class citizens and maintaining them properly.