Working with an AI assistant often leads to a familiar loop: on day one, you remind it to exclude test accounts when analyzing EMEA data; by day three, it includes test accounts again, so you casually correct it; by day five, the exact same issue resurfaces; entering the second week, you find yourself having to repeat the exact same sentence for the third time.
When encountering this situation, it is easy to assume that the model’s comprehension is limited, or to suspect that your prompt was not clear enough. But after observing this a few times, you will realize that an AI failing to remember your instructions is often unrelated to model intelligence or phrasing—the key is that the correction landed in the wrong location from the very start.
When facing the question of how to get an AI to remember experience, most people’s intuition lands on two extremes. On one end is the prompt: writing all rules and requirements into the input so the model follows them step by step. On the other end is fine-tuning: using data directly to modify the model’s weights and lock in specific behaviors.
These two approaches frequently clash in everyday discussions. Those who prefer prompts feel that current models are already powerful enough; as long as the conversation context is sufficient, there is no need to spend effort fine-tuning. Advocates for fine-tuning argue that repeating rules in every new conversation consumes too many resources, and that only solidifying behaviors into weights can solve the problem once and for all.
This opposition has a clear historical context. In the BERT era, the standard workflow for using a model was fine-tuning: for every new task, a GPU was scheduled to retrain it. When GPT-3 appeared in 2020, the paradigm shifted: describe the task clearly in natural language, add a few examples, and the model produces results directly. As I discussed in an earlier article on foundation models, the core reason foundation models are called “foundation” is that they can perform few-shot learning without modifying weights. At the time, I also observed that models like CLIP were difficult to make effective through fine-tuning; focusing on optimizing the input without altering weights was instead the better choice.
Interestingly, a few months later, I wrote an article on fine-tuning, documenting how fine-tuning allowed a model to learn my personal writing tone with very satisfying results. The same author, two sequentially published observations: one claiming foundation models do not need fine-tuning, and the other emphasizing that fine-tuning solves pain points prompts cannot handle.
These two experiences actually reflect realities in different scenarios. This leads to a boundary: what kind of experience works well in a prompt, and what kind of experience must be written into the model? The answer often lies in the blank space between these two extremes.
Breaking down how AI remembers an experience based on where state is stored, the target locations actually extend beyond the two endpoints of inside versus outside the model, and can be clearly divided into four layers. The shallowest layer exists in the input context of the current conversation. You state your requirements, and the AI follows them during that generation. This method takes effect fastest and has the shortest lifecycle; once the chat window closes, the memory resets to zero.
One layer down is external memory, such as vector databases or retrieval systems. Experience is organized into records and stored in a repository, retrieved on demand in future conversations. Its workflow is somewhat like Google: assuming information already rests somewhere, when a question is asked, similarity matching retrieves it. It breaks the lifespan limitation of a single conversation, but also introduces a series of new uncertainties: records might fail to write successfully, retrieval might miss relevant entries, recalled content might conflict with older data, or the records themselves may have long expired.
Another layer down consists of documentation, configuration files, and rule sets versioned alongside the project. Writing lessons learned or fixed workflows into separate files, store them in a directory readable by the AI. Before executing tasks each time, have the AI read this document first. It is more like early Yahoo’s manual directory curation: rather than relying on luck during queries, knowledge is organized in advance. OpenAI’s own Data Agent follows a similar approach, using large models offline to compile data warehouse code into structured descriptions and retrieving these pre-processed materials online, rather than running real-time searches directly over raw data (see this analysis for details). Unlike similarity-matching retrieval mechanisms, this approach features a deterministic trigger mechanism—as long as the corresponding file exists at the specified path, the model can read it reliably. Meanwhile, it supports complete version control, allowing rollbacks whenever deviations occur.
The deepest layer is fine-tuning, which writes specific behaviors directly into model parameters. By training a set of lightweight auxiliary parameters (such as LoRA) alongside a frozen foundation model, specific response habits are locked in. Once training is complete, there is no need to repeatedly provide context in the prompt; the model will naturally exhibit the expected behavior.
In daily practice, the third layer is most easily overlooked. People tend to swing between the two extremes of ad-hoc instructions and retraining models, often missing the versionable rule files in the middle. In fact, in most scenarios, the third layer is the very first one that should be implemented.
Memories in the first layer cannot persist across sessions, the second layer relies on retrieval luck, and the fourth layer carries a heavy build cost. In comparison, the third layer provides deterministic triggering conditions, a clear audit trail, and low rollback costs—without consuming any training resources. In my article on Skills, I proposed an implementation mindset of externalizing before executing: the core idea is to crystallize workflows and lessons learned into documents, allowing the AI to follow instructions step-by-step each time it reads them. This is a direct reflection of landing on the third layer.
These four layers are not simple replacements for one another; each has explicit application boundaries. Only by understanding what each layer excels at resolving—and where each breaks down—can we find the right landing spot for different experiences.
The first layer is best suited for temporary, one-off operational preferences. For example, if an analysis only needs to process data from the past three months, such a requirement takes effect immediately at minimal cost, fulfilling its purpose when the conversation ends. If you find yourself frequently repeating a particular sentence in your daily work, it signifies that this piece of experience needs to migrate to a deeper layer.
The second layer is suitable for storing dynamic information that requires verification, demands precision, and carries explicit sources. For instance, historical preference records of a client are neither suitable for hard-coding into model weights (since the information changes) nor maintainable solely through a single conversation. The strength of a knowledge retrieval system lies in flexibility, allowing arbitrary numbers of time-sensitive records to be appended at any time. However, the failure modes of this layer are often very subtle: frequently, you assume the system successfully fetched the context when it actually missed entirely. Although modern long-context windows give the illusion of infinite capacity, the volume of text that fits onto the screen is not equivalent to the memory capacity that actually takes effect within the model.
The third layer should be reserved for strict rules with clear triggers that demand long-term consistency. Examples include filtering out test accounts when analyzing EMEA data, avoiding em dashes when writing reports, or prioritizing Tavily when invoking search tools. Once such rules take shape, they should not rely on retrieval luck, nor should they require manual pasting every time. Organizing them into documents saved at designated system paths allows the model to reliably follow them whenever a task starts. More importantly, these rules remain fully controllable: the content is clearly visible, deviations can be corrected at any time, and modifications can easily be rolled back. When accumulating rules in practice, record only the errors the AI has actually made; do not predict potential pitfalls out of thin air, as excessive preset rules will only introduce noise.
The parametric writing of the fourth layer primarily addresses stable behavioral patterns that occur with high frequency and require fine-grained tuning. In practice, there are two typical application scenarios.
One scenario involves handling behavioral details that are difficult to define precisely using natural language. I documented the process of tuning writing tone in my previous fine-tuning practice. If constrained with a prompt—asking it not to be too formal—the generated content often becomes overly casual; asking for a plainer tone leaves it lacking nuance. This subtle balance falls into an intermediate zone between formal and casual that words can hardly pinpoint accurately. Fine-tuning bypasses this bottleneck of verbal expression: instead of struggling to define that balance in text, simply providing a large number of self-written authentic samples enables the model to automatically converge on the expected expression habits.
Another scenario is internalizing frequently used long prompts into weights. If every request requires appending hundreds of words of style guides, tool descriptions, and formatting constraints, single-inference latency remains high and computational costs continuously accumulate. By solidifying these fixed requirements into parameters via fine-tuning, only a very brief instruction is needed later for the model to directly output compliant results.
However, fine-tuning also has a clear boundary: it changes the model’s behavioral habits, but cannot impart entirely new factual knowledge. I once conducted an experiment fine-tuning an entire knowledge base into a model, and the results showed that it is difficult for a model to truly master brand-new factual knowledge through fine-tuning alone. When facing knowledge update demands, a retrieval system is clearly the more sensible choice. The role of fine-tuning is closer to awakening and activating the model’s existing capabilities; confusing the two is a major reason many people stumble with fine-tuning.
In reality, the fourth layer is not as simple as a single LoRA; how to combine multiple LoRAs has become a fairly mature research direction. One line of approach is multi-task combination: training multiple LoRAs, each specialized in a specific area, and loading or fusing them on demand during inference. For example, LoRAMoE uses a router to distribute inputs to different experts, mitigating partial conflicts on multi-task mixed data while preserving world knowledge in the base model as much as possible. Another line goes a step further, directly addressing continual learning over time series: tasks arrive sequentially, old data is inaccessible, and for each new task, a new set of LoRA experts is added while freezing old ones, with periodic backtesting across all old tasks. Representative work like D-MoLE freezes old experts, trains only new ones, and reports metrics on backward transfer for old tasks. These efforts indicate that the fourth layer is evolving from training a single plug-in parameter to managing the lifecycle of a set of plug-in parameters.
Parametric exploration continues to evolve as well. For instance, Macaron-V1 configures four LoRAs on top of a frozen foundation model—responsible for chat, agent, coding, and GenUI, respectively—selecting one of them during each round of inference. The project open-sources its model weights and an auditable code framework, though from its current state, it closer resembles an open architecture attempt geared toward continual learning, and has not yet demonstrated mature cross-generational learning capabilities. The fourth layer is transitioning from single plug-in parameters toward exploring multi-parameter group management, though this remains a developing roadmap rather than a fully realized technology.
Once the mechanisms of the four layers are understood, there is no need to fall into an all-or-nothing choice during daily use; one can directly follow an evolutionary path that progresses from light to heavy, upgrading on demand.
The vast majority of experiences should start by default at the first layer. Requirements that can be easily explained in conversation do not need to be rushed into documents; however, once you notice yourself repeating the same reminder more than three times in a short period, it is time to consider escalating it to the second or third layer. Information that requires dynamic updates and carries source attributions is well-suited for settling into the second layer’s retrieval repository; rigid workflows with clear requirements and explicit triggers should be organized into local rule files at the third layer. Only when a specific behavior occurs with persistent high frequency, is difficult to define accurately in text, or when long prompts impose a noticeable burden on response latency and costs, should fine-tuning attempts at the fourth layer be initiated.
This order of progression is based on risk and cost, unrelated directly to learning quality. Modifying a sentence in a conversation carries the lowest trial cost and risk; even if the effect is poor, it only impacts that single generation. Writing rule files slightly increases cost, but modifications and rollbacks remain highly controllable. Fine-tuning is the heaviest path, requiring data cleaning, training schedules, and regression testing—and once behavior shifts unexpectedly, identifying which training sample caused the side effect on a case-by-case basis is extremely difficult.
In practical production, choosing a context-first, parameter-later strategy is essentially about managing risk and release cadence. This also means that not all experience needs to ultimately be written into the parameter layer; retaining many rules within documents and retrieval systems is often far easier to maintain.
Regardless of which layer an experience ultimately lands on, there are five practical questions to check against during delivery and sign-off: Where exactly was the change written? How soon does it take effect? Can it persist across sessions? Can it be accurately retrieved in the future? How can it be rolled back if deviations occur?
Only by answering these five questions clearly is the closed loop of AI assimilating experience truly established. If the answers remain ambiguous, no matter how many memory entries are accumulated, how complex the skill documents written, or how many LoRA modules trained, it merely piles up more state in the system without genuinely digesting and absorbing that experience.