2026-03-24
If you’re an engineer using Python to build AI applications, you likely know LiteLLM. It lets you call APIs from OpenAI, Anthropic, Azure, Bedrock, and others through a single interface, saving you the trouble of managing multiple SDKs. Many use it directly as a model gateway. Others might not install it explicitly but find it in their dependency tree through frameworks like DSPy or CrewAI.
On March 24, 2026, the official LiteLLM package on PyPI was briefly
hijacked. Attackers used stolen publishing credentials to upload two
fake versions (1.82.7 and 1.82.8) containing
credential-stealing code. These versions don’t have corresponding tags
or releases in the LiteLLM GitHub repository. The exposure window lasted
about two to three hours (from 10:39 to roughly 11:25 UTC) before PyPI
isolated the packages. The last clean version is
1.82.6.
First, the most important point: this was a compromise of the Python package distribution channel. It has nothing to do with the model layer. There’s no evidence that model providers like OpenAI, Anthropic, or Google were breached, nor that model weights or inference APIs were tampered with. The attack targeted the publishing permissions for the LiteLLM Python package on PyPI.
The primary reason this incident deserves extra attention from AI
engineers is that the 1.82.8 version escalated this from a
package-specific problem to a Python-environment-level compromise.
During installation, it places a .pth file in the
site-packages directory. Python has a mechanism that most
developers rarely encounter: every time the interpreter starts, it scans
.pth files in site-packages. If a line starts
with import, it executes immediately. This is by design in
the standard library’s site
module. This means if 1.82.8 is installed in a Python
environment, any Python process started in that environment will run the
malicious code. It doesn’t matter if you’re running Flask, Jupyter,
pytest, or an unrelated script. You don’t even need to import LiteLLM.
Simply having the version installed affects every Python program in that
environment. Therefore, if you installed this version, every credential
accessible from that environment should be treated as compromised.
On top of that, LiteLLM’s position in the AI ecosystem amplifies the damage further. A typical LiteLLM deployment, whether used as a library or a proxy, already holds API keys for multiple model providers. It might also have database connection strings and cloud infrastructure credentials. The credential density in these environments is much higher than in a typical Python project. If malicious code runs in such an environment, it doesn’t just leak one key. It can leak an entire set of credentials across different providers and services.
Crucially, you might have LiteLLM in your environment without ever having installed it yourself. DSPy uses LiteLLM as its primary channel for model calls, and CrewAI depends on it for certain paths. If you installed or updated these frameworks during the exposure window without locked dependency versions, pip’s dependency resolution might have pulled in the malicious version.
What should you do now? If you installed or updated anything in your
Python environment between 10:39 and 11:25 UTC on March 24, and it
involved LiteLLM (directly or indirectly), run
pip show litellm to check the version. If it’s 1.82.7 or
1.82.8, treat your credentials as compromised. If your version has
stayed at 1.82.6 or earlier and you didn’t update during the window,
this is mostly a lesson to learn. I’ll provide more detailed self-check
steps later.
Both fake versions included credential-stealing code, but they were triggered differently.
1.82.7 hid the malicious code in
litellm/proxy/proxy_server.py using double Base64 encoding.
It only runs if you use the LiteLLM proxy feature, such as starting it
via litellm --proxy or importing
litellm.proxy.proxy_server in your code.
1.82.8 used the .pth mechanism mentioned
earlier. This affects the entire Python environment and isn’t limited to
LiteLLM’s code paths.
The malicious code performs three main actions, according to analyses
by Endor
Labs and Wiz.
First, it harvests every reachable credential, including environment
variables, SSH keys, AWS/GCP/Azure credentials, Kubernetes configs,
Docker configs, database passwords, and .env files. Second,
it encrypts the collected data with AES-256, wraps the key with an
embedded RSA public key, and sends it to the attacker’s domain. Third,
if it detects a Kubernetes service account token, it tries to move
laterally. It reads secrets from all namespaces, deploys privileged pods
on nodes, and uses chroot to enter the host machine to
install a persistent backdoor (sysmon.py + a systemd
service) that polls a C2 server every 50 minutes.
Since the data is encrypted before being sent, affected users can’t determine if their credentials were stolen just by analyzing network traffic.
LiteLLM’s own code didn’t have a vulnerability. The attack started further upstream with a tool many AI engineers use: Trivy.
Trivy is an open source security scanner maintained by Aqua Security. Many teams integrate it into CI/CD pipelines to check container images and code dependencies for known vulnerabilities. Because of what it does, Trivy usually has high permissions in a pipeline. It needs to read repository code and pull dependency info, often giving it access to tokens and secrets in the pipeline context.
In early March 2026, an attacker named TeamPCP compromised Trivy’s GitHub Actions. Aqua Security had previously discovered a credential leak and tried to rotate the affected credentials. However, the rotation process was incomplete. The replacement operation left an available access path for the attackers or exposed new tokens during the process. The attackers used this window to control Trivy’s GitHub Actions.
Once the Trivy CI component was compromised, the attackers could read secrets in the pipelines of downstream projects using that component. LiteLLM’s CI/CD pipeline used Trivy for security scanning and stored PyPI publishing credentials. The attackers used these to gain publishing rights for LiteLLM on PyPI and upload the malicious versions. The same path was used to compromise Checkmarx KICS.
The discovery of the incident was somewhat accidental. The security
firm FutureSearch
noticed that an MCP plugin running in Cursor pulled in LiteLLM as a
transitive dependency. A bug in the malicious code’s .pth
launcher triggered exponential process replication, which consumed all
the machine’s memory. Without this bug, the malicious code would have
run silently.
The main factor is whether your Python environment might have installed or updated LiteLLM during the exposure window (2026-03-24 10:39 to roughly 11:25 UTC).
You should treat your credentials as leaked if any of the following apply:
pip install litellm or a similar command during
the window.If you confirm that you installed version 1.82.7 or 1.82.8, assume
all reachable credentials in that environment have leaked. Rotate all
API keys, SSH keys, cloud credentials, and database passwords. If you’re
running in Kubernetes, audit the kube-system namespace for
unusual privileged pods (using the alpine:latest image and
names starting with node-setup-). Check if
~/.config/sysmon/sysmon.py exists.
You can mostly just take this as a lesson if all the following are true:
uv.lock, poetry.lock, or
pip-compile output.Quick check steps: Run pip show litellm
to see the version number. Search for litellm_init.pth in
your site-packages directory. Check if
~/.config/sysmon/sysmon.py exists. These three steps should
tell you if you’re affected.
Several points from this incident are worth reviewing for daily practice.
Lock your dependencies. The AI field moves fast, and
frameworks update often. Many people use loose version constraints or
run pip install without specifying a version. This habit
has turned from a theoretical risk into a real cost. If you use
uv, uv.lock handles locking by default. If you
use Poetry or pip-tools, ensure CI/CD and Docker builds only install
from the lock file.
Visibility of transitive dependencies. You might not
have chosen LiteLLM yourself, but frameworks like DSPy or CrewAI brought
it into your environment. Dependency trees for AI agent frameworks are
often deeper than those for traditional web apps because they integrate
multiple model SDKs, tools, and vector database clients. Regularly audit
what’s actually in your environment using pip list or
uv tree, paying close attention to packages you didn’t
explicitly choose.
Centralized model gateways are high-value targets.
Services like LiteLLM Proxy naturally hold multiple sets of API keys and
infrastructure credentials. Using official container images is safer
than running pip install directly. Injecting API keys
through a secrets manager is more controlled than using environment
variables. It’s also worth tightening network egress rules for gateway
nodes.
Security tools in CI/CD need security management too. The root cause here was the compromise of Trivy’s GitHub Actions, which allowed attackers to steal LiteLLM’s publishing credentials. Security scanners often run with high permissions in pipelines but aren’t always managed with the same version control as application code. Third-party actions in GitHub Actions should be locked to a full commit SHA rather than a version tag. Tags can be overwritten with a force-push, which is exactly what happened in the Trivy compromise.
This incident isn’t a reason to panic, but it does require serious attention.
The key takeaway isn’t the novelty of the attack. Credential theft
and PyPI poisoning are known patterns. What matters here is that the
.pth mechanism in 1.82.8 escalated a
package-level poisoning into an environment-level compromise: once
installed, every Python process in that environment could execute the
malicious code, and every credential accessible from that environment
should be treated as leaked. LiteLLM’s position in the AI engineering
stack then amplifies the problem further, because the environments where
it lives tend to already hold a high concentration of valuable API keys
and infrastructure credentials.
For most AI engineers, this is a good time to check dependency management practices and credential isolation strategies. You don’t have to wait until you’re a victim.
Core information sources: FutureSearch original disclosure, GitHub Issue #24512, GitHub Issue #24518, Endor Labs analysis, Wiz analysis, Aqua Security official statement, Sysdig analysis (KICS).