字数 1575,阅读大约需 8 分钟
前两周我在给一台旧机器装本地模型,3060 12G,想着跑个 7B 够了吧。
下了 Qwen 2.5 7B,行。又试了 Llama 3 8B,也行。膨胀了,下个 Qwen 2.5 14B,GGUF Q4_K_M,跑起来一秒三个字。也不是不能忍,但那个感觉就像你买了张 4090 回来发现打游戏还得开低画质。
我今年至少清了三次硬盘,删的全是那些「别人说好但我跑不动」的模型。每次流程都一样:在某处看到一个模型排名高,去 HuggingFace 找 GGUF,挑个看着顺眼的量化版本,下载,跑,要么龟速要么 OOM,删。
后来在 HelloGitHub 上看到有个叫 whichllm 的项目被收录了,点进去看了一眼 README,感觉挺合适我。使用之后,发现不得不替他点个赞了!

这个工具做的事说起来不复杂。你跑一下 whichllm,它扫一遍你的 GPU、CPU、内存,然后去 HuggingFace 拉模型列表,按照评分从高到低排好,告诉你哪些模型在你的机器上能跑、跑多快。
作者是 GitHub 上的 Andyyyy64,一个德国人,真名 Jannis,产品经理。工具本身 MIT 协议,Python 写的,代码结构挺清楚,不是什么屎山。
但光说「帮你选模型」太轻了。市面上干这个的工具不少,whichllm 让我觉得做的真不错的原因其实是它做的两个判断,跟那些「显存计算器」完全不在一个维度上。
第一个,它不看模型大小,看模型好不好。
举个例子。我以前用的那些工具逻辑很简单:你有 24G 显存,Q4 量化,最大能塞 32B 左右,那就给你推 32B。至于这个 32B 是不是好模型,它不管。
whichllm 不这么干。它拉取 LiveBench、Chatbot Arena ELO、Artificial Analysis、Aider 这些评测数据,综合打分。所以你经常会看到它在 RTX 4090 上推 Qwen3.6-27B Q5_K_M 而不是 Qwen3-32B Q4_K_M。因为新版 27B 跑分比老版 32B 高,虽然参数量小了 5B,但架构更新、训练更好。花差不多的显存和时间,拿到更好的回答。
第二个,它对评分来源做了分层,知道哪些分能信,哪些不能信。
这东西把每条评测数据打了来源标签:模型 ID 精确匹配的给全权重;同系列变体折算的打个折;从基础模型继承的再打折;上传者自己吹的权重最低。你在结果里看到分数前面有个 ~ 或者 !sr,就知道这个分可能有点水分。我跑了几次,发现很多看起来分很高的模型前面都挂着 ~,点进去一看果然是家族插值算出来的。
还有个我印象深的细节:它查跨家族分数继承。有人在 HuggingFace 上 fork 一个 70B 大模型,搞了个小变体,宣称继承了原版的评测分数。whichllm 对比参数量,发现差了 2 倍以上,直接拒掉这条数据。
这些事加在一起,能让 whichllm 跟那些只管显存不管质量的工具拉开差距。你不用再自己去翻排行榜、对型号、猜量化版本了,它帮你把脏活干了。
whichllm 用途挺多:
买卡之前的摸底。whichllm --gpu "RTX 5090" 模拟一下目标显卡上能跑什么。我之前纠结要不要从 4090 升 5090,跑了一下发现 5090 的 32G 显存能让 Qwen3.6-27B 用上 Q6_K 量化而不是 Q5_K_M,评分提了一档。后来实际对比了两个量化的回答质量差异,Q6_K 确实在一些逻辑推理题上明显更稳。值不值一万多块另说,至少我知道我花钱买的是什么提升。
选模型前的快速验证。 不是每个模型都值得花半小时下载。whichllm plan "llama 3 70b" 告诉我跑这东西要什么硬件。我之前想试 DeepSeek V3,计划命令跑完发现我的机器根本塞不下,省了一次白下载。
写代码集成的时候。whichllm snippet "qwen 7b" 直接输出一段 Python,用 llama-cpp-python 加载模型,复制粘贴就能跑。比自己翻 llama-cpp-python 的文档快太多了。
它还有一个 run 子命令,自动帮你隔离环境、装依赖、下载模型然后进对话。我很少用,因为我习惯在 Ollama 里管理模型。但这个功能对不想配环境的人很友好,相当于一个零配置的本地模型启动器。
说点我不爽的地方
断网就废了一半。数据走 HuggingFace API 实时拉,虽然本地有缓存(模型列表 6 小时过期,评测数据 24 小时过期),但没网的时候你就只能看缓存,新模型根本不知道。
默认输出是 Rich 做的表格,在终端里很漂亮,但想喂给 jq 必须加 --json。不加的话 stdout 就是纯展示,你没法在脚本里用。
还有个更根本的问题:它告诉你哪个模型在榜单上分高,不等于这个模型在你的具体任务上表现好。你写代码用的和写小说用的肯定不是同一个模型。虽然它支持 --profile coding 过滤,但粒度还是比较粗。希望之后 whichllm 能学类似 Arena 那样的做个分类的测评评分,编程、绘图、文本什么的排名分开的话就更好了!
装的话,Python 3.11 以上,一行就行:
uvx whichllm@latest
不需要预先安装。第一次跑 uvx 会自动拉下来,跑完即走。想常驻就 uv tool install whichllm 或者 pip install whichllm,Mac 上也能 brew。

这个项目五月在 GitHub 上热度涨得很快,属于势头正猛,现在 2.2k star。社区目前挺活跃的,Issue 里有人在报不同硬件的检测结果,作者回得也勤。
本地跑大模型这个事,真正费时间的不是下载,是你不知道哪个模型能在你的机器上好好干活。whichllm 不能替你选最好的模型,但它能让你别选到最烂的那个,省下来的时间刷点什么不好。
GitHub 地址:
https://github.com/Andyyyy64/whichllm
这个公众号曾分享过许多有趣的开源项目。如果你不想逐篇翻阅历史文章,也可以直接关注微信公众号“极客之家”,通过后台留言与我们互动交流

Character count: 1456, estimated reading time: ~7 minutes.
A couple of weeks ago, I was setting up local models on an old machine with a 3060 12G, thinking that 7B would be sufficient.
I downloaded Qwen 2.5 7B — worked. Then tried Llama 3 8B — also worked. Got carried away and downloaded Qwen 2.5 14B in GGUF Q4_K_M format. It ran, but only about three words per second — barely tolerable. It felt like buying a 4090 and having to play games on low settings.
Over the past year, I’ve cleared my hard drive at least three times, deleting models that “everyone said were good but my machine couldn’t handle.” The cycle was always the same: see a high-ranked model somewhere, head to HuggingFace for GGUF, pick what looks like a reasonable quantization, download, run, experience either snail-paced inference or OOM, then delete.
Recently, I came across whichllm listed in HelloGitHub and decided to check out the README. It seemed like just what I needed. After using it, I definitely think it deserves a shout-out!
What this tool does isn’t complicated — but it does it well. You run whichllm, it scans your GPU, CPU, and memory, then fetches the model list from HuggingFace, ranks them by score (from high-performance benchmarks), and tells you which ones will actually work on your setup and how fast.
The author is Andyyyy64 (real name Jannis) from Germany, a product manager. The project is MIT-licensed, written in Python, with clean code — no “spaghetti mess” here.
But “helping you pick a model” doesn’t fully capture the value. There are plenty of tools that do something similar. What sets whichllm apart is its approach to two key judgments — completely different from those simple “VRAM calculators.”
First: It cares about model quality, not just size.
Older tools operated on a simple logic: if you have 24G VRAM, Q4 quantization means you can fit up to ~32B. So they’d recommend the biggest 32B model available. They didn’t care if that 32B was actually a good model.
whichllm does. It pulls data from benchmarks like LiveBench, Chatbot Arena ELO, Artificial Analysis, and Aider, and aggregates scores. That’s why you might see it recommending Qwen3.6-27B Q5_K_M over Qwen3-32B Q4_K_M on a 4090 — because the newer 27B version scored higher despite having fewer parameters, thanks to improved architecture and training. You get better answers for roughly the same VRAM and time cost.
Second: It knows which scores to trust — and which to take with a grain of salt.
The tool assigns confidence levels to each rating source: exact model matches get full weight, while inherited scores from family variants get discounted, and self-reported claims are weighted minimally. When you see a score with ~ or !sr next to it, that’s a warning sign — often indicating interpolation from similar models rather than direct measurement. I’ve found that many seemingly high-scoring models turn out to be artificially boosted this way.
There’s also a clever check for cross-family inheritance abuse — if someone forks a 70B model into a 7B version and claims the same score, whichllm flags it as unreliable.
These features together make whichllm stand out from mindless VRAM calculators. It saves you the time of manually cross-referencing leaderboards, guessing at quantization effects, or downloading models only to find they’re too slow or too inaccurate.
Use cases abound:
Before buying a card: Run whichllm --gpu "RTX 5090" to see what you’ll be able to run with the next-gen hardware. I did this before considering an upgrade from 4090 to 5090, and saw that the extra 8G VRAM would allow Qwen3.6-27B to run at Q6_K instead of Q5_K_M, significantly improving accuracy on logic-heavy tasks.
Pre-download validation: No need to waste half an hour downloading unless you’re sure it works. whichllm plan "llama 3 70b" tells you whether your system is up to the task. I planned to try DeepSeek V3 but found out my setup couldn’t handle it — saved me the download.
Integration snippet: whichllm snippet "qwen 7b" outputs a ready-to-use Python script loading the model via llama-cpp-python. Copy, paste, and you’re off — much faster than hunting through API docs.
It even has a run command that isolates environments, installs dependencies, downloads, and starts a conversation — though I prefer using Ollama for model management; others might find this turnkey approach helpful.
Caveats:
Offline operation is limited. Data comes from HuggingFace API, cached locally (model lists expire after 6 hours, scores after 24), but without internet, you’re stuck with outdated info.
Output formatting: By default, it uses Rich for pretty terminal tables — great for human eyes, but not compatible with jq unless you add --json.
The biggest issue? It tells you which models are *ranked* high, not which ones are *best for your specific problem*. For coding vs. narrative writing, rankings may differ. Yes, it supports --profile coding, but the granularity isn’t perfect yet. I hope future versions adopt something like Arena’s domain-specific scoring (e.g., programming, art, text) to tailor recommendations better.
Installation couldn’t be easier: uvx whichllm runs it immediately without installation. For persistent use, try uv tool install whichllm or pip install whichllm. Mac users can brew install. Python 3.11+ is required.
The project is on a roll — over 2.2k GitHub stars as of now, active issue tracking, and a committed developer base.
Conclusion: Local model running isn’t just about storage or bandwidth — it’s about knowing what’s actually gonna work on your hardware. whichllm doesn’t guarantee perfection, but it prevents you from shooting yourself in the foot with bad choices. That alone makes it worth the space on your hard drive.
GitHub: https://github.com/Andyyyy64/whichllm
P.S. I run this blog through my Wechat account “极客之家” — if you enjoy reading about cool open-source stuff, send me a follow and chat anytime.