
No. 1 of the day
DeepSeek's open-source inference engine for V4.1-Flash

DeepInfer is DeepSeek's open-source inference framework for serving the DeepSeek-V4.1-Flash model at scale. It unifies optimized kernels like FlashMLA, DeepGEMM, and DeepEP into a single engine with expert parallelism, speculative decoding, and sliding-window cache recovery.
An engine, not a service.
The release includes deterministic modes and asynchronous scheduling, but omits KV cache, prefill/decode disaggregation, and load balancing—leaving those to the user. It supports NVIDIA SM100 and Huawei Ascend 950 hardware paths.
| Feature | Included |
|---|---|
| Optimized kernels | Yes |
| Expert parallelism | Yes |
| KV cache system | No |
Product Keywords
Loading comments...
Maker
DeepInfer is DeepSeek's open-source inference framework for serving the DeepSeek-V4.1-Flash model at scale. It assembles previously separate optimized kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels — into a single engine, released under Apache-2.0. Rather than a hosted service, it is a production-derived engine built around expert parallelism, speculative decoding, and sliding-window cache recovery, with hardware paths for NVIDIA SM100 and Huawei Ascend 950.
FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels are wired into one engine. The individual pieces had been released before; this release combines them.
Attention runs with data parallelism while the MoE portion uses expert parallelism. In DeepSeek's production design, decode spreads experts across 64–128 GPUs and rebalances hot experts with redundant copies.
Draft tokens are proposed ahead, and a confidence head predicts which ones are worth verifying. DeepSeek reports +4–7% throughput at its production operating point.
Instead of rebuilding the full sliding-window cache across every layer, the engine replays only the recent tail. The result is approximate, but the recompute cost becomes bounded.
An engine, not a service.
That distinction is the most important thing to understand about DeepInfer. A complete inference service also needs a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas — and none of the three is included. The gap is a deliberate design boundary rather than an oversight: DeepSeek kept the parts most tightly coupled to its internal infrastructure, which changes both who can use the code and what they should expect from it. Batch-invariant inference and deterministic sampling are available on NVIDIA, making benchmarks and agent runs auditable.
You want a close look at how DeepSeek serves a frontier open-weight model — wide expert parallelism at decode, confidence-scheduled speculation, and a cache policy that trades exactness for bounded recompute — and you are prepared to supply the KV cache, disaggregation, and load-balancing layers yourself. The ideas are useful even if you never run the code.
Other tools you might consider
Meet Mellum, a family of fast language models, including a next-generation model for ultra-low-latency and high-performance inference.
BeatAPI is a unified API platform for AI agents and the applications developers build. It brings together 70+ AI models, 1,000+ social data APIs, and web tools through one API key, one capability catalog, and one prepaid USD balance, reducing the need to manage separate provider accounts, integrations, and bills. MODELS, DATA, AND TOOLS IN ONE PLACE Access text, image, and video models alongside public social data from platforms such as TikTok, Instagram, YouTube, X, Reddit, and LinkedIn. Retrieve posts, profiles, comments, search results, and trends as structured data. Web tools support search, page reading, site maps, and research, while creative workflows cover use cases such as music videos and ecommerce videos. BUILT FOR AGENTS AND DEVELOPERS BeatAPI works with Claude Code, Codex, Cursor, OpenCode, Muse, and custom agents or applications that support MCP or HTTP requests. Its Search, Inspect, Run workflow helps agents discover a capability, review its input schema, outputs, pricing, and execution mode, and then execute it through the supported Run interface or its documented API endpoint. Developers can also integrate directly through REST and supported model-compatible APIs. CLEAR PRICING AND TRACEABLE EXECUTION Pricing is available before a call, with model rates listed alongside public provider prices and savings of up to 78% on selected models. Image, video, and workflow tasks expose task IDs, status, usage, outputs, and failure details. Completed media tasks return hosted file URLs, making it easier to deliver results without building a separate storage pipeline. FROM PROTOTYPE TO PRODUCT BeatAPI is designed for independent developers, startups, product and engineering teams, SaaS platforms, and automation builders. Use it to add AI generation to an application, research public social content, support creator discovery, automate web research, or give an agent access to models and data without wiring every service separately. Start with one key, explore the catalog, test a model in the playground, and reuse the same integration as your product grows.
PgDog is an open source connection pooler, load balancer, and sharding proxy for PostgreSQL. It's Postgres-compliant, fast, secure and built in the open by a community of database engineers.
MCP apps are the new website: they're real, interactive apps that run right inside AI assistants like ChatGPT and Claude. The tooling to build them didn't exist, so we built it. Meet Skybridge, the React framework that handles the infrastructure and the dev loop so you can focus on your features: MCP server, view rendering, client compatibility, testing tunnel, and more. Code once, ship everywhere. Already +500K downloads and dozens of apps in the stores!
Loading comments...