GitHub 每日热门开源仓库，过滤 AI/LLM/Agent/MCP 相关项目。追踪开发者社区关注的开源基础设施与框架变化，是技术选型的重要参考。
code-review-graph 是一个通过 Tree-sitter 构建代码 AST 结构图、利用 MCP 协议为 AI 编程助手提供精确上下文的开源工具，在基准测试中实现了每问题中位数约 82 倍的 token 缩减。
tirth8205 在 GitHub 上发布了 code-review-graph 项目，该项目使用 Tree-sitter 将代码库解析为包含函数、类、导入和调用关系节点的 AST 图结构，通过 MCP 协议为 Claude Code、Cursor、GitHub Copilot 等 AI 编程助手提供精确的代码审查上下文。工具在 fastapi、flask、express 等 6 个开源仓库的评估中实现了每问题中位数约 82 倍的 token 缩减，支持 30 多种编程语言，并提供了 GitHub Action 作为 CI 集成方式。
KTransformers 是清华大学 MADSys Lab 主导的开源项目，通过 CPU-GPU 异构计算实现大语言模型的高效推理与微调，支持 DeepSeek-V3/R1、Kimi-K2、GLM-5 等最新 MoE 模型，提供 kt-kernel 推理引擎和 LLaMA-Factory 微调集成两大能力，研究论文发表于 ACM SIGOPS SOSP 2025。
KTransformers 由清华大学 MADSys Lab、Approaching.AI 和 9#AISoft 联合开发，研究论文已被 ACM SIGOPS SOSP 2025 会议收录。项目通过 kt-kernel 推理引擎和 KTransformers × LLaMA-Factory 微调集成两条路径，实现 MoE 大模型在消费级硬件上的 CPU-GPU 异构运行。kt-kernel 利用 Intel AMX/AVX 指令集和 INT4/INT8 量化实现高效推理，微调路径支持 DeepSeek-V3 在 4×RTX 4090 上以 3.7 it/s 的速度训练。项目持续为 MiniMax-M3、GLM-5.2、DeepSeek-V4-Flash 等前沿模型提供 Day0 支持，并兼容 AMD ROCm、Intel Arc 和 Ascend NPU 等硬件后端。
Learn it. Build it. Ship it for others. From the creator of Agent Memory - #1 Persistent memory ⭐ which naturally works with any agents or chat assistants. ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ 84% of students already use AI tools. Only 18% feel prepared to use them professionally. This curriculum closes that gap. 503 lessons. 20 phases. ~320 hours. Python, TypeScript, Rust, Julia. Every lesson ships a reusable artifact: a prompt, a skill, an agent, an MCP server. Free, open source, MIT. You don't just learn AI. You build it. End-to-end. By hand. 150,639 readers · 241,669 page views in the last 30 days · as of 2026-06-07 How this works Most AI material teaches in scattered pieces. A paper here, a fine-tuning post there, a flashy agent demo somewhere else. The pieces rarely line up. You ship a chatbot but can't explain its loss curve. You hook a function to an agent but can't say what attention does inside the model that's calling it. This curriculum is the spine. 20 phases, 503 lessons, four languages: Python, TypeScript, Rust, Julia. Linear algebra at one end, autonomous swarms at the other. Every algorithm gets built from raw math first. Backprop. Tokenizer. Attention. Agent loop. By the time PyTorch shows up, you already know what it's doing under the hood. Each lesson runs the same loop: read the problem, derive the math, write the code, run the test, keep the artifact. No five-minute videos, no copy-paste deploys, no hand-holding. Free, open source, and built to run on your own laptop. ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ The shape of the curriculum Twenty phases stack on top of each other. Math is the floor. Agents and production are the roof. Skip ahead if you already know the lower layers, but don't skip and then wonder why something at the top is breaking. %%{init: {'theme':'base','themeVariables':{'primaryColor':'#fafaf5','primaryTextColor':'#1a1a1a','primaryBorderColor':'#3553ff','lineColor':'#3553ff','fontFamily':'JetBrains Mono','fontSize':'12px'}}}%% flowchart TB P0["Phase 0 — Setup & Tooling"] --> P1["Phase 1 — Math Foundations"] P1 --> P2["Phase 2 — ML Fundamentals"] P2 --> P3["Phase 3 — Deep Learning Core"] P3 --> P4["Phase 4 — Vision"] P3 --> P5["Phase 5 — NLP"] P3 --> P6["Phase 6 — Speech & Audio"] P3 --> P9["Phase 9 — RL"] P5 --> P7["Phase 7 — Transformers"] P7 --> P8["Phase 8 — GenAI"] P7 --> P10["Phase 10 — LLMs from Scratch"] P10 --> P11["Phase 11 — LLM Engineering"] P10 --> P12["Phase 12 — Multimodal"] P11 --> P13["Phase 13 — Tools & Protocols"] P13 --> P14["Phase 14 — Agent Engineering"] P14 --> P15["Phase 15 — Autonomous Systems"] P15 --> P16["Phase 16 — Multi-Agent & Swarms"] P14 --> P17["Phase 17 — Infrastructure & Production"] P15 --> P18["Phase 18 — Ethics & Alignment"] P16 --> P19["Phase 19 — Capstone Projects"] P17 --> P19 P18 --> P19 ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ The shape of a lesson Each lesson lives in its own folder, with the same structure across the entire curriculum: phases/<NN>-<phase-name>/<NN>-<lesson-name>/ ├── code/ runnable implementations (Python, TypeScript, Rust, Julia) ├── docs/ │ └── en.md lesson narrative └── outputs/ prompts, skills, agents, or MCP servers this lesson produces Every lesson follows six beats. The Build It / Use It split is the spine — you implement the algorithm from scratch first, then run the same thing through the production library. You understand what the framework is doing because you wrote the smaller version yourself. %%{init: {'theme':'base','themeVariables':{'primaryColor':'#fafaf5','primaryTextColor':'#1a1a1a','primaryBorderColor':'#3553ff','lineColor':'#3553ff','fontFamily':'JetBrains Mono','fontSize':'13px'}}}%% flowchart LR M["MOTTO<br/><sub>one-line core idea</sub>"] --> Pr["PROBLEM<br/><sub>concrete pain</sub>"] Pr --> C["CONCEPT<br/><sub>diagrams & intuition</sub>"] C --> B["BUILD IT<br/><sub>raw math, no frameworks</sub>"] B --> U["USE IT<br/><sub>same thing in PyTorch / sklearn</sub>"] U --> S["SHIP IT<br/><sub>prompt · skill · agent · MCP</sub>"] Getting started Three ways in. Pick one. Option A — read. Open any completed lesson on aiengineeringfromscratch.com or expand a phase under Contents. No setup, no cloning. Option B — clone and run. git clone https://github.com/rohitg00/ai-engineering-from-scratch.git cd ai-engineering-from-scratch python phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py Option C — find your level (recommended). Skip ahead intelligently. Inside Claude, Cursor, Codex, OpenClaw, Hermes, or any agent with the curriculum skills installed: /find-your-level Ten questions. Maps your knowledge to a starting phase, builds a personalized path with hour estimates. After each phase: /check-understanding 3 # quiz yourself on phase 3 ls phases/03-deep-learning-core/05-loss-functions/outputs/ # ├── prompt-loss-function-selector.md # └── prompt-loss-debugger.md Prerequisites You can write code (any language; Python helps). You want to understand how AI actually works, not just call APIs. Built-in agent skills (Claude, Cursor, Codex, OpenClaw, Hermes) Skill What it does /find-your-level Ten-question placement quiz. Maps your knowledge to a starting phase and produces a personalized path with hour estimates. /check-understanding <phase> Per-phase quiz, eight questions, with feedback and specific lessons to review. ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ Every lesson ships something Other curricula end with "congratulations, you learned X." Each lesson here ends with a reusable tool you can install or paste into your daily workflow. FIG_001 · APROMPTS FIG_001 · BSKILLS FIG_001 · CAGENTS FIG_001 · DMCP SERVERS Paste into any AI assistant for expert-level help on a narrow task. Drop into Claude, Cursor, Codex, OpenClaw, Hermes, or any agent that reads SKILL.md. Deploy as autonomous workers — you wrote the loop yourself in Phase 14. Plug into any MCP-compatible client. Built end-to-end in Phase 13. Install the lot with python3 scripts/install_skills.py. Real tools, not homework. By the end of the curriculum, you have a portfolio of 503 artifacts you actually understand because you built them. FIG_002 · A worked sample Phase 14, lesson 1: the agent loop. ~120 lines of pure Python, no dependencies. code/agent_loop.py build it def run(query, tools): history = [user(query)] for step in range(MAX_STEPS): msg = llm(history) if msg.tool_calls: for call in msg.tool_calls: result = tools[call.name](**call.args) history.append(tool_result(call.id, result)) continue return msg.content raise StepLimitExceeded outputs/skill-agent-loop.md ship it --- name: agent-loop description: ReAct-style loop for any tool list phase: 14 lesson: 01 --- Implement a minimal agent loop that... outputs/prompt-debug-agent.md You are an agent debugger. Given the trace of an agent run, identify the step where the agent went wrong and explain why... ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ Contents Twenty phases. Click any phase to expand its lesson list. Phase 0: Setup & Tooling 12 lessons Get your environment ready for everything that follows. # Lesson Type Lang 01 Dev Environment Build Python 02 Git & Collaboration Learn — 03 GPU Setup & Cloud Build Python 04 APIs & Keys Build Python 05 Jupyter Notebooks Build Python 06 Python Environments Build Shell 07 Docker for AI Build Docker 08 Editor Setup Build — 09 Data Management Build Python 10 Terminal & Shell Learn — 11 Linux for AI Learn — 12 Debugging & Profiling Build Python Phase 1 — Math Foundations 22 lessons The intuition behind every AI algorithm, through code. # Lesson Type Lang 01 Linear Algebra Intuition Learn Python, Julia 02 Vectors, Matrices & Operations Build Python, Julia 03 Matrix Transformations & Eigenvalues Build Python, Julia 04 Calculus for ML: Derivatives & Gradients Learn Python 05 Chain Rule & Automatic Differentiation Build Python 06 Probability & Distributions Learn Python 07 Bayes' Theorem & Statistical Thinking Build Python 08 Optimization: Gradient Descent Family Build Python 09 Information Theory: Entropy, KL Divergence Learn Python 10 Dimensionality Reduction: PCA, t-SNE, UMAP Build Python 11 Singular Value Decomposition Build Python, Julia 12 Tensor Operations Build Python 13 Numerical Stability Build Python 14 Norms & Distances Build Python 15 Statistics for ML Build Python 16 Sampling Methods Build Python 17 Linear Systems Build Python 18 Convex Optimization Build Python 19 Complex Numbers for AI Learn Python 20 The Fourier Transform Build Python 21 Graph Theory for ML Build Python 22 Stochastic Processes Learn Python Phase 2 — ML Fundamentals 18 lessons Classical ML — still the backbone of most production AI. # Lesson Type Lang 01 What Is Machine Learning Learn Python 02 Linear Regression from Scratch Build Python 03 Logistic Regression & Classification Build Python 04 Decision Trees & Random Forests Build Python 05 Support Vector Machines Build Python 06 KNN & Distance Metrics Build Python 07 Unsupervised Learning: K-Means, DBSCAN Build Python 08 Feature Engineering & Selection Build Python 09 Model Evaluation: Metrics, Cross-Validation Build Python 10 Bias, Variance & the Learning Curve Learn Python 11 Ensemble Methods: Boosting, Bagging, Stacking Build Python 12 Hyperparameter Tuning Build Python 13 ML Pipelines & Experiment Tracking Build Python 14 Naive Bayes Build Python 15 Time Series Fundamentals Build Python 16 Anomaly Detection Build Python 17 Handling Imbalanced Data Build Python 18 Feature Selection Build Python Phase 3 — Deep Learning Core 13 lessons Neural networks from first principles. No frameworks until you build one. # Lesson Type Lang 01 The Perceptron: Where It All Started Build Python 02 Multi-Layer Networks & Forward Pass Build Python 03 Backpropagation from Scratch Build Python 04 Activation Functions: ReLU, Sigmoid, GELU & Why Build Python 05 Loss Functions: MSE, Cross-Entropy, Contrastive Build Python 06 Optimizers: SGD, Momentum, Adam, AdamW Build Python 07 Regularization: Dropout, Weight Decay, BatchNorm Build Python 08 Weight Initialization & Training Stability Build Python 09 Learning Rate Schedules & Warmup Build Python 10 Build Your Own Mini Framework Build Python 11 Introduction to PyTorch Build Python 12 Introduction to JAX Build Python 13 Debugging Neural Networks Build Python Phase 4 — Computer Vision 28 lessons From pixels to understanding — image, video, 3D, VLMs, and world models. # Lesson Type Lang 01 Image Fundamentals: Pixels, Channels, Color Spaces Learn Python 02 Convolutions from Scratch Build Python 03 CNNs: LeNet to ResNet Build Python 04 Image Classification Build Python 05 Transfer Learning & Fine-Tuning Build Python 06 Object Detection — YOLO from Scratch Build Python 07 Semantic Segmentation — U-Net Build Python 08 Instance Segmentation — Mask R-CNN Build Python 09 Image Generation — GANs Build Python 10 Image Generation — Diffusion Models Build Python 11 Stable Diffusion — Architecture & Fine-Tuning Build Python 12 Video Understanding — Temporal Modeling Build Python 13 3D Vision: Point Clouds, NeRFs Build Python 14 Vision Transformers (ViT) Build Python 15 Real-Time Vision: Edge Deployment Build Python 16 Build a Complete Vision Pipeline Build Python 17 Self-Supervised Vision — SimCLR, DINO, MAE Build Python 18 Open-Vocabulary Vision — CLIP Build Python 19 OCR & Document Understanding Build Python 20 Image Retrieval & Metric Learning Build Python 21 Keypoint Detection & Pose Estimation Build Python 22 3D Gaussian Splatting from Scratch Build Python 23 Diffusion Transformers & Rectified Flow Build Python 24 SAM 3 & Open-Vocabulary Segmentation Build Python 25 Vision-Language Models (ViT-MLP-LLM) Build Python 26 Monocular Depth & Geometry Estimation Build Python 27 Multi-Object Tracking & Video Memory Build Python 28 World Models & Video Diffusion Build Python Phase 5 — NLP: Foundations to Advanced 29 lessons Language is the interface to intelligence. # Lesson Type Lang 01 Text Processing: Tokenization, Stemming, Lemmatization Build Python 02 Bag of Words, TF-IDF & Text Representation Build Python 03 Word Embeddings: Word2Vec from Scratch Build Python 04 GloVe, FastText & Subword Embeddings Build Python 05 Sentiment Analysis Build Python 06 Named Entity Recognition (NER) Build Python 07 POS Tagging & Syntactic Parsing Build Python 08 Text Classification — CNNs & RNNs for Text Build Python 09 Sequence-to-Sequence Models Build Python 10 Attention Mechanism — The Breakthrough Build Python 11 Machine Translation Build Python 12 Text Summarization Build Python 13 Question Answering Systems Build Python 14 Information Retrieval & Search Build Python 15 Topic Modeling: LDA, BERTopic Build Python 16 Text Generation Build Python 17 Chatbots: Rule-Based to Neural Build Python 18 Multilingual NLP Build Python 19 Subword Tokenization: BPE, WordPiece, Unigram, SentencePiece Learn Python 20 Structured Outputs & Constrained Decoding Build Python 21 NLI & Textual Entailment Learn Python 22 Embedding Models Deep Dive Learn Python 23 Chunking Strategies for RAG Build Python 24 Coreference Resolution Learn Python 25 Entity Linking & Disambiguation Build Python 26 Relation Extraction & Knowledge Graph Construction Build Python 27 LLM Evaluation: RAGAS, DeepEval, G-Eval Build Python 28 Long-Context Evaluation: NIAH, RULER, LongBench, MRCR Learn Python 29 Dialogue State Tracking Build Python Phase 6 — Speech & Audio 17 lessons Hear, understand, speak. # Lesson Type Lang 01 Audio Fundamentals: Waveforms, Sampling, FFT Learn Python 02 Spectrograms, Mel Scale & Audio Features Build Python 03 Audio Classification Build Python 04 Speech Recognition (ASR) Build Python 05 Whisper: Architecture & Fine-Tuning Build Python 06 Speaker Recognition & Verification Build Python 07 Text-to-Speech (TTS) Build Python 08 Voice Cloning & Voice Conversion Build Python 09 Music Generation Build Python 10 Audio-Language Models Build Python 11 Real-Time Audio Processing Build Python 12 Build a Voice Assistant Pipeline Build Python 13 Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC Learn Python 14 Voice Activity Detection & Turn-Taking Build Python 15 Streaming Speech-to-Speech — Moshi, Hibiki Learn Python 16 Voice Anti-Spoofing & Audio Watermarking Build Python 17 Audio Evaluation — WER, MOS, MMAU, Leaderboards Learn Python Phase 7 — Transformers Deep Dive 14 lessons The architecture that changed everything. # Lesson Type Lang 01 Why Transformers: The Problems with RNNs Learn Python 02 Self-Attention from Scratch Build Python 03 Multi-Head Attention Build Python 04 Positional Encoding: Sinusoidal, RoPE, ALiBi Build Python 05 The Full Transformer: Encoder + Decoder Build Python 06 BERT — Masked Language Modeling Build Python 07 GPT — Causal Language Modeling Build Python 08 T5, BART — Encoder-Decoder Models Learn Python 09 Vision Transformers (ViT) Build Python 10 Audio Transformers — Whisper Architecture Learn Python 11 Mixture of Experts (MoE) Build Python 12 KV Cache, Flash Attention & Inference Optimization Build Python 13 Scaling Laws Learn Python 14 Build a Transformer from Scratch Build Python 15 Attention Variants — Sliding Window, Sparse, Differential Build Python 16 Speculative Decoding — Draft, Verify, Repeat Build Python Phase 8 — Generative AI 14 lessons Create images, video, audio, 3D, and more. # Lesson Type Lang 01 Generative Models: Taxonomy & History Learn Python 02 Autoencoders & VAE Build Python 03 GANs: Generator vs Discriminator Build Python 04 Conditional GANs & Pix2Pix Build Python 05 StyleGAN Build Python 06 Diffusion Models — DDPM from Scratch Build Python 07 Latent Diffusion & Stable Diffusion Build Python 08 ControlNet, LoRA & Conditioning Build Python 09 Inpainting, Outpainting & Editing Build Python 10 Video Generation Build Python 11 Audio Generation Build Python 12 3D Generation Build Python 13 Flow Matching & Rectified Flows Build Python 14 Evaluation: FID, CLIP Score Build Python 19 Visual Autoregressive Modeling (VAR): Next-Scale Prediction Build Python Phase 9 — Reinforcement Learning 12 lessons The foundation of RLHF and game-playing AI. # Lesson Type Lang 01 MDPs, States, Actions & Rewards Learn Python 02 Dynamic Programming Build Python 03 Monte Carlo Methods Build Python 04 Q-Learning, SARSA Build Python 05 Deep Q-Networks (DQN) Build Python 06 Policy Gradients — REINFORCE Build Python 07 Actor-Critic — A2C, A3C Build Python 08 PPO Build Python 09 Reward Modeling & RLHF Build Python 10 Multi-Agent RL Build Python 11 Sim-to-Real Transfer Build Python 12 RL for Games Build Python Phase 10 — LLMs from Scratch 22 lessons Build, train, and understand large language models. # Lesson Type Lang 01 Tokenizers: BPE, WordPiece, SentencePiece Build Python, Rust 02 Building a Tokenizer from Scratch Build Python 03 Data Pipelines for Pre-Training Build Python 04 Pre-Training a Mini GPT (124M) Build Python 05 Distributed Training, FSDP, DeepSpeed Build Python 06 Instruction Tuning — SFT Build Python 07 RLHF — Reward Model + PPO Build Python 08 DPO — Direct Preference Optimization Build Python 09 Constitutional AI & Self-Improvement Build Python 10 Evaluation — Benchmarks, Evals Build Python 11 Quantization: INT8, GPTQ, AWQ, GGUF Build Python 12 Inference Optimization Build Python 13 Building a Complete LLM Pipeline Build Python 14 Open Models: Architecture Walkthroughs Learn Python 15 Speculative Decoding and EAGLE-3 Build Python 16 Differential Attention (V2) Build Python 17 Native Sparse Attention (DeepSeek NSA) Build Python 18 Multi-Token Prediction (MTP) Build Python 19 DualPipe Parallelism Learn Python 20 DeepSeek-V3 Architecture Walkthrough Learn Python 21 Jamba — Hybrid SSM-Transformer Learn Python 22 Async and Hogwild! Inference Build Python 25 Speculative Decoding and EAGLE Build Python 34 Gradient Checkpointing and Activation Recomputation Build Python Phase 11 — LLM Engineering 17 lessons Put LLMs to work in production. # Lesson Type Lang 01 Prompt Engineering: Techniques & Patterns Build Python 02 Few-Shot, CoT, Tree-of-Thought Build Python 03 Structured Outputs Build Python 04 Embeddings & Vector Representations Build Python 05 Context Engineering Build Python 06 RAG: Retrieval-Augmented Generation Build Python 07 Advanced RAG: Chunking, Reranking Build Python 08 Fine-Tuning with LoRA & QLoRA Build Python 09 Function Calling & Tool Use Build Python 10 Evaluation & Testing Build Python 11 Caching, Rate Limiting & Cost Build Python 12 Guardrails & Safety Build Python 13 Building a Production LLM App Build Python 14 Model Context Protocol (MCP) Build Python 15 Prompt Caching & Context Caching Build Python 16 LangGraph: State Machines for Agents Build Python 17 Agent Framework Tradeoffs Learn Python Phase 12 — Multimodal AI 25 lessons See, hear, read, and reason across modalities — from ViT patches to computer-use agents. # Lesson Type Lang 01 Vision Transformers and the Patch-Token Primitive Learn Python 02 CLIP and Contrastive Vision-Language Pretraining Build Python 03 BLIP-2 Q-Former as Modality Bridge Build Python 04 Flamingo and Gated Cross-Attention Learn Python 05 LLaVA and Visual Instruction Tuning Build Python 06 Any-Resolution Vision — Patch-n'-Pack and NaFlex Build Python 07 Open-Weight VLM Recipes: What Actually Matters Learn Python 08 LLaVA-OneVision: Single, Multi, Video Build Python 09 Qwen-VL Family and Dynamic-FPS Video Learn Python 10 InternVL3 Native Multimodal Pretraining Learn Python 11 Chameleon Early-Fusion Token-Only Build Python 12 Emu3 Next-Token Prediction for Generation Learn Python 13 Transfusion Autoregressive + Diffusion Build Python 14 Show-o Discrete-Diffusion Unified Learn Python 15 Janus-Pro Decoupled Encoders Build Python 16 MIO Any-to-Any Streaming Learn Python 17 Video-Language Temporal Grounding Build Python 18 Long-Video at Million-Token Context Build Python 19 Audio-Language Models: Whisper to AF3 Build Python 20 Omni Models: Thinker-Talker Streaming Build Python 21 Embodied VLAs: RT-2, OpenVLA, π0, GR00T Learn Python 22 Document and Diagram Understanding Build Python 23 ColPali Vision-Native Document RAG Build Python 24 Multimodal RAG and Cross-Modal Retrieval Build Python 25 Multimodal Agents and Computer-Use (Capstone) Build Python Phase 13 — Tools & Protocols 23 lessons The interfaces between AI and the real world. # Lesson Type Lang 01 The Tool Interface Learn Python 02 Function Calling Deep Dive Build Python 03 Parallel and Streaming Tool Calls Build Python 04 Structured Output Build Python 05 Tool Schema Design Learn Python 06 MCP Fundamentals Learn Python 07 Building an MCP Server Build Python 08 Building an MCP Client Build Python 09 MCP Transports Learn Python 10 MCP Resources and Prompts Build Python 11 MCP Sampling Build Python 12 MCP Roots and Elicitation Build Python 13 MCP Async Tasks Build Python 14 MCP Apps Build Python 15 MCP Security I — Tool Poisoning Learn Python 16 MCP Security II — OAuth 2.1 Build Python 17 MCP Gateways and Registries Learn Python 18 MCP Auth in Production — Enrollment, JWKS Refresh, Audience Pinning Build Python 19 A2A Protocol Build Python 20 OpenTelemetry GenAI Build Python 21 LLM Routing Layer Learn Python 22 Skills and Agent SDKs Learn Python 23 Capstone — Tool Ecosystem Build Python Phase 14 — Agent Engineering 42 lessons Build agents from first principles — loop, memory, planning, frameworks, benchmarks, production, workbench. # Lesson Type Lang 01 The Agent Loop Build Python 02 ReWOO and Plan-and-Execute Build Python 03 Reflexion and Verbal Reinforcement Learning Build Python 04 Tree of Thoughts and LATS Build Python 05 Self-Refine and CRITIC Build Python 06 Tool Use and Function Calling Build Python 07 Memory — Virtual Context and MemGPT Build Python 08 Memory Blocks and Sleep-Time Compute Build Python 09 Hybrid Memory — Mem0 Vector + Graph + KV Build Python 10 Skill Libraries and Lifelong Learning — Voyager Build Python 11 Planning with HTN and Evolutionary Search Build Python 12 Anthropic's Workflow Patterns Build Python 13 LangGraph — Stateful Graphs and Durable Execution Build Python 14 AutoGen v0.4 — Actor Model Build Python 15 CrewAI — Role-Based Crews and Flows Build Python 16 OpenAI Agents SDK — Handoffs, Guardrails, Tracing Build Python 17 Claude Agent SDK — Subagents and Session Store Build Python 18 Agno and Mastra — Production Runtimes Learn Python 19 Benchmarks — SWE-bench, GAIA, AgentBench Learn Python 20 Benchmarks — WebArena and OSWorld Learn Python 21 Computer Use — Claude, OpenAI CUA, Gemini Build Python 22 Voice Agents — Pipecat and LiveKit Build Python 23 OpenTelemetry GenAI Semantic Conventions Build Python 24 Agent Observability — Langfuse, Phoenix, Opik Learn Python 25 Multi-Agent Debate and Collaboration Build Python 26 Failure Modes — Why Agents Break Build Python 27 Prompt Injection and the PVE Defense Build Python 28 Orchestration Patterns — Supervisor, Swarm, Hierarchical Build Python 29 Production Runtimes — Queue, Event, Cron Learn Python 30 Eval-Driven Agent Development Build Python 31 Agent Workbench: Why Capable Models Still Fail Learn Python 32 The Minimal Agent Workbench Build Python 33 Agent Instructions as Executable Constraints Build Python 34 Repo Memory and Durable State Build Python 35 Initialization Scripts for Agents Build Python 36 Scope Contracts and Task Boundaries Build Python 37 Runtime Feedback Loops Build Python 38 Verification Gates Build Python 39 Reviewer Agent: Separate Builder from Marker Build Python 40 Multi-Session Handoff Build Python 41 The Workbench on a Real Repo Build Python 42 Capstone: Ship a Reusable Agent Workbench Pack Build Python Each Phase 14 workbench lesson (31-42) ships a mission.md briefing the agent before it opens the full lesson docs. Phase 15 — Autonomous Systems 22 lessons Long-horizon agents, self-improvement, and the 2026 safety stack. # Lesson Type Lang 01 From Chatbots to Long-Horizon Agents (METR) Learn Python 02 STaR, V-STaR, Quiet-STaR: Self-Taught Reasoning Learn Python 03 AlphaEvolve: Evolutionary Coding Agents Learn Python 04 Darwin Gödel Machine: Self-Modifying Agents Learn Python 05 AI Scientist v2: Workshop-Level Research Learn Python 06 Automated Alignment Research (Anthropic AAR) Learn Python 07 Recursive Self-Improvement: Capability vs Alignment Learn Python 08 Bounded Self-Improvement Designs Learn Python 09 Autonomous Coding Agent Landscape (SWE-bench, CodeAct) Learn Python 10 Claude Code Permission Modes and Auto Mode Learn Python 11 Browser Agents and Indirect Prompt Injection Learn Python 12 Durable Execution for Long-Running Agents Learn Python 13 Action Budgets, Iteration Caps, Cost Governors Learn Python 14 Kill Switches, Circuit Breakers, Canary Tokens Learn Python 15 HITL: Propose-Then-Commit Learn Python 16 Checkpoints and Rollback Learn Python 17 Constitutional AI and Rule Overrides Learn Python 18 Llama Guard and Input/Output Classification Learn Python 19 Anthropic Responsible Scaling Policy v3.0 Learn Python 20 OpenAI Preparedness Framework and DeepMind FSF Learn Python 21 METR Time Horizons and External Evaluation Learn Python 22 CAIS, CAISI, and Societal-Scale Risk Learn Python Phase 16 — Multi-Agent & Swarms 25 lessons Coordination, emergence, and collective intelligence. # Lesson Type Lang 01 Why Multi-Agent Learn TypeScript 02 FIPA-ACL Heritage and Speech Acts Learn Python 03 Communication Protocols Build TypeScript 04 The Multi-Agent Primitive Model Learn Python 05 Supervisor / Orchestrator-Worker Pattern Build Python 06 Hierarchical Architecture and Decomposition Drift Learn Python 07 Society of Mind and Multi-Agent Debate Build Python 08 Role Specialization — Planner / Critic / Executor / Verifier Build Python 09 Parallel Swarm and Networked Architectures Build Python 10 Group Chat and Speaker Selection Build Python 11 Handoffs and Routines (Stateless Orchestration) Build Python 12 A2A — The Agent-to-Agent Protocol Build Python 13 Shared Memory and Blackboard Patterns Build Python 14 Consensus and Byzantine Fault Tolerance Build Python 15 Voting, Self-Consistency, and Debate Topology Build Python 16 Negotiation and Bargaining Build Python 17 Generative Agents and Emergent Simulation Build Python 18 Theory of Mind and Emergent Coordination Build Python 19 Swarm Optimization (PSO, ACO) Build Python 20 MARL — MADDPG, QMIX, MAPPO Learn Python 21 Agent Economies, Token Incentives, Reputation Learn Python 22 Production Scaling — Queues, Checkpoints, Durability Build Python 23 Failure Modes — MAST, Groupthink, Monoculture Learn Python 24 Evaluation and Coordination Benchmarks Learn Python 25 Case Studies and 2026 State of the Art Learn Python Phase 17 — Infrastructure & Production 28 lessons Ship AI to the real world. # Lesson Type Lang 01 Managed LLM Platforms — Bedrock, Azure OpenAI, Vertex AI Learn Python 02 Inference Platform Economics — Fireworks, Together, Baseten, Modal Learn Python 03 GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler Learn Python 04 vLLM Serving Internals — PagedAttention, Continuous Batching, Chunked Prefill Learn Python 05 EAGLE-3 Speculative Decoding in Production Learn Python 06 SGLang and RadixAttention for Prefix-Heavy Workloads Learn Python 07 TensorRT-LLM on Blackwell with FP8 and NVFP4 Learn Python 08 Inference Metrics — TTFT, TPOT, ITL, Goodput, P99 Learn Python 09 Production Quantization — AWQ, GPTQ, GGUF, FP8, NVFP4 Learn Python 10 Cold Start Mitigation for Serverless LLMs Learn Python 11 Multi-Region LLM Serving and KV Cache Locality Learn Python 12 Edge Inference — ANE, Hexagon, WebGPU, Jetson Learn Python 13 LLM Observability Stack Selection Learn Python 14 Prompt Caching and Semantic Caching Economics Learn Python 15 Batch APIs — the 50% Discount as Industry Standard Learn Python 16 Model Routing as a Cost-Reduction Primitive Learn Python 17 Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d Learn Python 18 vLLM Production Stack with LMCache KV Offloading Learn Python 19 AI Gateways — LiteLLM, Portkey, Kong, Bifrost Learn Python 20 Shadow, Canary, and Progressive Deployment Learn Python 21 A/B Testing LLM Features — GrowthBook and Statsig Learn Python 22 Load Testing LLM APIs — k6, LLMPerf, GenAI-Perf Build Python 23 SRE for AI — Multi-Agent Incident Response Learn Python 24 Chaos Engineering for LLM Production Learn Python 25 Security — Secrets, PII Scrubbing, Audit Logs Learn Python 26 Compliance — SOC 2, HIPAA, GDPR, EU AI Act, ISO 42001 Learn Python 27 FinOps for LLMs — Unit Economics and Multi-Tenant Attribution Learn Python 28 Self-Hosted Serving Selection — llama.cpp, Ollama, TGI, vLLM, SGLang Learn Python Phase 18 — Ethics, Safety & Alignment 30 lessons Build AI that helps humanity. Not optional. # Lesson Type Lang 01 Instruction-Following as Alignment Signal Learn Python 02 Reward Hacking & Goodhart's Law Learn Python 03 Direct Preference Optimization Family Learn Python 04 Sycophancy as RLHF Amplification Learn Python 05 Constitutional AI & RLAIF Learn Python 06 Mesa-Optimization & Deceptive Alignment Learn Python 07 Sleeper Agents — Persistent Deception Learn Python 08 In-Context Scheming in Frontier Models Learn Python 09 Alignment Faking Learn Python 10 AI Control — Safety Despite Subversion Learn Python 11 Scalable Oversight & Weak-to-Strong Learn Python 12 Red-Teaming: PAIR & Automated Attacks Build Python 13 Many-Shot Jailbreaking Learn Python 14 ASCII Art & Visual Jailbreaks Build Python 15 Indirect Prompt Injection Build Python 16 Red-Team Tooling: Garak, Llama Guard, PyRIT Build Python 17 WMDP & Dual-Use Capability Evaluation Learn Python 18 Frontier Safety Frameworks — RSP, PF, FSF Learn Python 19 Model Welfare Research Learn Python 20 Bias & Representational Harm Build Python 21 Fairness Criteria: Group, Individual, Counterfactual Learn Python 22 Differential Privacy for LLMs Build Python 23 Watermarking: SynthID, Stable Signature, C2PA Build Python 24 Regulatory Frameworks: EU, US, UK, Korea Learn Python 25 EchoLeak & CVEs for AI Learn Python 26 Model, System & Dataset Cards Build Python 27 Data Provenance & Training-Data Governance Learn Python 28 Alignment Research Ecosystem: MATS, Redwood, Apollo, METR Learn Python 29 Moderation Systems: OpenAI, Perspective, Llama Guard Build Python 30 Dual-Use Risk: Cyber, Bio, Chem, Nuclear Learn Python Phase 19 — Capstone Projects 85 lessons 17 end-to-end products + 9 deep-build tracks. 20-40 hours per project; 4-12 lessons per track. # Project Combines Lang 01 Terminal-Native Coding Agent P0 P5 P7 P10 P11 P13 P14 P15 P17 P18 Python 02 RAG over Codebase (Cross-Repo Semantic Search) P5 P7 P11 P13 P17 Python 03 Real-Time Voice Assistant (ASR → LLM → TTS) P6 P7 P11 P13 P14 P17 Python 04 Multimodal Document QA (Vision-First) P4 P5 P7 P11 P12 P17 Python 05 Autonomous Research Agent (AI-Scientist Class) P0 P2 P3 P7 P10 P14 P15 P16 P18 Python 06 DevOps Troubleshooting Agent for Kubernetes P11 P13 P14 P15 P17 P18 Python 07 End-to-End Fine-Tuning Pipeline P2 P3 P7 P10 P11 P17 P18 Python 08 Production RAG Chatbot (Regulated Vertical) P5 P7 P11 P12 P17 P18 Python 09 Code Migration Agent (Repo-Level Upgrade) P5 P7 P11 P13 P14 P15 P17 Python 10 Multi-Agent Software Engineering Team P11 P13 P14 P15 P16 P17 Python 11 LLM Observability & Eval Dashboard P11 P13 P17 P18 Python 12 Video Understanding Pipeline (Scene → QA) P4 P6 P7 P11 P12 P17 Python 13 MCP Server with Registry and Governance P11 P13 P14 P17 P18 Python 14 Speculative-Decoding Inference Server P3 P7 P10 P17 Python 15 Constitutional Safety Harness + Red-Team Range P10 P11 P13 P14 P18 Python 16 GitHub Issue-to-PR Autonomous Agent P11 P13 P14 P15 P17 Python 17 Personal AI Tutor (Adaptive, Multimodal) P5 P6 P11 P12 P14 P17 P18 Python Deep-build tracks — multi-lesson series that build a complete subsystem from scratch. # Project Combines Lang 20 Agent Harness Loop Contract A. Agent harness Python 21 Tool Registry with Schema Validation A. Agent harness Python 22 JSON-RPC 2.0 Over Newline-Delimited Stdio A. Agent harness Python 23 Function Call Dispatcher A. Agent harness Python 24 Plan-Execute Control Flow A. Agent harness Python 25 Verification Gates and Observation Budget A. Agent harness Python 26 Sandbox Runner with Denylist and Path Jail A. Agent harness Python 27 Eval Harness with Fixture Tasks A. Agent harness Python 28 Observability with OTel GenAI Spans and Prometheus Metrics A. Agent harness Python 29 End-to-End Coding Agent on the Harness A. Agent harness Python 30 BPE Tokenizer From Scratch B. NLP LLM Python 31 Tokenized Dataset with Sliding Window B. NLP LLM Python 32 Token and Positional Embeddings B. NLP LLM Python 33 Multi-Head Self-Attention B. NLP LLM Python 34 Transformer Block from Scratch B. NLP LLM Python 35 GPT Model Assembly B. NLP LLM Python 36 Training Loop and Evaluation B. NLP LLM Python 37 Loading Pretrained Weights B. NLP LLM Python 38 Classifier Fine-Tuning by Head Swap B. NLP LLM Python 39 Instruction Tuning by Supervised Fine-Tuning B. NLP LLM Python 40 Direct Preference Optimization from Scratch B. NLP LLM Python 41 Full Evaluation Pipeline B. NLP LLM Python 42 Large Corpus Downloader C. Train end-to-end Python 43 HDF5 Tokenized Corpus C. Train end-to-end Python 44 Cosine LR with Linear Warmup C. Train end-to-end Python 45 Gradient Clipping and Mixed Precision C. Train end-to-end Python 46 Gradient Accumulation C. Train end-to-end Python 47 Checkpoint Save and Resume C. Train end-to-end Python 48 Distributed Data Parallel and FSDP from Scratch C. Train end-to-end Python 49 Language Model Evaluation Harness C. Train end-to-end Python 50 Hypothesis Generator D. Auto research Python 51 Literature Retrieval D. Auto research Python 52 Experiment Runner D. Auto research Python 53 Result Evaluator D. Auto research Python 54 Paper Writer D. Auto research Python 55 Critic Loop D. Auto research Python 56 Iteration Scheduler D. Auto research Python 57 End-to-End Research Demo D. Auto research Python 58 Vision Encoder Patches E. Multimodal VLM Python 59 Vision Transformer Encoder E. Multimodal VLM Python 60 Projection Layer for Modality Alignment E. Multimodal VLM Python 61 Cross-Attention Fusion E. Multimodal VLM Python 62 Vision-Language Pretraining E. Multimodal VLM Python 63 Multimodal Evaluation E. Multimodal VLM Python 64 Chunking Strategies, Compared F. Advanced RAG Python 65 Hybrid Retrieval with BM25 and Dense Embeddings F. Advanced RAG Python 66 Cross-Encoder Reranker F. Advanced RAG Python 67 Query Rewriting: HyDE, Multi-Query, and Decomposition F. Advanced RAG Python 68 RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer Relevance F. Advanced RAG Python 69 End-to-End RAG System F. Advanced RAG Python 70 Task Spec Format G. Eval framework Python 71 Classical Metrics G. Eval framework Python 72 Code Exec Metric G. Eval framework Python 73 Perplexity and Calibration G. Eval framework Python 74 Leaderboard Aggregation G. Eval framework Python 75 End-to-End Eval Runner G. Eval framework Python 76 Collective Ops From Scratch H. Distributed train Python 77 Data Parallel DDP From Scratch H. Distributed train Python 78 ZeRO Optimizer State Sharding H. Distributed train Python 79 Pipeline Parallel and Bubble Analysis H. Distributed train Python 80 Sharded Checkpoint and Atomic Resume H. Distributed train Python 81 End-to-End Distributed Training H. Distributed train Python 82 Jailbreak Taxonomy I. Safety harness Python 83 Prompt Injection Detector I. Safety harness Python 84 Refusal Evaluation I. Safety harness Python 85 Content Classifier Integration I. Safety harness Python 86 Constitutional Rules Engine I. Safety harness Python, YAML 87 End-to-End Safety Gate I. Safety harness Python ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ The toolkit Every lesson produces a reusable artifact. By the end you have: outputs/ ├── prompts/ prompt templates for every AI task └── skills/ SKILL.md files for AI coding agents Install them with npx skills add. Plug them into Claude, Cursor, Codex, OpenClaw, Hermes, or any agent that reads a SKILL.md / AGENTS.md directory. Real tools, not homework. Install every course skill into your agent The repo ships 388 skills and 99 prompts under phases/**/outputs/. Recommended: install via skills.sh. No clone, no Python, detects your agent's skills directory automatically: npx skills add rohitg00/ai-engineering-from-scratch # every skill npx skills add rohitg00/ai-engineering-from-scratch --skill agent-loop # one skill npx skills add rohitg00/ai-engineering-from-scratch --phase 14 # one phase skills writes to whichever directory your agent picks up: .claude/skills/, .cursor/skills/, .codex/skills/, OpenClaw's skills folder, Hermes's bundle path, or any SKILL.md-aware tool. One command, every agent. Advanced: offline / custom layout via scripts/install_skills.py. Requires cloning the repo. Useful when you need tag filters, dry-runs, or a non-default layout: python3 scripts/install_skills.py <target> # every skill, default --layout skills (nested) python3 scripts/install_skills.py <target> --layout skills # same as above, explicit python3 scripts/install_skills.py <target> --type all # skills + prompts + agents python3 scripts/install_skills.py <target> --phase 14 # one phase only python3 scripts/install_skills.py <target> --tag rag # filter by tag python3 scripts/install_skills.py <target> --layout flat # flat files python3 scripts/install_skills.py <target> --dry-run # preview without writing python3 scripts/install_skills.py <target> --force # overwrite existing files <target> is the skills directory for your agent (examples: ~/.claude/skills/, ~/.cursor/skills/, ~/.config/openclaw/skills/, .skills/, or any path your agent reads). By default the script refuses to overwrite an existing destination and exits with code 1 after listing every colliding path. Use --dry-run to preview collisions or --force to overwrite. Every non-dry-run run writes a manifest.json in the target with the full inventory grouped by type and phase. Pick the layout your agent reads: --layout Path written skills <target>/<name>/SKILL.md (nested convention, supported by Claude / Cursor / Codex / OpenClaw / Hermes) by-phase <target>/phase-NN/<name>.md flat <target>/<name>.md Drop the agent workbench into your own repo The Phase 14 capstone ships a reusable Agent Workbench pack (AGENTS.md, schemas, init / verify / handoff scripts). Scaffold it into any repo with: python3 scripts/scaffold_workbench.py path/to/your-repo # full pack + seeds python3 scripts/scaffold_workbench.py path/to/your-repo --minimal # skip docs/ python3 scripts/scaffold_workbench.py path/to/your-repo --dry-run # preview only python3 scripts/scaffold_workbench.py path/to/your-repo --force # overwrite You get the seven workbench surfaces wired up, a starter task_board.json, and a fresh agent_state.json at schema_version: 1. From there: edit the task, edit AGENTS.md, run scripts/init_agent.py, hand the contract to your agent. The pack source lives at phases/14-agent-engineering/42-agent-workbench-capstone/outputs/agent-workbench-pack/. Browse the entire course as JSON scripts/build_catalog.py walks every phase, every lesson, every artifact on disk and writes catalog.json at the repo root. One file, every course truth. python3 scripts/build_catalog.py # writes <repo>/catalog.json python3 scripts/build_catalog.py --stdout # to stdout, do not touch repo python3 scripts/build_catalog.py --out path/to/file.json The catalog is filesystem-derived, not README-derived, so counts always match what is actually on disk. Use it for site builds, downstream tooling, or to verify the README counts have not drifted. Schema is documented at the top of the script. A GitHub Action (.github/workflows/curriculum.yml) rebuilds catalog.json on every PR and fails the build if the committed file is stale. After editing any lesson, run python3 scripts/build_catalog.py and commit the result, or CI will reject the PR. The same workflow runs audit_lessons.py in warn-only mode (so existing drift does not block contributors). Smoke-check every lesson's Python code scripts/lesson_run.py byte-compiles every .py file under each lesson's code/ directory. Default mode is syntax-check only — no execution, no API keys, no heavy ML deps required. Catches the regressions contributors introduce most often (bad indentation, broken f-strings, stray edits). python3 scripts/lesson_run.py # syntax-check the whole curriculum python3 scripts/lesson_run.py --phase 14 # one phase only python3 scripts/lesson_run.py --json # JSON report on stdout python3 scripts/lesson_run.py --strict # exit 1 if any lesson fails python3 scripts/lesson_run.py --execute # actually run, 10s timeout per lesson --execute runs each lesson's code/main.py (or the first .py file) with a 10-second timeout. Lessons whose entry file starts with a # requires: pkg1, pkg2 comment listing non-stdlib deps are skipped with reason needs <deps>. The script is opt-in and not wired into CI. Stdlib only, Python 3.10+. Set LINK_CHECK_SKIP=domain1,domain2 to override the default skip-list (twitter.com, x.com, linkedin.com, instagram.com, medium.com — domains that aggressively block automated HEAD/GET). Where to start Background Start at Estimated time New to programming and AI Phase 0 — Setup ~306 hours Know Python, new to ML Phase 1 — Math Foundations ~270 hours Know ML, new to deep learning Phase 3 — Deep Learning Core ~200 hours Know deep learning, want LLMs and agents Phase 10 — LLMs from Scratch ~100 hours Senior engineer, only want agent engineering Phase 14 — Agent Engineering ~60 hours ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ Why this matters now FIG_003 · ATHE INDUSTRY SIGNAL FIG_003 · BFOUNDATIONAL PAPERS COVERED "The hottest new programming language is English." — Andrej Karpathy (tweet) "Software engineering is being remade in front of our eyes." — Boris Cherny, creator of Claude Code "Models will keep getting better. The skill that compounds is knowing what to build." — Industry consensus, 2026 Attention Is All You Need — Vaswani et al., 2017 → Phase 7 Language Models are Few-Shot Learners (GPT-3) → Phase 10 Denoising Diffusion Probabilistic Models → Phase 8 InstructGPT / RLHF → Phase 10 Direct Preference Optimization → Phase 10 Chain-of-Thought Prompting → Phase 11 ReAct: Reasoning + Acting in LLMs → Phase 14 Model Context Protocol — Anthropic → Phase 13 ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ Contributing Goal Read Contribute a lesson or fix CONTRIBUTING.md Fork for your team or school FORKING.md Lesson template LESSON_TEMPLATE.md Track progress ROADMAP.md Glossary glossary/terms.md Code of conduct CODE_OF_CONDUCT.md Before submitting a lesson, run the invariant check: python3 scripts/audit_lessons.py # full curriculum python3 scripts/audit_lessons.py --phase 14 # single phase python3 scripts/audit_lessons.py --json # CI-friendly output Exit code is non-zero when any rule fails. Rules (L001–L010) validate directory shape, docs/en.md presence + H1, code/ non-emptiness, quiz.json schema (rejects the legacy q/choices/answer keys that caused issue #102), and relative links inside lesson docs. ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ Sponsor the work Free, MIT-licensed, 503 lessons. The curriculum is maintained on sponsorship alone. Cash only. Reach (verified 2026-05-14): 55,593 monthly visitors · 90,709 page views · 7.5K stars · Twitter/X is the #1 acquisition channel. Current sponsors: CodeRabbit · iii Tier $/mo What you get Backer $25 Name in BACKERS.md Bronze $250 Text-only row in README sponsor block + launch-day tweet Silver $750 Small logo in README + listed as one supported provider in API lessons Gold $2,000 Medium logo in README + sponsor page + quarterly X / LinkedIn co-feature Platinum $5,000 Hero logo above the fold + one dedicated integration lesson, max 1 partner Full rate card, hard rules, pricing anchors, and reach data: SPONSORS.md. Sign up via GitHub Sponsors. ░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒░░░▒▒▒ Star history If this manual helped you, star the repo. It keeps the project alive. License MIT. Use it however you want — fork it, teach it, sell it, ship it. Attribution appreciated, not required. Maintained by Rohit Ghumare and the community. @ghumare64 · aiengineeringfromscratch.com · Report / Suggest
The open-source AI voice studio. Clone, dictate, create. Voicebox The open-source AI voice studio. Clone any voice. Generate speech. Dictate into any app. Talk to agents in voices you own. The full voice I/O stack, running locally on your machine. voicebox.sh • Docs • Download • Features • API • Troubleshooting Click the image above to watch the demo video on voicebox.sh What is Voicebox? Voicebox is a local-first AI voice studio — a free and open-source alternative to ElevenLabs and WisprFlow in one app. Clone voices from a few seconds of audio, generate speech in 23 languages across 7 TTS engines, dictate into any text field with a global hotkey, and give any MCP-aware AI agent a voice of your choosing. The two cloud incumbents sit on opposite halves of the voice I/O loop — ElevenLabs on output, WisprFlow on input. Voicebox does both, bridges them with a bundled local LLM for refinement and per-profile personas, and runs the whole thing on your machine. Complete privacy — models, voice data, and captures never leave your machine 7 TTS engines — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro Voice cloning and preset voices — zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro and Qwen CustomVoice 23 languages — from English to Arabic, Japanese, Hindi, Swahili, and more Post-processing effects — pitch shift, reverb, delay, chorus, compression, and filters Expressive speech — paralinguistic tags like [laugh], [sigh], [gasp] via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice Unlimited length — auto-chunking with crossfade for scripts, articles, and chapters Stories editor — multi-track timeline for conversations, podcasts, and narratives Voice input — global dictation hotkey with push-to-talk and toggle modes, accessibility-verified auto-paste on macOS, in-app mic on every text field, Whisper-based STT Agent voice output — one tool call (voicebox.speak) and any MCP-aware agent (Claude Code, Cursor, Cline) speaks to you in a voice you've cloned Voice personalities — attach a free-form persona to any voice profile, then Compose, Rewrite, or Respond via a bundled local LLM — agents can invoke the same modes over MCP API-first — REST API plus a built-in MCP server for integrating voice I/O into your own apps and agents Native performance — built with Tauri (Rust), not Electron Runs everywhere — macOS (MLX/Metal), Windows (CUDA), Linux, AMD ROCm, Intel Arc, Docker Download Platform Download macOS (Apple Silicon) Download DMG macOS (Intel) Download DMG Windows Download MSI Docker docker compose up View all binaries → Linux — Pre-built binaries are not yet available. See voicebox.sh/linux-install for build-from-source instructions. Having trouble? See the Troubleshooting Guide for common install, generation, model-download, and GPU issues. Features Multi-Engine Voice Cloning Seven TTS engines with different strengths, switchable per-generation: Engine Languages Strengths Qwen3-TTS (0.6B / 1.7B) 10 High-quality multilingual cloning, delivery instructions ("speak slowly", "whisper") Qwen CustomVoice 10 9 curated preset voices with natural-language delivery control — no reference audio required LuxTTS English Lightweight (~1GB VRAM), 48kHz output, 150x realtime on CPU Chatterbox Multilingual 23 Broadest language coverage — Arabic, Danish, Finnish, Greek, Hebrew, Hindi, Malay, Norwegian, Polish, Swahili, Swedish, Turkish and more Chatterbox Turbo English Fast 350M model with paralinguistic emotion/sound tags TADA (1B / 3B) 10 HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment Kokoro 8 50 curated preset voices, tiny 82M model, fast CPU inference Emotions & Paralinguistic Tags Only Chatterbox Turbo interprets paralinguistic tags like [laugh] and [sigh]. Qwen3-TTS, LuxTTS, Chatterbox Multilingual, and HumeAI TADA read them literally as text. With Chatterbox Turbo selected, type / in the text input to open the tag inserter and add expressive tags inline with speech: [laugh] [chuckle] [gasp] [cough] [sigh] [groan] [sniff] [shush] [clear throat] Post-Processing Effects 8 audio effects powered by Spotify's pedalboard library. Apply after generation, preview in real time, build reusable presets. Effect Description Pitch Shift Up or down by up to 12 semitones Reverb Configurable room size, damping, wet/dry mix Delay Echo with adjustable time, feedback, and mix Chorus / Flanger Modulated delay for metallic or lush textures Compressor Dynamic range compression Gain Volume adjustment (-40 to +40 dB) High-Pass Filter Remove low frequencies Low-Pass Filter Remove high frequencies Ships with 4 built-in presets (Robotic, Radio, Echo Chamber, Deep Voice) and supports custom presets. Effects can be assigned per-profile as defaults. Unlimited Generation Length Text is automatically split at sentence boundaries and each chunk is generated independently, then crossfaded together. Works with all engines. Configurable auto-chunking limit (100–5,000 chars) Crossfade slider (0–200ms) for smooth transitions Max text length: 50,000 characters Smart splitting respects abbreviations, CJK punctuation, and [tags] Generation Versions Every generation supports multiple versions with provenance tracking: Original — clean TTS output, always preserved Effects versions — apply different effects chains from any source version Takes — regenerate with a new seed for variation Source tracking — each version records its lineage Favorites — star generations for quick access Async Generation Queue Generation is non-blocking. Submit and immediately start typing the next one. Serial execution queue prevents GPU contention Real-time SSE status streaming Failed generations can be retried Stale generations from crashes auto-recover on startup Voice Profile Management Create profiles from audio files or record directly in-app Import/export profiles to share or back up Multi-sample support for higher quality cloning Per-profile default effects chains Organize with descriptions and language tags Stories Editor Multi-voice timeline editor for conversations, podcasts, and narratives. Multi-track composition with drag-and-drop Inline audio trimming and splitting Auto-playback with synchronized playhead Version pinning per track clip Global Dictation & Voice Input The other half of the voice I/O loop. Hold a hotkey anywhere on your system, speak, release — on macOS the transcript pastes straight into the focused text field. Or hit the mic on any Voicebox text input and dictate directly into the app. Configurable chord bindings — hold-to-speak and tap-to-toggle chords, each rebindable in the in-app chord picker. Holding push-to-talk and tapping Space mid-hold upgrades into a toggle session without a gap in audio Target-aware paste (macOS) — accessibility-verified injection into the focused text field, with atomic clipboard save/restore so your clipboard isn't clobbered First-run permissions UX — in-app gates walk you through the macOS Accessibility and Input Monitoring grants with deep-links to System Settings In-app mic button on every Voicebox text field — generation form, profile descriptions, story titles, anywhere you'd type LLM refinement — optional cleanup of ums, stutters, and false starts before paste On-screen pill — floating overlay surfacing recording, transcribing, refining, and speaking states. Same pill agents use when they speak to you, so there's one mental model for both directions of the loop Speech-to-Text Voicebox runs OpenAI Whisper for transcription — the same model that backs dictation, the Captures tab, and the /transcribe API. Running on MLX (Apple Silicon) or PyTorch (CUDA / ROCm / DirectML / CPU) depending on your platform. Size Notes Base / Small / Medium / Large Standard Whisper quality ladder Turbo ~8x faster than Whisper Large, minimal quality loss More engines (Parakeet v3, Qwen3-ASR) are planned — see Roadmap. Captures Every dictation, in-app recording, and uploaded audio file lands in the Captures tab — original audio paired with transcript, always preserved. Replay, re-transcribe, refine — rerun STT with any Whisper size, or re-run the raw transcript through the local LLM with different flags (filler cleanup, self-correction removal, technical-term preservation) Edit inline — tweak the transcript and save on blur Play as voice profile — turn any capture into speech with a cloned voice, one click Promote to voice sample — use a capture's audio + transcript as a reference sample on any voice profile Local capture storage — original audio and transcript stay in your Voicebox data directory, with a folder shortcut in Settings Agent Voice Output Every agent gets a voice. One tool call and any MCP-aware agent can speak to you in a voice you've cloned — task completions, questions, notifications. The same pill that surfaces during dictation surfaces during agent speech, so you always see what's coming out of your machine. // In any MCP-aware agent: await voicebox.speak({ text: "Deploy complete.", profile: "Morgan", }); Also exposed as POST /speak for anything that doesn't speak MCP — ACP, A2A, shell scripts, custom harnesses. Bidirectional pill — recording, transcribing, refining, and speaking are all states of the same OS-level overlay, so dictation and agent speech share one surface Per-agent voice binding — in Settings → MCP, pin Claude Code to Morgan and Cursor to Scarlett so you can tell which agent is talking without looking. Each client's last_seen_at timestamp confirms the install actually took Always visible — no silent background TTS; every agent-initiated speak surfaces the pill with the voice profile name for the full duration HTTP + stdio transports — install as a URL in Claude Code / Cursor / Windsurf / VS Code MCP, or point stdio-only clients at the bundled voicebox-mcp binary Voice Personalities Attach a free-form personality to any voice profile — who this voice is, how they speak, what they care about. Two actions appear on the generate box when a personality is set, powered by a bundled Qwen3 LLM running entirely locally. Compose — a shuffle button that drops a fresh in-character line into the textarea; edit and speak, or click again for a different take Speak in character — a toggle that routes your input text through the personality LLM to be rewritten in their voice before TTS Agents can reach the same rewrite path over MCP by passing personality: true to voicebox.speak, turning the tool into a text-in → personality-LLM → TTS pipeline. The same LLM backs dictation's refinement step — one LLM in the app, one model cache, one GPU-memory footprint. Local LLM options: Qwen3 0.6B / 1.7B / 4B, sharing the TTS runtime (MLX on Apple Silicon, PyTorch elsewhere). Use cases: agent dev loops (dictate a question, hear the answer in a cloned voice), interactive characters for games and narrative tools, speech assistance for people who can't speak in their original voice. Model Management Per-model unload to free GPU memory without deleting downloads Custom models directory via VOICEBOX_MODELS_DIR Model folder migration with progress tracking Download cancel/clear UI GPU Support Platform Backend Notes macOS (Apple Silicon) MLX (Metal) 4-5x faster via Neural Engine Windows / Linux (NVIDIA) PyTorch (CUDA) Auto-downloads CUDA binary from within the app Linux (AMD) PyTorch (ROCm) Auto-configures HSA_OVERRIDE_GFX_VERSION Windows (any GPU) DirectML Universal Windows GPU support Intel Arc IPEX/XPU Intel discrete GPU acceleration Any CPU Works everywhere, just slower API Voicebox exposes a REST API for integrating voice I/O into your own apps and agents. # Generate speech curl -X POST http://127.0.0.1:17493/generate \ -H "Content-Type: application/json" \ -d '{"text": "Hello world", "profile_id": "abc123", "language": "en"}' # Agent voice output — any app or script can speak in a cloned voice curl -X POST http://127.0.0.1:17493/speak \ -H "Content-Type: application/json" \ -H "X-Voicebox-Client-Id: my-script" \ -d '{"text": "Deploy complete.", "profile": "Morgan"}' # Transcribe an audio file curl -X POST http://127.0.0.1:17493/transcribe \ -F "audio=@recording.wav" \ -F "model=whisper-turbo" # List voice profiles curl http://127.0.0.1:17493/profiles POST /speak accepts profile as a name (case-insensitive) or id, and resolves via the same precedence as the MCP tool: explicit arg → per-client binding → capture_settings.default_playback_voice_id. MCP server Voicebox ships a built-in Model Context Protocol server so any MCP-aware agent (Claude Code, Cursor, Windsurf, Cline, VS Code MCP extensions) can speak, transcribe, and browse captures and profiles. Claude Code one-liner: claude mcp add voicebox \ --transport http \ --url http://127.0.0.1:17493/mcp \ --header "X-Voicebox-Client-Id: claude-code" Any HTTP MCP client (Cursor, Windsurf, VS Code, etc.): { "mcpServers": { "voicebox": { "url": "http://127.0.0.1:17493/mcp", "headers": { "X-Voicebox-Client-Id": "cursor" } } } } Stdio fallback for clients that don't speak HTTP MCP — point at the bundled voicebox-mcp binary inside the app: { "mcpServers": { "voicebox": { "command": "/Applications/Voicebox.app/Contents/MacOS/voicebox-mcp", "env": { "VOICEBOX_CLIENT_ID": "claude-desktop" } } } } Four tools ship: voicebox.speak, voicebox.transcribe, voicebox.list_captures, voicebox.list_profiles. Per-client voice bindings are managed in Voicebox → Settings → MCP. See the full MCP guide for tool signatures, resolution precedence, the speaking-pill contract, and security notes. // In any MCP-aware agent: await voicebox.speak({ text: "Tests passing. Ready to merge.", profile: "Morgan", // optional — falls back to the per-client binding personality: true, // optional — rewrites text through the profile's personality LLM first }); Use cases: agent dev loops (voice in, voice out), game dialogue, podcast production, accessibility tools, voice assistants, content automation. Full API documentation available at http://127.0.0.1:17493/docs. Tech Stack Layer Technology Desktop App Tauri (Rust) Frontend React, TypeScript, Tailwind CSS State Zustand, React Query Backend FastAPI (Python) TTS Engines Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro STT Whisper / Whisper Turbo (PyTorch or MLX) Local LLM Qwen3 (0.6B / 1.7B / 4B), shared runtime with TTS / STT MCP Server FastMCP mounted at /mcp (Streamable HTTP) + bundled stdio shim binary Native Shim Rust (inside Tauri) for global hotkey, paste injection, focus introspection Effects Pedalboard (Spotify) Inference MLX (Apple Silicon) / PyTorch (CUDA/ROCm/XPU/CPU) Database SQLite Audio WaveSurfer.js, librosa Roadmap Feature Description Windows / Linux auto-paste Dictation paste parity — SendInput on Windows, uinput / AT-SPI on Linux STT engine expansion Parakeet v3 and Qwen3-ASR joining Whisper — 50+ languages, better non-English quality Pipeline routing Configurable source → transform → sink chains with webhook + MCP sinks and a preset editor Streaming transcription WebSocket /transcribe/stream for partial transcripts as you speak End-to-end speech LLMs Moshi, GLM-4-Voice, Qwen2.5 Omni — real voice-to-voice, no text between Voice Design Create new voices from text descriptions Long-form capture Dual-stream recorder (mic + system audio) with summary LLM transform Platform sinks Apple Notes, Obsidian, and other opt-in integrations Plugin architecture Extend with custom models, transforms, and sinks Mobile companion Control Voicebox from your phone For the full engineering status, open-issue triage, and prioritized work queue, see docs/PROJECT_STATUS.md — a living document that tracks what's shipped, what's in-flight, candidate TTS engines under evaluation, and why we've accepted or backlogged specific integrations. Development See CONTRIBUTING.md for detailed setup and contribution guidelines. Quick Start git clone https://github.com/jamiepine/voicebox.git cd voicebox just setup # creates Python venv, installs all deps just dev # starts backend + desktop app Install just: brew install just or cargo install just. Run just --list to see all commands. Prerequisites: Bun, Rust, Python 3.11+, Tauri Prerequisites, and Xcode on macOS. The repo ships a pre-wired .mcp.json at the root — running Claude Code inside this checkout picks up the Voicebox MCP tools automatically once the dev app is running. Building Locally just build # Build CPU server binary + Tauri app just build-local # (Windows) Build CPU + CUDA server binaries + Tauri app Adding New Voice Models The multi-engine architecture makes adding new TTS engines straightforward. A step-by-step guide covers the full process: dependency research, backend protocol implementation, frontend wiring, and PyInstaller bundling. The guide is optimized for AI coding agents. An agent skill can pick up a model name and handle the entire integration autonomously — you just test the build locally. Project Structure voicebox/ ├── app/ # Shared React frontend ├── tauri/ # Desktop app (Tauri + Rust) ├── web/ # Web deployment ├── backend/ # Python FastAPI server ├── landing/ # Marketing website └── scripts/ # Build & release scripts Contributing Contributions welcome! See CONTRIBUTING.md for guidelines. Fork the repo Create a feature branch Make your changes Submit a PR Security Found a security vulnerability? Please report it responsibly. See SECURITY.md for details. License MIT License — see LICENSE for details. voicebox.sh
KnockOutEZ 发布了 wigolo，一个本地优先、无需 API 密钥的 AI 代理网络智能工具，提供搜索、抓取、提取、缓存等十种工具，通过 MCP、REST 和 SDK 三种方式与各类 AI 编码代理集成，所有数据默认存储在本地 ~/.wigolo/ 目录下。
KnockOutEZ 在 GitHub 上发布了 wigolo，一个面向 AI 代理的本地优先网络智能工具。它提供搜索、抓取、提取、缓存、研究等十种工具，核心功能无需 API 密钥即可使用。用户通过一条命令 npx wigolo init 即可完成安装配置，支持 Claude Code、Cursor、Codex 等主流 AI 编码代理的 MCP 集成。所有缓存、嵌入向量、模型和配置默认存储在本地 ~/.wigolo/ 目录下，不向第三方发送数据。文章通过与 Firecrawl、Exa、Tavily 的对比测试，展示了 wigolo 在逐字节引证摘录、可解释评分分解和本地持久化记忆方面的差异化优势。
GitHub 发布了多语言 Copilot SDK，覆盖 Python、TypeScript、Go、.NET、Java 和 Rust，允许开发者将 Copilot 的智能代理工作流嵌入自己的应用，并支持 BYOK 自定义模型和多种认证方式。
GitHub 于 2026 年正式发布了 Copilot SDK（通用可用版本），提供 Python、TypeScript、Go、.NET、Java 和 Rust 六种语言的客户端库。该 SDK 暴露了与 Copilot CLI 相同的生产级代理运行时引擎，通过 JSON-RPC 协议通信，自动管理 CLI 进程生命周期，开发者只需定义代理行为，由 Copilot 处理规划、工具调用和文件编辑。SDK 支持 BYOK（自带密钥）模式，可使用 OpenAI、Azure AI Foundry 和 Anthropic 等提供商的 API 密钥，无需 GitHub 认证。此外还支持 GitHub OAuth、环境变量令牌等多种认证方式，并允许自定义代理、技能和工具。
PostHog 是一个开源的全栈产品分析平台，集产品分析、会话回放、功能开关、A/B 测试、错误追踪、AI 可观测性和自助代理模式于一体，支持自托管部署和云服务，并通过 MCP 协议将平台能力接入 Claude Code、Cursor 等 AI 编辑器。
PostHog 是一个开源的全栈产品分析平台，提供了产品分析、网页分析、会话回放、功能开关、A/B 测试、错误追踪、日志管理、问卷调查、数据仓库、数据管道、AI 可观测性和自助代理模式等十余种工具。用户可选择免费注册使用 PostHog Cloud（US 或 EU 区域），或通过 Docker 一行命令自托管部署开源版本。每月免费配额包含 100 万事件、5000 条录制、100 万功能开关请求、10 万异常和 1500 条问卷回复，超出后按使用量付费。该平台还支持通过 MCP 协议将全部功能接入 Claude Code、Cursor 等 AI 编程工具。
AstrBotDevs 开源了 AstrBot，一个集成主流即时通讯的全能 Agent 聊天机器人平台。它支持 AI 对话、MCP、知识库、1000+ 插件扩展和多种部署方式（uv 一键部署、Docker、桌面应用等）。
AstrBotDevs 在 GitHub 上发布了 AstrBot 开源项目，这是一个全栈 Agent 聊天机器人平台，旨在为个人、开发者和团队提供可靠的对话 AI 基础设施。该平台集成了主流即时通讯应用（QQ、企业微信、飞书、钉钉、Telegram、Slack 等），支持 AI 大模型对话、多模态交互、Agent、MCP、技能系统、知识库和人物设定等核心功能。AstrBot 同时集成了 Dify、阿里云百炼、Coze 等 Agent 平台，提供 1000+ 社区插件扩展和 Agent 沙箱隔离执行环境，支持 uv、Docker、桌面应用、AUR 等多种部署方式。
Coding Agent Harness jcode The next generation coding agent harness to raise the skill ceiling. Built for multi-session workflows, infinite customizability, and performance. Website · Features · Install · Quick Start · Further Reading · Contributing Installation # macOS & Linux curl -fsSL https://jcode.sh/install | bash # Windows 11 (PowerShell 5.1+) irm https://jcode.sh/install.ps1 | iex Need Homebrew, source builds, provider setup, or want an agent to set it up for you? Jump to detailed installation. Performance & Resource Efficiency jcode is built to be as performant and resource efficient as possible. Every metric is optimized to the bone, which is important for scaling multi-session workflows. Here we sample a few metrics to show the difference: RAM usage and boot up. RAM comparison 1 active session Tool PSS Comparison jcode (local embedding off) 27.8 MB baseline jcode 167.1 MB 6.0× more RAM pi 144.4 MB 5.2× more RAM Codex CLI 140.0 MB 5.0× more RAM OpenCode 371.5 MB 13.4× more RAM GitHub Copilot CLI 333.3 MB 12.0× more RAM Cursor Agent 214.9 MB 7.7× more RAM Claude Code 386.6 MB 13.9× more RAM Antigravity CLI 243.7 MB 8.8× more RAM 10 active sessions Tool PSS Comparison jcode (local embedding off) 117.0 MB baseline jcode 260.8 MB 2.2× more RAM pi 833.0 MB 7.1× more RAM Codex CLI 334.8 MB 2.9× more RAM OpenCode 3237.2 MB 27.7× more RAM GitHub Copilot CLI 1756.5 MB 15.0× more RAM Cursor Agent 1632.4 MB 14.0× more RAM Claude Code 2300.6 MB 19.7× more RAM Antigravity CLI 1021.2 MB 8.7× more RAM Time to first frame Tool Time to first frame Range Comparison jcode 14.0 ms 10.1–19.3 ms baseline Antigravity CLI 383.5 ms 363.1–415.4 ms 27.4× slower pi 590.7 ms 369.6–934.8 ms 42.2× slower Codex CLI 882.8 ms 742.3–1640.9 ms 63.1× slower OpenCode 1035.9 ms 922.5–1104.4 ms 74.0× slower GitHub Copilot CLI 1518.6 ms 1357.4–1826.8 ms 108.5× slower Cursor Agent 1949.7 ms 1711.0–2104.8 ms 139.3× slower Claude Code 3436.9 ms 2032.7–8927.2 ms 245.5× slower Measured on this Linux machine across 10 interactive PTY launches. Time to first input (time until typed probe text appears on the rendered screen; Antigravity uses its internal input-ready log marker because the sign-in screen suppresses probe echo.) Tool Time to first input Range Comparison jcode 48.7 ms 30.3–62.7 ms baseline Antigravity CLI 383.7 ms 363.4–415.7 ms 7.9× slower pi 596.4 ms 373.9–955.2 ms 12.2× slower Codex CLI 905.8 ms 760.1–1675.7 ms 18.6× slower OpenCode 1047.9 ms 931.1–1116.9 ms 21.5× slower GitHub Copilot CLI 1583.4 ms 1422.8–1880.0 ms 32.5× slower Cursor Agent 1978.7 ms 1727.3–2130.0 ms 40.6× slower Claude Code 3512.8 ms 2137.4–9002.0 ms 72.2× slower Measured on this Linux machine across 10 interactive PTY launches. Antigravity CLI was unauthenticated for this run; its sign-in screen rendered normally and emitted an internal CLI ready for user input marker, but did not echo the typed probe. Additional clients / memory scaling Tool Extra PSS per added session Comparison jcode (local embedding off) ~9.9 MB baseline jcode ~10.4 MB 1.1× more RAM pi ~76.5 MB 7.7× more RAM Codex CLI ~21.6 MB 2.2× more RAM OpenCode ~318.4 MB 32.2× more RAM GitHub Copilot CLI ~158.1 MB 16.0× more RAM Cursor Agent ~157.5 MB 15.9× more RAM Claude Code ~212.7 MB 21.5× more RAM Antigravity CLI ~86.4 MB 8.7× more RAM versions tested for this corrected memory rerun: jcode v0.9.1888-dev (be386f2) pi 0.62.0 codex-cli 0.120.0 opencode 1.0.203 GitHub Copilot CLI 1.0.24 for the 1-session rerun, GitHub Copilot CLI 1.0.27 for the 10-session rerun Cursor Agent 2026.04.08-a41fba1 Claude Code 2.1.86 (Claude Code) Antigravity CLI 1.0.0 jcode performance demonstration Memory (Agent memory) Jcode embeds each turn/response as a semantic vector. Every turn does queries a graph of memories to efficiently find related memory entries via a cosine similarity check. The embedding hits are fed into the conversation, or optionally uses a memory sideagent which verifies the memories are relevant, and potentially does more work for information retreival before injecting into the conversation. This results in a human like memory system which allows the agent to automatically recall relevant information to the conversation without actively calling memory tools or being a token burner. ot To have memories which are retrieved, they must also be extracted and stored. Every so often (semantic drift, K turns since last extraction, session end, etc), memories are extracted via a memory sideagent, and put into the memory graph. The harness also provides explicit memory tools to allow the agent to actively search or store the memory without relying on a passive background process. The harness also provides session search for traditional RAG on previous sessions. Memories are automatically consolidated every so often via the ambient mode. This reorganizes, checks for staleness and conflicts, etc jcode memory demonstration UI: Side panels, Diagrams, Info Widgets, rendering, scrolling, alignment The side panel is a place for auxiliary information. Tell your jcode agent to load a file into the side panel and see it update in real time, or tell your agent to write directly to the side panel, or use it as a diff viewer. The side panel (and chat) is able to render mermaid diagrams inline. To make this possible, I created a new mermaid rendering library to render diagrams 1800x faster. It has no browser or Typescript dependency. See https://github.com/1jehuang/mermaid-rs-renderer To show you important information without taking space away from the screen that could be used for responses, I developed info widgets. Info widgets will only ever take up the negative space on the screen to show you information, and will get out of the way if there isn't any. Jcode can render at over a thousand fps. Your monitor will not have the refresh rate to show you, but this means you will not have silly flicker problems. The custom scrollback implementation of jcode allows it to do much more than a native scrollback. However, it is a terminal-level limitation that I cannot have smooth, partial line scrolling with a custom scrollback. To fix this, I made my own terminal. Handterm https://github.com/1jehuang/handterm implements a native scroll api, and also happens to be very effiecent. This is a work in progress. Scrolling is still well implemented for normal terminals. Jcode is left-aligned by default. You can switch to centered mode with the Alt+C hotkey, with the /alignment command, or in the config. Swarm Spawn two or more agents in the same repo, and they will automatically be managed by the server to allow native collaboration. When agent A edits a file that agent B has read (code shifting under its feet), the server notifies agent B. Agent B can ignore it if it is not relevant, or it can check the diff to make sure that it doesn't conflict. Each agent has messaging abilities, capable of DMing just one agent, broadcasting to all other agents hosted by the server, or just agents working in that repo. This allows you to spawn multiple sessions in the same repo, and have all conflicts automatically resolved. jcode swarm demonstration Agents are also able to spawn their own swarms autonomously. They have a swarm tool which allows them to spawn in their own teamates to accomplish tasks in parallel. Doing so turns the main agent into a coordinator and the spawned agents into workers. Groups of agents, their messaging channels, their completion statuses, etc are all automatically managed. This can be done headlessly or headed. OAuth and Providers jcode works with subscription-backed OAuth flows and many provider integrations, so you can use the models you already pay for and still fall back to direct API providers when needed. Supported built-in login flows Claude (jcode login --provider claude) OpenAI / ChatGPT / Codex (jcode login --provider openai) Google Gemini (jcode login --provider gemini) GitHub Copilot (jcode login --provider copilot) Azure OpenAI (jcode login --provider azure) Alibaba Cloud Coding Plan (jcode login --provider alibaba-coding-plan) Fireworks (jcode login --provider fireworks) MiniMax (jcode login --provider minimax) LM Studio (jcode login --provider lmstudio) Ollama (jcode login --provider ollama) Custom OpenAI-compatible endpoint (jcode login --provider openai-compatible) For custom OpenAI-compatible endpoints, jcode now prompts for the API base and supports local localhost servers without requiring an API key. Config-file setup for self-hosted endpoints and MCP If you prefer to configure things by editing files instead of using the login UI, jcode supports both a custom OpenAI-compatible endpoint config and MCP config files. OpenAI-compatible providers Many hosted services speak the standard OpenAI /v1/chat/completions API. jcode talks to them through one shared OpenAI-compatible provider, so you can use almost any such endpoint without waiting for a dedicated integration. There are two ways to set one up: Built-in named profiles — jcode ships ready-made profiles for several popular OpenAI-compatible services. Log in by id and jcode fills in the base URL and key environment variable for you: jcode login --provider <profile-id> # for example: jcode login --provider openrouter jcode login --provider deepseek jcode login --provider opencode # OpenCode Zen jcode login --provider moonshotai Built-in OpenAI-compatible profile ids include: openrouter, deepseek, zai, kimi, moonshotai, opencode (OpenCode Zen), opencode-go, 302ai, baseten, cortecs, huggingface, nebius, scaleway, stackit, and firmware. Each profile only sets the endpoint and key variable; you still pick the model with /model (or --model). Run jcode login with no provider to see the interactive list. Any other endpoint — point jcode at an arbitrary OpenAI-compatible API (hosted or local) with jcode login --provider openai-compatible or the scriptable jcode provider add command described below. Useful environment overrides for these endpoints: JCODE_STREAM_IDLE_TIMEOUT_SECS — raise the streaming idle timeout (default 180s) for slow reasoning models that think silently before emitting tokens. Also settable as [provider] stream_idle_timeout_secs in config.toml. Per-model context_window (alias context_limit) in a [[providers.<name>.models]] entry — set the context window when the endpoint has no usable /v1/models response, so jcode does not fall back to the generic 200k default. extra_body — inject non-standard top-level fields into every chat/completions request body for backends that require them. See Extra request-body fields below. For details on self-hosting, local runtimes, and the exact config file shape, see below. Self-hosted OpenAI-compatible endpoints, including vLLM For agents and scripts, the preferred path is the one-shot provider profile command. It writes a named profile to ~/.jcode/config.toml, stores secrets in jcode's private app config directory when requested, and prints exact run/validation commands: # Secret-safe setup for a hosted OpenAI-compatible API. printf '%s' "$MY_API_KEY" | jcode provider add my-api \ --base-url https://llm.example.com/v1 \ --model my-model-id \ --api-key-stdin \ --set-default \ --json # Smoke test the profile. jcode --provider-profile my-api auth-test --prompt 'Reply exactly JCODE_PROVIDER_SETUP_OK' # Use it directly. jcode --provider-profile my-api run 'hello' For local servers that do not require auth: jcode provider add local-vllm \ --base-url http://localhost:8000/v1 \ --model Qwen/Qwen3-Coder-30B-A3B-Instruct \ --no-api-key \ --set-default Built-in local profiles are available for the common desktop/local runtimes: # Ollama: start the local server and install a model first. ollama pull llama3.2 jcode login --provider ollama jcode --provider ollama --model llama3.2 run 'hello' # LM Studio: start the Local Server, load a chat model, then use the exact # model identifier shown by LM Studio or by curl http://localhost:1234/v1/models. jcode login --provider lmstudio jcode --provider lmstudio --model '<model-id>' run 'hello' Ollama and LM Studio both expose OpenAI-compatible /v1/models and /v1/chat/completions endpoints. jcode uses streaming chat completions, function/tool calling, and OpenAI-style image content for vision-capable local models. If a local server requires a token, enter it during jcode login or create a named profile with --api-key-stdin. Useful flags: --api-key-env NAME: reference an existing environment variable instead of storing a key. --api-key-stdin: read and store a key without putting it in shell history. --context-window TOKENS: persist the model context window for model selection and routing. --overwrite: replace an existing profile of the same name. --model-catalog: use the endpoint's /models response in addition to configured models. The generated profile can also be edited manually in ~/.jcode/config.toml: [provider] default_provider = "my-api" default_model = "my-model-id" [providers.my-api] type = "openai-compatible" base_url = "https://llm.example.com/v1" api_key_env = "JCODE_PROVIDER_MY_API_API_KEY" env_file = "provider-my-api.env" default_model = "my-model-id" [[providers.my-api.models]] id = "my-model-id" context_window = 128000 Extra request-body fields (extra_body) Some OpenAI-compatible backends require non-standard top-level request fields. For example, NVIDIA NIM DeepSeek-V4 reasoning models (deepseek-ai/deepseek-v4-flash, deepseek-ai/deepseek-v4-pro) only enable thinking when the request includes chat_template_kwargs; without it they reply without reasoning (or, for some deployments, hang). jcode lets you inject arbitrary top-level fields two ways. Per named profile, via extra_body in config.toml (a TOML table merged verbatim into the JSON body): [providers.my-nim] type = "openai-compatible" base_url = "https://integrate.api.nvidia.com/v1" api_key_env = "NVIDIA_API_KEY" default_model = "deepseek-ai/deepseek-v4-flash" [providers.my-nim.extra_body.chat_template_kwargs] thinking = true reasoning_effort = "high" For built-in profiles (e.g. nvidia-nim) or any endpoint, via the JCODE_OPENAI_EXTRA_BODY environment variable (a JSON object string). It can live in the provider's env file (~/.config/jcode/nvidia-nim.env) next to the API key: JCODE_OPENAI_EXTRA_BODY={"chat_template_kwargs":{"thinking":true,"reasoning_effort":"high"}} Keys from extra_body are merged last and override any jcode-generated body field with the same name (JCODE_OPENAI_EXTRA_BODY wins over the config extra_body on key collisions). Invalid values are logged and ignored rather than failing the request. The custom OpenAI-compatible provider reads overrides from environment variables or from an env file in jcode's app config directory. On Linux this is usually ~/.config/jcode/, so the default file is usually: ~/.config/jcode/openai-compatible.env Example for a local or LAN vLLM server: JCODE_OPENAI_COMPAT_API_BASE=http://192.168.1.50:8000/v1 JCODE_OPENAI_COMPAT_DEFAULT_MODEL=Qwen/Qwen3-Coder-30B-A3B-Instruct # Optional if your server expects auth OPENAI_COMPAT_API_KEY=your-token-here Notes: jcode login --provider openai-compatible can create or update this for you. Plain http:// is accepted for localhost and private LAN IPs. Public remote HTTP is still rejected. HTTPS endpoints work as usual. MCP config files MCP config is separate from config.toml. Primary config files: ~/.jcode/mcp.json for global MCP servers .jcode/mcp.json for project-local MCP servers Claude Code compatibility: ~/.claude.json (Claude Code's user config): top-level mcpServers, plus per-project servers under projects.<abs_path>.mcpServers for the current directory .mcp.json at the repo root (Claude Code's project config) .claude/mcp.json (legacy fallback) Both the canonical mcpServers key and jcode's historical servers key are accepted. jcode currently supports stdio (command-based) servers only; HTTP/SSE entries ("type": "http"/"sse") are recognized and skipped with a log line. Example MCP config: { "mcpServers": { "filesystem": { "command": "/path/to/mcp-server", "args": ["--root", "/workspace"], "env": {}, "shared": true } } } On first run, jcode also tries to import MCP servers from ~/.claude.json (falling back to the legacy ~/.claude/mcp.json) and ~/.codex/config.toml if ~/.jcode/mcp.json does not exist yet. For headless or SSH sessions, OAuth-style providers support jcode login --provider <provider> --no-browser (alias: --headless) so jcode prints the auth URL/QR and falls back to manual code or callback paste instead of trying to launch a local browser. For more scriptable remote flows, claude, openai, gemini, and antigravity also support a two-step pattern: # Step 1: print a resumable auth URL jcode login --provider openai --print-auth-url --json # Step 2: complete later with the callback URL or auth code jcode login --provider openai --callback-url 'http://localhost:1455/auth/callback?...' jcode login --provider gemini --auth-code '...' Additional scriptable cases: # Copilot device flow: print URL + user code, then complete later jcode login --provider copilot --print-auth-url --json jcode login --provider copilot --complete # Gmail/Google OAuth after credentials are already configured jcode login --provider google --print-auth-url --google-access-tier readonly jcode login --provider google --callback-url 'http://127.0.0.1:8456?...' Pending scriptable login state is stored under ~/.jcode/pending-login/, automatically expires, and stale entries are cleaned up when new scriptable logins start or resume. For the built-in OpenAI login flow, jcode opens a local callback on http://localhost:1455/auth/callback by default. The above image is the first page of provider logins Supported provider Native / first-party style providers: claude, openai, copilot, gemini, azure, alibaba-coding-plan Aggregator / compatibility providers: openrouter, openai-compatible Additional provider integrations: opencode, opencode-go, zai / kimi, 302ai, baseten, cortecs, deepseek, firmware, huggingface, moonshotai, nebius, scaleway, stackit, groq, mistral, perplexity, togetherai, deepinfra, fireworks, minimax, xai, lmstudio, ollama, chutes, cerebras, cursor, antigravity, google Jcode also supports easy multi-account switching. Ran out of tokens on your first ChatGPT Pro subscription? /account and quickly switch to your second. Customizability / Self-Dev Jcode is inventing a new form of customizability. One that doesn't limit you to what a plugin or extension can do. Tell your jcode agent to enter self dev mode, and it will start modifying its own source code. Jcode is optimized to iterate on itself. There is significant infrastructure around self developement, which allows it to edit, build, and test its own source code, then reload its own binary and continue work in your (potentially many) sessions, fully automatically. It is reccomended that you use a frontier model for this. The jcode codebase is not a simple one, and weaker models can make subtle, breaking changes. GPT 5.5 or the latest available frontier model works well. Misc. The devil is in the details. There are many undocumented optimizations and niceties that jcode implements. Some examples: Anthropic's Claude cache goes cold after 5 minutes. If you initiate Claude after these 5 minutes, you have a cache miss, potentially costing you lots of tokens. The ui warns you when the cache went cold, and notfies you if there was an unexpected cache miss. jcode comes with instructions on how to set up Firefox Agent Bridge. Ask you agent to set it up, and then you will have browser automation in jcode as well. Agent grep is a grep tool I made for the jcode agent. It adds file strucuture information (ie the list of functions, their displacement, etc) to the grep return, so that the agent can infer more of what the file doesn without actually reading the file. It also implements a harness-level integration that adaptively truncates returns based on what the agent has already seen. This saves on context a lot. Inputs are by default interleaved with the working agent. It sends the input as soon as it safely can without breaking the KV cache. Submit with shift enter instead, and it will send a queue send, and wait for the agent to fully finish its turn before sending. Resume sessions from different harnesses. Claude code broke on you? Resume the session from jcode and continue where you left off. Session resume is supported for codex, claude code, opencode, and pi. image of /Resume for codex sessions Skills are not all loaded on startup. The conversation is embedded as a semantic vector, and will automatically inject a skill if there is an embedding hit similar to memories. The agent has a skill tool for you to manually activate a skill at anytime. You may also activate via slash commands. iOS Application / Native OpenClaw A native iOS application version of jcode is coming soon. This will allow you to work with jcode on your personal machine's environment from your phone, via Tailscale. Openclaw like features will be bundled with this iOS application. Other planned features Agents dont like to commit in dirty git state with active changes. Git was clearly not built for multi-agent workflows, and git worktrees is not a good solution. Given this, I believe that is an opporunity for a new git like primitive to be born. Build speed improvements: An incremental debug cargo build with cache enabled takes about 1 minute on my machine. The goal is 5-20 seconds. Refactors and crates seams should be able to make this happen. Quick Start # Launch the TUI jcode # Run a single command non-interactively jcode run "say hello" # Resume a previous session by memorable name jcode --resume fox # Run as a persistent background server, then attach more clients jcode serve jcode connect # Send voice input from your configured STT command jcode dictate jcode supports interactive TUI use, non-interactive runs, persistent server/client workflows, and hotkey-friendly dictation without requiring a bundled speech-to-text stack. jcode workflow demonstration Browser Automation jcode includes a first-class built-in browser tool for browser control inside agent sessions. Current built-in backend: Firefox via Firefox Agent Bridge Current built-in tool actions include: status setup open snapshot get_content interactables click type fill_form select wait screenshot eval scroll upload press Quick setup: jcode browser status jcode browser setup Once setup is complete, the model can use the built-in browser tool directly. The UI also summarizes browser tool calls compactly, for example opening a URL, clicking a selector, or typing into a field without echoing sensitive typed text. Notes: the provider/tool architecture is in place for additional backends Firefox is the wired built-in backend today Chrome bridge / remote debugging style providers can be added on top of the same browser tool later Further Reading Ambient Mode / OpenClaw Browser Provider Protocol Memory Architecture Swarm Architecture Server Architecture Safety System Sponsored Discovery Sponsor Onboarding Windows Notes Wrappers and Shell Integration Refactoring Notes Detailed Installation Setup If you want another agent to set up jcode for you, give it this prompt: Set up jcode on this machine for me. 1. Detect the operating system, available package managers, and shell environment, then install jcode using the best matching command below instead of referring me somewhere else: - macOS with Homebrew available: brew tap 1jehuang/jcode brew install jcode - macOS or Linux via install script: curl -fsSL https://jcode.sh/install | bash - Windows PowerShell: irm https://jcode.sh/install.ps1 | iex - From source if the above paths are not appropriate: git clone https://github.com/1jehuang/jcode.git cd jcode cargo build --release scripts/install_release.sh - For local self-dev / refactor work on Linux x86_64, prefer: scripts/dev_cargo.sh build --release -p jcode --bin jcode scripts/dev_cargo.sh --print-setup scripts/install_release.sh 2. Verify that `jcode` is on my `PATH`. 3. Launch `jcode` once in a new terminal window/session to confirm it starts successfully. 4. Before attempting any interactive login flow, assess which providers are already available non-interactively and prefer those first. Check existing local credentials, config files, CLI sessions, and environment variables such as: - Claude: `~/.jcode/auth.json`, `~/.claude/.credentials.json`, `~/.local/share/opencode/auth.json`, `ANTHROPIC_API_KEY` - OpenAI: `~/.jcode/openai-auth.json`, `~/.codex/auth.json`, `OPENAI_API_KEY` - Gemini: `~/.jcode/gemini_oauth.json`, `~/.gemini/oauth_creds.json` - GitHub Copilot: existing auth under `~/.config/github-copilot/` - Azure OpenAI: `~/.config/jcode/azure-openai.env`, `AZURE_OPENAI_*`, or an existing `az login` - OpenRouter: `OPENROUTER_API_KEY` - Fireworks: `~/.config/jcode/fireworks.env`, `FIREWORKS_API_KEY` - MiniMax: `~/.config/jcode/minimax.env`, `MINIMAX_API_KEY` - NVIDIA NIM: `~/.config/jcode/nvidia-nim.env`, `NVIDIA_API_KEY` - Alibaba Cloud Coding Plan: existing jcode config/env if present 5. Prefer whichever provider is already configured and verify it with `jcode auth-test --all-configured` or a provider-specific auth test when appropriate. 6. Only if no usable provider is already configured, guide me through the minimal manual step needed: - Claude: `jcode login --provider claude` - GitHub Copilot: `jcode login --provider copilot` - OpenAI: `jcode login --provider openai` - Gemini: `jcode login --provider gemini` - Azure OpenAI: `jcode login --provider azure` - Fireworks: `jcode login --provider fireworks` - MiniMax: `jcode login --provider minimax` - NVIDIA NIM: `jcode login --provider nvidia-nim` - Alibaba Cloud Coding Plan: `jcode login --provider alibaba-coding-plan` - OpenRouter: help me set `OPENROUTER_API_KEY` - Anthropic direct API: help me set `ANTHROPIC_API_KEY` 7. After setup, run a simple smoke test with `jcode run "say hello"` and confirm it works. 8. If I want browser automation, also check `jcode browser status`. If browser automation is not ready, run `jcode browser setup`, verify the built-in `browser` tool works, and explain any remaining manual step. 9. Explain any manual step that still needs me, especially browser OAuth, device login, API key entry, or browser extension approval. This is intended to be a copy-paste bootstrap prompt for jcode itself or any other coding agent. Quick Install # macOS & Linux curl -fsSL https://jcode.sh/install | bash On Termux, install the glibc runtime and patchelf first so the installer can patch the downloaded Linux binary to Termux's glibc dynamic linker and create a launcher that avoids Termux's LD_PRELOAD shim: pkg install glibc patchelf curl -fsSL https://jcode.sh/install | bash # Windows 11 x64 or ARM64 (PowerShell 5.1+) irm https://jcode.sh/install.ps1 | iex The Windows installer selects the correct architecture and verifies the download against the release's SHA256SUMS. Alacritty and the optional global launch hotkey require explicit consent and are not installed by default. See Windows support, security, Defender, and SmartScreen notes. If a release does not contain a matching Windows asset, the installer stops instead of unexpectedly starting a long compilation. An explicit source build is available with -BuildFromSource and requires Git, Rust, and Visual Studio 2022 Build Tools with the Desktop development with C++ workload. macOS via Homebrew brew tap 1jehuang/jcode brew install jcode From Source (all platforms) git clone https://github.com/1jehuang/jcode.git cd jcode cargo build --release For local self-dev / refactor work on Linux x86_64, prefer: scripts/dev_cargo.sh build --release -p jcode --bin jcode scripts/dev_cargo.sh --print-setup That wrapper automatically uses sccache when available, prefers a fast working local linker setup (clang + lld) instead of assuming every machine's mold configuration is valid, and can print the active linker/cache setup via --print-setup so slow-path builds are easier to diagnose. Then symlink to your PATH: scripts/install_release.sh Uninstall Removes installed binaries and the launcher but keeps your config, auth, and sessions so a clean reinstall picks up where you left off: curl -fsSL https://raw.githubusercontent.com/1jehuang/jcode/master/scripts/uninstall.sh | bash -s -- --yes For a full wipe of everything including config, auth, sessions, logs, and memory (useful for recovering from a broken install): curl -fsSL https://raw.githubusercontent.com/1jehuang/jcode/master/scripts/uninstall.sh | bash -s -- --purge --yes Add --dry-run to preview what would be removed without deleting anything. Platform Support Platform Status Linux x86_64 / aarch64 Fully supported macOS Apple Silicon & Intel Supported Windows x86_64 Supported (native + WSL2) Termux aarch64 / x86_64 Supported with pkg install glibc patchelf
Scale computer-use 2.0 with open-source drivers, cross-OS fleets, and benchmarks for training, evaluation, and data generation. Scale computer-use 2.0 with open-source drivers, cross-OS fleets, and benchmarks for training, evaluation, and data generation. Choose Your Path Building your own agent? Start with Cua · Giving a coding agent a computer? Cua Drivers · Evaluating or training models? Cua Bench · Need macOS VMs? Lume Cua Drivers - Background computer-use on macOS, Windows, and Linux Drive native desktop apps in the background. Agents click, type, and verify without stealing the cursor or focus. Use the same CLI and MCP server on macOS, Windows, and Linux from Claude Code, Cursor, Codex, OpenClaw, and custom clients. Linux supports X11 and compositor-specific Wayland routes with explicit limits for raw background input. macOS / Linux /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" Windows (PowerShell) irm https://cua.ai/driver/install.ps1 | iex Then follow the post-install instructions. Drive your first app | Installation | CLI Reference Source documentation, architecture notes, and the optional agent skill pack live in libs/cua-driver/README.md. Cua - Agent-Ready Sandboxes for Any OS Build agents that see screens, click buttons, and complete tasks autonomously. One API for any VM or container image — cloud or local. pip install cua # Requires Python 3.11 or later from cua import Sandbox, Image # Same API regardless of OS or runtime async with Sandbox.ephemeral(Image.linux()) as sb: # or .macos() .windows() .android() result = await sb.shell.run("echo hello") screenshot = await sb.screenshot() await sb.mouse.click(100, 200) await sb.keyboard.type("Hello from Cua!") await sb.mobile.gesture((100, 500), (100, 200)) # multi-touch gestures Linux container Linux VM macOS Windows Android BYOI (.qcow2, .iso) Cloud (cua.ai) ✅ ✅ ✅ ✅ ✅ 🔜 soon Local (QEMU) ✅ ✅ ✅ ✅ ✅ ✅ Get Started | Examples | API Reference Cua-Bench - Benchmarks & RL Environments Evaluate computer-use agents on OSWorld, ScreenSpot, Windows Arena, and custom tasks. Export trajectories for training. # Clone, install, and create base image git clone https://github.com/trycua/cua && cd cua/cua-bench uv tool install -e . && cb image create linux-docker # Run benchmark with agent cb run dataset datasets/cua-bench-basic --agent cua-agent --max-parallel 4 Get Started | Partner With Us | Registry | CLI Reference Lume - macOS Virtualization Create and manage macOS/Linux VMs with near-native performance on Apple Silicon using Apple's Virtualization.Framework. # Install Lume /bin/bash -c "$(curl -fsSL https://cua.ai/lume/install.sh)" # Create and start a vanilla macOS VM from an Apple restore image curl -L "$(lume ipsw | tail -n 1)" -o ~/Downloads/macos-tahoe.ipsw lume create macos-tahoe --ipsw ~/Downloads/macos-tahoe.ipsw --unattended tahoe lume run macos-tahoe The --unattended option prepares the installed guest offline. The built-in sequoia and tahoe presets create the lume user, enable SSH, configure autologin, and disable sleep and screen locking. The default credentials are lume / lume. The Tahoe flow is E2E verified. Sequoia may still open the Accessibility step of Setup Assistant on its first display boot; see issue #2155. Get Started | FAQ | CLI Reference Packages Package Description cua-driver Background computer-use agent for macOS, Windows, and Linux cua-agent AI agent framework for computer-use tasks cua-sandbox SDK for creating and controlling sandboxes cua-computer-server Driver for UI interactions and code execution in sandboxes cua-bench Benchmarks and RL environments for computer-use lume macOS/Linux VM management on Apple Silicon lumier Docker-compatible interface for Lume VMs Resources Documentation — Guides, examples, and API reference Blog — Tutorials, updates, and research Discord — Community support and discussions GitHub Issues — Bug reports and feature requests Citation If Cua supports your research, please cite the software: @software{cua2025, author = {{Cua AI, Inc.}}, title = {Cua}, year = {2025}, url = {https://github.com/trycua/cua}, license = {MIT} } For reproducibility, include the Cua release or commit used in your experiments. Citation metadata is also available in CITATION.cff. Contributing We welcome contributions! See our Contributing Guidelines for details. License MIT License — see LICENSE for details. Third-party components have their own licenses: Kasm (MIT) OmniParser (CC-BY-4.0) Optional cua-agent[omni] includes ultralytics (AGPL-3.0) Trademarks Apple, macOS, Ubuntu, Canonical, and Microsoft are trademarks of their respective owners. This project is not affiliated with or endorsed by these companies. Thank you to all our GitHub Sponsors!
MoonshotAI 开源了 Kimi CLI 终端 AI 编程代理，支持代码编辑、Shell 命令执行和网页搜索，同时正逐步演进为 Kimi Code CLI，并内置 MCP 和 ACP 协议支持。
MoonshotAI 于 2026 年 7 月在 GitHub 发布了 Kimi CLI 项目，这是一个运行在终端中的 AI 代理，能够帮助开发者完成代码编辑、Shell 命令执行和网页搜索等任务。该项目正在逐步演进为 Kimi Code CLI，新版本会自动迁移配置和会话数据。Kimi CLI 内置了对 MCP（Model Context Protocol）和 ACP（Agent Client Protocol）的支持，可与 Zed、JetBrains 等 IDE 集成，同时提供了 Zsh 插件以增强终端体验。
Master programming by recreating your favorite technologies from scratch. Build your own <insert-technology-here> This repository is a compilation of well-written, step-by-step guides for re-creating our favorite technologies from scratch. What I cannot create, I do not understand — Richard Feynman. It's a great way to learn. 3D Renderer AI Model Augmented Reality BitTorrent Client Blockchain / Cryptocurrency Bot Command-Line Tool Database Docker Emulator / Virtual Machine Front-end Framework / Library Game Git Memory Allocator Network Stack Neural Network Operating System Physics Engine Processor Programming Language Regex Engine Search Engine Shell Template Engine Text Editor Visual Recognition System Voxel Engine Web Browser Web Server Uncategorized Tutorials Build your own Distributed Systems Java: Building Your Own Kafka-like System From Scratch: A Step-by-Step Guide Build your own 3D Renderer C++: Introduction to Ray Tracing: a Simple Method for Creating 3D Images C++: How OpenGL works: software rendering in 500 lines of code C++: Raycasting engine of Wolfenstein 3D C++: Physically Based Rendering:From Theory To Implementation C++: Ray Tracing in One Weekend C++: Rasterization: a Practical Implementation C# / TypeScript / JavaScript: Learning how to write a 3D soft engine from scratch in C#, TypeScript or JavaScript Java / JavaScript: Build your own 3D renderer Java: How to create your own simple 3D render engine in pure Java JavaScript / Pseudocode: Computer Graphics from scratch Python: A 3D Modeller Build your own AI Model Python: A Large Language Model (LLM) Python: Diffusion Models for Image Generation Python: RAG for Document Search Build your own Augmented Reality C#: How To: Augmented Reality App Tutorial for Beginners with Vuforia and Unity 3D [video] C#: How To Unity ARCore [video] C#: AR Portal Tutorial with Unity [video] C#: How to create a Dragon in Augmented Reality in Unity ARCore [video] C#: How to Augmented Reality AR Tutorial: ARKit Portal to the Upside Down [video] Python: Augmented Reality with Python and OpenCV Build your own BitTorrent Client C#: Building a BitTorrent client from scratch in C# Go: Building a BitTorrent client from the ground up in Go Nim: Writing a Bencode Parser Node.js: Write your own bittorrent client Python: A BitTorrent client in Python 3.5 Build your own Blockchain / Cryptocurrency ATS: Functional Blockchain Crystal: Write your own blockchain and PoW algorithm using Crystal Go: Building Blockchain in Go Go: Code your own blockchain in less than 200 lines of Go Java: Creating Your First Blockchain with Java JavaScript: A cryptocurrency implementation in less than 1500 lines of code JavaScript: Build your own Blockchain in JavaScript JavaScript: Learn & Build a JavaScript Blockchain JavaScript: Creating a blockchain with JavaScript JavaScript: How To Launch Your Own Production-Ready Cryptocurrency JavaScript: Writing a Blockchain in Node.js Kotlin: Let’s implement a cryptocurrency in Kotlin Python: Learn Blockchains by Building One Python: Build your own blockchain: a Python tutorial Python: A Practical Introduction to Blockchain with Python Python: Let’s Build the Tiniest Blockchain Ruby: Programming Blockchains Step-by-Step (Manuscripts Book Edition) Scala: How to build a simple actor-based blockchain TypeScript: Naivecoin: a tutorial for building a cryptocurrency TypeScript: NaivecoinStake: a tutorial for building a cryptocurrency with the Proof of Stake consensus Rust: Building A Blockchain in Rust & Substrate Build your own Bot Haskell: Roll your own IRC bot Node.js: Creating a Simple Facebook Messenger AI Bot with API.ai in Node.js Node.js: How to make a responsive telegram bot Node.js: Create a Discord bot Node.js: gifbot - Building a GitHub App Node.js: Building A Simple AI Chatbot With Web Speech API And Node.js Python: How to Build Your First Slack Bot with Python Python: How to build a Slack Bot with Python using Slack Events API & Django under 20 minute Python: Build a Reddit Bot Python: How To Make A Reddit Bot [video] Python: How To Create a Telegram Bot Using Python Python: Create a Twitter Bot in Python Using Tweepy Python: Creating Reddit Bot with Python & PRAW [video] R: Build A Cryptocurrency Trading Bot with R Rust: A bot for Starcraft in Rust, C or any other language Build your own Command-Line Tool Go: Visualize your local git contributions with Go Go: Build a command line app with Go: lolcat Go: Building a cli command with Go: cowsay Go: Go CLI tutorial: fortune clone Nim: Writing a stow alternative to manage dotfiles Node.js: Create a CLI tool in Javascript Rust: Command line apps in Rust Rust: Writing a Command Line Tool in Rust Zig: Build Your Own CLI App in Zig from Scratch Build your own Database C: Let's Build a Simple Database C++: Build Your Own Redis from Scratch C#: Build Your Own Database Clojure: An Archaeology-Inspired Database Crystal: Why you should build your own NoSQL Database Go: Build Your Own Database from Scratch: From B+Tree To SQL in 3000 Lines Go: Code a database in 45 steps: a series of test-driven small coding puzzles Go: Build Your Own Redis from Scratch JavaScript: Dagoba: an in-memory graph database Python: DBDB: Dog Bed Database Python: Write your own miniature Redis with Python Ruby: Build your own fast, persistent KV store in Ruby Rust: Build your own Redis client and server Build your own Docker C: Linux containers in 500 lines of code Go: Build Your Own Container Using Less than 100 Lines of Go Go: Building a container from scratch in Go [video] Python: A workshop on Linux containers: Rebuild Docker from Scratch Python: A proof-of-concept imitation of Docker, written in 100% Python Shell: Docker implemented in around 100 lines of bash Build your own Emulator / Virtual Machine C: Home-grown bytecode interpreters C: Virtual machine in C C: Write your Own Virtual Machine C: Writing a Game Boy emulator, Cinoop C++: How to write an emulator (CHIP-8 interpreter) C++: Emulation tutorial (CHIP-8 interpreter) C++: Emulation tutorial (GameBoy emulator) C++: Emulation tutorial (Master System emulator) C++: NES Emulator From Scratch [video] Common Lisp: CHIP-8 in Common Lisp JavaScript: GameBoy Emulation in JavaScript Python: Emulation Basics: Write your own Chip 8 Emulator/Interpreter Rust: 0dmg: Learning Rust by building a partial Game Boy emulator Build your own Front-end Framework / Library JavaScript: WTF is JSX (Let's Build a JSX Renderer) JavaScript: A DIY guide to build your own React JavaScript: Building React From Scratch [video] JavaScript: Gooact: React in 160 lines of JavaScript JavaScript: Learn how React Reconciler package works by building your own lightweight React DOM JavaScript: Build Yourself a Redux JavaScript: Let’s Write Redux! JavaScript: Redux: Implementing Store from Scratch [video] JavaScript: Build Your own Simplified AngularJS in 200 Lines of JavaScript JavaScript: Make Your Own AngularJS JavaScript: How to write your own Virtual DOM JavaScript: Building a frontend framework, from scratch, with components (templating, state, VDOM) JavaScript: Build your own React JavaScript: Building a Custom React Renderer [video] Build your own Game C: Handmade Hero C: How to Program an NES game in C C: Chess Engine In C [video] C: Let's Make: Dangerous Dave [video] C: Learn Video Game Programming in C [video] C: Coding A Sudoku Solver in C [video] C: Coding a Rogue/Nethack RPG in C [video] C: On Tetris and Reimplementation C++: Breakout C++: Beginning Game Programming v2.0 C++: Tetris tutorial in C++ platform independent focused in game logic for beginners C++: Remaking Cavestory in C++ [video] C++: Reconstructing Cave Story [video] C++: Space Invaders from Scratch C#: Learn C# by Building a Simple RPG C#: Creating a Roguelike Game in C# C#: Build a C#/WPF RPG Go: Games With Go [video] Java: Code a 2D Game Engine using Java - Full Course for Beginners [video] Java: 3D Game Development with LWJGL 3 JavaScript: 2D breakout game using Phaser JavaScript: How to Make Flappy Bird in HTML5 With Phaser JavaScript: Developing Games with React, Redux, and SVG JavaScript: Build your own 8-Ball Pool game from scratch [video] JavaScript: How to Make Your First Roguelike JavaScript: Think like a programmer: How to build Snake using only JavaScript, HTML & CSS Lua: BYTEPATH Python: Developing Games With PyGame Python: Making Games with Python & Pygame [pdf] Python: Roguelike Tutorial Revised Ruby: Developing Games With Ruby Ruby: Ruby Snake Rust: Adventures in Rust: A Basic 2D Game Rust: Roguelike Tutorial in Rust + tcod Build your own Git Haskell: Reimplementing “git clone” in Haskell from the bottom up JavaScript: Gitlet JavaScript: Build GIT - Learn GIT Python: Just enough of a Git client to create a repo, commit, and push itself to GitHub Python: Write yourself a Git! Python: ugit: Learn Git Internals by Building Git Yourself Ruby: Rebuilding Git in Ruby Build your own Memory Allocator C: Malloc is not magic -- Implementing your own memory allocator Build your own Network Stack C: Beej's Guide to Network Programming C: Let's code a TCP/IP stack C / Python: Build your own VPN/Virtual Switch Ruby: How to build a network stack in Ruby Build your own Neural Network C#: Neural Network OCR F#: Building Neural Networks in F#: Part 1 F#: Building Neural Networks in F#: Part 2 Go: Build a multilayer perceptron with Golang Go: How to build a simple artificial neural network with Go Go: Building a Neural Net from Scratch in Go JavaScript / Java: Neural Networks - The Nature of Code [video] JavaScript: Neural networks from scratch for JavaScript linguists (Part1 — The Perceptron) Python: A Neural Network in 11 lines of Python Python: Build Deep Learning From Scratch (reimplement PyTorch internals across 34 stages) Python: Implement a Neural Network from Scratch Python: Optical Character Recognition (OCR) Python: Traffic signs classification with a convolutional network Python: Generate Music using LSTM Neural Network in Keras Python: An Introduction to Convolutional Neural Networks Python: Neural Networks: Zero to Hero Python: SlowTorch: Implementation of PyTorch from the ground up in 100% pure Python Build your own Operating System Assembly: Writing a Tiny x86 Bootloader Assembly: Baking Pi – Operating Systems Development C: Building a software and hardware stack for a simple computer from scratch [video] C: Operating Systems: From 0 to 1 C: The little book about OS development C: Roll your own toy UNIX-clone OS C: Kernel 101 – Let’s write a Kernel C: Kernel 201 – Let’s write a Kernel with keyboard and screen support C: Build a minimal multi-tasking kernel for ARM from scratch C: How to create an OS from scratch C: Malloc tutorial C: Hack the virtual memory C: Learning operating system development using Linux kernel and Raspberry Pi C: Operating systems development for Dummies C++: Write your own Operating System [video] C++: Writing a Bootloader Rust: Writing an OS in Rust Rust: Add RISC-V Rust Operating System Tutorial (any): Linux from scratch Build your own Physics Engine C: Video Game Physics Tutorial C++: Game physics series by Allen Chou C++: How to Create a Custom Physics Engine C++: 3D Physics Engine Tutorial [video] JavaScript: How Physics Engines Work JavaScript: Broad Phase Collision Detection Using Spatial Partitioning JavaScript: Build a simple 2D physics engine for JavaScript games Build your own Processor Verilog: From Blinker to RISC-V Build your own Programming Language (any): mal - Make a Lisp Assembly: Jonesforth C: Baby's First Garbage Collector C: Build Your Own Lisp: Learn C and build your own programming language in 1000 lines of code C: Writing a Simple Garbage Collector in C C: C interpreter that interprets itself. C: A C & x86 version of the "Let's Build a Compiler" by Jack Crenshaw C: A journey explaining how to build a compiler from scratch C++: Writing Your Own Toy Compiler Using Flex C++: How to Create a Compiler [video] C++: Kaleidoscope: Implementing a Language with LLVM F#: Understanding Parser Combinators Elixir: Demystifying compilers by writing your own [video] Go: The Super Tiny Compiler Go: Lexical Scanning in Go [video] Haskell: Let's Build a Compiler Haskell: Write You a Haskell Haskell: Write Yourself a Scheme in 48 Hours Haskell: Write You A Scheme Java: Crafting interpreters: A handbook for making programming languages Java: Creating JVM Language JavaScript: The Super Tiny Compiler JavaScript: The Super Tiny Interpreter JavaScript: Little Lisp interpreter JavaScript: How to implement a programming language in JavaScript JavaScript: Let’s go write a Lisp OCaml: Writing a C Compiler OCaml: Writing a Lisp, the series Pascal: Let's Build a Compiler Python: A Python Interpreter Written in Python Python: lisp.py: Make your own Lisp interpreter Python: How to Write a Lisp Interpreter in Python Python: Let’s Build A Simple Interpreter Python: Make Your Own Simple Interpreted Programming Language [video] Python: From Source Code To Machine Code: Build Your Own Compiler From Scratch Racket: Beautiful Racket: How to make your own programming languages with Racket Ruby: A Compiler From Scratch Ruby: Markdown compiler from scratch in Ruby Rust: Learning Parser Combinators With Rust Swift: Building a LISP from scratch with Swift TypeScript: Build your own WebAssembly Compiler Build your own Regex Engine C: A Regular Expression Matcher C: Regular Expression Matching Can Be Simple And Fast Go: How to build a regex engine from scratch JavaScript: Build a Regex Engine in Less than 40 Lines of Code JavaScript: How to implement regular expressions in functional javascript using derivatives JavaScript: Implementing a Regular Expression Engine Perl: How Regexes Work Python: Build Your Own Regular Expression Engines: Backtracking, NFA, DFA Scala: No Magic: Regular Expressions Build your own Search Engine CSS: A search engine in CSS Python: Building a search engine using Redis and redis-py Python: Building a Vector Space Indexing Engine in Python Python: Building A Python-Based Search Engine [video] Python: Making text search learn from feedback Python: Finding Important Words in Text Using TF-IDF Build your own Shell C: Tutorial - Write a Shell in C C: Let's build a shell! C: Writing a UNIX Shell C: Build Your Own Shell C: Write a shell in C Go: Writing a simple shell in Go Rust: Build Your Own Shell using Rust Build your own Template Engine JavaScript: JavaScript template engine in just 20 lines JavaScript: Understanding JavaScript Micro-Templating Python: Approach: Building a toy template engine in Python Python: A Template Engine Ruby: How to write a template engine in less than 30 lines of code Build your own Text Editor C: Build Your Own Text Editor C++: Designing a Simple Text Editor Python: Python Tutorial: Make Your Own Text Editor [video] Python: Create a Simple Python Text Editor! Ruby: Build a Collaborative Text Editor Using Rails Rust: Hecto: Build your own text editor in Rust Build your own Visual Recognition System Python: Developing a License Plate Recognition System with Machine Learning in Python Python: Building a Facial Recognition Pipeline with Deep Learning in Tensorflow Build your own Voxel Engine C++: Let's Make a Voxel Engine Build your own Web Browser Rust: Let's build a browser engine Python: Browser Engineering Build your own Web Server C#: Writing a Web Server from Scratch Node.js: Build Your Own Web Server From Scratch In JavaScript Node.js: Let's code a web server from scratch with NodeJS Streams Node.js: lets-build-express PHP: Writing a webserver in pure PHP Python: A Simple Web Server Python: Let’s Build A Web Server. Python: Web application from scratch Python: Building a basic HTTP Server from scratch in Python Python: Implementing a RESTful Web API with Python & Flask Ruby: Building a simple websockets server from scratch in Ruby Uncategorized (any): From NAND to Tetris: Building a Modern Computer From First Principles (any): build-your-own-x-vibe-coding: BYOX-style tutorials adapted for vibe coding Alloy: The Same-Origin Policy C: How to Write a Video Player in Less Than 1000 Lines C: Learn how to write a hash table in C C: The very basics of a terminal emulator C: Write a System Call C: Sol - An MQTT broker from scratch C++: Build your own VR headset for $200 C++: How X Window Managers work and how to write one C++: Writing a Linux Debugger C++: How a 64k intro is made C++: Make your own Game Engine C#: C# Networking: Create a TCP chater server, TCP games, UDP Pong and more C#: Loading and rendering 3D skeletal animations from scratch in C# and GLSL Clojure: Building a spell-checker Go: Build A Simple Terminal Emulator In 100 Lines of Golang Go: Let's Create a Simple Load Balancer Go: Video Encoding from Scratch Java: How to Build an Android Reddit App [video] JavaScript: Build Your Own Module Bundler - Minipack JavaScript: Learn JavaScript Promises by Building a Promise from Scratch JavaScript: Implementing promises from scratch (TDD way) JavaScript: Implement your own — call(), apply() and bind() method in JavaScript JavaScript: JavaScript Algorithms and Data Structures JavaScript: Build a ride hailing app with React Native JavaScript: Build Your Own AdBlocker in (Literally) 10 Minutes Kotlin: Build Your Own Cache Lua: Building a CDN from Scratch to Learn about CDN Nim: Writing a Redis Protocol Parser Nim: Writing a Build system Nim: Writing a MiniTest Framework Nim: Writing a DMIDecode Parser Nim: Writing a INI Parser Nim: Writing a Link Checker Nim: Writing a URL Shortening Service Node.js: Build a static site generator in 40 lines with Node.js Node.js: Building A Simple Single Sign On(SSO) Server And Solution From Scratch In Node.js. Node.js: How to create a real-world Node CLI app with Node Node.js: Build a DNS Server in Node.js Node.js: Build a Synchronization Engine with Node.js and Y.js PHP: Write your own MVC from scratch in PHP PHP: Make your own blog PHP: Modern PHP Without a Framework PHP: Code a Web Search Engine in PHP Python: Build a Deep Learning Library [video] Python: How to Build a Kick-Ass Mobile Document Scanner in Just 5 Minutes Python: Continuous Integration System Python: Recommender Systems in Python: Beginner Tutorial Python: Write SMS-spam detector with Scikit-learn Python: A Simple Content-Based Recommendation Engine in Python Python: Stock Market Predictions with LSTM in Python Python: Building a simple Generative Adversarial Network (GAN) using Tensorflow Python: Learn ML Algorithms by coding: Decision Trees Python: JSON Decoding Algorithm Python: Build your own Git plugin with python Ruby: A Pedometer in the Real World Ruby: Creating a Linux Desktop application with Ruby Rust: Building a DNS server in Rust Rust: Writing Scalable Chat Service from Scratch Rust: WebGL + Rust: Basic Water Tutorial TypeScript: Tiny Package Manager: Learns how npm or Yarn works Contribute Submissions welcome, just send a PR, or create an issue Help us review pending submissions by leaving comments and "reactions" Origins & License This repository is the work of many contributors. It was started by Daniel Stefanovic, and is now maintained by CodeCrafters, Inc. To the extent possible under law, CodeCrafters, Inc. has waived all copyright and related or neighboring rights to this work.
AirLLM 70B inference with single 4GB GPU Quickstart | Configurations | MacOS | Example notebooks | FAQ AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB, and DeepSeek-V3 (671B) on ~12GB. AI Agents Recommendation: Best AI Game Sprite Generator Best AI Facial Expression Editor Bloome — build & run AI agent teams in the cloud, zero setup Updates [2026/06] v3.0: FP8 model support + the latest models. Run DeepSeek-V3 (671B) on ~12GB and Qwen3-235B on ~3GB, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single AutoModel. [2024/08/20] v2.11.0: Support Qwen2.5 [2024/08/18] v2.10.1 Support CPU inference. Support non sharded models. Thanks @NavodPeiris for the great work! [2024/07/30] Support Llama3.1 405B (example notebook). Support 8bit/4bit quantization. [2024/04/20] AirLLM supports Llama3 natively already. Run Llama3 70B on 4GB single GPU. [2023/12/25] v2.8.2: Support MacOS running 70B large language models. [2023/12/20] v2.7: Support AirLLMMixtral. [2023/12/20] v2.6: Added AutoModel, automatically detect model type, no need to provide model class to initialize model. [2023/12/18] v2.5: added prefetching to overlap the model loading and compute. 10% speed improvement. [2023/12/03] added support of ChatGLM, QWen, Baichuan, Mistral, InternLM! [2023/12/02] added support for safetensors. Now support all top 10 models in open llm leaderboard. [2023/12/01] airllm 2.0. Support compressions: 3x run time speed up! [2023/11/20] airllm Initial version! Star History Table of Contents Quick start Model Compression Configurations Run on MacOS Example notebooks Supported Models Acknowledgement FAQ Quickstart 1. Install package First, install the airllm pip package. pip install airllm 2. Inference Then, initialize AirLLMLlama2, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model. (You can also specify the path to save the splitted layered model through layer_shards_saving_path when init AirLLMLlama2. from airllm import AutoModel MAX_LENGTH = 128 # just pass a hugging face repo id — works with almost any popular model: model = AutoModel.from_pretrained("Qwen/Qwen3-32B") # go bigger with the exact same one line: #model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # 235B, runs in ~3GB #model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # 671B, runs in ~12GB # or use a model's local path... #model = AutoModel.from_pretrained("/home/ubuntu/.cache/huggingface/hub/models--Qwen--Qwen3-32B/snapshots/...") input_text = [ 'What is the capital of United States?', #'I like', ] input_tokens = model.tokenizer(input_text, return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH, padding=False) generation_output = model.generate( input_tokens['input_ids'].cuda(), max_new_tokens=20, use_cache=True, return_dict_in_generate=True) output = model.tokenizer.decode(generation_output.sequences[0]) print(output) Note: During inference, the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space in the huggingface cache directory. Model Compression - 3x Inference Speed Up! We just added model compression based on block-wise quantization-based model compression. Which can further speed up the inference speed for up to 3x , with almost ignorable accuracy loss! (see more performance evaluation and why we use block-wise quantization in this paper) How to enable model compression speed up: Step 1. make sure you have bitsandbytes installed by pip install -U bitsandbytes Step 2. make sure airllm verion later than 2.0.0: pip install -U airllm Step 3. when initialize the model, passing the argument compression ('4bit' or '8bit'): model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct", compression='4bit' # specify '8bit' for 8-bit block-wise quantization ) What are the differences between model compression and quantization? Quantization normally needs to quantize both weights and activations to really speed things up. Which makes it harder to maintain accuracy and avoid the impact of outliers in all kinds of inputs. While in our case the bottleneck is mainly at the disk loading, we only need to make the model loading size smaller. So, we get to only quantize the weights' part, which is easier to ensure the accuracy. Configurations When initialize the model, we support the following configurations: compression: supported options: 4bit, 8bit for 4-bit or 8-bit block-wise quantization, or by default None for no compression profiling_mode: supported options: True to output time consumptions or by default False layer_shards_saving_path: optionally another path to save the splitted model hf_token: huggingface token can be provided here if downloading gated models like: meta-llama/Llama-2-7b-hf prefetching: prefetching to overlap the model loading and compute. By default, turned on. For now, only AirLLMLlama2 supports this. delete_original: if you don't have too much disk space, you can set delete_original to true to delete the original downloaded hugging face model, only keep the transformed one to save half of the disk space. MacOS Just install airllm and run the code the same as on linux. See more in Quick Start. make sure you installed mlx and torch you probably need to install python native see more here only Apple silicon is supported Example [python notebook] (https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_on_macos.ipynb) Example Python Notebook Example colabs here: example of other models (ChatGLM, QWen, Baichuan, Mistral, etc): ChatGLM: from airllm import AutoModel MAX_LENGTH = 128 model = AutoModel.from_pretrained("THUDM/chatglm3-6b-base") input_text = ['What is the capital of China?',] input_tokens = model.tokenizer(input_text, return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH, padding=True) generation_output = model.generate( input_tokens['input_ids'].cuda(), max_new_tokens=5, use_cache= True, return_dict_in_generate=True) model.tokenizer.decode(generation_output.sequences[0]) QWen: from airllm import AutoModel MAX_LENGTH = 128 model = AutoModel.from_pretrained("Qwen/Qwen-7B") input_text = ['What is the capital of China?',] input_tokens = model.tokenizer(input_text, return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH) generation_output = model.generate( input_tokens['input_ids'].cuda(), max_new_tokens=5, use_cache=True, return_dict_in_generate=True) model.tokenizer.decode(generation_output.sequences[0]) Baichuan, InternLM, Mistral, etc: from airllm import AutoModel MAX_LENGTH = 128 model = AutoModel.from_pretrained("baichuan-inc/Baichuan2-7B-Base") #model = AutoModel.from_pretrained("internlm/internlm-20b") #model = AutoModel.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1") input_text = ['What is the capital of China?',] input_tokens = model.tokenizer(input_text, return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH) generation_output = model.generate( input_tokens['input_ids'].cuda(), max_new_tokens=5, use_cache=True, return_dict_in_generate=True) model.tokenizer.decode(generation_output.sequences[0]) To request other model support: here Supported Models AirLLM works out of the box with virtually every popular open LLM — just pass its Hugging Face ID to AutoModel.from_pretrained(...). That covers all the major families: Llama (2 / 3 / 3.1 / 3.3 / 4) · Qwen (1 / 2 / 2.5 / 3, including MoE and FP8) · DeepSeek (V2 / V3 / R1) · Mistral & Mixtral · Phi · Gemma · ChatGLM · Baichuan · InternLM · Yi — and most new models the day they're released. Tiny GPU, huge models The trick: AirLLM only ever keeps one layer on the GPU at a time, so the VRAM you need depends on the model's layer size — not its total size. That's how a 671B model fits on a hobbyist card: Model Size GPU VRAM Qwen3 / Mistral / Phi (≈8B) 8B ~1–2 GB Qwen3-30B / Mixtral (MoE) 30–47B ~1–3 GB Qwen3-235B (MoE) 235B ~3 GB Llama 3.x 70B (full precision) 70B ~4 GB Llama 3.1 405B 405B ~8 GB DeepSeek-V3 671B ~12 GB Same one line of code for all of them — no special setup. Acknowledgement A lot of the code are based on SimJeg's great work in the Kaggle exam competition. Big shoutout to SimJeg: GitHub account @SimJeg, the code on Kaggle, the associated discussion. FAQ 1. MetadataIncompleteBuffer safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer If you run into this error, most possible cause is you run out of disk space. The process of splitting model is very disk-consuming. See this. You may need to extend your disk space, clear huggingface .cache and rerun. 2. ValueError: max() arg is an empty sequence Most likely you are loading QWen or ChatGLM model with Llama2 class. Try the following: For QWen model: from airllm import AutoModel #<----- instead of AirLLMLlama2 AutoModel.from_pretrained(...) For ChatGLM model: from airllm import AutoModel #<----- instead of AirLLMLlama2 AutoModel.from_pretrained(...) 3. 401 Client Error....Repo model ... is gated. Some models are gated models, needs huggingface api token. You can provide hf_token: model = AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf", #hf_token='HF_API_TOKEN') 4. ValueError: Asking to pad but the tokenizer does not have a padding token. Some model's tokenizer doesn't have padding token, so you can set a padding token or simply turn the padding config off: input_tokens = model.tokenizer(input_text, return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH, padding=False #<----------- turn off padding ) Citing AirLLM If you find AirLLM useful in your research and wish to cite it, please use the following BibTex entry: @software{airllm2023, author = {Gavin Li}, title = {AirLLM: scaling large language models on low-end commodity computers}, url = {https://github.com/lyogavin/airllm/}, version = {0.0}, year = {2023}, } Sponsors Run AI Agent Teams in the Cloud — Bloome Bloome is an AI-agent IM platform: build and run AI agent teams in the cloud with zero setup. Add a skill as an agent in a group chat, run it in one click from web or mobile, and share it with your team — think of it as a group chat where your AI assistants are teammates you can @mention and assign tasks to. 👉 Try Bloome Contribution Welcomed contributions, ideas and discussions! If you find it useful, please ⭐ or buy me a coffee! 🙏
WrenAI 是一个开源的生成式商业智能（GenBI）引擎，让 AI 代理基于上下文语义层生成受治理的 SQL 查询和可部署的仪表盘，支持 22 种以上数据源，采用 Apache-2.0 许可证。
Canner 公司在 GitHub 上发布的开源项目 WrenAI 将 Wren Engine 合并至主仓库的 core/ 目录下，旧版 Wren GenBI Classic 保留在 legacy/v1 分支。WrenAI 提供 Generate（受治理 SQL 与图表生成）、Deploy（浏览器端仪表盘部署）、Know（版本化上下文知识管理）三层核心能力，基于 MDL 语义建模语言和 Apache DataFusion 引擎，支持通过 pip 安装的 CLI 工具和 npx 技能集成到 Claude Code、Cursor 等 AI 客户端中。
A feed-forward 3D foundation model for reconstructing scenes from streaming data LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstruction Robbyant Team https://github.com/user-attachments/assets/fe39e095-af2c-4ec9-b68d-a8ba97e505ab 🗺️ Meet LingBot-Map! We've built a feed-forward 3D foundation model for streaming 3D reconstruction! 🏗️🌍 LingBot-Map has focused on: Geometric Context Transformer: Architecturally unifies coordinate grounding, dense geometric cues, and long-range drift correction within a single streaming framework through anchor context, pose-reference window, and trajectory memory. High-Efficiency Streaming Inference: A feed-forward architecture with paged KV cache attention, enabling stable inference at ~20 FPS on 518×378 resolution over long sequences exceeding 10,000 frames. State-of-the-Art Reconstruction: Superior performance on diverse benchmarks compared to both existing streaming and iterative optimization-based approaches. 📑 Table of Contents Click to expand 📰 News 📋 TODO ⚙️ Installation 📦 Model Download 🚀 Quick Start 🎬 Interactive Demo (demo.py) Try the Example Scenes Streaming with Keyframe Interval Windowed Inference (for long sequences, >3000 frames) Sky Masking Visualization Options Performance & Memory 🎥 Offline Rendering Pipeline (demo_render/batch_demo.py) 📜 License 📖 Citation ✨ Acknowledgments 📰 News 2026-06-28 — Fixed an SDPA KV cache bug. The SDPA backend now performs better on long sequences. We still recommend the FlashInfer backend for the best performance. 2026-05-25 — 📊 Evaluation benchmark released. We released the evaluation scripts for KITTI and Oxford Spires — see benchmark/ for the pipeline, and run preprocess/oxford.py to prepare Oxford Spires data before evaluation. 2026-04-29 — 📹 Long-video demo released. We released a very-long-video example (~25 000 frames, 13-minute indoor walkthrough) rendered with the offline pipeline — see Worked Example for the command, flag rationale, and rendered output. 2026-04-27 — 🚀 LingBot-Map accelerated. Pull the latest main and run python demo.py --compile ... or python gct_profile.py --backend flashinfer --dtype bf16 --compile to verify on your hardware. 2026-04-24 — Fixed a FlashInfer KV cache bug where --keyframe_interval > 1 silently cached non-keyframes. You should now see better pose and reconstruction quality when running with more than 320 frames. 📋 TODO ✅ Release evaluation benchmark ✅ Oxford Spires dataset ✅ KITTI dataset ✅ VBR dataset ✅ Droid-W dataset ✅ TUM-D dataset ✅ 7-scenes dataset ✅ ETH3D dataset ✅ Tanks and Temples dataset ✅ NRGBD dataset ✅ Release demo scripts ✅ Indoor long-video demo (Featured indoor walkthrough) ✅ Outdoor long-video demo ✅ LingBot-World demo (Worked example) ✅ Aerial long-video demo ⚙️ Installation 1. Create conda environment conda create -n lingbot-map python=3.10 -y conda activate lingbot-map 2. Install PyTorch (CUDA 12.8) pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128 PyTorch 2.8.0 is the recommended version because NVIDIA Kaolin (required by the batch rendering pipeline) has prebuilt wheels for torch-2.8.0_cu128. If you only need demo.py you may use a newer PyTorch, but the batch renderer then requires building Kaolin from source. For other CUDA versions, see PyTorch Get Started. 3. Install lingbot-map pip install -e . 4. Install FlashInfer (recommended) FlashInfer provides paged KV cache attention for efficient streaming inference. It is a pure-Python package that JIT-compiles CUDA kernels on first use, so a single wheel works across CUDA/PyTorch versions: pip install --index-url https://pypi.org/simple flashinfer-python --index-url https://pypi.org/simple is only needed if your default pip index is an internal mirror that doesn't have flashinfer-python. (Optional) For faster first-use, you can additionally install a CUDA-specific JIT cache: pip install flashinfer-jit-cache -f https://flashinfer.ai/whl/cu128/flashinfer-jit-cache/. See FlashInfer installation for details. If FlashInfer is not installed, the model falls back to SDPA (PyTorch native attention) via --use_sdpa. 5. Visualization dependencies (optional) pip install -e ".[vis]" 📦 Model Download Model Name Huggingface Repository ModelScope Repository Description lingbot-map-long robbyant/lingbot-map Robbyant/lingbot-map Better suited for long sequences and large scale scenes. lingbot-map robbyant/lingbot-map Robbyant/lingbot-map Balanced checkpoint (used in paper, benchmark and offline demo) — trade off all-around performance across short and long sequences. lingbot-map-stage1 robbyant/lingbot-map Robbyant/lingbot-map Stage-1 training checkpoint of lingbot-map — can be loaded into the VGGT model for bidirectional inference (c2w). 🚧 Coming soon: we're training an stronger model that supports longer sequences — stay tuned. 🚀 Quick Start After installation, run your first scene with one command: python demo.py --model_path /path/to/lingbot-map-long.pt \ --image_folder example/courthouse --mask_sky This launches an interactive viser viewer at http://localhost:8080. See Interactive Demo below for the full set of scenes and flags, or jump to Offline Rendering Pipeline for long-sequence batch rendering. 🎬 Interactive Demo (demo.py) Run demo.py for interactive 3D visualization via a browser-based viser viewer (default http://localhost:8080). Try the Example Scenes We provide four example scenes in example/ that you can run out of the box: # courthouse scene python demo.py --model_path /path/to/lingbot-map-long.pt \ --image_folder example/courthouse --mask_sky https://github.com/user-attachments/assets/aa10f7ab-8024-43c7-92f8-d56159ec85c8 # University scene python demo.py --model_path /path/to/lingbot-map-long.pt \ --image_folder example/university --mask_sky https://github.com/user-attachments/assets/212a1744-6ff5-4ccf-9bd4-728608248b57 # Loop scene (loop closure trajectory) python demo.py --model_path /path/to/lingbot-map-long.pt \ --image_folder example/loop https://github.com/user-attachments/assets/5ae0a292-b081-40c6-838c-b7c1a0538d75 # Oxford scene with sky masking (outdoor, large scale scene) python demo.py --model_path /path/to/lingbot-map-long.pt \ --image_folder example/oxford --mask_sky https://github.com/user-attachments/assets/6b8daa95-9ed4-40b2-9902-7435779b886d 🎯 Featured: indoor walkthrough (~25 000 frames, 13 minutes) Sequence is too long for the interactive viser viewer — this clip was rendered with the Offline Rendering Pipeline. See that section for the full command. We will provide more examples in the follow-up. Streaming with Keyframe Interval Use --keyframe_interval to reduce KV cache memory by only keeping every N-th frame as a keyframe. Non-keyframe frames still produce predictions but are not stored in the cache. This is useful for long sequences which exceed 320 frames (We train with video RoPE on 320 views, so performance degrades when the KV cache stores more than 320 views. Using a keyframe strategy allows inference over longer sequences.). Dataset: Download the demo sequences from robbyant/lingbot-map-demo on Hugging Face. Example run on the travel sequence from the dataset above (sky masking on, 4 camera optimization iterations, keyframe every 2 frames): python demo.py \ --image_folder /path/to/lingbot-map-demo/travel/ \ --model_path /path/to/lingbot-map-long.pt \ --mask_sky \ --camera_num_iterations 4 \ --keyframe_interval 2 https://github.com/user-attachments/assets/d350b590-d036-4363-af8c-7af3918338ef Note on inference range. Our method does not perform state resetting by default, so the maximum inference range is bounded by the longest distance seen during training on the dataset. Beyond that distance, state resetting becomes necessary. If you observe pose collapse, switch to windowed mode (--mode windowed) — in most cases tuning --keyframe_interval alone is enough and the rest of the windowed parameters can stay at their defaults. Windowed Inference (for long sequences, >3000 frames) python demo.py --model_path /path/to/lingbot-map-long.pt \ --video_path video.mp4 --fps 10 \ --mode windowed --window_size 128 --overlap_keyframes 16 --keyframe_interval 2 Sky Masking Sky masking uses an ONNX sky segmentation model to filter out sky points from the reconstructed point cloud, which improves visualization quality for outdoor scenes. Setup: # Install onnxruntime (required) pip install onnxruntime # CPU # or pip install onnxruntime-gpu # GPU (faster for large image sets) The sky segmentation model (skyseg.onnx) will be automatically downloaded from HuggingFace on first use. Usage: python demo.py --model_path /path/to/checkpoint.pt \ --image_folder /path/to/images/ --mask_sky Sky masks are cached in <image_folder>_sky_masks/ so subsequent runs skip regeneration. You can also specify a custom cache directory with --sky_mask_dir, or save side-by-side mask visualizations with --sky_mask_visualization_dir: python demo.py --model_path /path/to/checkpoint.pt \ --image_folder /path/to/images/ --mask_sky \ --sky_mask_dir /path/to/cached_masks/ \ --sky_mask_visualization_dir /path/to/mask_viz/ Visualization Options Argument Default Description --port 8080 Viser viewer port --conf_threshold 1.5 Visibility threshold for filtering low-confidence points --point_size 0.00001 Point cloud point size --downsample_factor 10 Spatial downsampling for point cloud display Performance & Memory Without FlashInfer (SDPA fallback) python demo.py --model_path /path/to/checkpoint.pt \ --image_folder /path/to/images/ --use_sdpa Running on Limited GPU Memory If you run into out-of-memory issues, try one (or both) of the following: --offload_to_cpu — offload per-frame predictions to CPU during inference (on by default; use --no-offload_to_cpu only if you have memory to spare). --num_scale_frames 2 — reduce the number of bidirectional scale frames from the default 8 down to 2, which shrinks the activation peak of the initial scale phase. Faster Inference Lower the number of iterative refinement steps in the camera head to trade a small amount of pose accuracy for wall-clock speed: python demo.py --model_path /path/to/checkpoint.pt \ --image_folder /path/to/images/ --camera_num_iterations 1 --camera_num_iterations defaults to 4; setting it to 1 skips three refinement passes in the camera head (and shrinks its KV cache by 4×). 🎥 Offline Rendering Pipeline (demo_render/batch_demo.py) Use this pipeline when your sequence is too long for the interactive viser viewer — for example, the indoor walkthrough featured above. demo_render/batch_demo.py is the all-in-one offline entry point: feed it a video or a folder of images and it will run model inference and produce a headless point-cloud flythrough MP4 in a single command. It shares the same PyTorch / FlashInfer / checkpoint stack as demo.py. For those constrained by limited VRAM or GPU usage, you may also refer to the implementation at: https://github.com/ureeey/lingbot-map-rtx4060-8g/commit/eeee84a89cc97c1e39b736b46df4ee315275700b Install (extends the main install) 1. Rendering Python dependencies pip install -e ".[vis,render]" render pulls in open3d>=0.19 and pyyaml (the core numpy<2 constraint comes from the base lingbot-map install). Sky masking in this pipeline uses onnxruntime-gpu for batched segmentation; install it if you don't already have the CPU onnxruntime: pip install onnxruntime-gpu 2. Kaolin — matches the PyTorch 2.8.0 + CUDA 12.8 recommended above: pip install --index-url https://pypi.org/simple \ kaolin -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.8.0_cu128.html --index-url https://pypi.org/simple bypasses any internal mirror that might otherwise serve the PyPI placeholder wheel (which raises ImportError on import). NVIDIA Kaolin does not publish prebuilt wheels for PyTorch 2.9.x — if you're on 2.9 for other reasons, build Kaolin from source (pip install --no-build-isolation git+https://github.com/NVIDIAGameWorks/kaolin.git, needs local CUDA toolkit). For other torch/CUDA combinations see NVIDIA Kaolin installation. 3. ffmpeg sudo apt install ffmpeg # or: brew install ffmpeg 4. CUDA extensions (required before first run) cd demo_render/render_cuda_ext && python setup.py build_ext --inplace && cd ../.. This builds voxel_morton_ext and frustum_cull_ext in place — both are imported by rgbd_render for GPU voxelization and frustum culling. Worked Example — long indoor walkthrough (~25 000 frames, 13 minutes) Dataset: Download the example video from robbyant/lingbot-map-demo on Hugging Face. python demo_render/batch_demo.py \ --video_path /data/demo_videos/indoor_travel.MP4 \ --output_folder /data/outputs/indoor_travel/ \ --model_path /path/to/lingbot-map.pt \ --config demo_render/config/indoor.yaml \ --mode windowed --window_size 128 \ --keyframe_interval 10 --overlap_keyframes 8 \ --sky_mask_dir /data/outputs/sky_masks \ --sky_mask_visualization_dir /data/outputs/sky_mask_viz \ --camera_vis default --keyframes_only_points \ --frame_tag --frame_tag_position top_right \ --save_predictions Flag-by-flag rationale: Flag Why it's there --mode windowed --window_size 128 Sliding-window inference is required once the sequence exceeds the ~320-frame RoPE training range; each window resets the KV cache. window_size counts KV-cache slots, not actual frames — the first num_scale_frames (=8) slots hold the scale frames and the remaining 128 − 8 = 120 slots hold keyframes. With keyframe_interval = 13, one window therefore covers 8 + 120 × 13 = 1568 actual frames. --keyframe_interval 10 Cache only every 10th frame as a keyframe. Non-keyframes still emit per-frame predictions but don't grow the KV cache --overlap_keyframes 8 Adjacent windows share 8 keyframes of context, resolved internally to max(num_scale_frames, 8 × keyframe_interval) = 8 × 13 = 104 actual frames of overlap. Recommended whenever keyframe_interval > 1, to keep cross-window pose alignment stable. --config demo_render/config/indoor.yaml Seed render/scene/camera/overlay defaults from the indoor preset (short depth, tighter follow cam). Any CLI flag the user explicitly passes still overrides the YAML value. --sky_mask_dir / --sky_mask_visualization_dir Persist sky masks and their side-by-side visualizations to disk so subsequent reruns reuse them instead of re-running ONNX segmentation. (The render pipeline only consumes them when sky masking is enabled — by the YAML preset or by --mask_sky.) --camera_vis default Overlay the trajectory trail + recent-frame points on the rendered video. --keyframes_only_points Only unproject keyframe depth into the point cloud; non-keyframes still contribute their pose to the trajectory/frustum overlay. Keeps the cloud sparse for very long sequences. --frame_tag --frame_tag_position top_right Stamp a <i> / <N> Frames counter in the top-right corner of the MP4. --save_predictions Persist per-frame NPZs alongside the MP4. Useful for inspection or for re-rendering with different camera/overlay settings later. Replacing keyframe_interval = 10 with image_stride = 10 speeds up rendering. Then, comment out the camera follow section in demo_render/config/indoor.yaml and set the birdeye's ranges to [2000, 2500] to reproduce the indoor fly-through effect shown in the demo: Worked Example — outdoor drive scene Dataset: Download the example video from robbyant/lingbot-map-demo on Hugging Face. python demo_render/batch_demo.py \ --video_path /data/demo_videos/drive_frames.mp4 \ --output_folder /data/outputs/drive/ \ --model_path /path/to/lingbot-map.pt \ --config demo_render/config/outdoor_drive.yaml \ --mode windowed --window_size 128 \ --max_non_keyframe_gap 100 --overlap_keyframes 8 \ --image_stride 1 \ --sky_mask_dir /data/outputs/sky_masks \ --sky_mask_visualization_dir /data/outputs/sky_mask_viz \ --camera_vis default --keyframes_only_points \ --frame_tag --frame_tag_position top_right \ --save_predictions What differs from the indoor walkthrough above: Flag Why it's there --config demo_render/config/outdoor_drive.yaml Seed defaults from the outdoor preset: sky masking enabled, deeper render range (max_depth: 250), and a follow cam tuned for vehicle trajectories with a final birdeye reveal. --image_stride 1 Use every video frame. Increase it to subsample long or high-FPS drive footage. --max_non_keyframe_gap 100 Upper bound on consecutive non-keyframes before a keyframe is forced. Only active with flow-based keyframe selection (--flow_threshold > 0); in the default fixed-interval mode it has no effect. The remaining flags (--mode windowed --window_size 128, --overlap_keyframes 8, sky-mask caching, overlays, --save_predictions) carry over unchanged from the indoor example — see the flag-by-flag table above. Worked Example — LingBot-World scenes Reconstruct videos generated by LingBot-World, our world model — the same pipeline works on generated footage out of the box. Dataset: Download the example videos (lingbo_world_frames.mp4, lingbo_world2_frames.mp4) from robbyant/lingbot-map-demo on Hugging Face. python demo_render/batch_demo.py \ --video_path /data/demo_videos/lingbo_world_frames.mp4 \ --output_folder /data/outputs/lingbo_world/ \ --model_path /path/to/lingbot-map.pt \ --config demo_render/config/outdoor_drive.yaml \ --mode windowed --window_size 128 \ --max_non_keyframe_gap 100 --overlap_keyframes 8 \ --image_stride 1 \ --sky_mask_dir /data/outputs/sky_masks \ --sky_mask_visualization_dir /data/outputs/sky_mask_viz \ --camera_vis default --keyframes_only_points \ --frame_tag --frame_tag_position top_right \ --save_predictions For the second clip, run the same command with --video_path /data/demo_videos/lingbo_world2_frames.mp4 --output_folder /data/outputs/lingbo_world2/ (and separate --sky_mask_dir / --sky_mask_visualization_dir folders if you want to keep the cached masks apart). All flags are identical to the outdoor drive scene above — only the input video and output folder change. See the drive scene and indoor walkthrough tables for the flag-by-flag rationale. Camera Path (YAML) The virtual camera path is described by the camera.segments list in the YAML preset passed via --config. Edit the YAML to design your own shot — no need to touch CLI flags. Built-in presets live in demo_render/config/: default.yaml, indoor.yaml, outdoor_drive.yaml. Copy one and edit the camera: block. YAML structure camera: fov: 60.0 # camera field of view in degrees transition: 30 # frames blended between adjacent segments segments: - mode: follow # chase cam following the input trajectory frames: [0, 1500] # rendered-frame range this segment covers (-1 = end) back_offset: 0.3 # how far behind the input camera (fraction of scene scale) up_offset: 0.08 # vertical lift above the input camera look_offset: 0.4 # how far ahead the lookat target points smooth_window: 30 # trajectory smoothing window in frames - mode: birdeye # rise up for a top-down reveal of the whole scene frames: [1500, 1800] reveal_height_mult: 2.5 # birdeye height = scene scale × this factor - mode: follow # drop back into chase cam frames: [1800, -1] back_offset: 0.3 up_offset: 0.08 look_offset: 0.4 transition controls how many frames are blended between adjacent segments; frames: [0, -1] means "the whole sequence". Available modes mode Behavior Tunable fields follow Chase cam tracks the input trajectory with smooth offsets. The most cinematic option for walkthroughs. back_offset, up_offset, look_offset, smooth_window, scale_frames birdeye Top-down reveal of the whole scene. Useful for hero / overview shots. reveal_height_mult static Fixed eye + lookat, auto-derived from the segment's start frame. — pivot Fixed eye, lookat sweeps along the trajectory. — Single-shot YAML examples Pure follow (most common): camera: fov: 60.0 segments: - mode: follow frames: [0, -1] back_offset: 0.3 up_offset: 0.08 look_offset: 0.4 smooth_window: 30 Full birdeye (good for overview / hero shots): camera: fov: 60.0 segments: - mode: birdeye frames: [0, -1] reveal_height_mult: 2.5 Follow with birdeye inserts: just list multiple segments in order under segments: — adjacent segments are interpolated using transition frames. Caveat: when --config loads a YAML preset, passing any segment-shaping CLI flag (--camera_mode, --back_offset, --up_offset, --look_offset, --smooth_window, --follow_scale_frames, --birdeye_start, --birdeye_duration, --reveal_height_mult) discards the YAML's segments and rebuilds the camera path from those flags instead. To stay fully YAML-driven, don't pass any of them on the command line. Output files For a given output name (e.g. <scene> or <video_name>): File Description <name>_pointcloud.mp4 Rendered point-cloud flythrough <name>_pointcloud_rgb.mp4 Original RGB frames encoded as video <name>_pointcloud_config.yaml Full config snapshot of this run batch_results.json Per-scene success / duration summary 📜 License This project is released under the Apache License 2.0. See LICENSE file for details. 📖 Citation @article{chen2026geometric, title={Geometric Context Transformer for Streaming 3D Reconstruction}, author={Chen, Lin-Zhuo and Gao, Jian and Chen, Yihang and Cheng, Ka Leong and Sun, Yipengjing and Hu, Liangxiao and Xue, Nan and Zhu, Xing and Shen, Yujun and Yao, Yao and Xu, Yinghao}, journal={arXiv preprint arXiv:2604.14141}, year={2026} } ✨ Acknowledgments We thank Shangzhan Zhang, Jianyuan Wang, Yudong Jin, Christian Rupprecht, and Xun Cao for their helpful discussions and support. This work builds upon several excellent open-source projects: VGGT DINOv2 Flashinfer
Apache Ossie 是一个由 Apache 软件基金会托管的开源项目，致力于制定厂商中立的语义模型交换标准（前身为 Open Semantic Interchange），通过统一的 JSON/YAML 规范解决数据分析和 AI 生态中的语义碎片化问题。
Apache 软件基金会推出了 Apache Ossie 开源项目，旨在建立一套厂商中立的语义模型规范，让数据分析和 AI 工具之间能够无缝交换一致的语义定义。该项目通过统一的 JSON 和 YAML 格式规范，确保同一个 KPI 在不同 AI 代理、BI 平台和工具中保持定义一致。Ossie 提供核心规范、参考转换器、TPC-DS 示例模型和验证工具四个组成部分，并依托 GitHub Discussions 和 Slack 进行社区协作。
ibelick 发布了 ui-skills 开源项目，为设计工程师提供一套 CLI 工具，通过 npx ui-skills 命令将 AI 代理路由到适合特定任务的 UI 技能集。
ibelick 在 GitHub 上发布了 ui-skills 项目，这是一个面向设计工程师的技能工具集。该项目提供 npx ui-skills CLI 工具，支持 start、categories、list、get 等子命令，可将 AI 代理路由到适合特定任务的 UI 技能集。项目以 MIT 许可证开源，并配套有 ui-skills.com 网站提供更多信息。
G0DM0D3 是一个完全开源的隐私透明多模型聊天界面，支持 OpenRouter、Venice 和本地模型，集成了平行提示攻击（GODMODE CLASSIC）、多模型评估（ULTRAPLINIAN）、输入扰动（Parseltongue）和自适应采样（AutoTune）等红队研究工具。
elder-plinius（Pliny the Prompter）在 GitHub 上发布了 G0DM0D3 项目，这是一个完全开源的多模型聊天界面，旨在为红队测试、认知研究和去约束 AI 交互提供工具。该界面以单文件 index.html 形式提供，无需构建步骤，支持 OpenRouter 和 Venice 等多个模型提供商，也可连接本地 Ollama、LM Studio 等 OpenAI 兼容服务器。其核心功能包括五种模型提示组合并行竞速的 GODMODE CLASSIC、支持最多 60 个模型并行评估和复合评分的 ULTRAPLINIAN、提供 33 种输入扰动技术的 Parseltongue，以及覆盖 20 种查询上下文的自动采样参数引擎 AutoTune。项目采用 AGPL-3.0 许可证，数据流动和遥测策略在文档中透明公开。
HenryNdubuaku 发布开源教科书《Maths, CS & AI Compendium》，涵盖数学、计算机科学和人工智能共18章内容，并内置 MCP 服务器供 AI 编程助手作为知识库使用。
Henry Ndubuaku 于 2026 年在 GitHub 上发布了开源教科书《Maths, CS & AI Compendium》，覆盖从向量矩阵基础到多模态学习、自主系统和 GPU 编程等 18 个章节。该仓库内置 MCP 服务器，允许 Claude Code、Cursor、VS Code 等 AI 工具将其作为知识库查询。作者称早期笔记曾帮助朋友准备 DeepMind、OpenAI、Nvidia 的技术面试并成功入职，作者本人也于 2025 年入选 Y Combinator。
Nutlope/hallmark 是一个面向 AI 编程工具（Claude Code、Cursor、Codex）的设计技能，内置 20 套主题和 57 项反"AI 味"检测门，通过预输出自我批评机制生成不具 AI 痕迹的独特 UI 设计。
Together AI 发布了 Nutlope/hallmark，一个专为 Claude Code、Cursor 和 Codex 等 AI 编码助手开发的开源设计技能。该工具包含 20 套主题和四条核心指令（build、audit、redesign、study），每次输出前经过 57 项反套路检测和预输出自我批评，拒绝 LLM 默认的统计分布式输出，确保不同页面呈现独特的宏观结构和视觉风格。项目以 MIT 许可证在 GitHub 上开源，可通过 npx skills add nutlope/hallmark 安装。