编程语言接入_add-lang

以下为本文档的中文说明Add Lang 是一个专门用于为 CodeGraph 代码分析系统添加新编程语言支持的端到端自动化工具。它的工作流程完整覆盖了从语法接入到质量验证的全过程首先将 tree-sitter 语法接入 CodeGraph 的提取管道然后编写相应的测试用例最后通过基准测试评估提取质量和检索价值。使用场景非常明确每当需要为 CodeGraph 增加新的语言支持时触发典型目标语言包括 Lua、Elixir、Zig、OCaml 等。用户可以通过运行 /add-lang 命令来启动整个过程。该技能的核心特点在于高度自动化和质量导向。整个流程完全自主运行自动选择测试仓库、运行基准测试、更新文档并生成报告但严格遵守不提交、不推送、不发布的原则所有变更留待用户审阅。设计原则强调数据驱动的验证必须证明新语言支持在提取真实符号方面优于不使用 CodeGraph 的基准线确保每次添加都有明确的价值提升。对于维护代码分析工具的团队来说这显著降低了语言扩展的技术门槛使非专业人员也能为系统添加新的语言支持。该技能的价值不仅体现在直接的功能实现上还在于它与现有技术生态系统的良好兼容性和集成能力。无论是作为独立工具使用还是嵌入到更大的工作流中它都能发Add a language to CodeGraphWire a new tree-sitter language into codegraph’s extraction pipeline, prove itextracts real symbols on popular repos, and prove it beats no-codegraph for anagent. Runsfully autonomously— pick repos, benchmark, update docs, thenreport.Never commit, push, publish, or tag(house rule); leave all changesfor the user to review.The argument is the language token used throughout theLanguageunion, e.g.lua,elixir,zig. If none was given, ask which language. Use the lowercasesingle-token form everywhere (csharp, notc#).PrerequisitesRun from the codegraph repo root.node,git,gh, and a logged-inclaudeCLI (the benchmark spawns realclaude -pruns).The benchmark uses the local dev build — Step 8 builds links it on PATH.WorkflowCopy this checklist and work through it in order:- [ ] 1. Resolve language; bail early if already supported (just benchmark) - [ ] 2. Find a grammar health-check it (ABI / heap corruption) - [ ] 3. Discover the grammars AST node types (dump-ast.mjs) - [ ] 4. Wire the language (4 files; sometimes a 5th core touch) - [ ] 5. Build verify-extraction loop until PASS - [ ] 6. Add extraction tests; make them green - [ ] 7. Auto-pick 3 popular repos by size tier; add to corpus.json - [ ] 8. Benchmark all 3: extraction with/without A/B - [ ] 9. Update README CHANGELOG - [ ] 10. Report; do NOT commitStep 1 — Resolve short-circuitCheck whether the language is already wired: look for the token in theLANGUAGESconst (src/types.ts) and theEXTRACTORSmap(src/extraction/languages/index.ts). If it is already supported (e.g.typescript,rust),skip Steps 2–6and go straight to benchmarking(Steps 7–8) to validate/measure it — note in the report that no code changed.Step 2 — Find a grammar, then health-check itlsnode_modules/tree-sitter-wasms/out/|grep-ilang# csharp - c_sharpPresent→ likely off-the-shelf;grammars.tsresolves it fromtree-sitter-wasmsautomatically. (Many languages: elixir, zig, ocaml,solidity, toml, yaml, …)Absent→ vendor a.wasmintosrc/extraction/wasm/(likepascal/scala/lua) and add the token to the vendored branch in Step 4.Always health-check before writing an extractor — apresentgrammar canstill be unusable:nodescripts/add-lang/check-grammar.mjslangpath/to/valid-sample.extIt prints the grammar’s ABI version and parses a valid sample many times in amulti-grammar runtime. If itFAILs(ERROR trees on valid code — an old ABIcorrupting the shared WASM heap, which silently drops nested calls/imports onevery file after the first; e.g. the tree-sitter-wasmsLuagrammar is ABI 13and fails), do NOT use that wasm.Vendor a newer (ABI 14/15) build instead:npmpack tree-sitter-grammars/tree-sitter-lang# often ships a prebuilt *.wasm# or build one: npx tree-sitter build --wasm (needs Docker/emscripten)cpthe.wasm src/extraction/wasm/tree-sitter-lang.wasmthen add the token to the vendored branch in Step 4 and re-run check-grammar onthe vendored path until it PASSes.If you cannot obtain a healthy wasm, STOPand tell the user.Step 3 — Discover AST node typesGet a representative source file (write a small sample covering functions,classes/structs, imports, enums; orcurla raw file from a known repo), then:nodescripts/add-lang/dump-ast.mjslangpath/to/sample.ext# vendored grammar: pass the wasm path instead of the tokennodescripts/add-lang/dump-ast.mjs src/extraction/wasm/tree-sitter-lang.wasm sample.extThe frequency table field names (name:,parameters:,body:,return_type:) tell you what to map. Open the existing extractor closest to thelanguage’s paradigm as a model:rust.ts/scala.ts(functional, traits),java.ts/csharp.ts(OO),python.ts/ruby.ts(scripting),go.ts(top-level methods receivers).Step 4 — Wire the language (4 files)These are exact, fragile wiring — match the existing style precisely:src/types.ts— TWO edits:addlang,to theLANGUAGESconst (beforeunknown);add**/*.ext,toDEFAULT_CONFIG.include.Don’t skip this— it’sthe file-scan allowlist; without the glob,codegraph initfinds0fileseven though detection/extraction are wired.src/extraction/grammars.ts— three maps:WASM_GRAMMAR_FILES:lang: tree-sitter-lang.wasm,EXTENSION_MAP: each file extension →lang(e.g..lua: lua,)getLanguageDisplayName:lang: Display Name,vendored only: addlangto the(lang pascal || lang scala || …)wasm-path branch.src/extraction/languages/lang.ts— new file exportingexport const langExtractor: LanguageExtractor { … }. Map the node typesfrom Step 3. Required fields:functionTypes,classTypes,methodTypes,interfaceTypes,structTypes,enumTypes,typeAliasTypes,importTypes,callTypes,variableTypes,nameField,bodyField,paramsField. Add hooks as the grammar needs them (getSignature,getVisibility,isExported,extractImport,visitNode,getReceiverType,interfaceKind,enumMemberTypes, etc. — seesrc/extraction/tree-sitter-types.ts).src/extraction/languages/index.ts—import { langExtractor } from ./lang;and addlang: langExtractor,toEXTRACTORS.Sometimes a 5th, core touch insrc/extraction/tree-sitter.ts— variableextraction has per-language branches inextractVariable(the generic fallbackonly finds directidentifier/variable_declaratorchildren). If the grammarnests declared names (e.g. Lua’svariable_declaration → variable_list), add a} else if (this.language lang)branch there, mirroring the existingts/python/go ones. Import forms that aren’t a distinct node (Lua/Rubyrequireis acall) are handled in the extractor’svisitNodehook instead.Step 5 — Build verify loopnpmrun build# tsc copy-assets (copies any vendored *.wasm into dist/)Index a small sample repo and check extraction:(cdsample-repocodegraph init-i)nodescripts/add-lang/verify-extraction.mjssample-repolangverify-extraction.mjsfails (exit 1) if the language isn’t detected or onlyfile/importnodes were produced — the classic symptom of wrong node-typenames. On FAIL or a thin WARN: re-rundump-ast.mjson a richer file, fix themappings inlang.ts,npm run build, re-index, re-verify.Repeat untilPASS.Step 6 — TestsAdd to__tests__/extraction.test.ts, modeled on theRust Extractionblock:adetectLanguageassertion indescribe(Language Detection)adescribe(Lang Extraction)block asserting functions/classes/importsare extracted from an inline source string.npx vitest run __tests__/extraction.test.tsGreen before continuing.Step 7 — Auto-pick 3 repos corpusPickwithout asking. Find candidates, then curate 3 that are genuinelylang-dominant, one per size tier:gh search repos--languagelang--sortstars--limit40\\--jsonfullName,stargazerCount,descriptionTiers (matchcorpus.json):Small~150 files ·Medium~150–1500 ·Large~1500. Skip repos that are taggedlangbut mostly anotherlanguage. Write one cross-file architecturequestionper repo (the kind thatneeds tracing across files). Add aLanguageblock to.claude/skills/agent-eval/corpus.json(fields:name,repo,size,files,question) so/agent-evalcan reuse them.Step 8 — Benchmark all 3 (extraction A/B)Make the dev build the codegraph on PATHonce, then loop:npmrun build./scripts/local-install.sh scripts/add-lang/bench.shlangnameurlquestionheadless# ×3bench.shclones (shared/tmp/codegraph-corpus), wipes indexes, runsverify-extraction.mjs, then the with/without retrieval A/B viascripts/agent-eval/run-all.sh(skips the paid A/B if extraction is broken).Read eachparse-run.mjssummary printed byrun-all.sh: tool calls, fileReads, Grep/Bash, codegraph-tool calls, duration, andcost— for both thewithandwithoutarms. After the loop, restore the dev link if needed:./scripts/local-install.sh.Step 9 — Docs CHANGELOGREADME.md: addLangto the “19 Languages” feature bullet, and add arow to theSupported Languagestable:| Lang | \\.ext\| Full support (classes, methods, …) |.CHANGELOG.md: add an## [Unreleased]section at the top (above thelatest version) with### Added→ a user-perspective bullet, e.g.“CodeGraph now indexes (.ext) — functions, classes, imports, andcall edges.”If## [Unreleased]already exists, append under it. (It’sfolded into the next versioned block at release time.)Step 10 — Report (do NOT commit)Summarize for review:Files changed: the 4 wiring edits new extractor tests README CHANGELOG corpus.json ( any vendored.wasm).Extractionper repo: files / nodes / edges /verify-extractionresult.A/Bper repo:withvswithout(tool calls, file Reads, cost) and aone-line verdict — did codegraph reduce effort, and did both arms reach acorrect answer?Gaps / follow-ups(node types not yet mapped, resolution edges missing,framework routes, etc.).Hand the changes to the user.Do notrungit commit/pushor publish —releases go through the GitHub Actions Release workflow.NotesThe A/B spawns realpaidclaude -pruns (opus,--max-budget-usd),2 arms × 3 repos. The corpus dir/tmp/codegraph-corpusis shared with/agent-eval, so clones are reused across runs.Any new*.wasmmust live insrc/extraction/wasm/—copy-assets(run bynpm run build) ships it; otherwise it won’t be indist/.An index must be served by thesamebinary that built it. Step 8 builds links the dev build first, so this holds.If a grammar can’t be obtained, or extraction can’t reach PASS,STOP andreport— don’t ship a half-wired language.