新增: AST 代码智能 read_symbol 三态(治 prompt 爆,信息密度驱动)
tree-sitter 语法层符号解析,治 aichat read_file 全文回灌 prompt 爆(e46f5605 360K/8dfe0b94 5M)。 核心思想:信息密度≠压缩,read_symbol 按语义层级返符号骨架/全文,非物理读全文件。 - code_intel.rs: grammar_for 集中 lookup(Rust/TS/JS/Vue借TS,3 grammar 4 类)+ read_symbol 三态(骨架默认/全文)+ 不报错兜底(无grammar/解析失败/未找到→grep提示) - tool_registry 注册 read_symbol(对齐 read_file handler 闭包模式) - Cargo.toml 加 tree-sitter 0.25 + rust/typescript/javascript grammar - AST 设计文档 grammar 策略章节定稿(静态编译+lookup,不动态不trait) 实测: code_intel.rs 全文 21851B vs read_symbol 骨架 897B,降 24.4x(达设计目标一个量级+)。单测 14 全过。
This commit is contained in:
61
Cargo.lock
generated
61
Cargo.lock
generated
@@ -859,6 +859,10 @@ dependencies = [
|
||||
"tauri-plugin-window-state",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tree-sitter",
|
||||
"tree-sitter-javascript",
|
||||
"tree-sitter-rust",
|
||||
"tree-sitter-typescript",
|
||||
"uuid",
|
||||
]
|
||||
|
||||
@@ -3877,6 +3881,7 @@ version = "1.0.150"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9"
|
||||
dependencies = [
|
||||
"indexmap 2.14.0",
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
@@ -4140,6 +4145,12 @@ version = "1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
|
||||
|
||||
[[package]]
|
||||
name = "streaming-iterator"
|
||||
version = "0.1.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2b2231b7c3057d5e4ad0156fb3dc807d900806020c5ffa3ee6ff2c8c76fb8520"
|
||||
|
||||
[[package]]
|
||||
name = "string_cache"
|
||||
version = "0.9.0"
|
||||
@@ -5097,6 +5108,56 @@ dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter"
|
||||
version = "0.25.10"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "78f873475d258561b06f1c595d93308a7ed124d9977cb26b148c2084a4a3cc87"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"regex",
|
||||
"regex-syntax",
|
||||
"serde_json",
|
||||
"streaming-iterator",
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-javascript"
|
||||
version = "0.25.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "68204f2abc0627a90bdf06e605f5c470aa26fdcb2081ea553a04bdad756693f5"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-language"
|
||||
version = "0.1.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "009994f150cc0cd50ff54917d5bc8bffe8cad10ca10d81c34da2ec421ae61782"
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-rust"
|
||||
version = "0.24.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "439e577dbe07423ec2582ac62c7531120dbfccfa6e5f92406f93dd271a120e45"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-typescript"
|
||||
version = "0.23.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6c5f76ed8d947a75cc446d5fccd8b602ebf0cde64ccf2ffa434d873d7a575eff"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "try-lock"
|
||||
version = "0.2.5"
|
||||
|
||||
@@ -83,10 +83,30 @@ AST 是语法层(parse tree)。**语法层能**:
|
||||
|
||||
### 选型:tree-sitter
|
||||
- GitHub 出品,增量 AST 解析,几十种语言 grammar
|
||||
- Rust 生态 `tree-sitter` + `tree-sitter-rust`/`tree-sitter-typescript` 等(编译期 crate)
|
||||
- **按语言按需 enable grammar**(非全装,控体积)
|
||||
- Rust 生态 `tree-sitter` + 各语言 grammar crate(编译期静态链接)
|
||||
- 增量解析:文件改只重解变更(<1ms)
|
||||
|
||||
### grammar 策略(2026-06-24 定稿)
|
||||
|
||||
**静态编译 + 集中 lookup**(架构成本最低,详见 [插件机制设计](插件机制-设计-2026-06-24.md)):
|
||||
|
||||
- **不做**动态加载(grammar 编共享库运行期 dlopen)、**不抽** trait(YAGNI,无第二实现不抽象)
|
||||
- grammar 编译期静态链接进二进制(无 ABI 坑 / 无运行时状态 / 无分发负担)
|
||||
- grammar 获取集中到一个普通函数 `grammar_for(ext) -> Language`(未来加动态只改这一点,不返工调用方)
|
||||
|
||||
**Phase1 范围**(覆盖 devflow 源码 + 主流用户项目):
|
||||
|
||||
| 文件类型 | grammar | 备注 |
|
||||
|---|---|---|
|
||||
| `.rs` | tree-sitter-rust | 后端主体 |
|
||||
| `.ts`/`.tsx` | tree-sitter-typescript | 前端 script |
|
||||
| `.js`/`.jsx` | tree-sitter-javascript | |
|
||||
| `.vue` | **借 tree-sitter-typescript** | 切 `<script>` 段喂 ts;template/style 不做符号(Vue 逻辑在 script) |
|
||||
|
||||
**3 个 grammar 覆盖 4 类文件**。Vue 借 TS 省 grammar + 避 SFC 嵌套复杂度。
|
||||
|
||||
**扩展**:加标准语言 = 加 grammar crate + 一张节点映射表(表驱动,~30 行/语言)。小众(C++/Kotlin)按需;动态加载/集成插件化见插件机制设计文档。
|
||||
|
||||
### 三态读取(read_symbol,密度驱动)
|
||||
|
||||
read_symbol **不按物理行返、不截断**,按**语义结构层级**返(消解"截断多少行"争论):
|
||||
|
||||
@@ -57,6 +57,15 @@ keyring = { workspace = true }
|
||||
# 复用 df-ai 同款 reqwest 0.12(同版本锁定,避免双 TLS 后端)。默认 native-tls(与 df-ai 一致),
|
||||
# 启用 json(响应解析)/gzip/brotli(透明解压,常见 API 必备)。重定向手动接管(见 http.rs SSRF)。
|
||||
reqwest = { version = "0.12", features = ["json", "gzip", "brotli"] }
|
||||
# AST 代码智能(read_symbol 三态,信息密度驱动,见 docs/02-架构设计/专项设计/AST符号解析-设计-2026-06-24.md):
|
||||
# 治 aichat read_file 全文回灌 prompt 爆(e46f5605 360K/8dfe0b94 5M)。tree-sitter 语法层精准提取
|
||||
# 符号骨架/下钻/全文,替代物理读全文件。静态编译 + 集中 grammar_for(ext) lookup(不动态加载/不抽 trait,YAGNI)。
|
||||
# Phase1 范围:Rust(.rs)/TypeScript(.ts/.tsx)/JavaScript(.js/.jsx)/Vue(.vue 借 TS 切 script 段),3 grammar 覆盖 4 类。
|
||||
# 兼容:tree-sitter 0.25 主 crate 与 grammar crate 配对(各 grammar 声明 tree-sitter 依赖,0.25 系广泛兼容)。
|
||||
tree-sitter = "0.25"
|
||||
tree-sitter-rust = "0.24.2"
|
||||
tree-sitter-typescript = "0.23.2"
|
||||
tree-sitter-javascript = "0.25.0"
|
||||
|
||||
# 阶段3a/3b 统一审批模型开关(path_auth 审批链扎实重构 plan 阶段3)。
|
||||
# 3a 后端合并(单 HashMap + kind 字段)为编译期结构性变更,本 flag 用于文档标记 +
|
||||
|
||||
541
src-tauri/src/commands/ai/code_intel.rs
Normal file
541
src-tauri/src/commands/ai/code_intel.rs
Normal file
@@ -0,0 +1,541 @@
|
||||
//! AST 代码智能 — tree-sitter 符号解析(read_symbol 三态)
|
||||
//!
|
||||
//! 治 aichat read_file 全文回灌 prompt 爆(信息密度驱动 ≠ 压缩,见设计文档):
|
||||
//! read_symbol 按语义层级返符号 **骨架**(默认)/ **全文**(full),替代物理读全文件。
|
||||
//! LLM 默认拿骨架(签名 + 行范围 + 内部调用)→ 要细节再 full 取完整定义体 → 进 prompt 的都是主动要的,天然高密度。
|
||||
//! 设计:docs/02-架构设计/专项设计/AST符号解析-设计-2026-06-24.md
|
||||
//!
|
||||
//! **不报错底线**:无 grammar / 解析失败 / symbol 未找到 / Vue 无 script / 任何大小函数 →
|
||||
//! 返兜底提示(grep + read_file),不 Err / 不 panic / 不阻断。
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use tree_sitter::{Language, Node, Parser};
|
||||
|
||||
/// Phase1 支持的定义节点 kind(Rust + TS/JS 通用,Vue 借 TS)。
|
||||
///
|
||||
/// tree-sitter 各 grammar 的定义节点 kind 基本一致:
|
||||
/// - Rust:function_item / struct_item / impl_item 等(`_item` 后缀)
|
||||
/// - TS/JS:function_declaration / class_declaration / interface_declaration / method_definition 等
|
||||
/// 统一放一张表(find_definition 命中即提取),跨语言零分支。
|
||||
const DEFINITION_KINDS: &[&str] = &[
|
||||
// Rust
|
||||
"function_item",
|
||||
"function_signature_item",
|
||||
"struct_item",
|
||||
"enum_item",
|
||||
"impl_item",
|
||||
"trait_item",
|
||||
"mod_item",
|
||||
"const_item",
|
||||
"static_item",
|
||||
"type_item",
|
||||
"macro_definition",
|
||||
// TS / JS(与 Rust kind 不冲突)
|
||||
"function_declaration",
|
||||
"generator_function_declaration",
|
||||
"class_declaration",
|
||||
"abstract_class_declaration",
|
||||
"interface_declaration",
|
||||
"enum_declaration",
|
||||
"method_definition",
|
||||
];
|
||||
|
||||
/// 文件后缀 → tree-sitter grammar 集中 lookup(静态编译,不动态加载 / 不抽 trait)。
|
||||
///
|
||||
/// 设计(grammar 策略 2026-06-24 定稿,详见插件机制设计文档):
|
||||
/// **3 个 grammar 覆盖 4 类文件**,架构成本最低。
|
||||
/// - `.rs` → tree-sitter-rust
|
||||
/// - `.ts` → tree-sitter-typescript(language_typescript)
|
||||
/// - `.tsx` → tree-sitter-typescript(language_tsx)
|
||||
/// - `.js` / `.jsx` → tree-sitter-javascript
|
||||
/// - `.vue` → 返回 None 标记"需切 script 段"(Vue 借 TS,逻辑在 script,read_symbol 内 extract_vue_script 切段喂 ts)
|
||||
///
|
||||
/// 未来加语言 = 此处加一行 + 可能补 DEFINITION_KINDS(表驱动,~几行),不返工调用方。
|
||||
/// tree-sitter 动态加载技术上可行(Helix/Neovim 先例)但跨平台分发 + ABI 坑,当前不值(见插件设计)。
|
||||
pub fn grammar_for(ext: &str) -> Option<Language> {
|
||||
match ext {
|
||||
"rs" => Some(Language::from(tree_sitter_rust::LANGUAGE)),
|
||||
"ts" => Some(Language::from(tree_sitter_typescript::LANGUAGE_TYPESCRIPT)),
|
||||
"tsx" => Some(Language::from(tree_sitter_typescript::LANGUAGE_TSX)),
|
||||
"js" | "jsx" => Some(Language::from(tree_sitter_javascript::LANGUAGE)),
|
||||
// Vue 借 TS:返 None 标记需切 script 段(read_symbol 内 extract_vue_script + 喂 ts)
|
||||
"vue" => None,
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// 判断 ext 是否为 Phase1 支持语言(含 Vue 借 TS)。
|
||||
///
|
||||
/// 预留:当前 handler 直接调 read_symbol(内部已兜底 ext 判断),本函数供测试 + 未来
|
||||
/// handler 层提前判 ext 省一次 fs read(不支持直接返兜底不读文件)。测试消费,非真死代码。
|
||||
#[allow(dead_code)]
|
||||
pub fn is_supported_ext(ext: &str) -> bool {
|
||||
matches!(ext, "rs" | "ts" | "tsx" | "js" | "jsx" | "vue")
|
||||
}
|
||||
|
||||
/// 提取 Vue SFC 的 `<script>` 段内容(借 TS 解析)。
|
||||
///
|
||||
/// Vue 单文件组件逻辑在 `<script setup lang="ts">` 内,template/style 不做符号提取。
|
||||
/// 切出 `<script>...</script>` 内容喂 tree-sitter-typescript:
|
||||
/// - 省 1 个 grammar(tree-sitter-vue 不引,降依赖体积)
|
||||
/// - 避 SFC 嵌套复杂度(tree-sitter-vue 解析 template/style/script 混合,符号提取只关心 script)
|
||||
///
|
||||
/// 多个 script 段取首个(`<script setup>` 主逻辑);无 script 返 None(走兜底)。
|
||||
/// 返回 (script 内容, script 在原 source 中的字节起始偏移)用于行号映射。
|
||||
fn extract_vue_script(source: &str) -> Option<(String, usize)> {
|
||||
let start_tag = source.find("<script")?;
|
||||
// 跳过 <script ...> 的属性(到 >),取标签体开始
|
||||
let tag_close = source[start_tag..].find('>')? + start_tag + 1;
|
||||
let end = source[tag_close..].find("</script>")?;
|
||||
Some((source[tag_close..tag_close + end].to_string(), tag_close))
|
||||
}
|
||||
|
||||
/// read_symbol 三态读取(信息密度驱动)。
|
||||
///
|
||||
/// **不按物理行返、不截断**,按语义结构层级返(消解"截断多少行"争论):
|
||||
/// - `full=false`(默认 骨架态):返签名 + kind + 行范围 + 内部调用列表 + 总行数(极小,高密度)
|
||||
/// - `full=true`(全文态):返完整定义体(语法边界 100% 准)
|
||||
///
|
||||
/// **不报错底线**:无 grammar / 解析失败 / symbol 未找到 / Vue 无 script → 返兜底提示(grep + read_file),不 panic。
|
||||
///
|
||||
/// 参数:
|
||||
/// - `source`:文件全文(由 handler 读 + 算 hash 后传入,纯函数不碰 fs)
|
||||
/// - `file_hash`:文件 hash(handler 算,回传 LLM 供后续定位 / 兜底回取)
|
||||
/// - `path`:路径(取 ext + 兜底消息展示)
|
||||
/// - `symbol`:目标符号名(精确匹配定义节点的 name field)
|
||||
/// - `full`:true=完整定义体,false=骨架
|
||||
/// - `kind_hint`:可选 kind 过滤(如 "function"/"struct"),None 匹配任意定义 kind
|
||||
pub fn read_symbol(
|
||||
source: &str,
|
||||
file_hash: &str,
|
||||
path: &str,
|
||||
symbol: &str,
|
||||
full: bool,
|
||||
kind_hint: Option<&str>,
|
||||
) -> serde_json::Value {
|
||||
let ext = path.rsplit('.').next().unwrap_or("").to_lowercase();
|
||||
let is_vue = ext == "vue";
|
||||
|
||||
// Vue 借 TS:切 script 段(Vue 逻辑在 script,template/style 不做符号)
|
||||
let (parse_source, parse_ext, line_offset): (String, &str, usize) = if is_vue {
|
||||
match extract_vue_script(source) {
|
||||
Some((script, byte_off)) => {
|
||||
// Vue 切段后节点行号是 script 内相对行,加原文件中 script 起始行偏移映射回原行号
|
||||
let off_lines = source[..byte_off].matches('\n').count();
|
||||
(script, "ts", off_lines)
|
||||
}
|
||||
None => return fallback(symbol, path, file_hash, "vue_no_script",
|
||||
"Vue 文件无 <script> 段(纯 template/style 无可提取符号)"),
|
||||
}
|
||||
} else {
|
||||
(source.to_string(), ext.as_str(), 0)
|
||||
};
|
||||
|
||||
let grammar = match grammar_for(parse_ext) {
|
||||
Some(g) => g,
|
||||
None => {
|
||||
return fallback(symbol, path, file_hash, "no_grammar",
|
||||
&format!("不支持 {} 文件符号解析(Phase1 支持 .rs/.ts/.tsx/.js/.jsx/.vue)", ext));
|
||||
}
|
||||
};
|
||||
|
||||
let mut parser = Parser::new();
|
||||
if parser.set_language(&grammar).is_err() {
|
||||
return fallback(symbol, path, file_hash, "parse_failed",
|
||||
"grammar 加载失败(tree-sitter 版本/ABI 不兼容,疑似环境问题)");
|
||||
}
|
||||
let tree = match parser.parse(&parse_source, None) {
|
||||
Some(t) => t,
|
||||
None => return fallback(symbol, path, file_hash, "parse_failed", "解析失败(源码可能有语法错误)"),
|
||||
};
|
||||
|
||||
let root = tree.root_node();
|
||||
let mut found: Option<Node> = None;
|
||||
find_definition(&root, symbol, kind_hint, parse_source.as_bytes(), &mut found);
|
||||
|
||||
match found {
|
||||
Some(node) => extract_symbol(node, parse_source.as_bytes(), source, path, file_hash, full, line_offset),
|
||||
None => fallback(symbol, path, file_hash, "not_found",
|
||||
&format!("文件内未找到符号「{}」(可能拼写/作用域差异,试 grep 搜定义)", symbol)),
|
||||
}
|
||||
}
|
||||
|
||||
/// 递归遍历找匹配 symbol 名的定义节点。
|
||||
///
|
||||
/// 命中条件:node.kind() 在 DEFINITION_KINDS 内 + name field 文本 == symbol + kind_hint(若传)匹配。
|
||||
/// 首个命中即返(find_definition 调用方短路)。
|
||||
fn find_definition<'a>(
|
||||
node: &Node<'a>,
|
||||
symbol: &str,
|
||||
kind_hint: Option<&str>,
|
||||
source: &[u8],
|
||||
found: &mut Option<Node<'a>>,
|
||||
) {
|
||||
if found.is_some() {
|
||||
return;
|
||||
}
|
||||
if DEFINITION_KINDS.contains(&node.kind()) {
|
||||
let kind_ok = kind_hint.map_or(true, |h| kind_matches(node.kind(), h));
|
||||
if kind_ok {
|
||||
if let Some(name_node) = node.child_by_field_name("name") {
|
||||
let name = node_text(name_node, source);
|
||||
if name == symbol {
|
||||
*found = Some(*node);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut cursor = node.walk();
|
||||
for child in node.children(&mut cursor) {
|
||||
find_definition(&child, symbol, kind_hint, source, found);
|
||||
if found.is_some() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// kind_hint 宽松匹配:user 传 "function" 命中 function_item/function_declaration/method_definition 等。
|
||||
fn kind_matches(node_kind: &str, hint: &str) -> bool {
|
||||
let h = hint.to_lowercase();
|
||||
node_kind.contains(&h)
|
||||
|| (h == "function" && (node_kind.contains("function") || node_kind == "method_definition"))
|
||||
|| (h == "struct" && node_kind.contains("struct"))
|
||||
|| (h == "class" && node_kind.contains("class"))
|
||||
|| (h == "enum" && node_kind.contains("enum"))
|
||||
}
|
||||
|
||||
/// 取节点文本(字节切片避免 utf8_text 的 lifetime/Result 坑)。
|
||||
/// 返 owned String(符号体不大,避免借用 lifetime 纠缠调用方)。
|
||||
fn node_text(node: Node<'_>, source: &[u8]) -> String {
|
||||
std::str::from_utf8(&source[node.start_byte()..node.end_byte()])
|
||||
.unwrap_or("")
|
||||
.to_string()
|
||||
}
|
||||
|
||||
/// 提取符号三态信息,组装返回 Value。
|
||||
fn extract_symbol(
|
||||
node: Node<'_>,
|
||||
source: &[u8],
|
||||
_orig_source: &str,
|
||||
path: &str,
|
||||
file_hash: &str,
|
||||
full: bool,
|
||||
line_offset: usize,
|
||||
) -> serde_json::Value {
|
||||
let name = node
|
||||
.child_by_field_name("name")
|
||||
.map(|n| node_text(n, source).to_string())
|
||||
.unwrap_or_default();
|
||||
let kind = node.kind().to_string();
|
||||
|
||||
// 行号:script 内相对行(0-based)+ Vue 偏移 → 原文件 1-based 行
|
||||
let start_row = node.start_position().row + 1 + line_offset;
|
||||
let end_row = node.end_position().row + 1 + line_offset;
|
||||
let body_lines = end_row - start_row + 1;
|
||||
|
||||
let node_src = node_text(node, source);
|
||||
let signature = extract_signature(&node_src);
|
||||
let calls = collect_calls(node, source);
|
||||
|
||||
if full {
|
||||
serde_json::json!({
|
||||
"symbol": name,
|
||||
"kind": kind,
|
||||
"path": path,
|
||||
"file_hash": file_hash,
|
||||
"line_start": start_row,
|
||||
"line_end": end_row,
|
||||
"body_lines": body_lines,
|
||||
"signature": signature,
|
||||
"calls": calls,
|
||||
"mode": "full",
|
||||
"content": node_src,
|
||||
})
|
||||
} else {
|
||||
serde_json::json!({
|
||||
"symbol": name,
|
||||
"kind": kind,
|
||||
"path": path,
|
||||
"file_hash": file_hash,
|
||||
"line_start": start_row,
|
||||
"line_end": end_row,
|
||||
"body_lines": body_lines,
|
||||
"signature": signature,
|
||||
"calls": calls,
|
||||
"mode": "skeleton",
|
||||
"hint": "骨架态:传 full=true 取完整定义体(语法边界精准);下钻局部分支 drill 待 Phase2",
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// 提取签名:节点文本到首个 `{`(函数/结构体体开始)或 `;`(声明)之前的部分。
|
||||
/// 无 `{`/`;` 取首行。trim 收尾空白 + 注释。
|
||||
fn extract_signature(node_src: &str) -> String {
|
||||
let cut = node_src
|
||||
.find('{')
|
||||
.or_else(|| node_src.find(';'))
|
||||
.unwrap_or_else(|| node_src.len());
|
||||
let sig = node_src[..cut].trim();
|
||||
if sig.is_empty() {
|
||||
node_src.lines().next().unwrap_or("").trim().to_string()
|
||||
} else {
|
||||
sig.to_string()
|
||||
}
|
||||
}
|
||||
|
||||
/// 收集节点内所有 call_expression 的被调用者(同文件静态调用点)。
|
||||
///
|
||||
/// Phase1:扁平收集去重(Phase2 按分支结构组织调用图)。
|
||||
/// Rust/TS/JS 的 call_expression 都有 "function" field(callee),统一提取。
|
||||
fn collect_calls(node: Node, source: &[u8]) -> Vec<String> {
|
||||
let mut calls = Vec::new();
|
||||
let mut seen = HashSet::new();
|
||||
collect_calls_inner(node, source, &mut calls, &mut seen);
|
||||
calls
|
||||
}
|
||||
|
||||
fn collect_calls_inner(node: Node, source: &[u8], out: &mut Vec<String>, seen: &mut HashSet<String>) {
|
||||
let mut cursor = node.walk();
|
||||
for child in node.children(&mut cursor) {
|
||||
if child.kind() == "call_expression" {
|
||||
if let Some(func) = child.child_by_field_name("function") {
|
||||
let name = node_text(func, source).to_string();
|
||||
if !name.is_empty() && seen.insert(name.clone()) {
|
||||
out.push(name);
|
||||
}
|
||||
}
|
||||
}
|
||||
collect_calls_inner(child, source, out, seen);
|
||||
}
|
||||
}
|
||||
|
||||
/// 兜底返回(不报错底线):任何失败路径统一返此结构,LLM 据 fallback/suggestion 回退 grep/read_file。
|
||||
fn fallback(symbol: &str, path: &str, file_hash: &str, reason: &str, suggestion: &str) -> serde_json::Value {
|
||||
serde_json::json!({
|
||||
"symbol": symbol,
|
||||
"path": path,
|
||||
"file_hash": file_hash,
|
||||
"fallback": true,
|
||||
"reason": reason,
|
||||
"suggestion": suggestion,
|
||||
"tip": "回退:用 grep(pattern=symbol) 搜定义位置 → read_file(path, offset=行号) 精准读取",
|
||||
})
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 单测:三态提取 + 兜底 + grammar lookup + Vue 切段
|
||||
// ============================================================
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
const RUST_FIXTURE: &str = r#"use std::io;
|
||||
|
||||
pub fn read_symbol(path: &str) -> Result<(), io::Error> {
|
||||
let content = load(path)?;
|
||||
parse(&content);
|
||||
validate(path);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn parse(content: &str) {
|
||||
helper(content);
|
||||
}
|
||||
|
||||
fn helper(_x: &str) {}
|
||||
|
||||
struct Config {
|
||||
name: String,
|
||||
}
|
||||
"#;
|
||||
|
||||
#[test]
|
||||
fn test_grammar_for_all_exts() {
|
||||
assert!(grammar_for("rs").is_some());
|
||||
assert!(grammar_for("ts").is_some());
|
||||
assert!(grammar_for("tsx").is_some());
|
||||
assert!(grammar_for("js").is_some());
|
||||
assert!(grammar_for("jsx").is_some());
|
||||
// Vue 借 TS:grammar 本身返 None(切段后用 ts)
|
||||
assert!(grammar_for("vue").is_none());
|
||||
// 不支持语言
|
||||
assert!(grammar_for("py").is_none());
|
||||
assert!(grammar_for("").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_is_supported_ext() {
|
||||
assert!(is_supported_ext("rs"));
|
||||
assert!(is_supported_ext("vue"));
|
||||
assert!(!is_supported_ext("py"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_read_symbol_rust_skeleton() {
|
||||
let r = read_symbol(RUST_FIXTURE, "hash1", "test.rs", "read_symbol", false, None);
|
||||
assert_eq!(r["fallback"], serde_json::Value::Null);
|
||||
assert_eq!(r["symbol"], "read_symbol");
|
||||
assert_eq!(r["kind"], "function_item");
|
||||
assert_eq!(r["mode"], "skeleton");
|
||||
// 骨架态不返 content,返 signature + calls
|
||||
assert!(r["content"].is_null());
|
||||
assert!(r["signature"].as_str().unwrap().contains("read_symbol"));
|
||||
// calls 含同文件静态调用(parse/validate)+ load(跨文件也命中名,Phase1 扁平)
|
||||
let calls = r["calls"].as_array().unwrap();
|
||||
assert!(calls.iter().any(|c| c == "parse"));
|
||||
assert!(calls.iter().any(|c| c == "validate"));
|
||||
assert_eq!(r["body_lines"].as_u64().unwrap(), 6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_read_symbol_rust_full() {
|
||||
let r = read_symbol(RUST_FIXTURE, "hash1", "test.rs", "read_symbol", true, None);
|
||||
assert_eq!(r["mode"], "full");
|
||||
let content = r["content"].as_str().unwrap();
|
||||
// full 态返完整定义体(含函数体 {})
|
||||
assert!(content.contains("pub fn read_symbol"));
|
||||
assert!(content.contains("let content = load(path)?;"));
|
||||
assert!(content.trim_end().ends_with('}'));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_read_symbol_struct() {
|
||||
let r = read_symbol(RUST_FIXTURE, "h", "test.rs", "Config", false, None);
|
||||
assert_eq!(r["symbol"], "Config");
|
||||
assert_eq!(r["kind"], "struct_item");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_read_symbol_kind_hint_filter() {
|
||||
// kind_hint="struct" 时,"read_symbol"(function)不应被误匹配为 struct
|
||||
let r = read_symbol(RUST_FIXTURE, "h", "test.rs", "Config", false, Some("struct"));
|
||||
assert_eq!(r["symbol"], "Config");
|
||||
assert_eq!(r["kind"], "struct_item");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_read_symbol_not_found_fallback() {
|
||||
let r = read_symbol(RUST_FIXTURE, "h", "test.rs", "nonexistent_fn", false, None);
|
||||
assert_eq!(r["fallback"], true);
|
||||
assert_eq!(r["reason"], "not_found");
|
||||
assert!(r["suggestion"].as_str().unwrap().contains("grep"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_read_symbol_no_grammar_fallback() {
|
||||
let src = "def hello():\n print('hi')\n";
|
||||
let r = read_symbol(src, "h", "test.py", "hello", false, None);
|
||||
assert_eq!(r["fallback"], true);
|
||||
assert_eq!(r["reason"], "no_grammar");
|
||||
assert!(r["suggestion"].as_str().unwrap().contains("不支持"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_read_symbol_vue_script_extraction() {
|
||||
let vue = r#"<template><div>{{ msg }}</div></template>
|
||||
|
||||
<script setup lang="ts">
|
||||
function greet(name: string) {
|
||||
console.log(name);
|
||||
helper();
|
||||
}
|
||||
|
||||
function helper() {}
|
||||
</script>
|
||||
|
||||
<style scoped></style>
|
||||
"#;
|
||||
let r = read_symbol(vue, "h", "App.vue", "greet", false, None);
|
||||
assert_eq!(r["fallback"], serde_json::Value::Null);
|
||||
assert_eq!(r["symbol"], "greet");
|
||||
assert_eq!(r["kind"], "function_declaration");
|
||||
// Vue 切段后行号映射回原文件(greet 在原文件第 5 行附近)
|
||||
let start = r["line_start"].as_u64().unwrap();
|
||||
assert!(start >= 4, "Vue 行号应映射回原文件(script 段偏移后),实际 {}", start);
|
||||
let calls = r["calls"].as_array().unwrap();
|
||||
assert!(calls.iter().any(|c| c == "helper"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_read_symbol_vue_no_script_fallback() {
|
||||
let vue = "<template><div>hi</div></template>\n<style></style>";
|
||||
let r = read_symbol(vue, "h", "App.vue", "greet", false, None);
|
||||
assert_eq!(r["fallback"], true);
|
||||
assert_eq!(r["reason"], "vue_no_script");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_extract_vue_script() {
|
||||
let vue = "a\nb\n<script>let x = 1;</script>";
|
||||
let (script, off) = extract_vue_script(vue).unwrap();
|
||||
assert_eq!(script, "let x = 1;");
|
||||
// script 在第 3 行(0-indexed offset 后),行偏移应 = 2(\n 数)
|
||||
assert_eq!(vue[..off].matches('\n').count(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_extract_signature() {
|
||||
// function 带 body
|
||||
let s = extract_signature("pub fn foo(x: i32) -> i32 {\n x + 1\n}");
|
||||
assert_eq!(s, "pub fn foo(x: i32) -> i32");
|
||||
// struct 带 body
|
||||
let s = extract_signature("struct Config {\n name: String,\n}");
|
||||
assert_eq!(s, "struct Config");
|
||||
// 声明以 ; 结尾
|
||||
let s = extract_signature("fn foo();");
|
||||
assert_eq!(s, "fn foo()");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_collect_calls_dedup() {
|
||||
let src = "fn a() { b(); b(); c(); }";
|
||||
let mut parser = Parser::new();
|
||||
parser.set_language(&Language::from(tree_sitter_rust::LANGUAGE)).unwrap();
|
||||
let tree = parser.parse(src, None).unwrap();
|
||||
let calls = collect_calls(tree.root_node(), src.as_bytes());
|
||||
// b 去重只出现一次
|
||||
assert!(calls.iter().any(|c| c == "b"));
|
||||
assert_eq!(calls.iter().filter(|c| *c == "b").count(), 1);
|
||||
assert!(calls.iter().any(|c| c == "c"));
|
||||
}
|
||||
|
||||
/// 多角度验证(用户要求:实现后看效果 + 多角度测试):
|
||||
/// 解析 devflow 真实源码 code_intel.rs 自身,验骨架态密度提升(设计目标:降一个量级)。
|
||||
/// 对比 read_symbol 骨架(单符号签名+calls)vs read_file 全文(整个文件所有符号)的体积。
|
||||
#[test]
|
||||
fn test_read_symbol_real_devflow_token_density() {
|
||||
let manifest = env!("CARGO_MANIFEST_DIR"); // src-tauri/
|
||||
let path_str = format!("{}/src/commands/ai/code_intel.rs", manifest);
|
||||
let source = std::fs::read_to_string(&path_str)
|
||||
.expect("应能读到 devflow 自身 code_intel.rs");
|
||||
let full_bytes = source.len();
|
||||
|
||||
// 骨架态:解析自身 read_symbol 函数(只给该符号,非整个文件)
|
||||
let skeleton = read_symbol(&source, "h0", "code_intel.rs", "read_symbol", false, None);
|
||||
assert_ne!(skeleton["fallback"], true, "真实文件应解析出 read_symbol 符号");
|
||||
assert_eq!(skeleton["symbol"], "read_symbol");
|
||||
assert_eq!(skeleton["kind"], "function_item");
|
||||
let skeleton_str = serde_json::to_string(&skeleton).unwrap();
|
||||
let skeleton_bytes = skeleton_str.len();
|
||||
|
||||
// 密度断言:read_symbol 骨架(单符号)<< read_file 全文(整个文件)。
|
||||
// 正是治 prompt 爆的核心:LLM 要理解 read_symbol,工具只给该符号骨架,不给全文件。
|
||||
let ratio = full_bytes as f64 / skeleton_bytes.max(1) as f64;
|
||||
eprintln!(
|
||||
"[密度效果] code_intel.rs 全文 {} 字节 vs read_symbol 骨架 {} 字节 → 降 {:.1}x",
|
||||
full_bytes, skeleton_bytes, ratio
|
||||
);
|
||||
assert!(ratio > 5.0, "骨架密度应显著高于全文(降 >5x),实际 {:.1}x", ratio);
|
||||
|
||||
// full 态:完整定义体(语法边界 100% 准)
|
||||
let full_sym = read_symbol(&source, "h1", "code_intel.rs", "read_symbol", true, None);
|
||||
let content = full_sym["content"].as_str().expect("full 态应返 content");
|
||||
assert!(content.contains("pub fn read_symbol"));
|
||||
// calls 应含 read_symbol 内部调用的函数(grammar_for/extract_symbol/fallback 等)
|
||||
let calls = skeleton["calls"].as_array().expect("骨架应含 calls");
|
||||
assert!(!calls.is_empty(), "应提取出内部调用");
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,7 @@
|
||||
pub mod agentic;
|
||||
pub mod augmentation;
|
||||
pub mod audit;
|
||||
pub mod code_intel;
|
||||
pub mod commands;
|
||||
pub mod compress;
|
||||
pub mod conversation;
|
||||
|
||||
@@ -1042,6 +1042,50 @@ fn register_file_tools(
|
||||
})
|
||||
})},
|
||||
);
|
||||
registry.register(
|
||||
"read_symbol", "AST 符号解析(信息密度驱动,治 read_file 全文回灌 prompt 爆)。提取函数/结构体/类等符号:默认返回骨架(签名+行范围+内部调用,极小高密度);传 full=true 取完整定义体。不支持/未找到时回退提示用 grep+read_file。Phase1 支持 .rs/.ts/.tsx/.js/.jsx/.vue。",
|
||||
df_ai::ai_tools::object_schema(vec![("path", "string", true), ("symbol", "string", true), ("full", "boolean", false), ("kind", "string", false)]),
|
||||
RiskLevel::Low,
|
||||
{ let allowed_dirs = allowed_dirs.clone(); Box::new(move |args: serde_json::Value| {
|
||||
let allowed_dirs = allowed_dirs.clone();
|
||||
Box::pin(async move {
|
||||
let snap = allowed_dirs.read().await.clone();
|
||||
let resolved = resolve_workspace_path_with_allowed(
|
||||
args["path"].as_str().ok_or_else(|| anyhow::anyhow!("缺少 path 参数"))?,
|
||||
&snap,
|
||||
)?;
|
||||
let path = resolved.to_str().ok_or_else(|| anyhow::anyhow!("路径含非法字符"))?;
|
||||
let symbol = args["symbol"].as_str().ok_or_else(|| anyhow::anyhow!("缺少 symbol 参数"))?;
|
||||
let full = args["full"].as_bool().unwrap_or(false);
|
||||
let kind_hint = args["kind"].as_str();
|
||||
use tokio::fs::File;
|
||||
use tokio::io::AsyncReadExt;
|
||||
let mut file = File::open(path).await
|
||||
.map_err(|e| anyhow::anyhow!("无法访问文件 {}: {}", path, e))?;
|
||||
let metadata = file.metadata().await
|
||||
.map_err(|e| anyhow::anyhow!("读取元数据失败 {}: {}", path, e))?;
|
||||
if metadata.len() > 1_048_576 {
|
||||
anyhow::bail!("文件超过 1MB 限制 ({} 字节)", metadata.len());
|
||||
}
|
||||
let file_hash = compute_file_hash(&metadata);
|
||||
let mut content = String::new();
|
||||
if let Err(e) = file.read_to_string(&mut content).await {
|
||||
if e.kind() == std::io::ErrorKind::InvalidData {
|
||||
return Ok(serde_json::json!({
|
||||
"path": path, "binary": true, "size": metadata.len(), "file_hash": file_hash,
|
||||
"fallback": true, "reason": "binary",
|
||||
"suggestion": "文件非 UTF-8 文本,无法 AST 解析,用 grep 搜内容",
|
||||
}));
|
||||
}
|
||||
anyhow::bail!("读取文件失败: {}", e);
|
||||
}
|
||||
// 调 code_intel 纯函数(三态 + 兜底,不 panic)
|
||||
Ok(crate::commands::ai::code_intel::read_symbol(
|
||||
&content, &file_hash, path, symbol, full, kind_hint,
|
||||
))
|
||||
})
|
||||
})},
|
||||
);
|
||||
registry.register(
|
||||
"list_directory", "列出目录内容，返回文件和子目录列表（名称、类型、大小）",
|
||||
df_ai::ai_tools::object_schema(vec![("path", "string", true), ("recursive", "boolean", false), ("skip_noise_dirs", "boolean", false), ("max_depth", "integer", false)]),
|
||||
|
||||
Reference in New Issue
Block a user