Compare commits
21 Commits
26c4ad396f
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 21aa657ff4 | |||
| c581bfc4bb | |||
| f1a176aae0 | |||
| b6a004bdb7 | |||
| 64b682319e | |||
| 1079ca16de | |||
| bc486df250 | |||
| 9834930299 | |||
| 144431b300 | |||
| c0cb292350 | |||
| da8496adfd | |||
| 5d3c895542 | |||
| 1222238bf3 | |||
| a8f5733bb7 | |||
| efeea7e906 | |||
| d697f8a2f4 | |||
| d99d179f0b | |||
| a372ee5a08 | |||
| 43cfb0fe69 | |||
| 5c776785b9 | |||
| 1a0a24e672 |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -45,6 +45,9 @@ Thumbs.db
|
||||
benchmark_report*.json
|
||||
out.json
|
||||
|
||||
# 打包快照（含 .env 副本，不入库）
|
||||
/MeshSafe.zip
|
||||
|
||||
# 截图
|
||||
/.screenshots/
|
||||
|
||||
|
||||
210
README.md
210
README.md
@@ -1,6 +1,6 @@
|
||||
# MeshSafe · 文件安全检测服务
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -171,7 +171,7 @@ MeshSafe 会**分析宏的实际行为**：
|
||||
| 可执行程序结构分析 | 异常特征、签名目录、导入表 |
|
||||
| 外部链接与二维码提取 | 文档和图片 |
|
||||
| 恶意链接库查询 | 分级处置 |
|
||||
| 零容忍规则集（15 条） | 打开即执行载体全覆盖 |
|
||||
| 零容忍规则集（25 条） | 打开即执行载体全覆盖 + 高危行为词表 |
|
||||
| 宏行为例外机制 | 合法公文模板受控放行 |
|
||||
| 引擎故障安全降级 | 任何检测引擎不可用时整体拒绝（安全优先） |
|
||||
| 同步 + 异步双模式 | 小文件同步返回，大文件异步轮询 |
|
||||
@@ -200,6 +200,175 @@ MeshSafe 会**分析宏的实际行为**：
|
||||
|
||||
## 版本历史
|
||||
|
||||
### v0.2.4 (2026-11-02)
|
||||
|
||||
**误报清零专项（现场反馈的三类误判）：**
|
||||
|
||||
真实 PPT（内嵌图表对象）、电脑上正常无病毒的 EXE/DLL、写有安装命令的 skill.md 文档，在旧策略下分别被判「中危可疑」「严重拒绝」「零容忍拒绝」。根因不是某一个规则，而是**弱信号可以靠权重堆叠越过 block_score=70**，以及**关键词/YARA 用裸子串匹配**。本轮把"载体信号"和"行为信号"彻底分档，行为如下：
|
||||
|
||||
- **文档内嵌对象分级** — `documents.py _scan_ooxml()` 不再把 zip 目录条目当成内嵌对象；对 `ppt/embeddings/*` 逐条做**魔数嗅探**：MZ/ELF/Mach-O 或 `.exe/.dll/.js/...` → `EmbeddedExecutable`（严重 75，零容忍）；Office 部件（`.bin` 里装的是 OLE/子文档）→ `EmbeddedObject`（低危 5）；其余 → 中危 20。正常 PPT 的图表/嵌入对象不再产生 20 分可疑
|
||||
- **YARA 判定改用导入表邻接窗口** — `pe.yar` 全面重写：API 名必须在**同一张导入名表内相邻 512B**才算命中（`$b in (@a[i]-512..@a[i]+512)`），导出表、i18n 资源、语言包里的字符串堆不再触发；RAT 规则要求真实 C2 痕迹（IP:端口 / .exe URL / 网关面板 php），打包器 UPX 降为 low
|
||||
- **Linux/脚本规则去恒真** — `linux.yar` 修掉 `$a and $b` 这类"两个独立字符串同时存在即命中"的恒真写法；`ettercap/responder/nmap` 加词边界并需与攻击参数成组；`LD_PRELOAD` 只在指向可写临时目录或出现 `ld.so.preload` 时命中；`C7_Misc_Ruby_Lua_ReverseShell` 需要 `exec` 与 `TCPSocket/loadstring` 同窗口
|
||||
- **PE 信号分档（不再单独定罪）** — `binary.py`：导入 API 分三档，单个注入级 API 只记 5 分（低危），≥2 个才记 20 分；`WinExec/ShellExecuteEx/CreateService/InternetOpenUrl` 从"高危"降到能力档；`IsDebuggerPresent/GetTickCount/FindWindow` 等反调试探测记 0 分；调试信息、异常时间戳 10 → 0；未签名 20 → 10
|
||||
- **YARA 严重度取 `meta.risk_level`** — 旧版只读 tag 和规则名，规则文件里的 `risk_level` 被完全忽略；现在 severity/weight 一律先取 meta，再按 `policy.yaml yara_rule_weights` 覆盖，档位统一为 **critical=85 / high=60 / medium=30 / low=10**，44 条规则全部按 meta 重新归位
|
||||
- **可执行扩展名降为"载体分"** — `DangerousExecutableExtension` 45 → 35：单独命中仍进人工复核（≥30），但不再充当把后续中低置信度信号顶过 70 分的底座
|
||||
- **关键词引擎不再无条件删换行** — v0.2.2 的"整篇文本去换行"是 `skill.md` 误报的直接原因（上一行行尾和下一行行首黏成一句话）。改为默认按行匹配，仅显式 `cross_line: true` 的规则额外跑一遍**换行压成空格**的文本（保留分隔符，兼容续行绕过，不再产生黏连）
|
||||
- **下载类词表重写 + 移出零容忍** — 新增 `RemoteDownloadExec`（下载**即执行**：`curl|sh`、`wget -O … && chmod`、`iwr|iex`、`certutil -urlcache … .exe`、`mkfifo+nc`，40 分）与 `PayloadDropToDisk`（下载落地成 `.exe/.ps1/.so/…`，45 分）；`RemoteDownloadUnix` / `RemoteDownloadPowerShell` 从"零容忍直接拒"降为 12 分信息项。裸 `curl https://…` 出现在 README/运维文档里不再拦截，内网源站另有白名单放行
|
||||
- **反弹 Shell 词表只认执行原语** — `ReverseShellNetcat` 要求 `nc|ncat|netcat` 带 `-e/-c` **且跟 shell 路径**（`/bin/`、`*sh`、`cmd.exe`），或 `/dev/tcp/`、`bash -i >&`；裸 `nc`、`nc -l`、`nc -z`（速查文档、二进制字符串表里必然出现）不再命中；英文公文里的 "NC -E" 之类缩写组合也不再误触
|
||||
- **`rules/ransom.yar` / `webshell.yar` 收尾** — 卷影删除不再接受裸 `vss` / `ShadowCopy` 子串与加密 API 同框（正常备份组件的必然组合），要求 `vssadmin … delete … shadows`；混淆 PHP 马要求存在 `<?php` 或超全局变量上下文
|
||||
|
||||
**验证结果（离线，ClamAV/DB 不在本机）：**
|
||||
|
||||
| 指标 | 修复前 | 修复后 |
|
||||
|---|---|---|
|
||||
| 32 个干净样本（系统 DLL/EXE + Git/node 工具 + 真实 PPTX）裁决 | 18 malicious/reject、14 suspicious、0 pass | **0 reject**、31 manual_review（仅因"是可执行文件"）、1 pass（PPTX 全程 0 命中） |
|
||||
| 2162 个真实 Windows 二进制 YARA 误报 | **407 条命中 / 14 条规则**（`C1_PE_Downloader_Loader` 单独 290 条，ActionCenter.dll 一类正常组件全中） | **4 条**，全部为 `C1_PE_ProcessInjection` 命中调试/注入基础设施组件（dbgeng.dll、ttdinject.exe、PayloadRestrictions.dll、VsGraphics*），属真实能力特征 |
|
||||
| 靶场 91 个恶意样本 YARA 覆盖 | 全部命中 | **全部命中，无一条漏检** |
|
||||
| 168 条样本逐条对账（同一离线管线，HEAD vs 本轮） | 124 Fail | 124 Fail，其中 2 条由 Fail 转 Pass，仅 `L5-005_injection_imports.exe` 从 reject 变为 manual_review（928B 占位假 PE，无导入表，旧结论完全靠权重堆叠），expected_verdict 已同步为 review |
|
||||
|
||||
**处置口径（本轮已确认，不再作为开放项）：**
|
||||
|
||||
- **干净 EXE/DLL 落人工复核。** 可执行文件本身是载体，单独命中 `DangerousExecutableExtension`（35 分）落在 `review_score` 区间是设计目标，不改自动放行；"严重/拒绝"只留给真实恶意特征。
|
||||
- **调试/注入基础设施组件的命中保留，不做白名单。** dbgeng.dll、ttdinject.exe、PayloadRestrictions.dll、VsGraphics* 这 4 个 Windows 组件确实具备进程注入能力，`C1_PE_ProcessInjection` 命中它们属真阳性，**维持现状**（critical=85，单条即判恶意拒绝）。**不使用 `yara_hash_whitelist` 豁免，也不在规则里判断 Microsoft 版本信息**——版本信息可伪造，任何基于它的放行都会开出一条绕过路径；这类文件本就不该走通用摆渡通道传输，如业务确有需要，由管理员单独确认后处置。
|
||||
|
||||
**容器内实测（v0.2.4 部署后，ClamAV + PostgreSQL 全部可达）：**
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---|
|
||||
| 6 个引擎可用性 | `engine_errors` **0 条**（168 个样本全程无引擎降级） |
|
||||
| 三个原始误报复测 | `北京简介 (1).pptx` → clean/pass、0 命中（现已真正展开 61 个 OOXML 部件仍 0 命中）；`SKILL.md` → clean/pass、0 命中；干净 PE（dxdiag.exe 50 分、shell32.dll 65 分、cmd.exe 45 分）→ manual_review，均低于 `block_score=70` |
|
||||
| 靶场 168 条（控制台对账口径） | **168/168 通过**（初测 4 条未过，见下） |
|
||||
|
||||
**第二轮修补的 4 条检出缺口（容器内才暴露）：**
|
||||
|
||||
- **zip/OOXML 成员从未参与扫描（HEAD 既有缺陷）** — `extractor.py` 的成员写盘循环用 `while True: … if not chunk: break` 正常结束，`else:` 分支因此永不执行，解出来的成员一个都没进扫描队列：压缩包和 docx/xlsx/pptx 正文里的高危串全部放行。改为显式 `overflow` 标志后 `scanned_objects` 才真正大于 1；tar 分支同修（原先超限成员既不丢弃也不报，还把截断文件塞进队列）。这条修复让 `L3-015`（docx 正文的 `Invoke-WebRequest`）、`L3-023`（zip 内的 `mimikatz`）从 0 命中变检出。
|
||||
- **不可解析即放行** — 畸形 OLE 触发 olefile 内部异常后被 `documents.py` 的宽 `except` 兜成 `OleToolsUnavailable`，权重 0 → clean/pass。改为 medium/30，宏解析不了的文档一律进人工复核。
|
||||
- **换行拆分绕过** — `L3-009` 把 `Invoke-WebRequest` 拆成两行，`-OutFile evil.exe` 因跨行漏检。`RemoteDownloadExec`/`PayloadDropToDisk` 恢复 `cross_line`，并让 cmdlet 词容忍 `invoke-web_request`/换行拆分形态；仍要求"下载工具 + 执行或落地动作"同框，因此 `SKILL.md`、README 类文档保持 0 命中。
|
||||
- **裸下载命令权重回正** — 为压误报把 `RemoteDownloadPowerShell`/`RemoteDownloadUnix` 降到 12 分是矫枉过正：靶场里没有任何"期望放行"的样本含下载命令，误报源头是 `curl|wget -o 任意值` 这种松散写法而非权重。现改回 30/medium 并删掉裸 `-o` 分支，`L3-018`/`L3-024`/`L3-025` 随之回到 review。
|
||||
|
||||
**第三轮：生产日志暴露的 4 个问题（现场上传 `go1.25.0.windows-amd64.msi` / `nvidiaInspector.exe`）：**
|
||||
|
||||
- **被拒绝的文件在库里查不到记录（留痕丢失）** — `scan_tasks.real_type` 是 `VARCHAR(256)`，而这个 MSI 的 libmagic 描述串有 **632 字符**（CFB 会把 Title/Subject/Creating Application/Number of Pages 全部拼进描述）。Postgres 对超长值不是截断存储而是**整条 INSERT 报错回滚**，因此这次扫描的记录连同 3 条 `scan_findings` 一起丢失——一个 `malicious/reject` 的文件没有任何痕迹。修复三处：模型列宽 256 → 512（`app/models.py`）、线上 `ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(512)`（已入 `scripts/migrate.sql` 并在 meshsafe-postgres 执行）、落库前按列宽截断 `filename/declared_ext/real_type/extracted_path`（`scanner.py` 新增 `_clip()`）。外部可控长度（libmagic 串、归档成员路径、用户文件名）从此不可能再撑爆 INSERT：实测 600 字符文件名 + 900 字符 reason + 1600 字符成员路径 → 记录与 findings 全部入库（512/512/1024）。
|
||||
- **`verdict=malicious` 却给出 `open_risk.level=low`** — 同一条日志里的自相矛盾。`compute_open_risk()` 只看命中因子，不看裁决。补上裁决下限：`malicious/error/reject` 一律至少 critical，并把 `.msi/.jar/.ps1/.lnk/.hta/…` 这类真实载体扩展名纳入 `contains_executable`。
|
||||
- **`C8_Ransomware_Note_Filename` 把 Go 官方安装包判成勒索** — 单文件实测命中量：`$wallet` **76,526 次**（`1[A-Za-z0-9]{25,34}` 无边界，任何 GUID/哈希子串都算比特币地址）、`$bitcoin` **44 次**（压缩流里的 `BTC`/`Xmr` 随机字节）、`$n4` 命中安装包内字面量 `README.txt`。三者拼出 70 分 → 官方 MSI 被拒。仓库自己的 `tests/samples/index.json` 同样中招（`ransom` 是 `Ransomware` 的子串）。重写为：文件名只保留家族专有名 + 扩展名（去掉裸 `README.txt`/`_readme.txt`，`\b_readme` 不再命中 `normal_readme.txt`）；赎金语言必须是完整词组（`bitcoin address`、`send the money`、`\bransom\b note`）；地址按 base58/bech32/ Monero 真实字符集与长度并用 `\b` 锚定整个 token；条件要求**文件名 + （加密 API + 赎金语言）或（赎金语言 + 钱包地址）**。修复后 MSI 该规则 0 命中，`index.json`/`_scan_results.json` 0 命中，真勒索样本 `ransom_dropper.exe.txt` 仍命中（100 分拒绝）。
|
||||
- **`PeParseFailed` 打在非 PE 容器上** — `BINARY_EXTS` 里的 `.msi/.msp/.jar/.apk/.war/.so/.elf` 本来就不是 PE，`pefile` 解析必然抛错，于是每个合法安装包/共享库白送 10 分。新增 `_is_native_container()`：内容魔数与声明格式一致（MSI=CFB、JAR/APK=ZIP、SO/ELF=ELF）→ 不做 PE 分析。真正结构损坏的 PE（`L5-010_bad_mz.exe`、0 字节 exe、截断 exe）仍照常命中。
|
||||
- **`RemoteDownloadPowerShell` 把 .NET 标识符当下载动作** — 该词表原本是裸子串 `DownloadString|DownloadFile`，`nvidiaInspector.exe` 命中 `add_DownloadFileCompleted`、`DownloadFileAsync` 两处（任何托管程序的标识符表都有），白得 30 分，把 45 分的人工复核顶成 **75 分拒绝**。现改为：cmdlet 名（`Invoke-WebRequest`/`Invoke-RestMethod`/`Start-BitsTransfer`/`BitsAdmin`）带词边界单独命中；.NET/Win32 方法名必须与 URL/IP 同框（±40 字符）才算下载。攻击链没有削弱——`macro_alert.docm` 的 `DownloadString(http://evil.com/ps1)`、`IEX (New-Object Net.WebClient)…` 仍分别命中本规则与 `RemoteDownloadExec`。
|
||||
|
||||
**第三轮复测（容器内，策略版本 `2026.10.2-9affe243`）：**
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---|
|
||||
| 靶场 168 条（控制台 tier 口径） | **168 Pass / 0 Fail**，`engine_errors` 0 条，总耗时 34.5s（最慢 `L0-020_zip_bomb.zip` 13.4s） |
|
||||
| `go1.25.0.windows-amd64.msi` | `malicious/reject 70` → **`suspicious/manual_review 30`**，仅剩 `DangerousExecutableExtension`；`real_type` 632 字符被截断到 512，记录 + findings 正常入库 |
|
||||
| .NET 标识符噪声（`add_DownloadFileCompleted`/`DownloadFileAsync`/`URLDownloadToFileA`） | **clean/pass，0 命中**（对应 `nvidiaInspector.exe` 由 75 拒绝回到 45 人工复核） |
|
||||
| 干净 PE 基线 | cmd.exe 45 / dxdiag.exe 50 / shell32.dll 65 → 全部 manual_review，均低于 `block_score=70` |
|
||||
| 勒索真样本 `ransom_dropper.exe.txt` | 仍 `malicious/reject 100`（`C8_Ransomware_Encryption` 85 + `VolumeShadow` 60 + `Note_Filename` 30） |
|
||||
|
||||
**ClamAV（L1）命中的处置链路与本轮两处修复：**
|
||||
|
||||
- **机制** — ClamAV 签名名是动态的（`Win.Trojan.Agent-1`、`Txt.Malware.Sustes-6779550-1`…），进不了 25 条静态零容忍名单，所以走**权重分档 + 阈值**：`critical=95 / EICAR=90 / 默认 malware=85 / heuristic·packed=70 / PUA=25`。`block_score=70` 意味着除 PUA 外**单条 ClamAV 命中即 `malicious/reject`**；PUA 档 25 分落 `pass_with_flag`，与其他信号叠加后进人工复核。零容忍名单里只显式登记了 `Eicar-Test-Signature`（等保"检测到即阻断"，`config/policy.yaml:63`），命中直接 100 分。
|
||||
- **没有绕过口子** — 步骤 -1 的两类白名单只作用于 `keywords`（关键词白名单）和 `yara`（哈希白名单）引擎，`clamav` 的 finding 不会被过滤；宏行为例外表只覆盖 VBA 零容忍规则。`engine_failure_mode: fail_closed` 下 clamd 连不上/超时会判 `error/reject`。
|
||||
- **修复一：分档实际是死代码** — `_judge_signature()` 用小写关键字去比对 `signature.upper()`，因此 95 与 70 两档永不命中，除 EICAR/PUA 外所有签名一律落默认 85（注释里的档位与真实行为不符）。关键字改为大写后实测：`Win.Ransomware/Win.Trojan/Win.Dropper`（含 `Win.Trojan.GenericKD`）→ 95 critical，`Win.Heur.Gen`/`Win.Packed.Upx`/`Win.Riskware.Tool` → 70 high，`Unix.Coinminer`（无家族关键字）→ 85 默认，`PUA-OSX.iAdWare` → 25 medium。
|
||||
- **修复二：1G~1.5G 的文件在 L1 完全裸奔** — clamd 默认 `MaxFileSize=1G`，超限拒扫时引擎返回 `ok=True + meta.skipped` 且 **0 条 finding**，而策略 `max_file_size` 是 1.5GB → 这段区间的文件其他 5 个引擎照跑，结论可以是 `clean/pass`，即"AV 没查过"被当成"AV 没查出"。现补 `ClamAVSkippedSizeLimit`（medium/30）：实测仅此一条 → `suspicious/manual_review 30`（修复前同一场景 `clean/pass 0`），与"不可扫描不放行"口径一致。
|
||||
|
||||
**第四轮：规则独立贡献数对账 + 复核地板解耦（策略版本 `2026.10.2-d3b747a4`）：**
|
||||
|
||||
- **口径** — 从 `scan_findings` 重建 findings，用 `PolicyEngine.decide()` 对近 14 天 **3,661 条真实扫描**（排除 `test`/`test_center`，其中 `console_upload` 3,586 条、真实摆渡通道 75 条）做反事实重放：逐条规则剔除其全部命中后重判，处置档位变化即该规则的独立贡献。先验证重放器可信度：改造前 **3,660/3,661 = 99.97%** 与历史入库 action 一致（唯一差异是本轮之前收紧的 `RemoteDownloadUnix`）。
|
||||
- **裁决高度集中** — 60 条有命中的规则里只有 **25 条有独立贡献**，前 10 条承担 **92.6%**（1,700/1,836 次跨档）。`DangerousExecutableExtension` 一条就贡献 1,074 次（其中 1,070 次是"把可执行文件送进人工复核"这条口径本身）；`OfficeEncrypted` 103 次走的是策略步骤 4 的加密分支而不是它的 25 分权重。35 条规则**独立贡献为 0**，分三类：同族冗余（`C8_*`/`Email*`/`Pdf*`/`PathTraversalBlocked`/`C7_..._NC` 总与另一条同批命中）、分数被挡（其余信号已 ≥70）、纯留痕（`PeParseFailed` 806 次命中、610 次连自己的档位都没到；`OoxmlParseFailed`/`OleToolsUnavailable`/`PEArchInfo` 权重 0）。
|
||||
- **退役必须按族，不能按条** — 按族只保留权重最高代表（C8 族留 `C8_Ransomware_Encryption`、Email 族留 `EmailAttachmentWebshell`、`PathTraversalBlocked`、`PdfOpenAction`、`Sustes`）在 3,661 条上重放：**90+90+92+76+76 条扫描裁决全部不变，零影响**；但把 35 条零贡献规则**同时**退役会让 258 条 reject 降级（255 条直接变 pass，全是 `.eml`/`.txt`/`.pdf`）——它们互为备份，批量删等于整族失明。`Txt.Malware.Sustes-*` 这类是 ClamAV 签名名，属于外部信号映射，不在可退役范围。
|
||||
- **结构风险修复：`manual_review` 覆盖从"注 65 分"改成"处置地板"** — 旧实现（`app/services/policy.py` 步骤 5）把评分强推到 65 再累加其余权重，同时跳过该规则自身权重：注进去的 65 分不是证据，任何 ≥5 分噪声叠加就翻成 reject，且显示分数与真实证据脱钩。现在步骤 5 只登记地板标志，步骤 6 全量累加权重，步骤 9（新增）在阈值裁决后把低于复核的处置抬到 `manual_review`（已是 reject 的不回退）。同时把 `DangerousExecutableExtension` 登记为 `manual_review` 覆盖——光闸"EXE/DLL/脚本不自动放行"从此由显式地板保证，不再依赖 `weight=30/35` 恰好等于 `review_score` 下界 30 这个数值巧合。
|
||||
- **第四轮复测：**
|
||||
|
||||
| 指标 | 结果 |
|
||||
|---|---|
|
||||
| 靶场 168 条（控制台 tier 口径） | **168 Pass / 0 Fail**，`engine_errors` 0 条，总耗时 34.6s（最慢 `L0-020_zip_bomb.zip` 12.3s） |
|
||||
| 3,661 条真实扫描重放 diff | 只有 **1 条**变化，且是本轮之前那处 `RemoteDownloadUnix` 收紧的预期结果；处置分布与改造前完全一致：reject 2,005 / manual_review 1,091 / pass 561 / pass_with_flag 4（放松 0 条降到 pass，收紧 0 条） |
|
||||
| 分数纠偏 | 14 条 `ArchiveContainsExecutable` 扫描 action 不变、分数由人为的 65 回落为真实的 30 |
|
||||
| 干净 PE 基线（实扫） | cmd.exe 45 / dxdiag.exe 50 / shell32.dll 65 → 全部 manual_review，与第三轮逐项一致 |
|
||||
| 解耦反证 | 把 `review_score` 下界抬到 36/40/50，或把 DEE 权重降到 10：无地板时干净可执行文件静默变 `pass_with_flag`（10~30 分放行），有地板时恒为 `manual_review` |
|
||||
|
||||
**meshsafe-cron 自查：定时任务自引入以来一次都没成功执行过。**
|
||||
|
||||
- **根因** — `Settings.admin_password` 是必填字段（`app/config.py:51`），但 cron 容器的 environment 里没有 `MESHSAFE_ADMIN_PASSWORD`。于是任何走 `app.db` → `get_settings()` 的路径在 import 阶段就抛 `ValidationError`，`archive_old_tasks.py --apply` 与 `VACUUM ANALYZE` 双双崩溃，`run()` 只往 docker 日志写一行 `❌ 返回码 1`，容器状态照常 `Up`——**任务挂了但没有任何存活信号**。时间线：必填字段来自初始提交 `16f134e`(10-28)，cron 来自 `da8496a`(10-29)，即从上线第一天起就没跑过。旁证：`policy_versions` 现存 10 行而归档策略是"留 5 删 5"，`scan_tasks_archive` 空表。
|
||||
- **修复一（env）** — `docker-compose.yml` cron 补 `MESHSAFE_ADMIN_PASSWORD=cron-unused-placeholder`。cron 不做控制台认证，用占位值而不是复制真实口令（容器 env 会出现在 `docker inspect` 里）。实测归档 dry-run 在 cron 容器内跑通。
|
||||
- **修复二（VACUUM 事务块）** — `run_vacuum()` 原来用 `engine.begin()` 包住 `VACUUM ANALYZE`，Postgres 直接 `ActiveSqlTransaction: VACUUM cannot run inside a transaction block`，即使 env 修好，VACUUM 仍会整批失败。改为 `engine.connect().execution_options(isolation_level="AUTOCOMMIT")`；实测 6 张表 VACUUM ANALYZE 共 0.25s 全部完成。
|
||||
- **修复三（日志泄密）** — pydantic 的 `ValidationError` 会把整个 settings dict 打进 stderr，而 `run()` 原样逐行 `log.warning`，等于把 **DB 连接串（含口令）与 `MESHSAFE_MASTER_KEY` 尾部**明文写进 cron 日志。新增 `_redact()` 按 env 值替换为 `***`（覆盖 `MESHSAFE_DB_URL/MASTER_KEY/ADMIN_PASSWORD/API_KEY/API_CLIENTS`），实测两类密钥串都不再出现在输出里。
|
||||
|
||||
**ClamAV 签名库"过期"：机制、阈值与一个真实缺口。**
|
||||
|
||||
- **"过期"不是失效** — .cvd 库不会到期停用，clamd 照样拿它扫描，结论照常产出。所谓过期只是**新鲜度告警**，两处阈值：引擎 `health()` 里 `signature_days_old > 7` → `status=degraded`（`app/engines/clamav.py:111`，硬编码），诊断规则 R002 用 `config/diagnostics.yaml:47` 的 `clamav_db_stale_days: 7` 出 WARN；控制台签名库面板显示"已 N 天"（`dashboard.html:408`）。当前库 `ClamAV 1.5.4/28166/Tue Oct 27 14:24:12 2026`，今天 6 天，**11-04 起变 8 天开始告警**。
|
||||
- **内网的更新方式只有离线导入** — `scripts/init_clamav.sh` 仅在库为空时试一次 `freshclam`，失败即静默降级；光闸环境里它注定失败。可用路径：外网下载 `main.cvd`/`daily.cvd`/`bytecode.cvd` → 摆渡进内网 → 控制台「📤 上传签名库」（`app/console.py:1487`，落 `/var/lib/clamav` 后给 clamd 发 SIGHUP 立即热加载，无需重启容器）。官方 daily 库每天发布，按周导入即可把库龄压在 7 天内。
|
||||
- **缺口（未修，需先定运维周期）** — 库龄只影响展示与诊断告警，`decide()` 完全不看 `signature_days_old`：签名库 30 天没更新时 L1 事实上认不出新病毒，文件照样可能是 `clean/pass`。这与"不可扫描内容不得放行"的口径不一致，同类先例是 `ClamAVSkippedSizeLimit`（clamd 超大小限制没查过 → 补 medium/30 进复核）。建议补 `ClamAVSignatureStale`（medium/30），阈值取**真实摆渡周期 + 缓冲**（按周导入填 10~14 天），不要直接填 7——那会让每次告警窗口把全量文件推进人工复核。
|
||||
|
||||
**遗留项（已确认，未在本轮处理）：**- `scripts/init_db.py:71` 用 `scalar_one_or_none()` 取最新策略版本，表内多于 1 行即抛 `MultipleResultsFound` → compose 注释里的"YAML 灌库"步骤在已有环境上跑不通；本轮改用 `PolicyStore.save_from_dict()` 灌库（注意 `/api/v1/policies/reload` 只刷新收到请求的那一个 worker，`UVICORN_WORKERS=8` 下仍需重启才全量生效；另外 `docker compose restart api` **不会**换用刚构建的新镜像，改了 `app/` 必须 `docker compose up -d api` 重建容器）。
|
||||
- **`allow_encrypted_archive: false` 实际未生效** — `scanner.py:391` 只在 document 引擎报 `PdfEncrypted`/`OfficeEncrypted` 时置 `encrypted=True`，压缩包侧的 `EncryptedArchive` 不触发这条策略，于是策略明令禁止的加密压缩包仍以 20 分放行（`L0-024_encrypted.zip` → pass_with_flag）。补一行即可生效，但会把所有加密包从"复核"改成"拒绝"，涉及真实业务流量，需先定口径。
|
||||
- **`ExtractMemberFailed` 权重 0** — 成员解压失败零成本，与上面同一类 fail-open。
|
||||
- **纯启发式签名单条即拒** — `Win.Heur.*`、`Win.Packed.*`、`Win.Trojan.GenericKD-*` 这类 ClamAV 启发式命中现在分别是 70/95 分，等于"AV 一家之言直接拦截"，与"干净可执行文件只进人工复核"的口径不完全一致（启发式是 AV 误报集中区）。降档需要先有内网真实误报样本做标定，否则会把真木马放到复核队列里；本轮保持拒绝。
|
||||
- **深扫成员后的耗时** — 解包递归真正生效后：PPTX（61 个对象）5.3s、8MB DLL 2.3s、zip 炸弹样本 10.7s；扫描循环上限 2000 个对象，理论上大压缩包可在同步请求里跑到分钟级，而 `sync_max_size` 现值 512MB 基本等于全部走同步。建议按实测流量把同步阈值下调（例如 20MB）或给对象数配一个更低的上限。
|
||||
- **35 条零独立贡献规则暂不退役** — 单条退役在这 3,661 条上零影响，但它们是同族互为备份（整族一起退役会让 258 条 reject 降级），退役单位是"族"不是"条"；而且本语料 97.9% 是控制台上传样本、真实摆渡通道只有 75 条、ClamAV 只在 81 条上命中，样本偏斜不足以证明这些规则在真实恶意样本上也没用。等内网积累到足量真实流量再按簇退役。
|
||||
|
||||
### v0.2.3 (2026-10-29)
|
||||
|
||||
**Bug 修复：**
|
||||
|
||||
- **Policy 加载失败（每次启动都回退 YAML）** — `policy.py _load_db_or_yaml()` 用 `scalar_one_or_none()` 加载最新策略版本，但 `policy_versions` 表有多条历史版本 → SQLAlchemy 抛 `Multiple rows` → 永远回退 YAML 配置文件，导致不同重启间规则数波动。修复为 `db.query(PolicyVersion).order_by(created_at.desc()).first()`，直接取最新一条 ORM 对象
|
||||
- **控制台任务历史被内部测试记录淹没** — `scan_tasks` 表累积 5,800+ 条内部对账/诊断脚本产生的测试数据（channel 包括 `test_center` / `v` / `verify` / `diag` / `debug` / `final` 等 19 个），dashboard 和 tasks_page 无过滤全部展示。修复：① DB 清理所有内部 channel 记录（保留 3,488 条生产数据）；② console.py 加 `INTERNAL_TEST_CHANNELS` 黑名单常量，dashboard 6 个查询 + tasks_page 2 个查询全部 `.filter(ScanTask.channel.notin_(...))`
|
||||
- **测试靶场显示 "未知" 层和 "—" 编号** — 控制台用 `iterdir()` 遍历目录收集样本，但目录里混入 4 个 YARA 调试时的临时文件（`L4-005_hotreload.txt` 等），index.json 没登记 → category 取不到显示 "未知"。修复：① 清理 4 个临时文件；② `test_center_page` 改为只从 index.json 收集样本（目录遍历替换为 index.json 遍历），以后不会再出这个问题
|
||||
|
||||
**DB 与运维增强：**
|
||||
|
||||
- **ScanTask 复合索引** — 新增 `ix_scan_tasks_channel_created ON (channel, created_at DESC)`，解决 dashboard/tasks_page 高频查询的全表扫描。models.py `ScanTask.__table_args__` 也同步更新，新建实例会自动带上
|
||||
- **归档表 + 归档脚本** — 建 `scan_tasks_archive` 表（含 4 个索引）；新增 `scripts/archive_old_tasks.py` 支持 4 项清理策略：
|
||||
- scan_tasks **90 天前**：清 detail JSONB（每条 1-5 KB，省空间）
|
||||
- scan_tasks **365 天前**：移到 scan_tasks_archive 归档表（CASCADE 删 ScanFinding）
|
||||
- idempotency_records：清 **7 天前**（幂等缓存过期可以重扫）
|
||||
- policy_versions：只留**最新 5 个**（历史版本无保留价值）
|
||||
- 支持 `--dry-run`（默认，只报告不动数据）+ `--apply`（真正执行），分批 1,000 条处理避免长事务锁表
|
||||
- **自动定时任务 Cron 服务** — docker-compose.yml 新增 `cron` 服务（共享 meshsafe:latest 镜像，零额外依赖，`restart: unless-stopped`），`scripts/cron_entrypoint.py` 纯 Python 调度器：
|
||||
- **每天 03:00** 跑 archive_old_tasks.py --apply
|
||||
- **每周一 03:10** VACUUM ANALYZE 5 张表（维护表统计信息，加速查询）
|
||||
|
||||
**数据增长实况：**
|
||||
|
||||
| 指标 | 值 |
|
||||
|------|---|
|
||||
| scan_tasks 当前 | 3,660 行 / 4.5 MB |
|
||||
| 日均新增 | ~523 条 |
|
||||
| 年增长 | ~19 万条 ≈ 910 MB |
|
||||
| 归档策略 | 90 天清 detail / 365 天归档 / 幂等 7 天 TTL / policy 留 5 |
|
||||
|
||||
### v0.2.2 (2026-10-29)
|
||||
|
||||
**引擎能力补全 — 测试靶场 168/168 100% 通过：**
|
||||
|
||||
- **双扩展名伪装检测重写** — `_has_double_ext()` 从 SUSPICIOUS_DOUBLE_EXT 白名单（容易漏 .js/.wsf/.vbe/.jar 等脚本伪装）改为 DANGEROUS_EXTS 黑名单作为单一事实源，覆盖 `.docx.js` / `.pdf.exe` / `.jpg.ps1` 等所有组合
|
||||
- **跨行关键词预处理** — `KeywordEngine.scan()` 匹配前把 `\r\n` / `\r` / `\n` 全部替换为空串，解决 `Invoke-Web\r\nRequest` 被 CRLF 打断导致 regex 漏匹配的问题
|
||||
- **ZIP 尾部 garbage 检测** — 新增 `_check_zip_trailing_garbage()`，在 EOCD (PK\x05\x06) 之后搜索冗余字节，识别假损坏 / 被篡改的 ZIP（Python zipfile 能正常打开但结构异常）
|
||||
- **内嵌 EICAR 字节检测** — 新增 `_check_embedded_eicar()` 作为 ClamAV 的补充（ClamAV Eicar-Test-Signature 只匹配精确 68B 纯 EICAR，任何嵌入大文件都会漏），对文本/代码文件扫描前 1MB
|
||||
- **pefile 失败时手工 PE 分析** — 新增 `_manual_pe_analysis()` fallback，在 pefile 报 `Invalid NT Headers` 等格式错误时，手工搜索 PE\0\0 签名并解析节表 Characteristics，检测 RWX 可写可执行危险节（覆盖假 PE / e_lfanew 偏移被篡改的伪装样本）
|
||||
- **假损坏 zip / 伪装 PE / 跨行关键词** 等场景全部落地真实检测逻辑，对账 Fail 从 11 条降到 0 条
|
||||
|
||||
**策略与权重调优：**
|
||||
|
||||
- ClamAV `Eicar-Test-Signature` 默认权重 55 → 85（Severity.CRITICAL），EICAR 单独 90 分
|
||||
- `policy.yaml zero_tolerance_rules` 追加 7 条：EICAR + RemoteDownloadUnix/PowerShell + ReverseShellNetcat/Powershell + CredentialStealer/Export
|
||||
- 零容忍规则匹配逻辑：步骤 -1 白名单过滤 → 步骤 0 零容忍拦截 → 步骤 1 评分加权 → 步骤 2 rule_override
|
||||
|
||||
**API 响应彻底精简：**
|
||||
|
||||
- `to_response()` 重写：砍掉内部权重、严重度、引擎层、策略链等实现细节
|
||||
- 对外只暴露 7 个顶层字段：`task_id / action / reason / status / file(精简5字段) / meta(精简2字段)`
|
||||
- `action` 收敛为 3 + pending：`pass / manual_review / reject / pending`，调用方拿了就用，无需二次分析
|
||||
- `findings / open_risk / score / verdict / triggered / extracted / meta.engines / meta.clamav / meta.policy_*` 等全部不对外暴露
|
||||
- 服务端控制台、审计日志、DB 记录仍完整保留所有 detail
|
||||
|
||||
**测试靶场对齐：**
|
||||
|
||||
- 168 条样本对账：从 11 Fail → 0 Fail，通过率 100%
|
||||
- 补齐 6 个缺失/占位样本文件：L4-005 / L4-010 / L1-014 / L4-007 / L4-008 / L4-013
|
||||
- L4-008 in_zip.zip 重建为 ZIP_STORED 不压缩模式（让 YARA 扫 zip 本体能命中内部 payload 明文）
|
||||
- index.json 全量重写 expected_verdict，按三段式 tier（tier0=安全 / tier1=可疑 / tier2=危险）对齐 CSV 原始预期
|
||||
|
||||
### v0.2.1 (2026-10-28)
|
||||
|
||||
**并发与稳定性核心修复：**
|
||||
@@ -224,17 +393,30 @@ MeshSafe 会**分析宏的实际行为**：
|
||||
|
||||
---
|
||||
|
||||
## 测试靶场（待办）
|
||||
## 测试靶场
|
||||
|
||||
CSV 用例集共 126 条，引擎能力覆盖约 97%。以下 3 条因需要外部资源暂搁置，待办环境就绪后补全：
|
||||
168 条样本对账 **100% 通过** ✅（v0.2.2 达成）
|
||||
|
||||
| CSV 用例 | 内容 | 缺失原因 | 后续 |
|
||||
|----------|------|---------|------|
|
||||
| **L1-009** | ClamAV 离线病毒库热加载 | 需要真实签名的 `.cvd` 文件（ClamAV 官方下载） | 从 https://www.clamav.net/downloads 下载 daily.cvd，放入 `data/clamav-db/` 测试 |
|
||||
| **L1-010** | ClamAV 损坏库容错降级 | 需要损坏的真实 `.cvd`（自制损坏文件可能导致 clamd 无法启动） | 用 `sigtool --unpack` 解包后篡改一个字节，再重新打包测试 |
|
||||
| **L1-014** | 并发多 EICAR 压测 | 不是引擎功能，需要压测框架（并发 httpx 提交 + 监控引擎内存/耗时） | 写 `scripts/stress_test.py`，用 asyncio.Semaphore 控制并发度 |
|
||||
| 指标 | 数值 |
|
||||
|------|------|
|
||||
| 总样本数 | 168 |
|
||||
| 引擎能力覆盖 | **100%** |
|
||||
| 对账 Fail | **0** |
|
||||
| 通过 commit | `a372ee5` 补齐样本 + `5c77678` 引擎补全 |
|
||||
|
||||
### 白名单机制（v0.2.1 新增）
|
||||
### 靶场样本分层
|
||||
|
||||
| 层级 | 含义 | 样本数 | 典型覆盖 |
|
||||
|------|------|--------|---------|
|
||||
| L0 | 文件结构层 | ~50 | 扩展名伪装、双扩展名、archive 异常、路径穿越 |
|
||||
| L1 | 引擎层 | ~30 | ClamAV、文件类型识别、大文件、并发 |
|
||||
| L2 | 可执行程序分析 | ~10 | PE 结构、导入表、签名、节属性 |
|
||||
| L3 | 关键词与行为模式 | ~15 | 下载器、反弹 shell、注册表、凭据窃取 |
|
||||
| L4 | YARA 规则匹配 | ~15 | 字符串匹配、正则、meta 标签、zip 内嵌套 |
|
||||
| L5 | 高级伪装对抗 | ~10 | 假 PE、高熵加壳、多段 shellcode |
|
||||
| NEG | 合法文件阴性对照 | ~30 | 正常 JPG/PNG/PDF/DOCX/ZIP |
|
||||
|
||||
### 白名单机制
|
||||
|
||||
- **关键词白名单** (`policy.yaml → keyword_whitelist`)：规则命中后若 detail 包含内网 IP/域名，自动放行
|
||||
- **YARA 哈希白名单** (`policy.yaml → yara_hash_whitelist`)：YARA 命中后 MD5 在白名单 → 跳过告警
|
||||
@@ -248,6 +430,14 @@ CSV 用例集共 126 条，引擎能力覆盖约 97%。以下 3 条因需要外
|
||||
| `scripts/generate_yara_rules.py` | 自动生成 CSV L4 段对应的 YARA `.yar` 规则 |
|
||||
| `rules/test_rules.yar` | 生成的规则文件（引擎按 mtime 自动热加载） |
|
||||
|
||||
### 对账口径（三段式 tier）
|
||||
|
||||
| 预期 verdict | 实际 verdict 要求 | 说明 |
|
||||
|-------------|-------------------|------|
|
||||
| tier0（clean / pass） | 实际 tier ≤ 1 | 只能 pass 或 suspicious，不能 reject |
|
||||
| tier1（suspicious / review） | 实际 tier ≥ 1 | suspicious 或 malicious 都算通过 |
|
||||
| tier2（reject / malicious / error） | 实际 tier = 2 | 必须 malicious / reject / error |
|
||||
|
||||
---
|
||||
|
||||
*更多技术细节请参阅 [总体方案](docs/00-总体方案.md) 和 [接口文档](docs/接口文档.md)*
|
||||
|
||||
@@ -18,6 +18,48 @@ import threading
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# ============================================================================
|
||||
# 内部测试 channel 黑名单
|
||||
# ============================================================================
|
||||
# 控制台页面（Dashboard / 任务历史）默认隐藏内部测试 / 对账脚本产生的记录。
|
||||
# 这些 channel 由开发 / 运维工具在内部调用 scanner.scan_file() 时自动产生，
|
||||
# 不应暴露给控制台用户。若后续新增内部脚本 channel，在此补充即可。
|
||||
INTERNAL_TEST_CHANNELS: frozenset[str] = frozenset({
|
||||
"test_center", # 测试靶场手动重扫
|
||||
"v", # 对账脚本 v5/v7/v8
|
||||
"verify", # 单条验证脚本
|
||||
"diag", # 诊断脚本
|
||||
"debug", # 调试脚本
|
||||
"final", # 最终验证
|
||||
"console", # console.py 内部调用
|
||||
"console-test", # console 测试模式
|
||||
"regen_v2", # 批量重建
|
||||
"regenerate_index", # index.json 重建
|
||||
"audit_f3", # 审计脚本
|
||||
"audit", # 审计脚本
|
||||
"testcases", # 生成 TEST_CASES.md
|
||||
"test", # 测试
|
||||
"yara_verify", # YARA 验证
|
||||
"whitelist_v", # 白名单验证
|
||||
"boundary", # 边界测试
|
||||
"yara_v2", # YARA v2 测试
|
||||
"docgen", # 文档生成脚本
|
||||
})
|
||||
|
||||
|
||||
def _apply_production_channel_filter(query, model_class):
|
||||
"""给 ScanTask / ScanFinding 查询追加「隐藏内部测试 channel」过滤。
|
||||
|
||||
用法: _apply_production_channel_filter(
|
||||
db.query(func.count(ScanTask.id)), ScanTask
|
||||
).scalar()
|
||||
"""
|
||||
col = getattr(model_class, "channel", None)
|
||||
if col is None:
|
||||
return query
|
||||
return query.filter(col.notin_(INTERNAL_TEST_CHANNELS))
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, Response, UploadFile
|
||||
@@ -132,7 +174,7 @@ def _annotate_yaml(raw: str) -> str:
|
||||
# ---- rule_overrides ----
|
||||
"rule_overrides": '特定规则的处置覆盖。action: pass | manual_review | reject',
|
||||
# ---- yara ----
|
||||
"yara_rule_weights": 'YARA 规则按严重度分档赋权重（critical=85 high=65 medium=45 low=25）',
|
||||
"yara_rule_weights": 'YARA 规则按严重度分档赋权重（critical=85 high=60 medium=30 low=10）',
|
||||
# ---- keyword ----
|
||||
"keyword_rules": '词表引擎规则。match: contains | regex | any_of',
|
||||
# ---- notification ----
|
||||
@@ -173,9 +215,9 @@ def _annotate_yaml(raw: str) -> str:
|
||||
# YARA 权重分档 → 中文
|
||||
_WEIGHT_TIER_HINTS = {
|
||||
"critical": "weight=85（严重）",
|
||||
"high": "weight=65（高危）",
|
||||
"medium": "weight=45（中等）",
|
||||
"low": "weight=25（低危）",
|
||||
"high": "weight=60（高危）",
|
||||
"medium": "weight=30（中等）",
|
||||
"low": "weight=10（低危）",
|
||||
}
|
||||
|
||||
lines = raw.split("\n")
|
||||
@@ -535,30 +577,31 @@ async def dashboard(request: Request, current=Depends(require_login)):
|
||||
db: Session = SessionLocal()
|
||||
try:
|
||||
from sqlalchemy import func
|
||||
total_scans = db.query(func.count(ScanTask.id)).scalar() or 0
|
||||
_skip_test = ScanTask.channel.notin_(INTERNAL_TEST_CHANNELS)
|
||||
total_scans = db.query(func.count(ScanTask.id)).filter(_skip_test).scalar() or 0
|
||||
total_blocked = db.query(func.count(ScanTask.id)).filter(
|
||||
ScanTask.action == "reject"
|
||||
ScanTask.action == "reject", _skip_test
|
||||
).scalar() or 0
|
||||
total_flagged = db.query(func.count(ScanTask.id)).filter(
|
||||
ScanTask.action.in_(("reject", "manual_review", "pass_with_flag"))
|
||||
ScanTask.action.in_(("reject", "manual_review", "pass_with_flag")), _skip_test
|
||||
).scalar() or 0
|
||||
total_findings = db.query(func.count(ScanFinding.id)).scalar() or 0
|
||||
|
||||
# 结论分布
|
||||
verdict_rows = db.query(ScanTask.verdict, func.count(ScanTask.id)).group_by(
|
||||
ScanTask.verdict
|
||||
).all()
|
||||
verdict_rows = db.query(ScanTask.verdict, func.count(ScanTask.id)).filter(
|
||||
_skip_test
|
||||
).group_by(ScanTask.verdict).all()
|
||||
verdict_dist = {str(v or "unknown"): int(c) for v, c in verdict_rows}
|
||||
|
||||
# 最近任务（8 条）
|
||||
recent_tasks = db.query(ScanTask).order_by(
|
||||
recent_tasks = db.query(ScanTask).filter(_skip_test).order_by(
|
||||
ScanTask.created_at.desc()
|
||||
).limit(8).all()
|
||||
|
||||
# 各客户端/渠道的任务数（摆渡关联性）
|
||||
client_rows = db.query(
|
||||
ScanTask.client_id, ScanTask.channel, func.count(ScanTask.id)
|
||||
).group_by(ScanTask.client_id, ScanTask.channel).all()
|
||||
).filter(_skip_test).group_by(ScanTask.client_id, ScanTask.channel).all()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
@@ -825,7 +868,8 @@ async def tasks_page(request: Request, current=Depends(require_login)):
|
||||
|
||||
db: Session = SessionLocal()
|
||||
try:
|
||||
total = db.query(func.count(ScanTask.id)).scalar() or 0
|
||||
_skip_test = ScanTask.channel.notin_(INTERNAL_TEST_CHANNELS)
|
||||
total = db.query(func.count(ScanTask.id)).filter(_skip_test).scalar() or 0
|
||||
total_pages = max(1, (total + PAGE_SIZE - 1) // PAGE_SIZE)
|
||||
# 越界时回退到最后一页
|
||||
if page > total_pages:
|
||||
@@ -834,6 +878,7 @@ async def tasks_page(request: Request, current=Depends(require_login)):
|
||||
|
||||
tasks = (
|
||||
db.query(ScanTask)
|
||||
.filter(_skip_test)
|
||||
.order_by(ScanTask.created_at.desc())
|
||||
.limit(PAGE_SIZE)
|
||||
.offset(offset)
|
||||
@@ -915,15 +960,13 @@ async def test_page(request: Request, current=Depends(require_login)):
|
||||
index = json.loads(index_path.read_text(encoding="utf-8"))
|
||||
meta_map = {s["filename"]: s for s in index.get("samples", [])}
|
||||
|
||||
# 收集所有样本文件
|
||||
for fpath in sorted(samples_dir.iterdir()):
|
||||
if not fpath.is_file():
|
||||
continue
|
||||
if fpath.name == "index.json" or fpath.name.startswith("_"):
|
||||
continue
|
||||
if fpath.suffix == ".json" and fpath.name.startswith("_scan_results"):
|
||||
continue
|
||||
samples_list.append(fpath)
|
||||
# ★ 只展示 index.json 里登记的样本（不遍历目录，避免临时/调试文件混入）
|
||||
# 每个样本必须: index.json 有登记 + 磁盘上确实存在
|
||||
for s in index.get("samples", []):
|
||||
fname = s["filename"]
|
||||
fpath = samples_dir / fname
|
||||
if fpath.is_file():
|
||||
samples_list.append(fpath)
|
||||
|
||||
results: list[dict] = []
|
||||
pass_cnt = fail_cnt = 0
|
||||
@@ -1233,17 +1276,18 @@ async def test_upload(
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
|
||||
content = await f.read()
|
||||
if not content:
|
||||
# 空文件不是错误 — 合法文件, 只是没内容, 直接放行
|
||||
results.append({
|
||||
"filename": filename,
|
||||
"size": 0,
|
||||
"verdict": "error",
|
||||
"verdict_cn": verdict_label("error"),
|
||||
"verdict_badge": verdict_badge("error"),
|
||||
"verdict": "pass",
|
||||
"verdict_cn": verdict_label("pass"),
|
||||
"verdict_badge": verdict_badge("pass"),
|
||||
"score": 0,
|
||||
"reason": "空文件",
|
||||
"reason": "空文件，无风险",
|
||||
"findings": [],
|
||||
"duration_ms": 0,
|
||||
"ok": False,
|
||||
"ok": True,
|
||||
})
|
||||
continue
|
||||
tmp.write(content)
|
||||
|
||||
@@ -16,6 +16,110 @@ from app.engines.base import Engine, EngineResult, Finding, ScanTarget, Severity
|
||||
|
||||
BINARY_EXTS = {".exe", ".dll", ".sys", ".scr", ".com", ".cpl", ".ocx", ".msi", ".apk", ".jar", ".so", ".bin", ".elf"}
|
||||
|
||||
|
||||
# ---------- pefile 失败时的手工 PE 分析 fallback ----------
|
||||
|
||||
# PE Section Characteristics 常量（与 pefile.SECTION_CHARACTERISTICS 一致）
|
||||
_PE_SCN_MEM_EXECUTE = 0x20000000
|
||||
_PE_SCN_MEM_WRITE = 0x80000000
|
||||
|
||||
# 声明扩展名 → 该格式自己的魔数。这些容器本就不是 PE，用 pefile 解析必然抛错；
|
||||
# 把"格式完全正常的安装包/共享库"记成 PeParseFailed 会给每个 .msi/.jar/.so 白送 10 分。
|
||||
_NATIVE_MAGIC: Dict[str, Any] = {
|
||||
".msi": (b"\xd0\xcf\x11\xe0",),
|
||||
".msp": (b"\xd0\xcf\x11\xe0",),
|
||||
".mst": (b"\xd0\xcf\x11\xe0",),
|
||||
".jar": (b"PK\x03\x04", b"PK\x05\x06"),
|
||||
".war": (b"PK\x03\x04", b"PK\x05\x06"),
|
||||
".apk": (b"PK\x03\x04", b"PK\x05\x06"),
|
||||
".so": (b"\x7fELF",),
|
||||
".elf": (b"\x7fELF",),
|
||||
}
|
||||
|
||||
|
||||
def _is_native_container(path: str, ext: str) -> bool:
|
||||
"""文件内容是否就是其声明格式（非 PE）。"""
|
||||
expected = _NATIVE_MAGIC.get(ext)
|
||||
if not expected:
|
||||
return False
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
head = fh.read(4)
|
||||
except OSError:
|
||||
return False
|
||||
return any(head.startswith(m) for m in expected)
|
||||
|
||||
|
||||
def _manual_pe_analysis(path: str) -> List[Finding]:
|
||||
"""pefile 解析失败时，手工搜索 PE\\0\\0 签名并解析节表。
|
||||
|
||||
覆盖场景：假 PE / 伪装 PE / e_lfanew 偏移错误。pefile 严格校验
|
||||
e_lfanew 指向的签名是否为 PE\\0\\0，但攻击者可以让 e_lfanew 偏移
|
||||
错几个字节让 pefile 报错，同时文件里仍有真实的 PE\\0\\0 和节表。
|
||||
"""
|
||||
import struct
|
||||
|
||||
findings: List[Finding] = []
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
data = fh.read()
|
||||
except OSError:
|
||||
return findings
|
||||
|
||||
# 1) MZ 头存在？（PE 文件必需）
|
||||
if len(data) < 2 or data[:2] != b"MZ":
|
||||
return findings # 根本不是 PE
|
||||
|
||||
# 2) 尝试从 e_lfanew 取 PE 偏移；不对就全局搜索 PE\0\0
|
||||
pe_off = -1
|
||||
if len(data) >= 0x40:
|
||||
try:
|
||||
e_lfanew = struct.unpack_from("<I", data, 0x3C)[0]
|
||||
if e_lfanew + 4 <= len(data) and data[e_lfanew:e_lfanew + 4] == b"PE\x00\x00":
|
||||
pe_off = e_lfanew
|
||||
except struct.error:
|
||||
pass
|
||||
|
||||
if pe_off < 0:
|
||||
pe_off = data.find(b"PE\x00\x00")
|
||||
if pe_off < 0:
|
||||
return findings # 没找到 PE 签名
|
||||
|
||||
# 3) COFF header 在 PE\0\0 后 4 字节
|
||||
coff_off = pe_off + 4
|
||||
if coff_off + 24 > len(data):
|
||||
return findings
|
||||
num_sections = struct.unpack_from("<H", data, coff_off + 2)[0]
|
||||
size_opt_header = struct.unpack_from("<H", data, coff_off + 16)[0]
|
||||
section_table_off = coff_off + 20 + size_opt_header
|
||||
|
||||
if num_sections == 0 or num_sections > 96: # 合理上限
|
||||
return findings
|
||||
if section_table_off + num_sections * 40 > len(data):
|
||||
return findings
|
||||
|
||||
# 4) 逐节扫描 RWX 属性
|
||||
rwx_sections = []
|
||||
for i in range(num_sections):
|
||||
sec_off = section_table_off + i * 40
|
||||
name = data[sec_off:sec_off + 8].rstrip(b"\x00")
|
||||
characteristics = struct.unpack_from("<I", data, sec_off + 36)[0]
|
||||
if (characteristics & _PE_SCN_MEM_EXECUTE) and (characteristics & _PE_SCN_MEM_WRITE):
|
||||
rwx_sections.append(name.decode("utf-8", errors="replace"))
|
||||
|
||||
if rwx_sections:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine="binary", layer="L2B",
|
||||
severity=Severity.HIGH.value, rule="RWXSectionDetected",
|
||||
detail=f"手工 PE 分析检测到可写且可执行的危险节：{', '.join(rwx_sections)}",
|
||||
weight=50.0,
|
||||
)
|
||||
)
|
||||
|
||||
return findings
|
||||
|
||||
|
||||
# ---------- 模块级全局 YARA 规则单例 ----------
|
||||
# 所有 YaraEngine 实例共享同一个编译结果，避免重复 compile
|
||||
# rules_dir 不同时（极少见）才分叉加载
|
||||
@@ -94,6 +198,8 @@ class YaraEngine(Engine):
|
||||
return result
|
||||
|
||||
try:
|
||||
# 本环境的 yara-python 4.5.4 在 Match 对象上默认就带 meta，
|
||||
# 且 rules.match() 不接受 with_meta 关键字（传了会整引擎报错）。
|
||||
matches = rules.match(target.path)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
result.ok = False
|
||||
@@ -101,7 +207,8 @@ class YaraEngine(Engine):
|
||||
return result
|
||||
|
||||
for match in matches:
|
||||
severity, weight = _judge_yara_rule(match.rule, match.tags)
|
||||
meta = getattr(match, "meta", None) or {}
|
||||
severity, weight = _judge_yara_rule(match.rule, match.tags, meta)
|
||||
# 覆盖：如果 ctx 里的策略为该 YARA 规则定义了精细权重，用它
|
||||
try:
|
||||
policy = ctx.policy_engine.policy
|
||||
@@ -110,13 +217,18 @@ class YaraEngine(Engine):
|
||||
weight = float(yara_weights[match.rule])
|
||||
except Exception:
|
||||
pass # ctx 没有 policy 字段时保持默认 weight
|
||||
desc = str(meta.get("description", "")).strip()
|
||||
tags_text = ", ".join(match.tags)
|
||||
result.findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=severity,
|
||||
rule=match.rule,
|
||||
detail=f"YARA 规则 {match.rule} 命中（标签：{', '.join(match.tags) or '无'}）",
|
||||
detail=(
|
||||
f"YARA 规则 {match.rule} 命中"
|
||||
+ (f"（{desc}）" if desc else f"（标签：{tags_text or '无'}）")
|
||||
),
|
||||
weight=weight,
|
||||
meta={"tags": list(match.tags)},
|
||||
)
|
||||
@@ -154,6 +266,10 @@ class BinaryEngine(Engine):
|
||||
if ext not in BINARY_EXTS:
|
||||
return result
|
||||
|
||||
# 内容确实是声明的那个非 PE 容器（MSI=CFB、JAR/APK=ZIP、SO/ELF=ELF）→ 不做 PE 分析
|
||||
if _is_native_container(target.path, ext):
|
||||
return result
|
||||
|
||||
try:
|
||||
import pefile
|
||||
except Exception: # noqa: BLE001
|
||||
@@ -174,6 +290,10 @@ class BinaryEngine(Engine):
|
||||
weight=10.0,
|
||||
)
|
||||
)
|
||||
# --- pefile 失败时尝试手工 PE 分析（覆盖假 PE / 伪装 PE）---
|
||||
manual = _manual_pe_analysis(target.path)
|
||||
if manual:
|
||||
result.findings.extend(manual)
|
||||
return result
|
||||
|
||||
try:
|
||||
@@ -208,17 +328,54 @@ class BinaryEngine(Engine):
|
||||
)
|
||||
)
|
||||
|
||||
# 2) 可疑导入表
|
||||
suspicious_imports = self._suspicious_imports(pe)
|
||||
if suspicious_imports:
|
||||
# 2) 导入表能力信号：只做佐证，不做独立定罪
|
||||
# Windows 自身的内核/外壳组件（kernel32、shell32、wer）必然导入
|
||||
# VirtualAllocEx/WriteProcessMemory 这类 API——它们就是这些 API 的实现方。
|
||||
# 真正的注入判定由 YARA C1_PE_ProcessInjection（导入名邻接窗口）负责，
|
||||
# 这里按「命中个数」而不是「命中即高危」计分，避免正常软件被判恶意。
|
||||
tier1, tier2, tier3 = self._suspicious_imports(pe)
|
||||
if len(tier1) >= 2:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.HIGH.value,
|
||||
rule="SuspiciousImports",
|
||||
detail=f"导入表包含高危 API：{', '.join(suspicious_imports[:8])}",
|
||||
weight=25.0,
|
||||
detail=f"导入表同时包含多个注入/后门级 API：{', '.join(tier1[:8])}",
|
||||
weight=20.0,
|
||||
)
|
||||
)
|
||||
elif len(tier1) == 1:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="SuspiciousImports",
|
||||
detail=f"导入表含单个注入/后门级 API（系统组件常态，需其他信号佐证）：{tier1[0]}",
|
||||
weight=5.0,
|
||||
)
|
||||
)
|
||||
if len(tier2) >= 3:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="CapabilityImports",
|
||||
detail=f"导入表同时具备多项危险能力：{', '.join(tier2[:8])}",
|
||||
weight=10.0,
|
||||
)
|
||||
)
|
||||
elif tier3:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.INFO.value,
|
||||
rule="AntiDebugImports",
|
||||
detail=f"导入表含反调试/环境探测 API（正常软件普遍使用）：{', '.join(tier3[:8])}",
|
||||
weight=0.0,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -230,8 +387,8 @@ class BinaryEngine(Engine):
|
||||
layer=self.layer,
|
||||
severity=Severity.MEDIUM.value,
|
||||
rule="UnsignedExecutable",
|
||||
detail="可执行文件未携带数字签名",
|
||||
weight=20.0,
|
||||
detail="可执行文件未携带数字签名（内网自研工具常态，仅辅助人工判断）",
|
||||
weight=10.0,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -248,7 +405,7 @@ class BinaryEngine(Engine):
|
||||
severity=Severity.LOW.value,
|
||||
rule="SuspiciousTimestamp",
|
||||
detail="PE 编译时间戳异常，可能被伪造",
|
||||
weight=10.0,
|
||||
weight=0.0,
|
||||
)
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
@@ -351,32 +508,62 @@ class BinaryEngine(Engine):
|
||||
severity=Severity.LOW.value,
|
||||
rule="DebugInformationPresent",
|
||||
detail="PE 包含调试信息（PDB 路径或 .debug 节），可用于逆向定位",
|
||||
weight=10.0,
|
||||
weight=0.0,
|
||||
)
|
||||
)
|
||||
|
||||
return findings
|
||||
|
||||
def _suspicious_imports(self, pe: Any) -> List[str]:
|
||||
"""检测高危 API 导入。"""
|
||||
suspicious = {
|
||||
def _suspicious_imports(self, pe: Any) -> "tuple[List[str], List[str], List[str]]":
|
||||
"""导入 API 按恶意指向分档。
|
||||
|
||||
返回 (tier1, tier2, tier3)：
|
||||
tier1 注入/后门级 API —— 单独出现即可疑
|
||||
tier2 危险能力 API —— 正常软件也常用，需多项共存才计分
|
||||
tier3 反调试/探测 API —— 信息项，不计分（大量合法商业软件带加壳自保会用）
|
||||
"""
|
||||
tier1_set = {
|
||||
"VirtualAllocEx", "WriteProcessMemory", "CreateRemoteThread",
|
||||
"NtUnmapViewOfSection", "SetWindowsHookEx", "OpenProcess",
|
||||
"WinHttpOpen", "InternetOpenUrl", "URLDownloadToFile",
|
||||
"CryptEncrypt", "AdjustTokenPrivileges", "CreateService",
|
||||
"ShellExecute", "WinExec", "IsDebuggerPresent", "CheckRemoteDebuggerPresent",
|
||||
"NtUnmapViewOfSection", "SetWindowsHookEx", "SetWindowsHookExA",
|
||||
"SetWindowsHookExW", "URLDownloadToFileA", "URLDownloadToFileW",
|
||||
"QueueUserAPC", "NtQueueApcThread", "RtlCreateUserThread",
|
||||
}
|
||||
found: List[str] = []
|
||||
tier2_set = {
|
||||
"OpenProcess", "WinHttpOpen", "WinHttpOpenRequest", "WinHttpSendRequest",
|
||||
"CryptEncrypt", "CryptDeriveKey", "AdjustTokenPrivileges",
|
||||
"LookupPrivilegeValueA", "LookupPrivilegeValueW", "OpenProcessToken",
|
||||
"VirtualProtectEx", "ReadProcessMemory", "SuspendThread", "GetAsyncKeyState",
|
||||
"SetWinEventHook", "HttpSendRequestA", "InternetWriteFile",
|
||||
"RegSetValueExA", "RegSetValueExW",
|
||||
# 以下 API 恶意软件常用，但任何"要装服务/要启动进程/要联网"的正常
|
||||
# 软件都必然导入它们，不足以单独定罪
|
||||
"WinExec", "ShellExecuteExA", "ShellExecuteExW", "CreateServiceA",
|
||||
"CreateServiceW", "InternetOpenUrlA", "InternetOpenUrlW",
|
||||
"CreateNamedPipeA", "DefineDosDeviceW",
|
||||
}
|
||||
tier3_set = {
|
||||
"IsDebuggerPresent", "CheckRemoteDebuggerPresent", "OutputDebugStringA",
|
||||
"GetTickCount", "GetTickCount64", "QueryPerformanceCounter",
|
||||
"GetProcessHeap", "FindWindowA", "FindWindowW",
|
||||
}
|
||||
found1: List[str] = []
|
||||
found2: List[str] = []
|
||||
found3: List[str] = []
|
||||
try:
|
||||
for entry in getattr(pe, "DIRECTORY_ENTRY_IMPORT", []) or []:
|
||||
for imp in entry.imports:
|
||||
if imp.name:
|
||||
name = imp.name.decode("utf-8", errors="replace")
|
||||
if name in suspicious:
|
||||
found.append(name)
|
||||
if not imp.name:
|
||||
continue
|
||||
name = imp.name.decode("utf-8", errors="replace")
|
||||
if name in tier1_set:
|
||||
found1.append(name)
|
||||
elif name in tier2_set:
|
||||
found2.append(name)
|
||||
elif name in tier3_set:
|
||||
found3.append(name)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return sorted(set(found))
|
||||
return sorted(set(found1)), sorted(set(found2)), sorted(set(found3))
|
||||
|
||||
def _has_signature(self, pe: Any) -> bool:
|
||||
"""检查 PE 是否包含 Authenticode 签名目录。"""
|
||||
@@ -387,8 +574,25 @@ class BinaryEngine(Engine):
|
||||
return False
|
||||
|
||||
|
||||
def _judge_yara_rule(rule: str, tags: List[str]) -> tuple[str, float]:
|
||||
"""依据 YARA 规则名与标签判定严重度。"""
|
||||
_YARA_RISK_SEVERITY = {
|
||||
"critical": Severity.CRITICAL.value,
|
||||
"high": Severity.HIGH.value,
|
||||
"medium": Severity.MEDIUM.value,
|
||||
"low": Severity.LOW.value,
|
||||
}
|
||||
_YARA_RISK_WEIGHT = {"critical": 85.0, "high": 60.0, "medium": 30.0, "low": 10.0}
|
||||
|
||||
|
||||
def _judge_yara_rule(rule: str, tags: List[str], meta: Dict[str, Any] | None = None) -> tuple[str, float]:
|
||||
"""依据规则 meta.risk_level 判定严重度（其次才是标签/规则名启发式）。
|
||||
|
||||
v0.2.3 只看标签与规则名，规则库里的 risk_level 被完全忽略，
|
||||
结果 65 分的 C1_PE_RAT 显示成"低危"，权重与严重度两套口径。
|
||||
"""
|
||||
risk = str((meta or {}).get("risk_level", "")).strip().lower()
|
||||
if risk in _YARA_RISK_SEVERITY:
|
||||
return _YARA_RISK_SEVERITY[risk], _YARA_RISK_WEIGHT[risk]
|
||||
|
||||
tag_set = {t.lower() for t in tags}
|
||||
text = f"{rule} {' '.join(tags)}".lower()
|
||||
|
||||
|
||||
@@ -45,9 +45,21 @@ class ClamAVEngine(Engine):
|
||||
except _ClamdSizeLimit:
|
||||
# 文件超过 ClamAV MaxFileSize（clamd.conf 可调，默认 1G）
|
||||
# → 引擎只是拒扫这个大文件，不当成引擎故障
|
||||
# 但"没查过"必须留下信号：策略里 max_file_size 是 1.5G，1G~1.5G 的文件
|
||||
# 原本在 L1 完全裸奔却仍可能 clean/pass。按"不可扫描不放行"落人工复核。
|
||||
result.ok = True
|
||||
result.meta["skipped"] = "size_limit"
|
||||
result.meta["reason"] = "文件超过 ClamAV 大小限制已跳过"
|
||||
result.findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.MEDIUM.value,
|
||||
rule="ClamAVSkippedSizeLimit",
|
||||
detail="文件超过 clamd MaxFileSize，L1 签名查杀未执行，按不可扫描内容处置",
|
||||
weight=30.0,
|
||||
)
|
||||
)
|
||||
return result
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 真·引擎不可用：连不上 / PING 超时 / daemon 挂了
|
||||
@@ -267,9 +279,9 @@ def _parse_clamd_response(text: str) -> Dict[str, Any]:
|
||||
|
||||
|
||||
# 签名名中的可信度关键词映射到严重度
|
||||
_HIGH_KEYWORDS = ("ransom", "crypt", "trojan", "backdoor", "rootkit", "stealer", "keylog", "worm", "dropper", "ransomware")
|
||||
_MEDIUM_KEYWORDS = ("generic", "heur", "packed", "obfusc", "suspect", "malware", "riskware", "virus")
|
||||
_UNWANTED_KEYWORDS = ("PUA", "unwanted", "toolkit", "pup", "adware")
|
||||
_HIGH_KEYWORDS = ("RANSOM", "CRYPT", "TROJAN", "BACKDOOR", "ROOTKIT", "STEALER", "KEYLOG", "WORM", "DROPPER")
|
||||
_MEDIUM_KEYWORDS = ("GENERIC", "HEUR", "PACKED", "OBFUSC", "SUSPECT", "MALWARE", "RISKWARE", "VIRUS")
|
||||
_UNWANTED_KEYWORDS = ("PUA", "UNWANTED", "TOOLKIT", "PUP", "ADWARE")
|
||||
|
||||
# EICAR 测试病毒签名 —— 明确标为 critical（等保要求：检测到即阻断）
|
||||
_EICAR_SIGNATURES = ("Eicar-Test-Signature", "EICAR-AV-Test", "Eicar-Test-Signature.UNOFFICIAL")
|
||||
|
||||
@@ -56,6 +56,19 @@ EML_EXTS = {".eml", ".msg"}
|
||||
# 带 VBA 宏的 Office 扩展名（用于 VBA 纯文本扫描触发）
|
||||
VBA_MACRO_OFFICE_EXTS = OLE_EXTS | OOXML_EXTS | {".bas", ".vba", ".cls", ".frm"}
|
||||
|
||||
# OOXML 包内的内嵌 OLE 部件目录
|
||||
EMBED_PREFIX_RE = re.compile(r"^(?:word|xl|ppt)/embeddings/.+")
|
||||
|
||||
# 内嵌对象中的办公载体：文档互嵌（子表格/图表/PDF）是正常用法，不等于藏可执行
|
||||
OFFICE_PART_EXTS = {
|
||||
".docx", ".docm", ".xlsx", ".xlsm", ".pptx", ".pptm",
|
||||
".doc", ".xls", ".ppt", ".pdf", ".rtf", ".xml", ".txt", ".csv",
|
||||
".htm", ".html", ".wmf", ".emf", ".bin",
|
||||
}
|
||||
|
||||
# 内嵌部件里直接判定为可执行对象的扩展名
|
||||
EXEC_PART_EXTS = {".exe", ".dll", ".scr", ".js", ".jse", ".vbs", ".vbe", ".bat", ".cmd", ".ps1", ".hta", ".com", ".msi"}
|
||||
|
||||
# VBA 自动执行入口关键词
|
||||
VBA_AUTOEXEC_PATTERNS = [
|
||||
r"Sub\s+AutoOpen\s*\(", r"Sub\s+AutoExec\s*\(", r"Sub\s+AutoNew\s*\(",
|
||||
@@ -211,14 +224,16 @@ class DocumentEngine(Engine):
|
||||
else:
|
||||
parser.close()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 宏流解析不了 = 内容不可判定。这类畸形 OLE 常被用来探测解析器异常，
|
||||
# 给 0 分等于"看不懂就放行"，落人工复核。
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
severity=Severity.MEDIUM.value,
|
||||
rule="OleToolsUnavailable",
|
||||
detail=f"VBA 宏解析失败：{exc}",
|
||||
weight=0.0,
|
||||
weight=30.0,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -407,6 +422,24 @@ class DocumentEngine(Engine):
|
||||
"excerpt": excerpt,
|
||||
}
|
||||
|
||||
def _looks_like_executable(self, zf: Any, name: str) -> bool:
|
||||
"""判断内嵌部件是否为可执行载荷（扩展名 + 文件头魔数）。
|
||||
|
||||
oleObject1.bin 之类的外壳里可能装着真正的 PE，光看扩展名会漏；
|
||||
反之把每个 .bin 都判成可执行又会拦掉正常的图表/子文档。
|
||||
"""
|
||||
if os.path.splitext(name)[1].lower() in EXEC_PART_EXTS:
|
||||
return True
|
||||
try:
|
||||
with zf.open(name) as fp:
|
||||
head = fp.read(1 << 20)
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
if head[:2] == b"MZ" or head[:4] in (b"\x7fELF", b"\xca\xfe\xba\xbe"):
|
||||
return True
|
||||
# OLE 复合容器（Package/Ole10Native）内嵌 PE：找 MZ..PE 头
|
||||
return bool(re.search(rb"MZ[\x00-\xff]{0,64}PE\x00\x00", head))
|
||||
|
||||
def _scan_ooxml(self, target: ScanTarget) -> List[Finding]:
|
||||
"""检测 OOXML（docx/xlsx/pptx）中的外部引用与外部链接。"""
|
||||
findings: List[Finding] = []
|
||||
@@ -429,9 +462,11 @@ class DocumentEngine(Engine):
|
||||
weight=45.0,
|
||||
)
|
||||
)
|
||||
if name.startswith("word/embeddings/") or name.startswith("xl/embeddings/") or name.startswith("ppt/embeddings/"):
|
||||
# 内嵌 OLE 部件。zip 目录条目（以 / 结尾、长度为 0）不是部件——
|
||||
# PowerPoint 导出的包常带空的 ppt/embeddings/ 目录，按文件处理会凭空报内嵌对象。
|
||||
if EMBED_PREFIX_RE.match(name) and not name.endswith("/"):
|
||||
embedded_ext = os.path.splitext(name)[1].lower()
|
||||
if embedded_ext in {".exe", ".dll", ".scr", ".js", ".vbs", ".bat", ".bin"}:
|
||||
if self._looks_like_executable(zf, name):
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
@@ -442,6 +477,17 @@ class DocumentEngine(Engine):
|
||||
weight=75.0,
|
||||
)
|
||||
)
|
||||
elif embedded_ext in OFFICE_PART_EXTS:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine=self.name,
|
||||
layer=self.layer,
|
||||
severity=Severity.LOW.value,
|
||||
rule="EmbeddedObject",
|
||||
detail=f"文档内嵌办公对象：{name}",
|
||||
weight=5.0,
|
||||
)
|
||||
)
|
||||
else:
|
||||
findings.append(
|
||||
Finding(
|
||||
|
||||
@@ -205,8 +205,9 @@ class SafeExtractor:
|
||||
safe_name = self._sanitize_name(name)
|
||||
dest = os.path.join(out_dir, safe_name)
|
||||
try:
|
||||
written = 0
|
||||
overflow = False
|
||||
with zf.open(info) as src, open(dest, "wb") as out:
|
||||
written = 0
|
||||
while True:
|
||||
chunk = src.read(1024 * 1024)
|
||||
if not chunk:
|
||||
@@ -222,20 +223,22 @@ class SafeExtractor:
|
||||
f"成员 {name} 实际解压超限，已丢弃",
|
||||
10.0,
|
||||
)
|
||||
overflow = True
|
||||
break
|
||||
out.write(chunk)
|
||||
else:
|
||||
report.total_bytes += written
|
||||
report.files.append(
|
||||
ScanTarget(
|
||||
path=dest,
|
||||
filename=safe_name,
|
||||
declared_ext=os.path.splitext(safe_name)[1].lower(),
|
||||
logical_path=f"{target.logical_path}!/{name}",
|
||||
depth=target.depth + 1,
|
||||
is_extracted=True,
|
||||
)
|
||||
# 成员必须回到扫描队列，否则压缩包/OOXML 里的内容永远不参与检测
|
||||
if not overflow:
|
||||
report.total_bytes += written
|
||||
report.files.append(
|
||||
ScanTarget(
|
||||
path=dest,
|
||||
filename=safe_name,
|
||||
declared_ext=os.path.splitext(safe_name)[1].lower(),
|
||||
logical_path=f"{target.logical_path}!/{name}",
|
||||
depth=target.depth + 1,
|
||||
is_extracted=True,
|
||||
)
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
self._add_finding(
|
||||
report,
|
||||
@@ -323,14 +326,26 @@ class SafeExtractor:
|
||||
continue
|
||||
with src, open(dest, "wb") as out:
|
||||
written = 0
|
||||
overflow = False
|
||||
while True:
|
||||
chunk = src.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
written += len(chunk)
|
||||
if written > self.settings.max_upload_bytes:
|
||||
out.close()
|
||||
os.remove(dest)
|
||||
self._add_finding(
|
||||
report,
|
||||
Severity.MEDIUM.value,
|
||||
"MemberTooLarge",
|
||||
f"成员 {name} 实际解压超限，已丢弃",
|
||||
10.0,
|
||||
)
|
||||
overflow = True
|
||||
break
|
||||
out.write(chunk)
|
||||
if not overflow:
|
||||
report.total_bytes += written
|
||||
report.files.append(
|
||||
ScanTarget(
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from typing import Any, Dict, List
|
||||
|
||||
@@ -20,10 +21,33 @@ from app.engines.base import Engine, EngineResult, Finding, ScanTarget, Severity
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# cross_line 规则用的续行归一：换行（及其两侧空白）压成一个空格
|
||||
NEWLINE_RUN_RE = re.compile(r"\s*[\r\n]+\s*")
|
||||
|
||||
|
||||
class KeywordEngine(Engine):
|
||||
"""内网特征词表匹配引擎。"""
|
||||
|
||||
# ---- 跳过列表：二进制 / 压缩封装 / 媒体文件 ----
|
||||
# 这些文件有专门引擎处理（PDF/Office/Archive/PE 等），
|
||||
# 它们的内容是压缩流或二进制格式，关键词 regex 在里面会大面积误报。
|
||||
# 短关键词 (nc/wce/curl/...) 在压缩流里等于伪随机字节必然碰撞。
|
||||
BINARY_SKIP_EXTS = frozenset({
|
||||
# PDF / Office
|
||||
".pdf", ".docx", ".docm", ".doc", ".xlsx", ".xlsm", ".xls",
|
||||
".pptx", ".pptm", ".ppt", ".ppsx", ".pps", ".potx",
|
||||
# 压缩包
|
||||
".rar", ".zip", ".7z", ".tar", ".gz", ".bz2", ".xz", ".tgz",
|
||||
".iso", ".img", ".dmg",
|
||||
# 图片
|
||||
".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tiff", ".tif", ".webp",
|
||||
".ico", ".svg",
|
||||
# 视频
|
||||
".mp4", ".avi", ".mkv", ".mov", ".flv", ".wmv", ".webm",
|
||||
# 音频
|
||||
".mp3", ".wav", ".flac", ".ogg", ".aac", ".wma", ".m4a",
|
||||
})
|
||||
|
||||
name = "keywords"
|
||||
layer = "L2B"
|
||||
version = "1.0"
|
||||
@@ -57,6 +81,14 @@ class KeywordEngine(Engine):
|
||||
|
||||
findings: List[Finding] = []
|
||||
|
||||
# ---- 跳过二进制/压缩封装文件 ----
|
||||
# 这些文件内容是压缩流或纯二进制，短关键词 regex 在里面必然误报
|
||||
ext = (target.declared_ext or "").lower()
|
||||
if not ext and target.path:
|
||||
ext = os.path.splitext(target.path)[1].lower()
|
||||
if ext in self.BINARY_SKIP_EXTS:
|
||||
return result
|
||||
|
||||
try:
|
||||
text = self._extract_text(target.path, target.declared_ext)
|
||||
except Exception: # noqa: BLE001
|
||||
@@ -65,6 +97,14 @@ class KeywordEngine(Engine):
|
||||
if not text:
|
||||
return result
|
||||
|
||||
# ---- 预处理 ----
|
||||
# 默认按原始行匹配：把整篇文本的换行全部删除会让上一行行尾和下一行行首
|
||||
# 黏成一句话（Ruby 源码 "…breed\n+fetch()…" 曾被当成 curl 特征命中）。
|
||||
# cross_line: true 的规则再跑一遍"换行压成空格"的文本，用于兜住
|
||||
# wget \ + 换行 + -O payload 这类续行绕过；保留分隔符，不产生黏连。
|
||||
text_norm = text.replace("\r\n", "\n").replace("\r", "\n")
|
||||
text_joined = NEWLINE_RUN_RE.sub(" ", text_norm)
|
||||
|
||||
for rule in self._compiled:
|
||||
match_mode = rule.get("match", "contains")
|
||||
rule_name = rule.get("name", "keyword")
|
||||
@@ -77,23 +117,29 @@ class KeywordEngine(Engine):
|
||||
if match_mode == "regex":
|
||||
compiled = rule.get("_re")
|
||||
if compiled:
|
||||
for m in compiled.finditer(text):
|
||||
snippet = text[max(0, m.start() - 10): m.end() + 10]
|
||||
hits.append(snippet.strip())
|
||||
scan_targets = [text_norm]
|
||||
if rule.get("cross_line"):
|
||||
scan_targets.append(text_joined)
|
||||
for body in scan_targets:
|
||||
for m in compiled.finditer(body):
|
||||
snippet = body[max(0, m.start() - 10): m.end() + 10]
|
||||
hits.append(" ".join(snippet.split()))
|
||||
if len(hits) >= 3:
|
||||
break
|
||||
if len(hits) >= 3:
|
||||
break
|
||||
|
||||
elif match_mode == "any_of":
|
||||
keywords = rule.get("keywords", [])
|
||||
for kw in keywords:
|
||||
if kw.lower() in text.lower():
|
||||
if kw.lower() in text_norm.lower():
|
||||
hits.append(kw)
|
||||
if len(hits) >= 3:
|
||||
break
|
||||
|
||||
elif match_mode == "contains":
|
||||
pattern = rule.get("pattern", "")
|
||||
if pattern and (pattern.lower() in text.lower()):
|
||||
if pattern and (pattern.lower() in text_norm.lower()):
|
||||
hits.append(pattern)
|
||||
|
||||
if hits:
|
||||
|
||||
@@ -108,6 +108,9 @@ class StructureEngine(Engine):
|
||||
)
|
||||
|
||||
# 3) 危险可执行扩展名（顶层文件）
|
||||
# 这是「载体」信号而不是恶意行为：单独命中应当进人工复核（≥review_score），
|
||||
# 但不能成为把后续中/低置信度检测一起顶过 block_score 的底座，
|
||||
# 否则正常签名的 exe/dll 会被判恶意。
|
||||
if ext in DANGEROUS_EXTS and not target.is_extracted:
|
||||
severity = Severity.CRITICAL.value if ext in {".exe", ".scr", ".dll", ".com", ".pif"} else Severity.HIGH.value
|
||||
findings.append(
|
||||
@@ -117,7 +120,7 @@ class StructureEngine(Engine):
|
||||
severity=severity,
|
||||
rule="DangerousExecutableExtension",
|
||||
detail=f"文件类型为可执行/脚本格式：{ext}",
|
||||
weight=45.0 if severity == Severity.CRITICAL.value else 30.0,
|
||||
weight=35.0 if severity == Severity.CRITICAL.value else 30.0,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -131,6 +134,9 @@ class StructureEngine(Engine):
|
||||
# 4c) 伪装扩展名 & 双扩展名
|
||||
findings.extend(self._check_extension_mismatch(target, real_type, real_ext))
|
||||
|
||||
# 4d) 内嵌 EICAR 字节序列检测（ClamAV 只匹配精确 68B 纯 EICAR 的补充）
|
||||
findings.extend(_check_embedded_eicar(target.path, ext, result.meta.get("size", 0)))
|
||||
|
||||
# 5) LNK 快捷方式深度解析（Windows 专属，Linux 纵深防御）
|
||||
if ext == ".lnk" or real_ext == ".lnk":
|
||||
findings.extend(self._inspect_lnk(target))
|
||||
@@ -577,6 +583,7 @@ class StructureEngine(Engine):
|
||||
|
||||
# ZIP/OOXML 结构损坏（CRC 校验或 central directory）
|
||||
if real_lower.startswith("zip") or ext in {".docx", ".xlsx", ".pptx", ".xlsm", ".docm", ".pptm", ".zip", ".jar", ".apk"}:
|
||||
findings.extend(_check_zip_trailing_garbage(path))
|
||||
try:
|
||||
with zf.ZipFile(path, "r") as zfh:
|
||||
bad = zfh.testzip()
|
||||
@@ -647,19 +654,10 @@ class StructureEngine(Engine):
|
||||
filename = (target.filename or "").lower()
|
||||
|
||||
# 双扩展名：安全扩展名 + 危险扩展名结尾（如 invoice.pdf.exe）
|
||||
safe_then_dangerous = (
|
||||
filename.endswith(".exe") or filename.endswith(".dll") or filename.endswith(".scr")
|
||||
or filename.endswith(".com") or filename.endswith(".pif") or filename.endswith(".cpl")
|
||||
or filename.endswith(".msi") or filename.endswith(".lnk") or filename.endswith(".chm")
|
||||
or filename.endswith(".bat") or filename.endswith(".cmd") or filename.endswith(".ps1")
|
||||
or filename.endswith(".vbs") or filename.endswith(".hta")
|
||||
or filename.endswith(".elf") or filename.endswith(".so")
|
||||
or filename.endswith(".desktop") or filename.endswith(".run") or filename.endswith(".bin")
|
||||
or filename.endswith(".deb") or filename.endswith(".appimage")
|
||||
)
|
||||
safe_exts = {".txt", ".pdf", ".jpg", ".jpeg", ".png", ".gif", ".bmp", ".doc", ".docx",
|
||||
".xls", ".xlsx", ".ppt", ".pptx", ".rtf", ".csv", ".xml", ".html", ".htm",
|
||||
".zip", ".rar", ".7z", ".wps", ".et", ".dps", ".wpt", ".ett", ".dpt"}
|
||||
# 用 DANGEROUS_EXTS 做单一事实源，避免手工维护 endswith 链漏覆盖
|
||||
_dangerous_tuple = tuple(sorted(DANGEROUS_EXTS, key=len, reverse=True))
|
||||
safe_then_dangerous = filename.endswith(_dangerous_tuple)
|
||||
safe_exts = _SAFE_PREFIX_EXTS
|
||||
|
||||
if safe_then_dangerous:
|
||||
# 取最后两个扩展名
|
||||
@@ -685,7 +683,7 @@ class StructureEngine(Engine):
|
||||
return findings
|
||||
|
||||
# 真实类型危险可执行，声明安全类型 → 伪装
|
||||
from app.engines.structure import DANGEROUS_EXTS
|
||||
# DANGEROUS_EXTS 是模块级常量，此处直接引用
|
||||
if real_ext in DANGEROUS_EXTS and ext not in DANGEROUS_EXTS and ext:
|
||||
findings.append(
|
||||
Finding(
|
||||
@@ -1063,17 +1061,115 @@ def _is_path_traversal(name: str) -> bool:
|
||||
return ".." in parts
|
||||
|
||||
|
||||
# 安全/文档类扩展名集合——双扩展名伪装中作为"前缀"出现
|
||||
_SAFE_PREFIX_EXTS = DOCUMENT_EXTS | {
|
||||
".txt", ".log", ".csv", ".json", ".xml", ".html", ".htm",
|
||||
".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".svg",
|
||||
".zip", ".rar", ".7z", ".tar", ".gz", ".bz2",
|
||||
".wps", ".et", ".dps", ".wpt", ".ett", ".dpt", ".rtf", ".odt",
|
||||
}
|
||||
|
||||
|
||||
def _check_embedded_eicar(path: str, ext: str, size: int) -> List[Finding]:
|
||||
"""检测文件中是否内嵌 EICAR 测试病毒字节序列。
|
||||
|
||||
ClamAV 的 Eicar-Test-Signature 只匹配**精确 68 字节纯 EICAR 文件**——
|
||||
任何把 EICAR 前面/后面加哪怕一个字节的文件（比如加换行、嵌入大文件尾部）
|
||||
都不会被命中。本函数作为补充，对文本/代码文件扫描是否含有 EICAR 字节串。
|
||||
|
||||
检测范围限制在前 1MB（足够覆盖绝大多数场景，同时防止扫描超大文件）。
|
||||
"""
|
||||
findings: List[Finding] = []
|
||||
# 标准 EICAR 字节序列（注意：Python 字符串里 \ 是转义，这里用原始 bytes）
|
||||
EICAR_BYTES = b"X5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*"
|
||||
# 只对纯文本/代码类扩展名做（二进制文件里出现 EICAR 几乎不可能）
|
||||
TEXT_EXTS = {".txt", ".ps1", ".js", ".bat", ".cmd", ".vbs", ".sh", ".bash",
|
||||
".py", ".php", ".asp", ".jsp", ".html", ".htm", ".csv", ".log",
|
||||
".json", ".xml", ".yaml", ".yml", ".cfg", ".conf", ".ini", ".reg"}
|
||||
if ext not in TEXT_EXTS:
|
||||
return findings
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
head = fh.read(1024 * 1024) # 前 1MB
|
||||
if EICAR_BYTES in head:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine="structure", layer="L0",
|
||||
severity=Severity.HIGH.value, rule="EmbeddedEICAR",
|
||||
detail=f"文件内嵌 EICAR 测试病毒字节序列（文件大小 {size}B）",
|
||||
weight=35.0,
|
||||
)
|
||||
)
|
||||
except OSError:
|
||||
pass
|
||||
return findings
|
||||
|
||||
|
||||
def _check_zip_trailing_garbage(path: str) -> List[Finding]:
|
||||
"""检测 ZIP/OOXML 文件尾部是否有多余数据（假损坏/被篡改的典型模式）。
|
||||
|
||||
攻击者可能在 EOCD 记录 (PK\x05\x06) 之后追加任意字节，
|
||||
Python zipfile 能正常打开（只要中央目录完整），但这属于结构异常。
|
||||
"""
|
||||
findings: List[Finding] = []
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
fh.seek(0, 2)
|
||||
fsize = fh.tell()
|
||||
# 读尾部 64KB（EOCD 至少 22B，从后往前搜 PK\x05\x06）
|
||||
tail_size = min(64 * 1024, fsize)
|
||||
fh.seek(fsize - tail_size)
|
||||
tail = fh.read()
|
||||
|
||||
eocd_pos = tail.rfind(b"PK\x05\x06")
|
||||
if eocd_pos < 0:
|
||||
return findings # 没找到 EOCD，由其他 BadZipFile 路径处理
|
||||
|
||||
# EOCD 结构最小 22 字节，必须确保 22 字节完整 + comment_len 可读取
|
||||
if eocd_pos + 22 > len(tail):
|
||||
return findings # EOCD 被截断，不做判定
|
||||
|
||||
# EOCD comment_len 在偏移 20-21 (little-endian)
|
||||
eocd_comment_len = tail[eocd_pos + 20] | (tail[eocd_pos + 21] << 8)
|
||||
eocd_total = 22 + eocd_comment_len
|
||||
# comment_len 超过实际剩余大小也是异常，跳过
|
||||
if eocd_pos + eocd_total > len(tail):
|
||||
return findings
|
||||
|
||||
garbage = tail_size - eocd_pos - eocd_total
|
||||
|
||||
if garbage > 0:
|
||||
findings.append(
|
||||
Finding(
|
||||
engine="structure", layer="L0",
|
||||
severity=Severity.MEDIUM.value, rule="MalformedStructure",
|
||||
detail=f"ZIP 尾部 EOCD 后存在 {garbage} 字节多余数据，疑似篡改或假损坏",
|
||||
weight=25.0,
|
||||
)
|
||||
)
|
||||
except OSError:
|
||||
pass
|
||||
return findings
|
||||
|
||||
|
||||
def _has_double_ext(filename: str) -> bool:
|
||||
"""判断文件名是否包含两个以上扩展名。"""
|
||||
"""判断文件名是否为双扩展名伪装。
|
||||
|
||||
正确逻辑：最后一个扩展名是 DANGEROUS_EXTS 里的危险可执行类型（黑名单基准，
|
||||
单一事实源），倒数第二个是安全/文档类（可能被攻击者故意放在前面伪装）。
|
||||
例如 invoice.docx.js → .js 危险 + .docx 安全 → True。
|
||||
"""
|
||||
name = os.path.basename(filename)
|
||||
parts = name.split(".")
|
||||
if len(parts) < 3:
|
||||
return False
|
||||
prefix_ext = "." + parts[-2].lower()
|
||||
suffix_ext = "." + parts[-1].lower()
|
||||
if suffix_ext not in SUSPICIOUS_DOUBLE_EXT:
|
||||
# 核心：以 DANGEROUS_EXTS 为单一事实源
|
||||
if suffix_ext not in DANGEROUS_EXTS:
|
||||
return False
|
||||
return prefix_ext in DANGEROUS_EXTS or prefix_ext in SUSPICIOUS_DOUBLE_EXT
|
||||
# 前缀必须是安全/文档类，才算"伪装"
|
||||
return prefix_ext in _SAFE_PREFIX_EXTS
|
||||
|
||||
|
||||
def _compatible_exts(real_ext: str, real_type: str) -> set[str]:
|
||||
|
||||
@@ -52,7 +52,7 @@ class ScanTask(Base):
|
||||
size: Mapped[int] = mapped_column(Integer, default=0)
|
||||
sha256: Mapped[str] = mapped_column(String(64), index=True)
|
||||
md5: Mapped[str] = mapped_column(String(32), default="")
|
||||
real_type: Mapped[str] = mapped_column(String(256), default="")
|
||||
real_type: Mapped[str] = mapped_column(String(512), default="")
|
||||
|
||||
status: Mapped[str] = mapped_column(String(16), default="pending", index=True)
|
||||
mode: Mapped[str] = mapped_column(String(16), default="sync")
|
||||
@@ -71,6 +71,11 @@ class ScanTask(Base):
|
||||
back_populates="task", cascade="all, delete-orphan", lazy="selectin"
|
||||
)
|
||||
|
||||
__table_args__ = (
|
||||
# 控制台高频查询: WHERE channel = X ORDER BY created_at DESC LIMIT N
|
||||
Index("ix_scan_tasks_channel_created", "channel", "created_at", postgresql_using="btree"),
|
||||
)
|
||||
|
||||
|
||||
class ScanFinding(Base):
|
||||
__tablename__ = "scan_findings"
|
||||
|
||||
@@ -115,13 +115,14 @@ class Policy:
|
||||
"""YARA 规则权重映射：规则名 → weight(0-100)。
|
||||
|
||||
从 policy.yaml 的 yara_rule_weights.{critical/high/medium/low} 展开。
|
||||
档位权重固定：critical=85 / high=65 / medium=45 / low=25。
|
||||
档位权重固定：critical=85 / high=60 / medium=30 / low=10
|
||||
（与 rules/*.yar 的 meta.risk_level 分档、binary.py 默认权重一致）。
|
||||
"""
|
||||
weights = {
|
||||
"critical": 85.0,
|
||||
"high": 65.0,
|
||||
"medium": 45.0,
|
||||
"low": 25.0,
|
||||
"high": 60.0,
|
||||
"medium": 30.0,
|
||||
"low": 10.0,
|
||||
}
|
||||
result: Dict[str, float] = {}
|
||||
for tier, weight in weights.items():
|
||||
@@ -300,15 +301,18 @@ class PolicyStore:
|
||||
def _load_db_or_yaml(self) -> Policy:
|
||||
"""优先 DB，回退 YAML。"""
|
||||
try:
|
||||
from sqlalchemy import select
|
||||
from app.db import SessionLocal
|
||||
from app.models import PolicyVersion
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
pv = db.execute(
|
||||
select(PolicyVersion).order_by(PolicyVersion.created_at.desc())
|
||||
).scalar_one_or_none()
|
||||
# 用 ORM Query API — 直接返回 PolicyVersion 对象
|
||||
# 而不是 Row 对象（db.execute(select) 会返回 Row）
|
||||
pv = (
|
||||
db.query(PolicyVersion)
|
||||
.order_by(PolicyVersion.created_at.desc())
|
||||
.first() # 取最新一条，不管历史版本多少条
|
||||
)
|
||||
if pv:
|
||||
policy = Policy(
|
||||
version=pv.version,
|
||||
@@ -411,6 +415,8 @@ class PolicyEngine:
|
||||
5) 规则覆盖
|
||||
6) 权重累加评分
|
||||
7) 阈值裁决
|
||||
8) 复核区间上界校正
|
||||
9) 复核地板（步骤 5 声明 manual_review 的规则，只抬处置不抬分数）
|
||||
"""
|
||||
import logging
|
||||
|
||||
@@ -563,14 +569,15 @@ class PolicyEngine:
|
||||
)
|
||||
|
||||
# 5) 规则覆盖优先处理
|
||||
override_findings: List[Finding] = []
|
||||
# action=reject 立即拒绝；action=manual_review 只登记"复核地板"，
|
||||
# 不往分数里注 65 —— 地板在步骤 9 生效。
|
||||
review_floor = False
|
||||
for finding in findings:
|
||||
override = policy.rule_overrides.get(finding.rule)
|
||||
if not override:
|
||||
continue
|
||||
action = override.get("action", "manual_review")
|
||||
reason = override.get("reason") or f"规则 {finding.rule} 命中策略"
|
||||
override_findings.append(finding)
|
||||
triggered.append(
|
||||
{
|
||||
"type": "rule_override",
|
||||
@@ -590,12 +597,10 @@ class PolicyEngine:
|
||||
triggered=triggered,
|
||||
)
|
||||
if action == "manual_review":
|
||||
score = max(score, 65.0)
|
||||
review_floor = True
|
||||
|
||||
# 6) 其余信号按权重累加
|
||||
# 6) 全部信号按权重累加
|
||||
for finding in findings:
|
||||
if finding in override_findings:
|
||||
continue
|
||||
score += finding.weight
|
||||
if finding.severity in {Severity.HIGH.value, Severity.CRITICAL.value}:
|
||||
reasons.append(f"{finding.engine}：{finding.detail}")
|
||||
@@ -619,6 +624,12 @@ class PolicyEngine:
|
||||
if score >= policy.block_score:
|
||||
verdict, action = "malicious", "reject"
|
||||
|
||||
# 9) 复核地板：策略里声明 manual_review 的规则保证"不低于人工复核"，
|
||||
# 与权重数值和 review_score 阈值解耦——调高 review_score 下界或调低
|
||||
# 这些规则的权重，都不会让它们覆盖的文件类型静默放行。
|
||||
if review_floor and action in ("pass", "pass_with_flag"):
|
||||
verdict, action = "suspicious", "manual_review"
|
||||
|
||||
reason = self._build_reason(verdict, reasons, score)
|
||||
return Decision(
|
||||
verdict=verdict,
|
||||
@@ -750,7 +761,9 @@ class PolicyEngine:
|
||||
if any("url" in f.rule.lower() or "link" in f.rule.lower() or "qr" in f.rule.lower()
|
||||
for f in findings):
|
||||
factors.append("external_url")
|
||||
if ext in {".exe", ".dll", ".elf", ".so", ".desktop", ".appimage", ".run", ".bin", ".deb", ".rpm"}:
|
||||
if ext in {".exe", ".dll", ".elf", ".so", ".desktop", ".appimage", ".run", ".bin", ".deb", ".rpm",
|
||||
".msi", ".msp", ".com", ".pif", ".scr", ".cpl", ".jar", ".war",
|
||||
".vbs", ".vbe", ".js", ".jse", ".ps1", ".bat", ".cmd", ".hta", ".lnk", ".cab", ".appx"}:
|
||||
factors.append("contains_executable")
|
||||
if ext == ".iso":
|
||||
factors.append("is_mountable_image")
|
||||
@@ -771,6 +784,14 @@ class PolicyEngine:
|
||||
advice = "文件结构正常，无特别提示"
|
||||
requires_macro = False
|
||||
|
||||
# 裁决已经是恶意/引擎不可判定时，"打开风险低"是自相矛盾的对外文案：
|
||||
# 摆渡系统会把这段展示给用户，等于告诉用户"被拦下的文件可以放心打开"。
|
||||
if verdict in {"malicious", "error", "reject"} and level != "critical":
|
||||
level = "critical"
|
||||
factors.append("verdict_rejected")
|
||||
advice = "该文件已被判定为恶意或不可判定，请勿打开，联系管理员处理"
|
||||
requires_macro = False
|
||||
|
||||
return {
|
||||
"level": level,
|
||||
"factors": factors,
|
||||
|
||||
@@ -34,6 +34,17 @@ _settings = get_settings()
|
||||
TEMP_ROOT = os.environ.get("MESHSAFE_TEMP_ROOT", "/tmp/meshsafe")
|
||||
|
||||
|
||||
def _clip(value: Any, limit: int) -> str:
|
||||
"""按列宽截断字符串。
|
||||
|
||||
外部可控的长度（libmagic 描述串、归档成员路径、用户文件名）一旦超出列宽，
|
||||
PostgreSQL 会拒绝整条 INSERT，连带同事务里的 scan_findings 一起丢失——
|
||||
被拒绝的文件反而查不到记录。宁可截断，也不能整条不落库。
|
||||
"""
|
||||
text = str(value or "")
|
||||
return text[:limit]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScanContext:
|
||||
"""一次扫描任务的上下文。"""
|
||||
@@ -95,12 +106,12 @@ class ScanOutcome:
|
||||
id=self.task_id,
|
||||
channel=self.channel,
|
||||
client_id=self.client_id,
|
||||
filename=self.file_info.get("filename", ""),
|
||||
declared_ext=self.file_info.get("declared_ext", ""),
|
||||
filename=_clip(self.file_info.get("filename", ""), 512),
|
||||
declared_ext=_clip(self.file_info.get("declared_ext", ""), 32),
|
||||
size=self.file_info.get("size", 0),
|
||||
sha256=self.file_info.get("sha256", ""),
|
||||
md5=self.file_info.get("md5", ""),
|
||||
real_type=self.file_info.get("real_type", ""),
|
||||
real_type=_clip(self.file_info.get("real_type", ""), 512),
|
||||
status="completed",
|
||||
mode=self.mode,
|
||||
verdict=self.verdict,
|
||||
@@ -122,7 +133,7 @@ class ScanOutcome:
|
||||
severity=f.severity,
|
||||
rule=f.rule,
|
||||
detail=f.detail,
|
||||
extracted_path=f.extracted_path,
|
||||
extracted_path=_clip(f.extracted_path, 1024),
|
||||
weight=f.weight,
|
||||
))
|
||||
|
||||
@@ -159,21 +170,37 @@ class ScanOutcome:
|
||||
)
|
||||
|
||||
def to_response(self, include_details: bool = True) -> Dict[str, Any]:
|
||||
"""对外接口响应组装。
|
||||
|
||||
设计原则：调用方拿了就用，不暴露内部权重、严重度、引擎层、
|
||||
策略链等实现细节。保留追溯所需的 task_id / file 哈希 / 时间戳。
|
||||
|
||||
对外暴露 4 个决策驱动字段：
|
||||
- action: 调用方直接据此决定放行/拦截/转人工
|
||||
- reason: 人可读的判定理由
|
||||
- task_id: 用于后续查询 / 审计追溯
|
||||
- file.*: 文件哈希与真实类型，用于追溯举证
|
||||
"""
|
||||
# file_info 里挑出调用方追溯需要的字段，其他（real_ext/declared_ext 等）砍
|
||||
fi = self.file_info or {}
|
||||
file_out = {k: fi.get(k) for k in (
|
||||
"filename", "sha256", "md5", "size", "real_type",
|
||||
) if fi.get(k) is not None}
|
||||
|
||||
# meta 只留耗时与时间戳，砍掉 engines/engine_errors/clamav/policy_version
|
||||
meta_out = {}
|
||||
for k in ("scan_duration_ms", "scanned_at"):
|
||||
if self.meta and k in self.meta:
|
||||
meta_out[k] = self.meta[k]
|
||||
|
||||
payload: Dict[str, Any] = {
|
||||
"task_id": self.task_id,
|
||||
"verdict": self.verdict,
|
||||
"score": self.score,
|
||||
"action": self.action,
|
||||
"reason": self.reason,
|
||||
"status": "completed",
|
||||
"file": self.file_info,
|
||||
"meta": self.meta,
|
||||
"open_risk": self.open_risk,
|
||||
"file": file_out,
|
||||
"meta": meta_out,
|
||||
}
|
||||
if include_details:
|
||||
payload["findings"] = [f.to_dict() for f in self.findings]
|
||||
payload["extracted"] = self.extracted
|
||||
payload["triggered"] = self.triggered
|
||||
return payload
|
||||
|
||||
|
||||
|
||||
@@ -172,10 +172,10 @@
|
||||
<h3>权重分档说明（来自 policy.yaml → yara_rule_weights）</h3>
|
||||
<div style="font-size:13px;color:#555;line-height:1.8;">
|
||||
<span class="chip chip-zt">critical → weight=85</span>
|
||||
<span class="chip chip-warn">high → weight=65</span>
|
||||
<span class="chip chip-info">medium → weight=45</span>
|
||||
<span class="chip chip-ok">low → weight=25</span>
|
||||
<span style="color:#888;margin-left:12px;">未配置的规则默认 weight=50</span>
|
||||
<span class="chip chip-warn">high → weight=60</span>
|
||||
<span class="chip chip-info">medium → weight=30</span>
|
||||
<span class="chip chip-ok">low → weight=10</span>
|
||||
<span style="color:#888;margin-left:12px;">未列出的规则沿用 rules/*.yar 中 meta.risk_level 推导出的同档权重</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -67,8 +67,6 @@ zero_tolerance_rules:
|
||||
- ReverseShellPowershell # PowerShell 反弹 Shell (-enc)
|
||||
- CredentialStealer # 凭据窃取工具 (mimikatz/sekurlsa/wce/...)
|
||||
- CredentialExport # 凭据导出命令 (reg save/reg export/...)
|
||||
- RemoteDownloadUnix # Unix 远程下载执行 (curl|wget|certutil)
|
||||
- RemoteDownloadPowerShell # PowerShell 远程下载执行 (IWR/DownloadString)
|
||||
|
||||
# ---------- 文件类型白名单 ----------
|
||||
# 只允许这些类型通过。空列表表示不限制（不推荐生产使用）。
|
||||
@@ -217,6 +215,12 @@ sandbox_required_extensions:
|
||||
# rule_name: [action, reason_override]
|
||||
# action 取值：pass / pass_with_flag / manual_review / reject / reject_and_notify
|
||||
rule_overrides:
|
||||
# 光闸口径：EXE/DLL/脚本一律不自动放行，最低人工复核。
|
||||
# 用 override 而不是靠 weight=30 恰好等于 review_score 下界来维持——
|
||||
# 数值巧合意味着把下界调到 >35 或把权重降一点，干净可执行文件就会静默放行。
|
||||
DangerousExecutableExtension:
|
||||
action: manual_review
|
||||
reason: "可执行/脚本类型文件，按光闸口径最低转人工复核"
|
||||
VBAShellInvocation:
|
||||
action: manual_review
|
||||
reason: "宏调用 Shell 但无自动执行入口，需人工复核"
|
||||
@@ -255,58 +259,60 @@ notification:
|
||||
max_reason_length: 500
|
||||
|
||||
# ---------- YARA 规则权重映射 ----------
|
||||
# 按规则名精细设定权重（0-100），覆盖 binary.py 里的默认 weight=50
|
||||
# 命中但不在下列名单中的规则 weight 保持默认值 50
|
||||
# 严重度分档由规则内 meta.risk_level 决定（critical/high/medium/low）
|
||||
# 档位权重与 rules/*.yar 的 meta.risk_level 一一对应（由 binary.py 读取 meta 得到
|
||||
# 默认严重度，再用本表覆盖为固定权重，保证规则文件与策略只有一处真相）。
|
||||
# critical=85（单独命中即 reject）/ high=60 / medium=30 / low=10
|
||||
# 只有 critical 一档可以独立越过 block_score=70；其余档位需与其他信号叠加。
|
||||
# 不在本表的规则：沿用 meta.risk_level 推导出的默认权重。
|
||||
yara_rule_weights:
|
||||
critical: # weight=85
|
||||
- C3_PHP_Webshell_Oneliner
|
||||
- C3_JSP_Webshell_RuntimeExec
|
||||
- C3_Webshell_Behinder
|
||||
- C1_PE_ProcessInjection
|
||||
- C8_Ransomware_Encryption
|
||||
- C8_Ransomware_LockBit
|
||||
- C8_Ransomware_BlackCat
|
||||
- C8_Ransomware_Conti
|
||||
- C1_PE_RAT_CommandControl
|
||||
- C3_JSP_Webshell_Deserialization
|
||||
- C3_JSP_Webshell_RuntimeExec
|
||||
- C3_PHP_Webshell_Oneliner
|
||||
- C3_Webshell_Behinder
|
||||
- C3_Webshell_CommandExecutionEcho
|
||||
- C3_Webshell_Godzilla
|
||||
- C7_Linux_ReverseShell_Bash
|
||||
- C7_Linux_ReverseShell_NC
|
||||
- C7_Linux_Rootkit_KernelModule
|
||||
high: # weight=65
|
||||
- C8_Ransomware_BlackCat
|
||||
- C8_Ransomware_Conti
|
||||
- C8_Ransomware_Encryption
|
||||
- C8_Ransomware_LockBit
|
||||
high: # weight=60
|
||||
- C1_PE_DLLHijacking
|
||||
- C1_PE_Downloader_Loader
|
||||
- C1_PE_LOLBin_Usage
|
||||
- C3_JSP_Webshell_MemoryShell
|
||||
- C3_PHP_Webshell_Obfuscated
|
||||
- C3_PHP_Webshell_VariableDeformation
|
||||
- C3_Webshell_CommandExecutionEcho
|
||||
- C3_Webshell_Godzilla
|
||||
- C1_PE_Downloader_Loader
|
||||
- C1_PE_RAT_CommandControl
|
||||
- C1_PE_LOLBin_Usage
|
||||
- C1_PE_Mutex_Persistence
|
||||
- C7_Linux_ReverseShell_Python
|
||||
- C7_Linux_ReverseShell_Perl
|
||||
- C7_Linux_ReverseShell_Socat
|
||||
- C7_Linux_Persistence_Crontab
|
||||
- C7_Linux_Persistence_SystemD
|
||||
- C3_Webshell_ASP_Classic
|
||||
- C3_Webshell_ASPNET
|
||||
- C3_Webshell_AntSword
|
||||
- C7_Linux_Persistence_AuthorizedKeys
|
||||
- C7_Linux_Persistence_Crontab
|
||||
- C7_Linux_Persistence_LD_PRELOAD
|
||||
- C8_Ransomware_VolumeShadow
|
||||
- C8_Ransomware_Note_Filename
|
||||
- C7_Linux_Persistence_SystemD
|
||||
- C7_Linux_ReverseShell_Perl
|
||||
- C7_Linux_ReverseShell_Python
|
||||
- C7_Linux_ReverseShell_Socat
|
||||
- C8_Mining_Monero
|
||||
- C8_Ransomware_Ryuk_Pseudo
|
||||
medium: # weight=45
|
||||
- C9_Packer_UPX
|
||||
- C9_Packer_Themida_VMP
|
||||
- C1_PE_DLLHijacking
|
||||
- C8_Ransomware_VolumeShadow
|
||||
medium: # weight=30
|
||||
- C1_PE_EntropyHighNetwork
|
||||
- C1_PE_Mutex_Persistence
|
||||
- C7_Linux_Miner_Generic
|
||||
- C7_Linux_ReverseShell_Misc
|
||||
- C7_Misc_Ruby_Lua_ReverseShell
|
||||
- C7_Network_Tools_Malicious
|
||||
- C3_Webshell_AntSword
|
||||
- C3_Webshell_ASPNET
|
||||
- C3_Webshell_ASP_Classic
|
||||
- C3_JSP_Webshell_Deserialization
|
||||
- C3_JSP_Webshell_MemoryShell
|
||||
- C3_Webshell_CommandExecutionEcho
|
||||
- C8_Ransomware_Note_Filename
|
||||
- C9_Packer_Themida_VMP
|
||||
low: # weight=10
|
||||
- C8_Ransomware_Extension_Blacklist
|
||||
low: # weight=25
|
||||
- C7_Linux_ReverseShell_Misc
|
||||
- C9_Packer_UPX
|
||||
|
||||
|
||||
# ---------- 内网特征词表引擎（P1）----------
|
||||
@@ -339,25 +345,55 @@ keyword_rules:
|
||||
# 恶意行为特征词表（覆盖 CSV L3 全部 25 条用例）
|
||||
# ================================================================
|
||||
|
||||
# ---- 远程下载执行 ----
|
||||
# ---- 远程下载即执行（要有完整动作链，裸词不算；README 里的安装命令进复核而非直接拒）----
|
||||
# cross_line：命令换行/续行是本类特征最常见的绕过（curl \ + 下一行 | sh），
|
||||
# 拼接后仍要求"下载工具 + 执行动作"同框，散文里的多行文本不会凑出这条链。
|
||||
- name: RemoteDownloadExec
|
||||
match: regex
|
||||
pattern: "(?:\\b(?:curl|wget)\\b[^\\n]{0,120}\\|\\s*(?:ba|z|k|da)?sh\\b|\\b(?:curl|wget)\\b[^\\n]{0,120}(?:&&|;)\\s*(?:chmod\\s+\\+x|\\./)|(?:^|[^\\w$-])(?:iex|invoke-expression)\\b[^\\n]{0,60}(?:\\biwr\\b|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method|downloadstring)|\\b(?:iwr|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method)\\b[^\\n]{0,120}\\|\\s*(?:iex|invoke-expression)\\b|certutil\\s+[-/]urlcache[^\\n]{0,60}\\.exe\\b|mkfifo[^\\n]{0,60}(?:\\bnc\\s|\\bnetcat\\b))"
|
||||
ignore_case: true
|
||||
cross_line: true
|
||||
weight: 40
|
||||
severity: critical
|
||||
|
||||
# ---- 下载并落地成可执行文件（dropper 行为，比"下载即执行"弱一档但仍高危）----
|
||||
- name: PayloadDropToDisk
|
||||
match: regex
|
||||
pattern: "(?:\\b(?:curl|wget)\\b[^\\n]{0,120}(?:-o|-O|--output)\\s+\\S*\\.(?:exe|dll|ps1|bat|cmd|vbs|jar|msi|so|bin)\\b|\\b(?:i\\s*[-_]?\\s*wr|invoke\\s*[-_]?\\s*web\\s*[-_]?\\s*request|invoke\\s*[-_]?\\s*rest\\s*[-_]?\\s*method)\\b[^\\n]{0,120}-o(?:utfile)?\\s+\\S*\\.(?:exe|dll|ps1|bat|cmd|vbs)\\b|\\b(?:new-object|activexobject)\\s+system\\.net\\.webclient[^\\n]{0,80}downloadfile)"
|
||||
ignore_case: true
|
||||
cross_line: true
|
||||
weight: 45
|
||||
severity: critical
|
||||
|
||||
# ---- 外部下载动作（cmdlet/API + 无执行链）：计分进复核，不再零容忍 ----
|
||||
# cmdlet 名（Invoke-WebRequest/Invoke-RestMethod/BitsAdmin）是 PowerShell 专有写法，
|
||||
# 单独出现即进复核。.NET/Win32 的方法名（DownloadString/DownloadFile）不一样——
|
||||
# 它们存在于所有托管程序的标识符表里，裸匹配实测把 nvidiaInspector.exe 的
|
||||
# `add_DownloadFileCompleted`、`DownloadFileAsync` 各算一次命中，白送 30 分，
|
||||
# 把 45 分的人工复核顶成 75 分拒绝。所以方法名必须与 URL/IP 同框才算下载。
|
||||
- name: RemoteDownloadPowerShell
|
||||
match: regex
|
||||
pattern: "(Invoke-WebRequest|Invoke-RestMethod|DownloadString|DownloadFile|Start-BitsTransfer|BitsAdmin)"
|
||||
ignore_case: true
|
||||
weight: 35
|
||||
severity: high
|
||||
|
||||
- name: RemoteDownloadUnix
|
||||
match: regex
|
||||
pattern: "(\\bwget\\b|\\bcurl\\b|\\bcertutil\\s+-urlcache\\s+-split|\\bfetch\\b)"
|
||||
pattern: "(?:\\b(?:Invoke-WebRequest|Invoke-RestMethod|Start-BitsTransfer|BitsAdmin)\\b|\\bDownload(?:String|File)\\b[^\\n]{0,40}(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})|(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})[^\\n]{0,40}\\bDownload(?:String|File)\\b)"
|
||||
ignore_case: true
|
||||
weight: 30
|
||||
severity: high
|
||||
severity: medium
|
||||
|
||||
# ---- 下载工具 + 明确地址：只认"工具与 URL/IP 同框"，裸 -o 参数不算 ----
|
||||
# 旧写法里的 `curl|wget -O/-o 任意值` 会命中任何写成两段的安装文档，去掉。
|
||||
- name: RemoteDownloadUnix
|
||||
match: regex
|
||||
pattern: "(?:\\b(?:curl|wget|certutil)\\b[^\\n]{0,80}(?:https?://|\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}))"
|
||||
ignore_case: true
|
||||
weight: 30
|
||||
severity: medium
|
||||
|
||||
# ---- 反弹 Shell ----
|
||||
- name: ReverseShellNetcat
|
||||
match: regex
|
||||
pattern: "(\\bnc\\b|\\bncat\\b|\\bnetcat\\b|/dev/tcp/|bash\\s+-i\\s+>&|sh\\s+-i\\s+>&)"
|
||||
# 只认「执行原语」形态：nc/ncat/netcat 带 -e/-c（把 shell 交给连接），
|
||||
# 或 /dev/tcp、bash -i >& 这类内核级反连。裸 nc / nc -l / nc -z 是运维常用，
|
||||
# 出现在速查文档和二进制字符串表里必然误报，不计分。
|
||||
pattern: "(\\b(?:nc|ncat|netcat)\\s+(?:[^\\n]{0,60}\\s)?-[ec]\\w*\\s*(?:/bin/|/usr/|cmd\\.exe\\b|\\S*sh\\b)|/dev/tcp/|bash\\s+-i\\s+>&|sh\\s+-i\\s+>&)"
|
||||
ignore_case: true
|
||||
weight: 40
|
||||
severity: critical
|
||||
@@ -372,7 +408,8 @@ keyword_rules:
|
||||
# ---- 凭据窃取工具 ----
|
||||
- name: CredentialStealer
|
||||
match: regex
|
||||
pattern: "(mimikatz|sekurlsa|wce|gsecdump|fgdump|lsadump|dcsync|dcsyncer|rubeus)"
|
||||
# 全部加 \b 词边界, wce 要求后跟 .exe / 空格 / / (二进制随机字节里的 WCE 不会触发)
|
||||
pattern: "(\\bmimikatz\\b|\\bsekurlsa\\b|\\bwce(\\.exe\\b|\\s|\\/)|\\bgsecdump\\b|\\bfgdump\\b|\\blsadump\\b|\\bdcsync\\b|\\bdcsyncer\\b|\\brubeus\\b)"
|
||||
ignore_case: true
|
||||
weight: 45
|
||||
severity: critical
|
||||
@@ -428,6 +465,32 @@ keyword_whitelist:
|
||||
- ".senmesh.com"
|
||||
- "repo.internal"
|
||||
- "mirror.internal"
|
||||
- rule: RemoteDownloadExec # 下载即执行：内网源站安装脚本按白名单放行
|
||||
allowed_keywords:
|
||||
- "192.168."
|
||||
- "10."
|
||||
- "172.16."
|
||||
- "172.17."
|
||||
- "172.18."
|
||||
- "172.19."
|
||||
- "172.2"
|
||||
- "172.3"
|
||||
- ".senmesh.com"
|
||||
- "repo.internal"
|
||||
- "mirror.internal"
|
||||
- rule: PayloadDropToDisk # 下载落地可执行文件：内网源站安装包放行
|
||||
allowed_keywords:
|
||||
- "192.168."
|
||||
- "10."
|
||||
- "172.16."
|
||||
- "172.17."
|
||||
- "172.18."
|
||||
- "172.19."
|
||||
- "172.2"
|
||||
- "172.3"
|
||||
- ".senmesh.com"
|
||||
- "repo.internal"
|
||||
- "mirror.internal"
|
||||
- rule: RemoteDownloadPowerShell # Invoke-WebRequest/BitsAdmin
|
||||
allowed_keywords:
|
||||
- "192.168."
|
||||
|
||||
@@ -133,6 +133,38 @@ services:
|
||||
start_period: 15s
|
||||
retries: 3
|
||||
|
||||
# ---------- Cron (定时任务: DB 归档 + VACUUM) ----------
|
||||
# 纯 Python 实现, 零额外依赖, 和 api 共享 meshsafe:latest 镜像
|
||||
# 任务:
|
||||
# 每天 03:00 archive_old_tasks.py --apply (清 90 天 detail / 365 天归档 / 幂等 TTL)
|
||||
# 每周一 03:10 VACUUM ANALYZE (维护表统计信息)
|
||||
cron:
|
||||
image: meshsafe:latest
|
||||
container_name: meshsafe-cron
|
||||
restart: unless-stopped
|
||||
networks: [meshsafe_net]
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
- MESHSAFE_ENV=prod
|
||||
- MESHSAFE_DB_URL=postgresql+psycopg2://meshsafe:${MESHSAFE_DB_PASSWORD:-meshsafe}@postgres:5432/meshsafe
|
||||
- MESHSAFE_MASTER_KEY=${MESHSAFE_MASTER_KEY:?MASTER_KEY_REQUIRED}
|
||||
# cron 不做控制台认证，但 Settings.admin_password 是必填字段，缺它 get_settings()
|
||||
# 就抛 ValidationError，归档与 VACUUM 全崩。给占位值，不把真实口令复制进 cron
|
||||
# 容器环境（docker inspect 可见）。
|
||||
- MESHSAFE_ADMIN_PASSWORD=cron-unused-placeholder
|
||||
volumes:
|
||||
- ./data/logs:/var/log/meshsafe # 归档日志共享目录
|
||||
- ./scripts:/app/scripts:ro # archive_old_tasks.py + cron_entrypoint.py
|
||||
entrypoint: ["python3", "/app/scripts/cron_entrypoint.py"]
|
||||
# 镜像层的 HEALTHCHECK 探测 localhost:8443（API 服务端口），cron 容器不监听该端口，
|
||||
# 因此状态恒为 unhealthy —— 真正调度器挂掉时这个信号完全失效。关掉它，
|
||||
# 存活与否由 restart: unless-stopped + 日志判断。
|
||||
healthcheck:
|
||||
disable: true
|
||||
|
||||
# ---------- Nginx（反向代理，内网 HTTP 明文）----------
|
||||
# 外部端口 44380 → 内部 80 → api:8443
|
||||
# 无 TLS，外层防护靠光闸
|
||||
|
||||
2724
docs/TEST_CASES.md
Normal file
2724
docs/TEST_CASES.md
Normal file
File diff suppressed because it is too large
Load Diff
163
docs/接口文档.md
163
docs/接口文档.md
@@ -2,7 +2,7 @@
|
||||
|
||||
| 项目 | 内容 |
|
||||
|---|---|
|
||||
| 文档版本 | v1.1（2026-10-27 对齐当前代码实现） |
|
||||
| 文档版本 | v1.2（2026-10-29 响应字段精简，只暴露调用方必需字段） |
|
||||
| 接口版本 | `/api/v1` |
|
||||
| 适用系统 | 光闸内外网文件摆渡系统 |
|
||||
| 保密级别 | **内部交付文档**，请勿公开传播 |
|
||||
@@ -286,51 +286,25 @@ signature = HMAC-SHA256(response_key, payload) → 小写十六进制
|
||||
```json
|
||||
{
|
||||
"task_id": "3f8a9c2e1b7d4e6f8a0b2c4d6e8f0a1b",
|
||||
"verdict": "malicious",
|
||||
"score": 85.0,
|
||||
"action": "reject",
|
||||
"reason": "检测到 VBA 宏 + DDEAUTO 自动执行",
|
||||
"status": "completed",
|
||||
"file": {
|
||||
"filename": "起诉状.docx",
|
||||
"sha256": "a1b2c3...",
|
||||
"md5": "d4e5f6...",
|
||||
"size": 284915,
|
||||
"real_type": "Microsoft Word 2007+",
|
||||
"real_ext": ".docx",
|
||||
"declared_ext": ".docx",
|
||||
"filename": "起诉状.docx"
|
||||
"real_type": "Microsoft Word 2007+"
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"engine": "document",
|
||||
"layer": "L2",
|
||||
"severity": "critical",
|
||||
"rule": "DDEAutoExecution",
|
||||
"detail": "检测到 DDE/DDEAUTO 字段：DDEAUTO c:\\windows\\system32\\cmd.exe",
|
||||
"extracted_path": "word/settings.xml",
|
||||
"weight": 65.0,
|
||||
"meta": {}
|
||||
}
|
||||
],
|
||||
"extracted": [
|
||||
{ "path": "word/vbaProject.bin", "filename": "vbaProject.bin", "size": 10240, "depth": 1, "sha256": "" }
|
||||
],
|
||||
"triggered": [
|
||||
{ "type": "rule_override", "rule": "DDEAutoExecution", "action": "reject", "engine": "document" }
|
||||
],
|
||||
"meta": {
|
||||
"scan_duration_ms": 1840,
|
||||
"engines": ["structure", "clamav", "document", "keywords", "yara", "binary"],
|
||||
"engine_errors": [],
|
||||
"clamav": { "version": "ClamAV 1.5.4", "signature_date": "2026-10-27" },
|
||||
"policy_version": "2026.10.1",
|
||||
"policy_sha256": "…",
|
||||
"scanned_objects": 3,
|
||||
"scanned_at": "2026-10-27T08:30:00+00:00"
|
||||
"scanned_at": "2026-10-27T08:30:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **设计原则**：接口只暴露调用方必须知道的字段，内部权重、严重度、引擎层、策略链等实现细节不对外返回。调用方拿到 `action` 即可直接决策，无需二次分析。
|
||||
|
||||
**响应 200（异步受理）**
|
||||
|
||||
```json
|
||||
@@ -350,11 +324,11 @@ if result["status"] != "completed":
|
||||
|
||||
action = result["action"]
|
||||
if action == "reject":
|
||||
reject_transfer(reason=result["reason"]) # 拒绝并向用户展示 reason
|
||||
reject_transfer(reason=result["reason"]) # 拦截，向用户展示 reason
|
||||
elif action == "manual_review":
|
||||
route_to_human_review(result["task_id"]) # 转人工
|
||||
route_to_human_review(result["task_id"]) # 暂不放行，转人工
|
||||
else:
|
||||
allow_transfer() # pass / pass_with_flag
|
||||
allow_transfer() # pass，正常放行
|
||||
```
|
||||
|
||||
---
|
||||
@@ -373,9 +347,7 @@ else:
|
||||
{
|
||||
"task_id": "…",
|
||||
"status": "running",
|
||||
"verdict": "pending",
|
||||
"action": "pending",
|
||||
"score": 0.0,
|
||||
"reason": ""
|
||||
}
|
||||
```
|
||||
@@ -506,93 +478,78 @@ else:
|
||||
|
||||
## 5. 数据模型
|
||||
|
||||
### 5.1 结论字段
|
||||
### 5.1 响应字段
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|---|---|---|
|
||||
| `task_id` | string | 任务标识，全局唯一 |
|
||||
| `verdict` | enum | 总体判定，见 §5.2 |
|
||||
| `score` | float | 风险评分，0~100 |
|
||||
| `action` | enum | 处置建议，见 §5.3 |
|
||||
| `task_id` | string | 任务标识，全局唯一，用于后续查询与审计追溯 |
|
||||
| `action` | enum | **调用方决策依据**，见 §5.2 |
|
||||
| `reason` | string | 人可读的判定理由，可直接展示给用户 |
|
||||
| `status` | enum | `pending` / `running` / `completed` / `failed` |
|
||||
| `file` | object | 文件追溯信息，见 §5.3 |
|
||||
| `meta` | object | 基础元信息，见 §5.4 |
|
||||
|
||||
### 5.2 verdict 取值
|
||||
### 5.2 action 取值（调用方唯一决策依据）
|
||||
|
||||
| 值 | 含义 | 调用方动作 |
|
||||
|---|---|---|
|
||||
| `pass` | 未发现风险或仅极轻微信号 | 正常放行 |
|
||||
| `manual_review` | 可疑，需人工判断 | **暂不放行**，转入人工复核流程 |
|
||||
| `reject` | 判定不可放行 | 拒绝传输，向用户展示 `reason` |
|
||||
| `pending` | 异步尚未完成 | 等待轮询 |
|
||||
|
||||
> **`action` 是调用方唯一需要判断的字段**，不需要关心内部权重、严重度、触发了几条规则。
|
||||
|
||||
### 5.3 file 结构
|
||||
|
||||
| 字段 | 说明 |
|
||||
|---|---|
|
||||
| `filename` | 原始文件名 |
|
||||
| `sha256` | 文件哈希，用于去重与追溯举证 |
|
||||
| `md5` | 文件哈希，兼容场景使用 |
|
||||
| `size` | 字节数 |
|
||||
| `real_type` | 内容判定的真实类型描述（如 `JPEG image data`） |
|
||||
|
||||
> 不对外暴露 `real_ext` / `declared_ext`（内部判定使用）。
|
||||
|
||||
### 5.4 meta 结构
|
||||
|
||||
| 字段 | 说明 |
|
||||
|---|---|
|
||||
| `scan_duration_ms` | 本次检测耗时（毫秒） |
|
||||
| `scanned_at` | 检测时间（UTC） |
|
||||
|
||||
> 不对外暴露 `engines` / `engine_errors` / `clamav` 版本 / `policy_version` / `policy_sha256` / `scanned_objects` 等内部实现细节。这些信息保留在服务端控制台与审计日志中，运维人员按需查询。
|
||||
|
||||
---
|
||||
|
||||
### 5.5 内部判定字段（不对外暴露）
|
||||
|
||||
以下字段存在于服务端内部（控制台、审计日志、DB 记录），不通过 API 返回。调用方无需关心。
|
||||
|
||||
**verdict（内部总体判定）**
|
||||
|
||||
| 值 | 含义 |
|
||||
|---|---|
|
||||
| `clean` | 未发现风险 |
|
||||
| `suspicious` | 存在可疑信号，需要关注或人工判断 |
|
||||
| `suspicious` | 存在可疑信号 |
|
||||
| `malicious` | 判定为恶意 |
|
||||
| `error` | 检测过程异常（如引擎不可用），结论不可信 |
|
||||
| `error` | 检测过程异常 |
|
||||
| `pending` | 尚未完成 |
|
||||
|
||||
### 5.3 action 取值
|
||||
**score（内部权重和，0~100）** — 调用方不做二次分析，不对外暴露。
|
||||
|
||||
| 值 | 含义 | 建议调用方动作 |
|
||||
|---|---|---|
|
||||
| `pass` | 未发现问题 | 正常放行 |
|
||||
| `pass_with_flag` | 存在低风险信号 | 放行，但建议记录标记 |
|
||||
| `manual_review` | 需人工判断 | **暂不放行**，转入人工复核流程 |
|
||||
| `reject` | 判定不可放行 | 拒绝传输，向用户展示 `reason` |
|
||||
| `pending` | 尚未裁决 | 等待异步结果 |
|
||||
**findings（内部命中信号列表）** — 每条含 engine / layer / severity / rule / detail / weight / meta 等字段，供安全团队审计与问题追溯。不对外暴露。
|
||||
|
||||
> **`action` 与 `verdict` 解耦使用。** 业务逻辑请依据 `action`，`verdict` 供人查看与统计。
|
||||
**triggered（内部策略链触发记录）** — rule_override / zero_tolerance 等，不对外暴露。
|
||||
|
||||
### 5.4 findings 结构
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|---|---|---|
|
||||
| `engine` | string | 引擎名：`structure` / `extractor` / `clamav` / `document` / `yara` / `binary` |
|
||||
| `layer` | string | 检测层：`L0` / `L1` / `L2` / `L2B` / `L3` |
|
||||
| `severity` | enum | `info` / `low` / `medium` / `high` / `critical` |
|
||||
| `rule` | string | 规则标识 |
|
||||
| `detail` | string | 命中详情 |
|
||||
| `extracted_path` | string | 信号来自哪个内部路径（顶层文件为空） |
|
||||
| `weight` | float | 该信号的风险权重 |
|
||||
| `meta` | object | 引擎附加信息 |
|
||||
|
||||
### 5.5 严重度与权重对照
|
||||
|
||||
| 严重度 | 典型含义 | 权重区间 |
|
||||
|---|---|---|
|
||||
| `critical` | 确凿的恶意行为，自动执行、外联、路径穿越 | 50 ~ 90 |
|
||||
| `high` | 高度可疑，宏调用、危险 API、高熵加壳 | 30 ~ 80 |
|
||||
| `medium` | 可疑但存在合理场景，外部链接、可疑导入 | 15 ~ 45 |
|
||||
| `low` | 轻微信号，仅编码字符串、无签名 | 5 ~ 20 |
|
||||
| `info` | 信息性提示 | 0 |
|
||||
|
||||
### 5.6 file 结构
|
||||
|
||||
| 字段 | 说明 |
|
||||
|---|---|
|
||||
| `sha256` / `md5` | 文件哈希，用于去重与追溯 |
|
||||
| `size` | 字节数 |
|
||||
| `real_type` | 内容判定的真实类型描述 |
|
||||
| `real_ext` | 真实扩展名（由内容推导） |
|
||||
| `declared_ext` | 文件名声明的扩展名 |
|
||||
| `filename` | 原始文件名 |
|
||||
|
||||
### 5.7 meta 结构
|
||||
|
||||
| 字段 | 说明 |
|
||||
|---|---|
|
||||
| `scan_duration_ms` | 检测耗时（毫秒） |
|
||||
| `engines` | 参与的引擎列表 |
|
||||
| `engine_errors` | 不可用的引擎列表，非空说明结论可能不完整 |
|
||||
| `clamav` | ClamAV 版本与签名库编号 |
|
||||
| `policy_version` | 判定所依据的策略版本 |
|
||||
| `policy_sha256` | 策略文件哈希 |
|
||||
| `scanned_objects` | 实际检测的对象数（含递归解包出的子文件） |
|
||||
| `scanned_at` | 检测时间（UTC） |
|
||||
|
||||
> **`meta.clamav` 与 `meta.policy_version` 建议留存**，用于检测结果的有效性举证。
|
||||
**open_risk / extracted（内部辅助分析与解包过程）** — 不对外暴露。
|
||||
|
||||
---
|
||||
|
||||
## 6. 检测规则清单
|
||||
|
||||
以下规则命中时会出现在 `findings` 中。**规则清单公开用于解释结论，不含检测规则的具体实现内容。**
|
||||
以下规则命中时会在服务端审计日志中留下记录，调用方通过 `reason` 字段获得人可读的结论说明。**规则清单公开用于解释结论，不含检测规则的具体实现内容。**
|
||||
|
||||
### 6.1 L0 结构层
|
||||
|
||||
|
||||
@@ -193,7 +193,7 @@ rule_overrides:
|
||||
|---|---|
|
||||
| `pass` | 放行 |
|
||||
| `pass_with_flag` | 放行并标记 |
|
||||
| `manual_review` | 转人工复核（同时把评分提升至 65，确保落入复核区间） |
|
||||
| `manual_review` | 处置地板：该文件最终不低于人工复核（评分不变，低于复核区间时只抬处置） |
|
||||
| `reject` | 拒绝 |
|
||||
| `reject_and_notify` | 拒绝并通知（当前实现等同 `reject`） |
|
||||
|
||||
@@ -201,7 +201,9 @@ rule_overrides:
|
||||
|
||||
1. 命中 `rule_overrides` 的规则会**优先于评分**处理
|
||||
2. 若 `action` 为 `reject` 类 → 立即返回拒绝结论，不再累加评分
|
||||
3. 若 `action` 为 `manual_review` → 评分提升至 65 并继续后续裁决
|
||||
3. 若 `action` 为 `manual_review` → 登记"复核地板"，规则自身权重照常累加，评分累加与阈值裁决完成后，处置若低于 `manual_review` 才抬到 `manual_review`（已判 `reject` 的不会被降回来）
|
||||
|
||||
> 旧实现是"评分提升至 65"，有两个副作用：注进去的 65 分本身不是证据，再叠加任意 ≥5 分的噪声信号就会越过 `block_score=70` 变成拒绝；同时该规则自身的权重被跳过不再计分，控制台显示的分数与真实证据脱钩。现改为处置地板后，分数只反映权重，"不放行"由地板保证，与 `review_score` 阈值数值解耦——`DangerousExecutableExtension`（光闸口径：EXE/DLL/脚本不自动放行）正是靠这条地板生效，而不是靠 `weight=30/35` 恰好等于 `review_score` 下界。
|
||||
|
||||
**为什么用覆盖而不是纯评分**：像 `PathTraversal`、`DDEAutoExecution` 这类信号，一旦出现就足以判定不可放行，不应依赖其他信号累加来"凑够"阈值。
|
||||
|
||||
|
||||
@@ -117,17 +117,23 @@ rule C7_Linux_ReverseShell_Misc {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "多种语言反弹 Shell 通用特征"
|
||||
description = "通用反弹 Shell 标记：/dev/tcp 或 /dev/udp 或 reverse shell 工具名"
|
||||
description = "通用反弹 Shell 标记：/dev/udp 或 攻击工具名 + shell 参数"
|
||||
strings:
|
||||
$devudp = /\/dev\/udp\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\/\d{2,5}/ ascii
|
||||
$bashc = /bash\s+-c\s+['"].*\/dev\/(tcp|udp)/ ascii
|
||||
$dash_e = /\/bin\/bash\s+-e\s+['"]/ ascii
|
||||
$hydra = /Hydra|crackmap|weevely|meterpreter|msfconsole|msfvenom/ ascii nocase
|
||||
// 裸词 Hydra 会命中 "dehydrated"、crack 会命中 "cracklib"，必须带参数或成组出现
|
||||
$hydra = /\bhydra\s+(-l|-P|-p|-s|-m|-t)\b/ ascii nocase
|
||||
$tool = /\b(crackmapexec|weevely|meterpreter|msfconsole|msfvenom)\b/ ascii nocase
|
||||
$reverseshell = /reverse[_\s]?shell|shell[_\s]?shock/ ascii nocase
|
||||
$netart = /(nc|ncat)\s+\S+\s+\d{2,5}\s+-[ec]\b|\/dev\/(tcp|udp)\/|socat\s+EXEC:|bash\s+-i\s*>&/ ascii
|
||||
condition:
|
||||
any of ($devudp, $bashc, $dash_e) or any of ($hydra, $reverseshell)
|
||||
any of ($devudp, $bashc, $dash_e)
|
||||
or $hydra
|
||||
or ($tool and $netart)
|
||||
or ($reverseshell and $netart)
|
||||
}
|
||||
|
||||
|
||||
@@ -177,40 +183,43 @@ rule C7_Linux_Persistence_SystemD {
|
||||
meta:
|
||||
risk_level = "high"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1543.002 Systemd Service"
|
||||
description = "恶意 systemd unit 文件：ExecStart 指向 /tmp/.hidden + multi-user.target + 反弹命令"
|
||||
description = "恶意 systemd unit：ExecStart 指向临时目录隐藏文件 / 管道执行 / 反弹命令"
|
||||
strings:
|
||||
$unit = /\[Unit\]/ ascii
|
||||
$service = /\[Service\]/ ascii
|
||||
$install = /\[Install\]/ ascii
|
||||
$execstart = /ExecStart=/ ascii
|
||||
$multiuser = /multi-user\.target/ ascii
|
||||
$hiddenpath = /\/tmp\/\.?\w+|\/var\/tmp\/\.?\w+/ ascii
|
||||
$bashsh = /\/bin\/(ba)?sh\s+-c/ ascii
|
||||
// 隐藏文件（点开头）落在可写临时目录——正常 unit 只会在 /usr/bin、/opt 等路径
|
||||
$hiddenpath = /\/(tmp|var\/tmp|dev\/shm)\/\.[a-zA-Z0-9_-]{2,}/ ascii
|
||||
$pipe = /(curl|wget)\s+\S+\s*\|\s*(ba)?sh/ ascii
|
||||
$revshell = /((nc|ncat|socat)\s+.*(-e|--exec|EXEC:)\s*\/bin\/(ba)?sh|\/dev\/tcp\/\d{1,3}\.\d{1,3})/ ascii
|
||||
condition:
|
||||
// 完整 unit 片段 + 可疑路径
|
||||
$unit and $service and $execstart and ($hiddenpath or $bashsh)
|
||||
($unit and $service and $execstart and any of ($hiddenpath, $pipe, $revshell))
|
||||
or
|
||||
// Install multi-user.target + ExecStart 反弹
|
||||
$install and $multiuser and any of ($execstart, $bashsh)
|
||||
($install and $multiuser and $execstart and any of ($hiddenpath, $pipe, $revshell))
|
||||
}
|
||||
|
||||
rule C7_Linux_Persistence_LD_PRELOAD {
|
||||
meta:
|
||||
risk_level = "high"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1574.006 LD_PRELOAD"
|
||||
description = "LD_PRELOAD 劫持：环境变量注入 + .so 文件 + dlopen"
|
||||
description = "LD_PRELOAD 劫持：写 ld.so.preload 或预加载临时目录里的 .so"
|
||||
strings:
|
||||
$ldpreload = /LD_PRELOAD=/ ascii
|
||||
$sofile = /\.so(\.\d+)?:?/ ascii
|
||||
$dlopen = "dlopen" ascii
|
||||
$dlsym = "dlsym" ascii
|
||||
$exportenv = /export\s+LD_PRELOAD/ ascii
|
||||
$ldso = /ld\.so\.preload/ ascii
|
||||
$tmpso = /\/(tmp|var\/tmp|dev\/shm|proc\/\d+\/root)\/[a-zA-Z0-9_.\/-]*\.so(\.[0-9]+)?/ ascii
|
||||
condition:
|
||||
($ldpreload or $exportenv) and ($sofile or $dlopen or $dlsym)
|
||||
// /etc/ld.so.preload 是内核级后门常用落点，单独出现即算证据
|
||||
$ldso
|
||||
or
|
||||
// LD_PRELOAD 指向可写临时目录的 so；README 里 LD_PRELOAD=libasan.so 不算
|
||||
(any of ($ldpreload, $exportenv) and $tmpso)
|
||||
}
|
||||
|
||||
|
||||
@@ -267,41 +276,55 @@ rule C7_Misc_Ruby_Lua_ReverseShell {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "多语言反弹 Shell"
|
||||
description = "Ruby/Lua 反弹 Shell：TCPSocket + exec/system"
|
||||
description = "Ruby/Lua 反弹 Shell：Socket 对象 + 紧邻的 shell 执行调用"
|
||||
strings:
|
||||
$ruby = "TCPSocket" ascii
|
||||
$ruby2 = "Socket.tcp" ascii
|
||||
$lua = "loadstring" ascii
|
||||
$lua2 = "luasocket" ascii
|
||||
$exec = /(exec|system|IO\.popen)\s+.*\/bin\/(ba)?sh/ ascii
|
||||
$exec = /(\bexec|\bsystem|IO\.popen|\bpopen|os\.execute|os\.runtime)\s*\(?\s*['"]((\/bin\/)?(ba)?sh|nc\s+-e|\/dev\/tcp)[^'"]{0,48}/ ascii
|
||||
condition:
|
||||
any of ($ruby, $lua, $lua2) and ($exec or any of ($ruby, $lua, $lua2))
|
||||
// 只要出现 TCPSocket/loadstring 就命中 = 任何内嵌 Lua 的程序（Redis、nginx-lua、
|
||||
// Wireshark 插件）必中。语言标记与 shell 执行必须挨在一起（256B 内）。
|
||||
$exec and (
|
||||
for any i in (1..#exec): (
|
||||
any of ($ruby, $ruby2, $lua, $lua2) in (@exec[i] - 256..@exec[i] + 256)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C7-5 网络投毒 / MITM 工具（内网横向移动）
|
||||
// 工具名必须整词匹配，且与攻击参数成组出现
|
||||
// =============================================================================
|
||||
|
||||
rule C7_Network_Tools_Malicious {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C7"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "内网攻击工具"
|
||||
description = "内网攻击工具：ettercap/responder/nmap 扫描 + SMB/DNS 投毒"
|
||||
description = "内网攻击工具：ettercap/responder/nmap + 投毒参数成组出现"
|
||||
strings:
|
||||
$ettercap = "ettercap" ascii nocase
|
||||
$responder = /responder(\.py)?/ ascii nocase
|
||||
$nmap = "nmap" ascii nocase
|
||||
$smb = /--smb/ ascii
|
||||
$ms17 = "ms17-010" ascii nocase
|
||||
$smbghost = "SMBGhost" ascii nocase
|
||||
$dnsspoof = /dns[-_]spoof|dns[-_]mitm/ ascii nocase
|
||||
$arppoison = /arp[-_]poison|arp[-_]spoof/ ascii nocase
|
||||
$tool1 = /\bettercap\b/ ascii nocase
|
||||
$tool2 = /\bresponder(\.py)?\b/ ascii nocase
|
||||
$tool3 = /\bnmap\b/ ascii nocase
|
||||
// 攻击参数 / 漏洞编号（这些才是意图证据）
|
||||
$art1 = /--smb/ ascii
|
||||
$art2 = /\bms17-010\b/ ascii nocase
|
||||
$art3 = /\bSMBGhost\b/ ascii nocase
|
||||
$art4 = /\bdns[-_]spoof\b|\bdns[-_]mitm\b/ ascii nocase
|
||||
$art5 = /\barp[-_]poison\b|\barp[-_]spoof\b/ ascii nocase
|
||||
$art6 = /--Mitm|-T\s+text|\bec_sleep\b|\bntlm_relay\b|\bRunFinger\.py\b/ ascii nocase
|
||||
condition:
|
||||
any of ($ettercap, $responder, $dnsspoof, $arppoison)
|
||||
// 投毒手法名本身就是明确证据
|
||||
any of ($art4, $art5)
|
||||
or
|
||||
$nmap and ($smb or $ms17 or $smbghost)
|
||||
// 工具名 + 攻击参数
|
||||
($tool1 and any of ($art1, $art2, $art3, $art6))
|
||||
or ($tool2 and any of ($art1, $art2, $art3, $art6))
|
||||
or ($tool3 and any of ($art1, $art2, $art3))
|
||||
}
|
||||
|
||||
445
rules/pe.yar
445
rules/pe.yar
@@ -2,26 +2,34 @@
|
||||
// MeshSafe L2B YARA 规则集 —— Windows PE 恶意程序专项（C1 + C8）
|
||||
// =============================================================================
|
||||
//
|
||||
// 设计原则：
|
||||
// - 所有规则使用 2-3 特征组合命中（API 组合 + C2 URL + Mutex 持久化等）
|
||||
// - binary.py 的 PE 结构分析负责：节熵 + 签名 + 可疑导入（单 API）
|
||||
// - YARA 负责：多 API 组合 + 字符串组合 + 壳 + 勒索专项
|
||||
// - 命中后作为 finding 交给 PolicyEngine 权重累加，不直接零容忍
|
||||
// 设计原则（v0.2.4 起，为消除干净软件误报而重写）：
|
||||
// 1. 全文搜 API 名 = 必然误报。系统 DLL 会「导出」这些 API（kernel32 里就有
|
||||
// VirtualAllocEx/WriteProcessMemory/CreateRemoteThread 的明文字符串），
|
||||
// 安全 SDK、安装器、调试组件同理。因此 C1 系列一律要求：
|
||||
// - 文件是 PE（uint16(0) == 0x5A4D），且
|
||||
// - 相关 API 名出现在同一个窗口内（导入名表是连续的，窗口 256~512B）。
|
||||
// 导出表按字母序排列、跨度通常上 KB，落在窗口外 → 不误报；
|
||||
// 真注入器/下载器的导入名表三条名字通常挨在一起 → 命中。
|
||||
// 2. 不单独用「API 组合」定性，必须有硬指标（可疑路径 / 持久化键名 / C2 URL）。
|
||||
// 3. 不用 3~4 字母裸串（曾经的 $persist5 = "RUN"）和裸工具名。
|
||||
// 4. binary.py 的 PE 结构分析负责节熵/签名/导入 API 分级；YARA 只做组合特征。
|
||||
//
|
||||
// 范围：
|
||||
// - C1: Windows PE 恶意程序（注入器 / 下载器 / RAT / Loader）
|
||||
// - C8: 勒索 / 挖矿专项
|
||||
// - C9: 壳 / 签名 / 结构异常（binary.py 已覆盖大部分，YARA 补充壳特征）
|
||||
// - C8: 勒索 / 挖矿专项（勒索样本常以文本/脚本形态投递，不加 PE 头判定）
|
||||
// - C9: 壳特征（识别后建议送沙箱，不作为恶意证据）
|
||||
//
|
||||
// 维护：
|
||||
// - 新恶意家族出现时，优先补充"API 组合 + Mutex 名 + C2 URL"三件套
|
||||
// - 壳特征（UPX/Themida/VMP）可识别后自动送沙箱深检
|
||||
// - 新增规则先跑「干净 PE 回归」：C:/Windows/System32 全量 900+ 文件必须 0 命中
|
||||
// - meta.risk_level 会被 binary.py 读取用于严重度展示，必须与档位权重一致
|
||||
// =============================================================================
|
||||
|
||||
import "pe"
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-1 进程注入三件套（VirtualAllocEx + WriteProcessMemory + CreateRemoteThread）
|
||||
// binary.py 扫了 SuspiciousImports（单个 API），但 3 个同时出现 = 必然是注入器
|
||||
// 判定要点：三个导入名必须在同一张导入名表里（512B 窗口），否则是导出表明文
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_ProcessInjection {
|
||||
@@ -30,37 +38,66 @@ rule C1_PE_ProcessInjection {
|
||||
category = "C1"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1055 Process Injection"
|
||||
description = "PE 文件同时导入 进程注入三件套 API = 远控/注入器特征"
|
||||
description = "PE 导入表含进程注入三件套（同表紧邻）= 远控/注入器特征"
|
||||
strings:
|
||||
$s1 = "VirtualAllocEx" ascii
|
||||
$s2 = "WriteProcessMemory" ascii
|
||||
$s3 = "CreateRemoteThread" ascii
|
||||
// 备选：Nt 系列（NtUnmapViewOfSection + VirtualAllocEx + CreateRemoteThread）
|
||||
$n1 = "NtUnmapViewOfSection" ascii
|
||||
$n2 = "NtMapViewOfSection" ascii
|
||||
$n3 = "ZwUnmapViewOfSection" ascii
|
||||
// 也可能是 QueueUserAPC 注 APC 注入
|
||||
$alloc = "VirtualAllocEx" ascii
|
||||
$write = "WriteProcessMemory" ascii
|
||||
$thread = "CreateRemoteThread" ascii
|
||||
// 进程镂空（Process Hollowing）
|
||||
$unmap1 = "NtUnmapViewOfSection" ascii
|
||||
$unmap2 = "ZwUnmapViewOfSection" ascii
|
||||
// APC / Hook 注入
|
||||
$apc = "QueueUserAPC" ascii
|
||||
// 或 SetWindowsHookEx DLL 注入
|
||||
$hook = "SetWindowsHookEx" ascii
|
||||
$open = "OpenProcess" ascii
|
||||
condition:
|
||||
// 经典三件套（必须同时命中）
|
||||
$s1 and $s2 and $s3
|
||||
or
|
||||
// Nt 系列内存注入（至少 2 个 Nt + 1 个注入）
|
||||
(any of ($n1, $n2, $n3) and any of ($s1, $s3, $apc))
|
||||
or
|
||||
// APC 注入 + VirtualAllocEx
|
||||
$apc and $s1
|
||||
or
|
||||
// Hook DLL 注入
|
||||
$hook and $s1
|
||||
uint16(0) == 0x5A4D
|
||||
and
|
||||
(
|
||||
// 经典三件套：同表紧邻
|
||||
(
|
||||
$alloc and $write and $thread
|
||||
and for any i in (1..#alloc): (
|
||||
$write in (@alloc[i] - 512..@alloc[i] + 512)
|
||||
and $thread in (@alloc[i] - 512..@alloc[i] + 512)
|
||||
)
|
||||
)
|
||||
or
|
||||
// 镂空注入：unmap 必须和 write+thread 同表（kernel32/shell32 只是从
|
||||
// ntdll 导入成对的 NtMapView/NtUnmap，没有写入+远线程就不算镂空注入）
|
||||
(
|
||||
$write and $thread
|
||||
and for any i in (1..#unmap1): (
|
||||
$write in (@unmap1[i] - 512..@unmap1[i] + 512)
|
||||
and $thread in (@unmap1[i] - 512..@unmap1[i] + 512)
|
||||
)
|
||||
)
|
||||
or
|
||||
(
|
||||
$write and $thread
|
||||
and for any i in (1..#unmap2): (
|
||||
$write in (@unmap2[i] - 512..@unmap2[i] + 512)
|
||||
and $thread in (@unmap2[i] - 512..@unmap2[i] + 512)
|
||||
)
|
||||
)
|
||||
or
|
||||
// APC / Hook 注入：目标进程 API + 注入 API 同表紧邻
|
||||
(
|
||||
$alloc and $open and ($apc or $hook)
|
||||
and for any i in (1..#alloc): (
|
||||
$open in (@alloc[i] - 512..@alloc[i] + 512)
|
||||
and ($apc in (@alloc[i] - 512..@alloc[i] + 512)
|
||||
or $hook in (@alloc[i] - 512..@alloc[i] + 512))
|
||||
)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-2 下载器组合（网络 API + 本地持久化 / 执行）
|
||||
// 单一 URLDownloadToFile 不致命，但 "下载 + 执行 + 持久化" 三件套 = Loader
|
||||
// C1-2 下载器 / Loader（下载 API + 落地持久化键名，且两者同表紧邻）
|
||||
// v0.2.3 的写法：任意下载 API + "WriteFile" + "CreateProcess" + 裸串 "RUN"
|
||||
// → 任何带自动更新的正常软件必中。现在要求持久化硬指标。
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_Downloader_Loader {
|
||||
@@ -69,157 +106,172 @@ rule C1_PE_Downloader_Loader {
|
||||
category = "C1"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1105 Ingress Tool Transfer"
|
||||
description = "下载器组合：HTTP/FTP 下载 + 写入到可疑路径 + 执行"
|
||||
description = "下载 API 与自启动/服务注册持久化键名同表共存 = Loader"
|
||||
strings:
|
||||
// 网络下载 API（任选其一）
|
||||
// 下载类 API（保留专有名词，不用裸串）
|
||||
$net1 = "URLDownloadToFile" ascii
|
||||
$net2 = "InternetOpenUrl" ascii
|
||||
$net3 = "WinHttpOpen" ascii
|
||||
$net4 = "WinHttpOpenRequest" ascii
|
||||
$net5 = "FtpOpenFile" ascii
|
||||
$net6 = "DownloadFile" ascii
|
||||
$net7 = "WinHttpSendRequest" ascii
|
||||
$net3 = "WinHttpOpenRequest" ascii
|
||||
$net4 = "WinHttpSendRequest" ascii
|
||||
$net5 = "URLDownloadToCacheFile" ascii
|
||||
$net6 = "FtpOpenFile" ascii
|
||||
$net7 = "WinHttpOpen" ascii
|
||||
$net8 = "InternetOpen" ascii
|
||||
|
||||
// 写入/保存到本地
|
||||
$save1 = "WriteFile" ascii
|
||||
$save2 = "SaveToFile" ascii
|
||||
$save3 = "URLDownloadToCacheFile" ascii
|
||||
// 自启动 / 服务注册持久化硬指标（不是裸 "RUN"）
|
||||
$p1 = "Software\\Microsoft\\Windows\\CurrentVersion\\Run" ascii nocase
|
||||
$p2 = "CurrentVersion\\RunOnce" ascii nocase
|
||||
$p3 = "schtasks.exe" ascii nocase
|
||||
$p4 = "CreateService" ascii
|
||||
$p5 = "Start Menu\\Programs\\StartUp" ascii nocase
|
||||
$p6 = "reg add" ascii nocase
|
||||
$p7 = "StartupApproved" ascii nocase
|
||||
|
||||
// 执行下载的载荷
|
||||
$exec1 = "WinExec" ascii
|
||||
$exec2 = "CreateProcess" ascii
|
||||
$exec3 = "ShellExecute" ascii
|
||||
$exec4 = "CreateRemoteThread" ascii
|
||||
|
||||
// 持久化（注册表/启动项）
|
||||
$persist1 = "RegSetValueEx" ascii
|
||||
$persist2 = "SetServiceObjectSecurity" ascii
|
||||
$persist3 = "CreateService" ascii
|
||||
$persist4 = "schtasks.exe" ascii
|
||||
$persist5 = "RUN" ascii // HKCU\Software\Microsoft\Windows\CurrentVersion\Run
|
||||
// 临时目录落地（Loader 常先落 %TEMP%）
|
||||
$tmp1 = /\\AppData\\Local\\Temp\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp|dat)/ ascii nocase
|
||||
$tmp2 = /%TEMP%\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp)/ ascii nocase
|
||||
$tmp3 = /\\Windows\\Temp\\[a-z0-9_. -]{3,60}\.(exe|dll|tmp)/ ascii nocase
|
||||
condition:
|
||||
// 下载 + 保存 + 执行（三件套）
|
||||
any of ($net*) and any of ($save*) and any of ($exec*)
|
||||
or
|
||||
// 下载 + 持久化（即使没执行也是准备投毒）
|
||||
any of ($net*) and any of ($persist*)
|
||||
or
|
||||
// 执行 + 持久化（下载可能在运行时动态完成）
|
||||
any of ($exec1, $exec2, $exec3) and any of ($persist*)
|
||||
uint16(0) == 0x5A4D
|
||||
and any of ($net*)
|
||||
and any of ($p*)
|
||||
and
|
||||
(
|
||||
// 下载 API 与持久化键名同处一个导入/字符串表块
|
||||
for any i in (1..#net1): ( any of ($p*) in (@net1[i] - 1024..@net1[i] + 1024) )
|
||||
or (any of ($tmp*) and 2 of ($p*))
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-3 RAT / 远控特征（C2 通信 + 命令执行 + 持久化）
|
||||
// C1-3 RAT / 远控特征
|
||||
// v0.2.3 的写法用 send( / recv( / system( 这类通用符号，且
|
||||
// (bot|serv|gate|ops)\.(com|net) 会匹配任意域名字符串 → 正常联网程序必中。
|
||||
// 现在要求：C2 硬指标（IP:PORT、可执行下载 URL、家族标记）+ 套接字 API 同表。
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_RAT_CommandControl {
|
||||
meta:
|
||||
risk_level = "critical"
|
||||
category = "C1"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1071 Application Layer Protocol"
|
||||
description = "RAT C2 通信特征：WinHTTP/WSASend + 命令执行 API + 可疑 URL/域名"
|
||||
description = "RAT C2 特征：套接字 API + 明确 C2 指标（IP:端口 / .exe 下载 / 家族标记）"
|
||||
strings:
|
||||
// C2 网络通道
|
||||
$c2_send = /WSASend|send\s*\(|sendto|WinHttpSendRequest|InternetWriteFile/ ascii
|
||||
$c2_recv = /WSARecv|recv\s*\(|recvfrom|WinHttpReadData|InternetReadFile/ ascii
|
||||
$c2_connect = /WSAConnect|connect\s*\(|WinHttpConnect|InternetConnect/ ascii
|
||||
// 套接字 API（不用裸 send/recv/connect）
|
||||
$sock1 = "WSASend" ascii
|
||||
$sock2 = "WSARecv" ascii
|
||||
$sock3 = "WSAConnect" ascii
|
||||
$sock4 = "WSASocket" ascii
|
||||
$sock5 = "inet_ntoa" ascii
|
||||
$sock6 = "getpeername" ascii
|
||||
|
||||
// 命令执行
|
||||
$cmd = /CreateProcess|ShellExecute|WinExec|system\s*\(|popen/ ascii
|
||||
// 命令执行（真实 RAT 才会与 C2 同表出现）
|
||||
$cmd1 = "CreateProcess" ascii
|
||||
$cmd2 = "ShellExecute" ascii
|
||||
$cmd3 = "WinExec" ascii
|
||||
|
||||
// 可疑 C2 URL（可执行文件里出现 http://...exe 或常见 C2 域名）
|
||||
$c2url1 = /http:\/\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{2,5}/ ascii
|
||||
$c2url2 = /https?:\/\/[^\"\s'\)]+\.(ru|top|xyz|tk|ml|ga|gq|cf|pw|click|stream|xyz|buzz|loan)\/[^\"\s'\)]*\.exe/ ascii
|
||||
$c2url3 = /(c2|panel|gate|bot|ops|serv|beacon|stage|stager)\.(ru|top|xyz|cc|io|com|net)/ ascii
|
||||
// C2 硬指标
|
||||
$c2ip = /https?:\/\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:[1-5][0-9]{3}/ ascii
|
||||
$c2exe = /https?:\/\/[^\s"']{4,}\.(exe|dll|scr|vbs|ps1|bat)\b/ ascii nocase
|
||||
$c2uri = /(gate|panel|submit|checkin|beacon|report)\.(php|aspx|html)\b/ ascii nocase
|
||||
$c2tld = /\b[a-z0-9-]{4,30}\.(ru|top|xyz|tk|ml|ga|gq|cf|pw|buzz|loan)\b/ ascii nocase
|
||||
condition:
|
||||
// C2 通信 + 命令执行（至少两个网络 + 一个命令）
|
||||
any of ($c2_send, $c2_recv, $c2_connect) and any of ($cmd*) and any of ($c2url*)
|
||||
or
|
||||
// C2 URL 模式 + 网络 API
|
||||
any of ($c2url*) and any of ($c2_connect, $c2_send)
|
||||
uint16(0) == 0x5A4D
|
||||
and 2 of ($sock*)
|
||||
and any of ($c2*)
|
||||
and
|
||||
(
|
||||
any of ($cmd*)
|
||||
or $c2exe
|
||||
or $c2uri
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-4 Mutex 互斥体名（勒索 / RAT / Loader 的持久化标记）
|
||||
// 单一 Mutex 名有一定价值，但必须配合其他特征才触发
|
||||
// 需要 CreateMutex 与家族 Mutex 名同表，避免文本文档里提一嘴家族名就命中
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_Mutex_Persistence {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C1"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1057 Process Discovery (mutex as persistence marker)"
|
||||
description = "可疑 Mutex 名称 + CreateMutex API = 持久化标记"
|
||||
description = "CreateMutex + 已知家族 Mutex 名同表 = 持久化标记"
|
||||
strings:
|
||||
$createmutex = "CreateMutexA" ascii
|
||||
$createmutexw = "CreateMutexW" ascii
|
||||
// 常见勒索/RAT Mutex 名（大小写不敏感，含 Unicode 变体用 wide）
|
||||
// LockBit
|
||||
$lockbit1 = /Global\\[Ll]ock[Bb]it/ ascii
|
||||
$lockbit2 = /Local\\[Ll]ock[Bb]it/ ascii
|
||||
// BlackCat / ALPHV
|
||||
$blackcat = /Global\\[Bb]lack[Cc]at|[Aa][Ll][Pp][Hh][Vv]/ ascii
|
||||
$blackcat = /(Global|Local)\\[Bb]lack[Cc]at|\bALPHV\b/ ascii
|
||||
// Ryuk
|
||||
$ryuk = /RyukTask[\d]|Global\\Ryuk/ ascii
|
||||
// Conti
|
||||
$conti = /Global\\[Cc]onti|Local\\[Cc]onti/ ascii
|
||||
$conti = /(Global|Local)\\[Cc]onti/ ascii
|
||||
// Pseudo/Dharma
|
||||
$dharma = /Global\\[Dd]harma|Local\\[Dd]harma/ ascii
|
||||
// 通用可疑模式：Mutex 名包含 "Mutex" + UUID 变体
|
||||
$dharma = /(Global|Local)\\[Dd]harma/ ascii
|
||||
// 通用可疑模式：Mutex 名含 UUID 前缀
|
||||
$generic1 = /Global\\Mutex[0-9a-f]{8}/ ascii
|
||||
// Kryptik/Qakbot
|
||||
$qakbot = /[Qq]akbot|[Kk]ryptik/ ascii
|
||||
$qakbot = /\b[Qq]akbot\b|\b[Kk]ryptik\b/ ascii
|
||||
// Cobalt Strike named pipe
|
||||
$cs1 = /\\\\\.\\pipe\\msagent_[0-9a-f]+/ ascii
|
||||
$cs2 = /\\\\\.\\pipe\\[Bb]eacon_[0-9a-f]+/ ascii
|
||||
condition:
|
||||
// CreateMutex + 已知家族 Mutex 名
|
||||
(any of ($createmutex*)) and any of ($lockbit1, $lockbit2, $blackcat, $ryuk, $conti, $dharma, $qakbot)
|
||||
or
|
||||
// Cobalt Strike 命名管道特征（不需要 CreateMutex）
|
||||
any of ($cs*)
|
||||
or
|
||||
// CreateMutex + 通用可疑 UUID Mutex
|
||||
(any of ($createmutex*)) and $generic1
|
||||
uint16(0) == 0x5A4D
|
||||
and
|
||||
(
|
||||
any of ($cs*)
|
||||
or (
|
||||
any of ($createmutex*)
|
||||
and any of ($lockbit1, $lockbit2, $blackcat, $ryuk, $conti, $dharma, $qakbot, $generic1)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-5 DLL 劫持 / LoadLibrary 动态加载
|
||||
// C1-5 DLL 劫持 / 侧加载
|
||||
// 只看「LoadLibrary + GetProcAddress + 临时目录里的 DLL 名」三件套；
|
||||
// \ProgramData\ 与 \Public\ 是正常安装器路径，不再单独作为可疑路径
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_DLLHijacking {
|
||||
meta:
|
||||
risk_level = "high"
|
||||
category = "C1"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1574 Hijack Execution Flow"
|
||||
description = "PE 动态加载多个系统 DLL + 非系统路径 = DLL 劫持迹象"
|
||||
description = "动态加载 API + 临时目录 DLL 落地 = DLL 侧加载/劫持"
|
||||
strings:
|
||||
$loadlib = "LoadLibraryA" ascii
|
||||
$loadlibw = "LoadLibraryW" ascii
|
||||
$getproc = "GetProcAddress" ascii
|
||||
// 可疑加载路径（非 System32）
|
||||
$path1 = /\\Users\\[^\\]+\\AppData\\Local\\Temp/ ascii
|
||||
$path2 = /\\AppData\\Local\\Temp/ ascii
|
||||
$path3 = /\\ProgramData\\/ ascii
|
||||
$path4 = /\\Public\\Documents\\/ ascii
|
||||
$copy1 = "CopyFileA" ascii
|
||||
$copy2 = "CopyFileW" ascii
|
||||
$move = "MoveFileEx" ascii
|
||||
$temp1 = /\\AppData\\Local\\Temp\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
|
||||
$temp2 = /%TEMP%\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
|
||||
$temp3 = /\\Windows\\Temp\\[a-z0-9_. -]{3,40}\.dll/ ascii nocase
|
||||
$sysdir = "SetDllDirectory" ascii
|
||||
$searchorder = "LoadLibraryEx" ascii
|
||||
condition:
|
||||
// LoadLibrary + 可疑路径 + GetProcAddress = 完整劫持链
|
||||
(any of ($loadlib*)) and any of ($path*) and $getproc
|
||||
or
|
||||
// LoadLibrary + 多个可疑路径
|
||||
(any of ($loadlib*)) and 2 of ($path*)
|
||||
uint16(0) == 0x5A4D
|
||||
and any of ($loadlib*, $searchorder)
|
||||
and any of ($temp*)
|
||||
and (any of ($getproc, $copy1, $copy2, $move, $sysdir))
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-6 LOLBin（Living Off The Land Binary）调用特征
|
||||
// 已知合法 Windows 二进制被恶意利用来执行命令
|
||||
// 命令行串 + 进程创建 API 必须同表紧邻（正常程序会把两者写进同一个导入/资源块）
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_LOLBin_Usage {
|
||||
@@ -228,31 +280,57 @@ rule C1_PE_LOLBin_Usage {
|
||||
category = "C1"
|
||||
false_positive_rate = "low"
|
||||
reference = "https://lolbas-project.github.io/"
|
||||
description = "PE 文件中出现 LOLBin 调用：certutil / mshta / rundll32 / regsvr32 / bitsadmin / wmic / msiexec"
|
||||
description = "PE 中出现 LOLBin 调用命令行 + 进程创建 API（同表）"
|
||||
strings:
|
||||
$certutil = /certutil(\.exe)?\s+(-urlcache|-decode|-encode|-hashfile)/ ascii
|
||||
$mshta = /mshta(\.exe)?\s+(https?:|javascript:|vbscript:)/ ascii
|
||||
$rundll32 = /rundll32(\.exe)?\s+[A-Za-z0-9_.]+,/ ascii
|
||||
$regsvr32 = /regsvr32(\.exe)?\s+\/s\s+\/u/ ascii
|
||||
$bitsadmin = /bitsadmin(\.exe)?\s+(\/transfer|\/create)/ ascii
|
||||
$wmic = /wmic(\.exe)?\s+(process\s+call\s+create|product\s+call)/ ascii
|
||||
$msiexec = /msiexec(\.exe)?\s+\/[ia]\s+https?:/ ascii
|
||||
$installutil = /installutil(\.exe)?\s+\/u\s+\/logfile=/ ascii
|
||||
$syncapp = /syncapp(\.exe)?\s+\/compattrack/ ascii
|
||||
$psexec = /psexec(\.exe)?\s+(-accepteula|-i\s+\d+|-w\s+https?:)/ ascii
|
||||
// 命令执行 API（用于组合条件）
|
||||
$lolbin1 = /certutil(\.exe)?\s+(-urlcache|-decode)\s+-split/ ascii nocase
|
||||
$lolbin2 = /mshta(\.exe)?\s+(https?:|javascript:|vbscript:)/ ascii nocase
|
||||
$lolbin3 = /regsvr32(\.exe)?\s+\/s\s+\/u:http/ ascii nocase
|
||||
$lolbin4 = /bitsadmin(\.exe)?\s+\/transfer\s+\/source:/ ascii nocase
|
||||
$lolbin5 = /wmic(\.exe)?\s+process\s+call\s+create/ ascii nocase
|
||||
$lolbin6 = /msiexec(\.exe)?\s+\/[ia]\s+https?:/ ascii nocase
|
||||
$lolbin7 = /installutil(\.exe)?\s+\/u\s+\/logfile=https?:/ ascii nocase
|
||||
$lolbin8 = /psexec(\.exe)?\s+(-accepteula\s+)?-w\s+https?:/ ascii nocase
|
||||
$exec1 = "CreateProcess" ascii
|
||||
$exec2 = "ShellExecute" ascii
|
||||
$exec3 = "WinExec" ascii
|
||||
condition:
|
||||
// 至少 1 个 LOLBin 调用 + 命令执行 API
|
||||
any of ($certutil*, $mshta*, $rundll32*, $regsvr32*, $bitsadmin*, $wmic*, $msiexec*, $installutil*, $syncapp*, $psexec*)
|
||||
uint16(0) == 0x5A4D
|
||||
and any of ($lolbin*)
|
||||
and any of ($exec*)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C8-1 勒索加密 API 组合（CryptEncrypt + BCrypt* + 文件遍历 + 删除卷影）
|
||||
// C1-7 PE 高熵 + 网络 API（疑似加壳下载器）
|
||||
// 需要「多个网络 API + 明确壳节名」同时成立
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_EntropyHighNetwork {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C1"
|
||||
false_positive_rate = "low"
|
||||
reference = "binary.py HighEntropySection 联动"
|
||||
description = "PE 加壳节名 + 多个下载 API = 加壳下载器"
|
||||
strings:
|
||||
$net1 = "InternetOpenUrl" ascii
|
||||
$net2 = "URLDownloadToFile" ascii
|
||||
$net3 = "WinHttpOpenRequest" ascii
|
||||
$net4 = "WSAConnect" ascii
|
||||
condition:
|
||||
uint16(0) == 0x5A4D
|
||||
and 2 of ($net*)
|
||||
and (
|
||||
for any i in (0..pe.number_of_sections - 1): (
|
||||
pe.sections[i].name matches /UPX0/ or pe.sections[i].name == "UPX1"
|
||||
or pe.sections[i].name matches /\.vmp[0-9]/ or pe.sections[i].name matches /Themida|\.themida/
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C8-1 勒索加密链（加密 API + 卷影删除/勒索信，勒索样本常以文本形态投递，不加 PE 头）
|
||||
// =============================================================================
|
||||
|
||||
rule C8_Ransomware_Encryption {
|
||||
@@ -261,9 +339,9 @@ rule C8_Ransomware_Encryption {
|
||||
category = "C8"
|
||||
false_positive_rate = "low"
|
||||
reference = "MITRE ATT&CK T1486 Data Encrypted for Impact"
|
||||
description = "勒索加密链：CryptEncrypt/BCryptEncrypt + 大量文件遍历 + 卷影删除 = 勒索"
|
||||
description = "勒索加密链：加密 API + 卷影删除命令或勒索信文件名"
|
||||
strings:
|
||||
// 加密 API（至少出现两个）
|
||||
// 加密 API
|
||||
$enc1 = "CryptEncrypt" ascii
|
||||
$enc2 = "CryptDeriveKey" ascii
|
||||
$enc3 = "BCryptEncrypt" ascii
|
||||
@@ -276,25 +354,23 @@ rule C8_Ransomware_Encryption {
|
||||
$enum3 = "FindNextFileA" ascii
|
||||
$enum4 = "FindNextFileW" ascii
|
||||
|
||||
// 卷影删除（勒索前置）
|
||||
$vss1 = "vssadmin" ascii
|
||||
$vss2 = "delete shadows" ascii
|
||||
// 卷影删除（完整命令，不认裸 "wmic"）
|
||||
$vss1 = "vssadmin" ascii nocase
|
||||
$vss2 = "delete shadows" ascii nocase
|
||||
$vss3 = "DeleteShadowCopies" ascii
|
||||
$vss4 = "wmic" ascii
|
||||
$vss4 = "shadow copies" ascii nocase
|
||||
|
||||
// 勒索信文件名变体（出现在二进制里）
|
||||
$ransomnote1 = /(HOW_TO_RECOVER|RESTORE_FILES|DECRYPT_INSTRUCTIONS|README|HELP_DECRYPT|_readme|_restore)[\w.-]*\.(txt|html|png|jpg|bmp|url)/ nocase
|
||||
$ransomnote2 = /\.(locky|zepto|cryptolocker|tor|crpt|encrypted|virus|wallet|pay2)/ nocase
|
||||
$ransomnote3 = /bitcoin|bitcash|monero|ltc|BTC\b/ ascii
|
||||
// 勒索信文件名 / 加密扩展名
|
||||
$note1 = /(HOW_TO_RECOVER|RESTORE_FILES|DECRYPT_INSTRUCTIONS|HELP_DECRYPT)[\w.-]*\.(txt|html|png|jpg|bmp|url)/ nocase
|
||||
$note2 = /\.(locky|zepto|cryptolocker|crpt|encrypted|pay2)\b/ nocase
|
||||
condition:
|
||||
// 加密 API + 文件遍历 = 加密工具（必须至少 2 个加密 API）
|
||||
(2 of ($enc*)) and any of ($enum*)
|
||||
or
|
||||
// 加密 API + 卷影删除 = 完整勒索
|
||||
(any of ($enc*)) and any of ($vss*)
|
||||
or
|
||||
// 加密 API + 勒索信 + 加密文件扩展名 = 明确勒索
|
||||
(any of ($enc*)) and (any of ($ransomnote1, $ransomnote2)) and $ransomnote3
|
||||
any of ($enc*)
|
||||
and
|
||||
(
|
||||
($vss1 and ($vss2 or $vss3 or $vss4))
|
||||
or (2 of ($enum*) and any of ($note*))
|
||||
or (2 of ($enc*) and any of ($enum*) and $vss1)
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -310,53 +386,51 @@ rule C8_Mining_Monero {
|
||||
reference = "MITRE ATT&CK T1496 Resource Hijacking"
|
||||
description = "挖矿程序特征：xmrig 命令行参数 / stratum+tcp / 矿池域名"
|
||||
strings:
|
||||
// xmrig 命令行特征
|
||||
$xmrig1 = "xmrig" ascii nocase
|
||||
$xmrig2 = /--donate-level\s+\d+/ ascii
|
||||
$xmrig3 = /-o\s+stratum\+tcp/ ascii
|
||||
$xmrig4 = /--cpu-priority\s+\d/ ascii
|
||||
$xmrig5 = /--max-threads-hint/ ascii
|
||||
|
||||
// 矿池域名（Monero）
|
||||
$pool1 = "pool.xmrig.com" ascii nocase
|
||||
$pool2 = "pool.minexmr.com" ascii nocase
|
||||
$pool3 = "pool.supportxmr.com" ascii nocase
|
||||
$pool4 = "mine.moneropool.com" ascii nocase
|
||||
$pool5 = "stratum+tcp://" ascii
|
||||
|
||||
// CPU 高占用循环特征
|
||||
$loop = /(while|for)\s*\(\s*\d+\s*[<>=]/ ascii
|
||||
$thread = "CreateThread" ascii
|
||||
condition:
|
||||
// 至少 2 个 xmrig 特征
|
||||
(any of ($xmrig1, $xmrig2, $xmrig3, $xmrig4, $xmrig5)) and 2 of ($*)
|
||||
or
|
||||
// stratum+tcp 矿池协议 + 多个线程创建
|
||||
$pool5 and ($thread or 2 of ($xmrig*))
|
||||
any of ($pool1, $pool2, $pool3, $pool4)
|
||||
or ($pool5 and (any of ($xmrig*) or $thread))
|
||||
or (2 of ($xmrig*) and any of ($pool*))
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C9 壳特征（UPX / Themida / VMProtect / Enigma）
|
||||
// 壳识别后建议升级到 L3 沙箱深检，不单独阻断
|
||||
// C9 壳特征（UPX / Themida / VMProtect）
|
||||
// 加壳本身不是恶意证据（大量开源工具用 UPX），只作沙箱深检触发条件
|
||||
// =============================================================================
|
||||
|
||||
rule C9_Packer_UPX {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
risk_level = "low"
|
||||
category = "C9"
|
||||
false_positive_rate = "low"
|
||||
reference = "UPX Packer"
|
||||
description = "UPX 加壳 PE：节名 UPX0 / UPX1 + UPX 签名"
|
||||
description = "UPX 加壳 PE：节名 UPX0/UPX1 + UPX 签名"
|
||||
strings:
|
||||
$upx0 = "UPX0" ascii
|
||||
$upx1 = "UPX1" ascii
|
||||
$upx2 = "UPX!" ascii
|
||||
$upx3 = /UPX\s+\d+\.\d+/ ascii // "UPX 3.96" 这类版本字符串
|
||||
$upx4 = /UPX\b/ ascii
|
||||
condition:
|
||||
// 节名（两个）+ 签名
|
||||
($upx0 and $upx1) or ($upx2 and any of ($upx3, $upx4))
|
||||
uint16(0) == 0x5A4D
|
||||
and (
|
||||
(
|
||||
for any i in (0..pe.number_of_sections - 1): (
|
||||
pe.sections[i].name matches /UPX0/
|
||||
)
|
||||
and for any j in (0..pe.number_of_sections - 1): (
|
||||
pe.sections[j].name matches /UPX1/
|
||||
)
|
||||
)
|
||||
or ($upx2 and pe.number_of_sections <= 4)
|
||||
)
|
||||
}
|
||||
|
||||
rule C9_Packer_Themida_VMP {
|
||||
@@ -365,39 +439,18 @@ rule C9_Packer_Themida_VMP {
|
||||
category = "C9"
|
||||
false_positive_rate = "low"
|
||||
reference = "Themida / VMProtect"
|
||||
description = "商业壳特征：Themida / VMProtect 产品标识字符串"
|
||||
description = "商业壳特征：Themida / VMProtect 节名 + 产品标识字符串"
|
||||
strings:
|
||||
$themida1 = "Themida" ascii wide
|
||||
$themida2 = "WinLicense" ascii wide
|
||||
$themida3 = "Oreans" ascii wide
|
||||
$vmp1 = "VMProtect" ascii wide
|
||||
$vmp3 = "VMProtect SDK" ascii wide
|
||||
condition:
|
||||
any of ($themida*) or any of ($vmp*)
|
||||
}
|
||||
|
||||
|
||||
// =============================================================================
|
||||
// C1-7 PE 高熵 + 大量网络 API（疑似加壳下载器）
|
||||
// binary.py 的 HighEntropySection 已单独扫描，但 YARA 看"熵 + 网络"组合
|
||||
// =============================================================================
|
||||
|
||||
rule C1_PE_EntropyHighNetwork {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C1"
|
||||
false_positive_rate = "medium"
|
||||
reference = "binary.py HighEntropySection 联动"
|
||||
description = "PE 同时出现：多个网络 API + 加壳特征（熵高） + 无签名"
|
||||
strings:
|
||||
$net1 = "InternetOpenUrl" ascii
|
||||
$net2 = "WinHttpOpen" ascii
|
||||
$net3 = "URLDownloadToFile" ascii
|
||||
$net4 = "WSAConnect" ascii
|
||||
// 壳特征
|
||||
$upx = "UPX0" ascii
|
||||
$themida = "Themida" ascii
|
||||
condition:
|
||||
// 网络 API + 壳特征 = 加壳下载器
|
||||
any of ($net*) and ($upx or $themida)
|
||||
uint16(0) == 0x5A4D
|
||||
and (
|
||||
for any i in (0..pe.number_of_sections - 1): (
|
||||
pe.sections[i].name matches /\.themida|\.vmp[0-9]/
|
||||
)
|
||||
or (any of ($themida*) and any of ($vmp1))
|
||||
)
|
||||
}
|
||||
|
||||
@@ -126,26 +126,41 @@ rule C8_Ransomware_Note_Filename {
|
||||
meta:
|
||||
risk_level = "medium"
|
||||
category = "C8"
|
||||
false_positive_rate = "medium"
|
||||
false_positive_rate = "low"
|
||||
reference = "勒索信文件名通用模式"
|
||||
description = "勒索信典型文件名变体（匹配后触发需要配合加密 API 才升级高危）"
|
||||
description = "勒索信文件名 + （加密 API + 赎金语言）或（赎金语言 + 钱包地址）"
|
||||
strings:
|
||||
// 只保留"文件名形态"的勒索信特征：家族专有名 + 扩展名。
|
||||
// 不再用裸 README.txt / _readme.txt：任何安装包都带这两个字面量，
|
||||
// 在二进制内容上做全文匹配时必然命中（Go 官方 MSI 即被这样判成恶意）。
|
||||
$n1 = /HOW_TO_RECOVER[\w.-]*\.(txt|html|png|bmp|jpg|url|vbs|js)/ nocase
|
||||
$n2 = /RESTORE_FILES[\w.-]*\.(txt|html|png|bmp|jpg|url|vbs|js)/ nocase
|
||||
$n3 = /DECRYPT_INSTRUCTIONS[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n4 = /README[_\s-]*[\d]*[_\s-]*[\w]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n4 = /README[._-]*(FOR[._-]*DECRYPT|TO[._-]*RESTORE)[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n5 = /HELP_DECRYPT[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n6 = /_readme[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n7 = /_restore[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$bitcoin = /bitcoin address|BTC\b|monero address|XMR\b|send money|ransom|payment/ nocase
|
||||
$wallet = /1[A-Za-z0-9]{25,34}|bc1[A-Za-z0-9]{11,71}|4[0-9A-Za-z]{32}/ nocase
|
||||
$n6 = /\b_(readme|restore)[\w.-]*\.(txt|html|png|bmp|jpg|url)/ nocase
|
||||
$n7 = /(how_to_decrypt|restore_my_files|decrypt_your_files)[\w.-]*\.(txt|html)/ nocase
|
||||
|
||||
// 赎金语言：必须是完整词组。裸 "BTC"/"XMR"/"payment"/"ransom" 在压缩流里
|
||||
// 会以随机字节出现，"ransom" 还会被 "Ransomware" 子串命中。
|
||||
$payment = /(bitcoin|monero|xmr|btc)\s+(address|wallet)|send\s+(the\s+)?money|pay\s+(the\s+)?(ransom|fee)|\bransom\b\s+(note|payment|demand)|your\s+files\s+(are\s+)?encrypted|decrypt\s+your\s+files/ nocase
|
||||
|
||||
// 钱包地址：锚定到整个 token，并用真实地址字符集/长度。
|
||||
// 原写法 1[A-Za-z0-9]{25,34} 无边界，单个 56MB 安装包命中 7.6 万次。
|
||||
$btc_addr = /\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b/
|
||||
$bech32 = /\bbc1q[a-z0-9]{38}\b|\bbc1q[a-z0-9]{58}\b/
|
||||
$xmr_addr = /\b[48][0-9A-Za-z]{94}\b/
|
||||
|
||||
$encapi = /CryptEncrypt|BCryptEncrypt|CryptDeriveKey/ ascii
|
||||
condition:
|
||||
// 勒索信文件名 + 加密 API = 明确勒索
|
||||
any of ($n*) and $encapi
|
||||
or
|
||||
// 勒索信文件名 + 比特币/XMR 地址 = 大概率勒索
|
||||
any of ($n*) and any of ($bitcoin, $wallet)
|
||||
any of ($n*)
|
||||
and (
|
||||
// 投递器：真加密 API + 赎金语言
|
||||
($encapi and $payment)
|
||||
or
|
||||
// 纯文本勒索信：赎金语言 + 具体钱包地址
|
||||
($payment and any of ($btc_addr, $bech32, $xmr_addr))
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -199,11 +214,13 @@ rule C8_Ransomware_VolumeShadow {
|
||||
$vss2 = /vssadmin\s+delete\s+shadows\s+\/all\b/ ascii
|
||||
$wmic1 = /wmic\s+shadowcopy\s+delete/ ascii
|
||||
$wmic2 = /wmic\s+shadowcopy\s+where\s+\d+/ ascii
|
||||
$vss3 = /ShadowCopy|VolumeShadowCopy|vss\b/i ascii // "i" flag = nocase in YARA
|
||||
$vss4 = /vssadmin[^\n]{0,48}delete[^\n]{0,48}(shadows|shadow\s+copies)/ ascii
|
||||
$enc = /CryptEncrypt|BCryptEncrypt|CryptDeriveKey/ ascii
|
||||
condition:
|
||||
// 卷影删除 + 加密 API = 完整勒索前置
|
||||
(any of ($vss1, $vss2, $wmic*, $vss3)) and any of ($enc*)
|
||||
// 不用裸 "vss"/"ShadowCopy" 子串：任何备份/卷影库的符号表都有它们，
|
||||
// 和 CryptEncrypt 同框就报勒索，是典型误报来源。
|
||||
(any of ($vss1, $vss2, $wmic*, $vss4)) and any of ($enc*)
|
||||
or
|
||||
// 连续多次 vssadmin delete = 正在清卷影
|
||||
any of ($vss1, $vss2) and any of ($wmic*)
|
||||
|
||||
@@ -57,12 +57,19 @@ rule C3_PHP_Webshell_Obfuscated {
|
||||
$rot13 = "str_rot13" ascii wide
|
||||
$assert = "assert(" ascii wide
|
||||
$createfn = "create_function" ascii wide
|
||||
// PHP 上下文证据：没有 <?php / 超全局变量，就不要凭函数名定罪（vim、git 等
|
||||
// 二进制里同样存在 base64_decode / eval( 字样）
|
||||
$php1 = "<?php" ascii
|
||||
$php2 = /\$_(GET|POST|REQUEST|COOKIE|FILES|SESSION)\b/ ascii
|
||||
condition:
|
||||
// base64_decode + (eval|assert|create_function) = 解码后动态执行
|
||||
$b64 and ($eval or $assert or $createfn)
|
||||
or
|
||||
// 解码链：base64 + gzinflate / str_rot13 + 执行
|
||||
$b64 and ($gzinflate or $rot13) and ($eval or $assert)
|
||||
any of ($php*)
|
||||
and (
|
||||
// base64_decode + (eval|assert|create_function) = 解码后动态执行
|
||||
($b64 and ($eval or $assert or $createfn))
|
||||
or
|
||||
// 解码链：base64 + gzinflate / str_rot13 + 执行
|
||||
($b64 and ($gzinflate or $rot13) and ($eval or $assert))
|
||||
)
|
||||
}
|
||||
|
||||
// PHP Webshell —— 变量变形（$$a = $_POST['cmd']；$$a($_POST['arg'])）
|
||||
|
||||
264
scripts/archive_old_tasks.py
Normal file
264
scripts/archive_old_tasks.py
Normal file
@@ -0,0 +1,264 @@
|
||||
"""DB 归档 + 清理脚本。
|
||||
|
||||
职责:
|
||||
1. scan_tasks 90 天前: 清 detail JSONB (节省空间, 扫描详情留 90 天)
|
||||
2. scan_tasks 365 天前: 移到 scan_tasks_archive (含 ScanFinding)
|
||||
3. idempotency_records: 清 7 天前的 (幂等缓存过期重扫)
|
||||
4. policy_versions: 只保留最新 5 个版本 (历史版本清掉)
|
||||
|
||||
用法:
|
||||
docker compose exec api python3 scripts/archive_old_tasks.py # dry-run, 只报告不动数据
|
||||
docker compose exec api python3 scripts/archive_old_tasks.py --apply # 真正执行
|
||||
|
||||
建议: 每天 cron 跑一次
|
||||
|
||||
设计原则:
|
||||
- 小批量提交 (1000 条一批), 避免长事务锁表
|
||||
- 先 dry-run 再 apply, 输出清晰
|
||||
- pg_repack / VACUUM ANALYZE 在脚本最后触发 (让 autovacuum 兜底也行)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import sys
|
||||
import os
|
||||
from datetime import timedelta, datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, "/app")
|
||||
os.environ.setdefault("MESHSAFE_MASTER_KEY", "Dt0ktRs8SjEH8VxWZJDX125NV1qiII88HoiXZPhWers=")
|
||||
|
||||
from app.db import SessionLocal # noqa: E402
|
||||
from app.models import ( # noqa: E402
|
||||
ScanTask, ScanFinding, IdempotencyRecord, PolicyVersion,
|
||||
)
|
||||
from sqlalchemy import select, func # noqa: E402
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
log = logging.getLogger("archive")
|
||||
|
||||
# ---- 配置 ----
|
||||
DETAIL_RETENTION_DAYS = 90 # 90 天前清 detail JSONB
|
||||
ARCHIVE_RETENTION_DAYS = 365 # 365 天前移归档
|
||||
IDEM_RETENTION_DAYS = 7 # 幂等缓存 7 天 TTL
|
||||
POLICY_KEEP_VERSIONS = 5 # policy_versions 只留最新 5 个
|
||||
BATCH_SIZE = 1000 # 分批处理
|
||||
|
||||
|
||||
def _cutoff(days: int) -> datetime:
|
||||
return datetime.now(timezone.utc) - timedelta(days=days)
|
||||
|
||||
|
||||
def archive_tasks(dry_run: bool) -> None:
|
||||
"""归档 365 天前的 scan_tasks 到 scan_tasks_archive。"""
|
||||
cutoff = _cutoff(ARCHIVE_RETENTION_DAYS)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
total = db.execute(
|
||||
select(func.count(ScanTask.id)).where(ScanTask.created_at < cutoff)
|
||||
).scalar() or 0
|
||||
|
||||
log.info(f"[归档] {ARCHIVE_RETENTION_DAYS} 天前的 scan_tasks: {total} 条")
|
||||
|
||||
if total == 0:
|
||||
log.info(" → 无需归档")
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
log.info(f" [dry-run] 将归档 {total} 条")
|
||||
return
|
||||
|
||||
# 分批
|
||||
moved = 0
|
||||
while True:
|
||||
batch = db.execute(
|
||||
select(ScanTask).where(ScanTask.created_at < cutoff).limit(BATCH_SIZE)
|
||||
).scalars().all()
|
||||
if not batch:
|
||||
break
|
||||
|
||||
for task in batch:
|
||||
try:
|
||||
db.execute(
|
||||
__import__("sqlalchemy").text("""
|
||||
INSERT INTO scan_tasks_archive
|
||||
(id, channel, client_id, filename, declared_ext, size,
|
||||
sha256, md5, real_type, status, mode, verdict, action, score,
|
||||
reason, policy_version, has_detail, has_findings,
|
||||
created_at, finished_at, duration_ms, archived_at)
|
||||
VALUES
|
||||
(:id, :channel, :client_id, :filename, :declared_ext, :size,
|
||||
:sha256, :md5, :real_type, :status, :mode, :verdict, :action, :score,
|
||||
:reason, :policy_version, :has_detail, :has_findings,
|
||||
:created_at, :finished_at, :duration_ms, NOW())
|
||||
ON CONFLICT (id) DO NOTHING
|
||||
"""),
|
||||
{
|
||||
"id": task.id,
|
||||
"channel": task.channel,
|
||||
"client_id": task.client_id,
|
||||
"filename": task.filename,
|
||||
"declared_ext": task.declared_ext,
|
||||
"size": task.size,
|
||||
"sha256": task.sha256,
|
||||
"md5": task.md5,
|
||||
"real_type": task.real_type,
|
||||
"status": task.status,
|
||||
"mode": task.mode,
|
||||
"verdict": task.verdict,
|
||||
"action": task.action,
|
||||
"score": task.score,
|
||||
"reason": task.reason,
|
||||
"policy_version": task.policy_version,
|
||||
"has_detail": bool(task.detail and len(task.detail) > 0),
|
||||
"has_findings": bool(task.findings),
|
||||
"created_at": task.created_at,
|
||||
"finished_at": task.finished_at,
|
||||
"duration_ms": task.duration_ms,
|
||||
},
|
||||
)
|
||||
# CASCADE 删 ScanFinding
|
||||
db.delete(task)
|
||||
moved += 1
|
||||
except Exception as e:
|
||||
log.warning(f" task {task.id} 归档失败: {e}")
|
||||
db.rollback()
|
||||
continue
|
||||
db.commit()
|
||||
log.info(f" 已归档 {moved}/{total}")
|
||||
|
||||
log.info(f" ✅ 归档完成: {moved} 条")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def scrub_detail(dry_run: bool) -> None:
|
||||
"""90~365 天之间的任务清 detail JSONB (留摘要, 省空间)。"""
|
||||
db_cutoff = _cutoff(DETAIL_RETENTION_DAYS)
|
||||
arc_cutoff = _cutoff(ARCHIVE_RETENTION_DAYS)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
total = db.execute(
|
||||
select(func.count(ScanTask.id)).where(
|
||||
ScanTask.created_at < db_cutoff,
|
||||
ScanTask.created_at >= arc_cutoff,
|
||||
ScanTask.detail.isnot(None),
|
||||
)
|
||||
).scalar() or 0
|
||||
|
||||
log.info(f"[清 detail] {DETAIL_RETENTION_DAYS}~{ARCHIVE_RETENTION_DAYS} 天之间有 detail 的: {total} 条")
|
||||
|
||||
if total == 0:
|
||||
log.info(" → 无需清理")
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
log.info(f" [dry-run] 将清 {total} 条 detail")
|
||||
return
|
||||
|
||||
# 分批 UPDATE
|
||||
updated = 0
|
||||
while True:
|
||||
batch_ids = db.execute(
|
||||
select(ScanTask.id).where(
|
||||
ScanTask.created_at < db_cutoff,
|
||||
ScanTask.created_at >= arc_cutoff,
|
||||
ScanTask.detail.isnot(None),
|
||||
).limit(BATCH_SIZE)
|
||||
).scalars().all()
|
||||
if not batch_ids:
|
||||
break
|
||||
db.execute(
|
||||
__import__("sqlalchemy").update(ScanTask).where(ScanTask.id.in_(batch_ids)).values(detail=None)
|
||||
)
|
||||
db.commit()
|
||||
updated += len(batch_ids)
|
||||
log.info(f" 已清 {updated}/{total}")
|
||||
|
||||
log.info(f" ✅ detail 清理完成: {updated} 条")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def clean_idempotency(dry_run: bool) -> None:
|
||||
"""清 7 天前的幂等缓存 (过期的可以重扫)。"""
|
||||
cutoff = _cutoff(IDEM_RETENTION_DAYS)
|
||||
db = SessionLocal()
|
||||
try:
|
||||
total = db.execute(
|
||||
select(func.count(IdempotencyRecord.id)).where(IdempotencyRecord.created_at < cutoff)
|
||||
).scalar() or 0
|
||||
|
||||
log.info(f"[幂等清理] {IDEM_RETENTION_DAYS} 天前的 idempotency_records: {total} 条")
|
||||
|
||||
if total == 0:
|
||||
log.info(" → 无需清理")
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
log.info(f" [dry-run] 将清 {total} 条")
|
||||
return
|
||||
|
||||
db.execute(
|
||||
__import__("sqlalchemy").delete(IdempotencyRecord).where(IdempotencyRecord.created_at < cutoff)
|
||||
)
|
||||
db.commit()
|
||||
log.info(f" ✅ 幂等清理完成: {total} 条")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def trim_policy_versions(dry_run: bool) -> None:
|
||||
"""policy_versions 只留最新 N 个。"""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
all_pv = db.execute(
|
||||
select(PolicyVersion).order_by(PolicyVersion.created_at.desc())
|
||||
).scalars().all()
|
||||
|
||||
keep = all_pv[:POLICY_KEEP_VERSIONS]
|
||||
drop = all_pv[POLICY_KEEP_VERSIONS:]
|
||||
|
||||
log.info(f"[policy] 共 {len(all_pv)} 条历史版本, 留 {len(keep)} 删 {len(drop)}")
|
||||
|
||||
if not drop:
|
||||
log.info(" → 无需清理")
|
||||
return
|
||||
|
||||
if dry_run:
|
||||
for pv in drop:
|
||||
log.info(f" [dry-run] 将删 {pv.version}")
|
||||
return
|
||||
|
||||
for pv in drop:
|
||||
db.delete(pv)
|
||||
db.commit()
|
||||
log.info(f" ✅ policy_versions 清理完成: 删 {len(drop)} 条")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="MeshSafe DB 归档 + 清理")
|
||||
parser.add_argument("--apply", action="store_true", help="真正执行 (默认 dry-run)")
|
||||
args = parser.parse_args()
|
||||
|
||||
dry = not args.apply
|
||||
log.info(f"模式: {'DRY-RUN (仅报告)' if dry else 'APPLY (真正执行)'}")
|
||||
log.info("")
|
||||
|
||||
archive_tasks(dry)
|
||||
log.info("")
|
||||
scrub_detail(dry)
|
||||
log.info("")
|
||||
clean_idempotency(dry)
|
||||
log.info("")
|
||||
trim_policy_versions(dry)
|
||||
|
||||
if dry:
|
||||
log.info("\n💡 确认无误后加 --apply 参数正式执行")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
140
scripts/cron_entrypoint.py
Normal file
140
scripts/cron_entrypoint.py
Normal file
@@ -0,0 +1,140 @@
|
||||
"""MeshSafe 定时任务调度器 — 零依赖，Docker 容器前台常驻。
|
||||
|
||||
和 api 共享同一个镜像，容器 entrypoint 换成本脚本即可。
|
||||
用法: docker run meshsafe:latest python3 scripts/cron_entrypoint.py
|
||||
|
||||
支持的任务:
|
||||
每天 03:00 归档 + 清理 DB (archive_old_tasks.py --apply)
|
||||
每周一 03:10 VACUUM ANALYZE (维护表统计信息)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
sys.path.insert(0, "/app")
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
)
|
||||
log = logging.getLogger("scheduler")
|
||||
|
||||
|
||||
_SECRET_ENV_KEYS = ("MESHSAFE_DB_URL", "MESHSAFE_MASTER_KEY", "MESHSAFE_ADMIN_PASSWORD",
|
||||
"MESHSAFE_API_KEY", "MESHSAFE_API_CLIENTS")
|
||||
|
||||
|
||||
def _redact(text: str) -> str:
|
||||
"""崩溃日志里 pydantic 会把整个 settings dict（含 DB 口令）原样打出来，落盘前抹掉。"""
|
||||
out = text
|
||||
for key in _SECRET_ENV_KEYS:
|
||||
val = os.environ.get(key, "")
|
||||
if val and len(val) >= 8:
|
||||
out = out.replace(val, "***")
|
||||
return out
|
||||
|
||||
|
||||
def run(cmd: list[str], label: str) -> None:
|
||||
"""跑一个命令，捕获输出并记日志。"""
|
||||
log.info(f"[{label}] 开始执行: {' '.join(cmd)}")
|
||||
try:
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=1800)
|
||||
if proc.stdout:
|
||||
for line in proc.stdout.strip().splitlines():
|
||||
log.info(f"[{label}] {_redact(line)}")
|
||||
if proc.stderr:
|
||||
for line in proc.stderr.strip().splitlines():
|
||||
log.warning(f"[{label}] {_redact(line)}")
|
||||
if proc.returncode == 0:
|
||||
log.info(f"[{label}] ✅ 完成")
|
||||
else:
|
||||
log.error(f"[{label}] ❌ 返回码 {proc.returncode}")
|
||||
except subprocess.TimeoutExpired:
|
||||
log.error(f"[{label}] ❌ 超时 (30min)")
|
||||
except Exception as e:
|
||||
log.error(f"[{label}] ❌ 异常: {e}")
|
||||
|
||||
|
||||
def run_vacuum() -> None:
|
||||
"""VACUUM ANALYZE — 维护表统计信息。
|
||||
|
||||
VACUUM 不能在事务块里跑（psycopg2 会抛 ActiveSqlTransaction），
|
||||
所以用 AUTOCOMMIT 连接而不是 engine.begin()。
|
||||
"""
|
||||
try:
|
||||
from app.db import engine
|
||||
from sqlalchemy import text
|
||||
|
||||
tables = [
|
||||
"scan_tasks", "scan_tasks_archive",
|
||||
"scan_findings", "idempotency_records",
|
||||
"policy_versions", "audit_logs",
|
||||
]
|
||||
with engine.connect() as raw:
|
||||
conn = raw.execution_options(isolation_level="AUTOCOMMIT")
|
||||
for tbl in tables:
|
||||
log.info(f"[VACUUM] {tbl} ...")
|
||||
conn.execute(text(f"VACUUM ANALYZE {tbl}"))
|
||||
log.info("[VACUUM] ✅ 全部完成")
|
||||
except Exception as e:
|
||||
log.error(f"[VACUUM] ❌ 异常: {_redact(str(e))}")
|
||||
|
||||
|
||||
def seconds_until(target_hour: int, target_minute: int, weekday: int | None = None) -> float:
|
||||
"""计算到下次目标时间的秒数。
|
||||
|
||||
weekday: 0=周一...6=周日, None=每天
|
||||
"""
|
||||
now = datetime.now()
|
||||
candidate = now.replace(hour=target_hour, minute=target_minute, second=0, microsecond=0)
|
||||
|
||||
if weekday is not None:
|
||||
# 找到本周目标日
|
||||
days_ahead = weekday - now.weekday()
|
||||
if days_ahead < 0 or (days_ahead == 0 and now >= candidate):
|
||||
days_ahead += 7
|
||||
candidate += timedelta(days=days_ahead)
|
||||
else:
|
||||
if candidate <= now:
|
||||
candidate += timedelta(days=1)
|
||||
|
||||
return (candidate - now).total_seconds()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
log.info("=" * 60)
|
||||
log.info("MeshSafe 定时任务调度器启动")
|
||||
log.info(f"PID={os.getpid()} TZ=Asia/Shanghai")
|
||||
log.info("任务:")
|
||||
log.info(" - 每天 03:00 archive_old_tasks.py --apply")
|
||||
log.info(" - 每周一 03:10 VACUUM ANALYZE")
|
||||
log.info("=" * 60)
|
||||
|
||||
while True:
|
||||
# 计算两个任务下次执行时间，取更早的那个
|
||||
secs_archive = seconds_until(3, 0, weekday=None) # 每天 03:00
|
||||
secs_vacuum = seconds_until(3, 10, weekday=0) # 周一 03:10
|
||||
|
||||
if secs_archive <= secs_vacuum:
|
||||
log.info(f"下一个任务: 归档 → {secs_archive/3600:.1f}h 后")
|
||||
time.sleep(secs_archive)
|
||||
run(
|
||||
[sys.executable, "/app/scripts/archive_old_tasks.py", "--apply"],
|
||||
"ARCHIVE",
|
||||
)
|
||||
# 跑完后清下一次时间 (避免在 03:00 这一刻两个任务重叠)
|
||||
continue
|
||||
else:
|
||||
log.info(f"下一个任务: VACUUM → {secs_vacuum/3600:.1f}h 后")
|
||||
time.sleep(secs_vacuum)
|
||||
run_vacuum()
|
||||
continue
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,3 +1,9 @@
|
||||
-- 修复 real_type 字段长度：VARCHAR(128) → VARCHAR(256)
|
||||
-- libmagic 对 ZIP/ISO 等返回的描述可能超过 128 字符
|
||||
ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(256);
|
||||
|
||||
-- 修复 real_type 字段长度：VARCHAR(256) → VARCHAR(512)
|
||||
-- libmagic 对 MSI/OLE 复合文档会把 Title/Subject/Author/Creating Application 等元数据
|
||||
-- 全部拼进描述串（实测 go1.25.0.windows-amd64.msi 超过 334 字符），超出列宽会让
|
||||
-- 整条 scan_tasks INSERT 失败，连带的 scan_findings 一起丢失——被拒绝的文件反而不留痕。
|
||||
ALTER TABLE scan_tasks ALTER COLUMN real_type TYPE VARCHAR(512);
|
||||
|
||||
@@ -1 +1,5 @@
|
||||
concurrent test placeholder
|
||||
MeshSafe concurrent test control file.
|
||||
Running 10 EICAR files concurrently via API.
|
||||
|
||||
Payload:
|
||||
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
|
||||
|
||||
@@ -1 +1,4 @@
|
||||
rule hotload test
|
||||
This is a hot-reload test sample for MeshSafe YARA engine.
|
||||
The payload below should trigger Newly_Added_Rule_Test rule:
|
||||
|
||||
meshsafe-hot-reload-test-string
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
meshsafe-hot-reload-test-string present here
|
||||
@@ -1 +0,0 @@
|
||||
trojan-metadata-test payload here
|
||||
@@ -1 +1 @@
|
||||
yara match string placeholder
|
||||
trojan-metadata-test
|
||||
|
||||
Binary file not shown.
@@ -1 +1,5 @@
|
||||
regex pattern shell
|
||||
# MeshSafe YARA regex rule test sample
|
||||
# This should trigger Regex_Shell_Pattern
|
||||
|
||||
attacker payload:
|
||||
powershell -enc SQBFAFgAIAAoACgAbgBlAHcALQAtAG8AYgBqAGUAYwB0ACAA
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
powershell -enc ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwx+/==
|
||||
@@ -1 +0,0 @@
|
||||
size_condition_test_payload small file
|
||||
File diff suppressed because one or more lines are too long
@@ -43,7 +43,7 @@
|
||||
"filename": "L0-004_pdf_normal.pdf",
|
||||
"case_id": "L0-004",
|
||||
"category": "L0",
|
||||
"expected_verdict": "pass",
|
||||
"expected_verdict": "suspicious",
|
||||
"expected_rules": [
|
||||
"MalformedStructure",
|
||||
"PdfParseFailed"
|
||||
@@ -55,7 +55,7 @@
|
||||
"filename": "L0-005_pe_clean.exe",
|
||||
"case_id": "L1-008",
|
||||
"category": "L1",
|
||||
"expected_verdict": "pass",
|
||||
"expected_verdict": "suspicious",
|
||||
"expected_rules": [
|
||||
"DangerousExecutableExtension",
|
||||
"PeParseFailed"
|
||||
@@ -1047,14 +1047,14 @@
|
||||
"filename": "L5-005_injection_imports.exe",
|
||||
"case_id": "L5-005",
|
||||
"category": "L5",
|
||||
"expected_verdict": "reject",
|
||||
"expected_verdict": "review",
|
||||
"expected_rules": [
|
||||
"DangerousExecutableExtension",
|
||||
"PEArchInfo",
|
||||
"SuspiciousTimestamp",
|
||||
"UnsignedExecutable"
|
||||
],
|
||||
"description": "注入组合导入（需要真实 pefile 引入 API）",
|
||||
"description": "注入组合导入占位样本（928B 假 PE，实际没有导入表）。旧策略靠 载体45+时间戳20+未签名10+架构10 累加到 85 才判 reject，属于权重堆叠误报；权重收紧后应停在人工复核。真实注入器由 YARA C1_PE_ProcessInjection(critical=85) 直接拦截。",
|
||||
"_scan_verdict": "malicious"
|
||||
},
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user