diff --git a/.gitattributes b/.gitattributes index f8e512d..09d0ea4 100644 --- a/.gitattributes +++ b/.gitattributes @@ -5,3 +5,7 @@ *.md text eol=lf *.json text eol=lf *.txt text eol=lf +*.kt text eol=lf +*.kts text eol=lf +*.properties text eol=lf +android-buyer/gradlew text eol=lf diff --git a/AGENTS.md b/AGENTS.md index bb461e6..5b14ba7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,12 +77,8 @@ ## 验证 -当前尚无生产代码和可运行构建命令。完成 `T-001` 后，必须把真实命令同步到： - -- `init.ps1` / `init.sh` -- `docs/00-ai-start-here.md` -- `docs/03-tech-stack.md` -- `docs/05-coding-rules.md` -- `docs/current-state.md` +Android 基线使用根目录 `init.ps1` 或 `init.sh` 验证。默认执行单元测试任务并构建 +Debug APK；设置 `RUN_START_COMMAND=1` 时还会在已授权真机上安装和启动 App。 +后端建立后再把后端测试加入同一标准入口。 结束任务前按 `docs/clean-state-checklist.md` 收尾。 diff --git a/README.md b/README.md index 8c9645c..91fe31b 100644 --- a/README.md +++ b/README.md @@ -21,12 +21,12 @@ MVP 不自动提交订单、不支付、不绕过验证码或平台风控。 ```text cmroubao/ -├── android-buyer/ # 基于 Roubao 二次开发的采购 App，待接入 +├── android-buyer/ # Roubao main 固定 commit 的 Android 基线 ├── backend-api/ # Go-Gin API、管理页面和任务数据，待创建 ├── docs/ # Harness Coding 项目事实和执行约束 ├── AGENTS.md # AI coding agent 权威入口 -├── init.ps1 # Windows 标准验证入口，待 T-001 配置 -└── init.sh # Unix/WSL 等价入口，待 T-001 配置 +├── init.ps1 # Windows 标准构建/真机启动入口 +└── init.sh # Unix 等价构建/真机启动入口 ``` 管理人员使用 Web 管理端，采购人员使用 Android App；二者共享同一后端、任务数据 @@ -53,5 +53,5 @@ Android 8.0（API 26），目标为 Android 14（API 34）。具体版本和本 - [当前实现状态](docs/current-state.md) - [完整文档导航](docs/README.md) -当前仓库只有文档，还没有接入 Roubao、后端代码或可构建 APK。真实状态以 -[`docs/current-state.md`](docs/current-state.md) 为准。 +Roubao Android 基线已经接入并完成 Debug APK 构建和真机启动；Go 后端尚未建立。 +真实状态以 [`docs/current-state.md`](docs/current-state.md) 为准。 diff --git a/android-buyer/.gitignore b/android-buyer/.gitignore new file mode 100644 index 0000000..0609db4 --- /dev/null +++ b/android-buyer/.gitignore @@ -0,0 +1,114 @@ +# Built application files +*.apk +*.aar +*.ap_ +*.aab + +# Files for the ART/Dalvik VM +*.dex + +# Java class files +*.class + +# Generated files +bin/ +gen/ +out/ +release/ + +# Gradle files +.gradle/ +build/ +app/build/ + +# Local configuration file (sdk path, etc) +local.properties + +# Proguard folder generated by Eclipse +proguard/ + +# Log Files +*.log + +# Android Studio Navigation editor temp files +.navigation/ + +# Android Studio captures folder +captures/ + +# IntelliJ +*.iml +.idea/ +.idea/workspace.xml +.idea/tasks.xml +.idea/gradle.xml +.idea/assetWizardSettings.xml +.idea/dictionaries +.idea/libraries +.idea/caches +.idea/modules.xml +.idea/misc.xml +.idea/vcs.xml +.idea/deploymentTargetSelector.xml +.idea/compiler.xml +.idea/kotlinc.xml + +# Android Studio 3.1+ Annotation Processors +.cxx/ + +# Keystore files +# Uncomment the following lines if you do not want to check your keystore files in. +#*.jks +#*.keystore + +# External native build folder generated in Android Studio 2.2 and later +.externalNativeBuild +.cxx/ + +# Google Services (e.g. APIs or Firebase) +google-services.json + +# Freeline +freeline.py +freeline/ +freeline_project_description.json + +# fastlane +fastlane/report.xml +fastlane/Preview.html +fastlane/screenshots +fastlane/test_output +fastlane/readme.md + +# Version control +vcs.xml + +# lint +lint/intermediates/ +lint/generated/ +lint/outputs/ +lint/tmp/ + +# Android Profiling +*.hprof + +# Jetpack Compose +.compose/ + +# Kotlin +*.kotlin_module + +# OS generated files +.DS_Store +.DS_Store? +._* +.Spotlight-V100 +.Trashes +ehthumbs.db +Thumbs.db + +# Sensitive files +*.env +.env.* +secrets.properties +api_keys.properties diff --git a/android-buyer/LICENSE b/android-buyer/LICENSE new file mode 100644 index 0000000..1d48eaf --- /dev/null +++ b/android-buyer/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025 Roubao Team + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/android-buyer/README.md b/android-buyer/README.md new file mode 100644 index 0000000..bd904dc --- /dev/null +++ b/android-buyer/README.md @@ -0,0 +1,540 @@ +

+ 肉包 Roubao - AI Android Automation +

+ +

肉包 Roubao

+ +

+ 首款无需电脑的开源 AI 手机自动化助手 | AI Phone Automation Assistant +

+ +

+ 基于视觉语言模型 (VLM) · 原生 Android Kotlin · 多 Agent 协作架构 +

+ +

+ English | 简体中文 +

+ +

+ Platform + Min SDK + License + Language +

+ +

+ 演示 +

+ +

+ 首页 + 能力 + 执行记录 + 设置 +

+ +--- + +## 项目背景 + +2025 年 12 月，字节跳动联合中兴发布了「豆包手机助手」，一款能够自动操作手机完成复杂任务的 AI 助手。它能帮你比价下单、批量投简历、刷视频，甚至代打游戏。 + +首批 3 万台工程机定价 3499 元，上线当天即告售罄，二手市场一度炒到 5000+。 + +**买不到？那就自己做一个。** + +于是有了肉包——一个完全开源的 AI 手机自动化助手。 + +为什么叫「肉包」？因为作者不爱吃素。🥟 + +--- + +## 与同类项目的对比 + +| 特性 | 肉包 | 豆包手机 | 其他开源方案 | +|------|------|----------|--------------| +| 需要电脑 | ❌ 不需要 | ❌ 不需要 | ✅ 大多需要 | +| 需要购买硬件 | ❌ 不需要 | ✅ 需要 3499+ | ❌ 不需要 | +| 原生 Android 实现 | ✅ Kotlin | ✅ 原生 | ❌ Python | +| 开源 | ✅ MIT | ❌ 闭源 | ✅ 开源 | +| Skills/Tools 架构 | ✅ 完整 | ❓ 未知 | ❌ 无 | +| UI 设计 | ⭐⭐⭐½ | ⭐⭐⭐⭐ | ⭐⭐ | +| 自定义模型 | ✅ 支持 | ❌ 仅豆包 | ✅ 部分支持 | + +### 我们解决了什么问题？ + +**传统的手机自动化方案痛点：** + +- 必须连接电脑运行 ADB 命令 +- 需要部署 Python 环境和各种依赖 +- 只能在电脑端操作，手机必须通过数据线连接 +- 技术门槛高，普通用户难以使用 + +**肉包的解决方案：** + +一个 App，装上就能用。无需电脑、无需数据线、无需任何技术背景。 + +打开 App → 配置 API Key → 说出你想做的事 → 完成。 + +--- + +## 为什么选择肉包？ + +### 原生 Android 实现，不是 Python 脚本的封装 + +市面上几乎所有手机自动化开源项目（包括阿里的 MobileAgent）都是 **Python 实现**，需要： +- 在电脑上运行 Python 脚本 +- 手机通过 USB/WiFi ADB 连接电脑 +- 截图通过 ADB 传输到电脑，处理后再把操作指令传回手机 + +**肉包完全不同。** + +我们用 **Kotlin 重写了整个 MobileAgent 框架**，原生运行在 Android 设备上： +- 截图、分析、执行全部在手机本地完成 +- 无需电脑中转，延迟更低 +- 利用 Shizuku 获得系统级权限，而非繁琐的 ADB 命令 + +### 为什么需要 Shizuku？ + +Android 系统出于安全考虑，普通 App 无法： +- 模拟用户点击、滑动屏幕 +- 读取其他 App 的界面内容 +- 执行 `input tap`、`screencap` 等系统命令 + +传统方案需要连接电脑执行 ADB 命令。而 **Shizuku** 是一个优雅的解决方案： + +1. 通过无线调试或电脑 ADB **启动一次** Shizuku 服务 +2. 之后普通 App 就可以获得 ADB 级别的权限 +3. **无需 Root**，无需每次都连接电脑 + +这让肉包可以直接在手机上执行截图、点击、输入等操作，真正实现「一个 App 搞定一切」。 + +### 类 Claude Code 的 Tools/Skills 双层架构 + +受 [Claude Code](https://claude.ai/claude-code) 架构启发，肉包实现了 **Tools + Skills 双层 Agent 框架**： + +``` +用户: "帮我点份外卖" + │ + ▼ + ┌─────────────┐ + │ SkillManager │ ← 意图识别 + └─────────────┘ + │ + ┌────┴────┐ + │ │ + ▼ ▼ +🚀 快速路径 🤖 标准路径 +(Delegation) (GUI 自动化) + │ │ + ▼ ▼ +直接 DeepLink Agent 循环 +打开小美 AI 操作美团 App +``` + +**Tools 层（原子能力）** + +底层工具集，每个 Tool 完成一个独立操作： + +| Tool | 功能 | +|------|------| +| `search_apps` | 智能搜索已安装应用（支持拼音、语义） | +| `open_app` | 打开应用 | +| `deep_link` | 通过 DeepLink 跳转到 App 特定页面 | +| `clipboard` | 读写剪贴板 | +| `shell` | 执行 Shell 命令 | +| `http` | HTTP 请求（调用外部 API） | + +**Skills 层（用户意图）** + +面向用户的任务层，将自然语言映射到具体操作： + +| Skill | 类型 | 描述 | +|-------|------|------| +| 点外卖(小美) | Delegation | 直接打开小美 AI 让它帮你点 | +| 点外卖(美团) | GUI 自动化 | 在美团 App 上一步步操作 | +| 导航(高德) | Delegation | DeepLink 直达高德搜索 | +| 生成图片(即梦) | Delegation | 打开即梦 AI 生成图片 | +| 发微信 | GUI 自动化 | 自动操作微信发消息 | + +**两种执行模式：** + +1. **Delegation（委托）**：高置信度匹配时，直接通过 DeepLink 打开有 AI 能力的 App（如小美、豆包、即梦），让它们完成任务。**快速、一步到位。** + +2. **GUI 自动化**：没有 AI 能力的 App（如美团、微信），通过传统的截图-分析-操作循环完成。Skill 会提供操作步骤指导，提高成功率。 + +--- + +## 核心特性 + +### 🤖 智能 AI Agent + +- 基于先进的视觉语言模型（VLM），能够"看懂"屏幕内容 +- 自然语言指令，说人话就能操作手机 +- 智能决策，根据屏幕状态自动规划下一步操作 + +### 🎨 精心设计的 UI + +**这可能是所有手机自动化开源项目中 UI 做得最好看的。** + +- 现代化 Material 3 设计语言 +- 流畅的动画效果 +- 深色/浅色主题自适应 +- 精心设计的首次使用引导 +- 完整的中英文双语支持 + +### 🔧 高度可定制 + +- 支持多种 VLM：阿里云通义千问、OpenAI GPT-4V、Claude 等 +- 预设 API 服务商：阿里云、OpenAI、OpenRouter 一键切换 +- 从 API 动态获取可用模型列表，支持模糊搜索 +- 可配置自定义 API 端点，支持本地模型（Ollama、vLLM 等） + +### 🔐 安全保护 + +- API Key 使用 AES-256-GCM 加密存储 +- 检测到支付、密码等敏感页面自动停止 +- 任务执行全程可视，悬浮窗显示进度 +- 随时可以手动停止任务 +- 可选的云端崩溃上报（可在设置中关闭） + +### 🔓 Root 模式支持 + +当 Shizuku 以 Root 权限运行时，肉包可以启用 Root 模式： + +- **Root 模式**：解锁更多系统级操作能力 +- **su 命令**：允许执行 `su -c` 命令（需谨慎使用） +- **自动检测**：自动检测 Shizuku 权限等级（ADB/Root），非 Root 环境下该选项为灰色不可用 + +--- + +## 快速开始 + +### 前置要求 + +1. **Android 8.0 (API 26)** 或更高版本 +2. **WiFi 网络** - Shizuku 无线调试依赖 WiFi 连接，确保手机已连接 WiFi +3. **Shizuku** - 用于获取系统级控制权限 +4. **VLM API Key** - 需要视觉语言模型的 API 密钥（如阿里云通义千问） + +### 安装步骤 + +#### 1. 安装并启动 Shizuku + +Shizuku 是一个开源工具，可以让普通应用获得 ADB 权限，无需 Root。 + +- [Google Play](https://play.google.com/store/apps/details?id=moe.shizuku.privileged.api) +- [GitHub Releases](https://github.com/RikkaApps/Shizuku/releases) + +**启动方式（二选一）：** + +**无线调试（推荐，需 Android 11+）** +1. 进入 `设置 > 开发者选项 > 无线调试` +2. 开启无线调试 +3. 在 Shizuku App 中选择"无线调试"方式启动 + +**电脑 ADB** +1. 手机连接电脑，开启 USB 调试 +2. 执行：`adb shell sh /storage/emulated/0/Android/data/moe.shizuku.privileged.api/start.sh` + +#### 2. 安装肉包 + +从[上游 Releases](https://github.com/Turbo1123/roubao/releases) 页面下载最新 APK 安装。 + +#### 3. 授权与配置 + +1. 打开肉包 App +2. 在 Shizuku 中授权肉包 +3. **⚠️ 重要：进入设置页面，配置你的 API Key** + +### 获取 API Key + +**阿里云通义千问（推荐国内用户）** +1. 访问 [阿里云百炼平台](https://bailian.console.aliyun.com/) +2. 开通 DashScope 服务 +3. 在 API-KEY 管理中创建密钥 + +**OpenAI（需要代理）** +1. 访问 [OpenAI Platform](https://platform.openai.com/) +2. 创建 API Key + +--- + +## 使用示例 + +``` +帮我点个附近好吃的汉堡 +打开网易云音乐播放每日推荐 +帮我把最后一张照片发送到微博 +帮我在美团点一份猪脚饭 +打开B站看热门视频 +``` + +--- + +## 技术架构 + +``` +┌──────────────────────────────────────────────────────────────┐ +│ 肉包 App │ +├──────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ UI 层 (Compose) │ │ +│ │ HomeScreen / Settings / History │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────────▼────────────────────────────┐ │ +│ │ Skills 层 │ │ +│ │ SkillManager → 意图识别 → 快速路径/标准路径 │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ 点外卖 │ 导航 │ 打车 │ 发微信 │ AI画图 │ ... │ │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────────▼────────────────────────────┐ │ +│ │ Tools 层 │ │ +│ │ ToolManager → 原子能力封装 │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ search_apps │ open_app │ deep_link │ clipboard │ │ +│ │ │ shell │ http │ screenshot │ tap │ swipe │ type │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────────▼────────────────────────────┐ │ +│ │ Agent 层 │ │ +│ │ MobileAgent (移植自 MobileAgent-v3) │ │ +│ │ ┌───────────┬───────────┬───────────┬──────────┐ │ │ +│ │ │ Manager │ Executor │ Reflector │ Notetaker│ │ │ +│ │ │ (规划) │ (执行) │ (反思) │ (记录) │ │ │ +│ │ └───────────┴───────────┴───────────┴──────────┘ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────────▼────────────────────────────┐ │ +│ │ VLM Client │ │ +│ │ Qwen-VL / GPT-4V / Claude │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +├────────────────────────────┼────────────────────────────────┤ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Shizuku │ │ +│ │ System-level Control │ │ +│ │ screencap │ input tap │ input swipe │ am start │ │ +│ └─────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────┘ +``` + +### 工作流程 + +``` +用户输入指令 + │ + ▼ +┌─────────────────┐ +│ Skills 匹配 │ ← 检查是否有对应的 Skill +└─────────────────┘ + │ + ├── 高置信度 Delegation Skill ──▶ 直接 DeepLink 跳转 ──▶ 完成 + │ + ▼ +┌─────────────────┐ +│ 标准 Agent 循环 │ +└─────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────┐ + │ 1. 截图 - Shizuku screencap │ + │ 2. Manager 规划 - VLM 分析当前状态 │ + │ 3. Executor 决策 - 确定下一步操作 │ + │ 4. 执行动作 - tap/swipe/type/open_app │ + │ 5. Reflector 反思 - 评估操作效果 │ + │ 6. 循环直到完成或安全限制 │ + └──────────────────────────────────────────────┘ +``` + +### 项目结构 + +``` +app/src/main/java/com/roubao/autopilot/ +├── agent/ # AI Agent 核心 (移植自 MobileAgent-v3) +│ ├── MobileAgent.kt # Agent 主循环 +│ ├── Manager.kt # 规划 Agent +│ ├── Executor.kt # 执行 Agent +│ ├── ActionReflector.kt # 反思 Agent +│ ├── Notetaker.kt # 笔记 Agent +│ └── InfoPool.kt # 状态池 +│ +├── tools/ # Tools 层 - 原子能力 +│ ├── Tool.kt # Tool 接口定义 +│ ├── ToolManager.kt # 工具管理器 +│ ├── SearchAppsTool.kt # 应用搜索 +│ ├── OpenAppTool.kt # 打开应用 +│ ├── DeepLinkTool.kt # DeepLink 跳转 +│ ├── ClipboardTool.kt # 剪贴板操作 +│ ├── ShellTool.kt # Shell 命令 +│ └── HttpTool.kt # HTTP 请求 +│ +├── skills/ # Skills 层 - 用户意图 +│ ├── Skill.kt # Skill 接口定义 +│ ├── SkillRegistry.kt # Skill 注册表 +│ └── SkillManager.kt # Skill 管理器 +│ +├── controller/ # 设备控制 +│ ├── DeviceController.kt # Shizuku 控制器 +│ └── AppScanner.kt # 应用扫描 (拼音/语义搜索) +│ +├── vlm/ # VLM 客户端 +│ └── VLMClient.kt # API 调用封装 +│ +├── ui/ # 用户界面 +│ ├── screens/ # 各个页面 +│ ├── theme/ # 主题定义 +│ └── OverlayService.kt # 悬浮窗服务 +│ +├── data/ # 数据层 +│ └── SettingsManager.kt # 设置管理 +│ +└── App.kt # Application 入口 + +app/src/main/assets/ +└── skills.json # Skills 配置文件 +``` + +--- + +## 路线图 + +### 已完成 (v1.x) + +- [x] **原生 Android 实现** - Kotlin 重写 MobileAgent，摆脱 Python 依赖 +- [x] **Tools 层** - 原子能力封装（search_apps、deep_link、clipboard 等） +- [x] **Skills 层** - 用户意图映射，支持 Delegation 和 GUI 自动化两种模式 +- [x] **智能应用搜索** - 拼音、语义、分类多维度匹配 +- [x] **快速路径** - 高置信度 Skill 直接 DeepLink 跳转 + +### 🚀 v2.0 开发中 + +> 正在开发的重大更新，目前在 `roubao2.0+AccessibilityService` 分支 + +- [ ] **无障碍服务混合模式** - 集成 AccessibilityService，实现更精准的 UI 操作 + - 优先使用元素索引点击（不受屏幕变化影响） + - 智能回退：索引模式失败时自动切换到坐标模式 + - 无需 Root，进一步降低使用门槛 + +- [ ] **UI 树感知** - Agent 能够获取完整的 UI 结构 + - 识别可点击元素、输入框、滚动区域 + - 为 LLM 提供结构化 UI 上下文 + - 减少纯视觉误判 + +- [ ] **宏脚本系统** - 录制、存储、回放操作序列 + - 将执行过程录制为可重复播放的脚本 + - 支持循环播放、延时控制 + - 脚本管理界面（新增"脚本"导航页） + +- [ ] **设置增强** + - 无障碍服务开关与引导 + - 混合模式状态展示 + +### 近期计划 + +- [ ] **MCP (Model Context Protocol)** - 接入更多能力扩展，如日历、邮件、文件管理等 +- [ ] **执行录屏** - 保存任务执行过程视频，方便回顾和调试 +- [ ] **更多 Skills** - 扩充内置 Skills，支持用户自定义 + +### 中期计划 + +- [ ] **更多设备支持** - 适配更多 Android 设备和定制系统（MIUI、ColorOS、HarmonyOS 等） +- [ ] **本地模型** - 支持在设备端运行小型 VLM，实现离线使用 +- [ ] **任务模板** - 保存和分享常用任务 + +### 长期愿景 + +- [ ] **多应用协作** - 跨 App 联动完成复杂工作流 +- [ ] **智能学习** - 从用户操作习惯中学习，优化执行策略 +- [ ] **语音控制** - 语音唤醒和语音指令 + +--- + +## 开发 + +### 环境要求 + +- Android Studio Hedgehog 或更高版本 +- JDK 17 +- Android SDK 34 + +### 构建 + +```bash +# 克隆仓库 +git clone https://github.com/yourusername/roubao.git +cd roubao + +# 构建 Debug 版本 +./gradlew assembleDebug + +# 安装到设备 +./gradlew installDebug +``` + +--- + +## 问题反馈 + +遇到崩溃或 Bug？请通过以下方式反馈： + +### 导出日志 + +1. 打开肉包 App → 设置 +2. 找到「反馈与调试」分组 +3. 点击「导出日志」 +4. 选择分享方式（微信、邮件等）发送给开发者 + +### 日志包含的信息 + +- 设备型号和 Android 版本 +- 应用版本号 +- 崩溃堆栈信息（如有） +- 操作日志 + +> 💡 日志文件不包含您的 API Key 或个人隐私信息 + +### 提交 Issue + +请在 [GitHub Issues](https://github.com/Turbo1123/roubao/issues) 提交问题，附上： +- 问题描述 +- 复现步骤 +- 导出的日志文件 + +--- + +## 贡献 + +欢迎提交 Issue 和 Pull Request！ + +1. Fork 本仓库 +2. 创建特性分支 (`git checkout -b feature/amazing-feature`) +3. 提交更改 (`git commit -m 'Add some amazing feature'`) +4. 推送到分支 (`git push origin feature/amazing-feature`) +5. 开启 Pull Request + +--- + +## 许可证 + +本项目基于 MIT 许可证开源。详见 [LICENSE](LICENSE) 文件。 + +--- + +## 致谢 + +- [MobileAgent](https://github.com/X-PLUG/MobileAgent) - 阿里达摩院 X-PLUG 团队开源的移动端 Agent 框架，为本项目提供了重要的技术参考 +- [Shizuku](https://github.com/RikkaApps/Shizuku) - 优秀的 Android 权限管理框架 + +--- + +

+ Made with ❤️ by Roubao Team +

diff --git a/android-buyer/README_EN.md b/android-buyer/README_EN.md new file mode 100644 index 0000000..6198105 --- /dev/null +++ b/android-buyer/README_EN.md @@ -0,0 +1,540 @@ +

+ Roubao - AI Android Automation Assistant +

+ +

Roubao (肉包)

+ +

+ The First Open-Source AI Phone Automation Assistant Without PC +

+ +

+ Vision-Language Model (VLM) · Native Android Kotlin · Multi-Agent Architecture +

+ +

+ English | 简体中文 +

+ +

+ Platform + Min SDK + License + Language +

+ +

+ Demo +

+ +

+ Home + Capabilities + History + Settings +

+ +--- + +## Background + +In December 2025, ByteDance partnered with ZTE to release "Doubao Phone Assistant" - an AI assistant that can automatically operate your phone to complete complex tasks. It can compare prices and place orders, batch submit job applications, scroll through videos, and even play games for you. + +The first batch of 30,000 engineering units priced at 3,499 CNY (~$480) sold out on launch day, with resale prices reaching 5,000+ CNY. + +**Can't buy one? Let's build our own.** + +And so Roubao was born - a fully open-source AI phone automation assistant. + +Why "Roubao" (肉包, meaning "meat bun")? Because the author doesn't like vegetables. 🥟 + +--- + +## Comparison + +| Feature | Roubao | Doubao Phone | Other Open Source | +|---------|--------|--------------|-------------------| +| Requires PC | ❌ No | ❌ No | ✅ Most do | +| Requires Hardware | ❌ No | ✅ $480+ | ❌ No | +| Native Android | ✅ Kotlin | ✅ Native | ❌ Python | +| Open Source | ✅ MIT | ❌ Closed | ✅ Yes | +| Skills/Tools Architecture | ✅ Full | ❓ Unknown | ❌ No | +| UI Design | ⭐⭐⭐½ | ⭐⭐⭐⭐ | ⭐⭐ | +| Custom Models | ✅ Yes | ❌ Doubao only | ✅ Partial | + +### What Problem Do We Solve? + +**Pain points of traditional phone automation:** + +- Must connect to a computer to run ADB commands +- Need to set up Python environment and various dependencies +- Can only operate from computer, phone must be connected via USB +- High technical barrier, difficult for regular users + +**Roubao's Solution:** + +One app, install and use. No computer, no cables, no technical background required. + +Open App → Configure API Key → Tell it what you want → Done. + +--- + +## Why Choose Roubao? + +### Native Android Implementation, Not a Python Script Wrapper + +Almost all phone automation open-source projects (including Alibaba's MobileAgent) are **Python implementations**, requiring: +- Running Python scripts on a computer +- Phone connected to computer via USB/WiFi ADB +- Screenshots transferred to computer, processed, then commands sent back to phone + +**Roubao is completely different.** + +We **rewrote the entire MobileAgent framework in Kotlin**, running natively on Android: +- Screenshot, analysis, and execution all happen locally on the phone +- No computer relay, lower latency +- Uses Shizuku for system-level permissions instead of cumbersome ADB commands + +### Why Shizuku? + +For security reasons, regular Android apps cannot: +- Simulate user taps and swipes on screen +- Read UI content from other apps +- Execute system commands like `input tap` or `screencap` + +Traditional solutions require connecting to a computer for ADB commands. **Shizuku** is an elegant solution: + +1. Start Shizuku service **once** via wireless debugging or computer ADB +2. After that, regular apps can gain ADB-level permissions +3. **No Root required**, no need to connect to computer each time + +This allows Roubao to execute screenshots, taps, and input directly on the phone, truly achieving "one app does it all." + +### Claude Code-Inspired Tools/Skills Architecture + +Inspired by [Claude Code](https://claude.ai/claude-code), Roubao implements a **Tools + Skills dual-layer Agent framework**: + +``` +User: "Order me some food" + │ + ▼ + ┌─────────────┐ + │ SkillManager │ ← Intent Recognition + └─────────────┘ + │ + ┌────┴────┐ + │ │ + ▼ ▼ +🚀 Fast Path 🤖 Standard Path +(Delegation) (GUI Automation) + │ │ + ▼ ▼ +Direct DeepLink Agent Loop +Open Xiaomei AI Operate Meituan App +``` + +**Tools Layer (Atomic Capabilities)** + +Low-level toolkit where each Tool performs an independent operation: + +| Tool | Function | +|------|----------| +| `search_apps` | Smart app search (pinyin, semantic support) | +| `open_app` | Open application | +| `deep_link` | Jump to specific app page via DeepLink | +| `clipboard` | Read/write clipboard | +| `shell` | Execute Shell commands | +| `http` | HTTP requests (call external APIs) | + +**Skills Layer (User Intent)** + +User-facing task layer that maps natural language to specific operations: + +| Skill | Type | Description | +|-------|------|-------------| +| Order Food (Xiaomei) | Delegation | Directly open Xiaomei AI to help order | +| Order Food (Meituan) | GUI Automation | Step-by-step operation on Meituan App | +| Navigate (Amap) | Delegation | DeepLink directly to Amap search | +| Generate Image (Jimeng) | Delegation | Open Jimeng AI to generate images | +| Send WeChat | GUI Automation | Auto-operate WeChat to send messages | + +**Two Execution Modes:** + +1. **Delegation**: For high-confidence matches, directly open AI-capable apps (like Xiaomei, Doubao, Jimeng) via DeepLink to complete tasks. **Fast, one-step.** + +2. **GUI Automation**: For apps without AI capability (like Meituan, WeChat), complete tasks through traditional screenshot-analyze-operate loops. Skills provide step guidance for better success rates. + +--- + +## Key Features + +### 🤖 Intelligent AI Agent + +- Based on advanced Vision Language Models (VLM), can "see" and understand screen content +- Natural language commands - just speak normally +- Smart decision making, automatically plans next steps based on screen state + +### 🎨 Beautifully Designed UI + +**This is probably the best-looking UI among all open-source phone automation projects.** + +- Modern Material 3 design language +- Smooth animations +- Dark/Light theme auto-adaptation +- Carefully designed onboarding experience +- Full English and Chinese language support + +### 🔒 Safety Protection + +- API Key encrypted with AES-256-GCM +- Automatically stops when detecting payment or password pages +- Full visibility during task execution with overlay progress display +- Can manually stop tasks anytime +- Optional cloud crash reporting (can be disabled in settings) + +### 🔓 Root Mode Support + +When Shizuku runs with Root privileges, Roubao can enable Root mode: + +- **Root Mode**: Unlock more system-level operation capabilities +- **su Commands**: Allow execution of `su -c` commands (use with caution) +- **Auto Detection**: Automatically detects Shizuku privilege level (ADB/Root), option is grayed out in non-Root environments + +### 🔧 Highly Customizable + +- Supports multiple VLMs: Alibaba Qwen-VL, OpenAI GPT-4V, Claude, etc. +- Configurable custom API endpoints +- Can add custom models + +--- + +## Quick Start + +### Prerequisites + +1. **Android 8.0 (API 26)** or higher +2. **WiFi Network** - Shizuku wireless debugging requires WiFi connection, ensure your phone is connected to WiFi +3. **Shizuku** - For system-level control permissions +4. **VLM API Key** - Requires a Vision Language Model API key (e.g., Alibaba Qwen-VL) + +### Installation Steps + +#### 1. Install and Start Shizuku + +Shizuku is an open-source tool that allows regular apps to gain ADB-level permissions without Root. + +- [Google Play](https://play.google.com/store/apps/details?id=moe.shizuku.privileged.api) +- [GitHub Releases](https://github.com/RikkaApps/Shizuku/releases) + +**Startup Methods (choose one):** + +**Wireless Debugging (Recommended, requires Android 11+)** +1. Go to `Settings > Developer Options > Wireless Debugging` +2. Enable Wireless Debugging +3. In Shizuku app, select "Wireless Debugging" to start + +**Computer ADB** +1. Connect phone to computer, enable USB Debugging +2. Run: `adb shell sh /storage/emulated/0/Android/data/moe.shizuku.privileged.api/start.sh` + +#### 2. Install Roubao + +Download the latest APK from the [upstream Releases](https://github.com/Turbo1123/roubao/releases) page. + +#### 3. Authorization & Configuration + +1. Open Roubao app +2. Authorize Roubao in Shizuku +3. **⚠️ Important: Go to Settings and configure your API Key** + +### Getting an API Key + +**Alibaba Qwen-VL (Recommended for China users)** +1. Visit [Alibaba Cloud Bailian Platform](https://bailian.console.aliyun.com/) +2. Enable DashScope service +3. Create API key in API-KEY management + +**OpenAI (Requires proxy in some regions)** +1. Visit [OpenAI Platform](https://platform.openai.com/) +2. Create an API Key + +--- + +## Usage Examples + +``` +Order a tasty burger nearby +Open NetEase Music and play daily recommendations +Post my latest photo to Weibo +Order pork trotter rice on Meituan +Watch trending videos on Bilibili +``` + +--- + +## Architecture + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Roubao App │ +├──────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ UI Layer (Compose) │ │ +│ │ HomeScreen / Settings / History │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────────▼────────────────────────────┐ │ +│ │ Skills Layer │ │ +│ │ SkillManager → Intent Recognition → Fast/Standard │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ Order Food │ Navigate │ Taxi │ WeChat │ AI Art │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────────▼────────────────────────────┐ │ +│ │ Tools Layer │ │ +│ │ ToolManager → Atomic Capability Wrapper │ │ +│ │ ┌─────────────────────────────────────────────┐ │ │ +│ │ │ search_apps │ open_app │ deep_link │ clipboard │ │ +│ │ │ shell │ http │ screenshot │ tap │ swipe │ type │ │ +│ │ └─────────────────────────────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────────▼────────────────────────────┐ │ +│ │ Agent Layer │ │ +│ │ MobileAgent (ported from MobileAgent-v3) │ │ +│ │ ┌───────────┬───────────┬───────────┬──────────┐ │ │ +│ │ │ Manager │ Executor │ Reflector │ Notetaker│ │ │ +│ │ │ (Planning)│(Execution)│(Reflection)│ (Notes) │ │ │ +│ │ └───────────┴───────────┴───────────┴──────────┘ │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌────────────────────────▼────────────────────────────┐ │ +│ │ VLM Client │ │ +│ │ Qwen-VL / GPT-4V / Claude │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ │ +├────────────────────────────┼────────────────────────────────┤ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ Shizuku │ │ +│ │ System-level Control │ │ +│ │ screencap │ input tap │ input swipe │ am start │ │ +│ └─────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────┘ +``` + +### Workflow + +``` +User Input + │ + ▼ +┌─────────────────┐ +│ Skills Match │ ← Check for matching Skill +└─────────────────┘ + │ + ├── High-confidence Delegation Skill ──▶ Direct DeepLink ──▶ Done + │ + ▼ +┌─────────────────┐ +│ Standard Agent │ +│ Loop │ +└─────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────┐ + │ 1. Screenshot - Shizuku screencap │ + │ 2. Manager Planning - VLM analyzes state │ + │ 3. Executor Decision - Determine next step │ + │ 4. Execute Action - tap/swipe/type/open_app │ + │ 5. Reflector - Evaluate action outcome │ + │ 6. Loop until done or safety limit │ + └──────────────────────────────────────────────┘ +``` + +### Project Structure + +``` +app/src/main/java/com/roubao/autopilot/ +├── agent/ # AI Agent Core (ported from MobileAgent-v3) +│ ├── MobileAgent.kt # Agent main loop +│ ├── Manager.kt # Planning Agent +│ ├── Executor.kt # Execution Agent +│ ├── ActionReflector.kt # Reflection Agent +│ ├── Notetaker.kt # Notes Agent +│ └── InfoPool.kt # State pool +│ +├── tools/ # Tools Layer - Atomic Capabilities +│ ├── Tool.kt # Tool interface definition +│ ├── ToolManager.kt # Tool manager +│ ├── SearchAppsTool.kt # App search +│ ├── OpenAppTool.kt # Open app +│ ├── DeepLinkTool.kt # DeepLink jump +│ ├── ClipboardTool.kt # Clipboard operations +│ ├── ShellTool.kt # Shell commands +│ └── HttpTool.kt # HTTP requests +│ +├── skills/ # Skills Layer - User Intent +│ ├── Skill.kt # Skill interface definition +│ ├── SkillRegistry.kt # Skill registry +│ └── SkillManager.kt # Skill manager +│ +├── controller/ # Device Control +│ ├── DeviceController.kt # Shizuku controller +│ └── AppScanner.kt # App scanner (pinyin/semantic search) +│ +├── vlm/ # VLM Client +│ └── VLMClient.kt # API wrapper +│ +├── ui/ # User Interface +│ ├── screens/ # Screen composables +│ ├── theme/ # Theme definitions +│ └── OverlayService.kt # Overlay service +│ +├── data/ # Data Layer +│ └── SettingsManager.kt # Settings management +│ +└── App.kt # Application entry + +app/src/main/assets/ +└── skills.json # Skills configuration file +``` + +--- + +## Roadmap + +### Completed (v1.x) + +- [x] **Native Android Implementation** - Kotlin rewrite of MobileAgent, no Python dependency +- [x] **Tools Layer** - Atomic capability wrapper (search_apps, deep_link, clipboard, etc.) +- [x] **Skills Layer** - User intent mapping with Delegation and GUI Automation modes +- [x] **Smart App Search** - Multi-dimensional matching via pinyin, semantic, and category +- [x] **Fast Path** - High-confidence Skills direct DeepLink jump + +### 🚀 v2.0 In Development + +> Major update in progress on `roubao2.0+AccessibilityService` branch + +- [ ] **Accessibility Service Hybrid Mode** - Integrate AccessibilityService for more precise UI control + - Prioritize element index-based clicking (unaffected by screen changes) + - Smart fallback: auto-switch to coordinate mode when index mode fails + - No Root required, further lowering the barrier to entry + +- [ ] **UI Tree Awareness** - Agent can access complete UI structure + - Identify clickable elements, input fields, scrollable areas + - Provide structured UI context to LLM + - Reduce pure-visual misjudgments + +- [ ] **Macro Script System** - Record, store, and replay action sequences + - Record execution as replayable scripts + - Support loop playback, delay control + - Script management UI (new "Scripts" navigation tab) + +- [ ] **Settings Enhancement** + - Accessibility service toggle with guidance + - Hybrid mode status display + +### Near-term + +- [ ] **MCP (Model Context Protocol)** - Extended capabilities like calendar, email, file management +- [ ] **Execution Recording** - Save task execution videos for review and debugging +- [ ] **More Skills** - Expand built-in Skills, support user customization + +### Mid-term + +- [ ] **More Device Support** - Support more Android devices and custom systems (MIUI, ColorOS, HarmonyOS, etc.) +- [ ] **Local Models** - Support running small VLMs on-device for offline use +- [ ] **Task Templates** - Save and share common tasks + +### Long-term Vision + +- [ ] **Multi-app Collaboration** - Cross-app workflows for complex tasks +- [ ] **Smart Learning** - Learn from user habits to optimize execution strategies +- [ ] **Voice Control** - Voice activation and commands + +--- + +## Development + +### Requirements + +- Android Studio Hedgehog or later +- JDK 17 +- Android SDK 34 + +### Building + +```bash +# Clone repository +git clone https://github.com/yourusername/roubao.git +cd roubao + +# Build Debug version +./gradlew assembleDebug + +# Install to device +./gradlew installDebug +``` + +--- + +## Bug Reports + +Encountered a crash or bug? Here's how to report: + +### Export Logs + +1. Open Roubao App → Settings +2. Find "Feedback & Debug" section +3. Tap "Export Logs" +4. Choose a sharing method (Email, etc.) to send to developers + +### Log Contents + +- Device model and Android version +- App version +- Crash stack traces (if any) +- Operation logs + +> 💡 Log files do NOT contain your API Key or personal information + +### Submit an Issue + +Please submit issues on [GitHub Issues](https://github.com/Turbo1123/roubao/issues) with: +- Problem description +- Steps to reproduce +- Exported log file + +--- + +## Contributing + +Issues and Pull Requests are welcome! + +1. Fork this repository +2. Create a feature branch (`git checkout -b feature/amazing-feature`) +3. Commit your changes (`git commit -m 'Add some amazing feature'`) +4. Push to branch (`git push origin feature/amazing-feature`) +5. Open a Pull Request + +--- + +## License + +This project is open-sourced under the MIT License. See [LICENSE](LICENSE) file for details. + +--- + +## Acknowledgments + +- [MobileAgent](https://github.com/X-PLUG/MobileAgent) - Mobile Agent framework open-sourced by Alibaba DAMO Academy X-PLUG team, provided important technical reference for this project +- [Shizuku](https://github.com/RikkaApps/Shizuku) - Excellent Android permission management framework + +--- + +

+ Made with ❤️ by Roubao Team +

diff --git a/android-buyer/UPSTREAM.md b/android-buyer/UPSTREAM.md new file mode 100644 index 0000000..5cdc158 --- /dev/null +++ b/android-buyer/UPSTREAM.md @@ -0,0 +1,21 @@ +# Roubao Upstream Baseline + +- Repository: https://github.com/Turbo1123/roubao +- Imported branch: `main` +- Imported commit: `c8a6d7f03422eb01744b01f3ee77bf7757741f7e` +- Imported on: 2026-07-25 +- Upstream license: MIT; see `LICENSE` + +The source was imported as a vendored snapshot without the upstream `.git` +directory. Local changes are tracked by the parent `cmroubao` repository. + +## Baseline Adjustments + +Firebase Analytics and Crashlytics were removed from the imported baseline. +The upstream repository does not include `google-services.json`, and telemetry +is not required for the procurement feasibility probe. Local crash logs remain +available through the existing `CrashHandler`. + +The upstream `roubao2.0+AccessibilityService` branch is not the imported +baseline. Its accessibility implementation must be evaluated and ported in a +separate task after this baseline builds successfully. diff --git a/android-buyer/app/build.gradle.kts b/android-buyer/app/build.gradle.kts new file mode 100644 index 0000000..5bb7554 --- /dev/null +++ b/android-buyer/app/build.gradle.kts @@ -0,0 +1,91 @@ +plugins { + id("com.android.application") + id("org.jetbrains.kotlin.android") +} + +android { + namespace = "com.roubao.autopilot" + compileSdk = 34 + + defaultConfig { + applicationId = "com.roubao.autopilot" + minSdk = 26 + targetSdk = 34 + versionCode = 7 + versionName = "1.4.2" + + vectorDrawables { + useSupportLibrary = true + } + } + + buildTypes { + release { + isMinifyEnabled = false + proguardFiles( + getDefaultProguardFile("proguard-android-optimize.txt"), + "proguard-rules.pro" + ) + } + } + + compileOptions { + sourceCompatibility = JavaVersion.VERSION_17 + targetCompatibility = JavaVersion.VERSION_17 + } + + kotlinOptions { + jvmTarget = "17" + } + + buildFeatures { + compose = true + aidl = true + buildConfig = true + } + + composeOptions { + kotlinCompilerExtensionVersion = "1.5.5" + } + + packaging { + resources { + excludes += "/META-INF/{AL2.0,LGPL2.1}" + } + } +} + +dependencies { + // AndroidX Core + implementation("androidx.core:core-ktx:1.12.0") + implementation("androidx.core:core-splashscreen:1.0.1") + implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.6.2") + implementation("androidx.activity:activity-compose:1.8.1") + + // Security (Encrypted SharedPreferences) + implementation("androidx.security:security-crypto:1.1.0-alpha06") + + // Compose + implementation(platform("androidx.compose:compose-bom:2023.10.01")) + implementation("androidx.compose.ui:ui") + implementation("androidx.compose.ui:ui-graphics") + implementation("androidx.compose.ui:ui-tooling-preview") + implementation("androidx.compose.material3:material3") + + // Coroutines + implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.7.3") + + // OkHttp for API calls + implementation("com.squareup.okhttp3:okhttp:4.12.0") + + // Shizuku + implementation("dev.rikka.shizuku:api:13.1.5") + implementation("dev.rikka.shizuku:provider:13.1.5") + + // JSON + implementation("org.json:json:20231013") + + // Debug + debugImplementation("androidx.compose.ui:ui-tooling") + debugImplementation("androidx.compose.ui:ui-test-manifest") +} diff --git a/android-buyer/app/proguard-rules.pro b/android-buyer/app/proguard-rules.pro new file mode 100644 index 0000000..9e13270 --- /dev/null +++ b/android-buyer/app/proguard-rules.pro @@ -0,0 +1,3 @@ +# Add project specific ProGuard rules here. +-keep class com.example.autopilot.** { *; } +-keep class rikka.shizuku.** { *; } diff --git a/android-buyer/app/src/main/AndroidManifest.xml b/android-buyer/app/src/main/AndroidManifest.xml new file mode 100644 index 0000000..0031f86 --- /dev/null +++ b/android-buyer/app/src/main/AndroidManifest.xml @@ -0,0 +1,112 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/android-buyer/app/src/main/aidl/com/roubao/autopilot/IShellService.aidl b/android-buyer/app/src/main/aidl/com/roubao/autopilot/IShellService.aidl new file mode 100644 index 0000000..e42e675 --- /dev/null +++ b/android-buyer/app/src/main/aidl/com/roubao/autopilot/IShellService.aidl @@ -0,0 +1,6 @@ +package com.roubao.autopilot; + +interface IShellService { + void destroy() = 16777114; + String exec(String command) = 1; +} diff --git a/android-buyer/app/src/main/assets/skills.json b/android-buyer/app/src/main/assets/skills.json new file mode 100644 index 0000000..b942473 --- /dev/null +++ b/android-buyer/app/src/main/assets/skills.json @@ -0,0 +1,709 @@ +[ + { + "id": "order_food", + "name": "点外卖", + "description": "帮你点外卖、订餐", + "category": "外卖", + "keywords": ["点外卖", "叫外卖", "外卖", "点餐", "订餐", "吃饭", "饿了", "点吃的", "送餐", "订个餐"], + "params": [ + { + "name": "food", + "type": "string", + "description": "想吃什么", + "required": false, + "examples": ["汉堡", "猪脚饭", "麻辣烫", "炸鸡"] + } + ], + "related_apps": [ + { + "package": "com.meituan.android.beam", + "name": "小美", + "type": "delegation", + "deep_link": "beam://www.meituan.com/home", + "priority": 100, + "description": "美团 AI 助手，直接对话点餐，最快捷" + }, + { + "package": "me.ele", + "name": "饿了么", + "type": "gui_automation", + "priority": 60, + "steps": ["打开饿了么", "搜索食物", "选择商家", "下单"] + } + ] + }, + { + "id": "find_food", + "name": "找美食", + "description": "搜索附近好吃的、美食推荐", + "category": "外卖", + "keywords": ["附近有什么好吃的", "好吃的", "美食", "推荐吃的", "附近美食", "吃什么", "有啥好吃的", "找吃的", "想吃点什么"], + "params": [ + { + "name": "food", + "type": "string", + "description": "想吃的类型", + "required": false, + "examples": ["火锅", "烧烤", "日料", "川菜"] + } + ], + "related_apps": [ + { + "package": "com.meituan.android.beam", + "name": "小美", + "type": "delegation", + "deep_link": "beam://www.meituan.com/home", + "priority": 100, + "description": "美团 AI 助手，智能推荐附近美食" + }, + { + "package": "com.dianping.v1", + "name": "大众点评", + "type": "gui_automation", + "priority": 70, + "steps": ["打开大众点评", "搜索或浏览美食", "查看评价"] + } + ] + }, + { + "id": "find_fun", + "name": "找好玩的", + "description": "搜索附近好玩的地方、娱乐推荐", + "category": "生活", + "keywords": ["附近有什么好玩的", "好玩的", "玩什么", "周末去哪", "附近景点", "有啥好玩的", "去哪玩", "找地方玩"], + "params": [ + { + "name": "keyword", + "type": "string", + "description": "想玩的类型", + "required": false, + "examples": ["密室逃脱", "剧本杀", "电影院", "游乐园"] + } + ], + "related_apps": [ + { + "package": "com.meituan.android.beam", + "name": "小美", + "type": "delegation", + "deep_link": "beam://www.meituan.com/home", + "priority": 100, + "description": "美团 AI 助手，智能推荐附近娱乐" + }, + { + "package": "com.dianping.v1", + "name": "大众点评", + "type": "gui_automation", + "priority": 70, + "steps": ["打开大众点评", "搜索或浏览娱乐", "查看评价"] + } + ] + }, + { + "id": "book_hotel", + "name": "订酒店", + "description": "预订酒店、民宿", + "category": "生活", + "keywords": ["订酒店", "住酒店", "订房", "找酒店", "住宿", "民宿", "开房"], + "params": [ + { + "name": "keyword", + "type": "string", + "description": "地点或酒店名", + "required": false + } + ], + "related_apps": [ + { + "package": "com.meituan.android.beam", + "name": "小美", + "type": "delegation", + "deep_link": "beam://www.meituan.com/home", + "priority": 100, + "description": "美团 AI 助手，智能推荐酒店" + }, + { + "package": "com.Qunar", + "name": "去哪儿", + "type": "gui_automation", + "priority": 70, + "steps": ["打开去哪儿", "搜索酒店", "选择日期", "预订"] + }, + { + "package": "ctrip.android.view", + "name": "携程", + "type": "gui_automation", + "priority": 60, + "steps": ["打开携程", "搜索酒店", "选择日期", "预订"] + } + ] + }, + { + "id": "book_movie", + "name": "买电影票", + "description": "购买电影票", + "category": "生活", + "keywords": ["买电影票", "看电影", "电影票", "订电影票", "电影院"], + "params": [ + { + "name": "keyword", + "type": "string", + "description": "电影名", + "required": false + } + ], + "related_apps": [ + { + "package": "com.meituan.android.beam", + "name": "小美", + "type": "delegation", + "deep_link": "beam://www.meituan.com/home", + "priority": 100, + "description": "美团 AI 助手，查电影、买票" + }, + { + "package": "com.taobao.movie.android", + "name": "淘票票", + "type": "gui_automation", + "priority": 70, + "steps": ["打开淘票票", "选择电影", "选择场次", "购票"] + } + ] + }, + { + "id": "meituan_general", + "name": "问小美", + "description": "让美团 AI 助手帮忙处理各种生活服务", + "category": "生活", + "keywords": ["问小美", "小美", "美团", "帮我查", "帮我找", "帮我订"], + "params": [ + { + "name": "message", + "type": "string", + "description": "问题或需求", + "required": false + } + ], + "related_apps": [ + { + "package": "com.meituan.android.beam", + "name": "小美", + "type": "delegation", + "deep_link": "beam://www.meituan.com/home", + "priority": 100, + "description": "美团 AI 助手，支持外卖、酒店、电影、美食推荐等" + } + ] + }, + { + "id": "navigate", + "name": "导航", + "description": "导航到目的地", + "category": "出行", + "keywords": ["导航", "去", "怎么走", "路线", "地图", "到"], + "params": [ + { + "name": "destination", + "type": "string", + "description": "目的地", + "required": true + } + ], + "related_apps": [ + { + "package": "com.autonavi.minimap", + "name": "高德地图", + "type": "delegation", + "deep_link": "amap://search?keyword={destination}&sourceApplication=roubao", + "priority": 100 + }, + { + "package": "com.baidu.BaiduMap", + "name": "百度地图", + "type": "delegation", + "deep_link": "baidumap://map/direction?destination={destination}&mode=driving&src=roubao", + "priority": 90 + }, + { + "package": "com.tencent.map", + "name": "腾讯地图", + "type": "delegation", + "deep_link": "qqmap://map/search?keyword={destination}", + "priority": 80 + } + ] + }, + { + "id": "call_taxi", + "name": "打车", + "description": "叫车出行", + "category": "出行", + "keywords": ["打车", "叫车", "出租车", "网约车", "叫个车"], + "params": [ + { + "name": "destination", + "type": "string", + "description": "目的地", + "required": false + } + ], + "related_apps": [ + { + "package": "com.sdu.didi.psnger", + "name": "滴滴出行", + "type": "delegation", + "deep_link": "diditaxi://", + "priority": 100 + }, + { + "package": "com.autonavi.minimap", + "name": "高德地图", + "type": "gui_automation", + "priority": 80, + "description": "高德打车", + "steps": ["打开高德", "点击打车", "输入目的地", "叫车"] + }, + { + "package": "com.dache.caocao", + "name": "曹操出行", + "type": "delegation", + "deep_link": "caocao://", + "priority": 70 + }, + { + "package": "com.t3go.passenger", + "name": "T3出行", + "type": "delegation", + "deep_link": "t3go://", + "priority": 70 + } + ] + }, + { + "id": "generate_image", + "name": "AI画图", + "description": "使用 AI 生成图片", + "category": "AI", + "keywords": ["画图", "生成图片", "AI画图", "生成图", "画一张", "生成一张"], + "params": [ + { + "name": "prompt", + "type": "string", + "description": "图片描述", + "required": true + } + ], + "related_apps": [ + { + "package": "com.volcengine.jimeng", + "name": "即梦", + "type": "delegation", + "deep_link": "jimeng://create?prompt={prompt}", + "priority": 100 + }, + { + "package": "com.lark.bot", + "name": "豆包", + "type": "delegation", + "deep_link": "doubao://chat?message=帮我画{prompt}", + "priority": 90 + } + ] + }, + { + "id": "ai_chat", + "name": "AI对话", + "description": "与 AI 助手对话", + "category": "AI", + "keywords": ["问AI", "AI对话", "AI助手", "问一下", "帮我查"], + "params": [ + { + "name": "message", + "type": "string", + "description": "问题内容", + "required": false + } + ], + "related_apps": [ + { + "package": "com.lark.bot", + "name": "豆包", + "type": "delegation", + "deep_link": "doubao://chat?message={message}", + "priority": 100 + }, + { + "package": "com.moonshot.kimichat", + "name": "Kimi", + "type": "delegation", + "deep_link": "kimi://chat", + "priority": 90 + }, + { + "package": "com.alibaba.tongyi", + "name": "通义千问", + "type": "delegation", + "deep_link": "tongyi://chat", + "priority": 80 + } + ] + }, + { + "id": "play_music", + "name": "听音乐", + "description": "播放音乐", + "category": "音乐", + "keywords": ["听歌", "放歌", "音乐", "播放", "来首歌", "听音乐"], + "params": [ + { + "name": "song", + "type": "string", + "description": "歌曲名或歌手", + "required": false + } + ], + "related_apps": [ + { + "package": "com.netease.cloudmusic", + "name": "网易云音乐", + "type": "gui_automation", + "priority": 100, + "steps": ["打开网易云音乐", "搜索歌曲", "播放"] + }, + { + "package": "com.tencent.qqmusic", + "name": "QQ音乐", + "type": "gui_automation", + "priority": 90, + "steps": ["打开QQ音乐", "搜索歌曲", "播放"] + }, + { + "package": "com.kugou.android", + "name": "酷狗音乐", + "type": "gui_automation", + "priority": 80, + "steps": ["打开酷狗音乐", "搜索歌曲", "播放"] + }, + { + "package": "com.spotify.music", + "name": "Spotify", + "type": "gui_automation", + "priority": 70, + "steps": ["打开Spotify", "搜索歌曲", "播放"] + } + ] + }, + { + "id": "watch_video", + "name": "看视频", + "description": "看视频/刷视频", + "category": "视频", + "keywords": ["看视频", "刷视频", "追剧", "看电影", "视频"], + "params": [ + { + "name": "keyword", + "type": "string", + "description": "搜索关键词", + "required": false + } + ], + "related_apps": [ + { + "package": "tv.danmaku.bili", + "name": "哔哩哔哩", + "type": "gui_automation", + "priority": 100, + "steps": ["打开B站", "搜索或浏览视频", "播放"] + }, + { + "package": "com.ss.android.ugc.aweme", + "name": "抖音", + "type": "delegation", + "deep_link": "snssdk1128://", + "priority": 90 + }, + { + "package": "com.smile.gifmaker", + "name": "快手", + "type": "delegation", + "deep_link": "kwai://", + "priority": 80 + }, + { + "package": "com.youku.phone", + "name": "优酷", + "type": "gui_automation", + "priority": 70, + "steps": ["打开优酷", "搜索视频", "播放"] + }, + { + "package": "com.qiyi.video", + "name": "爱奇艺", + "type": "gui_automation", + "priority": 70, + "steps": ["打开爱奇艺", "搜索视频", "播放"] + } + ] + }, + { + "id": "send_message", + "name": "发消息", + "description": "给好友发消息", + "category": "社交", + "keywords": ["发消息", "发微信", "聊天", "给谁说"], + "params": [ + { + "name": "contact", + "type": "string", + "description": "联系人", + "required": false + }, + { + "name": "message", + "type": "string", + "description": "消息内容", + "required": false + } + ], + "related_apps": [ + { + "package": "com.tencent.mm", + "name": "微信", + "type": "gui_automation", + "priority": 100, + "steps": ["打开微信", "搜索联系人", "进入聊天", "输入消息", "发送"] + }, + { + "package": "com.tencent.mobileqq", + "name": "QQ", + "type": "gui_automation", + "priority": 80, + "steps": ["打开QQ", "搜索联系人", "进入聊天", "输入消息", "发送"] + } + ] + }, + { + "id": "post_social", + "name": "发动态", + "description": "发布社交动态", + "category": "社交", + "keywords": ["发微博", "发朋友圈", "发动态", "发状态"], + "params": [ + { + "name": "content", + "type": "string", + "description": "内容", + "required": false + } + ], + "related_apps": [ + { + "package": "com.tencent.mm", + "name": "微信", + "type": "gui_automation", + "priority": 100, + "description": "发朋友圈", + "steps": ["打开微信", "进入朋友圈", "点击发布", "输入内容", "发布"] + }, + { + "package": "com.sina.weibo", + "name": "微博", + "type": "gui_automation", + "priority": 90, + "steps": ["打开微博", "点击发布", "输入内容", "发布"] + }, + { + "package": "com.ss.android.ugc.aweme", + "name": "抖音", + "type": "gui_automation", + "priority": 80, + "description": "发抖音", + "steps": ["打开抖音", "点击发布", "录制或选择视频", "发布"] + } + ] + }, + { + "id": "post_xiaohongshu", + "name": "发小红书", + "description": "在小红书发布笔记", + "category": "社交", + "keywords": ["发小红书", "小红书", "发笔记", "小红书笔记", "红书"], + "params": [ + { + "name": "content", + "type": "string", + "description": "笔记内容", + "required": false + } + ], + "prompt_hint": "小红书笔记内容不能超过100字，请精炼表达", + "related_apps": [ + { + "package": "com.xingin.xhs", + "name": "小红书", + "type": "gui_automation", + "priority": 100, + "steps": ["打开小红书", "点击底部+号发布", "选择图片或视频", "输入标题和正文（正文不超过100字）", "添加话题标签", "发布"] + } + ] + }, + { + "id": "pay", + "name": "付款", + "description": "付款/扫码支付", + "category": "支付", + "keywords": ["付款", "支付", "付钱", "扫码付款", "买单"], + "params": [], + "related_apps": [ + { + "package": "com.eg.android.AlipayGphone", + "name": "支付宝", + "type": "delegation", + "deep_link": "alipays://platformapi/startapp?appId=20000056", + "priority": 100, + "description": "支付宝付款码" + }, + { + "package": "com.tencent.mm", + "name": "微信", + "type": "delegation", + "deep_link": "weixin://pay", + "priority": 90, + "description": "微信付款码" + } + ] + }, + { + "id": "scan", + "name": "扫码", + "description": "扫描二维码", + "category": "支付", + "keywords": ["扫码", "扫一扫", "扫描"], + "params": [], + "related_apps": [ + { + "package": "com.eg.android.AlipayGphone", + "name": "支付宝", + "type": "delegation", + "deep_link": "alipays://platformapi/startapp?appId=10000007", + "priority": 100 + }, + { + "package": "com.tencent.mm", + "name": "微信", + "type": "delegation", + "deep_link": "weixin://scanqrcode", + "priority": 90 + } + ] + }, + { + "id": "take_photo", + "name": "拍照", + "description": "打开相机拍照", + "category": "工具", + "keywords": ["拍照", "照相", "自拍", "拍摄"], + "params": [], + "related_apps": [ + { + "package": "com.android.camera", + "name": "相机", + "type": "gui_automation", + "priority": 100, + "steps": ["打开相机"] + } + ] + }, + { + "id": "set_alarm", + "name": "设闹钟", + "description": "设置闹钟提醒", + "category": "工具", + "keywords": ["闹钟", "提醒", "定时", "叫我起床", "设个闹钟"], + "params": [ + { + "name": "time", + "type": "string", + "description": "时间", + "required": false + } + ], + "related_apps": [ + { + "package": "com.android.deskclock", + "name": "时钟", + "type": "gui_automation", + "priority": 100, + "steps": ["打开时钟", "进入闹钟", "新建闹钟", "设置时间", "保存"] + } + ] + }, + { + "id": "shopping", + "name": "购物", + "description": "网上购物", + "category": "购物", + "keywords": ["买", "购物", "下单", "网购", "买东西"], + "params": [ + { + "name": "item", + "type": "string", + "description": "商品", + "required": false + } + ], + "related_apps": [ + { + "package": "com.taobao.taobao", + "name": "淘宝", + "type": "gui_automation", + "priority": 100, + "steps": ["打开淘宝", "搜索商品", "选择", "下单"] + }, + { + "package": "com.jingdong.app.mall", + "name": "京东", + "type": "gui_automation", + "priority": 90, + "steps": ["打开京东", "搜索商品", "选择", "下单"] + }, + { + "package": "com.xunmeng.pinduoduo", + "name": "拼多多", + "type": "gui_automation", + "priority": 80, + "steps": ["打开拼多多", "搜索商品", "选择", "下单"] + } + ] + }, + { + "id": "read_book", + "name": "看书", + "description": "阅读电子书", + "category": "阅读", + "keywords": ["看书", "读书", "阅读", "看小说"], + "params": [ + { + "name": "book", + "type": "string", + "description": "书名", + "required": false + } + ], + "related_apps": [ + { + "package": "com.tencent.weread", + "name": "微信读书", + "type": "gui_automation", + "priority": 100, + "steps": ["打开微信读书", "搜索或打开书架", "阅读"] + }, + { + "package": "com.dragon.read", + "name": "番茄小说", + "type": "gui_automation", + "priority": 80, + "steps": ["打开番茄小说", "搜索或打开书架", "阅读"] + } + ] + } +] diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/App.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/App.kt new file mode 100644 index 0000000..6711d2e --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/App.kt @@ -0,0 +1,77 @@ +package com.roubao.autopilot + +import android.app.Application +import android.content.pm.PackageManager +import com.roubao.autopilot.controller.AppScanner +import com.roubao.autopilot.controller.DeviceController +import com.roubao.autopilot.skills.SkillManager +import com.roubao.autopilot.tools.ToolManager +import com.roubao.autopilot.utils.CrashHandler +import rikka.shizuku.Shizuku + +class App : Application() { + + lateinit var deviceController: DeviceController + private set + lateinit var appScanner: AppScanner + private set + + override fun onCreate() { + super.onCreate() + instance = this + + // 初始化崩溃捕获（本地日志） + CrashHandler.getInstance().init(this) + + // 初始化 Shizuku + Shizuku.addRequestPermissionResultListener(REQUEST_PERMISSION_RESULT_LISTENER) + + // 初始化核心组件 + initializeComponents() + } + + private fun initializeComponents() { + // 初始化设备控制器 + deviceController = DeviceController(this) + deviceController.setCacheDir(cacheDir) + + // 初始化应用扫描器 + appScanner = AppScanner(this) + + // 初始化 Tools 层 + val toolManager = ToolManager.init(this, deviceController, appScanner) + + // 异步预扫描应用列表（避免 ANR） + println("[App] 开始异步扫描已安装应用...") + Thread { + appScanner.refreshApps() + println("[App] 已扫描 ${appScanner.getApps().size} 个应用") + }.start() + + // 初始化 Skills 层（传入 appScanner 用于检测已安装应用） + val skillManager = SkillManager.init(this, toolManager, appScanner) + println("[App] SkillManager 已加载 ${skillManager.getAllSkills().size} 个 Skills") + + println("[App] 组件初始化完成") + } + + override fun onTerminate() { + super.onTerminate() + Shizuku.removeRequestPermissionResultListener(REQUEST_PERMISSION_RESULT_LISTENER) + } + + companion object { + @Volatile + private var instance: App? = null + + fun getInstance(): App { + return instance ?: throw IllegalStateException("App 未初始化") + } + + private val REQUEST_PERMISSION_RESULT_LISTENER = + Shizuku.OnRequestPermissionResultListener { requestCode, grantResult -> + val granted = grantResult == PackageManager.PERMISSION_GRANTED + println("[Shizuku] Permission result: $granted") + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/MainActivity.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/MainActivity.kt new file mode 100644 index 0000000..c0466f7 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/MainActivity.kt @@ -0,0 +1,625 @@ +package com.roubao.autopilot + +import android.content.pm.PackageManager +import android.os.Bundle +import android.widget.Toast +import androidx.activity.ComponentActivity +import androidx.activity.SystemBarStyle +import androidx.activity.compose.setContent +import androidx.activity.enableEdgeToEdge +import androidx.core.splashscreen.SplashScreen.Companion.installSplashScreen +import androidx.activity.compose.BackHandler +import androidx.compose.animation.* +import androidx.compose.foundation.background +import androidx.compose.foundation.layout.* +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.* +import androidx.compose.material.icons.outlined.* +import androidx.compose.material3.* +import androidx.compose.runtime.* +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.unit.dp +import androidx.compose.ui.graphics.vector.ImageVector +import androidx.compose.ui.platform.LocalContext +import androidx.lifecycle.lifecycleScope +import android.net.Uri +import android.provider.Settings +import com.roubao.autopilot.agent.MobileAgent +import com.roubao.autopilot.controller.AppScanner +import com.roubao.autopilot.controller.DeviceController +import com.roubao.autopilot.data.* +import com.roubao.autopilot.ui.screens.* +import com.roubao.autopilot.ui.theme.* +import androidx.compose.ui.graphics.toArgb +import androidx.core.view.WindowCompat +import com.roubao.autopilot.vlm.GUIOwlClient +import com.roubao.autopilot.vlm.MAIUIClient +import com.roubao.autopilot.vlm.VLMClient +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.launch +import rikka.shizuku.Shizuku +import android.util.Log + +private const val TAG = "MainActivity" + +sealed class Screen(val route: String, val title: String, val icon: ImageVector, val selectedIcon: ImageVector) { + object Home : Screen("home", "肉包", Icons.Outlined.Home, Icons.Filled.Home) + object Capabilities : Screen("capabilities", "能力", Icons.Outlined.Star, Icons.Filled.Star) + object History : Screen("history", "记录", Icons.Outlined.List, Icons.Filled.List) + object Settings : Screen("settings", "设置", Icons.Outlined.Settings, Icons.Filled.Settings) +} + +class MainActivity : ComponentActivity() { + + private lateinit var deviceController: DeviceController + private lateinit var settingsManager: SettingsManager + private lateinit var executionRepository: ExecutionRepository + + private val mobileAgent = mutableStateOf(null) + private var shizukuAvailable = mutableStateOf(false) + + // 当前执行的协程 Job（用于停止任务） + private var currentExecutionJob: kotlinx.coroutines.Job? = null + + // 执行记录列表 + private val executionRecords = mutableStateOf>(emptyList()) + + // 是否正在执行（点击发送后立即为 true） + private val isExecuting = mutableStateOf(false) + + // 当前执行的记录 ID（用于停止后跳转） + private val currentRecordId = mutableStateOf(null) + + // 是否需要跳转到记录详情（悬浮窗停止后触发） + private val shouldNavigateToRecord = mutableStateOf(false) + + private val binderReceivedListener = Shizuku.OnBinderReceivedListener { + Log.d(TAG, "Shizuku binder received") + shizukuAvailable.value = true + if (checkShizukuPermission()) { + Log.d(TAG, "Shizuku permission granted, binding service") + deviceController.bindService() + } else { + Log.d(TAG, "Shizuku permission not granted") + } + } + + private val binderDeadListener = Shizuku.OnBinderDeadListener { + Log.d(TAG, "Shizuku binder dead") + shizukuAvailable.value = false + } + + private val permissionResultListener = Shizuku.OnRequestPermissionResultListener { _, grantResult -> + Log.d(TAG, "Shizuku permission result: $grantResult") + if (grantResult == PackageManager.PERMISSION_GRANTED) { + deviceController.bindService() + Toast.makeText(this, "Shizuku 权限已获取", Toast.LENGTH_SHORT).show() + } + } + + override fun onCreate(savedInstanceState: Bundle?) { + installSplashScreen() + super.onCreate(savedInstanceState) + + // 设置边到边显示，深色状态栏和导航栏 + enableEdgeToEdge( + statusBarStyle = SystemBarStyle.dark(android.graphics.Color.TRANSPARENT), + navigationBarStyle = SystemBarStyle.dark(android.graphics.Color.TRANSPARENT) + ) + + deviceController = DeviceController(this) + deviceController.setCacheDir(cacheDir) + settingsManager = SettingsManager(this) + executionRepository = ExecutionRepository(this) + + // 加载执行记录 + lifecycleScope.launch { + executionRecords.value = executionRepository.getAllRecords() + } + + // 添加 Shizuku 监听器 + Shizuku.addBinderReceivedListenerSticky(binderReceivedListener) + Shizuku.addBinderDeadListener(binderDeadListener) + Shizuku.addRequestPermissionResultListener(permissionResultListener) + + // 检查 Shizuku 状态 + checkAndUpdateShizukuStatus() + + // 预加载已安装应用 + lifecycleScope.launch(Dispatchers.IO) { + AppScanner(this@MainActivity).getApps() + } + + setContent { + val settings by settingsManager.settings.collectAsState() + BaoziTheme(themeMode = settings.themeMode) { + val colors = BaoziTheme.colors + // 动态更新系统栏颜色 + SideEffect { + val window = this@MainActivity.window + window.statusBarColor = colors.background.toArgb() + window.navigationBarColor = colors.backgroundCard.toArgb() + WindowCompat.getInsetsController(window, window.decorView).apply { + isAppearanceLightStatusBars = !colors.isDark + isAppearanceLightNavigationBars = !colors.isDark + } + } + + // 首次启动显示引导画面 + if (!settings.hasSeenOnboarding) { + OnboardingScreen( + onComplete = { + settingsManager.setOnboardingSeen() + } + ) + } else { + MainApp() + } + } + } + } + + @OptIn(ExperimentalMaterial3Api::class) + @Composable + fun MainApp() { + var currentScreen by remember { mutableStateOf(Screen.Home) } + var selectedRecord by remember { mutableStateOf(null) } + var showShizukuHelpDialog by remember { mutableStateOf(false) } + var hasShownShizukuHelp by remember { mutableStateOf(false) } + + val settings by settingsManager.settings.collectAsState() + val colors = BaoziTheme.colors + val agent = mobileAgent.value + val agentState by agent?.state?.collectAsState() ?: remember { mutableStateOf(null) } + val logs by agent?.logs?.collectAsState() ?: remember { mutableStateOf(emptyList()) } + val records by remember { executionRecords } + val isShizukuAvailable = shizukuAvailable.value && checkShizukuPermission() + val executing by remember { isExecuting } + val navigateToRecord by remember { shouldNavigateToRecord } + val recordId by remember { currentRecordId } + + // 监听跳转事件 + LaunchedEffect(navigateToRecord, recordId) { + if (navigateToRecord && recordId != null) { + // 找到对应的记录并跳转 + val record = records.find { it.id == recordId } + if (record != null) { + selectedRecord = record + currentScreen = Screen.History + } + shouldNavigateToRecord.value = false + } + } + + // 首次进入且 Shizuku 未连接时，显示帮助引导（只显示一次） + LaunchedEffect(Unit) { + if (!isShizukuAvailable && settings.hasSeenOnboarding && !hasShownShizukuHelp) { + hasShownShizukuHelp = true + showShizukuHelpDialog = true + } + } + + Scaffold( + modifier = Modifier.background(colors.background), + containerColor = colors.background, + bottomBar = { + if (selectedRecord == null) { + NavigationBar( + containerColor = colors.background, + contentColor = colors.textPrimary, + tonalElevation = 0.dp + ) { + listOf(Screen.Home, Screen.Capabilities, Screen.History, Screen.Settings).forEach { screen -> + val selected = currentScreen == screen + NavigationBarItem( + icon = { + Icon( + imageVector = if (selected) screen.selectedIcon else screen.icon, + contentDescription = screen.title + ) + }, + label = { Text(screen.title) }, + selected = selected, + onClick = { currentScreen = screen }, + colors = NavigationBarItemDefaults.colors( + selectedIconColor = if (colors.isDark) colors.textPrimary else Color.White, + selectedTextColor = colors.primary, + unselectedIconColor = colors.textSecondary, + unselectedTextColor = colors.textSecondary, + indicatorColor = colors.primary + ) + ) + } + } + } + } + ) { padding -> + Box( + modifier = Modifier + .fillMaxSize() + .padding(padding) + ) { + // 处理系统返回手势 + BackHandler(enabled = selectedRecord != null) { + selectedRecord = null + } + + // 详情页优先显示 + if (selectedRecord != null) { + HistoryDetailScreen( + record = selectedRecord!!, + onBack = { selectedRecord = null } + ) + } else { + // 主页面切换 + AnimatedContent( + targetState = currentScreen, + transitionSpec = { + fadeIn() togetherWith fadeOut() + }, + label = "screen" + ) { screen -> + when (screen) { + Screen.Home -> { + // 每次进入首页都检测 Shizuku 状态 + LaunchedEffect(Unit) { + checkAndUpdateShizukuStatus() + } + HomeScreen( + agentState = agentState, + logs = logs, + onExecute = { instruction -> + runAgent( + instruction = instruction, + apiKey = settings.apiKey, + baseUrl = settings.baseUrl, + model = settings.model, + maxSteps = settings.maxSteps, + isGUIAgent = settings.currentProvider.isGUIAgent, + providerId = settings.currentProviderId + ) + }, + onStop = { + mobileAgent.value?.stop() + }, + shizukuAvailable = isShizukuAvailable, + currentModel = settings.model, + onRefreshShizuku = { refreshShizukuStatus() }, + onShizukuRequired = { showShizukuHelpDialog = true }, + isExecuting = executing + ) + } + Screen.Capabilities -> CapabilitiesScreen() + Screen.History -> HistoryScreen( + records = records, + onRecordClick = { record -> selectedRecord = record }, + onDeleteRecord = { id -> deleteRecord(id) } + ) + Screen.Settings -> SettingsScreen( + settings = settings, + onUpdateApiKey = { settingsManager.updateApiKey(it) }, + onUpdateBaseUrl = { settingsManager.updateBaseUrl(it) }, + onUpdateModel = { settingsManager.updateModel(it) }, + onUpdateCachedModels = { settingsManager.updateCachedModels(it) }, + onUpdateThemeMode = { settingsManager.updateThemeMode(it) }, + onUpdateMaxSteps = { settingsManager.updateMaxSteps(it) }, + onUpdateRootModeEnabled = { settingsManager.updateRootModeEnabled(it) }, + onUpdateSuCommandEnabled = { settingsManager.updateSuCommandEnabled(it) }, + onSelectProvider = { settingsManager.selectProvider(it) }, + shizukuAvailable = isShizukuAvailable, + shizukuPrivilegeLevel = if (isShizukuAvailable) { + when (deviceController.getShizukuPrivilegeLevel()) { + DeviceController.ShizukuPrivilegeLevel.ROOT -> "ROOT" + DeviceController.ShizukuPrivilegeLevel.ADB -> "ADB" + else -> "NONE" + } + } else "NONE", + onFetchModels = { onSuccess, onError -> + lifecycleScope.launch { + val result = VLMClient.fetchModels(settings.baseUrl, settings.apiKey) + result.onSuccess { models -> + onSuccess(models) + }.onFailure { error -> + onError(error.message ?: "未知错误") + } + } + } + ) + } + } + } + } + } + + // Shizuku 帮助对话框 + if (showShizukuHelpDialog) { + ShizukuHelpDialog(onDismiss = { showShizukuHelpDialog = false }) + } + } + + private fun deleteRecord(id: String) { + lifecycleScope.launch { + executionRepository.deleteRecord(id) + executionRecords.value = executionRepository.getAllRecords() + } + } + + override fun onDestroy() { + super.onDestroy() + Shizuku.removeBinderReceivedListener(binderReceivedListener) + Shizuku.removeBinderDeadListener(binderDeadListener) + Shizuku.removeRequestPermissionResultListener(permissionResultListener) + deviceController.unbindService() + } + + private fun checkShizukuPermission(): Boolean { + return try { + val granted = Shizuku.checkSelfPermission() == PackageManager.PERMISSION_GRANTED + Log.d(TAG, "checkShizukuPermission: $granted") + granted + } catch (e: Exception) { + Log.e(TAG, "checkShizukuPermission error", e) + false + } + } + + private fun checkAndUpdateShizukuStatus() { + Log.d(TAG, "checkAndUpdateShizukuStatus called") + try { + val binderAlive = Shizuku.pingBinder() + Log.d(TAG, "Shizuku pingBinder: $binderAlive") + + if (binderAlive) { + shizukuAvailable.value = true + val hasPermission = checkShizukuPermission() + Log.d(TAG, "Shizuku hasPermission: $hasPermission") + + if (hasPermission) { + Log.d(TAG, "Binding Shizuku service") + deviceController.bindService() + } else { + Log.d(TAG, "Requesting Shizuku permission") + requestShizukuPermission() + } + } else { + Log.d(TAG, "Shizuku binder not alive") + shizukuAvailable.value = false + } + } catch (e: Exception) { + Log.e(TAG, "checkAndUpdateShizukuStatus error", e) + shizukuAvailable.value = false + } + } + + private fun refreshShizukuStatus() { + Log.d(TAG, "refreshShizukuStatus called by user") + Toast.makeText(this, "正在检查 Shizuku 状态...", Toast.LENGTH_SHORT).show() + checkAndUpdateShizukuStatus() + + if (shizukuAvailable.value && checkShizukuPermission()) { + Toast.makeText(this, "Shizuku 已连接", Toast.LENGTH_SHORT).show() + } else if (shizukuAvailable.value) { + Toast.makeText(this, "请在弹窗中授权 Shizuku", Toast.LENGTH_SHORT).show() + } else { + Toast.makeText(this, "请先启动 Shizuku App", Toast.LENGTH_SHORT).show() + } + } + + private fun requestShizukuPermission() { + try { + if (!Shizuku.pingBinder()) { + Toast.makeText(this, "请先启动 Shizuku App", Toast.LENGTH_SHORT).show() + return + } + + if (Shizuku.isPreV11()) { + Toast.makeText(this, "Shizuku 版本过低", Toast.LENGTH_SHORT).show() + return + } + + if (Shizuku.checkSelfPermission() == PackageManager.PERMISSION_GRANTED) { + Toast.makeText(this, "Shizuku 权限已获取", Toast.LENGTH_SHORT).show() + shizukuAvailable.value = true + deviceController.bindService() + return + } + + Shizuku.requestPermission(0) + } catch (e: Exception) { + Toast.makeText(this, "请先启动 Shizuku App", Toast.LENGTH_SHORT).show() + } + } + + private fun runAgent( + instruction: String, + apiKey: String, + baseUrl: String, + model: String, + maxSteps: Int, + isGUIAgent: Boolean = false, + providerId: String = "" + ) { + if (instruction.isBlank()) { + Toast.makeText(this, "请输入指令", Toast.LENGTH_SHORT).show() + return + } + // MAI-UI 本地部署不需要 API Key + val requiresApiKey = providerId != "mai_ui" + if (requiresApiKey && apiKey.isBlank()) { + Toast.makeText(this, "请输入 API Key", Toast.LENGTH_SHORT).show() + return + } + + // 检查悬浮窗权限 + if (!Settings.canDrawOverlays(this)) { + Toast.makeText(this, "请授予悬浮窗权限", Toast.LENGTH_LONG).show() + val intent = android.content.Intent( + Settings.ACTION_MANAGE_OVERLAY_PERMISSION, + Uri.parse("package:$packageName") + ) + startActivity(intent) + return + } + + // 立即设置执行状态为 true，显示停止按钮 + isExecuting.value = true + + // 根据服务商类型创建相应的客户端 + if (isGUIAgent) { + // GUI-Owl 模式 + val guiOwlClient = GUIOwlClient( + apiKey = apiKey, + model = model.ifBlank { "pre-gui_owl_7b" } + ) + mobileAgent.value = MobileAgent( + vlmClient = null, + controller = deviceController, + context = this, + guiOwlClient = guiOwlClient + ) + } else if (providerId == "mai_ui") { + // MAI-UI 模式 + val maiuiClient = MAIUIClient( + baseUrl = baseUrl.ifBlank { "http://localhost:8000/v1" }, + model = model.ifBlank { "MAI-UI-2B" } + ) + mobileAgent.value = MobileAgent( + vlmClient = null, + controller = deviceController, + context = this, + maiuiClient = maiuiClient + ) + } else { + // OpenAI 兼容模式 (阿里云 Qwen-VL, OpenAI, OpenRouter 等) + val vlmClient = VLMClient( + apiKey = apiKey, + baseUrl = baseUrl.ifBlank { "https://dashscope.aliyuncs.com/compatible-mode/v1" }, + model = model.ifBlank { "qwen3-vl-plus" } + ) + mobileAgent.value = MobileAgent(vlmClient, deviceController, this) + } + + // 设置停止回调，用于取消协程 + mobileAgent.value?.onStopRequested = { + currentExecutionJob?.cancel() + currentExecutionJob = null + } + + // 创建执行记录 + val record = ExecutionRecord( + title = generateTitle(instruction), + instruction = instruction, + startTime = System.currentTimeMillis(), + status = ExecutionStatus.RUNNING + ) + + // 保存当前记录 ID，用于停止后跳转 + currentRecordId.value = record.id + + // 取消之前的任务（如果有） + currentExecutionJob?.cancel() + + currentExecutionJob = lifecycleScope.launch { + // 保存初始记录 + executionRepository.saveRecord(record) + executionRecords.value = executionRepository.getAllRecords() + + try { + val result = mobileAgent.value!!.runInstruction(instruction, maxSteps) + + // 更新记录状态 + val agentState = mobileAgent.value?.state?.value + val steps = agentState?.executionSteps ?: emptyList() + val currentLogs = mobileAgent.value?.logs?.value ?: emptyList() + + val updatedRecord = record.copy( + endTime = System.currentTimeMillis(), + status = if (result.success) ExecutionStatus.COMPLETED else ExecutionStatus.FAILED, + steps = steps, + logs = currentLogs, + resultMessage = result.message + ) + executionRepository.saveRecord(updatedRecord) + executionRecords.value = executionRepository.getAllRecords() + + Toast.makeText(this@MainActivity, result.message, Toast.LENGTH_LONG).show() + + // 重置执行状态 + isExecuting.value = false + + // 延迟3秒后清空日志，恢复默认状态 + kotlinx.coroutines.delay(3000) + mobileAgent.value?.clearLogs() + } catch (e: kotlinx.coroutines.CancellationException) { + // 用户取消任务 - 使用 NonCancellable 确保清理操作完成 + kotlinx.coroutines.withContext(kotlinx.coroutines.NonCancellable) { + val agentState = mobileAgent.value?.state?.value + val steps = agentState?.executionSteps ?: emptyList() + val currentLogs = mobileAgent.value?.logs?.value ?: emptyList() + + println("[MainActivity] 取消任务 - steps: ${steps.size}, logs: ${currentLogs.size}") + + val updatedRecord = record.copy( + endTime = System.currentTimeMillis(), + status = ExecutionStatus.STOPPED, + steps = steps, + logs = currentLogs, + resultMessage = "已取消" + ) + executionRepository.saveRecord(updatedRecord) + executionRecords.value = executionRepository.getAllRecords() + + // 重置执行状态 + isExecuting.value = false + + Toast.makeText(this@MainActivity, "任务已停止", Toast.LENGTH_SHORT).show() + mobileAgent.value?.clearLogs() + + // 触发跳转到记录详情页 + shouldNavigateToRecord.value = true + } + } catch (e: Exception) { + // 更新失败记录 + val currentLogs = mobileAgent.value?.logs?.value ?: emptyList() + val updatedRecord = record.copy( + endTime = System.currentTimeMillis(), + status = ExecutionStatus.FAILED, + logs = currentLogs, + resultMessage = "错误: ${e.message}" + ) + executionRepository.saveRecord(updatedRecord) + executionRecords.value = executionRepository.getAllRecords() + + // 重置执行状态 + isExecuting.value = false + + Toast.makeText(this@MainActivity, "错误: ${e.message}", Toast.LENGTH_LONG).show() + + // 延迟3秒后清空日志，恢复默认状态 + kotlinx.coroutines.delay(3000) + mobileAgent.value?.clearLogs() + } + } + } + + private fun generateTitle(instruction: String): String { + // 生成简短标题 + val keywords = listOf( + "打开" to "打开应用", + "点" to "点餐", + "发" to "发送消息", + "看" to "浏览内容", + "搜" to "搜索", + "设置" to "调整设置", + "播放" to "播放媒体" + ) + for ((key, title) in keywords) { + if (instruction.contains(key)) { + return title + } + } + return if (instruction.length > 10) instruction.take(10) + "..." else instruction + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/agent/ActionReflector.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/ActionReflector.kt new file mode 100644 index 0000000..6750e49 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/ActionReflector.kt @@ -0,0 +1,78 @@ +package com.roubao.autopilot.agent + +/** + * ActionReflector Agent - 反思动作是否成功 + */ +class ActionReflector { + + /** + * 生成反思 Prompt + */ + fun getPrompt(infoPool: InfoPool): String = buildString { + append("You are an agent verifying whether the last action produced the expected behavior.\n\n") + + append("### User Request ###\n") + append("${infoPool.instruction}\n\n") + + append("### Progress Status ###\n") + if (infoPool.completedPlan.isNotEmpty()) { + append("${infoPool.completedPlan}\n\n") + } else { + append("No progress yet.\n\n") + } + + append("---\n") + append("The two attached images are phone screenshots taken BEFORE and AFTER your last action.\n\n") + + append("### Latest Action ###\n") + append("Action: ${infoPool.lastAction}\n") + append("Expectation: ${infoPool.lastSummary}\n\n") + + append("---\n") + append("Carefully examine whether the last action produced the expected behavior.\n\n") + + append("Note: For swiping to scroll, if the content before and after is exactly the same, ") + append("the swipe is considered Failed (C) - the page may have reached the bottom.\n\n") + + append("Provide your output in the following format:\n\n") + + append("### Outcome ###\n") + append("Choose from:\n") + append("A: Successful. The result meets the expectation.\n") + append("B: Failed. The action resulted in a wrong page. Need to return to previous state.\n") + append("C: Failed. The action produced no changes.\n\n") + + append("### Error Description ###\n") + append("If failed, describe the error. If successful, put \"None\".\n") + } + + /** + * 解析反思响应 + */ + fun parseResponse(response: String): ReflectorResult { + val outcomeSection = response + .substringAfter("### Outcome", "") + .substringBefore("### Error Description") + .replace("###", "") + .trim() + + val outcome = when { + outcomeSection.contains("A") -> "A" + outcomeSection.contains("B") -> "B" + outcomeSection.contains("C") -> "C" + else -> "C" + } + + val errorDescription = response + .substringAfter("### Error Description", "") + .replace("###", "") + .trim() + + return ReflectorResult(outcome, errorDescription) + } +} + +data class ReflectorResult( + val outcome: String, // A, B, C + val errorDescription: String +) diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/agent/ConversationMemory.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/ConversationMemory.kt new file mode 100644 index 0000000..6eae8df --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/ConversationMemory.kt @@ -0,0 +1,163 @@ +package com.roubao.autopilot.agent + +import android.graphics.Bitmap +import android.util.Base64 +import org.json.JSONArray +import org.json.JSONObject +import java.io.ByteArrayOutputStream + +/** + * 对话记忆管理 - 保存完整对话历史，图片用完即删 + * + * 参考 Open-AutoGLM 的设计： + * - 保持完整对话历史，让模型看到之前所有操作 + * - 图片用完即删，节省 token + */ +class ConversationMemory { + + private val messages = mutableListOf() + + /** + * 消息类型 + */ + data class Message( + val role: String, // "system", "user", "assistant" + val textContent: String, + var imageBase64: String? = null // 图片用完后置为 null + ) + + /** + * 添加系统消息（通常只在开始时添加一次） + */ + fun addSystemMessage(text: String) { + messages.add(Message(role = "system", textContent = text)) + } + + /** + * 添加用户消息（带截图） + */ + fun addUserMessage(text: String, image: Bitmap? = null) { + val imageBase64 = image?.let { bitmapToBase64(it) } + messages.add(Message(role = "user", textContent = text, imageBase64 = imageBase64)) + } + + /** + * 添加助手消息 + */ + fun addAssistantMessage(text: String) { + messages.add(Message(role = "assistant", textContent = text)) + } + + /** + * 删除最后一条用户消息中的图片（节省 token） + * 在获取模型响应后调用 + */ + fun stripLastUserImage() { + for (i in messages.indices.reversed()) { + if (messages[i].role == "user" && messages[i].imageBase64 != null) { + messages[i].imageBase64 = null + println("[ConversationMemory] 已删除第 $i 条消息的图片") + break + } + } + } + + /** + * 获取最近 N 条消息（不包括系统消息） + */ + fun getRecentMessages(count: Int): List { + val nonSystemMessages = messages.filter { it.role != "system" } + return nonSystemMessages.takeLast(count) + } + + /** + * 构建 OpenAI 兼容的 messages JSON + * @param includeImages 是否包含图片（最后一条用户消息的图片） + */ + fun toMessagesJson(includeImages: Boolean = true): JSONArray { + val jsonArray = JSONArray() + + for ((index, msg) in messages.withIndex()) { + val msgJson = JSONObject() + msgJson.put("role", msg.role) + + // 判断是否需要包含图片 + val shouldIncludeImage = includeImages && + msg.imageBase64 != null && + index == messages.indexOfLast { it.role == "user" } + + if (shouldIncludeImage && msg.imageBase64 != null) { + // 多模态消息格式 + val contentArray = JSONArray() + contentArray.put(JSONObject().apply { + put("type", "text") + put("text", msg.textContent) + }) + contentArray.put(JSONObject().apply { + put("type", "image_url") + put("image_url", JSONObject().apply { + put("url", "data:image/jpeg;base64,${msg.imageBase64}") + }) + }) + msgJson.put("content", contentArray) + } else { + // 纯文本消息 + msgJson.put("content", msg.textContent) + } + + jsonArray.put(msgJson) + } + + return jsonArray + } + + /** + * 获取消息数量 + */ + fun size(): Int = messages.size + + /** + * 清空所有消息 + */ + fun clear() { + messages.clear() + } + + /** + * 获取 token 估算（粗略） + * 中文约 2 字符/token，英文约 4 字符/token + */ + fun estimateTokens(): Int { + var total = 0 + for (msg in messages) { + // 文本 token + total += msg.textContent.length / 3 + // 图片 token（约 1000 token 每张） + if (msg.imageBase64 != null) { + total += 1000 + } + } + return total + } + + /** + * Bitmap 转 Base64 + */ + private fun bitmapToBase64(bitmap: Bitmap): String { + val outputStream = ByteArrayOutputStream() + bitmap.compress(Bitmap.CompressFormat.JPEG, 70, outputStream) + val bytes = outputStream.toByteArray() + return Base64.encodeToString(bytes, Base64.NO_WRAP) + } + + companion object { + /** + * 创建带系统提示的记忆 + */ + fun withSystemPrompt(systemPrompt: String): ConversationMemory { + return ConversationMemory().apply { + addSystemMessage(systemPrompt) + } + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Executor.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Executor.kt new file mode 100644 index 0000000..2cc08fd --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Executor.kt @@ -0,0 +1,194 @@ +package com.roubao.autopilot.agent + +/** + * Executor Agent - 决定具体执行什么动作 + */ +class Executor { + + companion object { + val GUIDELINES = """ +General: +- For any pop-up window, close it (e.g., by clicking 'Don't Allow' or 'Accept') before proceeding. +- For requests that are questions, remember to use the `answer` action to reply before finish! +- If the desired state is already achieved, you can just complete the task. + +Action Related: +- Use `open_app` to open an app, do not use the app drawer. +- Consider using `swipe` to reveal additional content. +- If swiping doesn't change the page, it may have reached the bottom. + +Text Related: +- To input text: first click the input box, make sure keyboard is visible, then use `type` action. +- To clear text: long press the backspace button in the keyboard. + """.trimIndent() + } + + /** + * 生成执行 Prompt + */ + fun getPrompt(infoPool: InfoPool): String = buildString { + append("You are an agent who can operate an Android phone. ") + append("Decide the next action based on the current state.\n\n") + + append("### User Request ###\n") + append("${infoPool.instruction}\n\n") + + append("### Overall Plan ###\n") + append("${infoPool.plan}\n\n") + + append("### Current Subgoal ###\n") + val subgoals = infoPool.plan.split(Regex("(?<=\\d)\\. ")).take(3) + append("${subgoals.joinToString(". ")}\n\n") + + append("### Progress Status ###\n") + if (infoPool.progressStatus.isNotEmpty()) { + append("${infoPool.progressStatus}\n\n") + } else { + append("No progress yet.\n\n") + } + + append("### Guidelines ###\n") + append("$GUIDELINES\n\n") + + append("---\n") + append("Examine all information and decide on the next action.\n\n") + + append("#### Atomic Actions ####\n") + append("- click(coordinate): Click at (x, y). Example: {\"action\": \"click\", \"coordinate\": [x, y]}\n") + append("- double_tap(coordinate): Double tap at (x, y) for zoom or like. Example: {\"action\": \"double_tap\", \"coordinate\": [x, y]}\n") + append("- long_press(coordinate): Long press at (x, y). Example: {\"action\": \"long_press\", \"coordinate\": [x, y]}\n") + append("- type(text): Type text into activated input box. Example: {\"action\": \"type\", \"text\": \"hello\"}\n") + append("- swipe(coordinate, coordinate2): Swipe from point1 to point2. Example: {\"action\": \"swipe\", \"coordinate\": [x1, y1], \"coordinate2\": [x2, y2]}\n") + append("- system_button(button): Press Back/Home/Enter. Example: {\"action\": \"system_button\", \"button\": \"Back\"}\n") + append("- open_app(text): Open an app by name. ALWAYS use this instead of looking for app icons on screen! Example: {\"action\": \"open_app\", \"text\": \"设置\"}\n") + if (infoPool.installedApps.isNotEmpty()) { + append(" Available apps: ${infoPool.installedApps}\n") + } + append("- wait(duration): Wait for page loading. Duration in seconds (1-10). Example: {\"action\": \"wait\", \"duration\": 3}\n") + append("- take_over(message): Request user to manually complete login/captcha/verification. Example: {\"action\": \"take_over\", \"message\": \"请完成登录验证\"}\n") + append("- answer(text): Answer user's question. Example: {\"action\": \"answer\", \"text\": \"The answer is...\"}\n") + append("\n") + + append("#### Sensitive Operations ####\n") + append("For payment, password, or privacy-related actions, add 'message' field to request user confirmation:\n") + append("Example: {\"action\": \"click\", \"coordinate\": [500, 800], \"message\": \"确认支付 ¥100\"}\n") + append("The user will see a confirmation dialog and can choose to confirm or cancel.\n") + append("\n") + + append("### Latest Action History ###\n") + if (infoPool.actionHistory.isNotEmpty()) { + val numActions = minOf(5, infoPool.actionHistory.size) + val latestActions = infoPool.actionHistory.takeLast(numActions) + val latestSummaries = infoPool.summaryHistory.takeLast(numActions) + val latestOutcomes = infoPool.actionOutcomes.takeLast(numActions) + val latestErrors = infoPool.errorDescriptions.takeLast(numActions) + + latestActions.forEachIndexed { i, act -> + val outcome = latestOutcomes.getOrNull(i) ?: "?" + if (outcome == "A") { + append("- Action: $act | Description: ${latestSummaries.getOrNull(i)} | Outcome: Successful\n") + } else { + append("- Action: $act | Description: ${latestSummaries.getOrNull(i)} | Outcome: Failed | Error: ${latestErrors.getOrNull(i)}\n") + } + } + } else { + append("No actions have been taken yet.\n") + } + append("\n") + + append("---\n") + append("IMPORTANT:\n") + append("1. Do NOT repeat previously failed actions. Try a different approach.\n") + append("2. Prioritize the current subgoal.\n") + append("3. Always analyze the screen BEFORE deciding on an action.\n\n") + + append("Provide your output in the following format:\n\n") + append("### Thought ###\n") + append("1. **Screen**: What app/page is currently shown? Is it the expected page for the current subgoal?\n") + append("2. **Blocker**: Any popup, dialog, keyboard, or loading state that needs to be handled first?\n") + append("3. **Target**: Is the target element visible? If not, should I scroll or navigate?\n") + append("4. **Decision**: Based on the above analysis, what action should I take and why?\n\n") + append("### Action ###\n") + append("A valid JSON specifying the action. Example: {\"action\":\"click\", \"coordinate\": [500, 800]}\n\n") + append("### Description ###\n") + append("A brief description of the chosen action.\n") + } + + /** + * 解析执行响应 + * 支持两种格式: + * 1. 标准格式 (### Thought / ### Action / ### Description) + * 2. MAI-UI 格式 (......) + */ + fun parseResponse(response: String): ExecutorResult { + // 检测是否为 MAI-UI 格式 + if (response.contains("") || response.contains("")) { + return parseMAIUIResponse(response) + } + + // 标准格式解析 + val thought = response + .substringAfter("### Thought", "") + .substringBefore("### Action") + .replace("###", "") + .trim() + + val actionStr = response + .substringAfter("### Action", "") + .substringBefore("### Description") + .replace("###", "") + .replace("```json", "") + .replace("```", "") + .trim() + + val description = response + .substringAfter("### Description", "") + .replace("###", "") + .trim() + + val action = Action.fromJson(actionStr) + + return ExecutorResult(thought, action, actionStr, description) + } + + /** + * 解析 MAI-UI 格式响应 + * 格式: ...{"name": "mobile_use", "arguments": {...}} + */ + private fun parseMAIUIResponse(response: String): ExecutorResult { + val thought = Action.extractThinking(response) + val action = Action.fromMAIUIFormat(response) + + // 从 action 生成描述 + val description = when (action?.type) { + "click" -> "点击坐标 (${action.x}, ${action.y})" + "long_press" -> "长按坐标 (${action.x}, ${action.y})" + "double_tap" -> "双击坐标 (${action.x}, ${action.y})" + "swipe" -> { + if (action.direction != null) { + "向${action.direction}滑动" + } else { + "从 (${action.x}, ${action.y}) 滑动到 (${action.x2}, ${action.y2})" + } + } + "type" -> "输入文字: ${action.text}" + "open_app" -> "打开应用: ${action.text}" + "system_button" -> "按${action.button}键" + "wait" -> "等待" + "terminate" -> "任务${if (action.status == "success") "完成" else "失败"}" + "answer" -> "回答: ${action.text?.take(30)}" + "take_over" -> "请求用户: ${action.text}" + else -> action?.type ?: "未知动作" + } + + val actionStr = action?.toJson() ?: "" + return ExecutorResult(thought, action, actionStr, description) + } +} + +data class ExecutorResult( + val thought: String, + val action: Action?, + val actionStr: String, + val description: String +) diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/agent/InfoPool.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/InfoPool.kt new file mode 100644 index 0000000..1a5bb2d --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/InfoPool.kt @@ -0,0 +1,226 @@ +package com.roubao.autopilot.agent + +import org.json.JSONObject + +/** + * Agent 状态池 - 保存所有执行过程中的信息 + */ +data class InfoPool( + // 用户指令 + var instruction: String = "", + + // 规划相关 + var plan: String = "", + var completedPlan: String = "", + var progressStatus: String = "", + var currentSubgoal: String = "", + + // 动作历史 + val actionHistory: MutableList = mutableListOf(), + val summaryHistory: MutableList = mutableListOf(), + val actionOutcomes: MutableList = mutableListOf(), // A=成功, B=错误页面, C=无变化 + val errorDescriptions: MutableList = mutableListOf(), + + // 最近一次动作 + var lastAction: Action? = null, + var lastActionThought: String = "", + var lastSummary: String = "", + + // 笔记 + var importantNotes: String = "", + + // 错误处理 + var errorFlagPlan: Boolean = false, + val errToManagerThresh: Int = 2, + + // 屏幕尺寸 + var screenWidth: Int = 1080, + var screenHeight: Int = 2400, + + // 额外知识 + var additionalKnowledge: String = "", + + // Skill 上下文（从 SkillManager 获取的相关技能信息） + var skillContext: String = "", + + // 对话记忆（保存完整对话历史，用于 Executor） + var executorMemory: ConversationMemory? = null, + + // 已安装应用列表（用于 open_app 动作） + var installedApps: String = "" +) + +/** + * 动作定义 + * 支持的动作类型: + * - click, double_tap, long_press: 点击类操作 + * - swipe, drag: 滑动类操作 + * - type: 输入文字 + * - system_button: 系统按键 (Back, Home, Enter) + * - open_app, open: 打开应用 + * - answer: 回答问题 + * - wait: 等待 + * - take_over, ask_user: 人机交互 + * - terminate: 任务结束 + */ +data class Action( + val type: String, + val x: Int? = null, + val y: Int? = null, + val x2: Int? = null, + val y2: Int? = null, + val text: String? = null, + val button: String? = null, // Back, Home, Enter, menu + val duration: Int? = null, // wait 动作的等待时长（秒） + val message: String? = null, // take_over/ask_user 动作的提示消息 + val needConfirm: Boolean = false, + val direction: String? = null, // swipe 方向: up, down, left, right + val status: String? = null // terminate 状态: success, fail +) { + companion object { + private const val SCALE_FACTOR = 999 // MAI-UI 坐标缩放因子 + + /** + * 从 JSON 字符串解析 Action + * 支持两种格式: + * 1. 标准格式: {"action": "click", "coordinate": [x, y]} + * 2. MAI-UI 格式: {"name": "mobile_use", "arguments": {...}} + */ + fun fromJson(json: String): Action? { + val cleanJson = json.trim() + .replace("```json", "") + .replace("```", "") + .trim() + + // 检查是否是 MAI-UI 的 格式 + if (cleanJson.contains("")) { + return fromMAIUIFormat(cleanJson) + } + + return fromStandardJson(cleanJson) + } + + /** + * 解析标准 JSON 格式 + */ + private fun fromStandardJson(json: String): Action? { + return try { + val obj = JSONObject(json) + val type = obj.optString("action", "") + + Action( + type = type, + x = obj.optJSONArray("coordinate")?.optInt(0), + y = obj.optJSONArray("coordinate")?.optInt(1), + x2 = obj.optJSONArray("coordinate2")?.optInt(0), + y2 = obj.optJSONArray("coordinate2")?.optInt(1), + text = obj.optString("text", null), + button = obj.optString("button", null), + duration = if (obj.has("duration")) obj.optInt("duration", 3) else null, + message = obj.optString("message", null), + needConfirm = obj.optBoolean("need_confirm", false), + direction = obj.optString("direction", null), + status = obj.optString("status", null) + ) + } catch (e: Exception) { + null + } + } + + /** + * 解析 MAI-UI 的 格式 + * 格式: {"name": "mobile_use", "arguments": {"action": "click", "coordinate": [x, y]}} + */ + fun fromMAIUIFormat(response: String): Action? { + return try { + // 提取 内容 + val toolCallRegex = Regex("\\s*(.+?)\\s*", RegexOption.DOT_MATCHES_ALL) + val match = toolCallRegex.find(response) ?: return null + val toolCallJson = match.groupValues[1].trim() + + val obj = JSONObject(toolCallJson) + val arguments = obj.optJSONObject("arguments") ?: return null + + val type = arguments.optString("action", "") + + // MAI-UI 坐标是 0-999 归一化的，需要标记（在执行时处理） + val coordinate = arguments.optJSONArray("coordinate") + var x = coordinate?.optInt(0) + var y = coordinate?.optInt(1) + + // MAI-UI drag 动作使用 start_coordinate 和 end_coordinate + val startCoord = arguments.optJSONArray("start_coordinate") + val endCoord = arguments.optJSONArray("end_coordinate") + var x2: Int? = null + var y2: Int? = null + + if (startCoord != null && endCoord != null) { + x = startCoord.optInt(0) + y = startCoord.optInt(1) + x2 = endCoord.optInt(0) + y2 = endCoord.optInt(1) + } + + // 映射 MAI-UI 的动作类型到我们的类型 + val mappedType = when (type) { + "open" -> "open_app" + "double_click" -> "double_tap" + "drag" -> "swipe" // drag 和 swipe 在执行层面相同 + "ask_user" -> "take_over" + else -> type + } + + Action( + type = mappedType, + x = x, + y = y, + x2 = x2, + y2 = y2, + text = arguments.optString("text", null), + button = arguments.optString("button", null), + direction = arguments.optString("direction", null), + status = arguments.optString("status", null) + ) + } catch (e: Exception) { + println("[Action] MAI-UI 格式解析失败: ${e.message}") + null + } + } + + /** + * 从 MAI-UI 响应中提取思考过程 + */ + fun extractThinking(response: String): String { + val thinkingRegex = Regex("\\s*(.+?)\\s*", RegexOption.DOT_MATCHES_ALL) + val match = thinkingRegex.find(response) + return match?.groupValues?.get(1)?.trim() ?: "" + } + } + + fun toJson(): String { + val obj = JSONObject() + obj.put("action", type) + + if (x != null && y != null) { + val coord = org.json.JSONArray() + coord.put(x) + coord.put(y) + obj.put("coordinate", coord) + } + if (x2 != null && y2 != null) { + val coord2 = org.json.JSONArray() + coord2.put(x2) + coord2.put(y2) + obj.put("coordinate2", coord2) + } + text?.let { obj.put("text", it) } + button?.let { obj.put("button", it) } + duration?.let { obj.put("duration", it) } + message?.let { obj.put("message", it) } + if (needConfirm) obj.put("need_confirm", true) + + return obj.toString() + } + + override fun toString(): String = toJson() +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Manager.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Manager.kt new file mode 100644 index 0000000..f58006b --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Manager.kt @@ -0,0 +1,178 @@ +package com.roubao.autopilot.agent + +/** + * Manager Agent - 负责规划和进度管理 + */ +class Manager { + + /** + * 生成规划 Prompt + */ + fun getPrompt(infoPool: InfoPool): String { + return if (infoPool.plan.isEmpty()) { + getInitialPlanPrompt(infoPool) + } else { + getUpdatePlanPrompt(infoPool) + } + } + + private fun getInitialPlanPrompt(infoPool: InfoPool): String = buildString { + append("You are an agent who can operate an Android phone on behalf of a user. ") + append("Your goal is to track progress and devise high-level plans to achieve the user's requests.\n\n") + + append("### User Request ###\n") + append("${infoPool.instruction}\n\n") + + append("---\n") + append("Make a high-level plan to achieve the user's request. ") + append("If the request is complex, break it down into subgoals. ") + append("The screenshot displays the starting state of the phone.\n\n") + + append("IMPORTANT: For requests that explicitly require an answer, ") + append("always add 'perform the `answer` action' as the last step to the plan!\n\n") + + // Skill 上下文 + if (infoPool.skillContext.isNotEmpty()) { + append("### Available Skills ###\n") + append("${infoPool.skillContext}\n\n") + } + + append("### Guidelines ###\n") + append("1. IMPORTANT: If you see the \"肉包\" or \"Baozi\" app interface (this automation tool), press Home button first to go back to the home screen, then proceed with the task.\n") + append("2. ALWAYS use `open_app` action to open apps - NEVER go to home screen to look for app icons! The open_app action can launch any installed app directly.\n") + append("3. Use search to quickly find a file or entry with a specific name.\n") + append("4. If there are relevant skills listed above, follow their suggested steps for better efficiency.\n") + if (infoPool.additionalKnowledge.isNotEmpty()) { + append("5. ${infoPool.additionalKnowledge}\n") + } + append("\n") + + append("Provide your output in the following format:\n\n") + append("### Thought ###\n") + append("A detailed explanation of your rationale for the plan.\n\n") + append("### Plan ###\n") + append("1. first subgoal\n") + append("2. second subgoal\n") + append("...\n") + } + + private fun getUpdatePlanPrompt(infoPool: InfoPool): String = buildString { + append("You are an agent who can operate an Android phone on behalf of a user. ") + append("Your goal is to track progress and update plans.\n\n") + + append("### User Request ###\n") + append("${infoPool.instruction}\n\n") + + if (infoPool.completedPlan.isNotEmpty() && infoPool.completedPlan != "No completed subgoal.") { + append("### Historical Operations ###\n") + append("${infoPool.completedPlan}\n\n") + } + + append("### Current Plan ###\n") + append("${infoPool.plan}\n\n") + + append("### Last Action ###\n") + append("${infoPool.lastAction}\n\n") + + append("### Last Action Description ###\n") + append("${infoPool.lastSummary}\n\n") + + // 最近的动作结果 + if (infoPool.actionOutcomes.isNotEmpty()) { + val recentOutcomes = infoPool.actionOutcomes.takeLast(3) + val failCount = recentOutcomes.count { it in listOf("B", "C") } + if (failCount > 0) { + append("### Recent Action Results ###\n") + append("Last ${recentOutcomes.size} actions: ${recentOutcomes.joinToString(", ")} ") + append("(A=success, B=partial, C=failed)\n") + append("Failed attempts: $failCount\n\n") + } + } + + if (infoPool.importantNotes.isNotEmpty()) { + append("### Important Notes ###\n") + append("${infoPool.importantNotes}\n\n") + } + + // 错误升级 + if (infoPool.errorFlagPlan) { + append("### ⚠️ STUCK - Multiple Failed Attempts! ###\n") + append("You have encountered several consecutive failed attempts:\n") + val k = infoPool.errToManagerThresh + val recentActions = infoPool.actionHistory.takeLast(k) + val recentSummaries = infoPool.summaryHistory.takeLast(k) + val recentErrors = infoPool.errorDescriptions.takeLast(k) + + recentActions.forEachIndexed { i, act -> + append("- Action: $act | Description: ${recentSummaries.getOrNull(i)} | Failed: ${recentErrors.getOrNull(i)}\n") + } + append("\nIMPORTANT: DO NOT mark as \"Finished\" when there are failed attempts! ") + append("Try a different approach:\n") + append("- Wait for page to load (the UI might be loading)\n") + append("- Try clicking at different coordinates\n") + append("- Scroll to find the correct button\n") + append("- Press Back and try again\n\n") + } + + append("---\n") + append("Assess the current status.\n\n") + + append("### ⛔ SECURITY: Sensitive Pages - MUST STOP ###\n") + append("ONLY output \"STOP_SENSITIVE\" when the screen is ACTIVELY REQUESTING one of these:\n") + append("- A payment confirmation button that will charge money (确认支付, 立即付款, 确认付款)\n") + append("- A password input field that is focused and waiting for input\n") + append("- Face ID or fingerprint verification dialog\n") + append("DO NOT stop for: price displays, payment method selection, cart pages, or general app navigation.\n\n") + + append("### CRITICAL: When to mark \"Finished\" ###\n") + append("ONLY mark as \"Finished\" when ALL of these are true:\n") + append("1. The user's request has been FULLY completed (not partially)\n") + append("2. You can SEE the final success state in the screenshot (e.g., order confirmed, message sent, setting changed)\n") + append("3. The last action was SUCCESSFUL (outcome A), not failed (outcome C)\n") + append("4. There are NO recent consecutive failures\n\n") + append("If any action failed or the task is incomplete, DO NOT say \"Finished\". Instead, update the plan with a new approach.\n\n") + + append("Provide your output in the following format:\n\n") + append("### Thought ###\n") + append("Your rationale for the updated plan.\n\n") + append("### Historical Operations ###\n") + append("Add newly completed subgoals on top of existing ones.\n\n") + append("### Plan ###\n") + append("Updated plan or \"Finished\" if truly done.\n") + } + + /** + * 解析规划响应 + */ + fun parseResponse(response: String): PlanResult { + val thought = response + .substringAfter("### Thought", "") + .substringBefore("### Historical Operations") + .substringBefore("### Plan") + .replace("###", "") + .trim() + + val completedSubgoal = if (response.contains("### Historical Operations")) { + response + .substringAfter("### Historical Operations") + .substringBefore("### Plan") + .replace("###", "") + .trim() + } else { + "No completed subgoal." + } + + val plan = response + .substringAfter("### Plan") + .replace("###", "") + .trim() + + return PlanResult(thought, completedSubgoal, plan) + } +} + +data class PlanResult( + val thought: String, + val completedSubgoal: String, + val plan: String +) diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/agent/MobileAgent.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/MobileAgent.kt new file mode 100644 index 0000000..75bc395 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/MobileAgent.kt @@ -0,0 +1,1167 @@ +package com.roubao.autopilot.agent + +import android.content.Context +import android.content.Intent +import android.graphics.Bitmap +import com.roubao.autopilot.App +import com.roubao.autopilot.controller.AppScanner +import com.roubao.autopilot.controller.DeviceController +import com.roubao.autopilot.data.ExecutionStep +import com.roubao.autopilot.skills.SkillManager +import com.roubao.autopilot.ui.OverlayService +import com.roubao.autopilot.vlm.GUIOwlClient +import com.roubao.autopilot.vlm.MAIUIClient +import com.roubao.autopilot.vlm.VLMClient +import kotlinx.coroutines.CancellationException +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.delay +import kotlinx.coroutines.ensureActive +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.suspendCancellableCoroutine +import kotlinx.coroutines.withContext +import kotlin.coroutines.coroutineContext +import kotlin.coroutines.resume + +/** + * Mobile Agent 主循环 - 移植自 MobileAgent-v3 + * + * 新增 Skill 层支持： + * - 快速路径：高置信度 delegation Skill 直接执行 + * - 增强模式：GUI 自动化 Skill 提供上下文指导 + * + * 支持三种模式： + * - OpenAI 兼容模式：使用 VLMClient (Manager → Executor → Reflector) + * - GUI-Owl 模式：使用 GUIOwlClient (直接返回操作指令) + * - MAI-UI 模式：使用 MAIUIClient (专用 prompt 和对话历史) + */ +class MobileAgent( + private val vlmClient: VLMClient?, + private val controller: DeviceController, + private val context: Context, + private val guiOwlClient: GUIOwlClient? = null, // GUI-Owl 专用客户端 + private val maiuiClient: MAIUIClient? = null // MAI-UI 专用客户端 +) { + // 是否使用 GUI-Owl 模式 + private val useGUIOwlMode: Boolean = guiOwlClient != null + // 是否使用 MAI-UI 模式 + private val useMAIUIMode: Boolean = maiuiClient != null + // App 扫描器 (使用 App 单例中的实例) + private val appScanner: AppScanner = App.getInstance().appScanner + private val manager = Manager() + private val executor = Executor() + private val reflector = ActionReflector() + private val notetaker = Notetaker() + + // Skill 管理器 + private val skillManager: SkillManager? = try { + SkillManager.getInstance().also { + println("[肉包] SkillManager 已加载，共 ${it.getAllSkills().size} 个 Skills") + // 设置 VLM 客户端用于意图匹配（仅在 OpenAI 兼容模式下） + vlmClient?.let { client -> it.setVLMClient(client) } + } + } catch (e: Exception) { + println("[肉包] SkillManager 加载失败: ${e.message}") + null + } + + // 状态流 + private val _state = MutableStateFlow(AgentState()) + val state: StateFlow = _state + + private val _logs = MutableStateFlow>(emptyList()) + val logs: StateFlow> = _logs + + /** + * 执行指令 + */ + suspend fun runInstruction( + instruction: String, + maxSteps: Int = 25, + useNotetaker: Boolean = false + ): AgentResult { + log("开始执行: $instruction") + + // 根据模式选择执行路径 + if (useGUIOwlMode && guiOwlClient != null) { + log("使用 GUI-Owl 模式") + return runInstructionWithGUIOwl(instruction, maxSteps) + } + + if (useMAIUIMode && maiuiClient != null) { + log("使用 MAI-UI 模式") + return runInstructionWithMAIUI(instruction, maxSteps) + } + + // OpenAI 兼容模式需要 VLMClient + if (vlmClient == null) { + log("错误: VLMClient 未初始化") + return AgentResult(success = false, message = "VLMClient 未初始化") + } + + log("使用 OpenAI 兼容模式") + + // 使用 LLM 匹配 Skill，生成上下文信息给 Agent（不执行任何操作） + log("正在分析意图...") + val skillContext = skillManager?.generateAgentContextWithLLM(instruction) + + val infoPool = InfoPool(instruction = instruction) + + // 初始化 Executor 的对话记忆 + val executorSystemPrompt = buildString { + append("You are an agent who can operate an Android phone. ") + append("Decide the next action based on the current state.\n\n") + append("User Request: $instruction\n") + } + infoPool.executorMemory = ConversationMemory.withSystemPrompt(executorSystemPrompt) + log("已初始化对话记忆") + + // 如果有 Skill 上下文，添加到 InfoPool，让 Manager 知道可用的工具 + if (!skillContext.isNullOrEmpty() && skillContext != "未找到相关技能或可用应用，请使用通用 GUI 自动化完成任务。") { + infoPool.skillContext = skillContext + log("已匹配到可用技能:\n$skillContext") + } else { + log("未匹配到特定技能，使用通用 GUI 自动化") + } + + // 获取屏幕尺寸 + val (width, height) = controller.getScreenSize() + infoPool.screenWidth = width + infoPool.screenHeight = height + log("屏幕尺寸: ${width}x${height}") + + // 获取已安装应用列表（只取非系统应用，限制数量避免 prompt 过长） + val apps = appScanner.getApps() + .filter { !it.isSystem } + .take(50) + .map { it.appName } + infoPool.installedApps = apps.joinToString(", ") + log("已加载 ${apps.size} 个应用") + + // 显示悬浮窗 (带停止按钮) + OverlayService.show(context, "开始执行...") { + // 停止回调 - 设置状态为停止 + // 注意：协程取消需要在 MainActivity 中处理 + updateState { copy(isRunning = false) } + // 调用 stop() 方法确保清理 + stop() + } + + updateState { copy(isRunning = true, currentStep = 0, instruction = instruction) } + + try { + for (step in 0 until maxSteps) { + // 检查协程是否被取消 + coroutineContext.ensureActive() + + // 检查是否被用户停止 + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + updateState { copy(currentStep = step + 1) } + log("\n========== Step ${step + 1} ==========") + OverlayService.update("Step ${step + 1}/$maxSteps") + + // 1. 截图 (先隐藏悬浮窗避免被识别) + log("截图中...") + OverlayService.setVisible(false) + delay(100) // 等待悬浮窗隐藏 + val screenshotResult = controller.screenshotWithFallback() + OverlayService.setVisible(true) + val screenshot = screenshotResult.bitmap + + // 处理敏感页面（截图被系统阻止） + if (screenshotResult.isSensitive) { + log("⚠️ 检测到敏感页面（截图被阻止），请求人工接管") + val confirmed = withContext(Dispatchers.Main) { + waitForUserConfirm("检测到敏感页面，是否继续执行？") + } + if (!confirmed) { + log("用户取消，任务终止") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "敏感页面，用户取消") + } + log("用户确认继续（使用黑屏占位图）") + } else if (screenshotResult.isFallback) { + log("⚠️ 截图失败，使用黑屏占位图继续") + } + + // 再次检查停止状态（截图后） + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + // 2. 检查错误升级 + checkErrorEscalation(infoPool) + + // 3. 跳过 Manager 的情况 + val skipManager = !infoPool.errorFlagPlan && + infoPool.actionHistory.isNotEmpty() && + infoPool.actionHistory.last().type == "invalid" + + // 4. Manager 规划 + if (!skipManager) { + log("Manager 规划中...") + + // 检查停止状态 + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + val planPrompt = manager.getPrompt(infoPool) + val planResponse = vlmClient.predict(planPrompt, listOf(screenshot)) + + // VLM 调用后检查停止状态 + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + if (planResponse.isFailure) { + log("Manager 调用失败: ${planResponse.exceptionOrNull()?.message}") + continue + } + + val planResult = manager.parseResponse(planResponse.getOrThrow()) + infoPool.completedPlan = planResult.completedSubgoal + infoPool.plan = planResult.plan + + log("计划: ${planResult.plan.take(100)}...") + + // 检查是否遇到敏感页面 + if (planResult.plan.contains("STOP_SENSITIVE")) { + log("检测到敏感页面（支付/密码等），已停止执行") + OverlayService.update("敏感页面，已停止") + delay(2000) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = false) } + bringAppToFront() + return AgentResult(success = false, message = "检测到敏感页面（支付/密码），已安全停止") + } + + // 检查是否完成 + if (planResult.plan.contains("Finished") && planResult.plan.length < 20) { + log("任务完成!") + OverlayService.update("完成!") + delay(1500) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = true) } + bringAppToFront() + return AgentResult(success = true, message = "任务完成") + } + } + + // 5. Executor 决定动作 (使用上下文记忆) + log("Executor 决策中...") + + // 检查停止状态 + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + val actionPrompt = executor.getPrompt(infoPool) + + // 使用上下文记忆调用 VLM + val memory = infoPool.executorMemory + val actionResponse = if (memory != null) { + // 添加用户消息（带截图） + memory.addUserMessage(actionPrompt, screenshot) + log("记忆消息数: ${memory.size()}, 估算 token: ${memory.estimateTokens()}") + + // 调用 VLM + val response = vlmClient.predictWithContext(memory.toMessagesJson()) + + // 删除图片节省 token + memory.stripLastUserImage() + + response + } else { + // 降级：使用普通方式 + vlmClient.predict(actionPrompt, listOf(screenshot)) + } + + // VLM 调用后检查停止状态 + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + if (actionResponse.isFailure) { + log("Executor 调用失败: ${actionResponse.exceptionOrNull()?.message}") + continue + } + + val responseText = actionResponse.getOrThrow() + val executorResult = executor.parseResponse(responseText) + + // 将助手响应添加到记忆 + memory?.addAssistantMessage(responseText) + val action = executorResult.action + + log("思考: ${executorResult.thought.take(80)}...") + log("动作: ${executorResult.actionStr}") + log("描述: ${executorResult.description}") + + infoPool.lastActionThought = executorResult.thought + infoPool.lastSummary = executorResult.description + + if (action == null) { + log("动作解析失败") + infoPool.actionHistory.add(Action(type = "invalid")) + infoPool.summaryHistory.add(executorResult.description) + infoPool.actionOutcomes.add("C") + infoPool.errorDescriptions.add("Invalid action format") + continue + } + + // 特殊处理: answer 动作 + if (action.type == "answer") { + log("回答: ${action.text}") + OverlayService.update("${action.text?.take(20)}...") + delay(1500) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = true, answer = action.text) } + bringAppToFront() + return AgentResult(success = true, message = "回答: ${action.text}") + } + + // 特殊处理: terminate 动作 (MAI-UI) + if (action.type == "terminate") { + val success = action.status == "success" + log("任务${if (success) "完成" else "失败"}") + OverlayService.update(if (success) "完成!" else "失败") + delay(1500) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = success) } + bringAppToFront() + return AgentResult(success = success, message = if (success) "任务完成" else "任务失败") + } + + // 6. 敏感操作确认 + if (action.needConfirm || action.message != null && action.type in listOf("click", "double_tap", "long_press")) { + val confirmMessage = action.message ?: "确认执行此操作？" + log("⚠️ 敏感操作: $confirmMessage") + + val confirmed = withContext(Dispatchers.Main) { + waitForUserConfirm(confirmMessage) + } + + if (!confirmed) { + log("❌ 用户取消操作") + infoPool.actionHistory.add(action) + infoPool.summaryHistory.add("用户取消: ${executorResult.description}") + infoPool.actionOutcomes.add("C") + infoPool.errorDescriptions.add("User cancelled") + continue + } + log("✅ 用户确认，继续执行") + } + + // 7. 执行动作 + log("执行动作: ${action.type}") + OverlayService.update("${action.type}: ${executorResult.description.take(15)}...") + executeAction(action, infoPool) + infoPool.lastAction = action + + // 立即记录执行步骤（outcome 暂时为 "?" 表示进行中） + val currentStepIndex = _state.value.executionSteps.size + val executionStep = ExecutionStep( + stepNumber = step + 1, + timestamp = System.currentTimeMillis(), + action = action.type, + description = executorResult.description, + thought = executorResult.thought, + outcome = "?" // 进行中 + ) + updateState { copy(executionSteps = executionSteps + executionStep) } + + // 等待动作生效 + delay(if (step == 0) 5000 else 2000) + + // 检查停止状态 + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + // 8. 截图 (动作后，隐藏悬浮窗) + OverlayService.setVisible(false) + delay(100) + val afterScreenshotResult = controller.screenshotWithFallback() + OverlayService.setVisible(true) + val afterScreenshot = afterScreenshotResult.bitmap + if (afterScreenshotResult.isFallback) { + log("动作后截图失败，使用黑屏占位图") + } + + // 9. Reflector 反思 + log("Reflector 反思中...") + + // 检查停止状态 + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + val reflectPrompt = reflector.getPrompt(infoPool) + val reflectResponse = vlmClient.predict(reflectPrompt, listOf(screenshot, afterScreenshot)) + + val reflectResult = if (reflectResponse.isSuccess) { + reflector.parseResponse(reflectResponse.getOrThrow()) + } else { + ReflectorResult("C", "Failed to call reflector") + } + + log("结果: ${reflectResult.outcome} - ${reflectResult.errorDescription.take(50)}") + + // 更新历史 + infoPool.actionHistory.add(action) + infoPool.summaryHistory.add(executorResult.description) + infoPool.actionOutcomes.add(reflectResult.outcome) + infoPool.errorDescriptions.add(reflectResult.errorDescription) + infoPool.progressStatus = infoPool.completedPlan + + // 更新执行步骤的 outcome（之前添加的步骤 outcome 是 "?"） + updateState { + val updatedSteps = executionSteps.toMutableList() + if (currentStepIndex < updatedSteps.size) { + updatedSteps[currentStepIndex] = updatedSteps[currentStepIndex].copy( + outcome = reflectResult.outcome + ) + } + copy(executionSteps = updatedSteps) + } + + // 10. Notetaker (可选) + if (useNotetaker && reflectResult.outcome == "A" && action.type != "answer") { + log("Notetaker 记录中...") + + // 检查停止状态 + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + val notePrompt = notetaker.getPrompt(infoPool) + val noteResponse = vlmClient.predict(notePrompt, listOf(afterScreenshot)) + if (noteResponse.isSuccess) { + infoPool.importantNotes = notetaker.parseResponse(noteResponse.getOrThrow()) + } + } + } + } catch (e: CancellationException) { + log("任务被取消") + OverlayService.hide(context) + updateState { copy(isRunning = false) } + bringAppToFront() + throw e + } + + log("达到最大步数限制") + OverlayService.update("达到最大步数") + delay(1500) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = false) } + bringAppToFront() + return AgentResult(success = false, message = "达到最大步数限制") + } + + /** + * GUI-Owl 模式执行指令 + * 简化流程：截图 → GUI-Owl API → 解析操作 → 执行 + */ + private suspend fun runInstructionWithGUIOwl( + instruction: String, + maxSteps: Int + ): AgentResult { + val client = guiOwlClient ?: return AgentResult(false, "GUIOwlClient 未初始化") + + // 重置会话 + client.resetSession() + log("GUI-Owl 会话已重置") + + // 获取屏幕尺寸 + val (screenWidth, screenHeight) = controller.getScreenSize() + log("屏幕尺寸: ${screenWidth}x${screenHeight}") + + // 显示悬浮窗 + OverlayService.show(context, "GUI-Owl 模式") { + updateState { copy(isRunning = false) } + stop() + } + + updateState { copy(isRunning = true, currentStep = 0, instruction = instruction) } + + try { + for (step in 0 until maxSteps) { + coroutineContext.ensureActive() + + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + updateState { copy(currentStep = step + 1) } + log("\n========== Step ${step + 1} (GUI-Owl) ==========") + OverlayService.update("Step ${step + 1}/$maxSteps") + + // 1. 截图 + log("截图中...") + OverlayService.setVisible(false) + delay(100) + val screenshotResult = controller.screenshotWithFallback() + OverlayService.setVisible(true) + val screenshot = screenshotResult.bitmap + + if (screenshotResult.isSensitive) { + log("⚠️ 检测到敏感页面") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "敏感页面，已停止") + } + + // 2. 调用 GUI-Owl API + log("调用 GUI-Owl API...") + val response = client.predict(instruction, screenshot) + + if (response.isFailure) { + log("GUI-Owl 调用失败: ${response.exceptionOrNull()?.message}") + continue + } + + val result = response.getOrThrow() + log("思考: ${result.thought.take(100)}...") + log("操作: ${result.operation}") + log("说明: ${result.explanation}") + + // 3. 解析操作指令 + val parsedAction = client.parseOperation(result.operation) + if (parsedAction == null) { + log("无法解析操作: ${result.operation}") + continue + } + + // 记录执行步骤 + val executionStep = ExecutionStep( + stepNumber = step + 1, + timestamp = System.currentTimeMillis(), + action = parsedAction.type, + description = result.explanation, + thought = result.thought, + outcome = "?" + ) + updateState { copy(executionSteps = executionSteps + executionStep) } + + // 检查是否完成 + if (parsedAction.type == "finish") { + log("任务完成!") + OverlayService.update("完成!") + delay(1500) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = true) } + bringAppToFront() + return AgentResult(success = true, message = "任务完成") + } + + // 4. 执行动作 + log("执行动作: ${parsedAction.type}") + OverlayService.update("${parsedAction.type}: ${result.explanation.take(15)}...") + executeGUIOwlAction(parsedAction, screenWidth, screenHeight) + + // 更新步骤状态 + updateState { + val updatedSteps = executionSteps.toMutableList() + if (step < updatedSteps.size) { + updatedSteps[step] = updatedSteps[step].copy(outcome = "A") + } + copy(executionSteps = updatedSteps) + } + + // 等待动作生效 + delay(if (step == 0) 3000 else 1500) + } + } catch (e: CancellationException) { + log("任务被取消") + OverlayService.hide(context) + updateState { copy(isRunning = false) } + bringAppToFront() + throw e + } + + log("达到最大步数限制") + OverlayService.update("达到最大步数") + delay(1500) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = false) } + bringAppToFront() + return AgentResult(success = false, message = "达到最大步数限制") + } + + /** + * 执行 GUI-Owl 解析的动作 + */ + private suspend fun executeGUIOwlAction( + action: GUIOwlClient.ParsedAction, + screenWidth: Int, + screenHeight: Int + ) = withContext(Dispatchers.IO) { + when (action.type) { + "click" -> { + val x = action.x ?: return@withContext + val y = action.y ?: return@withContext + log("点击: ($x, $y)") + controller.tap(x, y) + } + "swipe" -> { + val x1 = action.x ?: return@withContext + val y1 = action.y ?: return@withContext + val x2 = action.x2 ?: return@withContext + val y2 = action.y2 ?: return@withContext + log("滑动: ($x1, $y1) -> ($x2, $y2)") + controller.swipe(x1, y1, x2, y2) + } + "long_press" -> { + val x = action.x ?: return@withContext + val y = action.y ?: return@withContext + log("长按: ($x, $y)") + controller.longPress(x, y) + } + "type" -> { + val text = action.text ?: return@withContext + log("输入: $text") + controller.type(text) + } + "scroll" -> { + val direction = action.text ?: "down" + val centerX = screenWidth / 2 + val centerY = screenHeight / 2 + log("滚动: $direction") + when (direction) { + "up" -> controller.swipe(centerX, centerY + 300, centerX, centerY - 300) + "down" -> controller.swipe(centerX, centerY - 300, centerX, centerY + 300) + "left" -> controller.swipe(centerX + 300, centerY, centerX - 300, centerY) + "right" -> controller.swipe(centerX - 300, centerY, centerX + 300, centerY) + } + } + "system_button" -> { + when (action.text?.lowercase()) { + "back" -> { + log("按返回键") + controller.back() + } + "home" -> { + log("按 Home 键") + controller.home() + } + } + } + else -> { + log("未知动作类型: ${action.type}") + } + } + } + + /** + * MAI-UI 模式执行指令 + * 使用专用的 MAI-UI prompt 和对话历史管理 + */ + private suspend fun runInstructionWithMAIUI( + instruction: String, + maxSteps: Int + ): AgentResult { + val client = maiuiClient ?: return AgentResult(false, "MAIUIClient 未初始化") + + // 重置会话 + client.reset() + log("MAI-UI 会话已重置") + + // 获取屏幕尺寸 + val (screenWidth, screenHeight) = controller.getScreenSize() + log("屏幕尺寸: ${screenWidth}x${screenHeight}") + + // 设置可用应用列表 + val installedApps = appScanner.getApps().map { it.appName } + client.setAvailableApps(installedApps) + log("已加载 ${installedApps.size} 个应用") + + // 显示悬浮窗 + OverlayService.show(context, "MAI-UI 模式") { + updateState { copy(isRunning = false) } + stop() + } + + updateState { copy(isRunning = true, currentStep = 0, instruction = instruction) } + + try { + for (step in 0 until maxSteps) { + coroutineContext.ensureActive() + + if (!_state.value.isRunning) { + log("用户停止执行") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "用户停止") + } + + updateState { copy(currentStep = step + 1) } + log("\n========== Step ${step + 1} (MAI-UI) ==========") + OverlayService.update("Step ${step + 1}/$maxSteps") + + // 1. 截图 + log("截图中...") + OverlayService.setVisible(false) + delay(100) + val screenshotResult = controller.screenshotWithFallback() + OverlayService.setVisible(true) + val screenshot = screenshotResult.bitmap + + if (screenshotResult.isSensitive) { + log("⚠️ 检测到敏感页面") + OverlayService.hide(context) + bringAppToFront() + return AgentResult(success = false, message = "敏感页面，已停止") + } + + // 2. 调用 MAI-UI API + log("调用 MAI-UI API...") + val response = client.predict(instruction, screenshot) + + if (response.isFailure) { + log("MAI-UI 调用失败: ${response.exceptionOrNull()?.message}") + continue + } + + val result = response.getOrThrow() + log("思考: ${result.thinking.take(150)}...") + + val action = result.action + if (action == null) { + log("无法解析动作") + continue + } + + log("动作: ${action.type}") + + // 记录执行步骤 + val executionStep = ExecutionStep( + stepNumber = step + 1, + timestamp = System.currentTimeMillis(), + action = action.type, + description = result.thinking.take(50), + thought = result.thinking, + outcome = "?" + ) + updateState { copy(executionSteps = executionSteps + executionStep) } + + // 检查是否完成 + if (action.type == "terminate") { + val success = action.status == "success" + log(if (success) "任务完成!" else "任务失败") + OverlayService.update(if (success) "完成!" else "失败") + delay(1500) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = success) } + bringAppToFront() + return AgentResult(success = success, message = if (success) "任务完成" else "任务失败") + } + + // 检查是否需要人工接管 + if (action.type == "ask_user") { + log("请求用户介入: ${action.text}") + OverlayService.update("请手动操作: ${action.text?.take(20)}") + // 等待用户操作后继续 + delay(5000) + continue + } + + // 检查是否是回答 + if (action.type == "answer") { + log("回答: ${action.text}") + OverlayService.update("答案: ${action.text?.take(30)}") + delay(3000) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = true) } + bringAppToFront() + return AgentResult(success = true, message = "回答: ${action.text}") + } + + // 3. 执行动作 + log("执行动作: ${action.type}") + OverlayService.update("${action.type}...") + executeMAIUIAction(action, screenWidth, screenHeight) + + // 更新步骤状态 + updateState { + val updatedSteps = executionSteps.toMutableList() + if (step < updatedSteps.size) { + updatedSteps[step] = updatedSteps[step].copy(outcome = "A") + } + copy(executionSteps = updatedSteps) + } + + // 等待动作生效 + delay(if (step == 0) 2000 else 1000) + } + } catch (e: CancellationException) { + log("任务被取消") + OverlayService.hide(context) + updateState { copy(isRunning = false) } + bringAppToFront() + throw e + } + + log("达到最大步数限制") + OverlayService.update("达到最大步数") + delay(1500) + OverlayService.hide(context) + updateState { copy(isRunning = false, isCompleted = false) } + bringAppToFront() + return AgentResult(success = false, message = "达到最大步数限制") + } + + /** + * 执行 MAI-UI 解析的动作 + */ + private suspend fun executeMAIUIAction( + action: com.roubao.autopilot.vlm.MAIUIAction, + screenWidth: Int, + screenHeight: Int + ) = withContext(Dispatchers.IO) { + // 转换归一化坐标到屏幕像素 + val screenAction = action.toScreenCoordinates(screenWidth, screenHeight) + + when (action.type) { + "click" -> { + val x = screenAction.x?.toInt() ?: return@withContext + val y = screenAction.y?.toInt() ?: return@withContext + log("点击: ($x, $y)") + controller.tap(x, y) + } + "long_press" -> { + val x = screenAction.x?.toInt() ?: return@withContext + val y = screenAction.y?.toInt() ?: return@withContext + log("长按: ($x, $y)") + controller.longPress(x, y) + } + "double_click" -> { + val x = screenAction.x?.toInt() ?: return@withContext + val y = screenAction.y?.toInt() ?: return@withContext + log("双击: ($x, $y)") + controller.tap(x, y) + delay(100) + controller.tap(x, y) + } + "type" -> { + val text = action.text ?: return@withContext + log("输入: $text") + controller.type(text) + } + "swipe" -> { + // 支持方向式滑动和坐标式滑动 + if (action.direction != null) { + val centerX = screenWidth / 2 + val centerY = screenHeight / 2 + val distance = screenHeight / 3 + log("滑动: ${action.direction}") + when (action.direction.lowercase()) { + "up" -> controller.swipe(centerX, centerY + distance, centerX, centerY - distance) + "down" -> controller.swipe(centerX, centerY - distance, centerX, centerY + distance) + "left" -> controller.swipe(centerX + distance, centerY, centerX - distance, centerY) + "right" -> controller.swipe(centerX - distance, centerY, centerX + distance, centerY) + } + } else if (screenAction.x != null && screenAction.y != null) { + // 从指定位置滑动 + val startX = screenAction.x.toInt() + val startY = screenAction.y.toInt() + val distance = screenHeight / 3 + val direction = action.direction ?: "up" + when (direction.lowercase()) { + "up" -> controller.swipe(startX, startY, startX, startY - distance) + "down" -> controller.swipe(startX, startY, startX, startY + distance) + "left" -> controller.swipe(startX, startY, startX - distance, startY) + "right" -> controller.swipe(startX, startY, startX + distance, startY) + } + } + } + "drag" -> { + val x1 = screenAction.startX?.toInt() ?: return@withContext + val y1 = screenAction.startY?.toInt() ?: return@withContext + val x2 = screenAction.endX?.toInt() ?: return@withContext + val y2 = screenAction.endY?.toInt() ?: return@withContext + log("拖拽: ($x1, $y1) -> ($x2, $y2)") + controller.swipe(x1, y1, x2, y2, 500) + } + "open" -> { + val appName = action.text ?: return@withContext + log("打开应用: $appName") + controller.openApp(appName) + } + "system_button" -> { + when (action.button?.lowercase()) { + "back" -> { + log("按返回键") + controller.back() + } + "home" -> { + log("按 Home 键") + controller.home() + } + "menu" -> { + log("按菜单键") + // 通过 keyevent 模拟菜单键 + controller.back() // 大多数场景用返回代替 + } + "enter" -> { + log("按回车键") + controller.enter() + } + } + } + "wait" -> { + log("等待...") + delay(2000) + } + else -> { + log("未知动作类型: ${action.type}") + } + } + } + + /** + * 执行具体动作 (在 IO 线程执行，避免 ANR) + */ + private suspend fun executeAction(action: Action, infoPool: InfoPool) = withContext(Dispatchers.IO) { + // 动态获取屏幕尺寸（处理横竖屏切换） + val (screenWidth, screenHeight) = controller.getScreenSize() + + when (action.type) { + "click" -> { + val x = mapCoordinate(action.x ?: 0, screenWidth) + val y = mapCoordinate(action.y ?: 0, screenHeight) + controller.tap(x, y) + } + "double_tap" -> { + val x = mapCoordinate(action.x ?: 0, screenWidth) + val y = mapCoordinate(action.y ?: 0, screenHeight) + controller.doubleTap(x, y) + } + "long_press" -> { + val x = mapCoordinate(action.x ?: 0, screenWidth) + val y = mapCoordinate(action.y ?: 0, screenHeight) + controller.longPress(x, y) + } + "swipe" -> { + // 支持两种 swipe 方式: + // 1. 坐标方式: coordinate + coordinate2 + // 2. 方向方式: direction (up/down/left/right) + optional coordinate + if (action.direction != null) { + // 方向方式 (MAI-UI 格式) + val centerX = action.x?.let { mapCoordinate(it, screenWidth) } ?: (screenWidth / 2) + val centerY = action.y?.let { mapCoordinate(it, screenHeight) } ?: (screenHeight / 2) + val distance = 400 // 滑动距离 + when (action.direction.lowercase()) { + "up" -> controller.swipe(centerX, centerY + distance, centerX, centerY - distance) + "down" -> controller.swipe(centerX, centerY - distance, centerX, centerY + distance) + "left" -> controller.swipe(centerX + distance, centerY, centerX - distance, centerY) + "right" -> controller.swipe(centerX - distance, centerY, centerX + distance, centerY) + else -> log("未知滑动方向: ${action.direction}") + } + } else { + // 坐标方式 + val x1 = mapCoordinate(action.x ?: 0, screenWidth) + val y1 = mapCoordinate(action.y ?: 0, screenHeight) + val x2 = mapCoordinate(action.x2 ?: 0, screenWidth) + val y2 = mapCoordinate(action.y2 ?: 0, screenHeight) + controller.swipe(x1, y1, x2, y2) + } + } + "type" -> { + action.text?.let { controller.type(it) } + } + "system_button" -> { + when (action.button) { + "Back", "back" -> controller.back() + "Home", "home" -> controller.home() + "Enter", "enter" -> controller.enter() + else -> log("未知系统按钮: ${action.button}") + } + } + "open_app" -> { + action.text?.let { appName -> + // 智能匹配包名 (客户端模糊搜索，省 token) + val packageName = appScanner.findPackage(appName) + if (packageName != null) { + log("找到应用: $appName -> $packageName") + controller.openApp(packageName) + } else { + log("未找到应用: $appName，尝试直接打开") + controller.openApp(appName) + } + } + } + "wait" -> { + // 智能等待：模型决定等待时长 + val duration = (action.duration ?: 3).coerceIn(1, 10) + log("等待 ${duration} 秒...") + delay(duration * 1000L) + } + "take_over" -> { + // 人机协作：暂停等待用户手动完成操作 + val message = action.message ?: "请完成操作后点击继续" + log("🖐 人机协作: $message") + withContext(Dispatchers.Main) { + waitForUserTakeOver(message) + } + log("✅ 用户已完成，继续执行") + } + else -> { + log("未知动作类型: ${action.type}") + } + } + } + + /** + * 等待用户完成手动操作（人机协作） + */ + @OptIn(kotlinx.coroutines.ExperimentalCoroutinesApi::class) + private suspend fun waitForUserTakeOver(message: String) = suspendCancellableCoroutine { continuation -> + com.roubao.autopilot.ui.OverlayService.showTakeOver(message) { + if (continuation.isActive) { + continuation.resume(Unit) {} + } + } + } + + /** + * 等待用户确认敏感操作 + * @return true = 用户确认，false = 用户取消 + */ + @OptIn(kotlinx.coroutines.ExperimentalCoroutinesApi::class) + private suspend fun waitForUserConfirm(message: String) = suspendCancellableCoroutine { continuation -> + com.roubao.autopilot.ui.OverlayService.showConfirm(message) { confirmed -> + if (continuation.isActive) { + continuation.resume(confirmed) {} + } + } + } + + /** + * 坐标映射 - 支持相对坐标和绝对坐标 + * + * 坐标格式判断: + * - 0-999: Qwen-VL 相对坐标 (0-999 映射到屏幕) + * - >= 1000: 绝对像素坐标，直接使用 + * + * @param value 模型输出的坐标值 + * @param screenMax 屏幕实际尺寸 + */ + private fun mapCoordinate(value: Int, screenMax: Int): Int { + return if (value < 1000) { + // 相对坐标 (0-999) -> 绝对像素 + (value * screenMax / 999) + } else { + // 绝对坐标，限制在屏幕范围内 + value.coerceAtMost(screenMax) + } + } + + /** + * 检查错误升级 + */ + private fun checkErrorEscalation(infoPool: InfoPool) { + infoPool.errorFlagPlan = false + val thresh = infoPool.errToManagerThresh + + if (infoPool.actionOutcomes.size >= thresh) { + val recentOutcomes = infoPool.actionOutcomes.takeLast(thresh) + val failCount = recentOutcomes.count { it in listOf("B", "C") } + if (failCount == thresh) { + infoPool.errorFlagPlan = true + } + } + } + + // 停止回调（由 MainActivity 设置，用于取消协程） + var onStopRequested: (() -> Unit)? = null + + /** + * 停止执行 + */ + fun stop() { + OverlayService.hide(context) + updateState { copy(isRunning = false) } + // 通知 MainActivity 取消协程 + onStopRequested?.invoke() + } + + /** + * 清空日志 + */ + fun clearLogs() { + _logs.value = emptyList() + updateState { copy(executionSteps = emptyList()) } + } + + /** + * 返回肉包App + */ + private fun bringAppToFront() { + try { + val intent = context.packageManager.getLaunchIntentForPackage(context.packageName) + intent?.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_REORDER_TO_FRONT) + context.startActivity(intent) + } catch (e: Exception) { + log("返回App失败: ${e.message}") + } + } + + private fun log(message: String) { + println("[肉包] $message") + _logs.value = _logs.value + message + } + + private fun updateState(update: AgentState.() -> AgentState) { + _state.value = _state.value.update() + } + +} + +data class AgentState( + val isRunning: Boolean = false, + val isCompleted: Boolean = false, + val currentStep: Int = 0, + val instruction: String = "", + val answer: String? = null, + val executionSteps: List = emptyList() +) + +data class AgentResult( + val success: Boolean, + val message: String +) diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Notetaker.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Notetaker.kt new file mode 100644 index 0000000..b315922 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/agent/Notetaker.kt @@ -0,0 +1,52 @@ +package com.roubao.autopilot.agent + +/** + * Notetaker Agent - 记录重要信息 + */ +class Notetaker { + + /** + * 生成笔记 Prompt + */ + fun getPrompt(infoPool: InfoPool): String = buildString { + append("You are a helpful AI assistant for operating mobile phones. ") + append("Your goal is to take notes of important content relevant to the user's request.\n\n") + + append("### User Request ###\n") + append("${infoPool.instruction}\n\n") + + append("### Progress Status ###\n") + append("${infoPool.progressStatus}\n\n") + + append("### Existing Important Notes ###\n") + if (infoPool.importantNotes.isNotEmpty()) { + append("${infoPool.importantNotes}\n\n") + } else { + append("No important notes recorded.\n\n") + } + + append("---\n") + append("Examine the current screen to identify any important content that needs to be recorded.\n\n") + + append("IMPORTANT:\n") + append("- Do not take notes on low-level actions\n") + append("- Only keep track of significant textual or visual information relevant to the request\n") + append("- Do not repeat user request or progress status\n") + append("- Do not make up content that you are not sure about\n\n") + + append("Provide your output in the following format:\n\n") + append("### Important Notes ###\n") + append("The updated important notes, combining old and new ones. ") + append("If nothing new to record, copy the existing important notes.\n") + } + + /** + * 解析笔记响应 + */ + fun parseResponse(response: String): String { + return response + .substringAfter("### Important Notes", "") + .replace("###", "") + .trim() + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/controller/AppScanner.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/controller/AppScanner.kt new file mode 100644 index 0000000..e4d0dc6 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/controller/AppScanner.kt @@ -0,0 +1,435 @@ +package com.roubao.autopilot.controller + +import android.content.Context +import android.content.pm.ApplicationInfo +import android.content.pm.PackageManager +import org.json.JSONArray +import org.json.JSONObject +import java.io.File + +/** + * App 扫描器 - 获取所有已安装应用信息 + * 支持：预扫描缓存、拼音匹配、分类、语义搜索 + */ +class AppScanner(private val context: Context) { + + companion object { + private const val CACHE_FILE = "installed_apps.json" + + // 内存缓存 (应用生命周期内有效) + @Volatile + private var cachedApps: List? = null + + // 预编译的正则表达式（避免重复创建） + private val PINYIN_CLEAN_REGEX = Regex("[^a-z0-9\\u4e00-\\u9fa5]") + + // 应用分类关键词映射 + private val CATEGORY_KEYWORDS = mapOf( + "社交" to listOf("微信", "QQ", "钉钉", "飞书", "Telegram", "WhatsApp", "Line", "微博", "陌陌", "探探"), + "购物" to listOf("淘宝", "京东", "拼多多", "天猫", "苏宁", "唯品会", "得物", "闲鱼", "当当"), + "外卖" to listOf("美团", "饿了么", "肯德基", "麦当劳", "星巴克", "瑞幸", "小美"), + "出行" to listOf("滴滴", "高德", "百度地图", "腾讯地图", "嘀嗒", "哈啰", "曹操", "T3", "花小猪"), + "地图" to listOf("高德", "百度地图", "腾讯地图", "谷歌地图", "导航"), + "音乐" to listOf("网易云", "QQ音乐", "酷狗", "酷我", "Spotify", "Apple Music", "虾米"), + "视频" to listOf("抖音", "快手", "B站", "bilibili", "优酷", "爱奇艺", "腾讯视频", "芒果TV", "西瓜视频"), + "支付" to listOf("支付宝", "微信", "云闪付", "翼支付"), + "笔记" to listOf("印象笔记", "有道云", "Notion", "备忘录", "便签", "笔记", "记事", "OneNote", "滴答"), + "相机" to listOf("相机", "Camera", "拍照", "美颜", "美图", "轻颜"), + "图片" to listOf("相册", "图库", "Photos", "Gallery", "照片"), + "浏览器" to listOf("Chrome", "Safari", "Firefox", "Edge", "浏览器", "UC", "夸克", "Via"), + "办公" to listOf("WPS", "Office", "钉钉", "飞书", "企业微信", "Slack", "Teams"), + "AI" to listOf("ChatGPT", "Claude", "豆包", "文心", "通义", "讯飞", "Copilot", "即梦", "Midjourney"), + "工具" to listOf("计算器", "手电筒", "指南针", "时钟", "闹钟", "日历", "天气", "文件管理"), + "阅读" to listOf("微信读书", "Kindle", "掌阅", "番茄小说", "起点", "知乎", "今日头条"), + "游戏" to listOf("王者荣耀", "和平精英", "原神", "崩坏", "阴阳师", "游戏") + ) + + // 拼音映射表 (常用应用) + private val PINYIN_MAP = mapOf( + // 社交 + "weixin" to "微信", "wechat" to "微信", "wx" to "微信", + "qq" to "QQ", + "dingding" to "钉钉", "dingtalk" to "钉钉", + "feishu" to "飞书", "lark" to "飞书", + "weibo" to "微博", + + // 购物 + "taobao" to "淘宝", "tb" to "淘宝", + "jingdong" to "京东", "jd" to "京东", + "pinduoduo" to "拼多多", "pdd" to "拼多多", + "xianyu" to "闲鱼", + + // 外卖/餐饮 + "meituan" to "美团", "mt" to "美团", + "eleme" to "饿了么", "elm" to "饿了么", + "xiaomei" to "小美", + "kfc" to "肯德基", "kendeji" to "肯德基", + "maidanglao" to "麦当劳", "mcdonald" to "麦当劳", + "starbucks" to "星巴克", "xingbake" to "星巴克", + "ruixing" to "瑞幸", "luckin" to "瑞幸", + + // 出行/地图 + "didi" to "滴滴", "dd" to "滴滴", + "gaode" to "高德", "amap" to "高德", + "baidu" to "百度", "baidumap" to "百度地图", + "ditu" to "地图", + "daohang" to "导航", + + // 支付 + "zhifubao" to "支付宝", "alipay" to "支付宝", "zfb" to "支付宝", + + // 音乐 + "wangyiyun" to "网易云", "netease" to "网易云", + "qqmusic" to "QQ音乐", "qqyinyue" to "QQ音乐", + "kugou" to "酷狗", + "yinyue" to "音乐", "music" to "音乐", + + // 视频 + "douyin" to "抖音", "tiktok" to "抖音", "dy" to "抖音", + "kuaishou" to "快手", "ks" to "快手", + "bilibili" to "哔哩哔哩", "bzhan" to "哔哩哔哩", "b站" to "哔哩哔哩", + "youku" to "优酷", + "iqiyi" to "爱奇艺", "aiqiyi" to "爱奇艺", + "shipin" to "视频", "video" to "视频", + + // 笔记/办公 + "biji" to "笔记", "note" to "笔记", "notes" to "笔记", + "beiwanglu" to "备忘录", "memo" to "备忘录", + "bianjian" to "便签", + "wps" to "WPS", + + // 系统 + "shezhi" to "设置", "settings" to "设置", + "xiangji" to "相机", "camera" to "相机", "paizhao" to "相机", + "xiangce" to "相册", "photos" to "相册", "gallery" to "相册", "tuku" to "图库", + "dianhua" to "电话", "phone" to "电话", + "duanxin" to "短信", "message" to "短信", "sms" to "短信", + "liulanqi" to "浏览器", "browser" to "浏览器", + "jisuanqi" to "计算器", "calculator" to "计算器", + "shizhong" to "时钟", "clock" to "时钟", + "naozhong" to "闹钟", "alarm" to "闹钟", + "rili" to "日历", "calendar" to "日历", + "tianqi" to "天气", "weather" to "天气", + "wenjian" to "文件", "file" to "文件" + ) + + // 语义查询映射 (用户可能说的自然语言) + private val SEMANTIC_MAP = mapOf( + // 功能描述 -> 分类 + "拍照" to "相机", "照相" to "相机", "自拍" to "相机", "拍摄" to "相机", + "看照片" to "图片", "看图" to "图片", "图片" to "图片", + "聊天" to "社交", "发消息" to "社交", "通讯" to "社交", + "买东西" to "购物", "购物" to "购物", "网购" to "购物", "下单" to "购物", + "点餐" to "外卖", "叫外卖" to "外卖", "点外卖" to "外卖", "吃饭" to "外卖", "订餐" to "外卖", + "打车" to "出行", "叫车" to "出行", "出行" to "出行", "坐车" to "出行", + "导航" to "地图", "找路" to "地图", "去哪" to "地图", "怎么走" to "地图", + "听歌" to "音乐", "听音乐" to "音乐", "放歌" to "音乐", "播放音乐" to "音乐", + "看视频" to "视频", "刷视频" to "视频", "追剧" to "视频", "看电影" to "视频", "看剧" to "视频", + "付款" to "支付", "支付" to "支付", "扫码" to "支付", "收款" to "支付", + "记笔记" to "笔记", "记事" to "笔记", "记录" to "笔记", "写笔记" to "笔记", + "上网" to "浏览器", "搜索" to "浏览器", "查资料" to "浏览器", + "办公" to "办公", "工作" to "办公", "文档" to "办公", + "画图" to "AI", "生成图片" to "AI", "AI画图" to "AI", "AI" to "AI", + "看书" to "阅读", "阅读" to "阅读", "读书" to "阅读", "看小说" to "阅读", + "玩游戏" to "游戏", "游戏" to "游戏" + ) + } + + /** + * 应用信息 + */ + data class AppInfo( + val packageName: String, + val appName: String, + val pinyin: String, // 拼音（自动生成） + val category: String?, // 分类 + val isSystem: Boolean, + val keywords: List // 关键词（用于搜索） + ) + + /** + * 搜索结果 + */ + data class SearchResult( + val app: AppInfo, + val score: Float, // 匹配分数 0-1 + val matchType: String // 匹配类型：exact/contains/pinyin/category/semantic + ) + + /** + * 获取应用列表 (优先内存 -> 文件 -> 扫描) + */ + fun getApps(): List { + cachedApps?.let { return it } + + val cacheFile = File(context.filesDir, CACHE_FILE) + if (cacheFile.exists()) { + val loaded = loadFromFile(cacheFile) + if (loaded.isNotEmpty()) { + cachedApps = loaded + println("[AppScanner] 从文件加载 ${loaded.size} 个应用") + return loaded + } + } + + return refreshApps() + } + + /** + * 强制刷新应用列表 + */ + fun refreshApps(): List { + println("[AppScanner] 扫描已安装应用...") + val apps = scanAllApps() + cachedApps = apps + + val cacheFile = File(context.filesDir, CACHE_FILE) + saveToFile(apps, cacheFile) + println("[AppScanner] 已缓存 ${apps.size} 个应用") + + return apps + } + + /** + * 扫描所有已安装应用 + */ + private fun scanAllApps(): List { + val pm = context.packageManager + val apps = mutableListOf() + + try { + // 使用 0 作为 flag，获取所有应用（不过滤） + val packages = pm.getInstalledApplications(0) + for (appInfo in packages) { + val appName = pm.getApplicationLabel(appInfo).toString() + val isSystem = (appInfo.flags and ApplicationInfo.FLAG_SYSTEM) != 0 + val pinyin = toPinyin(appName) + val category = detectCategory(appName, appInfo.packageName) + val keywords = generateKeywords(appName, appInfo.packageName, category) + + apps.add(AppInfo( + packageName = appInfo.packageName, + appName = appName, + pinyin = pinyin, + category = category, + isSystem = isSystem, + keywords = keywords + )) + } + } catch (e: Exception) { + e.printStackTrace() + } + + return apps.sortedBy { it.appName } + } + + /** + * 智能搜索应用 + * @param query 搜索词（支持：应用名、拼音、分类、语义描述） + * @param topK 返回前 K 个结果 + * @param includeSystem 是否包含系统应用 + */ + fun searchApps(query: String, topK: Int = 5, includeSystem: Boolean = true): List { + val apps = getApps() + val lowerQuery = query.lowercase().trim() + val results = mutableListOf() + + // 先检查是否是语义查询，转换为分类 + val semanticCategory = SEMANTIC_MAP[lowerQuery] + val pinyinMapped = PINYIN_MAP[lowerQuery] + + for (app in apps) { + if (!includeSystem && app.isSystem) continue + + var score = 0f + var matchType = "" + + // 1. 精确匹配应用名 (最高优先级) + if (app.appName.equals(query, ignoreCase = true)) { + score = 1.0f + matchType = "exact" + } + // 2. 拼音映射精确匹配 + else if (pinyinMapped != null && app.appName.contains(pinyinMapped)) { + score = 0.95f + matchType = "pinyin_exact" + } + // 3. 应用名包含查询词 + else if (app.appName.lowercase().contains(lowerQuery)) { + score = 0.9f + matchType = "contains" + } + // 4. 拼音包含 + else if (app.pinyin.contains(lowerQuery)) { + score = 0.8f + matchType = "pinyin" + } + // 5. 关键词匹配 + else if (app.keywords.any { it.contains(lowerQuery) || lowerQuery.contains(it) }) { + score = 0.7f + matchType = "keyword" + } + // 6. 分类匹配（语义查询） + else if (semanticCategory != null && app.category == semanticCategory) { + score = 0.6f + matchType = "semantic" + } + // 7. 包名包含 + else if (app.packageName.lowercase().contains(lowerQuery)) { + score = 0.5f + matchType = "package" + } + + if (score > 0) { + // 非系统应用加分 + if (!app.isSystem) score += 0.05f + results.add(SearchResult(app, score.coerceAtMost(1f), matchType)) + } + } + + return results + .sortedByDescending { it.score } + .take(topK) + } + + /** + * 根据名称模糊搜索包名 (兼容旧接口) + */ + fun findPackage(query: String): String? { + val results = searchApps(query, topK = 1) + return results.firstOrNull()?.app?.packageName + } + + /** + * 按分类获取应用 + */ + fun getAppsByCategory(category: String): List { + return getApps().filter { it.category == category } + } + + /** + * 获取所有分类 + */ + fun getAllCategories(): List { + return getApps().mapNotNull { it.category }.distinct().sorted() + } + + /** + * 格式化搜索结果给 LLM + */ + fun formatSearchResultsForLLM(results: List): String { + if (results.isEmpty()) return "未找到匹配的应用" + + return buildString { + append("找到以下应用，请选择最合适的：\n") + results.forEachIndexed { index, result -> + val app = result.app + val categoryStr = app.category?.let { " [$it]" } ?: "" + append("${index + 1}. ${app.appName}$categoryStr (${app.packageName})\n") + } + } + } + + // ========== 辅助方法 ========== + + /** + * 简单拼音转换（仅处理常见中文字符） + */ + private fun toPinyin(text: String): String { + // 简单实现：保留英文数字，中文用首字母拼音 + // 完整拼音需要引入 pinyin4j 库，这里用简化版本 + return text.lowercase() + .replace(PINYIN_CLEAN_REGEX, "") + } + + /** + * 检测应用分类 + */ + private fun detectCategory(appName: String, packageName: String): String? { + val lowerName = appName.lowercase() + val lowerPackage = packageName.lowercase() + + for ((category, keywords) in CATEGORY_KEYWORDS) { + for (keyword in keywords) { + if (lowerName.contains(keyword.lowercase()) || + lowerPackage.contains(keyword.lowercase())) { + return category + } + } + } + return null + } + + /** + * 生成搜索关键词 + */ + private fun generateKeywords(appName: String, packageName: String, category: String?): List { + val keywords = mutableListOf() + + // 从包名提取关键词 + val packageParts = packageName.split(".") + keywords.addAll(packageParts.filter { it.length > 2 }) + + // 添加分类 + category?.let { keywords.add(it) } + + // 从拼音映射反向添加 + for ((pinyin, name) in PINYIN_MAP) { + if (appName.contains(name)) { + keywords.add(pinyin) + } + } + + return keywords.distinct() + } + + // ========== 缓存相关 ========== + + private fun saveToFile(apps: List, file: File) { + try { + val jsonArray = JSONArray() + for (app in apps) { + val obj = JSONObject() + obj.put("package", app.packageName) + obj.put("name", app.appName) + obj.put("pinyin", app.pinyin) + obj.put("category", app.category ?: "") + obj.put("system", app.isSystem) + obj.put("keywords", JSONArray(app.keywords)) + jsonArray.put(obj) + } + file.writeText(jsonArray.toString()) + } catch (e: Exception) { + e.printStackTrace() + } + } + + private fun loadFromFile(file: File): List { + val apps = mutableListOf() + try { + val jsonArray = JSONArray(file.readText()) + for (i in 0 until jsonArray.length()) { + val obj = jsonArray.getJSONObject(i) + val keywordsArray = obj.optJSONArray("keywords") + val keywords = mutableListOf() + if (keywordsArray != null) { + for (j in 0 until keywordsArray.length()) { + keywords.add(keywordsArray.getString(j)) + } + } + + apps.add(AppInfo( + packageName = obj.getString("package"), + appName = obj.getString("name"), + pinyin = obj.optString("pinyin", ""), + category = obj.optString("category", null)?.takeIf { it.isNotEmpty() }, + isSystem = obj.optBoolean("system", false), + keywords = keywords + )) + } + } catch (e: Exception) { + e.printStackTrace() + } + return apps + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/controller/DeviceController.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/controller/DeviceController.kt new file mode 100644 index 0000000..6a69a5a --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/controller/DeviceController.kt @@ -0,0 +1,552 @@ +package com.roubao.autopilot.controller + +import android.content.ClipData +import android.content.ClipboardManager +import android.content.ComponentName +import android.content.Context +import android.content.ServiceConnection +import android.graphics.Bitmap +import android.graphics.BitmapFactory +import android.os.Handler +import android.os.IBinder +import android.os.Looper +import com.roubao.autopilot.App +import com.roubao.autopilot.IShellService +import com.roubao.autopilot.service.ShellService +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.delay +import kotlinx.coroutines.withContext +import rikka.shizuku.Shizuku +import java.io.BufferedReader +import java.io.File +import java.io.InputStreamReader +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit + +/** + * 设备控制器 - 通过 Shizuku UserService 执行 shell 命令 + */ +class DeviceController(private val context: Context? = null) { + + companion object { + // 使用 /data/local/tmp，shell 用户有权限访问 + private const val SCREENSHOT_PATH = "/data/local/tmp/autopilot_screen.png" + } + + private var shellService: IShellService? = null + private var serviceBound = false + private val mainHandler = Handler(Looper.getMainLooper()) + private val clipboardManager: ClipboardManager? by lazy { + context?.getSystemService(Context.CLIPBOARD_SERVICE) as? ClipboardManager + } + + private val userServiceArgs = Shizuku.UserServiceArgs( + ComponentName( + "com.roubao.autopilot", + ShellService::class.java.name + ) + ) + .daemon(false) + .processNameSuffix("shell") + .debuggable(true) + .version(1) + + private val serviceConnection = object : ServiceConnection { + override fun onServiceConnected(name: ComponentName?, service: IBinder?) { + shellService = IShellService.Stub.asInterface(service) + serviceBound = true + println("[DeviceController] ShellService connected") + } + + override fun onServiceDisconnected(name: ComponentName?) { + shellService = null + serviceBound = false + println("[DeviceController] ShellService disconnected") + } + } + + /** + * 绑定 Shizuku UserService + */ + fun bindService() { + if (!isShizukuAvailable()) { + println("[DeviceController] Shizuku not available") + return + } + try { + Shizuku.bindUserService(userServiceArgs, serviceConnection) + } catch (e: Exception) { + e.printStackTrace() + } + } + + /** + * 解绑服务 + */ + fun unbindService() { + try { + Shizuku.unbindUserService(userServiceArgs, serviceConnection, true) + } catch (e: Exception) { + e.printStackTrace() + } + } + + /** + * 检查 Shizuku 是否可用 + */ + fun isShizukuAvailable(): Boolean { + return try { + Shizuku.checkSelfPermission() == android.content.pm.PackageManager.PERMISSION_GRANTED + } catch (e: Exception) { + false + } + } + + /** + * 检查服务是否可用 + */ + fun isAvailable(): Boolean { + return serviceBound && shellService != null + } + + /** + * Shizuku 权限级别 + */ + enum class ShizukuPrivilegeLevel { + NONE, // 未连接 + ADB, // ADB 模式 (UID 2000) + ROOT // Root 模式 (UID 0) + } + + /** + * 获取当前 Shizuku 权限级别 + * UID 0 = root, UID 2000 = shell (ADB) + */ + fun getShizukuPrivilegeLevel(): ShizukuPrivilegeLevel { + if (!isAvailable()) { + return ShizukuPrivilegeLevel.NONE + } + return try { + val uid = Shizuku.getUid() + println("[DeviceController] Shizuku UID: $uid") + when (uid) { + 0 -> ShizukuPrivilegeLevel.ROOT + else -> ShizukuPrivilegeLevel.ADB + } + } catch (e: Exception) { + e.printStackTrace() + ShizukuPrivilegeLevel.NONE + } + } + + /** + * 执行 shell 命令 (本地，无权限) + */ + private fun execLocal(command: String): String { + return try { + val process = Runtime.getRuntime().exec(arrayOf("sh", "-c", command)) + val reader = BufferedReader(InputStreamReader(process.inputStream)) + val output = reader.readText() + process.waitFor() + reader.close() + output + } catch (e: Exception) { + e.printStackTrace() + "" + } + } + + /** + * 执行 shell 命令 (通过 Shizuku) + */ + private fun exec(command: String): String { + return try { + shellService?.exec(command) ?: execLocal(command) + } catch (e: Exception) { + e.printStackTrace() + execLocal(command) + } + } + + /** + * 点击屏幕 + */ + fun tap(x: Int, y: Int) { + exec("input tap $x $y") + } + + /** + * 长按 + */ + fun longPress(x: Int, y: Int, durationMs: Int = 1000) { + exec("input swipe $x $y $x $y $durationMs") + } + + /** + * 双击 + */ + fun doubleTap(x: Int, y: Int) { + exec("input tap $x $y && input tap $x $y") + } + + /** + * 滑动 + */ + fun swipe(x1: Int, y1: Int, x2: Int, y2: Int, durationMs: Int = 500) { + exec("input swipe $x1 $y1 $x2 $y2 $durationMs") + } + + /** + * 输入文本 (使用剪贴板方式，支持中文) + */ + fun type(text: String) { + // 检查是否包含非 ASCII 字符 + val hasNonAscii = text.any { it.code > 127 } + + if (hasNonAscii) { + // 中文等使用剪贴板方式 + typeViaClipboard(text) + } else { + // 纯英文数字使用 input text + val escaped = text.replace("'", "'\\''") + exec("input text '$escaped'") + } + } + + /** + * 通过剪贴板方式输入中文 + * 使用 Android ClipboardManager API 设置剪贴板，然后发送粘贴按键 + */ + private fun typeViaClipboard(text: String) { + println("[DeviceController] 尝试输入中文: $text") + + // 方法1: 使用 Android 剪贴板 API + 粘贴 (最可靠，不需要额外 App) + if (clipboardManager != null) { + try { + // 使用 CountDownLatch 等待剪贴板设置完成 + val latch = CountDownLatch(1) + var clipboardSet = false + + // 必须在主线程操作剪贴板 + mainHandler.post { + try { + val clip = ClipData.newPlainText("baozi_input", text) + clipboardManager?.setPrimaryClip(clip) + clipboardSet = true + println("[DeviceController] ✅ 已设置剪贴板: $text") + } catch (e: Exception) { + println("[DeviceController] ❌ 设置剪贴板异常: ${e.message}") + } finally { + latch.countDown() + } + } + + // 等待剪贴板设置完成 (最多等 1 秒) + val success = latch.await(1, TimeUnit.SECONDS) + if (!success) { + println("[DeviceController] ❌ 等待剪贴板超时") + return + } + + if (!clipboardSet) { + println("[DeviceController] ❌ 剪贴板设置失败") + return + } + + // 稍等一下确保剪贴板生效 + Thread.sleep(200) + + // 发送粘贴按键 (KEYCODE_PASTE = 279) + exec("input keyevent 279") + println("[DeviceController] ✅ 已发送粘贴按键") + return + } catch (e: Exception) { + println("[DeviceController] ❌ 剪贴板方式失败: ${e.message}") + e.printStackTrace() + } + } else { + println("[DeviceController] ❌ ClipboardManager 为 null，Context 未设置") + } + + // 方法2: 使用 ADB Keyboard 广播 (备选，需要安装 ADBKeyboard) + val escaped = text.replace("\"", "\\\"") + val adbKeyboardResult = exec("am broadcast -a ADB_INPUT_TEXT --es msg \"$escaped\"") + println("[DeviceController] ADBKeyboard 广播结果: $adbKeyboardResult") + + if (adbKeyboardResult.contains("result=0")) { + println("[DeviceController] ✅ ADBKeyboard 输入成功") + return + } + + // 方法3: 使用 cmd input text (Android 12+ 可能支持 UTF-8) + println("[DeviceController] 尝试 cmd input text...") + exec("cmd input text '$text'") + } + + /** + * 输入文本 (逐字符，兼容性更好) + */ + fun typeCharByChar(text: String) { + text.forEach { char -> + when { + char == ' ' -> exec("input text %s") + char == '\n' -> exec("input keyevent 66") + char.isLetterOrDigit() && char.code <= 127 -> exec("input text $char") + char in "-.,!?@'/:;()" -> exec("input text \"$char\"") + else -> { + // 非 ASCII 字符使用广播 + exec("am broadcast -a ADB_INPUT_TEXT --es msg \"$char\"") + } + } + } + } + + /** + * 返回键 + */ + fun back() { + exec("input keyevent 4") + } + + /** + * Home 键 + */ + fun home() { + exec("input keyevent 3") + } + + /** + * 回车键 + */ + fun enter() { + exec("input keyevent 66") + } + + private var cacheDir: File? = null + + fun setCacheDir(dir: File) { + cacheDir = dir + } + + /** + * 截图结果 + */ + data class ScreenshotResult( + val bitmap: Bitmap, + val isSensitive: Boolean = false, // 是否是敏感页面（截图失败） + val isFallback: Boolean = false // 是否是降级的黑屏占位图 + ) + + /** + * 截图 - 使用 /data/local/tmp 并设置全局可读权限 + * 失败时返回黑屏占位图（降级处理） + */ + suspend fun screenshotWithFallback(): ScreenshotResult = withContext(Dispatchers.IO) { + try { + // 截图到 /data/local/tmp 并设置权限让 App 可读 + val output = exec("screencap -p $SCREENSHOT_PATH && chmod 666 $SCREENSHOT_PATH") + delay(500) + + // 检查是否截图失败（敏感页面保护） + if (output.contains("Status: -1") || output.contains("Failed") || output.contains("error")) { + println("[DeviceController] Screenshot blocked (sensitive screen), returning fallback") + return@withContext createFallbackScreenshot(isSensitive = true) + } + + // 尝试直接读取 + val file = File(SCREENSHOT_PATH) + if (file.exists() && file.canRead() && file.length() > 0) { + println("[DeviceController] Reading screenshot from: $SCREENSHOT_PATH, size: ${file.length()}") + val bitmap = BitmapFactory.decodeFile(SCREENSHOT_PATH) + if (bitmap != null) { + return@withContext ScreenshotResult(bitmap) + } + } + + // 如果无法直接读取，通过 shell cat 读取二进制数据 + println("[DeviceController] Cannot read directly, trying shell cat...") + val process = Runtime.getRuntime().exec(arrayOf("su", "-c", "cat $SCREENSHOT_PATH")) + val bytes = process.inputStream.readBytes() + process.waitFor() + + if (bytes.isNotEmpty()) { + println("[DeviceController] Read ${bytes.size} bytes via shell") + val bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.size) + if (bitmap != null) { + return@withContext ScreenshotResult(bitmap) + } + } + + println("[DeviceController] Screenshot file empty or not accessible, returning fallback") + createFallbackScreenshot(isSensitive = false) + } catch (e: Exception) { + e.printStackTrace() + println("[DeviceController] Screenshot exception, returning fallback") + createFallbackScreenshot(isSensitive = false) + } + } + + /** + * 创建黑屏占位图（降级处理） + */ + private fun createFallbackScreenshot(isSensitive: Boolean): ScreenshotResult { + val (width, height) = getScreenSize() + val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888) + // 默认是黑色，无需填充 + return ScreenshotResult( + bitmap = bitmap, + isSensitive = isSensitive, + isFallback = true + ) + } + + /** + * 截图 - 使用 /data/local/tmp 并设置全局可读权限 + * @deprecated 使用 screenshotWithFallback() 代替 + */ + suspend fun screenshot(): Bitmap? = withContext(Dispatchers.IO) { + try { + // 截图到 /data/local/tmp 并设置权限让 App 可读 + exec("screencap -p $SCREENSHOT_PATH && chmod 666 $SCREENSHOT_PATH") + delay(500) + + // 尝试直接读取 + val file = File(SCREENSHOT_PATH) + if (file.exists() && file.canRead() && file.length() > 0) { + println("[DeviceController] Reading screenshot from: $SCREENSHOT_PATH, size: ${file.length()}") + return@withContext BitmapFactory.decodeFile(SCREENSHOT_PATH) + } + + // 如果无法直接读取，通过 shell cat 读取二进制数据 + println("[DeviceController] Cannot read directly, trying shell cat...") + val process = Runtime.getRuntime().exec(arrayOf("su", "-c", "cat $SCREENSHOT_PATH")) + val bytes = process.inputStream.readBytes() + process.waitFor() + + if (bytes.isNotEmpty()) { + println("[DeviceController] Read ${bytes.size} bytes via shell") + BitmapFactory.decodeByteArray(bytes, 0, bytes.size) + } else { + println("[DeviceController] Screenshot file empty or not accessible") + null + } + } catch (e: Exception) { + e.printStackTrace() + null + } + } + + /** + * 获取屏幕尺寸（考虑屏幕方向） + */ + fun getScreenSize(): Pair { + val output = exec("wm size") + // 输出格式: Physical size: 1080x2400 + val match = Regex("(\\d+)x(\\d+)").find(output) + val (physicalWidth, physicalHeight) = if (match != null) { + val (w, h) = match.destructured + Pair(w.toInt(), h.toInt()) + } else { + Pair(1080, 2400) + } + + // 检测屏幕方向 + val orientation = getScreenOrientation() + return if (orientation == 1 || orientation == 3) { + // 横屏：交换宽高 + Pair(physicalHeight, physicalWidth) + } else { + // 竖屏 + Pair(physicalWidth, physicalHeight) + } + } + + /** + * 获取屏幕方向 + * @return 0=竖屏, 1=横屏(90°), 2=倒置竖屏, 3=横屏(270°) + */ + private fun getScreenOrientation(): Int { + val output = exec("dumpsys window displays | grep mCurrentOrientation") + // 输出格式: mCurrentOrientation=0 或 mCurrentOrientation=1 + val match = Regex("mCurrentOrientation=(\\d)").find(output) + return match?.groupValues?.get(1)?.toIntOrNull() ?: 0 + } + + /** + * 打开 App - 支持包名或应用名 + */ + fun openApp(appNameOrPackage: String) { + // 常见应用名到包名的映射 (作为备选) + val packageMap = mapOf( + "settings" to "com.android.settings", + "设置" to "com.android.settings", + "chrome" to "com.android.chrome", + "浏览器" to "com.android.browser", + "camera" to "com.android.camera", + "相机" to "com.android.camera", + "phone" to "com.android.dialer", + "电话" to "com.android.dialer", + "contacts" to "com.android.contacts", + "联系人" to "com.android.contacts", + "messages" to "com.android.mms", + "短信" to "com.android.mms", + "gallery" to "com.android.gallery3d", + "相册" to "com.android.gallery3d", + "clock" to "com.android.deskclock", + "时钟" to "com.android.deskclock", + "calculator" to "com.android.calculator2", + "计算器" to "com.android.calculator2", + "calendar" to "com.android.calendar", + "日历" to "com.android.calendar", + "files" to "com.android.documentsui", + "文件" to "com.android.documentsui" + ) + + val lowerName = appNameOrPackage.lowercase().trim() + val finalPackage: String + + if (appNameOrPackage.contains(".")) { + // 已经是包名格式 + finalPackage = appNameOrPackage + } else if (packageMap.containsKey(lowerName)) { + // 从内置映射中查找 + finalPackage = packageMap[lowerName]!! + } else { + // 使用 AppScanner 搜索应用 + val appScanner = App.getInstance().appScanner + val searchResults = appScanner.searchApps(appNameOrPackage, topK = 1) + if (searchResults.isNotEmpty()) { + finalPackage = searchResults[0].app.packageName + println("[DeviceController] AppScanner found: ${searchResults[0].app.appName} -> $finalPackage") + } else { + // 找不到，直接用原始输入尝试 + finalPackage = appNameOrPackage + println("[DeviceController] App not found in AppScanner: $appNameOrPackage") + } + } + + // 使用 monkey 命令启动应用 (最可靠) + val result = exec("monkey -p $finalPackage -c android.intent.category.LAUNCHER 1 2>/dev/null") + println("[DeviceController] openApp: $appNameOrPackage -> $finalPackage, result: $result") + } + + /** + * 通过 Intent 打开 + */ + fun openIntent(action: String, data: String? = null) { + val cmd = buildString { + append("am start -a $action") + if (data != null) { + append(" -d \"$data\"") + } + } + exec(cmd) + } + + /** + * 打开 DeepLink + */ + fun openDeepLink(uri: String) { + exec("am start -a android.intent.action.VIEW -d \"$uri\"") + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/data/ExecutionHistory.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/data/ExecutionHistory.kt new file mode 100644 index 0000000..596561c --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/data/ExecutionHistory.kt @@ -0,0 +1,209 @@ +package com.roubao.autopilot.data + +import android.content.Context +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext +import org.json.JSONArray +import org.json.JSONObject +import java.io.File +import java.text.SimpleDateFormat +import java.util.* + +/** + * 执行步骤记录 + */ +data class ExecutionStep( + val stepNumber: Int, + val timestamp: Long, + val action: String, + val description: String, + val thought: String, + val outcome: String, // A=成功, B=部分成功, C=失败 + val screenshotPath: String? = null +) { + fun toJson(): JSONObject = JSONObject().apply { + put("stepNumber", stepNumber) + put("timestamp", timestamp) + put("action", action) + put("description", description) + put("thought", thought) + put("outcome", outcome) + put("screenshotPath", screenshotPath ?: "") + } + + companion object { + fun fromJson(json: JSONObject): ExecutionStep = ExecutionStep( + stepNumber = json.optInt("stepNumber", 0), + timestamp = json.optLong("timestamp", 0), + action = json.optString("action", ""), + description = json.optString("description", ""), + thought = json.optString("thought", ""), + outcome = json.optString("outcome", ""), + screenshotPath = json.optString("screenshotPath", "").ifEmpty { null } + ) + } +} + +/** + * 执行记录 + */ +data class ExecutionRecord( + val id: String = UUID.randomUUID().toString(), + val title: String, + val instruction: String, + val startTime: Long, + val endTime: Long = 0, + val status: ExecutionStatus = ExecutionStatus.RUNNING, + val steps: List = emptyList(), + val logs: List = emptyList(), + val resultMessage: String = "" +) { + val duration: Long get() = if (endTime > 0) endTime - startTime else System.currentTimeMillis() - startTime + + val formattedStartTime: String get() { + val sdf = SimpleDateFormat("MM-dd HH:mm", Locale.getDefault()) + return sdf.format(Date(startTime)) + } + + val formattedDuration: String get() { + val seconds = duration / 1000 + return if (seconds < 60) "${seconds}秒" else "${seconds / 60}分${seconds % 60}秒" + } + + fun toJson(): JSONObject = JSONObject().apply { + put("id", id) + put("title", title) + put("instruction", instruction) + put("startTime", startTime) + put("endTime", endTime) + put("status", status.name) + put("resultMessage", resultMessage) + put("steps", JSONArray().apply { + steps.forEach { put(it.toJson()) } + }) + put("logs", JSONArray().apply { + logs.forEach { put(it) } + }) + } + + companion object { + fun fromJson(json: JSONObject): ExecutionRecord { + val stepsArray = json.optJSONArray("steps") ?: JSONArray() + val steps = mutableListOf() + for (i in 0 until stepsArray.length()) { + steps.add(ExecutionStep.fromJson(stepsArray.getJSONObject(i))) + } + val logsArray = json.optJSONArray("logs") ?: JSONArray() + val logs = mutableListOf() + for (i in 0 until logsArray.length()) { + logs.add(logsArray.optString(i, "")) + } + return ExecutionRecord( + id = json.optString("id", UUID.randomUUID().toString()), + title = json.optString("title", ""), + instruction = json.optString("instruction", ""), + startTime = json.optLong("startTime", 0), + endTime = json.optLong("endTime", 0), + status = try { + ExecutionStatus.valueOf(json.optString("status", "COMPLETED")) + } catch (e: Exception) { + ExecutionStatus.COMPLETED + }, + steps = steps, + logs = logs, + resultMessage = json.optString("resultMessage", "") + ) + } + } +} + +enum class ExecutionStatus { + RUNNING, + COMPLETED, + FAILED, + STOPPED +} + +/** + * 执行记录仓库 + */ +class ExecutionRepository(private val context: Context) { + + private val historyFile: File + get() = File(context.filesDir, "execution_history.json") + + /** + * 获取所有记录 + */ + suspend fun getAllRecords(): List = withContext(Dispatchers.IO) { + try { + if (!historyFile.exists()) return@withContext emptyList() + val json = historyFile.readText() + val array = JSONArray(json) + val records = mutableListOf() + for (i in 0 until array.length()) { + records.add(ExecutionRecord.fromJson(array.getJSONObject(i))) + } + records.sortedByDescending { it.startTime } + } catch (e: Exception) { + e.printStackTrace() + emptyList() + } + } + + /** + * 获取单条记录 + */ + suspend fun getRecord(id: String): ExecutionRecord? = withContext(Dispatchers.IO) { + getAllRecords().find { it.id == id } + } + + /** + * 保存记录 + */ + suspend fun saveRecord(record: ExecutionRecord) = withContext(Dispatchers.IO) { + try { + val records = getAllRecords().toMutableList() + val existingIndex = records.indexOfFirst { it.id == record.id } + if (existingIndex >= 0) { + records[existingIndex] = record + } else { + records.add(0, record) + } + // 只保留最近100条记录 + val trimmedRecords = records.take(100) + val array = JSONArray().apply { + trimmedRecords.forEach { put(it.toJson()) } + } + historyFile.writeText(array.toString()) + } catch (e: Exception) { + e.printStackTrace() + } + } + + /** + * 删除记录 + */ + suspend fun deleteRecord(id: String) = withContext(Dispatchers.IO) { + try { + val records = getAllRecords().filter { it.id != id } + val array = JSONArray().apply { + records.forEach { put(it.toJson()) } + } + historyFile.writeText(array.toString()) + } catch (e: Exception) { + e.printStackTrace() + } + } + + /** + * 清空所有记录 + */ + suspend fun clearAll() = withContext(Dispatchers.IO) { + try { + historyFile.delete() + } catch (e: Exception) { + e.printStackTrace() + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/data/SettingsManager.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/data/SettingsManager.kt new file mode 100644 index 0000000..24ab391 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/data/SettingsManager.kt @@ -0,0 +1,352 @@ +package com.roubao.autopilot.data + +import android.content.Context +import android.content.SharedPreferences +import androidx.security.crypto.EncryptedSharedPreferences +import androidx.security.crypto.MasterKey +import com.roubao.autopilot.ui.theme.ThemeMode +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow + +/** + * API 提供商配置 + */ +data class ApiProvider( + val id: String, + val name: String, + val baseUrl: String, + val defaultModel: String, + val isGUIAgent: Boolean = false // 是否为 GUI Agent 专用协议（非 OpenAI 兼容） +) { + companion object { + val GUI_OWL = ApiProvider( + id = "gui_owl", + name = "GUI-Owl (阿里云)", + baseUrl = "https://dashscope.aliyuncs.com/api/v2/apps/gui-owl/gui_agent_server", + defaultModel = "pre-gui_owl_7b", + isGUIAgent = true + ) + val MAI_UI = ApiProvider( + id = "mai_ui", + name = "MAI-UI (本地部署)", + baseUrl = "http://localhost:8000/v1", // vLLM 默认地址 + defaultModel = "MAI-UI-2B" // 支持 MAI-UI-2B 或 MAI-UI-8B + ) + val ALIYUN = ApiProvider( + id = "aliyun", + name = "阿里云 (Qwen-VL)", + baseUrl = "https://dashscope.aliyuncs.com/compatible-mode/v1", + defaultModel = "qwen3-vl-plus" + ) + val OPENAI = ApiProvider( + id = "openai", + name = "OpenAI", + baseUrl = "https://api.openai.com/v1", + defaultModel = "gpt-4o" + ) + val OPENROUTER = ApiProvider( + id = "openrouter", + name = "OpenRouter", + baseUrl = "https://openrouter.ai/api/v1", + defaultModel = "anthropic/claude-3.5-sonnet" + ) + val CUSTOM = ApiProvider( + id = "custom", + name = "自定义", + baseUrl = "", + defaultModel = "" + ) + + val ALL = listOf(GUI_OWL, MAI_UI, ALIYUN, OPENAI, OPENROUTER, CUSTOM) + } +} + +/** + * 服务商配置（每个服务商独立保存） + */ +data class ProviderConfig( + val apiKey: String = "", + val model: String = "", + val cachedModels: List = emptyList(), + val customBaseUrl: String = "" // 仅 custom 服务商使用 +) + +/** + * 默认推荐模型 + */ +const val DEFAULT_MODEL = "qwen3-vl-plus" + +/** + * 应用设置 + */ +data class AppSettings( + val currentProviderId: String = ApiProvider.ALIYUN.id, // 当前选中的服务商 + val providerConfigs: Map = emptyMap(), // 每个服务商的配置 + val themeMode: ThemeMode = ThemeMode.SYSTEM, + val hasSeenOnboarding: Boolean = false, + val maxSteps: Int = 25, + val rootModeEnabled: Boolean = false, + val suCommandEnabled: Boolean = false +) { + // 便捷属性：获取当前服务商的配置 + val currentConfig: ProviderConfig + get() = providerConfigs[currentProviderId] ?: ProviderConfig() + + val currentProvider: ApiProvider + get() = ApiProvider.ALL.find { it.id == currentProviderId } ?: ApiProvider.ALIYUN + + val apiKey: String get() = currentConfig.apiKey + val model: String get() = currentConfig.model.ifEmpty { currentProvider.defaultModel } + val cachedModels: List get() = currentConfig.cachedModels + + val baseUrl: String + get() = when { + currentProviderId == "custom" -> currentConfig.customBaseUrl + // MAI-UI 支持自定义 URL（用于远程部署） + currentProviderId == "mai_ui" && currentConfig.customBaseUrl.isNotEmpty() -> currentConfig.customBaseUrl + else -> currentProvider.baseUrl + } +} + +/** + * 设置管理器 + */ +class SettingsManager(context: Context) { + + // 普通设置存储 + private val prefs: SharedPreferences = + context.getSharedPreferences("baozi_settings", Context.MODE_PRIVATE) + + // 加密存储（用于敏感数据如 API Key） + private val securePrefs: SharedPreferences by lazy { + try { + val masterKey = MasterKey.Builder(context) + .setKeyScheme(MasterKey.KeyScheme.AES256_GCM) + .build() + + EncryptedSharedPreferences.create( + context, + "baozi_secure_settings", + masterKey, + EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV, + EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM + ) + } catch (e: Exception) { + // 加密失败时回退到普通存储（不应该发生） + android.util.Log.e("SettingsManager", "Failed to create encrypted prefs", e) + prefs + } + } + + private val _settings = MutableStateFlow(loadSettings()) + val settings: StateFlow = _settings + + init { + // 迁移旧的明文 API Key 到加密存储 + migrateApiKeyToSecureStorage() + } + + /** + * 迁移旧的明文 API Key 到加密存储 + */ + private fun migrateApiKeyToSecureStorage() { + val oldApiKey = prefs.getString("api_key", null) + if (!oldApiKey.isNullOrEmpty()) { + // 保存到加密存储 + securePrefs.edit().putString("api_key", oldApiKey).apply() + // 删除旧的明文存储 + prefs.edit().remove("api_key").apply() + android.util.Log.d("SettingsManager", "API Key migrated to secure storage") + } + } + + private fun loadSettings(): AppSettings { + val themeModeStr = prefs.getString("theme_mode", ThemeMode.SYSTEM.name) ?: ThemeMode.SYSTEM.name + val themeMode = try { + ThemeMode.valueOf(themeModeStr) + } catch (e: Exception) { + ThemeMode.SYSTEM + } + + // 加载当前选中的服务商 + val currentProviderId = prefs.getString("current_provider_id", ApiProvider.ALIYUN.id) ?: ApiProvider.ALIYUN.id + + // 加载每个服务商的配置 + val providerConfigs = mutableMapOf() + for (provider in ApiProvider.ALL) { + val config = loadProviderConfig(provider.id) + providerConfigs[provider.id] = config + } + + // 迁移旧数据（如果有） + val oldApiKey = securePrefs.getString("api_key", null) + val oldModel = prefs.getString("model", null) + val oldBaseUrl = prefs.getString("base_url", null) + val oldCachedModels = prefs.getStringSet("cached_models", null) + + if (oldApiKey != null || oldModel != null) { + // 找到旧数据对应的服务商 + val oldProviderId = when (oldBaseUrl) { + ApiProvider.ALIYUN.baseUrl -> ApiProvider.ALIYUN.id + ApiProvider.OPENAI.baseUrl -> ApiProvider.OPENAI.id + ApiProvider.OPENROUTER.baseUrl -> ApiProvider.OPENROUTER.id + else -> "custom" + } + + // 迁移到新格式 + val migratedConfig = ProviderConfig( + apiKey = oldApiKey ?: "", + model = oldModel ?: "", + cachedModels = oldCachedModels?.toList() ?: emptyList(), + customBaseUrl = if (oldProviderId == "custom") oldBaseUrl ?: "" else "" + ) + providerConfigs[oldProviderId] = migratedConfig + saveProviderConfig(oldProviderId, migratedConfig) + + // 清除旧数据 + securePrefs.edit().remove("api_key").apply() + prefs.edit() + .remove("model") + .remove("base_url") + .remove("cached_models") + .putString("current_provider_id", oldProviderId) + .apply() + + android.util.Log.d("SettingsManager", "Migrated old settings to provider: $oldProviderId") + } + + return AppSettings( + currentProviderId = currentProviderId, + providerConfigs = providerConfigs, + themeMode = themeMode, + hasSeenOnboarding = prefs.getBoolean("has_seen_onboarding", false), + maxSteps = prefs.getInt("max_steps", 25), + rootModeEnabled = prefs.getBoolean("root_mode_enabled", false), + suCommandEnabled = prefs.getBoolean("su_command_enabled", false) + ) + } + + /** + * 加载指定服务商的配置 + */ + private fun loadProviderConfig(providerId: String): ProviderConfig { + val prefix = "provider_${providerId}_" + return ProviderConfig( + apiKey = securePrefs.getString("${prefix}api_key", "") ?: "", + model = prefs.getString("${prefix}model", "") ?: "", + cachedModels = prefs.getStringSet("${prefix}cached_models", emptySet())?.toList() ?: emptyList(), + customBaseUrl = prefs.getString("${prefix}custom_base_url", "") ?: "" + ) + } + + /** + * 保存指定服务商的配置 + */ + private fun saveProviderConfig(providerId: String, config: ProviderConfig) { + val prefix = "provider_${providerId}_" + securePrefs.edit().putString("${prefix}api_key", config.apiKey).apply() + prefs.edit() + .putString("${prefix}model", config.model) + .putStringSet("${prefix}cached_models", config.cachedModels.toSet()) + .putString("${prefix}custom_base_url", config.customBaseUrl) + .apply() + } + + /** + * 更新当前服务商的配置 + */ + private fun updateCurrentConfig(update: (ProviderConfig) -> ProviderConfig) { + val currentId = _settings.value.currentProviderId + val currentConfig = _settings.value.currentConfig + val newConfig = update(currentConfig) + + saveProviderConfig(currentId, newConfig) + + val newConfigs = _settings.value.providerConfigs.toMutableMap() + newConfigs[currentId] = newConfig + _settings.value = _settings.value.copy(providerConfigs = newConfigs) + } + + fun updateApiKey(apiKey: String) { + updateCurrentConfig { it.copy(apiKey = apiKey) } + } + + fun updateBaseUrl(baseUrl: String) { + // 自定义服务商和 MAI-UI 可以修改 URL + val providerId = _settings.value.currentProviderId + if (providerId == "custom" || providerId == "mai_ui") { + updateCurrentConfig { it.copy(customBaseUrl = baseUrl) } + } + } + + fun updateModel(model: String) { + updateCurrentConfig { it.copy(model = model) } + } + + /** + * 更新缓存的模型列表（从 API 获取后调用） + */ + fun updateCachedModels(models: List) { + val distinctModels = models.distinct() + updateCurrentConfig { it.copy(cachedModels = distinctModels) } + } + + /** + * 清空缓存的模型列表 + */ + fun clearCachedModels() { + updateCurrentConfig { it.copy(cachedModels = emptyList()) } + } + + /** + * 选择服务商（切换时自动加载该服务商的配置） + */ + fun selectProvider(provider: ApiProvider) { + prefs.edit().putString("current_provider_id", provider.id).apply() + _settings.value = _settings.value.copy(currentProviderId = provider.id) + } + + /** + * 获取当前服务商 + */ + fun getCurrentProvider(): ApiProvider { + return _settings.value.currentProvider + } + + /** + * 判断是否使用自定义 URL + */ + fun isCustomUrl(): Boolean { + return _settings.value.currentProviderId == "custom" + } + + fun updateThemeMode(themeMode: ThemeMode) { + prefs.edit().putString("theme_mode", themeMode.name).apply() + _settings.value = _settings.value.copy(themeMode = themeMode) + } + + fun setOnboardingSeen() { + prefs.edit().putBoolean("has_seen_onboarding", true).apply() + _settings.value = _settings.value.copy(hasSeenOnboarding = true) + } + + fun updateMaxSteps(maxSteps: Int) { + val validSteps = maxSteps.coerceIn(5, 100) // 限制范围 5-100 + prefs.edit().putInt("max_steps", validSteps).apply() + _settings.value = _settings.value.copy(maxSteps = validSteps) + } + + fun updateRootModeEnabled(enabled: Boolean) { + prefs.edit().putBoolean("root_mode_enabled", enabled).apply() + _settings.value = _settings.value.copy(rootModeEnabled = enabled) + // 关闭 Root 模式时，同时关闭 su -c + if (!enabled) { + updateSuCommandEnabled(false) + } + } + + fun updateSuCommandEnabled(enabled: Boolean) { + prefs.edit().putBoolean("su_command_enabled", enabled).apply() + _settings.value = _settings.value.copy(suCommandEnabled = enabled) + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/service/ShellService.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/service/ShellService.kt new file mode 100644 index 0000000..876e6fa --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/service/ShellService.kt @@ -0,0 +1,35 @@ +package com.roubao.autopilot.service + +import com.roubao.autopilot.IShellService +import java.io.BufferedReader +import java.io.InputStreamReader +import kotlin.system.exitProcess + +/** + * Shizuku UserService - 在 shell/root 权限下执行命令 + */ +class ShellService : IShellService.Stub() { + + override fun destroy() { + exitProcess(0) + } + + override fun exec(command: String): String { + return try { + val process = Runtime.getRuntime().exec(arrayOf("sh", "-c", command)) + val reader = BufferedReader(InputStreamReader(process.inputStream)) + val errorReader = BufferedReader(InputStreamReader(process.errorStream)) + + val output = reader.readText() + val error = errorReader.readText() + + process.waitFor() + reader.close() + errorReader.close() + + if (output.isNotEmpty()) output else error + } catch (e: Exception) { + "Error: ${e.message}" + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/skills/Skill.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/skills/Skill.kt new file mode 100644 index 0000000..5d18456 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/skills/Skill.kt @@ -0,0 +1,299 @@ +package com.roubao.autopilot.skills + +import com.roubao.autopilot.tools.ToolManager + +/** + * 执行类型 + */ +enum class ExecutionType { + /** 委托：通过 DeepLink 打开 App */ + DELEGATION, + /** GUI 自动化：通过截图-操作循环 */ + GUI_AUTOMATION +} + +/** + * 关联应用配置 + */ +data class RelatedApp( + val packageName: String, + val name: String, + val type: ExecutionType, + val deepLink: String? = null, + val steps: List? = null, + val priority: Int = 0, + val description: String? = null +) + +/** + * Skill 参数定义 + */ +data class SkillParam( + val name: String, + val type: String, // string, int, boolean + val description: String, + val required: Boolean = false, + val defaultValue: Any? = null, + val examples: List = emptyList() +) + +/** + * Skill 配置（意图定义） + */ +data class SkillConfig( + val id: String, + val name: String, + val description: String, + val category: String, + val keywords: List, + val params: List, + val relatedApps: List, + val promptHint: String? = null // 提示词约束，如"内容不超过100字" +) + +/** + * Skill 执行计划 + * + * 根据用户意图和本地已安装 App 生成的执行方案 + */ +data class ExecutionPlan( + val skillId: String, + val skillName: String, + val app: RelatedApp, + val params: Map, + val isInstalled: Boolean, + val promptHint: String? = null // 提示词约束 +) { + /** + * 生成给 Agent 的上下文信息 + */ + fun toAgentContext(): String { + return buildString { + append("【任务】${skillName}\n") + append("【目标应用】${app.name} (${app.packageName})\n") + append("【执行方式】${if (app.type == ExecutionType.DELEGATION) "快捷跳转" else "GUI 自动化"}\n") + + if (!promptHint.isNullOrBlank()) { + append("【重要提示】⚠️ $promptHint\n") + } + + if (!app.steps.isNullOrEmpty()) { + append("【操作步骤】\n") + app.steps.forEachIndexed { index, step -> + append(" ${index + 1}. $step\n") + } + } + + if (params.isNotEmpty()) { + append("【参数】\n") + params.forEach { (key, value) -> + if (key != "_raw_query" && value != null) { + append(" $key: $value\n") + } + } + } + } + } +} + +/** + * Skill 执行结果 + */ +sealed class SkillResult { + /** + * 委托成功：已通过 DeepLink 跳转 + */ + data class Delegated( + val app: RelatedApp, + val deepLink: String, + val message: String + ) : SkillResult() + + /** + * GUI 自动化：返回执行计划给 Agent + */ + data class NeedAutomation( + val plan: ExecutionPlan, + val message: String + ) : SkillResult() + + /** + * 失败 + */ + data class Failed( + val error: String, + val suggestion: String? = null + ) : SkillResult() + + /** + * 无可用应用 + */ + data class NoAvailableApp( + val skillName: String, + val requiredApps: List + ) : SkillResult() +} + +/** + * Skill 意图匹配器 + */ +class Skill(val config: SkillConfig) { + + /** + * 计算与用户查询的匹配分数 + * @return 0-1 之间的分数 + */ + fun matchScore(query: String): Float { + val lowerQuery = query.lowercase() + + // 精确匹配关键词（最高分） + for (keyword in config.keywords) { + if (lowerQuery.contains(keyword.lowercase())) { + return 0.9f + } + } + + // 匹配 Skill 名称 + if (lowerQuery.contains(config.name.lowercase())) { + return 0.8f + } + + // 模糊匹配描述 + val descWords = config.description.split(" ", "，", "、", "/") + val matchedWords = descWords.count { lowerQuery.contains(it.lowercase()) } + if (matchedWords > 0) { + return (0.3f + 0.3f * matchedWords / descWords.size).coerceAtMost(0.7f) + } + + return 0f + } + + /** + * 从查询中提取参数 + */ + fun extractParams(query: String): Map { + val params = mutableMapOf() + + for (param in config.params) { + when (param.name) { + "food", "item", "song", "book", "keyword" -> { + // 提取关键内容（去掉意图关键词后的部分） + var content = query + for (kw in config.keywords) { + content = content.replace(kw, "", ignoreCase = true) + } + content = content.trim() + if (content.isNotEmpty()) { + params[param.name] = content + } + } + "destination", "address", "location" -> { + // 提取目的地 + val patterns = listOf( + "去(.+?)$", + "到(.+?)$", + "导航(.+?)$", + "(.+?)怎么走" + ) + for (pattern in patterns) { + val match = Regex(pattern).find(query) + if (match != null) { + params[param.name] = match.groupValues[1].trim() + break + } + } + } + "contact" -> { + // 提取联系人 + val patterns = listOf( + "给(.+?)发", + "跟(.+?)说", + "告诉(.+?)" + ) + for (pattern in patterns) { + val match = Regex(pattern).find(query) + if (match != null) { + params[param.name] = match.groupValues[1].trim() + break + } + } + } + "message", "content", "prompt" -> { + // 保存原始查询作为内容 + params[param.name] = query + } + "time" -> { + // 提取时间 + val patterns = listOf( + "(\\d{1,2}[点:：]\\d{0,2})", + "(\\d{1,2}点)", + "(早上|上午|中午|下午|晚上|明天).{0,5}(\\d{1,2}[点:：]?\\d{0,2}?)" + ) + for (pattern in patterns) { + val match = Regex(pattern).find(query) + if (match != null) { + params[param.name] = match.value + break + } + } + } + } + + // 设置默认值 + if (!params.containsKey(param.name) && param.defaultValue != null) { + params[param.name] = param.defaultValue + } + } + + // 保存原始查询 + params["_raw_query"] = query + + return params + } + + /** + * 生成 DeepLink（替换参数） + */ + fun generateDeepLink(app: RelatedApp, params: Map): String { + var deepLink = app.deepLink ?: return "" + + for ((key, value) in params) { + if (value != null && key != "_raw_query") { + deepLink = deepLink.replace("{$key}", value.toString()) + } + } + + // 清理未替换的占位符 + deepLink = deepLink.replace(Regex("\\{[^}]+\\}"), "") + + return deepLink + } +} + +/** + * Skill 匹配结果 + */ +data class SkillMatch( + val skill: Skill, + val score: Float, + val params: Map +) + +/** + * 可用应用匹配结果 + */ +data class AvailableAppMatch( + val skill: Skill, + val app: RelatedApp, + val params: Map, + val score: Float +) + +/** + * LLM 意图匹配结果 + */ +data class LLMIntentMatch( + val skillId: String, + val confidence: Float, + val reasoning: String +) diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/skills/SkillManager.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/skills/SkillManager.kt new file mode 100644 index 0000000..d3f2e7b --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/skills/SkillManager.kt @@ -0,0 +1,503 @@ +package com.roubao.autopilot.skills + +import android.content.Context +import android.content.Intent +import android.net.Uri +import com.roubao.autopilot.controller.AppScanner +import com.roubao.autopilot.tools.ToolManager +import com.roubao.autopilot.vlm.VLMClient +import org.json.JSONObject + +/** + * Skill 管理器 + * + * 作为 Skill 层的统一入口，负责： + * - 初始化和加载 Skills + * - 意图识别和 Skill 匹配（使用 LLM 语义理解） + * - 基于已安装 App 选择最佳执行方案 + * - Skill 执行调度 + */ +class SkillManager private constructor( + private val context: Context, + private val toolManager: ToolManager, + private val appScanner: AppScanner +) { + + private val registry: SkillRegistry = SkillRegistry.init(context, appScanner) + + // VLM 客户端（用于意图匹配） + private var vlmClient: VLMClient? = null + + /** + * 设置 VLM 客户端（用于 LLM 意图匹配） + */ + fun setVLMClient(client: VLMClient) { + this.vlmClient = client + } + + /** + * 初始化：加载 Skills 配置 + */ + fun initialize() { + val loadedCount = registry.loadFromAssets("skills.json") + println("[SkillManager] 已加载 $loadedCount 个 Skills") + } + + /** + * 刷新已安装应用列表 + */ + fun refreshInstalledApps() { + registry.refreshInstalledApps() + } + + /** + * 处理用户意图（新方法：返回最佳可用应用） + * + * @param query 用户输入 + * @return 可用应用匹配结果，如果没有则返回 null + */ + fun matchAvailableApp(query: String): AvailableAppMatch? { + return registry.getBestAvailableApp(query, minScore = 0.3f) + } + + /** + * 获取所有匹配的可用应用 + */ + fun matchAllAvailableApps(query: String): List { + return registry.matchAvailableApps(query, minScore = 0.2f) + } + + /** + * 使用 LLM 进行意图匹配（异步方法） + * + * @param query 用户输入 + * @return 匹配的 Skill ID，如果没有匹配返回 null + */ + suspend fun matchIntentWithLLM(query: String): LLMIntentMatch? { + val client = vlmClient ?: return null + + // 构建 Skills 列表描述 + val skillsInfo = buildString { + append("可用技能列表：\n") + for (skill in registry.getAll()) { + val config = skill.config + // 只展示有已安装应用的 Skill + val installedApps = config.relatedApps.filter { registry.isAppInstalled(it.packageName) } + if (installedApps.isNotEmpty()) { + append("- ID: ${config.id}\n") + append(" 名称: ${config.name}\n") + append(" 描述: ${config.description}\n") + append(" 关键词: ${config.keywords.joinToString(", ")}\n") + append(" 可用应用: ${installedApps.joinToString(", ") { it.name }}\n\n") + } + } + } + + val prompt = """你是一个意图识别助手。根据用户输入，判断最匹配的技能。 + +$skillsInfo + +用户输入: "$query" + +请分析用户意图，返回 JSON 格式： +{ + "skill_id": "匹配的技能ID，如果没有匹配返回 null", + "confidence": 0.0-1.0 的置信度, + "reasoning": "简短的匹配理由" +} + +注意： +1. 只返回 JSON，不要有其他文字 +2. 如果用户意图明确匹配某个技能，即使措辞不同也要识别 +3. 如果确实没有匹配的技能，skill_id 返回 null +4. 例如"点个汉堡"、"帮我点外卖"、"想吃炸鸡" 都应该匹配 order_food +5. "附近好吃的"、"推荐美食" 应该匹配 find_food""" + + return try { + val result = client.predict(prompt) + result.getOrNull()?.let { response -> + parseIntentResponse(response) + } + } catch (e: Exception) { + println("[SkillManager] LLM 意图匹配失败: ${e.message}") + null + } + } + + /** + * 解析 LLM 返回的意图匹配结果 + */ + private fun parseIntentResponse(response: String): LLMIntentMatch? { + return try { + // 提取 JSON（可能被 markdown 包裹） + val jsonStr = response + .replace("```json", "") + .replace("```", "") + .trim() + + val json = JSONObject(jsonStr) + val skillId = json.optString("skill_id", null)?.takeIf { it != "null" && it.isNotEmpty() } + val confidence = json.optDouble("confidence", 0.0).toFloat() + val reasoning = json.optString("reasoning", "") + + if (skillId != null) { + LLMIntentMatch( + skillId = skillId, + confidence = confidence, + reasoning = reasoning + ) + } else { + null + } + } catch (e: Exception) { + println("[SkillManager] 解析意图响应失败: ${e.message}") + null + } + } + + /** + * 使用 LLM 匹配意图并返回可用应用（组合方法） + */ + suspend fun matchAvailableAppWithLLM(query: String): AvailableAppMatch? { + // 先尝试 LLM 匹配 + val llmMatch = matchIntentWithLLM(query) + + if (llmMatch != null && llmMatch.confidence >= 0.5f) { + println("[SkillManager] LLM 匹配: ${llmMatch.skillId} (置信度: ${llmMatch.confidence})") + println("[SkillManager] 理由: ${llmMatch.reasoning}") + + // 获取对应的 Skill 和已安装应用 + val skill = registry.get(llmMatch.skillId) + if (skill != null) { + println("[SkillManager] 找到 Skill: ${skill.config.name}") + println("[SkillManager] 关联应用: ${skill.config.relatedApps.map { "${it.name}(${it.packageName})" }}") + + // 检查每个应用的安装状态 + for (app in skill.config.relatedApps) { + val installed = registry.isAppInstalled(app.packageName) + println("[SkillManager] ${app.name}(${app.packageName}): ${if (installed) "已安装" else "未安装"}") + } + + val availableApp = skill.config.relatedApps + .filter { registry.isAppInstalled(it.packageName) } + .maxByOrNull { it.priority } + + if (availableApp != null) { + println("[SkillManager] 选中应用: ${availableApp.name}") + val params = skill.extractParams(query) + return AvailableAppMatch( + skill = skill, + app = availableApp, + params = params, + score = llmMatch.confidence + ) + } else { + println("[SkillManager] 没有可用应用（都未安装）") + } + } else { + println("[SkillManager] 未找到 Skill: ${llmMatch.skillId}") + } + } + + // 如果 LLM 匹配失败，回退到关键词匹配 + println("[SkillManager] LLM 未匹配或无可用应用，回退到关键词匹配") + return matchAvailableApp(query) + } + + /** + * 生成给 Agent 的上下文提示（使用 LLM 匹配） + */ + suspend fun generateAgentContextWithLLM(query: String): String { + // 使用 LLM 匹配 + val match = matchAvailableAppWithLLM(query) + + if (match == null) { + return "未找到相关技能或可用应用，请使用通用 GUI 自动化完成任务。" + } + + return buildString { + val config = match.skill.config + val app = match.app + + append("根据用户意图，已匹配到技能：\n\n") + append("【${config.name}】(置信度: ${(match.score * 100).toInt()}%)\n") + append("描述: ${config.description}\n\n") + + // 显示提示词约束（如小红书100字限制） + if (!config.promptHint.isNullOrBlank()) { + append("⚠️ 重要提示: ${config.promptHint}\n\n") + } + + val typeLabel = when (app.type) { + ExecutionType.DELEGATION -> "🚀委托(快速)" + ExecutionType.GUI_AUTOMATION -> "🤖GUI自动化" + } + + append("推荐应用: ${app.name} $typeLabel\n") + + if (app.type == ExecutionType.DELEGATION && app.deepLink != null) { + append("DeepLink: ${app.deepLink}\n") + } + + if (!app.steps.isNullOrEmpty()) { + append("操作步骤: ${app.steps.joinToString(" → ")}\n") + } + + app.description?.let { + append("说明: $it\n") + } + + append("\n建议：") + if (app.type == ExecutionType.DELEGATION) { + append("使用 DeepLink 直接打开 ${app.name}，可快速完成任务。") + } else { + append("通过 GUI 自动化操作 ${app.name} 完成任务。") + } + } + } + + /** + * 执行 Skill（核心执行方法） + * + * @param match 可用应用匹配结果 + * @return 执行结果 + */ + suspend fun execute(match: AvailableAppMatch): SkillResult { + val skill = match.skill + val app = match.app + val params = match.params + + println("[SkillManager] 执行: ${skill.config.name} -> ${app.name} (${app.type})") + + return when (app.type) { + ExecutionType.DELEGATION -> { + // 委托模式：通过 DeepLink 打开 + executeDelegation(skill, app, params) + } + ExecutionType.GUI_AUTOMATION -> { + // GUI 自动化模式：返回执行计划 + executeAutomation(skill, app, params) + } + } + } + + /** + * 执行委托（DeepLink） + */ + private fun executeDelegation( + skill: Skill, + app: RelatedApp, + params: Map + ): SkillResult { + val deepLink = skill.generateDeepLink(app, params) + + if (deepLink.isEmpty()) { + return SkillResult.Failed( + error = "无法生成 DeepLink", + suggestion = "尝试使用 GUI 自动化方式" + ) + } + + return try { + val intent = Intent(Intent.ACTION_VIEW, Uri.parse(deepLink)).apply { + addFlags(Intent.FLAG_ACTIVITY_NEW_TASK) + // 明确指定目标包名，避免系统选择其他能响应此 scheme 的应用 + setPackage(app.packageName) + } + context.startActivity(intent) + + SkillResult.Delegated( + app = app, + deepLink = deepLink, + message = "已打开 ${app.name}" + ) + } catch (e: Exception) { + // 如果指定包名失败，尝试不指定包名的方式 + println("[SkillManager] 指定包名打开失败，尝试通用方式: ${e.message}") + try { + val fallbackIntent = Intent(Intent.ACTION_VIEW, Uri.parse(deepLink)).apply { + addFlags(Intent.FLAG_ACTIVITY_NEW_TASK) + } + context.startActivity(fallbackIntent) + + SkillResult.Delegated( + app = app, + deepLink = deepLink, + message = "已打开 ${app.name}（通用方式）" + ) + } catch (e2: Exception) { + SkillResult.Failed( + error = "打开 ${app.name} 失败: ${e2.message}", + suggestion = "请确认应用已安装并支持 DeepLink" + ) + } + } + } + + /** + * 执行 GUI 自动化（返回执行计划给 Agent） + */ + private fun executeAutomation( + skill: Skill, + app: RelatedApp, + params: Map + ): SkillResult { + val plan = ExecutionPlan( + skillId = skill.config.id, + skillName = skill.config.name, + app = app, + params = params, + isInstalled = true, + promptHint = skill.config.promptHint + ) + + return SkillResult.NeedAutomation( + plan = plan, + message = "需要通过 GUI 自动化操作 ${app.name}" + ) + } + + /** + * 判断是否应该使用快速路径 + * + * 条件： + * 1. 高置信度匹配 (score >= 0.8) + * 2. 最佳应用是委托类型 (delegation) + * 3. 应用已安装 + */ + fun shouldUseFastPath(query: String): AvailableAppMatch? { + val match = matchAvailableApp(query) ?: return null + + // 只有委托类型且高置信度才走快速路径 + if (match.app.type == ExecutionType.DELEGATION && match.score >= 0.8f) { + return match + } + + return null + } + + /** + * 生成给 Agent 的上下文提示 + * + * 包含：匹配的意图、可用应用列表、推荐操作步骤 + */ + fun generateAgentContext(query: String): String { + val matches = matchAllAvailableApps(query) + + if (matches.isEmpty()) { + return "未找到相关技能或可用应用，请使用通用 GUI 自动化完成任务。" + } + + return buildString { + append("根据用户意图，匹配到以下可用方案：\n\n") + + // 按 Skill 分组 + val groupedBySkill = matches.groupBy { it.skill.config.id } + + for ((_, skillMatches) in groupedBySkill) { + val firstMatch = skillMatches.first() + val config = firstMatch.skill.config + + append("【${config.name}】(置信度: ${(firstMatch.score * 100).toInt()}%)\n") + + for ((index, match) in skillMatches.withIndex()) { + val app = match.app + val typeLabel = when (app.type) { + ExecutionType.DELEGATION -> "🚀委托(快速)" + ExecutionType.GUI_AUTOMATION -> "🤖GUI自动化" + } + + append(" ${index + 1}. ${app.name} $typeLabel (优先级: ${app.priority})\n") + + if (app.type == ExecutionType.DELEGATION && app.deepLink != null) { + append(" DeepLink: ${app.deepLink}\n") + } + + if (!app.steps.isNullOrEmpty()) { + append(" 步骤: ${app.steps.joinToString(" → ")}\n") + } + + app.description?.let { + append(" 说明: $it\n") + } + } + append("\n") + } + + append("建议：优先使用委托模式(🚀)，速度更快。如果委托失败再使用 GUI 自动化(🤖)。") + } + } + + /** + * 获取 Skill 信息 + */ + fun getSkillInfo(skillId: String): SkillConfig? { + return registry.get(skillId)?.config + } + + /** + * 获取所有 Skills 描述（给 LLM） + */ + fun getSkillsDescription(): String { + return registry.getSkillsDescription() + } + + /** + * 获取所有 Skills + */ + fun getAllSkills(): List { + return registry.getAll() + } + + /** + * 按分类获取 Skills + */ + fun getSkillsByCategory(category: String): List { + return registry.getByCategory(category) + } + + /** + * 检查意图是否有可用应用 + */ + fun hasAvailableApp(query: String): Boolean { + return matchAvailableApp(query) != null + } + + /** + * 获取意图的所有关联应用（不管是否安装） + */ + fun getAllRelatedApps(query: String): List { + val skillMatch = registry.matchBest(query) ?: return emptyList() + return skillMatch.skill.config.relatedApps + } + + /** + * 获取缺失的应用推荐（用户没装但可以装的） + */ + fun getMissingAppSuggestions(query: String): List { + val skillMatch = registry.matchBest(query) ?: return emptyList() + return skillMatch.skill.config.relatedApps + .filter { !registry.isAppInstalled(it.packageName) } + .sortedByDescending { it.priority } + } + + companion object { + @Volatile + private var instance: SkillManager? = null + + fun init(context: Context, toolManager: ToolManager, appScanner: AppScanner): SkillManager { + return instance ?: synchronized(this) { + instance ?: SkillManager(context.applicationContext, toolManager, appScanner).also { + it.initialize() + instance = it + } + } + } + + fun getInstance(): SkillManager { + return instance ?: throw IllegalStateException("SkillManager 未初始化，请先调用 init()") + } + + fun isInitialized(): Boolean = instance != null + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/skills/SkillRegistry.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/skills/SkillRegistry.kt new file mode 100644 index 0000000..9e336ef --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/skills/SkillRegistry.kt @@ -0,0 +1,341 @@ +package com.roubao.autopilot.skills + +import android.content.Context +import com.roubao.autopilot.controller.AppScanner +import org.json.JSONArray +import org.json.JSONObject +import java.io.IOException + +/** + * Skill 注册表 + * + * 管理所有 Skills 的注册、查找和匹配 + * 核心功能： + * - 从 skills.json 加载意图定义 + * - 查询本地已安装 App，筛选可用应用 + * - 根据优先级选择最佳执行方案 + */ +class SkillRegistry private constructor( + private val context: Context, + private val appScanner: AppScanner +) { + + private val skills = mutableMapOf() + private val categoryIndex = mutableMapOf>() + + // 缓存已安装 App 的包名集合（启动时刷新） + private var installedPackages: Set = emptySet() + + /** + * 初始化：刷新已安装应用列表 + */ + fun refreshInstalledApps() { + val apps = appScanner.getApps() + installedPackages = apps.map { it.packageName }.toSet() + println("[SkillRegistry] 已缓存 ${installedPackages.size} 个已安装应用") + + // 调试：检查美团相关的应用 + val meituanApps = installedPackages.filter { it.contains("meituan") || it.contains("dianping") } + println("[SkillRegistry] 美团相关应用: $meituanApps") + + // 检查小美的 DeepLink 是否可用（间接检测安装状态） + try { + val pm = context.packageManager + val intent = android.content.Intent(android.content.Intent.ACTION_VIEW).apply { + data = android.net.Uri.parse("beam://www.meituan.com/home") + } + val resolveInfo = pm.resolveActivity(intent, 0) + if (resolveInfo != null) { + val pkgName = resolveInfo.activityInfo.packageName + println("[SkillRegistry] 小美 DeepLink 可用，包名: $pkgName") + if (!installedPackages.contains(pkgName)) { + installedPackages = installedPackages + pkgName + println("[SkillRegistry] 添加 $pkgName 到已安装列表") + } + } else { + println("[SkillRegistry] 小美 DeepLink 不可用") + } + } catch (e: Exception) { + println("[SkillRegistry] 检查小美失败: ${e.message}") + } + } + + /** + * 检查包名是否已安装 + */ + fun isAppInstalled(packageName: String): Boolean { + return installedPackages.contains(packageName) + } + + /** + * 从 assets/skills.json 加载 Skills + */ + fun loadFromAssets(filename: String = "skills.json"): Int { + try { + val jsonString = context.assets.open(filename).bufferedReader().use { it.readText() } + return loadFromJson(jsonString) + } catch (e: IOException) { + println("[SkillRegistry] 无法加载 $filename: ${e.message}") + return 0 + } + } + + /** + * 从 JSON 字符串加载 Skills + */ + fun loadFromJson(jsonString: String): Int { + var loadedCount = 0 + try { + val jsonArray = JSONArray(jsonString) + for (i in 0 until jsonArray.length()) { + val obj = jsonArray.getJSONObject(i) + val config = parseSkillConfig(obj) + register(Skill(config)) + loadedCount++ + } + println("[SkillRegistry] 已加载 $loadedCount 个 Skills") + } catch (e: Exception) { + println("[SkillRegistry] JSON 解析错误: ${e.message}") + e.printStackTrace() + } + return loadedCount + } + + /** + * 解析单个 Skill 配置（新结构） + */ + private fun parseSkillConfig(obj: JSONObject): SkillConfig { + // 解析参数 + val params = mutableListOf() + val paramsArray = obj.optJSONArray("params") + if (paramsArray != null) { + for (i in 0 until paramsArray.length()) { + val paramObj = paramsArray.getJSONObject(i) + val examples = mutableListOf() + val examplesArray = paramObj.optJSONArray("examples") + if (examplesArray != null) { + for (j in 0 until examplesArray.length()) { + examples.add(examplesArray.getString(j)) + } + } + params.add(SkillParam( + name = paramObj.getString("name"), + type = paramObj.optString("type", "string"), + description = paramObj.optString("description", ""), + required = paramObj.optBoolean("required", false), + defaultValue = paramObj.opt("default"), + examples = examples + )) + } + } + + // 解析关键词 + val keywords = mutableListOf() + val keywordsArray = obj.optJSONArray("keywords") + if (keywordsArray != null) { + for (i in 0 until keywordsArray.length()) { + keywords.add(keywordsArray.getString(i)) + } + } + + // 解析关联应用列表（新结构） + val relatedApps = mutableListOf() + val appsArray = obj.optJSONArray("related_apps") + if (appsArray != null) { + for (i in 0 until appsArray.length()) { + val appObj = appsArray.getJSONObject(i) + + // 解析执行类型 + val typeStr = appObj.optString("type", "gui_automation") + val type = when (typeStr.lowercase()) { + "delegation" -> ExecutionType.DELEGATION + else -> ExecutionType.GUI_AUTOMATION + } + + // 解析操作步骤 + val steps = mutableListOf() + val stepsArray = appObj.optJSONArray("steps") + if (stepsArray != null) { + for (j in 0 until stepsArray.length()) { + steps.add(stepsArray.getString(j)) + } + } + + relatedApps.add(RelatedApp( + packageName = appObj.getString("package"), + name = appObj.getString("name"), + type = type, + deepLink = appObj.optString("deep_link", null)?.takeIf { it.isNotEmpty() }, + steps = if (steps.isEmpty()) null else steps, + priority = appObj.optInt("priority", 0), + description = appObj.optString("description", null)?.takeIf { it.isNotEmpty() } + )) + } + } + + return SkillConfig( + id = obj.getString("id"), + name = obj.getString("name"), + description = obj.optString("description", ""), + category = obj.optString("category", "通用"), + keywords = keywords, + params = params, + relatedApps = relatedApps, + promptHint = obj.optString("prompt_hint", null)?.takeIf { it.isNotEmpty() } + ) + } + + /** + * 注册 Skill + */ + fun register(skill: Skill) { + skills[skill.config.id] = skill + + // 更新分类索引 + val category = skill.config.category + categoryIndex.getOrPut(category) { mutableListOf() }.add(skill) + + println("[SkillRegistry] 注册 Skill: ${skill.config.id} (${skill.config.relatedApps.size} 关联应用)") + } + + /** + * 获取 Skill + */ + fun get(id: String): Skill? = skills[id] + + /** + * 获取所有 Skills + */ + fun getAll(): List = skills.values.toList() + + /** + * 按分类获取 Skills + */ + fun getByCategory(category: String): List { + return categoryIndex[category] ?: emptyList() + } + + /** + * 获取所有分类 + */ + fun getAllCategories(): List = categoryIndex.keys.toList() + + /** + * 匹配用户意图（基于关键词） + */ + fun match(query: String, topK: Int = 3, minScore: Float = 0.3f): List { + val matches = mutableListOf() + + for (skill in skills.values) { + val score = skill.matchScore(query) + if (score >= minScore) { + val params = skill.extractParams(query) + matches.add(SkillMatch(skill, score, params)) + } + } + + return matches + .sortedByDescending { it.score } + .take(topK) + } + + /** + * 获取最佳匹配 + */ + fun matchBest(query: String, minScore: Float = 0.3f): SkillMatch? { + return match(query, topK = 1, minScore = minScore).firstOrNull() + } + + /** + * 匹配意图并返回可用应用（核心方法） + * + * 1. 匹配用户意图到 Skill + * 2. 筛选出已安装的关联应用 + * 3. 按优先级排序 + */ + fun matchAvailableApps( + query: String, + minScore: Float = 0.3f + ): List { + val skillMatches = match(query, topK = 5, minScore = minScore) + val results = mutableListOf() + + for (skillMatch in skillMatches) { + val skill = skillMatch.skill + val params = skillMatch.params + + // 筛选已安装的应用，按优先级排序 + val availableApps = skill.config.relatedApps + .filter { isAppInstalled(it.packageName) } + .sortedByDescending { it.priority } + + for (app in availableApps) { + results.add(AvailableAppMatch( + skill = skill, + app = app, + params = params, + score = skillMatch.score + )) + } + } + + // 按 (匹配分数 * 0.5 + 应用优先级 * 0.01) 综合排序 + return results.sortedByDescending { it.score * 0.5f + it.app.priority * 0.01f } + } + + /** + * 获取意图的最佳可用应用 + */ + fun getBestAvailableApp(query: String, minScore: Float = 0.3f): AvailableAppMatch? { + return matchAvailableApps(query, minScore).firstOrNull() + } + + /** + * 生成 Skills 描述（给 LLM） + */ + fun getSkillsDescription(): String { + return buildString { + append("可用技能列表：\n\n") + for ((category, categorySkills) in categoryIndex) { + append("【$category】\n") + for (skill in categorySkills) { + val config = skill.config + append("- ${config.name}: ${config.description}\n") + if (config.keywords.isNotEmpty()) { + append(" 关键词: ${config.keywords.joinToString(", ")}\n") + } + // 显示已安装的应用 + val installedApps = config.relatedApps.filter { isAppInstalled(it.packageName) } + if (installedApps.isNotEmpty()) { + val appNames = installedApps.map { + val typeIcon = if (it.type == ExecutionType.DELEGATION) "🚀" else "🤖" + "$typeIcon${it.name}" + } + append(" 可用应用: ${appNames.joinToString(", ")}\n") + } + } + append("\n") + } + } + } + + companion object { + @Volatile + private var instance: SkillRegistry? = null + + fun init(context: Context, appScanner: AppScanner): SkillRegistry { + return instance ?: synchronized(this) { + instance ?: SkillRegistry(context.applicationContext, appScanner).also { + it.refreshInstalledApps() + instance = it + } + } + } + + fun getInstance(): SkillRegistry { + return instance ?: throw IllegalStateException("SkillRegistry 未初始化，请先调用 init()") + } + + fun isInitialized(): Boolean = instance != null + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ClipboardTool.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ClipboardTool.kt new file mode 100644 index 0000000..52cbe4d --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ClipboardTool.kt @@ -0,0 +1,152 @@ +package com.roubao.autopilot.tools + +import android.content.ClipData +import android.content.ClipboardManager +import android.content.Context +import android.os.Handler +import android.os.Looper +import kotlinx.coroutines.suspendCancellableCoroutine +import kotlin.coroutines.resume + +/** + * 剪贴板工具 + * + * 提供剪贴板的读写功能 + */ +class ClipboardTool(private val context: Context) : Tool { + + override val name = "clipboard" + override val displayName = "剪贴板" + override val description = "读取或写入系统剪贴板内容" + + override val params = listOf( + ToolParam( + name = "action", + type = "string", + description = "操作类型：read（读取）或 write（写入）", + required = true + ), + ToolParam( + name = "text", + type = "string", + description = "要写入的文本（action=write 时必填）", + required = false + ), + ToolParam( + name = "label", + type = "string", + description = "剪贴板标签（可选）", + required = false, + defaultValue = "roubao" + ) + ) + + private val clipboardManager: ClipboardManager by lazy { + context.getSystemService(Context.CLIPBOARD_SERVICE) as ClipboardManager + } + + private val mainHandler = Handler(Looper.getMainLooper()) + + override suspend fun execute(params: Map): ToolResult { + val action = params["action"] as? String + ?: return ToolResult.Error("缺少 action 参数") + + return when (action.lowercase()) { + "read" -> readClipboard() + "write" -> { + val text = params["text"] as? String + ?: return ToolResult.Error("write 操作需要 text 参数") + val label = params["label"] as? String ?: "roubao" + writeClipboard(text, label) + } + else -> ToolResult.Error("不支持的操作: $action（只支持 read/write）") + } + } + + /** + * 读取剪贴板内容 + */ + private suspend fun readClipboard(): ToolResult = suspendCancellableCoroutine { cont -> + mainHandler.post { + try { + val clip = clipboardManager.primaryClip + if (clip == null || clip.itemCount == 0) { + cont.resume(ToolResult.Success(data = "", message = "剪贴板为空")) + } else { + val text = clip.getItemAt(0).coerceToText(context).toString() + cont.resume(ToolResult.Success( + data = text, + message = "已读取剪贴板内容（${text.length} 字符）" + )) + } + } catch (e: Exception) { + cont.resume(ToolResult.Error("读取剪贴板失败: ${e.message}")) + } + } + } + + /** + * 写入剪贴板内容 + */ + private suspend fun writeClipboard(text: String, label: String): ToolResult = suspendCancellableCoroutine { cont -> + mainHandler.post { + try { + val clip = ClipData.newPlainText(label, text) + clipboardManager.setPrimaryClip(clip) + cont.resume(ToolResult.Success( + data = text, + message = "已写入剪贴板（${text.length} 字符）" + )) + } catch (e: Exception) { + cont.resume(ToolResult.Error("写入剪贴板失败: ${e.message}")) + } + } + } + + /** + * 同步读取（用于非协程环境） + */ + fun readSync(): String? { + var result: String? = null + val latch = java.util.concurrent.CountDownLatch(1) + + mainHandler.post { + try { + val clip = clipboardManager.primaryClip + if (clip != null && clip.itemCount > 0) { + result = clip.getItemAt(0).coerceToText(context).toString() + } + } catch (e: Exception) { + e.printStackTrace() + } finally { + latch.countDown() + } + } + + latch.await(1, java.util.concurrent.TimeUnit.SECONDS) + return result + } + + /** + * 同步写入（用于非协程环境） + */ + fun writeSync(text: String, label: String = "roubao"): Boolean { + var success = false + val latch = java.util.concurrent.CountDownLatch(1) + + mainHandler.post { + try { + val clip = ClipData.newPlainText(label, text) + clipboardManager.setPrimaryClip(clip) + success = true + } catch (e: Exception) { + e.printStackTrace() + } finally { + latch.countDown() + } + } + + latch.await(1, java.util.concurrent.TimeUnit.SECONDS) + return success + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/tools/DeepLinkTool.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/DeepLinkTool.kt new file mode 100644 index 0000000..d18037d --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/DeepLinkTool.kt @@ -0,0 +1,108 @@ +package com.roubao.autopilot.tools + +import com.roubao.autopilot.controller.DeviceController + +/** + * DeepLink 工具 + * + * 通过 Intent 打开应用的特定页面或功能 + * 这是实现 delegation 类型 Skill 的核心工具 + */ +class DeepLinkTool(private val deviceController: DeviceController) : Tool { + + override val name = "deep_link" + override val displayName = "深度链接" + override val description = "通过 DeepLink/Intent 打开应用的特定页面或功能" + + override val params = listOf( + ToolParam( + name = "uri", + type = "string", + description = "DeepLink URI（如：weixin://、alipays://、amap://）", + required = true + ), + ToolParam( + name = "action", + type = "string", + description = "Intent Action（默认 VIEW）", + required = false, + defaultValue = "android.intent.action.VIEW" + ) + ) + + override suspend fun execute(params: Map): ToolResult { + val uri = params["uri"] as? String + ?: return ToolResult.Error("缺少 uri 参数") + + return try { + deviceController.openDeepLink(uri) + ToolResult.Success( + data = mapOf("uri" to uri), + message = "已打开: $uri" + ) + } catch (e: Exception) { + ToolResult.Error("打开 DeepLink 失败: ${e.message}") + } + } + + companion object { + /** + * 常用 DeepLink 模板 + */ + val TEMPLATES = mapOf( + // ========== 外卖类 ========== + "meituan_food" to "imeituan://www.meituan.com/waimai/home", + "meituan_search" to "imeituan://www.meituan.com/search?q={query}", + "eleme_home" to "eleme://search", + "xiaomei_chat" to "xiaomei://chat?message={query}", // 小美 AI 对话 + + // ========== 出行类 ========== + "amap_route" to "amap://route/plan?sourceApplication=roubao&dlat={lat}&dlon={lon}&dname={destination}&dev=0&t=0", + "amap_navi" to "amap://navi?sourceApplication=roubao&lat={lat}&lon={lon}&dev=0", + "amap_search" to "amap://search?keyword={query}&sourceApplication=roubao", + "didi_call" to "diditaxi://", + "baidu_map_route" to "baidumap://map/direction?destination={destination}&mode=driving&src=roubao", + + // ========== 社交类 ========== + "weixin_scan" to "weixin://scanqrcode", + "weixin_pay" to "weixin://pay", + "alipay_scan" to "alipays://platformapi/startapp?appId=10000007", + "alipay_pay" to "alipays://platformapi/startapp?appId=20000056", + + // ========== 支付类 ========== + "alipay_transfer" to "alipays://platformapi/startapp?appId=20000200&actionType=toAccount&account={account}", + + // ========== 音乐类 ========== + "netease_play" to "orpheus://song/{id}", + "netease_search" to "orpheus://search?keyword={query}", + "qqmusic_search" to "qqmusic://qq.com/ui/search?key={query}", + + // ========== 视频类 ========== + "douyin_search" to "snssdk1128://search?keyword={query}", + "bilibili_search" to "bilibili://search?keyword={query}", + "bilibili_video" to "bilibili://video/{bvid}", + + // ========== AI 类（delegation 目标）========== + "doubao_chat" to "doubao://chat?message={query}", + "tongyi_chat" to "tongyi://chat?message={query}", + + // ========== 通用 ========== + "web" to "{url}", + "tel" to "tel:{phone}", + "sms" to "sms:{phone}?body={message}", + "email" to "mailto:{email}?subject={subject}&body={body}" + ) + + /** + * 根据模板生成 DeepLink + */ + fun fromTemplate(templateName: String, params: Map): String? { + val template = TEMPLATES[templateName] ?: return null + var result = template + for ((key, value) in params) { + result = result.replace("{$key}", value) + } + return result + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/tools/HttpTool.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/HttpTool.kt new file mode 100644 index 0000000..f412821 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/HttpTool.kt @@ -0,0 +1,164 @@ +package com.roubao.autopilot.tools + +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext +import org.json.JSONObject +import java.io.BufferedReader +import java.io.InputStreamReader +import java.io.OutputStreamWriter +import java.net.HttpURLConnection +import java.net.URL + +/** + * HTTP 请求工具 + * + * 用于调用外部 API，如： + * - 调用 AI 服务 + * - 获取天气信息 + * - 查询数据 + */ +class HttpTool : Tool { + + override val name = "http_request" + override val displayName = "HTTP 请求" + override val description = "发送 HTTP 请求调用外部 API" + + override val params = listOf( + ToolParam( + name = "url", + type = "string", + description = "请求 URL", + required = true + ), + ToolParam( + name = "method", + type = "string", + description = "HTTP 方法（GET/POST/PUT/DELETE）", + required = false, + defaultValue = "GET" + ), + ToolParam( + name = "headers", + type = "object", + description = "请求头（JSON 格式）", + required = false + ), + ToolParam( + name = "body", + type = "string", + description = "请求体（POST/PUT 时使用）", + required = false + ), + ToolParam( + name = "timeout", + type = "int", + description = "超时时间（毫秒）", + required = false, + defaultValue = 30000 + ) + ) + + override suspend fun execute(params: Map): ToolResult = withContext(Dispatchers.IO) { + val urlStr = params["url"] as? String + ?: return@withContext ToolResult.Error("缺少 url 参数") + + val method = (params["method"] as? String)?.uppercase() ?: "GET" + val timeout = (params["timeout"] as? Number)?.toInt() ?: 30000 + val body = params["body"] as? String + + @Suppress("UNCHECKED_CAST") + val headers = params["headers"] as? Map ?: emptyMap() + + try { + val url = URL(urlStr) + val connection = url.openConnection() as HttpURLConnection + + connection.requestMethod = method + connection.connectTimeout = timeout + connection.readTimeout = timeout + connection.doInput = true + + // 设置请求头 + headers.forEach { (key, value) -> + connection.setRequestProperty(key, value) + } + + // 默认 Content-Type + if (body != null && !headers.containsKey("Content-Type")) { + connection.setRequestProperty("Content-Type", "application/json") + } + + // 发送请求体 + if (body != null && (method == "POST" || method == "PUT")) { + connection.doOutput = true + OutputStreamWriter(connection.outputStream).use { writer -> + writer.write(body) + writer.flush() + } + } + + // 读取响应 + val responseCode = connection.responseCode + val inputStream = if (responseCode >= 400) { + connection.errorStream + } else { + connection.inputStream + } + + val response = BufferedReader(InputStreamReader(inputStream)).use { reader -> + reader.readText() + } + + connection.disconnect() + + if (responseCode >= 400) { + return@withContext ToolResult.Error( + "HTTP $responseCode: $response", + code = responseCode + ) + } + + ToolResult.Success( + data = mapOf( + "status_code" to responseCode, + "body" to response + ), + message = "请求成功 (HTTP $responseCode)" + ) + } catch (e: Exception) { + ToolResult.Error("请求失败: ${e.message}") + } + } + + /** + * 简化的 GET 请求 + */ + suspend fun get(url: String, headers: Map = emptyMap()): ToolResult { + return execute(mapOf( + "url" to url, + "method" to "GET", + "headers" to headers + )) + } + + /** + * 简化的 POST 请求 + */ + suspend fun post(url: String, body: String, headers: Map = emptyMap()): ToolResult { + return execute(mapOf( + "url" to url, + "method" to "POST", + "body" to body, + "headers" to headers + )) + } + + /** + * POST JSON 请求 + */ + suspend fun postJson(url: String, json: JSONObject, headers: Map = emptyMap()): ToolResult { + val allHeaders = headers.toMutableMap() + allHeaders["Content-Type"] = "application/json" + return post(url, json.toString(), allHeaders) + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/tools/OpenAppTool.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/OpenAppTool.kt new file mode 100644 index 0000000..bc44ff1 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/OpenAppTool.kt @@ -0,0 +1,56 @@ +package com.roubao.autopilot.tools + +import com.roubao.autopilot.controller.AppScanner +import com.roubao.autopilot.controller.DeviceController + +/** + * 打开应用工具 + * + * 支持： + * - 通过包名打开 + * - 通过应用名打开（自动搜索包名） + */ +class OpenAppTool( + private val deviceController: DeviceController, + private val appScanner: AppScanner +) : Tool { + + override val name = "open_app" + override val displayName = "打开应用" + override val description = "打开指定的应用程序" + + override val params = listOf( + ToolParam( + name = "app", + type = "string", + description = "应用名称或包名（如：微信、com.tencent.mm）", + required = true + ) + ) + + override suspend fun execute(params: Map): ToolResult { + val app = params["app"] as? String + ?: return ToolResult.Error("缺少 app 参数") + + // 判断是包名还是应用名 + val packageName = if (app.contains(".")) { + // 已经是包名格式 + app + } else { + // 需要搜索包名 + val results = appScanner.searchApps(app, topK = 1) + results.firstOrNull()?.app?.packageName + ?: return ToolResult.Error("未找到应用: $app") + } + + return try { + deviceController.openApp(packageName) + ToolResult.Success( + data = mapOf("package_name" to packageName), + message = "已打开应用: $app" + ) + } catch (e: Exception) { + ToolResult.Error("打开应用失败: ${e.message}") + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/tools/SearchAppsTool.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/SearchAppsTool.kt new file mode 100644 index 0000000..1ad854b --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/SearchAppsTool.kt @@ -0,0 +1,83 @@ +package com.roubao.autopilot.tools + +import com.roubao.autopilot.controller.AppScanner + +/** + * 搜索应用工具 + * + * 支持： + * - 应用名搜索 + * - 拼音搜索 + * - 语义搜索（如"点外卖"会匹配外卖类应用） + * - 分类搜索 + */ +class SearchAppsTool(private val appScanner: AppScanner) : Tool { + + override val name = "search_apps" + override val displayName = "搜索应用" + override val description = "在已安装应用中搜索，支持应用名、拼音、语义描述等多种方式" + + override val params = listOf( + ToolParam( + name = "query", + type = "string", + description = "搜索词（应用名/拼音/描述，如：微信、weixin、聊天）", + required = true + ), + ToolParam( + name = "top_k", + type = "int", + description = "返回结果数量", + required = false, + defaultValue = 5 + ), + ToolParam( + name = "include_system", + type = "boolean", + description = "是否包含系统应用", + required = false, + defaultValue = true + ) + ) + + override suspend fun execute(params: Map): ToolResult { + val query = params["query"] as? String + ?: return ToolResult.Error("缺少 query 参数") + + val topK = (params["top_k"] as? Number)?.toInt() ?: 5 + val includeSystem = params["include_system"] as? Boolean ?: true + + val results = appScanner.searchApps(query, topK, includeSystem) + + if (results.isEmpty()) { + return ToolResult.Success( + data = emptyList>(), + message = "未找到匹配\"$query\"的应用" + ) + } + + val data = results.map { result -> + mapOf( + "package_name" to result.app.packageName, + "app_name" to result.app.appName, + "category" to (result.app.category ?: ""), + "score" to result.score, + "match_type" to result.matchType, + "is_system" to result.app.isSystem + ) + } + + return ToolResult.Success( + data = data, + message = "找到 ${results.size} 个匹配的应用" + ) + } + + /** + * 快捷方法：直接获取最佳匹配的包名 + */ + fun findBestMatch(query: String): String? { + val results = appScanner.searchApps(query, topK = 1) + return results.firstOrNull()?.app?.packageName + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ShellTool.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ShellTool.kt new file mode 100644 index 0000000..2cb493e --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ShellTool.kt @@ -0,0 +1,150 @@ +package com.roubao.autopilot.tools + +import com.roubao.autopilot.controller.DeviceController +import com.roubao.autopilot.data.SettingsManager + +/** + * Shell 命令工具 + * + * 通过 Shizuku 执行 shell 命令 + * 注意：这是一个底层工具，主要供其他工具或高级场景使用 + */ +class ShellTool( + private val deviceController: DeviceController, + private val settingsManager: SettingsManager? = null +) : Tool { + + override val name = "shell" + override val displayName = "Shell 命令" + override val description = "执行 shell 命令（需要 Shizuku 权限）" + + override val params = listOf( + ToolParam( + name = "command", + type = "string", + description = "要执行的 shell 命令", + required = true + ) + ) + + // 安全白名单：允许执行的命令前缀 + private val ALLOWED_PREFIXES = listOf( + "input ", // 输入操作 + "am ", // Activity Manager + "pm ", // Package Manager + "wm ", // Window Manager + "screencap ", // 截图 + "monkey ", // 启动应用 + "dumpsys ", // 系统信息 + "getprop ", // 系统属性 + "settings ", // 系统设置 + "content ", // Content Provider + "cmd ", // 通用命令 + "ls ", // 文件列表 + "cat ", // 读文件 + "echo " // 输出 + ) + + // 基础黑名单：始终禁止的危险命令 + private val BASE_BLOCKED_COMMANDS = listOf( + "rm -rf", + "rm -r /", + "format", + "mkfs", + "dd if=", + "reboot", + "shutdown", + "> /dev", + "chmod 777 /" + ) + + // su -c 命令（需要特殊权限才能使用） + private val SU_COMMAND = "su -c" + + /** + * 获取当前生效的黑名单 + * 根据设置决定是否允许 su -c + */ + private fun getBlockedCommands(): List { + val settings = settingsManager?.settings?.value + val suEnabled = settings?.rootModeEnabled == true && settings.suCommandEnabled == true + return if (suEnabled) { + BASE_BLOCKED_COMMANDS // su -c 已启用，不在黑名单中 + } else { + BASE_BLOCKED_COMMANDS + SU_COMMAND // su -c 禁用 + } + } + + override suspend fun execute(params: Map): ToolResult { + val command = params["command"] as? String + ?: return ToolResult.Error("缺少 command 参数") + + // 安全检查 + val securityCheck = checkSecurity(command) + if (securityCheck != null) { + return ToolResult.Error(securityCheck) + } + + // 由于 DeviceController.exec 是 private，这里需要通过已有的公开方法 + // 或者扩展 DeviceController + // 暂时只支持特定的安全命令 + return try { + val result = executeCommand(command) + ToolResult.Success( + data = result, + message = "命令执行完成" + ) + } catch (e: Exception) { + ToolResult.Error("执行失败: ${e.message}") + } + } + + /** + * 安全检查 + */ + private fun checkSecurity(command: String): String? { + val lowerCmd = command.lowercase().trim() + val blockedCommands = getBlockedCommands() + + // 检查黑名单 + for (blocked in blockedCommands) { + if (lowerCmd.contains(blocked.lowercase())) { + // 特殊提示 su -c 命令 + if (blocked == SU_COMMAND) { + return "安全限制：su -c 命令需要在设置中开启「Root 模式」和「允许 su -c」" + } + return "安全限制：禁止执行此类命令" + } + } + + // 检查白名单（可选，如果需要更严格的控制可以启用） + // val isAllowed = ALLOWED_PREFIXES.any { lowerCmd.startsWith(it.lowercase()) } + // if (!isAllowed) { + // return "安全限制：命令不在允许列表中" + // } + + return null + } + + /** + * 执行命令 + * 注意：这里需要扩展 DeviceController 或使用反射 + * 暂时使用简化实现 + */ + private fun executeCommand(command: String): String { + return try { + val process = Runtime.getRuntime().exec(arrayOf("sh", "-c", command)) + val output = process.inputStream.bufferedReader().readText() + val error = process.errorStream.bufferedReader().readText() + process.waitFor() + + if (error.isNotBlank()) { + "Output: $output\nError: $error" + } else { + output + } + } catch (e: Exception) { + "Error: ${e.message}" + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/tools/Tool.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/Tool.kt new file mode 100644 index 0000000..af81b3d --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/Tool.kt @@ -0,0 +1,144 @@ +package com.roubao.autopilot.tools + +import org.json.JSONObject + +/** + * Tool 执行结果 + */ +sealed class ToolResult { + data class Success(val data: Any?, val message: String = "") : ToolResult() + data class Error(val error: String, val code: Int = -1) : ToolResult() + + val isSuccess: Boolean get() = this is Success + + fun getDataOrNull(): Any? = (this as? Success)?.data + + fun toJson(): JSONObject = JSONObject().apply { + when (this@ToolResult) { + is Success -> { + put("success", true) + put("data", data?.toString() ?: "") + put("message", message) + } + is Error -> { + put("success", false) + put("error", error) + put("code", code) + } + } + } +} + +/** + * Tool 参数定义 + */ +data class ToolParam( + val name: String, + val type: String, // string, int, boolean, list + val description: String, + val required: Boolean = true, + val defaultValue: Any? = null +) + +/** + * Tool 接口 - 所有工具的基类 + * + * Tool 是原子能力层，提供单一、可复用的功能 + * - 每个 Tool 完成一个独立的操作 + * - Tool 之间可以组合使用 + * - Tool 的输入输出是结构化的 + */ +interface Tool { + /** 工具名称（唯一标识） */ + val name: String + + /** 工具显示名称 */ + val displayName: String + + /** 工具描述 */ + val description: String + + /** 参数定义 */ + val params: List + + /** + * 执行工具 + * @param params 参数 Map + * @return 执行结果 + */ + suspend fun execute(params: Map): ToolResult + + /** + * 验证参数 + */ + fun validateParams(params: Map): List { + val errors = mutableListOf() + for (param in this.params) { + if (param.required && !params.containsKey(param.name)) { + errors.add("缺少必填参数: ${param.name}") + } + } + return errors + } + + /** + * 生成给 LLM 的工具描述 + */ + fun toLLMDescription(): String { + val paramsDesc = params.joinToString("\n") { p -> + val required = if (p.required) "(必填)" else "(可选)" + " - ${p.name}: ${p.type} $required - ${p.description}" + } + return """ + |工具: $name + |说明: $description + |参数: + |$paramsDesc + """.trimMargin() + } +} + +/** + * Tool 注册表 - 管理所有可用工具 + */ +object ToolRegistry { + private val tools = mutableMapOf() + + fun register(tool: Tool) { + tools[tool.name] = tool + println("[ToolRegistry] 注册工具: ${tool.name}") + } + + fun get(name: String): Tool? = tools[name] + + fun getAll(): List = tools.values.toList() + + fun getNames(): List = tools.keys.toList() + + fun contains(name: String): Boolean = tools.containsKey(name) + + /** + * 生成所有工具的描述（给 LLM） + */ + fun getAllDescriptions(): String { + return tools.values.joinToString("\n\n") { it.toLLMDescription() } + } + + /** + * 根据名称执行工具 + */ + suspend fun execute(toolName: String, params: Map): ToolResult { + val tool = tools[toolName] ?: return ToolResult.Error("未找到工具: $toolName") + + val errors = tool.validateParams(params) + if (errors.isNotEmpty()) { + return ToolResult.Error("参数错误: ${errors.joinToString(", ")}") + } + + return try { + tool.execute(params) + } catch (e: Exception) { + ToolResult.Error("执行失败: ${e.message}") + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ToolManager.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ToolManager.kt new file mode 100644 index 0000000..2bb02a8 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/tools/ToolManager.kt @@ -0,0 +1,109 @@ +package com.roubao.autopilot.tools + +import android.content.Context +import com.roubao.autopilot.controller.AppScanner +import com.roubao.autopilot.controller.DeviceController + +/** + * 工具管理器 + * + * 负责初始化、注册和管理所有 Tools + * 作为 Tool 层的统一入口 + */ +class ToolManager private constructor( + private val context: Context, + private val deviceController: DeviceController, + private val appScanner: AppScanner +) { + + // 持有各个工具的引用（方便直接调用） + lateinit var searchAppsTool: SearchAppsTool + private set + lateinit var openAppTool: OpenAppTool + private set + lateinit var clipboardTool: ClipboardTool + private set + lateinit var deepLinkTool: DeepLinkTool + private set + lateinit var shellTool: ShellTool + private set + lateinit var httpTool: HttpTool + private set + + /** + * 初始化所有工具 + */ + private fun initialize() { + // 创建工具实例 + searchAppsTool = SearchAppsTool(appScanner) + openAppTool = OpenAppTool(deviceController, appScanner) + clipboardTool = ClipboardTool(context) + deepLinkTool = DeepLinkTool(deviceController) + shellTool = ShellTool(deviceController) + httpTool = HttpTool() + + // 注册到全局 Registry + ToolRegistry.register(searchAppsTool) + ToolRegistry.register(openAppTool) + ToolRegistry.register(clipboardTool) + ToolRegistry.register(deepLinkTool) + ToolRegistry.register(shellTool) + ToolRegistry.register(httpTool) + + println("[ToolManager] 已初始化 ${ToolRegistry.getAll().size} 个工具") + } + + /** + * 执行工具 + */ + suspend fun execute(toolName: String, params: Map): ToolResult { + return ToolRegistry.execute(toolName, params) + } + + /** + * 获取所有工具描述（给 LLM） + */ + fun getToolDescriptions(): String { + return ToolRegistry.getAllDescriptions() + } + + /** + * 获取可用工具列表 + */ + fun getAvailableTools(): List { + return ToolRegistry.getAll() + } + + companion object { + @Volatile + private var instance: ToolManager? = null + + /** + * 初始化单例 + */ + fun init( + context: Context, + deviceController: DeviceController, + appScanner: AppScanner + ): ToolManager { + return instance ?: synchronized(this) { + instance ?: ToolManager(context, deviceController, appScanner).also { + it.initialize() + instance = it + } + } + } + + /** + * 获取单例 + */ + fun getInstance(): ToolManager { + return instance ?: throw IllegalStateException("ToolManager 未初始化，请先调用 init()") + } + + /** + * 检查是否已初始化 + */ + fun isInitialized(): Boolean = instance != null + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/ui/OverlayService.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/OverlayService.kt new file mode 100644 index 0000000..9ee23ea --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/OverlayService.kt @@ -0,0 +1,501 @@ +package com.roubao.autopilot.ui + +import android.animation.ValueAnimator +import android.annotation.SuppressLint +import android.app.NotificationChannel +import android.app.NotificationManager +import android.app.PendingIntent +import android.app.Service +import android.content.Context +import android.content.Intent +import android.graphics.* +import android.graphics.drawable.GradientDrawable +import android.os.Build +import android.os.IBinder +import android.view.Gravity +import android.view.MotionEvent +import android.view.View +import android.view.WindowManager +import android.widget.LinearLayout +import android.widget.TextView +import androidx.core.app.NotificationCompat +import androidx.core.content.ContextCompat +import com.roubao.autopilot.MainActivity +import com.roubao.autopilot.R + +/** + * 七彩悬浮窗服务 - 显示当前执行步骤 + * 放在屏幕顶部状态栏下方，不影响截图识别 + */ +class OverlayService : Service() { + + private var windowManager: WindowManager? = null + private var overlayView: View? = null + private var textView: TextView? = null + private var actionButton: TextView? = null + private var cancelButton: TextView? = null // 确认模式下的取消按钮 + private var divider: View? = null + private var divider2: View? = null // 确认模式下第二个分隔线 + private var animator: ValueAnimator? = null + + companion object { + private var instance: OverlayService? = null + private var stopCallback: (() -> Unit)? = null + private var continueCallback: (() -> Unit)? = null + private var confirmCallback: ((Boolean) -> Unit)? = null // 敏感操作确认回调 + private var isTakeOverMode = false + private var isConfirmMode = false // 敏感操作确认模式 + + // 等待 instance 回调队列 + private val pendingCallbacks = mutableListOf<() -> Unit>() + + fun show(context: Context, text: String, onStop: (() -> Unit)? = null) { + stopCallback = onStop + isTakeOverMode = false + isConfirmMode = false + instance?.updateText(text) ?: run { + val intent = Intent(context, OverlayService::class.java).apply { + putExtra("text", text) + } + ContextCompat.startForegroundService(context, intent) + } + instance?.setNormalMode() + } + + fun hide(context: Context) { + stopCallback = null + continueCallback = null + confirmCallback = null + isTakeOverMode = false + isConfirmMode = false + pendingCallbacks.clear() + // 只有当 service 已经启动完成时才停止它 + // 否则会导致 ForegroundServiceDidNotStartInTimeException + if (instance != null) { + context.stopService(Intent(context, OverlayService::class.java)) + } + } + + fun update(text: String) { + instance?.updateText(text) + } + + /** 截图时临时隐藏悬浮窗 */ + fun setVisible(visible: Boolean) { + instance?.overlayView?.post { + instance?.overlayView?.visibility = if (visible) View.VISIBLE else View.INVISIBLE + } + } + + /** 显示人机协作模式 - 等待用户手动完成操作 */ + fun showTakeOver(message: String, onContinue: () -> Unit) { + val action: () -> Unit = { + println("[OverlayService] showTakeOver: $message") + continueCallback = onContinue + isTakeOverMode = true + isConfirmMode = false + instance?.setTakeOverMode(message) + Unit + } + + if (instance != null) { + action() + } else { + // 悬浮窗尚未启动，加入等待队列 + println("[OverlayService] showTakeOver: instance is null, queuing...") + pendingCallbacks.add(action) + } + } + + /** 显示敏感操作确认模式 - 用户确认或取消 */ + fun showConfirm(message: String, onConfirm: (Boolean) -> Unit) { + val action: () -> Unit = { + println("[OverlayService] showConfirm: $message") + confirmCallback = onConfirm + isConfirmMode = true + isTakeOverMode = false + instance?.setConfirmMode(message) + Unit + } + + if (instance != null) { + action() + } else { + // 悬浮窗尚未启动，加入等待队列 + println("[OverlayService] showConfirm: instance is null, queuing...") + pendingCallbacks.add(action) + } + } + + /** 当 instance 可用时执行等待中的回调 */ + private fun processPendingCallbacks() { + println("[OverlayService] processPendingCallbacks: ${pendingCallbacks.size} pending") + pendingCallbacks.forEach { it.invoke() } + pendingCallbacks.clear() + } + } + + override fun onBind(intent: Intent?): IBinder? = null + + override fun onCreate() { + super.onCreate() + instance = this + windowManager = getSystemService(WINDOW_SERVICE) as WindowManager + + // 必须第一时间调用 startForeground，否则会崩溃 + startForegroundNotification() + + // 创建悬浮窗（可能因权限问题失败） + try { + createOverlayView() + } catch (e: Exception) { + println("[OverlayService] createOverlayView failed: ${e.message}") + } + + // 处理在 service 启动前排队的回调 + processPendingCallbacks() + } + + private fun startForegroundNotification() { + val channelId = "baozi_overlay" + val channelName = "肉包状态" + + try { + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) { + val channel = NotificationChannel( + channelId, + channelName, + NotificationManager.IMPORTANCE_LOW + ).apply { + description = "显示肉包执行状态" + setShowBadge(false) + } + val notificationManager = getSystemService(NotificationManager::class.java) + notificationManager.createNotificationChannel(channel) + } + + val pendingIntent = PendingIntent.getActivity( + this, + 0, + Intent(this, MainActivity::class.java), + PendingIntent.FLAG_IMMUTABLE + ) + + val notification = NotificationCompat.Builder(this, channelId) + .setContentTitle("肉包运行中") + .setContentText("正在执行自动化任务...") + .setSmallIcon(R.mipmap.ic_launcher) + .setContentIntent(pendingIntent) + .setOngoing(true) + .setPriority(NotificationCompat.PRIORITY_LOW) + .build() + + startForeground(1001, notification) + } catch (e: Exception) { + println("[OverlayService] startForegroundNotification error: ${e.message}") + // 降级：使用最简单的通知确保 startForeground 被调用 + try { + val fallbackNotification = NotificationCompat.Builder(this, channelId) + .setContentTitle("肉包") + .setSmallIcon(R.mipmap.ic_launcher) + .build() + startForeground(1001, fallbackNotification) + } catch (e2: Exception) { + println("[OverlayService] fallback startForeground also failed: ${e2.message}") + } + } + } + + override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int { + val text = intent?.getStringExtra("text") ?: "AutoPilot" + updateText(text) + return START_STICKY + } + + override fun onDestroy() { + super.onDestroy() + instance = null + animator?.cancel() + overlayView?.let { windowManager?.removeView(it) } + } + + @SuppressLint("ClickableViewAccessibility") + private fun createOverlayView() { + // 容器 + val container = LinearLayout(this).apply { + orientation = LinearLayout.HORIZONTAL + gravity = Gravity.CENTER_VERTICAL + setPadding(16, 12, 16, 12) + } + + // 七彩渐变背景 + val gradientDrawable = GradientDrawable().apply { + shape = GradientDrawable.RECTANGLE + cornerRadius = 30f + setStroke(2, Color.WHITE) + } + container.background = gradientDrawable + + // 状态文字 + textView = TextView(this).apply { + text = "肉包" + textSize = 13f + setTextColor(Color.WHITE) + gravity = Gravity.CENTER + setPadding(16, 4, 16, 4) + setShadowLayer(4f, 0f, 0f, Color.BLACK) + typeface = Typeface.DEFAULT_BOLD + } + container.addView(textView) + + // 分隔线 + divider = View(this).apply { + setBackgroundColor(Color.WHITE) + alpha = 0.5f + } + val dividerParams = LinearLayout.LayoutParams(2, 36).apply { + setMargins(12, 0, 12, 0) + } + container.addView(divider, dividerParams) + + // 动作按钮（停止/继续/确认） + actionButton = TextView(this).apply { + text = "⏹ 停止" + textSize = 13f + setTextColor(Color.WHITE) + gravity = Gravity.CENTER + setPadding(16, 4, 16, 4) + setShadowLayer(4f, 0f, 0f, Color.BLACK) + typeface = Typeface.DEFAULT_BOLD + setOnClickListener { + when { + isConfirmMode -> { + // 确认模式：点击确认 + confirmCallback?.invoke(true) + confirmCallback = null + isConfirmMode = false + setNormalMode() + } + isTakeOverMode -> { + // 人机协作模式：点击继续 + continueCallback?.invoke() + continueCallback = null + isTakeOverMode = false + setNormalMode() + } + else -> { + // 正常模式：点击停止 + stopCallback?.invoke() + hide(this@OverlayService) + } + } + } + } + container.addView(actionButton) + + // 第二个分隔线（确认模式用） + divider2 = View(this).apply { + setBackgroundColor(Color.WHITE) + alpha = 0.5f + visibility = View.GONE + } + val divider2Params = LinearLayout.LayoutParams(2, 36).apply { + setMargins(12, 0, 12, 0) + } + container.addView(divider2, divider2Params) + + // 取消按钮（确认模式用） + cancelButton = TextView(this).apply { + text = "❌ 取消" + textSize = 13f + setTextColor(Color.parseColor("#FF6B6B")) // 红色 + gravity = Gravity.CENTER + setPadding(16, 4, 16, 4) + setShadowLayer(4f, 0f, 0f, Color.BLACK) + typeface = Typeface.DEFAULT_BOLD + visibility = View.GONE + setOnClickListener { + if (isConfirmMode) { + confirmCallback?.invoke(false) + confirmCallback = null + isConfirmMode = false + setNormalMode() + } + } + } + container.addView(cancelButton) + + // 动画：七彩渐变流动效果 + startRainbowAnimation(gradientDrawable) + + val params = WindowManager.LayoutParams().apply { + type = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) { + WindowManager.LayoutParams.TYPE_APPLICATION_OVERLAY + } else { + @Suppress("DEPRECATION") + WindowManager.LayoutParams.TYPE_PHONE + } + flags = WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or + WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL or + WindowManager.LayoutParams.FLAG_LAYOUT_IN_SCREEN or + WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON // 保持屏幕常亮 + format = PixelFormat.TRANSLUCENT + width = WindowManager.LayoutParams.WRAP_CONTENT + height = WindowManager.LayoutParams.WRAP_CONTENT + gravity = Gravity.TOP or Gravity.START + x = 100 + y = 200 + } + + // 添加拖动功能（只拦截文字区域，不影响按钮点击） + var initialX = 0 + var initialY = 0 + var initialTouchX = 0f + var initialTouchY = 0f + var isDragging = false + val dragThreshold = 20f // 增大阈值，避免误触 + + // 只在文字区域启用拖动，按钮区域不拦截 + textView?.setOnTouchListener { _, event -> + when (event.action) { + MotionEvent.ACTION_DOWN -> { + initialX = params.x + initialY = params.y + initialTouchX = event.rawX + initialTouchY = event.rawY + isDragging = false + true + } + MotionEvent.ACTION_MOVE -> { + val deltaX = event.rawX - initialTouchX + val deltaY = event.rawY - initialTouchY + if (kotlin.math.abs(deltaX) > dragThreshold || kotlin.math.abs(deltaY) > dragThreshold) { + isDragging = true + } + if (isDragging) { + params.x = initialX + deltaX.toInt() + params.y = initialY + deltaY.toInt() + windowManager?.updateViewLayout(container, params) + } + true + } + MotionEvent.ACTION_UP -> { + isDragging + } + else -> false + } + } + + overlayView = container + windowManager?.addView(overlayView, params) + } + + private fun startRainbowAnimation(drawable: GradientDrawable) { + val colors = intArrayOf( + Color.parseColor("#FF6B6B"), // 红 + Color.parseColor("#FFA94D"), // 橙 + Color.parseColor("#FFE066"), // 黄 + Color.parseColor("#69DB7C"), // 绿 + Color.parseColor("#4DABF7"), // 蓝 + Color.parseColor("#9775FA"), // 紫 + Color.parseColor("#F783AC"), // 粉 + Color.parseColor("#FF6B6B") // 回到红 + ) + + animator = ValueAnimator.ofFloat(0f, 1f).apply { + duration = 3000 + repeatCount = ValueAnimator.INFINITE + repeatMode = ValueAnimator.RESTART + + addUpdateListener { animation -> + val fraction = animation.animatedValue as Float + val index = (fraction * (colors.size - 1)).toInt() + val nextIndex = minOf(index + 1, colors.size - 1) + val localFraction = (fraction * (colors.size - 1)) - index + + val color1 = interpolateColor(colors[index], colors[nextIndex], localFraction) + val color2 = interpolateColor( + colors[(index + 2) % colors.size], + colors[(nextIndex + 2) % colors.size], + localFraction + ) + val color3 = interpolateColor( + colors[(index + 4) % colors.size], + colors[(nextIndex + 4) % colors.size], + localFraction + ) + + drawable.colors = intArrayOf(color1, color2, color3) + drawable.orientation = GradientDrawable.Orientation.LEFT_RIGHT + } + start() + } + } + + private fun interpolateColor(startColor: Int, endColor: Int, fraction: Float): Int { + val startA = Color.alpha(startColor) + val startR = Color.red(startColor) + val startG = Color.green(startColor) + val startB = Color.blue(startColor) + + val endA = Color.alpha(endColor) + val endR = Color.red(endColor) + val endG = Color.green(endColor) + val endB = Color.blue(endColor) + + return Color.argb( + (startA + (endA - startA) * fraction).toInt(), + (startR + (endR - startR) * fraction).toInt(), + (startG + (endG - startG) * fraction).toInt(), + (startB + (endB - startB) * fraction).toInt() + ) + } + + private fun updateText(text: String) { + textView?.post { + textView?.text = text + } + } + + /** 切换到人机协作模式 */ + private fun setTakeOverMode(message: String) { + println("[OverlayService] setTakeOverMode: $message") + overlayView?.post { + // 确保悬浮窗可见 + overlayView?.visibility = View.VISIBLE + textView?.text = "🖐 $message" + actionButton?.text = "✅ 继续" + actionButton?.setTextColor(Color.parseColor("#90EE90")) // 浅绿色 + // 隐藏取消按钮（人机协作只有继续按钮） + divider2?.visibility = View.GONE + cancelButton?.visibility = View.GONE + } + } + + /** 切换到正常模式 */ + private fun setNormalMode() { + println("[OverlayService] setNormalMode") + overlayView?.post { + actionButton?.text = "⏹ 停止" + actionButton?.setTextColor(Color.WHITE) + // 隐藏取消按钮和第二分隔线 + divider2?.visibility = View.GONE + cancelButton?.visibility = View.GONE + } + } + + /** 切换到敏感操作确认模式 */ + private fun setConfirmMode(message: String) { + println("[OverlayService] setConfirmMode: $message") + overlayView?.post { + // 确保悬浮窗可见 + overlayView?.visibility = View.VISIBLE + textView?.text = "⚠️ $message" + actionButton?.text = "✅ 确认" + actionButton?.setTextColor(Color.parseColor("#90EE90")) // 浅绿色 + // 显示取消按钮和第二分隔线 + divider2?.visibility = View.VISIBLE + cancelButton?.visibility = View.VISIBLE + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/CapabilitiesScreen.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/CapabilitiesScreen.kt new file mode 100644 index 0000000..eaa1f6b --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/CapabilitiesScreen.kt @@ -0,0 +1,452 @@ +package com.roubao.autopilot.ui.screens + +import androidx.compose.animation.* +import androidx.compose.foundation.background +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.* +import androidx.compose.foundation.lazy.LazyColumn +import androidx.compose.foundation.lazy.items +import androidx.compose.foundation.shape.CircleShape +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.KeyboardArrowDown +import androidx.compose.material.icons.filled.KeyboardArrowUp +import androidx.compose.material3.* +import androidx.compose.runtime.* +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.clip +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.style.TextOverflow +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp +import com.roubao.autopilot.tools.ToolManager +import com.roubao.autopilot.ui.theme.BaoziTheme + +/** + * 工具信息（用于展示） + */ +data class ToolInfo( + val name: String, + val description: String +) + +/** + * Agent 角色信息 + */ +data class AgentInfo( + val name: String, + val icon: String, + val role: String, + val description: String, + val responsibilities: List +) + +/** + * 预定义的 Agents 列表 + */ +val agentsList = listOf( + AgentInfo( + name = "Manager", + icon = "🎯", + role = "规划者", + description = "负责理解用户意图，制定高层次的执行计划，并跟踪任务进度。", + responsibilities = listOf( + "分析用户请求，理解真实意图", + "将复杂任务分解为可执行的子目标", + "制定执行计划和步骤顺序", + "根据执行反馈动态调整计划" + ) + ), + AgentInfo( + name = "Executor", + icon = "⚡", + role = "执行者", + description = "负责分析当前屏幕状态，决定具体的操作动作。", + responsibilities = listOf( + "分析屏幕截图，理解界面元素", + "根据计划选择下一步操作", + "确定点击、滑动、输入等具体动作", + "输出精确的操作坐标和参数" + ) + ), + AgentInfo( + name = "Reflector", + icon = "🔍", + role = "反思者", + description = "负责评估操作结果，判断动作是否成功执行。", + responsibilities = listOf( + "对比操作前后的屏幕变化", + "判断操作是否达到预期效果", + "识别异常情况（如弹窗、错误）", + "提供反馈帮助调整后续策略" + ) + ), + AgentInfo( + name = "Notetaker", + icon = "📝", + role = "记录者", + description = "负责记录执行过程中的关键信息，供其他 Agent 参考。", + responsibilities = listOf( + "记录任务执行的重要节点", + "保存中间结果和状态信息", + "为后续步骤提供上下文参考", + "生成执行摘要和日志" + ) + ) +) + +/** + * 能力展示页面 + * + * 展示 Agents 和 Tools（只读） + */ +@Composable +fun CapabilitiesScreen() { + val colors = BaoziTheme.colors + + // 获取 Tools + val tools = remember { + if (ToolManager.isInitialized()) { + ToolManager.getInstance().getAvailableTools().map { tool -> + ToolInfo(name = tool.name, description = tool.description) + } + } else { + emptyList() + } + } + + // 额外的内置工具（不在 ToolManager 中但是系统能力） + val builtInTools = listOf( + ToolInfo("screenshot", "截取当前屏幕，获取界面图像供 AI 分析"), + ToolInfo("tap", "点击屏幕指定坐标位置"), + ToolInfo("swipe", "在屏幕上滑动，支持上下左右方向"), + ToolInfo("type", "输入文本内容到当前焦点位置"), + ToolInfo("press_key", "按下系统按键（Home、Back、Enter 等）") + ) + + val allTools = tools + builtInTools + + // Tab 状态 + var selectedTab by remember { mutableStateOf(0) } + val tabs = listOf("Agents (${agentsList.size})", "Tools (${allTools.size})") + + Column( + modifier = Modifier + .fillMaxSize() + .background(colors.background) + ) { + // 顶部标题 + Box( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 20.dp, vertical = 16.dp) + ) { + Column { + Text( + text = "能力", + fontSize = 28.sp, + fontWeight = FontWeight.Bold, + color = colors.primary + ) + Text( + text = "${agentsList.size} 个 Agent，${allTools.size} 个工具", + fontSize = 14.sp, + color = colors.textSecondary + ) + } + } + + // Tab 切换 + TabRow( + selectedTabIndex = selectedTab, + containerColor = colors.background, + contentColor = colors.primary + ) { + tabs.forEachIndexed { index, title -> + Tab( + selected = selectedTab == index, + onClick = { selectedTab = index }, + text = { + Text( + text = title, + color = if (selectedTab == index) colors.primary else colors.textSecondary + ) + } + ) + } + } + + // 内容区域 + when (selectedTab) { + 0 -> AgentsListView() + 1 -> ToolsListView(tools = allTools) + } + } +} + +@Composable +fun AgentsListView() { + val colors = BaoziTheme.colors + + LazyColumn( + modifier = Modifier.fillMaxSize(), + contentPadding = PaddingValues(16.dp), + verticalArrangement = Arrangement.spacedBy(16.dp) + ) { + // 架构说明卡片 + item(key = "arch_intro") { + Card( + modifier = Modifier.fillMaxWidth(), + shape = RoundedCornerShape(16.dp), + colors = CardDefaults.cardColors(containerColor = colors.primary.copy(alpha = 0.1f)) + ) { + Column(modifier = Modifier.padding(16.dp)) { + Text( + text = "🧠 多 Agent 协作架构", + fontSize = 16.sp, + fontWeight = FontWeight.Bold, + color = colors.primary + ) + Spacer(modifier = Modifier.height(8.dp)) + Text( + text = "肉包采用多 Agent 协作架构，每个 Agent 专注于特定职责，通过协作完成复杂的手机自动化任务。", + fontSize = 13.sp, + color = colors.textSecondary + ) + Spacer(modifier = Modifier.height(8.dp)) + Text( + text = "Manager → Executor → Reflector → Notetaker", + fontSize = 12.sp, + fontWeight = FontWeight.Medium, + color = colors.textHint + ) + } + } + } + + // Agent 列表 + items(agentsList, key = { it.name }) { agent -> + AgentCard(agent = agent) + } + } +} + +@Composable +fun AgentCard(agent: AgentInfo) { + val colors = BaoziTheme.colors + var expanded by remember { mutableStateOf(false) } + + Card( + modifier = Modifier + .fillMaxWidth() + .clickable { expanded = !expanded }, + shape = RoundedCornerShape(16.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Column(modifier = Modifier.padding(16.dp)) { + Row( + modifier = Modifier.fillMaxWidth(), + verticalAlignment = Alignment.CenterVertically + ) { + // Agent 图标 + Box( + modifier = Modifier + .size(56.dp) + .clip(CircleShape) + .background(colors.primary.copy(alpha = 0.15f)), + contentAlignment = Alignment.Center + ) { + Text( + text = agent.icon, + fontSize = 28.sp + ) + } + + Spacer(modifier = Modifier.width(16.dp)) + + Column(modifier = Modifier.weight(1f)) { + Row(verticalAlignment = Alignment.CenterVertically) { + Text( + text = agent.name, + fontSize = 18.sp, + fontWeight = FontWeight.Bold, + color = colors.textPrimary + ) + Spacer(modifier = Modifier.width(8.dp)) + Box( + modifier = Modifier + .clip(RoundedCornerShape(8.dp)) + .background(colors.secondary.copy(alpha = 0.2f)) + .padding(horizontal = 8.dp, vertical = 2.dp) + ) { + Text( + text = agent.role, + fontSize = 11.sp, + color = colors.secondary + ) + } + } + Spacer(modifier = Modifier.height(4.dp)) + Text( + text = agent.description, + fontSize = 13.sp, + color = colors.textSecondary, + maxLines = if (expanded) Int.MAX_VALUE else 2, + overflow = TextOverflow.Ellipsis + ) + } + + Icon( + imageVector = if (expanded) Icons.Default.KeyboardArrowUp else Icons.Default.KeyboardArrowDown, + contentDescription = if (expanded) "收起" else "展开", + tint = colors.textHint + ) + } + + // 展开显示职责列表 + AnimatedVisibility( + visible = expanded, + enter = expandVertically() + fadeIn(), + exit = shrinkVertically() + fadeOut() + ) { + Column( + modifier = Modifier.padding(top = 16.dp) + ) { + Text( + text = "职责", + fontSize = 14.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary + ) + Spacer(modifier = Modifier.height(8.dp)) + + agent.responsibilities.forEach { responsibility -> + Row( + modifier = Modifier + .fillMaxWidth() + .padding(vertical = 4.dp), + verticalAlignment = Alignment.Top + ) { + Text( + text = "•", + fontSize = 14.sp, + color = colors.primary, + modifier = Modifier.padding(end = 8.dp) + ) + Text( + text = responsibility, + fontSize = 13.sp, + color = colors.textSecondary + ) + } + } + } + } + } + } +} + +@Composable +fun ToolsListView(tools: List) { + if (tools.isEmpty()) { + EmptyState(message = "暂无工具") + } else { + LazyColumn( + modifier = Modifier.fillMaxSize(), + contentPadding = PaddingValues(16.dp), + verticalArrangement = Arrangement.spacedBy(12.dp) + ) { + items(tools, key = { it.name }) { tool -> + ToolCard(tool = tool) + } + } + } +} + +@Composable +fun ToolCard(tool: ToolInfo) { + val colors = BaoziTheme.colors + + // 根据工具名获取图标 + val toolIcon = when (tool.name) { + "search_apps" -> "🔍" + "open_app" -> "📱" + "deep_link" -> "🔗" + "clipboard" -> "📋" + "shell" -> "💻" + "http" -> "🌐" + else -> "🔧" + } + + Card( + modifier = Modifier.fillMaxWidth(), + shape = RoundedCornerShape(16.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + verticalAlignment = Alignment.CenterVertically + ) { + // 工具图标 + Box( + modifier = Modifier + .size(48.dp) + .clip(CircleShape) + .background(colors.secondary.copy(alpha = 0.2f)), + contentAlignment = Alignment.Center + ) { + Text( + text = toolIcon, + fontSize = 20.sp + ) + } + + Spacer(modifier = Modifier.width(16.dp)) + + Column(modifier = Modifier.weight(1f)) { + Text( + text = tool.name, + fontSize = 16.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary + ) + Spacer(modifier = Modifier.height(4.dp)) + Text( + text = tool.description, + fontSize = 13.sp, + color = colors.textSecondary, + maxLines = 2, + overflow = TextOverflow.Ellipsis + ) + } + } + } +} + +@Composable +fun EmptyState(message: String) { + val colors = BaoziTheme.colors + + Box( + modifier = Modifier + .fillMaxSize() + .padding(32.dp), + contentAlignment = Alignment.Center + ) { + Column(horizontalAlignment = Alignment.CenterHorizontally) { + Text( + text = "📦", + fontSize = 64.sp + ) + Spacer(modifier = Modifier.height(16.dp)) + Text( + text = message, + fontSize = 16.sp, + color = colors.textSecondary + ) + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/HistoryScreen.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/HistoryScreen.kt new file mode 100644 index 0000000..1cf272e --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/HistoryScreen.kt @@ -0,0 +1,595 @@ +package com.roubao.autopilot.ui.screens + +import androidx.compose.animation.* +import androidx.compose.foundation.background +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.* +import androidx.compose.foundation.lazy.LazyColumn +import androidx.compose.foundation.lazy.items +import androidx.compose.foundation.shape.CircleShape +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.ArrowBack +import androidx.compose.material.icons.filled.Delete +import androidx.compose.material.icons.filled.CheckCircle +import androidx.compose.material.icons.filled.Warning +import androidx.compose.material.icons.filled.PlayArrow +import androidx.compose.material3.* +import androidx.compose.runtime.* +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.clip +import androidx.compose.ui.graphics.Brush +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.style.TextOverflow +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp +import com.roubao.autopilot.data.ExecutionRecord +import com.roubao.autopilot.data.ExecutionStatus +import com.roubao.autopilot.data.ExecutionStep +import com.roubao.autopilot.ui.theme.BaoziTheme +import com.roubao.autopilot.ui.theme.Primary +import com.roubao.autopilot.ui.theme.Secondary + +@Composable +fun HistoryScreen( + records: List, + onRecordClick: (ExecutionRecord) -> Unit, + onDeleteRecord: (String) -> Unit +) { + val colors = BaoziTheme.colors + Column( + modifier = Modifier + .fillMaxSize() + .background(colors.background) + ) { + // 顶部标题 + Box( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 20.dp, vertical = 16.dp) + ) { + Column { + Text( + text = "执行记录", + fontSize = 28.sp, + fontWeight = FontWeight.Bold, + color = colors.primary + ) + Text( + text = "共 ${records.size} 条记录", + fontSize = 14.sp, + color = colors.textSecondary + ) + } + } + + if (records.isEmpty()) { + // 空状态 + Box( + modifier = Modifier + .fillMaxSize() + .padding(32.dp), + contentAlignment = Alignment.Center + ) { + Column(horizontalAlignment = Alignment.CenterHorizontally) { + Text( + text = "📝", + fontSize = 64.sp + ) + Spacer(modifier = Modifier.height(16.dp)) + Text( + text = "暂无执行记录", + fontSize = 16.sp, + color = colors.textSecondary + ) + Text( + text = "执行任务后记录会显示在这里", + fontSize = 14.sp, + color = colors.textHint + ) + } + } + } else { + // 记录列表 + LazyColumn( + modifier = Modifier.fillMaxSize(), + contentPadding = PaddingValues(horizontal = 16.dp, vertical = 8.dp), + verticalArrangement = Arrangement.spacedBy(12.dp) + ) { + items( + items = records, + key = { it.id } + ) { record -> + HistoryRecordCard( + record = record, + onClick = { onRecordClick(record) }, + onDelete = { onDeleteRecord(record.id) } + ) + } + } + } + } +} + +@OptIn(ExperimentalMaterial3Api::class) +@Composable +fun HistoryRecordCard( + record: ExecutionRecord, + onClick: () -> Unit, + onDelete: () -> Unit +) { + val colors = BaoziTheme.colors + var showDeleteDialog by remember { mutableStateOf(false) } + + if (showDeleteDialog) { + AlertDialog( + onDismissRequest = { showDeleteDialog = false }, + containerColor = colors.backgroundCard, + title = { Text("删除记录", color = colors.textPrimary) }, + text = { Text("确定要删除这条执行记录吗？", color = colors.textSecondary) }, + confirmButton = { + TextButton(onClick = { + onDelete() + showDeleteDialog = false + }) { + Text("删除", color = colors.error) + } + }, + dismissButton = { + TextButton(onClick = { showDeleteDialog = false }) { + Text("取消", color = colors.textSecondary) + } + } + ) + } + + Card( + modifier = Modifier + .fillMaxWidth() + .clickable(onClick = onClick), + shape = RoundedCornerShape(16.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + verticalAlignment = Alignment.CenterVertically + ) { + // 状态图标 + Box( + modifier = Modifier + .size(48.dp) + .clip(CircleShape) + .background( + when (record.status) { + ExecutionStatus.COMPLETED -> colors.success.copy(alpha = 0.2f) + ExecutionStatus.FAILED -> colors.error.copy(alpha = 0.2f) + ExecutionStatus.STOPPED -> colors.warning.copy(alpha = 0.2f) + ExecutionStatus.RUNNING -> colors.primary.copy(alpha = 0.2f) + } + ), + contentAlignment = Alignment.Center + ) { + Icon( + imageVector = when (record.status) { + ExecutionStatus.COMPLETED -> Icons.Default.CheckCircle + ExecutionStatus.FAILED -> Icons.Default.Warning + ExecutionStatus.STOPPED -> Icons.Default.PlayArrow + ExecutionStatus.RUNNING -> Icons.Default.PlayArrow + }, + contentDescription = null, + tint = when (record.status) { + ExecutionStatus.COMPLETED -> colors.success + ExecutionStatus.FAILED -> colors.error + ExecutionStatus.STOPPED -> colors.warning + ExecutionStatus.RUNNING -> colors.primary + }, + modifier = Modifier.size(24.dp) + ) + } + + Spacer(modifier = Modifier.width(16.dp)) + + // 内容 + Column(modifier = Modifier.weight(1f)) { + Text( + text = record.title, + fontSize = 16.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary, + maxLines = 1, + overflow = TextOverflow.Ellipsis + ) + Spacer(modifier = Modifier.height(4.dp)) + Text( + text = record.instruction, + fontSize = 13.sp, + color = colors.textSecondary, + maxLines = 2, + overflow = TextOverflow.Ellipsis + ) + Spacer(modifier = Modifier.height(8.dp)) + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.spacedBy(8.dp), + verticalAlignment = Alignment.CenterVertically + ) { + // 状态标签 + val (statusText, statusColor) = when (record.status) { + ExecutionStatus.COMPLETED -> "已完成" to colors.success + ExecutionStatus.FAILED -> "失败" to colors.error + ExecutionStatus.STOPPED -> "已取消" to colors.warning + ExecutionStatus.RUNNING -> "执行中" to colors.primary + } + Text( + text = statusText, + fontSize = 11.sp, + fontWeight = FontWeight.Medium, + color = statusColor, + modifier = Modifier + .background( + statusColor.copy(alpha = 0.15f), + RoundedCornerShape(4.dp) + ) + .padding(horizontal = 6.dp, vertical = 2.dp) + ) + Text( + text = "·", + fontSize = 12.sp, + color = colors.textHint + ) + Text( + text = record.formattedStartTime, + fontSize = 12.sp, + color = colors.textHint, + maxLines = 1 + ) + Text( + text = "·", + fontSize = 12.sp, + color = colors.textHint + ) + Text( + text = "${record.steps.size}步", + fontSize = 12.sp, + color = colors.textHint, + maxLines = 1 + ) + Text( + text = "·", + fontSize = 12.sp, + color = colors.textHint + ) + Text( + text = record.formattedDuration, + fontSize = 12.sp, + color = colors.textHint, + maxLines = 1 + ) + } + } + + // 删除按钮 + IconButton( + onClick = { showDeleteDialog = true } + ) { + Icon( + imageVector = Icons.Default.Delete, + contentDescription = "删除", + tint = colors.textHint + ) + } + } + } +} + +@OptIn(ExperimentalMaterial3Api::class) +@Composable +fun HistoryDetailScreen( + record: ExecutionRecord, + onBack: () -> Unit +) { + val colors = BaoziTheme.colors + // Tab 状态：0 = 时间线，1 = 日志 + var selectedTab by remember { mutableStateOf(0) } + + Column( + modifier = Modifier + .fillMaxSize() + .background(colors.background) + ) { + // 顶部栏 + TopAppBar( + title = { + Column { + Text( + text = record.title, + fontSize = 18.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary + ) + Text( + text = record.formattedStartTime, + fontSize = 12.sp, + color = colors.textSecondary + ) + } + }, + navigationIcon = { + IconButton(onClick = onBack) { + Icon( + imageVector = Icons.Default.ArrowBack, + contentDescription = "返回", + tint = colors.textPrimary + ) + } + }, + colors = TopAppBarDefaults.topAppBarColors( + containerColor = colors.background + ) + ) + + // 任务信息卡片 + Card( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + shape = RoundedCornerShape(16.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Column(modifier = Modifier.padding(16.dp)) { + Text( + text = "任务指令", + fontSize = 12.sp, + color = colors.textHint + ) + Spacer(modifier = Modifier.height(4.dp)) + Text( + text = record.instruction, + fontSize = 15.sp, + color = colors.textPrimary + ) + Spacer(modifier = Modifier.height(12.dp)) + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.SpaceBetween + ) { + Column { + Text("状态", fontSize = 12.sp, color = colors.textHint) + Text( + text = when (record.status) { + ExecutionStatus.COMPLETED -> "已完成" + ExecutionStatus.FAILED -> "失败" + ExecutionStatus.STOPPED -> "已停止" + ExecutionStatus.RUNNING -> "执行中" + }, + fontSize = 14.sp, + color = when (record.status) { + ExecutionStatus.COMPLETED -> colors.success + ExecutionStatus.FAILED -> colors.error + ExecutionStatus.STOPPED -> colors.warning + ExecutionStatus.RUNNING -> colors.primary + } + ) + } + Column { + Text("步骤数", fontSize = 12.sp, color = colors.textHint) + Text("${record.steps.size}", fontSize = 14.sp, color = colors.textPrimary) + } + Column { + Text("耗时", fontSize = 12.sp, color = colors.textHint) + Text(record.formattedDuration, fontSize = 14.sp, color = colors.textPrimary) + } + } + } + } + + // Tab 切换 + Row( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp), + horizontalArrangement = Arrangement.spacedBy(8.dp) + ) { + // 时间线 Tab + Box( + modifier = Modifier + .weight(1f) + .clip(RoundedCornerShape(12.dp)) + .background( + if (selectedTab == 0) colors.primary + else colors.backgroundCard + ) + .clickable { selectedTab = 0 } + .padding(vertical = 12.dp), + contentAlignment = Alignment.Center + ) { + Text( + text = "执行时间线", + fontSize = 14.sp, + fontWeight = if (selectedTab == 0) FontWeight.Medium else FontWeight.Normal, + color = if (selectedTab == 0) Color.White else colors.textSecondary + ) + } + + // 日志 Tab + Box( + modifier = Modifier + .weight(1f) + .clip(RoundedCornerShape(12.dp)) + .background( + if (selectedTab == 1) colors.primary + else colors.backgroundCard + ) + .clickable { selectedTab = 1 } + .padding(vertical = 12.dp), + contentAlignment = Alignment.Center + ) { + Text( + text = "执行日志", + fontSize = 14.sp, + fontWeight = if (selectedTab == 1) FontWeight.Medium else FontWeight.Normal, + color = if (selectedTab == 1) Color.White else colors.textSecondary + ) + } + } + + Spacer(modifier = Modifier.height(8.dp)) + + // 内容区域 + when (selectedTab) { + 0 -> { + // 时间线列表 + if (record.steps.isEmpty()) { + Box( + modifier = Modifier + .fillMaxSize() + .padding(32.dp), + contentAlignment = Alignment.Center + ) { + Text( + text = "暂无执行步骤", + fontSize = 14.sp, + color = colors.textHint + ) + } + } else { + LazyColumn( + modifier = Modifier.fillMaxSize(), + contentPadding = PaddingValues(horizontal = 16.dp, vertical = 8.dp) + ) { + items(record.steps) { step -> + TimelineItem(step = step, isLast = step == record.steps.lastOrNull()) + } + } + } + } + 1 -> { + // 日志列表 + if (record.logs.isEmpty()) { + Box( + modifier = Modifier + .fillMaxSize() + .padding(32.dp), + contentAlignment = Alignment.Center + ) { + Text( + text = "暂无执行日志", + fontSize = 14.sp, + color = colors.textHint + ) + } + } else { + LazyColumn( + modifier = Modifier.fillMaxSize(), + contentPadding = PaddingValues(horizontal = 16.dp, vertical = 8.dp) + ) { + items(record.logs) { log -> + val logColor = when { + log.contains("❌") -> colors.error + log.contains("✅") -> colors.success + log.contains("📋") || log.contains("🎬") -> colors.secondary + log.contains("Step") || log.contains("=====") -> colors.primary + log.contains("⛔") -> colors.error + else -> colors.textSecondary + } + Text( + text = log, + fontSize = 12.sp, + color = logColor, + modifier = Modifier.padding(vertical = 2.dp) + ) + } + } + } + } + } + } +} + +@Composable +fun TimelineItem( + step: ExecutionStep, + isLast: Boolean +) { + val colors = BaoziTheme.colors + Row(modifier = Modifier.fillMaxWidth()) { + // 时间线指示器 + Column( + horizontalAlignment = Alignment.CenterHorizontally + ) { + // 圆点 + Box( + modifier = Modifier + .size(16.dp) + .clip(CircleShape) + .background( + when (step.outcome) { + "A" -> colors.success + "B" -> colors.warning + "?" -> colors.textHint // 进行中被取消 + else -> colors.error + } + ) + ) + // 连接线 + if (!isLast) { + Box( + modifier = Modifier + .width(2.dp) + .height(80.dp) + .background(colors.backgroundInput) + ) + } + } + + Spacer(modifier = Modifier.width(16.dp)) + + // 步骤内容 + Card( + modifier = Modifier + .fillMaxWidth() + .padding(bottom = if (isLast) 0.dp else 8.dp), + shape = RoundedCornerShape(12.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Column(modifier = Modifier.padding(12.dp)) { + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.SpaceBetween + ) { + Text( + text = "Step ${step.stepNumber}", + fontSize = 14.sp, + fontWeight = FontWeight.Medium, + color = colors.primary + ) + Text( + text = step.action, + fontSize = 12.sp, + color = colors.secondary + ) + } + Spacer(modifier = Modifier.height(8.dp)) + Text( + text = step.description, + fontSize = 13.sp, + color = colors.textPrimary + ) + if (step.thought.isNotBlank()) { + Spacer(modifier = Modifier.height(8.dp)) + Text( + text = step.thought, + fontSize = 12.sp, + color = colors.textSecondary, + maxLines = 3, + overflow = TextOverflow.Ellipsis + ) + } + } + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/HomeScreen.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/HomeScreen.kt new file mode 100644 index 0000000..ca81ab8 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/HomeScreen.kt @@ -0,0 +1,557 @@ +package com.roubao.autopilot.ui.screens + +import androidx.compose.animation.* +import androidx.compose.animation.core.* +import androidx.compose.foundation.background +import androidx.compose.foundation.border +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.* +import androidx.compose.foundation.layout.imePadding +import androidx.compose.foundation.lazy.LazyColumn +import androidx.compose.foundation.lazy.LazyRow +import androidx.compose.foundation.lazy.items +import androidx.compose.foundation.lazy.rememberLazyListState +import androidx.compose.foundation.shape.CircleShape +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.foundation.text.BasicTextField +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.Send +import androidx.compose.material.icons.filled.Close +import androidx.compose.material.icons.filled.Refresh +import androidx.compose.material3.* +import androidx.compose.runtime.* +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.clip +import androidx.compose.ui.graphics.Brush +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.SolidColor +import androidx.compose.ui.platform.LocalFocusManager +import androidx.compose.ui.platform.LocalSoftwareKeyboardController +import androidx.compose.ui.text.TextStyle +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp +import com.roubao.autopilot.agent.AgentState +import com.roubao.autopilot.ui.theme.BaoziTheme +import com.roubao.autopilot.ui.theme.Primary +import com.roubao.autopilot.ui.theme.Secondary + +/** + * 预设命令 + */ +data class PresetCommand( + val icon: String, + val title: String, + val command: String +) + +val presetCommands = listOf( + PresetCommand("🍔", "点汉堡", "帮我点个附近好吃的汉堡"), + PresetCommand("📕", "发小红书", "帮我发一条小红书，内容是今日份好心情"), + PresetCommand("📺", "刷B站", "打开B站搜索肉包，找到第一个视频点个赞"), + PresetCommand("✈️", "旅游攻略", "用小美帮我查一下三亚旅游攻略"), + PresetCommand("🎵", "听音乐", "打开网易云音乐播放每日推荐"), + PresetCommand("🛒", "点外卖", "帮我在美团点一份猪脚饭") +) + +@OptIn(androidx.compose.ui.ExperimentalComposeUiApi::class) +@Composable +fun HomeScreen( + agentState: AgentState?, + logs: List, + onExecute: (String) -> Unit, + onStop: () -> Unit, + shizukuAvailable: Boolean, + currentModel: String = "", + onRefreshShizuku: () -> Unit = {}, + onShizukuRequired: () -> Unit = {}, + isExecuting: Boolean = false +) { + val colors = BaoziTheme.colors + var inputText by remember { mutableStateOf("") } + // 使用 isExecuting 或 agentState?.isRunning 来判断是否运行中 + val isRunning = isExecuting || agentState?.isRunning == true + val listState = rememberLazyListState() + val keyboardController = LocalSoftwareKeyboardController.current + val focusManager = LocalFocusManager.current + + // 记录上一次的运行状态，用于检测任务结束 + var wasRunning by remember { mutableStateOf(false) } + + // 任务结束时清空输入框 + LaunchedEffect(isRunning) { + if (wasRunning && !isRunning) { + // 从运行中变为未运行，说明任务结束 + inputText = "" + } + wasRunning = isRunning + } + + // 自动滚动到底部 + LaunchedEffect(logs.size) { + if (logs.isNotEmpty()) { + listState.animateScrollToItem(logs.size - 1) + } + } + + Column( + modifier = Modifier + .fillMaxSize() + .background(colors.background) + .imePadding() + ) { + // 顶部标题 + Box( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 20.dp, vertical = 16.dp) + ) { + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.SpaceBetween, + verticalAlignment = Alignment.CenterVertically + ) { + Column { + Text( + text = "肉包", + fontSize = 28.sp, + fontWeight = FontWeight.Bold, + color = colors.primary + ) + Text( + text = if (shizukuAvailable) "准备就绪，告诉我你想做什么" else "请先连接 Shizuku", + fontSize = 14.sp, + color = if (shizukuAvailable) colors.textSecondary else colors.error + ) + } + + // 未连接时显示刷新按钮 + if (!shizukuAvailable) { + IconButton( + onClick = onRefreshShizuku, + modifier = Modifier + .clip(RoundedCornerShape(12.dp)) + .background(colors.backgroundCard) + ) { + Icon( + imageVector = Icons.Default.Refresh, + contentDescription = "刷新 Shizuku 状态", + tint = colors.primary + ) + } + } + } + } + + // 内容区域 + Box( + modifier = Modifier + .weight(1f) + .fillMaxWidth() + ) { + if (isRunning || logs.isNotEmpty()) { + // 执行中或有日志时显示日志 + ExecutionLogView( + logs = logs, + isRunning = isRunning, + currentStep = agentState?.currentStep ?: 0, + currentModel = currentModel, + listState = listState, + modifier = Modifier.fillMaxSize() + ) + } else { + // 空闲时显示预设命令 + PresetCommandsView( + onCommandClick = { command -> + if (shizukuAvailable) { + inputText = command + } else { + onShizukuRequired() + } + }, + modifier = Modifier.fillMaxSize() + ) + } + } + + // 底部输入区域 + InputArea( + inputText = inputText, + onInputChange = { inputText = it }, + onExecute = { + if (inputText.isNotBlank()) { + // 收起键盘并清除焦点 + keyboardController?.hide() + focusManager.clearFocus() + onExecute(inputText) + } + }, + onStop = { + // 停止任务并清空输入框 + inputText = "" + onStop() + }, + isRunning = isRunning, + enabled = shizukuAvailable, + onInputClick = { + if (!shizukuAvailable) { + onShizukuRequired() + } + } + ) + } +} + +@Composable +fun PresetCommandsView( + onCommandClick: (String) -> Unit, + modifier: Modifier = Modifier +) { + val colors = BaoziTheme.colors + Column( + modifier = modifier.padding(horizontal = 16.dp), + verticalArrangement = Arrangement.Center + ) { + Text( + text = "试试这些指令", + fontSize = 16.sp, + fontWeight = FontWeight.Medium, + color = colors.textSecondary, + modifier = Modifier.padding(bottom = 16.dp) + ) + + presetCommands.chunked(2).forEach { rowCommands -> + Row( + modifier = Modifier + .fillMaxWidth() + .padding(bottom = 12.dp), + horizontalArrangement = Arrangement.spacedBy(12.dp) + ) { + rowCommands.forEach { preset -> + PresetCommandCard( + preset = preset, + onClick = { onCommandClick(preset.command) }, + modifier = Modifier.weight(1f) + ) + } + // 如果是奇数，补一个空白 + if (rowCommands.size == 1) { + Spacer(modifier = Modifier.weight(1f)) + } + } + } + } +} + +@Composable +fun PresetCommandCard( + preset: PresetCommand, + onClick: () -> Unit, + modifier: Modifier = Modifier +) { + val colors = BaoziTheme.colors + Card( + modifier = modifier + .clickable(onClick = onClick), + shape = RoundedCornerShape(16.dp), + colors = CardDefaults.cardColors( + containerColor = colors.backgroundCard + ) + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + verticalAlignment = Alignment.CenterVertically + ) { + Text( + text = preset.icon, + fontSize = 24.sp + ) + Spacer(modifier = Modifier.width(12.dp)) + Column { + Text( + text = preset.title, + fontSize = 14.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary + ) + Text( + text = preset.command, + fontSize = 11.sp, + color = colors.textSecondary, + maxLines = 1 + ) + } + } + } +} + +@Composable +fun ExecutionLogView( + logs: List, + isRunning: Boolean, + currentStep: Int, + currentModel: String, + listState: androidx.compose.foundation.lazy.LazyListState, + modifier: Modifier = Modifier +) { + Column(modifier = modifier) { + // 执行状态指示器 + if (isRunning) { + ExecutingIndicator(currentStep = currentStep, currentModel = currentModel) + } + + // 日志列表 + LazyColumn( + state = listState, + modifier = Modifier + .fillMaxSize() + .padding(horizontal = 16.dp), + verticalArrangement = Arrangement.spacedBy(4.dp) + ) { + items(logs) { log -> + LogItem(log = log) + } + } + } +} + +@Composable +fun ExecutingIndicator(currentStep: Int, currentModel: String = "") { + val colors = BaoziTheme.colors + val infiniteTransition = rememberInfiniteTransition(label = "executing") + val animatedProgress by infiniteTransition.animateFloat( + initialValue = 0f, + targetValue = 1f, + animationSpec = infiniteRepeatable( + animation = tween(1500, easing = LinearEasing), + repeatMode = RepeatMode.Restart + ), + label = "progress" + ) + + Box( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 8.dp) + ) { + Card( + shape = RoundedCornerShape(12.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Column( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp) + ) { + Row( + modifier = Modifier.fillMaxWidth(), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.SpaceBetween + ) { + Row( + verticalAlignment = Alignment.CenterVertically + ) { + // 动画圆点 + Box( + modifier = Modifier + .size(12.dp) + .clip(CircleShape) + .background( + Brush.sweepGradient( + listOf(Primary, Secondary, Primary) + ) + ) + ) + Spacer(modifier = Modifier.width(12.dp)) + Text( + text = "正在执行 Step $currentStep", + fontSize = 14.sp, + fontWeight = FontWeight.Medium, + color = colors.primary + ) + } + // 显示当前模型 + if (currentModel.isNotEmpty()) { + Text( + text = currentModel, + fontSize = 11.sp, + color = colors.textHint, + maxLines = 1 + ) + } + } + + Spacer(modifier = Modifier.height(8.dp)) + + // 进度条 + Box( + modifier = Modifier + .fillMaxWidth() + .height(4.dp) + .clip(RoundedCornerShape(2.dp)) + .background(colors.backgroundInput) + ) { + Box( + modifier = Modifier + .fillMaxWidth(animatedProgress) + .fillMaxHeight() + .background( + Brush.horizontalGradient( + listOf(Primary, Secondary) + ) + ) + ) + } + } + } + } +} + +@Composable +fun LogItem(log: String) { + val colors = BaoziTheme.colors + val logColor = when { + log.contains("❌") -> colors.error + log.contains("✅") -> colors.success + log.contains("📋") || log.contains("🎬") -> colors.secondary + log.contains("Step") || log.contains("=====") -> colors.primary + log.contains("⛔") -> colors.error + else -> colors.textSecondary + } + + Text( + text = log, + fontSize = 12.sp, + color = logColor, + modifier = Modifier.padding(vertical = 2.dp) + ) +} + +@Composable +fun InputArea( + inputText: String, + onInputChange: (String) -> Unit, + onExecute: () -> Unit, + onStop: () -> Unit, + isRunning: Boolean, + enabled: Boolean, + onInputClick: () -> Unit = {} +) { + val colors = BaoziTheme.colors + Surface( + modifier = Modifier.fillMaxWidth(), + color = colors.backgroundCard, + shadowElevation = 8.dp + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 12.dp), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = if (isRunning) Arrangement.Center else Arrangement.Start + ) { + if (isRunning) { + // 运行中只显示停止按钮 + Button( + onClick = onStop, + colors = ButtonDefaults.buttonColors( + containerColor = colors.error + ), + shape = RoundedCornerShape(24.dp), + modifier = Modifier + .fillMaxWidth() + .height(48.dp) + ) { + Icon( + imageVector = Icons.Default.Close, + contentDescription = "停止", + tint = Color.White, + modifier = Modifier.size(20.dp) + ) + Spacer(modifier = Modifier.width(8.dp)) + Text( + text = "停止执行", + color = Color.White, + fontSize = 15.sp, + fontWeight = FontWeight.Medium + ) + } + } else { + // 非运行状态显示输入框和发送按钮 + // 输入框 + Box( + modifier = Modifier + .weight(1f) + .clip(RoundedCornerShape(24.dp)) + .background(colors.backgroundInput) + .then( + if (!enabled) { + Modifier.clickable { onInputClick() } + } else { + Modifier + } + ) + .padding(horizontal = 20.dp, vertical = 14.dp) + ) { + if (enabled) { + // Shizuku 已连接，显示可编辑的输入框 + BasicTextField( + value = inputText, + onValueChange = onInputChange, + textStyle = TextStyle( + color = colors.textPrimary, + fontSize = 15.sp + ), + cursorBrush = SolidColor(colors.primary), + modifier = Modifier.fillMaxWidth(), + decorationBox = { innerTextField -> + Box { + if (inputText.isEmpty()) { + Text( + text = "告诉肉包你想做什么...", + color = colors.textHint, + fontSize = 15.sp + ) + } + innerTextField() + } + } + ) + } else { + // Shizuku 未连接，显示提示文字 + Text( + text = "请先连接 Shizuku", + color = colors.textHint, + fontSize = 15.sp + ) + } + } + + Spacer(modifier = Modifier.width(12.dp)) + + // 发送按钮 + IconButton( + onClick = onExecute, + enabled = enabled && inputText.isNotBlank(), + modifier = Modifier + .size(48.dp) + .clip(CircleShape) + .background( + if (inputText.isNotBlank() && enabled) colors.primary + else colors.backgroundInput + ) + ) { + Icon( + imageVector = Icons.Default.Send, + contentDescription = "发送", + tint = Color.White, + modifier = Modifier.size(24.dp) + ) + } + } + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/OnboardingScreen.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/OnboardingScreen.kt new file mode 100644 index 0000000..06c4c7b --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/OnboardingScreen.kt @@ -0,0 +1,247 @@ +package com.roubao.autopilot.ui.screens + +import androidx.compose.animation.* +import androidx.compose.animation.core.* +import androidx.compose.foundation.ExperimentalFoundationApi +import androidx.compose.foundation.background +import androidx.compose.foundation.layout.* +import androidx.compose.foundation.pager.HorizontalPager +import androidx.compose.foundation.pager.rememberPagerState +import androidx.compose.foundation.shape.CircleShape +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.* +import androidx.compose.material.icons.outlined.* +import androidx.compose.material3.* +import androidx.compose.runtime.* +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.clip +import androidx.compose.ui.graphics.Brush +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.vector.ImageVector +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.style.TextAlign +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp +import com.roubao.autopilot.ui.theme.BaoziTheme +import com.roubao.autopilot.ui.theme.Primary +import kotlinx.coroutines.launch + +data class OnboardingPage( + val icon: ImageVector, + val title: String, + val description: String, + val iconColor: Color = Primary +) + +val onboardingPages = listOf( + OnboardingPage( + icon = Icons.Outlined.Star, + title = "欢迎使用肉包", + description = "肉包是一个智能自动化助手，\n可以帮你操作手机完成各种任务", + iconColor = Color(0xFF6366F1) // Indigo + ), + OnboardingPage( + icon = Icons.Outlined.Settings, + title = "AI 驱动", + description = "基于先进的视觉语言模型，\n肉包能够理解屏幕内容并做出智能决策", + iconColor = Color(0xFF8B5CF6) // Violet + ), + OnboardingPage( + icon = Icons.Outlined.Home, + title = "简单易用", + description = "只需用自然语言描述你想做的事，\n肉包会自动帮你完成", + iconColor = Color(0xFF06B6D4) // Cyan + ), + OnboardingPage( + icon = Icons.Filled.Lock, + title = "安全可靠", + description = "遇到敏感页面（如支付、密码）会自动停止，\n保护你的账户安全", + iconColor = Color(0xFF10B981) // Emerald + ) +) + +@OptIn(ExperimentalFoundationApi::class) +@Composable +fun OnboardingScreen( + onComplete: () -> Unit +) { + val colors = BaoziTheme.colors + val pagerState = rememberPagerState(pageCount = { onboardingPages.size }) + val coroutineScope = rememberCoroutineScope() + + Box( + modifier = Modifier + .fillMaxSize() + .background( + Brush.verticalGradient( + colors = listOf( + colors.background, + colors.backgroundCard + ) + ) + ) + ) { + Column( + modifier = Modifier.fillMaxSize(), + horizontalAlignment = Alignment.CenterHorizontally + ) { + // 页面内容 + HorizontalPager( + state = pagerState, + modifier = Modifier + .weight(1f) + .fillMaxWidth() + ) { page -> + OnboardingPageContent( + page = onboardingPages[page], + modifier = Modifier.fillMaxSize() + ) + } + + // 指示器 + Row( + modifier = Modifier.padding(16.dp), + horizontalArrangement = Arrangement.spacedBy(8.dp) + ) { + repeat(onboardingPages.size) { index -> + val isSelected = pagerState.currentPage == index + Box( + modifier = Modifier + .size(if (isSelected) 24.dp else 8.dp, 8.dp) + .clip(CircleShape) + .background( + if (isSelected) Primary else colors.textHint.copy(alpha = 0.3f) + ) + .animateContentSize() + ) + } + } + + Spacer(modifier = Modifier.height(24.dp)) + + // 底部按钮 + Row( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 24.dp, vertical = 16.dp), + horizontalArrangement = Arrangement.SpaceBetween + ) { + // 跳过按钮 + TextButton( + onClick = onComplete, + modifier = Modifier.weight(1f) + ) { + Text( + text = "跳过", + color = colors.textSecondary, + fontSize = 16.sp + ) + } + + Spacer(modifier = Modifier.width(16.dp)) + + // 下一步/开始按钮 + Button( + onClick = { + if (pagerState.currentPage < onboardingPages.size - 1) { + coroutineScope.launch { + pagerState.animateScrollToPage(pagerState.currentPage + 1) + } + } else { + onComplete() + } + }, + modifier = Modifier + .weight(2f) + .height(52.dp), + shape = RoundedCornerShape(26.dp), + colors = ButtonDefaults.buttonColors( + containerColor = Primary + ) + ) { + Text( + text = if (pagerState.currentPage < onboardingPages.size - 1) "下一步" else "开始使用", + fontSize = 16.sp, + fontWeight = FontWeight.Medium, + color = Color.White + ) + } + } + + Spacer(modifier = Modifier.height(32.dp)) + } + } +} + +@Composable +fun OnboardingPageContent( + page: OnboardingPage, + modifier: Modifier = Modifier +) { + val colors = BaoziTheme.colors + + Column( + modifier = modifier.padding(horizontal = 32.dp), + horizontalAlignment = Alignment.CenterHorizontally, + verticalArrangement = Arrangement.Center + ) { + // 动画图标 + val infiniteTransition = rememberInfiniteTransition(label = "icon") + val scale by infiniteTransition.animateFloat( + initialValue = 1f, + targetValue = 1.08f, + animationSpec = infiniteRepeatable( + animation = tween(1200, easing = EaseInOutSine), + repeatMode = RepeatMode.Reverse + ), + label = "scale" + ) + + Box( + modifier = Modifier + .size(160.dp) + .clip(CircleShape) + .background( + Brush.radialGradient( + colors = listOf( + page.iconColor.copy(alpha = 0.15f), + page.iconColor.copy(alpha = 0.05f), + colors.background.copy(alpha = 0f) + ) + ) + ), + contentAlignment = Alignment.Center + ) { + Icon( + imageVector = page.icon, + contentDescription = page.title, + modifier = Modifier.size((72 * scale).dp), + tint = page.iconColor + ) + } + + Spacer(modifier = Modifier.height(48.dp)) + + // 标题 + Text( + text = page.title, + fontSize = 28.sp, + fontWeight = FontWeight.Bold, + color = colors.textPrimary, + textAlign = TextAlign.Center + ) + + Spacer(modifier = Modifier.height(16.dp)) + + // 描述 + Text( + text = page.description, + fontSize = 16.sp, + color = colors.textSecondary, + textAlign = TextAlign.Center, + lineHeight = 26.sp + ) + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/SettingsScreen.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/SettingsScreen.kt new file mode 100644 index 0000000..92c5f9d --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/screens/SettingsScreen.kt @@ -0,0 +1,1812 @@ +package com.roubao.autopilot.ui.screens + +import androidx.compose.animation.* +import androidx.compose.foundation.background +import androidx.compose.foundation.border +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.* +import androidx.compose.foundation.rememberScrollState +import androidx.compose.foundation.verticalScroll +import androidx.compose.foundation.lazy.LazyColumn +import androidx.compose.foundation.shape.CircleShape +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.foundation.text.BasicTextField +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.Add +import androidx.compose.material.icons.filled.CheckCircle +import androidx.compose.material.icons.filled.Close +import androidx.compose.material.icons.filled.Info +import androidx.compose.material.icons.filled.KeyboardArrowRight +import androidx.compose.material.icons.filled.Lock +import androidx.compose.material.icons.filled.Search +import androidx.compose.material.icons.filled.Settings +import androidx.compose.material.icons.filled.Build +import androidx.compose.material.icons.filled.Star +import androidx.compose.material.icons.filled.Warning +import androidx.compose.material.icons.outlined.Star +import androidx.compose.material3.* +import androidx.compose.runtime.* +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.clip +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.SolidColor +import androidx.compose.ui.graphics.vector.ImageVector +import androidx.compose.ui.text.TextStyle +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.input.PasswordVisualTransformation +import androidx.compose.ui.text.input.VisualTransformation +import androidx.compose.ui.unit.dp +import androidx.compose.ui.unit.sp +import androidx.compose.ui.platform.LocalContext +import com.roubao.autopilot.BuildConfig +import com.roubao.autopilot.data.ApiProvider +import com.roubao.autopilot.data.AppSettings +import com.roubao.autopilot.ui.theme.BaoziTheme +import com.roubao.autopilot.ui.theme.ThemeMode +import com.roubao.autopilot.utils.CrashHandler + +@Composable +fun SettingsScreen( + settings: AppSettings, + onUpdateApiKey: (String) -> Unit, + onUpdateBaseUrl: (String) -> Unit, + onUpdateModel: (String) -> Unit, + onUpdateCachedModels: (List) -> Unit, + onUpdateThemeMode: (ThemeMode) -> Unit, + onUpdateMaxSteps: (Int) -> Unit, + onUpdateRootModeEnabled: (Boolean) -> Unit, + onUpdateSuCommandEnabled: (Boolean) -> Unit, + onSelectProvider: (ApiProvider) -> Unit, + shizukuAvailable: Boolean, + shizukuPrivilegeLevel: String = "ADB", // "ADB", "ROOT", "NONE" + onFetchModels: ((onSuccess: (List) -> Unit, onError: (String) -> Unit) -> Unit)? = null +) { + val colors = BaoziTheme.colors + var showApiKeyDialog by remember { mutableStateOf(false) } + var showModelDialog by remember { mutableStateOf(false) } + var showThemeDialog by remember { mutableStateOf(false) } + var showMaxStepsDialog by remember { mutableStateOf(false) } + var showBaseUrlDialog by remember { mutableStateOf(false) } + var showShizukuHelpDialog by remember { mutableStateOf(false) } + var showOverlayHelpDialog by remember { mutableStateOf(false) } + var showRootModeWarningDialog by remember { mutableStateOf(false) } + var showSuCommandWarningDialog by remember { mutableStateOf(false) } + + LazyColumn( + modifier = Modifier + .fillMaxSize() + .background(colors.background) + ) { + // 顶部标题 + item { + Box( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 20.dp, vertical = 16.dp) + ) { + Column { + Text( + text = "设置", + fontSize = 28.sp, + fontWeight = FontWeight.Bold, + color = colors.primary + ) + Text( + text = "配置 API 和应用选项", + fontSize = 14.sp, + color = colors.textSecondary + ) + } + } + } + + // 连接状态卡片 + item { + StatusCard(shizukuAvailable = shizukuAvailable) + } + + // 外观设置分组 + item { + SettingsSection(title = "外观") + } + + // 主题模式设置 + item { + SettingsItem( + icon = if (colors.isDark) Icons.Default.Star else Icons.Outlined.Star, + title = "主题模式", + subtitle = when (settings.themeMode) { + ThemeMode.LIGHT -> "浅色模式" + ThemeMode.DARK -> "深色模式" + ThemeMode.SYSTEM -> "跟随系统" + }, + onClick = { showThemeDialog = true } + ) + } + + // 执行设置分组 + item { + SettingsSection(title = "执行设置") + } + + // 最大步数设置 + item { + SettingsItem( + icon = Icons.Default.Settings, + title = "最大执行步数", + subtitle = "${settings.maxSteps} 步", + onClick = { showMaxStepsDialog = true } + ) + } + + // Shizuku 高级设置分组（仅在 Shizuku 可用时显示） + if (shizukuAvailable) { + item { + SettingsSection(title = "Shizuku 高级选项") + } + + // 显示当前权限级别 + item { + Card( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 4.dp), + shape = RoundedCornerShape(12.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + verticalAlignment = Alignment.CenterVertically + ) { + Box( + modifier = Modifier + .size(40.dp) + .clip(RoundedCornerShape(10.dp)) + .background( + when (shizukuPrivilegeLevel) { + "ROOT" -> colors.error.copy(alpha = 0.15f) + else -> colors.primary.copy(alpha = 0.15f) + } + ), + contentAlignment = Alignment.Center + ) { + Icon( + imageVector = Icons.Default.Info, + contentDescription = null, + tint = when (shizukuPrivilegeLevel) { + "ROOT" -> colors.error + else -> colors.primary + }, + modifier = Modifier.size(22.dp) + ) + } + Spacer(modifier = Modifier.width(16.dp)) + Column(modifier = Modifier.weight(1f)) { + Text( + text = "当前权限级别", + fontSize = 15.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary + ) + Text( + text = when (shizukuPrivilegeLevel) { + "ROOT" -> "Root 模式 (UID 0)" + "ADB" -> "ADB 模式 (UID 2000)" + else -> "未连接" + }, + fontSize = 13.sp, + color = when (shizukuPrivilegeLevel) { + "ROOT" -> colors.error + else -> colors.textSecondary + }, + maxLines = 1 + ) + } + } + } + } + + // Root 模式开关（仅在 Shizuku 以 Root 权限运行时可用） + item { + val isShizukuRoot = shizukuPrivilegeLevel == "ROOT" + Card( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 4.dp), + shape = RoundedCornerShape(12.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + verticalAlignment = Alignment.CenterVertically + ) { + Box( + modifier = Modifier + .size(40.dp) + .clip(RoundedCornerShape(10.dp)) + .background( + if (isShizukuRoot) colors.error.copy(alpha = 0.15f) + else colors.textHint.copy(alpha = 0.15f) + ), + contentAlignment = Alignment.Center + ) { + Icon( + imageVector = Icons.Default.Warning, + contentDescription = null, + tint = if (isShizukuRoot) colors.error else colors.textHint, + modifier = Modifier.size(22.dp) + ) + } + Spacer(modifier = Modifier.width(16.dp)) + Column(modifier = Modifier.weight(1f)) { + Text( + text = "Root 模式", + fontSize = 15.sp, + fontWeight = FontWeight.Medium, + color = if (isShizukuRoot) colors.textPrimary else colors.textHint + ) + Text( + text = when { + !isShizukuRoot -> "需要 Shizuku 以 Root 权限运行" + settings.rootModeEnabled -> "已启用高级权限" + else -> "启用后可使用 Root 功能" + }, + fontSize = 13.sp, + color = when { + !isShizukuRoot -> colors.textHint + settings.rootModeEnabled -> colors.error + else -> colors.textSecondary + }, + maxLines = 1 + ) + } + Switch( + checked = settings.rootModeEnabled, + onCheckedChange = { enabled -> + if (enabled) { + showRootModeWarningDialog = true + } else { + onUpdateRootModeEnabled(false) + } + }, + enabled = isShizukuRoot, + colors = SwitchDefaults.colors( + checkedThumbColor = colors.error, + checkedTrackColor = colors.error.copy(alpha = 0.5f), + uncheckedThumbColor = colors.textHint, + uncheckedTrackColor = colors.backgroundInput, + disabledCheckedThumbColor = colors.textHint, + disabledCheckedTrackColor = colors.backgroundInput, + disabledUncheckedThumbColor = colors.textHint.copy(alpha = 0.5f), + disabledUncheckedTrackColor = colors.backgroundInput.copy(alpha = 0.5f) + ) + ) + } + } + } + + // su -c 开关（仅在 Root 模式开启时显示） + if (settings.rootModeEnabled) { + item { + Card( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 4.dp), + shape = RoundedCornerShape(12.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + verticalAlignment = Alignment.CenterVertically + ) { + Box( + modifier = Modifier + .size(40.dp) + .clip(RoundedCornerShape(10.dp)) + .background(colors.error.copy(alpha = 0.15f)), + contentAlignment = Alignment.Center + ) { + Icon( + imageVector = Icons.Default.Lock, + contentDescription = null, + tint = colors.error, + modifier = Modifier.size(22.dp) + ) + } + Spacer(modifier = Modifier.width(16.dp)) + Column(modifier = Modifier.weight(1f)) { + Text( + text = "允许 su -c 命令", + fontSize = 15.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary + ) + Text( + text = if (settings.suCommandEnabled) "AI 可执行 Root 命令" else "禁止执行 su -c", + fontSize = 13.sp, + color = if (settings.suCommandEnabled) colors.error else colors.textSecondary, + maxLines = 1 + ) + } + Switch( + checked = settings.suCommandEnabled, + onCheckedChange = { enabled -> + if (enabled) { + showSuCommandWarningDialog = true + } else { + onUpdateSuCommandEnabled(false) + } + }, + colors = SwitchDefaults.colors( + checkedThumbColor = colors.error, + checkedTrackColor = colors.error.copy(alpha = 0.5f), + uncheckedThumbColor = colors.textHint, + uncheckedTrackColor = colors.backgroundInput + ) + ) + } + } + } + } + } + + // API 设置分组 + item { + SettingsSection(title = "API 配置") + } + + // Base URL 设置 + item { + SettingsItem( + icon = Icons.Default.Settings, + title = "API 服务商", + subtitle = settings.currentProvider.name, + onClick = { showBaseUrlDialog = true } + ) + } + + // API Key 设置 + item { + SettingsItem( + icon = Icons.Default.Lock, + title = "API Key", + subtitle = if (settings.apiKey.isNotEmpty()) "已设置 (${maskApiKey(settings.apiKey)})" else "未设置", + onClick = { showApiKeyDialog = true } + ) + } + + // 模型设置 + item { + SettingsItem( + icon = Icons.Default.Build, + title = "模型", + subtitle = settings.model, + onClick = { showModelDialog = true } + ) + } + + // 反馈分组 + item { + SettingsSection(title = "反馈与调试") + } + + item { + val context = LocalContext.current + val logStats = remember { mutableStateOf(CrashHandler.getLogStats(context)) } + + SettingsItem( + icon = Icons.Default.Info, + title = "导出日志", + subtitle = logStats.value, + onClick = { + CrashHandler.shareLogs(context) + } + ) + } + + item { + val context = LocalContext.current + var showClearDialog by remember { mutableStateOf(false) } + + SettingsItem( + icon = Icons.Default.Close, + title = "清除日志", + subtitle = "删除所有本地日志文件", + onClick = { showClearDialog = true } + ) + + if (showClearDialog) { + AlertDialog( + onDismissRequest = { showClearDialog = false }, + containerColor = BaoziTheme.colors.backgroundCard, + title = { Text("确认清除", color = BaoziTheme.colors.textPrimary) }, + text = { Text("确定要删除所有日志文件吗？", color = BaoziTheme.colors.textSecondary) }, + confirmButton = { + TextButton(onClick = { + CrashHandler.clearLogs(context) + showClearDialog = false + android.widget.Toast.makeText(context, "日志已清除", android.widget.Toast.LENGTH_SHORT).show() + }) { + Text("确定", color = BaoziTheme.colors.error) + } + }, + dismissButton = { + TextButton(onClick = { showClearDialog = false }) { + Text("取消", color = BaoziTheme.colors.textSecondary) + } + } + ) + } + } + + // 帮助分组 + item { + SettingsSection(title = "帮助") + } + + item { + SettingsItem( + icon = Icons.Default.Info, + title = "Shizuku 使用指南", + subtitle = "了解如何安装和配置 Shizuku", + onClick = { showShizukuHelpDialog = true } + ) + } + + item { + SettingsItem( + icon = Icons.Default.Settings, + title = "悬浮窗权限说明", + subtitle = "了解为什么需要悬浮窗权限", + onClick = { showOverlayHelpDialog = true } + ) + } + + // 关于分组 + item { + SettingsSection(title = "关于") + } + + item { + SettingsItem( + icon = Icons.Default.Info, + title = "版本", + subtitle = BuildConfig.VERSION_NAME, + onClick = { } + ) + } + + item { + SettingsItem( + icon = Icons.Default.Build, + title = "肉包 Autopilot", + subtitle = "基于视觉语言模型的 Android 自动化工具", + onClick = { } + ) + } + + // 底部间距 + item { + Spacer(modifier = Modifier.height(32.dp)) + } + } + + // 主题选择对话框 + if (showThemeDialog) { + ThemeSelectDialog( + currentTheme = settings.themeMode, + onDismiss = { showThemeDialog = false }, + onSelect = { + onUpdateThemeMode(it) + showThemeDialog = false + } + ) + } + + // 最大步数设置对话框 + if (showMaxStepsDialog) { + MaxStepsDialog( + currentSteps = settings.maxSteps, + onDismiss = { showMaxStepsDialog = false }, + onConfirm = { + onUpdateMaxSteps(it) + showMaxStepsDialog = false + } + ) + } + + // API Key 编辑对话框 + if (showApiKeyDialog) { + ApiKeyDialog( + currentKey = settings.apiKey, + onDismiss = { showApiKeyDialog = false }, + onConfirm = { + onUpdateApiKey(it) + showApiKeyDialog = false + } + ) + } + + // 模型选择对话框（合并了自定义输入和从 API 获取） + if (showModelDialog) { + ModelSelectDialogWithFetch( + currentModel = settings.model, + cachedModels = settings.cachedModels, + hasApiKey = settings.apiKey.isNotEmpty(), + onDismiss = { showModelDialog = false }, + onSelect = { + onUpdateModel(it) + showModelDialog = false + }, + onFetchModels = onFetchModels, + onUpdateCachedModels = onUpdateCachedModels + ) + } + + // 服务商选择对话框 + if (showBaseUrlDialog) { + ProviderSelectDialog( + currentProviderId = settings.currentProviderId, + customBaseUrl = settings.currentConfig.customBaseUrl, + onDismiss = { showBaseUrlDialog = false }, + onSelectProvider = { provider -> + onSelectProvider(provider) + showBaseUrlDialog = false + }, + onUpdateCustomUrl = { url -> + onUpdateBaseUrl(url) + } + ) + } + + // Shizuku 帮助对话框 + if (showShizukuHelpDialog) { + ShizukuHelpDialog(onDismiss = { showShizukuHelpDialog = false }) + } + + // 悬浮窗权限帮助对话框 + if (showOverlayHelpDialog) { + OverlayHelpDialog(onDismiss = { showOverlayHelpDialog = false }) + } + + // Root 模式警告对话框 + if (showRootModeWarningDialog) { + RootModeWarningDialog( + onDismiss = { showRootModeWarningDialog = false }, + onConfirm = { + onUpdateRootModeEnabled(true) + showRootModeWarningDialog = false + } + ) + } + + // su -c 命令警告对话框 + if (showSuCommandWarningDialog) { + SuCommandWarningDialog( + onDismiss = { showSuCommandWarningDialog = false }, + onConfirm = { + onUpdateSuCommandEnabled(true) + showSuCommandWarningDialog = false + } + ) + } +} + +@Composable +fun StatusCard(shizukuAvailable: Boolean) { + val colors = BaoziTheme.colors + Card( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 8.dp), + shape = RoundedCornerShape(16.dp), + colors = CardDefaults.cardColors( + containerColor = if (shizukuAvailable) colors.success.copy(alpha = 0.15f) else colors.error.copy(alpha = 0.15f) + ) + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + verticalAlignment = Alignment.CenterVertically + ) { + Icon( + imageVector = if (shizukuAvailable) Icons.Default.CheckCircle else Icons.Default.Close, + contentDescription = null, + tint = if (shizukuAvailable) colors.success else colors.error, + modifier = Modifier.size(32.dp) + ) + Spacer(modifier = Modifier.width(16.dp)) + Column { + Text( + text = if (shizukuAvailable) "Shizuku 已连接" else "Shizuku 未连接", + fontSize = 16.sp, + fontWeight = FontWeight.Medium, + color = if (shizukuAvailable) colors.success else colors.error + ) + Text( + text = if (shizukuAvailable) "设备控制功能可用" else "请启动 Shizuku 并授权", + fontSize = 13.sp, + color = colors.textSecondary + ) + } + } + } +} + +@Composable +fun SettingsSection(title: String) { + val colors = BaoziTheme.colors + Text( + text = title, + fontSize = 14.sp, + fontWeight = FontWeight.Medium, + color = colors.primary, + modifier = Modifier.padding(horizontal = 20.dp, vertical = 12.dp) + ) +} + +@Composable +fun SettingsItem( + icon: ImageVector, + title: String, + subtitle: String, + onClick: () -> Unit, + trailing: @Composable (() -> Unit)? = null +) { + val colors = BaoziTheme.colors + Card( + modifier = Modifier + .fillMaxWidth() + .padding(horizontal = 16.dp, vertical = 4.dp) + .clickable(onClick = onClick), + shape = RoundedCornerShape(12.dp), + colors = CardDefaults.cardColors(containerColor = colors.backgroundCard) + ) { + Row( + modifier = Modifier + .fillMaxWidth() + .padding(16.dp), + verticalAlignment = Alignment.CenterVertically + ) { + Box( + modifier = Modifier + .size(40.dp) + .clip(RoundedCornerShape(10.dp)) + .background(colors.primary.copy(alpha = 0.15f)), + contentAlignment = Alignment.Center + ) { + Icon( + imageVector = icon, + contentDescription = null, + tint = colors.primary, + modifier = Modifier.size(22.dp) + ) + } + Spacer(modifier = Modifier.width(16.dp)) + Column(modifier = Modifier.weight(1f)) { + Text( + text = title, + fontSize = 15.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary + ) + Text( + text = subtitle, + fontSize = 13.sp, + color = colors.textSecondary, + maxLines = 1 + ) + } + if (trailing != null) { + trailing() + } else { + Icon( + imageVector = Icons.Default.KeyboardArrowRight, + contentDescription = null, + tint = colors.textHint, + modifier = Modifier.size(20.dp) + ) + } + } + } +} + + + +@Composable +fun ThemeSelectDialog( + currentTheme: ThemeMode, + onDismiss: () -> Unit, + onSelect: (ThemeMode) -> Unit +) { + val colors = BaoziTheme.colors + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + title = { + Text("选择主题", color = colors.textPrimary) + }, + text = { + Column { + listOf( + ThemeMode.LIGHT to "浅色模式", + ThemeMode.DARK to "深色模式", + ThemeMode.SYSTEM to "跟随系统" + ).forEach { (mode, label) -> + val isSelected = mode == currentTheme + Surface( + modifier = Modifier + .fillMaxWidth() + .padding(vertical = 4.dp) + .clickable { onSelect(mode) }, + shape = RoundedCornerShape(8.dp), + color = if (isSelected) colors.primary.copy(alpha = 0.15f) else Color.Transparent + ) { + Row( + modifier = Modifier.padding(12.dp), + verticalAlignment = Alignment.CenterVertically + ) { + if (isSelected) { + Icon( + imageVector = Icons.Default.CheckCircle, + contentDescription = null, + tint = colors.primary, + modifier = Modifier.size(20.dp) + ) + } else { + Box( + modifier = Modifier + .size(20.dp) + .border(2.dp, colors.textHint, CircleShape) + ) + } + Spacer(modifier = Modifier.width(12.dp)) + Text( + text = label, + fontSize = 14.sp, + color = if (isSelected) colors.primary else colors.textPrimary + ) + } + } + } + } + }, + confirmButton = { + TextButton(onClick = onDismiss) { + Text("关闭", color = colors.textSecondary) + } + } + ) +} + +@Composable +fun ApiKeyDialog( + currentKey: String, + onDismiss: () -> Unit, + onConfirm: (String) -> Unit +) { + val colors = BaoziTheme.colors + var key by remember { mutableStateOf(currentKey) } + var showKey by remember { mutableStateOf(false) } + + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + title = { + Text("API Key", color = colors.textPrimary) + }, + text = { + Column { + Text( + text = "请输入您的 API Key", + fontSize = 14.sp, + color = colors.textSecondary, + modifier = Modifier.padding(bottom = 16.dp) + ) + Box( + modifier = Modifier + .fillMaxWidth() + .clip(RoundedCornerShape(12.dp)) + .background(colors.backgroundInput) + .padding(4.dp) + ) { + Row( + verticalAlignment = Alignment.CenterVertically + ) { + BasicTextField( + value = key, + onValueChange = { key = it }, + textStyle = TextStyle( + color = colors.textPrimary, + fontSize = 14.sp + ), + cursorBrush = SolidColor(colors.primary), + visualTransformation = if (showKey) VisualTransformation.None else PasswordVisualTransformation(), + modifier = Modifier + .weight(1f) + .padding(12.dp) + ) + TextButton(onClick = { showKey = !showKey }) { + Text( + text = if (showKey) "隐藏" else "显示", + fontSize = 12.sp, + color = colors.textHint + ) + } + } + } + } + }, + confirmButton = { + TextButton(onClick = { onConfirm(key) }) { + Text("确定", color = colors.primary) + } + }, + dismissButton = { + TextButton(onClick = onDismiss) { + Text("取消", color = colors.textSecondary) + } + } + ) +} + + +/** + * 模型选择对话框（合并了自定义输入和从 API 获取） + */ +@Composable +fun ModelSelectDialogWithFetch( + currentModel: String, + cachedModels: List, + hasApiKey: Boolean, + onDismiss: () -> Unit, + onSelect: (String) -> Unit, + onFetchModels: ((onSuccess: (List) -> Unit, onError: (String) -> Unit) -> Unit)? = null, + onUpdateCachedModels: (List) -> Unit +) { + val colors = BaoziTheme.colors + val context = LocalContext.current + var customModel by remember { mutableStateOf("") } + var searchQuery by remember { mutableStateOf("") } + var isLoading by remember { mutableStateOf(false) } + + // 默认推荐模型 + val defaultModel = "qwen3-vl-plus" + + // 过滤后的模型列表 + val filteredModels = remember(cachedModels, searchQuery) { + if (searchQuery.isBlank()) { + cachedModels + } else { + cachedModels.filter { it.contains(searchQuery, ignoreCase = true) } + } + } + + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + title = { + Text("选择模型", color = colors.textPrimary) + }, + text = { + Column( + modifier = Modifier.verticalScroll(rememberScrollState()) + ) { + // 默认推荐模型 + Text( + text = "推荐模型", + fontSize = 12.sp, + fontWeight = FontWeight.Medium, + color = colors.textHint, + modifier = Modifier.padding(bottom = 8.dp) + ) + + val isDefaultSelected = currentModel == defaultModel + Surface( + modifier = Modifier + .fillMaxWidth() + .clickable { onSelect(defaultModel) }, + shape = RoundedCornerShape(10.dp), + color = if (isDefaultSelected) colors.primary.copy(alpha = 0.15f) else colors.backgroundInput + ) { + Row( + modifier = Modifier.padding(horizontal = 12.dp, vertical = 12.dp), + verticalAlignment = Alignment.CenterVertically + ) { + if (isDefaultSelected) { + Icon( + imageVector = Icons.Default.CheckCircle, + contentDescription = null, + tint = colors.primary, + modifier = Modifier.size(20.dp) + ) + } else { + Box( + modifier = Modifier + .size(20.dp) + .border(2.dp, colors.textHint, CircleShape) + ) + } + Spacer(modifier = Modifier.width(10.dp)) + Column { + Text( + text = defaultModel, + fontSize = 14.sp, + fontWeight = FontWeight.Medium, + color = if (isDefaultSelected) colors.primary else colors.textPrimary + ) + Text( + text = "阿里云通义千问视觉模型", + fontSize = 11.sp, + color = colors.textHint + ) + } + } + } + + Spacer(modifier = Modifier.height(16.dp)) + + // 自定义模型输入 + Text( + text = "自定义模型", + fontSize = 12.sp, + fontWeight = FontWeight.Medium, + color = colors.textHint, + modifier = Modifier.padding(bottom = 8.dp) + ) + + Row( + modifier = Modifier.fillMaxWidth(), + verticalAlignment = Alignment.CenterVertically + ) { + Box( + modifier = Modifier + .weight(1f) + .clip(RoundedCornerShape(12.dp)) + .background(colors.backgroundInput) + .padding(horizontal = 12.dp, vertical = 12.dp) + ) { + if (customModel.isEmpty()) { + Text( + text = "输入模型名称，如 gpt-4o", + color = colors.textHint, + fontSize = 14.sp + ) + } + BasicTextField( + value = customModel, + onValueChange = { customModel = it }, + textStyle = TextStyle( + color = colors.textPrimary, + fontSize = 14.sp + ), + cursorBrush = SolidColor(colors.primary), + modifier = Modifier.fillMaxWidth(), + singleLine = true + ) + } + + Spacer(modifier = Modifier.width(8.dp)) + + // 确认按钮 + Surface( + modifier = Modifier + .size(44.dp) + .clickable(enabled = customModel.isNotBlank()) { + onSelect(customModel.trim()) + }, + shape = RoundedCornerShape(12.dp), + color = if (customModel.isNotBlank()) colors.primary else colors.backgroundInput + ) { + Box(contentAlignment = Alignment.Center) { + Icon( + imageVector = Icons.Default.CheckCircle, + contentDescription = "确认", + tint = if (customModel.isNotBlank()) Color.White else colors.textHint, + modifier = Modifier.size(22.dp) + ) + } + } + } + + Spacer(modifier = Modifier.height(16.dp)) + + // 从 API 获取模型 - 更明显的按钮 + if (onFetchModels != null) { + Button( + onClick = { + if (!hasApiKey) { + android.widget.Toast.makeText(context, "请先设置 API Key", android.widget.Toast.LENGTH_SHORT).show() + return@Button + } + isLoading = true + onFetchModels( + { models -> + isLoading = false + onUpdateCachedModels(models) + android.widget.Toast.makeText(context, "获取到 ${models.size} 个模型", android.widget.Toast.LENGTH_SHORT).show() + }, + { error -> + isLoading = false + android.widget.Toast.makeText(context, "获取失败: $error", android.widget.Toast.LENGTH_SHORT).show() + } + ) + }, + enabled = !isLoading && hasApiKey, + modifier = Modifier.fillMaxWidth(), + shape = RoundedCornerShape(12.dp), + colors = ButtonDefaults.buttonColors( + containerColor = colors.primary, + disabledContainerColor = colors.backgroundInput + ) + ) { + if (isLoading) { + CircularProgressIndicator( + modifier = Modifier.size(18.dp), + strokeWidth = 2.dp, + color = Color.White + ) + Spacer(modifier = Modifier.width(8.dp)) + Text("获取中...", fontSize = 14.sp, color = Color.White) + } else { + Icon( + imageVector = Icons.Default.Add, + contentDescription = null, + modifier = Modifier.size(18.dp), + tint = if (hasApiKey) Color.White else colors.textHint + ) + Spacer(modifier = Modifier.width(6.dp)) + Text( + "从 API 获取可用模型", + fontSize = 14.sp, + color = if (hasApiKey) Color.White else colors.textHint + ) + } + } + + if (cachedModels.isNotEmpty()) { + Spacer(modifier = Modifier.height(12.dp)) + + Text( + text = "API 模型列表 (${cachedModels.size})", + fontSize = 12.sp, + fontWeight = FontWeight.Medium, + color = colors.textHint, + modifier = Modifier.padding(bottom = 8.dp) + ) + } + + Spacer(modifier = Modifier.height(4.dp)) + + // 搜索框（模型数量超过 10 个时显示） + if (cachedModels.size > 10) { + Box( + modifier = Modifier + .fillMaxWidth() + .clip(RoundedCornerShape(10.dp)) + .background(colors.backgroundInput) + .padding(horizontal = 12.dp, vertical = 10.dp) + ) { + Row( + verticalAlignment = Alignment.CenterVertically + ) { + Icon( + imageVector = Icons.Default.Search, + contentDescription = null, + tint = colors.textHint, + modifier = Modifier.size(18.dp) + ) + Spacer(modifier = Modifier.width(8.dp)) + Box(modifier = Modifier.weight(1f)) { + if (searchQuery.isEmpty()) { + Text( + text = "搜索模型...", + color = colors.textHint, + fontSize = 14.sp + ) + } + BasicTextField( + value = searchQuery, + onValueChange = { searchQuery = it }, + textStyle = TextStyle( + color = colors.textPrimary, + fontSize = 14.sp + ), + cursorBrush = SolidColor(colors.primary), + modifier = Modifier.fillMaxWidth(), + singleLine = true + ) + } + if (searchQuery.isNotEmpty()) { + Icon( + imageVector = Icons.Default.Close, + contentDescription = "清除", + tint = colors.textHint, + modifier = Modifier + .size(18.dp) + .clickable { searchQuery = "" } + ) + } + } + } + Spacer(modifier = Modifier.height(8.dp)) + } + } + + // 模型列表 + if (cachedModels.isEmpty()) { + Box( + modifier = Modifier + .fillMaxWidth() + .padding(vertical = 16.dp), + contentAlignment = Alignment.Center + ) { + Text( + text = if (hasApiKey) "点击「从 API 获取」加载模型列表" else "请先设置 API Key", + fontSize = 13.sp, + color = colors.textHint + ) + } + } else if (filteredModels.isEmpty()) { + Box( + modifier = Modifier + .fillMaxWidth() + .padding(vertical = 16.dp), + contentAlignment = Alignment.Center + ) { + Text( + text = "没有匹配「$searchQuery」的模型", + fontSize = 13.sp, + color = colors.textHint + ) + } + } else { + // 显示过滤结果数量 + if (searchQuery.isNotBlank()) { + Text( + text = "找到 ${filteredModels.size} 个模型", + fontSize = 11.sp, + color = colors.textHint, + modifier = Modifier.padding(bottom = 4.dp) + ) + } + filteredModels.forEach { model -> + val isSelected = model == currentModel + Surface( + modifier = Modifier + .fillMaxWidth() + .padding(vertical = 3.dp) + .clickable { onSelect(model) }, + shape = RoundedCornerShape(8.dp), + color = if (isSelected) colors.primary.copy(alpha = 0.15f) else Color.Transparent + ) { + Row( + modifier = Modifier.padding(horizontal = 12.dp, vertical = 10.dp), + verticalAlignment = Alignment.CenterVertically + ) { + if (isSelected) { + Icon( + imageVector = Icons.Default.CheckCircle, + contentDescription = null, + tint = colors.primary, + modifier = Modifier.size(18.dp) + ) + } else { + Box( + modifier = Modifier + .size(18.dp) + .border(2.dp, colors.textHint, CircleShape) + ) + } + Spacer(modifier = Modifier.width(10.dp)) + Text( + text = model, + fontSize = 14.sp, + color = if (isSelected) colors.primary else colors.textPrimary + ) + } + } + } + } + } + }, + confirmButton = { + TextButton(onClick = onDismiss) { + Text("关闭", color = colors.textSecondary) + } + } + ) +} + +private fun maskApiKey(key: String): String { + return if (key.length > 8) { + "${key.take(4)}****${key.takeLast(4)}" + } else { + "****" + } +} + +@Composable +fun ShizukuHelpDialog(onDismiss: () -> Unit) { + val colors = BaoziTheme.colors + val context = LocalContext.current + + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + title = { + Text("Shizuku 使用指南", color = colors.textPrimary) + }, + text = { + Column( + modifier = Modifier.verticalScroll(rememberScrollState()) + ) { + HelpStep( + number = "1", + title = "下载 Shizuku", + description = "从 Google Play 或 GitHub 下载 Shizuku 应用" + ) + Spacer(modifier = Modifier.height(8.dp)) + // 下载按钮 + Button( + onClick = { + val intent = android.content.Intent( + android.content.Intent.ACTION_VIEW, + android.net.Uri.parse("https://github.com/RikkaApps/Shizuku/releases") + ) + context.startActivity(intent) + }, + colors = ButtonDefaults.buttonColors( + containerColor = colors.primary + ), + modifier = Modifier.fillMaxWidth() + ) { + Text("前往下载 Shizuku", color = Color.White) + } + Spacer(modifier = Modifier.height(16.dp)) + HelpStep( + number = "2", + title = "启动 Shizuku", + description = "打开 Shizuku 应用，根据您的设备选择启动方式：\n\n• 无线调试（推荐）：需要 Android 11+，在开发者选项中开启无线调试\n• 连接电脑：通过 ADB 命令启动" + ) + Spacer(modifier = Modifier.height(16.dp)) + HelpStep( + number = "3", + title = "授权肉包", + description = "在 Shizuku 的「应用管理」中找到「肉包」，点击授权按钮" + ) + Spacer(modifier = Modifier.height(16.dp)) + HelpStep( + number = "4", + title = "开始使用", + description = "授权完成后，返回肉包应用，即可开始使用" + ) + } + }, + confirmButton = { + TextButton(onClick = onDismiss) { + Text("知道了", color = colors.primary) + } + } + ) +} + +@Composable +fun OverlayHelpDialog(onDismiss: () -> Unit) { + val colors = BaoziTheme.colors + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + title = { + Text("悬浮窗权限说明", color = colors.textPrimary) + }, + text = { + Column( + modifier = Modifier.verticalScroll(rememberScrollState()) + ) { + Text( + text = "为什么需要悬浮窗权限？", + fontSize = 16.sp, + fontWeight = FontWeight.Medium, + color = colors.primary + ) + Spacer(modifier = Modifier.height(8.dp)) + Text( + text = "肉包在执行任务时需要显示悬浮窗来：", + fontSize = 14.sp, + color = colors.textPrimary + ) + Spacer(modifier = Modifier.height(8.dp)) + BulletPoint("显示当前执行进度") + BulletPoint("提供停止按钮，随时中断任务") + BulletPoint("在其他应用上方显示状态信息") + + Spacer(modifier = Modifier.height(16.dp)) + Text( + text = "如何开启？", + fontSize = 16.sp, + fontWeight = FontWeight.Medium, + color = colors.primary + ) + Spacer(modifier = Modifier.height(8.dp)) + Text( + text = "1. 点击执行任务时会自动提示\n2. 或前往：设置 > 应用 > 肉包 > 悬浮窗权限\n3. 开启「允许显示在其他应用上层」", + fontSize = 14.sp, + color = colors.textPrimary, + lineHeight = 22.sp + ) + + Spacer(modifier = Modifier.height(16.dp)) + Text( + text = "隐私安全", + fontSize = 16.sp, + fontWeight = FontWeight.Medium, + color = colors.primary + ) + Spacer(modifier = Modifier.height(8.dp)) + Text( + text = "悬浮窗仅在任务执行期间显示，不会收集任何个人信息。任务完成后悬浮窗会自动消失。", + fontSize = 14.sp, + color = colors.textSecondary + ) + } + }, + confirmButton = { + TextButton(onClick = onDismiss) { + Text("知道了", color = colors.primary) + } + } + ) +} + +@Composable +private fun HelpStep( + number: String, + title: String, + description: String +) { + val colors = BaoziTheme.colors + Row { + Box( + modifier = Modifier + .size(28.dp) + .clip(CircleShape) + .background(colors.primary), + contentAlignment = Alignment.Center + ) { + Text( + text = number, + fontSize = 14.sp, + fontWeight = FontWeight.Bold, + color = Color.White + ) + } + Spacer(modifier = Modifier.width(12.dp)) + Column(modifier = Modifier.weight(1f)) { + Text( + text = title, + fontSize = 15.sp, + fontWeight = FontWeight.Medium, + color = colors.textPrimary + ) + Spacer(modifier = Modifier.height(4.dp)) + Text( + text = description, + fontSize = 13.sp, + color = colors.textSecondary, + lineHeight = 20.sp + ) + } + } +} + +@Composable +private fun BulletPoint(text: String) { + val colors = BaoziTheme.colors + Row( + modifier = Modifier.padding(start = 8.dp, top = 4.dp) + ) { + Text( + text = "•", + fontSize = 14.sp, + color = colors.primary + ) + Spacer(modifier = Modifier.width(8.dp)) + Text( + text = text, + fontSize = 14.sp, + color = colors.textPrimary + ) + } +} + +@Composable +fun MaxStepsDialog( + currentSteps: Int, + onDismiss: () -> Unit, + onConfirm: (Int) -> Unit +) { + val colors = BaoziTheme.colors + var steps by remember { mutableStateOf(currentSteps.toFloat()) } + + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + title = { + Text("最大执行步数", color = colors.textPrimary) + }, + text = { + Column { + Text( + text = "设置 Agent 单次任务的最大执行步数。步数越多，能完成的任务越复杂，但消耗的 token 也越多。", + fontSize = 14.sp, + color = colors.textSecondary, + modifier = Modifier.padding(bottom = 16.dp) + ) + + // 当前值显示 + Text( + text = "${steps.toInt()} 步", + fontSize = 24.sp, + fontWeight = FontWeight.Bold, + color = colors.primary, + modifier = Modifier + .fillMaxWidth() + .padding(bottom = 8.dp), + textAlign = androidx.compose.ui.text.style.TextAlign.Center + ) + + // 滑块 + Slider( + value = steps, + onValueChange = { steps = it }, + valueRange = 5f..100f, + steps = 18, // (100-5)/5 - 1 = 18 个刻度点，每 5 步一个 + colors = SliderDefaults.colors( + thumbColor = colors.primary, + activeTrackColor = colors.primary, + inactiveTrackColor = colors.backgroundInput + ), + modifier = Modifier.fillMaxWidth() + ) + + // 范围提示 + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.SpaceBetween + ) { + Text( + text = "5", + fontSize = 12.sp, + color = colors.textHint + ) + Text( + text = "100", + fontSize = 12.sp, + color = colors.textHint + ) + } + + Spacer(modifier = Modifier.height(8.dp)) + + // 快捷选项 + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.spacedBy(8.dp) + ) { + listOf(15, 25, 50).forEach { preset -> + Surface( + modifier = Modifier + .weight(1f) + .clickable { steps = preset.toFloat() }, + shape = RoundedCornerShape(8.dp), + color = if (steps.toInt() == preset) colors.primary else colors.backgroundInput + ) { + Text( + text = "$preset", + fontSize = 14.sp, + color = if (steps.toInt() == preset) Color.White else colors.textSecondary, + modifier = Modifier.padding(vertical = 8.dp), + textAlign = androidx.compose.ui.text.style.TextAlign.Center + ) + } + } + } + } + }, + confirmButton = { + TextButton(onClick = { onConfirm(steps.toInt()) }) { + Text("确定", color = colors.primary) + } + }, + dismissButton = { + TextButton(onClick = onDismiss) { + Text("取消", color = colors.textSecondary) + } + } + ) +} + +/** + * 服务商选择对话框 + */ +@Composable +fun ProviderSelectDialog( + currentProviderId: String, + customBaseUrl: String, + onDismiss: () -> Unit, + onSelectProvider: (ApiProvider) -> Unit, + onUpdateCustomUrl: (String) -> Unit +) { + val colors = BaoziTheme.colors + var selectedProviderId by remember { mutableStateOf(currentProviderId) } + var customUrl by remember { mutableStateOf(customBaseUrl) } + + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + title = { + Text("API 服务商", color = colors.textPrimary) + }, + text = { + Column( + modifier = Modifier.verticalScroll(rememberScrollState()) + ) { + Text( + text = "选择 API 服务商（支持 OpenAI 兼容接口）", + fontSize = 14.sp, + color = colors.textSecondary, + modifier = Modifier.padding(bottom = 16.dp) + ) + + // 服务商列表 + ApiProvider.ALL.forEach { provider -> + val isSelected = provider.id == selectedProviderId + Surface( + modifier = Modifier + .fillMaxWidth() + .padding(vertical = 4.dp) + .clickable { + selectedProviderId = provider.id + onSelectProvider(provider) + }, + shape = RoundedCornerShape(8.dp), + color = if (isSelected) colors.primary.copy(alpha = 0.15f) else Color.Transparent + ) { + Column( + modifier = Modifier.padding(12.dp) + ) { + Row( + verticalAlignment = Alignment.CenterVertically + ) { + if (isSelected) { + Icon( + imageVector = Icons.Default.CheckCircle, + contentDescription = null, + tint = colors.primary, + modifier = Modifier.size(18.dp) + ) + } else { + Box( + modifier = Modifier + .size(18.dp) + .border(2.dp, colors.textHint, CircleShape) + ) + } + Spacer(modifier = Modifier.width(10.dp)) + Text( + text = provider.name, + fontSize = 14.sp, + color = if (isSelected) colors.primary else colors.textPrimary + ) + } + // 对于非自定义服务商，显示其 URL + if (provider.id != "custom") { + Text( + text = provider.baseUrl, + fontSize = 11.sp, + color = colors.textHint, + modifier = Modifier.padding(start = 28.dp, top = 2.dp) + ) + } + } + } + + // 自定义服务商或 MAI-UI 选中时显示 URL 输入框 + if ((provider.id == "custom" || provider.id == "mai_ui") && isSelected) { + Spacer(modifier = Modifier.height(8.dp)) + Box( + modifier = Modifier + .fillMaxWidth() + .padding(start = 28.dp) + .clip(RoundedCornerShape(12.dp)) + .background(colors.backgroundInput) + .padding(horizontal = 16.dp, vertical = 14.dp) + ) { + if (customUrl.isEmpty()) { + Text( + text = if (provider.id == "mai_ui") "http://your-server:8000/v1" else "https://api.example.com/v1", + color = colors.textHint, + fontSize = 14.sp + ) + } + BasicTextField( + value = customUrl, + onValueChange = { newUrl -> + customUrl = newUrl + onUpdateCustomUrl(newUrl) + }, + textStyle = TextStyle( + color = colors.textPrimary, + fontSize = 14.sp + ), + cursorBrush = SolidColor(colors.primary), + modifier = Modifier.fillMaxWidth(), + singleLine = true + ) + } + Text( + text = if (provider.id == "mai_ui") "留空使用默认地址 (localhost:8000)" else "输入自定义 API 端点地址", + fontSize = 11.sp, + color = colors.textHint, + modifier = Modifier.padding(start = 28.dp, top = 4.dp) + ) + } + } + } + }, + confirmButton = { + TextButton(onClick = onDismiss) { + Text("完成", color = colors.primary) + } + } + ) +} + +/** + * Root 模式警告对话框 + */ +@Composable +fun RootModeWarningDialog( + onDismiss: () -> Unit, + onConfirm: () -> Unit +) { + val colors = BaoziTheme.colors + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + icon = { + Icon( + imageVector = Icons.Default.Warning, + contentDescription = null, + tint = colors.error, + modifier = Modifier.size(48.dp) + ) + }, + title = { + Text( + "启用 Root 模式", + color = colors.error, + fontWeight = FontWeight.Bold + ) + }, + text = { + Column { + Text( + text = "Root 模式将允许应用使用更高级的系统权限。", + fontSize = 14.sp, + color = colors.textPrimary, + modifier = Modifier.padding(bottom = 12.dp) + ) + Text( + text = "警告：", + fontSize = 14.sp, + fontWeight = FontWeight.Bold, + color = colors.error + ) + Spacer(modifier = Modifier.height(8.dp)) + BulletPoint("Root 权限可能导致系统不稳定") + BulletPoint("不当操作可能损坏设备数据") + BulletPoint("请确保您了解 Root 权限的风险") + Spacer(modifier = Modifier.height(12.dp)) + Text( + text = "仅在您完全了解风险并需要高级功能时才启用此选项。", + fontSize = 13.sp, + color = colors.textSecondary + ) + } + }, + confirmButton = { + Button( + onClick = onConfirm, + colors = ButtonDefaults.buttonColors(containerColor = colors.error) + ) { + Text("我了解风险，启用", color = Color.White) + } + }, + dismissButton = { + TextButton(onClick = onDismiss) { + Text("取消", color = colors.textSecondary) + } + } + ) +} + +/** + * su -c 命令警告对话框 + */ +@Composable +fun SuCommandWarningDialog( + onDismiss: () -> Unit, + onConfirm: () -> Unit +) { + val colors = BaoziTheme.colors + AlertDialog( + onDismissRequest = onDismiss, + containerColor = colors.backgroundCard, + icon = { + Icon( + imageVector = Icons.Default.Warning, + contentDescription = null, + tint = colors.error, + modifier = Modifier.size(48.dp) + ) + }, + title = { + Text( + "允许 su -c 命令", + color = colors.error, + fontWeight = FontWeight.Bold + ) + }, + text = { + Column { + Text( + text = "此选项将允许 AI 执行 su -c 命令，这意味着 AI 可以以 Root 权限执行任意 Shell 命令。", + fontSize = 14.sp, + color = colors.textPrimary, + modifier = Modifier.padding(bottom = 12.dp) + ) + Text( + text = "极度危险：", + fontSize = 14.sp, + fontWeight = FontWeight.Bold, + color = colors.error + ) + Spacer(modifier = Modifier.height(8.dp)) + BulletPoint("AI 可能执行危险的系统命令") + BulletPoint("可能导致数据丢失或系统损坏") + BulletPoint("可能被恶意指令利用") + BulletPoint("不建议在日常使用中启用") + Spacer(modifier = Modifier.height(12.dp)) + Text( + text = "强烈建议：仅在完全可控的测试环境中使用，并在使用完毕后立即关闭。", + fontSize = 13.sp, + color = colors.error, + fontWeight = FontWeight.Medium + ) + } + }, + confirmButton = { + Button( + onClick = onConfirm, + colors = ButtonDefaults.buttonColors(containerColor = colors.error) + ) { + Text("我了解风险，启用", color = Color.White) + } + }, + dismissButton = { + TextButton(onClick = onDismiss) { + Text("取消", color = colors.textSecondary) + } + } + ) +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/ui/theme/Theme.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/theme/Theme.kt new file mode 100644 index 0000000..55e943e --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/ui/theme/Theme.kt @@ -0,0 +1,180 @@ +package com.roubao.autopilot.ui.theme + +import androidx.compose.foundation.isSystemInDarkTheme +import androidx.compose.material3.* +import androidx.compose.runtime.Composable +import androidx.compose.runtime.CompositionLocalProvider +import androidx.compose.runtime.staticCompositionLocalOf +import androidx.compose.ui.graphics.Color + +// 主色调 +val Primary = Color(0xFFD67744) +val PrimaryDark = Color(0xFFB85E2E) +val PrimaryLight = Color(0xFFE89060) +val Secondary = Color(0xFFEFB773) +val SecondaryDark = Color(0xFFD69B52) +val SecondaryLight = Color(0xFFF5CB94) + +// 深色主题背景色 +val BackgroundDark = Color(0xFF1A1A1A) +val BackgroundCard = Color(0xFF252525) +val BackgroundInput = Color(0xFF2A2A2A) +val SurfaceVariant = Color(0xFF303030) + +// 深色主题文字颜色 +val TextPrimary = Color(0xFFFFFFFF) +val TextSecondary = Color(0xFFB0B0B0) +val TextHint = Color(0xFF666666) + +// 浅色主题背景色 - 纯白简洁风格 +val BackgroundLight = Color(0xFFFFFFFF) +val BackgroundCardLight = Color(0xFFF5F5F5) +val BackgroundInputLight = Color(0xFFEEEEEE) +val SurfaceVariantLight = Color(0xFFE0E0E0) + +// 浅色主题文字颜色 +val TextPrimaryLight = Color(0xFF212121) +val TextSecondaryLight = Color(0xFF757575) +val TextHintLight = Color(0xFFBDBDBD) + +// 状态颜色 +val Success = Color(0xFF4CAF50) +val Error = Color(0xFFF44336) +val Warning = Color(0xFFFF9800) + +// 主题颜色数据类 +data class BaoziColors( + val primary: Color, + val primaryDark: Color, + val primaryLight: Color, + val secondary: Color, + val background: Color, + val backgroundCard: Color, + val backgroundInput: Color, + val surfaceVariant: Color, + val textPrimary: Color, + val textSecondary: Color, + val textHint: Color, + val success: Color, + val error: Color, + val warning: Color, + val isDark: Boolean +) + +// 深色主题颜色 +val DarkBaoziColors = BaoziColors( + primary = Primary, + primaryDark = PrimaryDark, + primaryLight = PrimaryLight, + secondary = Secondary, + background = BackgroundDark, + backgroundCard = BackgroundCard, + backgroundInput = BackgroundInput, + surfaceVariant = SurfaceVariant, + textPrimary = TextPrimary, + textSecondary = TextSecondary, + textHint = TextHint, + success = Success, + error = Error, + warning = Warning, + isDark = true +) + +// 浅色主题颜色 +val LightBaoziColors = BaoziColors( + primary = Primary, + primaryDark = PrimaryDark, + primaryLight = PrimaryLight, + secondary = Secondary, + background = BackgroundLight, + backgroundCard = BackgroundCardLight, + backgroundInput = BackgroundInputLight, + surfaceVariant = SurfaceVariantLight, + textPrimary = TextPrimaryLight, + textSecondary = TextSecondaryLight, + textHint = TextHintLight, + success = Success, + error = Error, + warning = Warning, + isDark = false +) + +// CompositionLocal 用于访问当前主题颜色 +val LocalBaoziColors = staticCompositionLocalOf { DarkBaoziColors } + +// Material 3 深色配色方案 +private val DarkColorScheme = darkColorScheme( + primary = Primary, + onPrimary = Color.White, + primaryContainer = PrimaryDark, + onPrimaryContainer = Color.White, + secondary = Secondary, + onSecondary = Color.Black, + secondaryContainer = SecondaryDark, + onSecondaryContainer = Color.White, + background = BackgroundDark, + onBackground = TextPrimary, + surface = BackgroundCard, + onSurface = TextPrimary, + surfaceVariant = SurfaceVariant, + onSurfaceVariant = TextSecondary, + error = Error, + onError = Color.White +) + +// Material 3 浅色配色方案 +private val LightColorScheme = lightColorScheme( + primary = Primary, + onPrimary = Color.White, + primaryContainer = PrimaryLight, + onPrimaryContainer = Color.Black, + secondary = Secondary, + onSecondary = Color.Black, + secondaryContainer = SecondaryLight, + onSecondaryContainer = Color.Black, + background = BackgroundLight, + onBackground = TextPrimaryLight, + surface = BackgroundCardLight, + onSurface = TextPrimaryLight, + surfaceVariant = SurfaceVariantLight, + onSurfaceVariant = TextSecondaryLight, + error = Error, + onError = Color.White +) + +// 主题模式枚举 +enum class ThemeMode { + LIGHT, + DARK, + SYSTEM +} + +@Composable +fun BaoziTheme( + themeMode: ThemeMode = ThemeMode.DARK, + content: @Composable () -> Unit +) { + val isDarkTheme = when (themeMode) { + ThemeMode.LIGHT -> false + ThemeMode.DARK -> true + ThemeMode.SYSTEM -> isSystemInDarkTheme() + } + + val colorScheme = if (isDarkTheme) DarkColorScheme else LightColorScheme + val baoziColors = if (isDarkTheme) DarkBaoziColors else LightBaoziColors + + CompositionLocalProvider(LocalBaoziColors provides baoziColors) { + MaterialTheme( + colorScheme = colorScheme, + typography = Typography(), + content = content + ) + } +} + +// 便捷访问当前主题颜色 +object BaoziTheme { + val colors: BaoziColors + @Composable + get() = LocalBaoziColors.current +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/utils/CrashHandler.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/utils/CrashHandler.kt new file mode 100644 index 0000000..ccda35c --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/utils/CrashHandler.kt @@ -0,0 +1,245 @@ +package com.roubao.autopilot.utils + +import android.content.Context +import android.content.Intent +import android.os.Build +import androidx.core.content.FileProvider +import java.io.File +import java.io.FileWriter +import java.io.PrintWriter +import java.text.SimpleDateFormat +import java.util.* + +/** + * 全局崩溃捕获器 + * 捕获未处理的异常并保存到本地文件 + */ +class CrashHandler private constructor() : Thread.UncaughtExceptionHandler { + + private var context: Context? = null + private var defaultHandler: Thread.UncaughtExceptionHandler? = null + + companion object { + private const val LOG_DIR = "crash_logs" + private const val MAX_LOG_FILES = 10 // 最多保留10个日志文件 + + @Volatile + private var instance: CrashHandler? = null + + fun getInstance(): CrashHandler { + return instance ?: synchronized(this) { + instance ?: CrashHandler().also { instance = it } + } + } + + /** + * 获取日志目录 + */ + fun getLogDir(context: Context): File { + val dir = File(context.filesDir, LOG_DIR) + if (!dir.exists()) { + dir.mkdirs() + } + return dir + } + + /** + * 获取所有日志文件 + */ + fun getLogFiles(context: Context): List { + val dir = getLogDir(context) + return dir.listFiles() + ?.filter { it.isFile && it.name.endsWith(".log") } + ?.sortedByDescending { it.lastModified() } + ?: emptyList() + } + + /** + * 导出日志（合并所有日志到一个文件） + */ + fun exportLogs(context: Context): File? { + val logFiles = getLogFiles(context) + if (logFiles.isEmpty()) { + return null + } + + val exportFile = File(context.cacheDir, "roubao_logs_${System.currentTimeMillis()}.txt") + try { + FileWriter(exportFile).use { writer -> + // 写入设备信息 + writer.write("========== 设备信息 ==========\n") + writer.write("设备: ${Build.MANUFACTURER} ${Build.MODEL}\n") + writer.write("Android: ${Build.VERSION.RELEASE} (API ${Build.VERSION.SDK_INT})\n") + writer.write("应用版本: ${getAppVersion(context)}\n") + writer.write("导出时间: ${SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.getDefault()).format(Date())}\n") + writer.write("\n") + + // 合并所有日志 + logFiles.forEach { file -> + writer.write("========== ${file.name} ==========\n") + writer.write(file.readText()) + writer.write("\n\n") + } + } + return exportFile + } catch (e: Exception) { + e.printStackTrace() + return null + } + } + + /** + * 分享日志文件 + */ + fun shareLogs(context: Context) { + val exportFile = exportLogs(context) + if (exportFile == null) { + android.widget.Toast.makeText(context, "没有日志可导出", android.widget.Toast.LENGTH_SHORT).show() + return + } + + try { + val uri = FileProvider.getUriForFile( + context, + "${context.packageName}.fileprovider", + exportFile + ) + + val intent = Intent(Intent.ACTION_SEND).apply { + type = "text/plain" + putExtra(Intent.EXTRA_STREAM, uri) + putExtra(Intent.EXTRA_SUBJECT, "肉包 App 日志") + putExtra(Intent.EXTRA_TEXT, "请查看附件中的日志文件") + addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION) + } + + context.startActivity(Intent.createChooser(intent, "分享日志")) + } catch (e: Exception) { + e.printStackTrace() + android.widget.Toast.makeText(context, "分享失败: ${e.message}", android.widget.Toast.LENGTH_SHORT).show() + } + } + + /** + * 清除所有日志 + */ + fun clearLogs(context: Context) { + val dir = getLogDir(context) + dir.listFiles()?.forEach { it.delete() } + } + + /** + * 获取日志统计信息 + */ + fun getLogStats(context: Context): String { + val files = getLogFiles(context) + if (files.isEmpty()) { + return "暂无日志" + } + val totalSize = files.sumOf { it.length() } + val sizeStr = if (totalSize > 1024) "${totalSize / 1024} KB" else "$totalSize B" + return "${files.size} 个文件, $sizeStr" + } + + private fun getAppVersion(context: Context): String { + return try { + val pInfo = context.packageManager.getPackageInfo(context.packageName, 0) + "${pInfo.versionName} (${pInfo.longVersionCode})" + } catch (e: Exception) { + "Unknown" + } + } + } + + /** + * 初始化 + */ + fun init(context: Context) { + this.context = context.applicationContext + defaultHandler = Thread.getDefaultUncaughtExceptionHandler() + Thread.setDefaultUncaughtExceptionHandler(this) + + // 清理旧日志 + cleanOldLogs() + } + + override fun uncaughtException(thread: Thread, throwable: Throwable) { + // 保存崩溃日志 + saveCrashLog(throwable) + + // 调用默认处理器（让系统显示崩溃对话框或直接退出） + defaultHandler?.uncaughtException(thread, throwable) + } + + /** + * 保存崩溃日志 + */ + private fun saveCrashLog(throwable: Throwable) { + val ctx = context ?: return + + try { + val timestamp = SimpleDateFormat("yyyyMMdd_HHmmss", Locale.getDefault()).format(Date()) + val fileName = "crash_$timestamp.log" + val file = File(getLogDir(ctx), fileName) + + PrintWriter(FileWriter(file)).use { writer -> + // 时间 + writer.println("========== 崩溃时间 ==========") + writer.println(SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.getDefault()).format(Date())) + writer.println() + + // 设备信息 + writer.println("========== 设备信息 ==========") + writer.println("设备: ${Build.MANUFACTURER} ${Build.MODEL}") + writer.println("Android: ${Build.VERSION.RELEASE} (API ${Build.VERSION.SDK_INT})") + writer.println("CPU ABI: ${Build.SUPPORTED_ABIS.joinToString()}") + writer.println("应用版本: ${getAppVersion(ctx)}") + writer.println() + + // 异常信息 + writer.println("========== 异常信息 ==========") + throwable.printStackTrace(writer) + } + + println("[CrashHandler] 崩溃日志已保存: $fileName") + } catch (e: Exception) { + e.printStackTrace() + } + } + + /** + * 记录普通日志（非崩溃） + */ + fun log(tag: String, message: String, throwable: Throwable? = null) { + val ctx = context ?: return + + try { + val today = SimpleDateFormat("yyyyMMdd", Locale.getDefault()).format(Date()) + val fileName = "log_$today.log" + val file = File(getLogDir(ctx), fileName) + + FileWriter(file, true).use { writer -> + val time = SimpleDateFormat("HH:mm:ss", Locale.getDefault()).format(Date()) + writer.appendLine("[$time] [$tag] $message") + throwable?.let { + val sw = java.io.StringWriter() + it.printStackTrace(PrintWriter(sw)) + writer.appendLine(sw.toString()) + } + } + } catch (e: Exception) { + e.printStackTrace() + } + } + + /** + * 清理旧日志，只保留最近的文件 + */ + private fun cleanOldLogs() { + val ctx = context ?: return + val files = getLogFiles(ctx) + if (files.size > MAX_LOG_FILES) { + files.drop(MAX_LOG_FILES).forEach { it.delete() } + } + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/GUIOwlClient.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/GUIOwlClient.kt new file mode 100644 index 0000000..a5e2231 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/GUIOwlClient.kt @@ -0,0 +1,295 @@ +package com.roubao.autopilot.vlm + +import android.graphics.Bitmap +import android.util.Base64 +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.delay +import kotlinx.coroutines.withContext +import okhttp3.ConnectionPool +import okhttp3.MediaType.Companion.toMediaType +import okhttp3.OkHttpClient +import okhttp3.Request +import okhttp3.RequestBody.Companion.toRequestBody +import org.json.JSONArray +import org.json.JSONObject +import java.io.ByteArrayOutputStream +import java.util.concurrent.TimeUnit + +/** + * GUI-Owl API 客户端 + * 专用于阿里云 GUI Agent 服务（非 OpenAI 兼容格式） + * + * API 特点： + * - 专用端点: https://dashscope.aliyuncs.com/api/v2/apps/gui-owl/gui_agent_server + * - 返回直接的操作指令: Click(x,y), Swipe(x1,y1,x2,y2) 等 + * - 支持 session_id 管理多轮操作 + */ +class GUIOwlClient( + private val apiKey: String, + private val model: String = "pre-gui_owl_7b", + private val deviceType: String = "mobile", + private val thoughtLanguage: String = "chinese" +) { + companion object { + private const val TAG = "GUIOwlClient" + private const val ENDPOINT = "https://dashscope.aliyuncs.com/api/v2/apps/gui-owl/gui_agent_server" + private const val MAX_RETRIES = 3 + private const val RETRY_DELAY_MS = 1000L + } + + private val client = OkHttpClient.Builder() + .connectTimeout(30, TimeUnit.SECONDS) + .readTimeout(90, TimeUnit.SECONDS) + .writeTimeout(60, TimeUnit.SECONDS) + .retryOnConnectionFailure(true) + .connectionPool(ConnectionPool(5, 1, TimeUnit.MINUTES)) + .build() + + // 会话 ID（用于关联多轮操作） + private var sessionId: String = "" + + /** + * GUI-Owl 响应结果 + */ + data class GUIOwlResponse( + val thought: String, // 思考过程 + val operation: String, // 操作指令: Click (x, y, x, y) / Swipe (x1, y1, x2, y2) 等 + val explanation: String, // 操作说明 + val sessionId: String, // 会话 ID + val rawResponse: String // 原始响应 + ) + + /** + * 解析操作指令为 Action + */ + data class ParsedAction( + val type: String, // click, swipe, type, etc. + val x: Int? = null, + val y: Int? = null, + val x2: Int? = null, + val y2: Int? = null, + val text: String? = null + ) + + /** + * 调用 GUI-Owl 进行界面理解和操作推理 + * + * @param instruction 用户指令 + * @param imageUrl 截图 URL（支持 http/https 或 data:image/... base64） + * @param addInfo 额外的操作提示信息 + * @return GUIOwlResponse + */ + suspend fun predict( + instruction: String, + imageUrl: String, + addInfo: String = "" + ): Result = withContext(Dispatchers.IO) { + var lastException: Exception? = null + + for (attempt in 1..MAX_RETRIES) { + try { + val messagesArray = JSONArray().apply { + put(JSONObject().put("image", imageUrl)) + put(JSONObject().put("instruction", instruction)) + put(JSONObject().put("session_id", sessionId)) + put(JSONObject().put("device_type", deviceType)) + put(JSONObject().put("pipeline_type", "agent")) + put(JSONObject().put("model_name", model)) + put(JSONObject().put("thought_language", thoughtLanguage)) + put(JSONObject().put("param_list", JSONArray().apply { + put(JSONObject().put("add_info", addInfo)) + })) + } + + val dataObj = JSONObject().apply { + put("messages", messagesArray) + } + + val contentArray = JSONArray().apply { + put(JSONObject().apply { + put("type", "data") + put("data", dataObj) + }) + } + + val inputArray = JSONArray().apply { + put(JSONObject().apply { + put("role", "user") + put("content", contentArray) + }) + } + + val requestBody = JSONObject().apply { + put("app_id", "gui-owl") + put("input", inputArray) + } + + val request = Request.Builder() + .url(ENDPOINT) + .addHeader("Authorization", "Bearer $apiKey") + .addHeader("Content-Type", "application/json") + .post(requestBody.toString().toRequestBody("application/json".toMediaType())) + .build() + + println("[$TAG] 请求: instruction=$instruction") + val response = client.newCall(request).execute() + val responseBody = response.body?.string() ?: "" + + if (response.isSuccessful) { + val json = JSONObject(responseBody) + + // 更新 session_id + val newSessionId = json.optString("session_id", "") + if (newSessionId.isNotEmpty()) { + sessionId = newSessionId + } + + // 解析 output + val outputArray = json.optJSONArray("output") + if (outputArray != null && outputArray.length() > 0) { + val output = outputArray.getJSONObject(0) + val contentArr = output.optJSONArray("content") + + if (contentArr != null && contentArr.length() > 0) { + val content = contentArr.getJSONObject(0) + val data = content.optJSONObject("data") + + if (data != null) { + val result = GUIOwlResponse( + thought = data.optString("Thought", ""), + operation = data.optString("Operation", ""), + explanation = data.optString("Explanation", ""), + sessionId = sessionId, + rawResponse = responseBody + ) + println("[$TAG] 响应: operation=${result.operation}") + return@withContext Result.success(result) + } + } + } + + lastException = Exception("Invalid response format: $responseBody") + } else { + lastException = Exception("API error: ${response.code} - $responseBody") + } + } catch (e: Exception) { + println("[$TAG] 请求失败 (attempt $attempt): ${e.message}") + lastException = e + if (attempt < MAX_RETRIES) { + delay(RETRY_DELAY_MS * attempt) + } + } + } + + Result.failure(lastException ?: Exception("Unknown error")) + } + + /** + * 使用 Bitmap 调用 GUI-Owl + * 自动将 Bitmap 转换为 base64 data URL + */ + suspend fun predict( + instruction: String, + image: Bitmap, + addInfo: String = "" + ): Result { + val imageUrl = bitmapToDataUrl(image) + return predict(instruction, imageUrl, addInfo) + } + + /** + * 解析操作指令字符串为 ParsedAction + * + * 支持的格式: + * - Click (x, y, x, y) 或 Click (x, y) + * - Swipe (x1, y1, x2, y2) + * - Type (text) + * - Long_press (x, y) + * - Scroll (direction) + */ + fun parseOperation(operation: String): ParsedAction? { + val trimmed = operation.trim() + + // Click (x, y, x, y) 或 Click (x, y) + val clickPattern = Regex("""Click\s*\(\s*(\d+)\s*,\s*(\d+)(?:\s*,\s*\d+\s*,\s*\d+)?\s*\)""", RegexOption.IGNORE_CASE) + clickPattern.find(trimmed)?.let { match -> + val x = match.groupValues[1].toIntOrNull() ?: return null + val y = match.groupValues[2].toIntOrNull() ?: return null + return ParsedAction(type = "click", x = x, y = y) + } + + // Swipe (x1, y1, x2, y2) + val swipePattern = Regex("""Swipe\s*\(\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*\)""", RegexOption.IGNORE_CASE) + swipePattern.find(trimmed)?.let { match -> + val x1 = match.groupValues[1].toIntOrNull() ?: return null + val y1 = match.groupValues[2].toIntOrNull() ?: return null + val x2 = match.groupValues[3].toIntOrNull() ?: return null + val y2 = match.groupValues[4].toIntOrNull() ?: return null + return ParsedAction(type = "swipe", x = x1, y = y1, x2 = x2, y2 = y2) + } + + // Long_press (x, y) 或 LongPress (x, y) + val longPressPattern = Regex("""Long[_\s]?[Pp]ress\s*\(\s*(\d+)\s*,\s*(\d+)\s*\)""", RegexOption.IGNORE_CASE) + longPressPattern.find(trimmed)?.let { match -> + val x = match.groupValues[1].toIntOrNull() ?: return null + val y = match.groupValues[2].toIntOrNull() ?: return null + return ParsedAction(type = "long_press", x = x, y = y) + } + + // Type (text) 或 Input (text) + val typePattern = Regex("""(?:Type|Input)\s*\(\s*["\']?(.+?)["\']?\s*\)""", RegexOption.IGNORE_CASE) + typePattern.find(trimmed)?.let { match -> + val text = match.groupValues[1] + return ParsedAction(type = "type", text = text) + } + + // Scroll (direction) 或 Scroll_down / Scroll_up + val scrollPattern = Regex("""Scroll[_\s]?(up|down|left|right)?""", RegexOption.IGNORE_CASE) + scrollPattern.find(trimmed)?.let { match -> + val direction = match.groupValues.getOrNull(1)?.lowercase() ?: "down" + return ParsedAction(type = "scroll", text = direction) + } + + // Back + if (trimmed.contains("Back", ignoreCase = true)) { + return ParsedAction(type = "system_button", text = "Back") + } + + // Home + if (trimmed.contains("Home", ignoreCase = true)) { + return ParsedAction(type = "system_button", text = "Home") + } + + // FINISH / DONE / COMPLETE + if (trimmed.contains(Regex("FINISH|DONE|COMPLETE|Finished", RegexOption.IGNORE_CASE))) { + return ParsedAction(type = "finish") + } + + println("[$TAG] 无法解析操作: $operation") + return null + } + + /** + * 重置会话（开始新任务时调用） + */ + fun resetSession() { + sessionId = "" + } + + /** + * 获取当前会话 ID + */ + fun getSessionId(): String = sessionId + + /** + * Bitmap 转换为 data URL + */ + private fun bitmapToDataUrl(bitmap: Bitmap): String { + val outputStream = ByteArrayOutputStream() + bitmap.compress(Bitmap.CompressFormat.JPEG, 70, outputStream) + val bytes = outputStream.toByteArray() + println("[$TAG] 图片压缩: ${bitmap.width}x${bitmap.height}, ${bytes.size / 1024}KB") + val base64 = Base64.encodeToString(bytes, Base64.NO_WRAP) + return "data:image/jpeg;base64,$base64" + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/MAIUIClient.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/MAIUIClient.kt new file mode 100644 index 0000000..d35a18a --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/MAIUIClient.kt @@ -0,0 +1,390 @@ +package com.roubao.autopilot.vlm + +import android.graphics.Bitmap +import android.util.Base64 +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.delay +import kotlinx.coroutines.withContext +import okhttp3.ConnectionPool +import okhttp3.MediaType.Companion.toMediaType +import okhttp3.OkHttpClient +import okhttp3.Request +import okhttp3.RequestBody.Companion.toRequestBody +import org.json.JSONArray +import org.json.JSONObject +import java.io.ByteArrayOutputStream +import java.util.concurrent.TimeUnit + +/** + * MAI-UI 专用客户端 + * 实现 MAI-UI 的特定 prompt 格式和对话历史管理 + */ +class MAIUIClient( + private val baseUrl: String = "http://localhost:8000/v1", + private val model: String = "MAI-UI-2B", + private val historyN: Int = 3 // 保留的历史图片数量 +) { + companion object { + private const val SCALE_FACTOR = 999 + private const val MAX_RETRIES = 3 + private const val RETRY_DELAY_MS = 1000L + } + + private val client = OkHttpClient.Builder() + .connectTimeout(30, TimeUnit.SECONDS) + .readTimeout(120, TimeUnit.SECONDS) + .writeTimeout(60, TimeUnit.SECONDS) + .retryOnConnectionFailure(true) + .connectionPool(ConnectionPool(5, 1, TimeUnit.MINUTES)) + .build() + + // 对话历史 + private val historyImages = mutableListOf() // Base64 encoded images + private val historyResponses = mutableListOf() // Assistant responses + + // 可用应用列表 (会在运行时更新) + private var availableApps: List = emptyList() + + /** + * 系统提示词 (参考 MAI-UI 官方实现) + */ + private fun getSystemPrompt(): String = """ +You are a GUI agent. You are given a task and your action history, with screenshots. You need to perform the next action to complete the task. + +## Output Format +For each function call, return the thinking process in tags, and a json object with function name and arguments within XML tags: +``` + +... + + +{"name": "mobile_use", "arguments": } + +``` + +## Action Space + +{"action": "click", "coordinate": [x, y]} +{"action": "long_press", "coordinate": [x, y]} +{"action": "type", "text": ""} +{"action": "swipe", "direction": "up or down or left or right", "coordinate": [x, y]} +{"action": "open", "text": "app_name"} +{"action": "drag", "start_coordinate": [x1, y1], "end_coordinate": [x2, y2]} +{"action": "system_button", "button": "button_name"} +{"action": "wait"} +{"action": "terminate", "status": "success or fail"} +{"action": "answer", "text": "xxx"} +{"action": "ask_user", "text": "xxx"} + +## Note +- Write a small plan and finally summarize your next action (with its target element) in one sentence in part. +- Available Apps: `${if (availableApps.isNotEmpty()) availableApps.toString() else "[请通过open动作打开应用]"}`. +- You should use the `open` action to open the app as possible as you can, because it is the fast way to open the app. +- You must follow the Action Space strictly, and return the correct json object within and XML tags. + """.trimIndent() + + /** + * 设置可用应用列表 + */ + fun setAvailableApps(apps: List) { + availableApps = apps + } + + /** + * 重置对话历史 + */ + fun reset() { + historyImages.clear() + historyResponses.clear() + } + + /** + * 预测下一步动作 + * @param instruction 用户指令 + * @param screenshot 当前截图 + * @return Result + */ + suspend fun predict( + instruction: String, + screenshot: Bitmap + ): Result = withContext(Dispatchers.IO) { + var lastException: Exception? = null + + // 编码当前截图 + val currentImageBase64 = bitmapToBase64(screenshot) + + for (attempt in 1..MAX_RETRIES) { + try { + // 构建消息 + val messages = buildMessages(instruction, currentImageBase64) + + val requestBody = JSONObject().apply { + put("model", model) + put("messages", messages) + put("max_tokens", 2048) + put("temperature", 0.0) + put("top_p", 1.0) + } + + val request = Request.Builder() + .url("${normalizeUrl(baseUrl)}/chat/completions") + .addHeader("Content-Type", "application/json") + .post(requestBody.toString().toRequestBody("application/json".toMediaType())) + .build() + + val response = client.newCall(request).execute() + val responseBody = response.body?.string() ?: "" + + if (response.isSuccessful) { + val json = JSONObject(responseBody) + val choices = json.getJSONArray("choices") + if (choices.length() > 0) { + val message = choices.getJSONObject(0).getJSONObject("message") + val content = message.getString("content") + + println("[MAIUIClient] Raw response: $content") + + // 解析响应 + val parsed = parseResponse(content) + + // 保存到历史 + historyImages.add(currentImageBase64) + historyResponses.add(content) + + // 限制历史数量 + while (historyImages.size > historyN) { + historyImages.removeAt(0) + historyResponses.removeAt(0) + } + + return@withContext Result.success(parsed) + } else { + lastException = Exception("No response from model") + } + } else { + lastException = Exception("API error: ${response.code} - $responseBody") + } + } catch (e: Exception) { + println("[MAIUIClient] Error on attempt $attempt: ${e.message}") + lastException = e + if (attempt < MAX_RETRIES) { + delay(RETRY_DELAY_MS * attempt) + } + } + } + + Result.failure(lastException ?: Exception("Unknown error")) + } + + /** + * 构建消息列表 (参考 MAI-UI 官方实现) + */ + private fun buildMessages(instruction: String, currentImageBase64: String): JSONArray { + val messages = JSONArray() + + // 1. System message + messages.put(JSONObject().apply { + put("role", "system") + put("content", JSONArray().apply { + put(JSONObject().apply { + put("type", "text") + put("text", getSystemPrompt()) + }) + }) + }) + + // 2. User instruction + messages.put(JSONObject().apply { + put("role", "user") + put("content", JSONArray().apply { + put(JSONObject().apply { + put("type", "text") + put("text", instruction) + }) + }) + }) + + // 3. History (image + assistant response pairs) + val startIdx = maxOf(0, historyImages.size - (historyN - 1)) + for (i in startIdx until historyImages.size) { + // User message with image + messages.put(JSONObject().apply { + put("role", "user") + put("content", JSONArray().apply { + put(JSONObject().apply { + put("type", "image_url") + put("image_url", JSONObject().apply { + put("url", "data:image/jpeg;base64,${historyImages[i]}") + }) + }) + }) + }) + + // Assistant response + messages.put(JSONObject().apply { + put("role", "assistant") + put("content", JSONArray().apply { + put(JSONObject().apply { + put("type", "text") + put("text", historyResponses[i]) + }) + }) + }) + } + + // 4. Current image + messages.put(JSONObject().apply { + put("role", "user") + put("content", JSONArray().apply { + put(JSONObject().apply { + put("type", "image_url") + put("image_url", JSONObject().apply { + put("url", "data:image/jpeg;base64,$currentImageBase64") + }) + }) + }) + }) + + return messages + } + + /** + * 解析 MAI-UI 响应 + */ + private fun parseResponse(text: String): MAIUIResponse { + var processedText = text.trim() + + // 处理 thinking model 输出格式 ( instead of ) + if (processedText.contains("") && !processedText.contains("")) { + processedText = processedText.replace("", "") + processedText = "$processedText" + } + + // 提取 thinking + val thinkingRegex = Regex("(.*?)", RegexOption.DOT_MATCHES_ALL) + val thinking = thinkingRegex.find(processedText)?.groupValues?.get(1)?.trim() ?: "" + + // 提取 tool_call + val toolCallRegex = Regex("\\s*(.+?)\\s*", RegexOption.DOT_MATCHES_ALL) + val toolCallMatch = toolCallRegex.find(processedText) + + var action: MAIUIAction? = null + if (toolCallMatch != null) { + try { + val toolCallJson = JSONObject(toolCallMatch.groupValues[1].trim()) + val arguments = toolCallJson.optJSONObject("arguments") + if (arguments != null) { + action = parseAction(arguments) + } + } catch (e: Exception) { + println("[MAIUIClient] Failed to parse tool_call: ${e.message}") + } + } + + return MAIUIResponse( + thinking = thinking, + action = action, + rawResponse = text + ) + } + + /** + * 解析动作 + */ + private fun parseAction(arguments: JSONObject): MAIUIAction { + val actionType = arguments.optString("action", "") + + // 解析坐标 (0-999 归一化) + val coordinate = arguments.optJSONArray("coordinate") + var x: Float? = null + var y: Float? = null + if (coordinate != null && coordinate.length() >= 2) { + x = coordinate.getDouble(0).toFloat() / SCALE_FACTOR + y = coordinate.getDouble(1).toFloat() / SCALE_FACTOR + } + + // 解析 drag 坐标 + val startCoord = arguments.optJSONArray("start_coordinate") + val endCoord = arguments.optJSONArray("end_coordinate") + var startX: Float? = null + var startY: Float? = null + var endX: Float? = null + var endY: Float? = null + if (startCoord != null && endCoord != null) { + startX = startCoord.getDouble(0).toFloat() / SCALE_FACTOR + startY = startCoord.getDouble(1).toFloat() / SCALE_FACTOR + endX = endCoord.getDouble(0).toFloat() / SCALE_FACTOR + endY = endCoord.getDouble(1).toFloat() / SCALE_FACTOR + } + + return MAIUIAction( + type = actionType, + x = x, + y = y, + startX = startX, + startY = startY, + endX = endX, + endY = endY, + text = arguments.optString("text", null), + button = arguments.optString("button", null), + direction = arguments.optString("direction", null), + status = arguments.optString("status", null) + ) + } + + private fun bitmapToBase64(bitmap: Bitmap): String { + val outputStream = ByteArrayOutputStream() + bitmap.compress(Bitmap.CompressFormat.JPEG, 70, outputStream) + val bytes = outputStream.toByteArray() + println("[MAIUIClient] Image compressed: ${bitmap.width}x${bitmap.height}, ${bytes.size / 1024}KB") + return Base64.encodeToString(bytes, Base64.NO_WRAP) + } + + private fun normalizeUrl(url: String): String { + var normalized = url.trim().removeSuffix("/") + if (!normalized.startsWith("http://") && !normalized.startsWith("https://")) { + normalized = "http://$normalized" + } + return normalized + } +} + +/** + * MAI-UI 响应 + */ +data class MAIUIResponse( + val thinking: String, + val action: MAIUIAction?, + val rawResponse: String +) + +/** + * MAI-UI 动作 (坐标已归一化为 0-1) + */ +data class MAIUIAction( + val type: String, + val x: Float? = null, // 归一化坐标 0-1 + val y: Float? = null, + val startX: Float? = null, // drag 起点 + val startY: Float? = null, + val endX: Float? = null, // drag 终点 + val endY: Float? = null, + val text: String? = null, + val button: String? = null, + val direction: String? = null, // swipe 方向: up, down, left, right + val status: String? = null // terminate 状态: success, fail +) { + /** + * 转换为屏幕像素坐标 + */ + fun toScreenCoordinates(screenWidth: Int, screenHeight: Int): MAIUIAction { + return copy( + x = x?.let { it * screenWidth }, + y = y?.let { it * screenHeight }, + startX = startX?.let { it * screenWidth }, + startY = startY?.let { it * screenHeight }, + endX = endX?.let { it * screenWidth }, + endY = endY?.let { it * screenHeight } + ) + } +} diff --git a/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/VLMClient.kt b/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/VLMClient.kt new file mode 100644 index 0000000..406e879 --- /dev/null +++ b/android-buyer/app/src/main/java/com/roubao/autopilot/vlm/VLMClient.kt @@ -0,0 +1,351 @@ +package com.roubao.autopilot.vlm + +import android.graphics.Bitmap +import android.util.Base64 +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.delay +import kotlinx.coroutines.withContext +import okhttp3.ConnectionPool +import okhttp3.MediaType.Companion.toMediaType +import okhttp3.OkHttpClient +import okhttp3.Request +import okhttp3.RequestBody.Companion.toRequestBody +import org.json.JSONArray +import org.json.JSONObject +import java.io.ByteArrayOutputStream +import java.net.UnknownHostException +import java.util.concurrent.TimeUnit + +/** + * VLM (Vision Language Model) API 客户端 + * 支持 OpenAI 兼容接口 (GPT-4V, Qwen-VL, Claude, etc.) + */ +class VLMClient( + private val apiKey: String, + baseUrl: String = "https://api.openai.com/v1", + private val model: String = "gpt-4-vision-preview" +) { + // 规范化 URL：自动添加 https:// 前缀，移除末尾斜杠 + private val baseUrl: String = normalizeUrl(baseUrl) + + private val client = OkHttpClient.Builder() + .connectTimeout(30, TimeUnit.SECONDS) + .readTimeout(90, TimeUnit.SECONDS) + .writeTimeout(60, TimeUnit.SECONDS) + .retryOnConnectionFailure(true) + .connectionPool(ConnectionPool(5, 1, TimeUnit.MINUTES)) + .build() + + companion object { + private const val MAX_RETRIES = 3 + private const val RETRY_DELAY_MS = 1000L + + /** 规范化 URL：自动添加 https:// 前缀，移除末尾斜杠 */ + private fun normalizeUrl(url: String): String { + var normalized = url.trim().removeSuffix("/") + if (!normalized.startsWith("http://") && !normalized.startsWith("https://")) { + normalized = "https://$normalized" + } + return normalized + } + + /** + * 从 API 获取可用模型列表 + * @param baseUrl API 基础地址 + * @param apiKey API 密钥 + * @return 模型 ID 列表 + */ + suspend fun fetchModels(baseUrl: String, apiKey: String): Result> = withContext(Dispatchers.IO) { + // 验证 baseUrl 是否为空 + if (baseUrl.isBlank()) { + return@withContext Result.failure(Exception("Base URL 不能为空")) + } + + val client = OkHttpClient.Builder() + .connectTimeout(10, TimeUnit.SECONDS) + .readTimeout(10, TimeUnit.SECONDS) + .build() + + // 清理 URL，确保正确拼接 + val cleanBaseUrl = normalizeUrl(baseUrl.removeSuffix("/chat/completions")) + + val request = try { + Request.Builder() + .url("$cleanBaseUrl/models") + .apply { + if (apiKey.isNotBlank()) { + addHeader("Authorization", "Bearer $apiKey") + } + } + .get() + .build() + } catch (e: IllegalArgumentException) { + return@withContext Result.failure(Exception("Base URL 格式无效: ${e.message}")) + } + + try { + client.newCall(request).execute().use { response -> + val responseBody = response.body?.string() ?: "" + + if (response.isSuccessful) { + val json = JSONObject(responseBody) + val data = json.optJSONArray("data") ?: JSONArray() + val models = mutableListOf() + for (i in 0 until data.length()) { + val item = data.optJSONObject(i) + if (item != null) { + val id = item.optString("id", "").trim() + if (id.isNotEmpty()) { + models.add(id) + } + } + } + Result.success(models) + } else { + Result.failure(Exception("HTTP ${response.code}: $responseBody")) + } + } + } catch (e: Exception) { + Result.failure(e) + } + } + } + + /** + * 调用 VLM 进行多模态推理 (带重试) + */ + suspend fun predict( + prompt: String, + images: List = emptyList() + ): Result = withContext(Dispatchers.IO) { + var lastException: Exception? = null + + // 预先编码图片 (避免重试时重复编码) + val encodedImages = images.map { bitmapToBase64Url(it) } + + for (attempt in 1..MAX_RETRIES) { + try { + val content = JSONArray().apply { + put(JSONObject().apply { + put("type", "text") + put("text", prompt) + }) + encodedImages.forEach { imageUrl -> + put(JSONObject().apply { + put("type", "image_url") + put("image_url", JSONObject().apply { + put("url", imageUrl) + }) + }) + } + } + + val messages = JSONArray().apply { + put(JSONObject().apply { + put("role", "user") + put("content", content) + }) + } + + val requestBody = JSONObject().apply { + put("model", model) + put("messages", messages) + put("max_tokens", 4096) + put("temperature", 0.0) + put("top_p", 0.85) + put("frequency_penalty", 0.2) // 减少重复输出 + } + + val request = Request.Builder() + .url("$baseUrl/chat/completions") + .apply { + if (apiKey.isNotBlank()) { + addHeader("Authorization", "Bearer $apiKey") + } + } + .addHeader("Content-Type", "application/json") + .post(requestBody.toString().toRequestBody("application/json".toMediaType())) + .build() + + val response = client.newCall(request).execute() + val responseBody = response.body?.string() ?: "" + + if (response.isSuccessful) { + val json = JSONObject(responseBody) + val choices = json.getJSONArray("choices") + if (choices.length() > 0) { + val message = choices.getJSONObject(0).getJSONObject("message") + val responseContent = message.getString("content") + return@withContext Result.success(responseContent) + } else { + lastException = Exception("No response from model") + } + } else { + lastException = Exception("API error: ${response.code} - $responseBody") + } + } catch (e: UnknownHostException) { + // DNS 解析失败，重试 + println("[VLMClient] DNS 解析失败，重试 $attempt/$MAX_RETRIES...") + lastException = e + if (attempt < MAX_RETRIES) { + delay(RETRY_DELAY_MS * attempt) + } + } catch (e: java.net.SocketTimeoutException) { + // 超时，重试 + println("[VLMClient] 请求超时，重试 $attempt/$MAX_RETRIES...") + lastException = e + if (attempt < MAX_RETRIES) { + delay(RETRY_DELAY_MS * attempt) + } + } catch (e: java.io.IOException) { + // IO 错误，重试 + println("[VLMClient] IO 错误: ${e.message}，重试 $attempt/$MAX_RETRIES...") + lastException = e + if (attempt < MAX_RETRIES) { + delay(RETRY_DELAY_MS * attempt) + } + } catch (e: Exception) { + // 其他错误，不重试 + return@withContext Result.failure(e) + } + } + + Result.failure(lastException ?: Exception("Unknown error")) + } + + /** + * 调用 VLM 进行多模态推理 (使用完整对话历史) + * @param messagesJson OpenAI 兼容的 messages JSON 数组 + */ + suspend fun predictWithContext( + messagesJson: JSONArray + ): Result = withContext(Dispatchers.IO) { + var lastException: Exception? = null + + for (attempt in 1..MAX_RETRIES) { + try { + val requestBody = JSONObject().apply { + put("model", model) + put("messages", messagesJson) + put("max_tokens", 4096) + put("temperature", 0.0) + } + + val request = Request.Builder() + .url("$baseUrl/chat/completions") + .apply { + if (apiKey.isNotBlank()) { + addHeader("Authorization", "Bearer $apiKey") + } + } + .addHeader("Content-Type", "application/json") + .post(requestBody.toString().toRequestBody("application/json".toMediaType())) + .build() + + val response = client.newCall(request).execute() + val responseBody = response.body?.string() ?: "" + + if (response.isSuccessful) { + val json = JSONObject(responseBody) + val choices = json.getJSONArray("choices") + if (choices.length() > 0) { + val message = choices.getJSONObject(0).getJSONObject("message") + val responseContent = message.getString("content") + return@withContext Result.success(responseContent) + } else { + lastException = Exception("No response from model") + } + } else { + lastException = Exception("API error: ${response.code} - $responseBody") + } + } catch (e: UnknownHostException) { + println("[VLMClient] DNS 解析失败，重试 $attempt/$MAX_RETRIES...") + lastException = e + if (attempt < MAX_RETRIES) { + delay(RETRY_DELAY_MS * attempt) + } + } catch (e: java.net.SocketTimeoutException) { + println("[VLMClient] 请求超时，重试 $attempt/$MAX_RETRIES...") + lastException = e + if (attempt < MAX_RETRIES) { + delay(RETRY_DELAY_MS * attempt) + } + } catch (e: java.io.IOException) { + println("[VLMClient] IO 错误: ${e.message}，重试 $attempt/$MAX_RETRIES...") + lastException = e + if (attempt < MAX_RETRIES) { + delay(RETRY_DELAY_MS * attempt) + } + } catch (e: Exception) { + return@withContext Result.failure(e) + } + } + + Result.failure(lastException ?: Exception("Unknown error")) + } + + /** + * Bitmap 转 Base64 URL (只压缩质量，不压缩分辨率) + * 保持原始分辨率以确保坐标准确 + */ + private fun bitmapToBase64Url(bitmap: Bitmap): String { + val outputStream = ByteArrayOutputStream() + // 使用 JPEG 格式，质量 70%，保持原始分辨率 + bitmap.compress(Bitmap.CompressFormat.JPEG, 70, outputStream) + val bytes = outputStream.toByteArray() + println("[VLMClient] 图片压缩: ${bitmap.width}x${bitmap.height}, ${bytes.size / 1024}KB") + val base64 = Base64.encodeToString(bytes, Base64.NO_WRAP) + return "data:image/jpeg;base64,$base64" + } + + /** + * 调整图片大小 + */ + private fun resizeBitmap(bitmap: Bitmap, maxWidth: Int, maxHeight: Int): Bitmap { + val width = bitmap.width + val height = bitmap.height + + if (width <= maxWidth && height <= maxHeight) { + return bitmap + } + + val ratio = minOf(maxWidth.toFloat() / width, maxHeight.toFloat() / height) + val newWidth = (width * ratio).toInt() + val newHeight = (height * ratio).toInt() + + return Bitmap.createScaledBitmap(bitmap, newWidth, newHeight, true) + } +} + +/** + * 常用 VLM 配置 + */ +object VLMConfigs { + // OpenAI GPT-4V + fun gpt4v(apiKey: String) = VLMClient( + apiKey = apiKey, + baseUrl = "https://api.openai.com/v1", + model = "gpt-4-vision-preview" + ) + + // Qwen-VL (阿里云) + fun qwenVL(apiKey: String) = VLMClient( + apiKey = apiKey, + baseUrl = "https://dashscope.aliyuncs.com/compatible-mode/v1", + model = "qwen-vl-max" + ) + + // Claude (Anthropic) + fun claude(apiKey: String) = VLMClient( + apiKey = apiKey, + baseUrl = "https://api.anthropic.com/v1", + model = "claude-3-5-sonnet-20241022" + ) + + // 自定义 (vLLM / Ollama / LocalAI) + fun custom(apiKey: String, baseUrl: String, model: String) = VLMClient( + apiKey = apiKey, + baseUrl = baseUrl, + model = model + ) +} diff --git a/android-buyer/app/src/main/res/drawable-xxxhdpi/splash_icon.png b/android-buyer/app/src/main/res/drawable-xxxhdpi/splash_icon.png new file mode 100644 index 0000000..1adfa36 Binary files /dev/null and b/android-buyer/app/src/main/res/drawable-xxxhdpi/splash_icon.png differ diff --git a/android-buyer/app/src/main/res/drawable/splash_icon.png b/android-buyer/app/src/main/res/drawable/splash_icon.png new file mode 100644 index 0000000..c18b25b Binary files /dev/null and b/android-buyer/app/src/main/res/drawable/splash_icon.png differ diff --git a/android-buyer/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml b/android-buyer/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml new file mode 100644 index 0000000..80b730f --- /dev/null +++ b/android-buyer/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml @@ -0,0 +1,5 @@ + + + + + diff --git a/android-buyer/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml b/android-buyer/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml new file mode 100644 index 0000000..80b730f --- /dev/null +++ b/android-buyer/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml @@ -0,0 +1,5 @@ + + + + + diff --git a/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher.png b/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher.png new file mode 100644 index 0000000..20395d4 Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher.png differ diff --git a/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher_foreground.png b/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher_foreground.png new file mode 100644 index 0000000..52e4862 Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher_foreground.png differ diff --git a/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher_round.png b/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher_round.png new file mode 100644 index 0000000..5dbd0a9 Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-hdpi/ic_launcher_round.png differ diff --git a/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher.png b/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher.png new file mode 100644 index 0000000..d89b472 Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher.png differ diff --git a/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher_foreground.png b/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher_foreground.png new file mode 100644 index 0000000..f597172 Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher_foreground.png differ diff --git a/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher_round.png b/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher_round.png new file mode 100644 index 0000000..e7ae6df Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-mdpi/ic_launcher_round.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher.png b/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher.png new file mode 100644 index 0000000..eaebb0a Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher_foreground.png b/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher_foreground.png new file mode 100644 index 0000000..aac76b7 Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher_foreground.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher_round.png b/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher_round.png new file mode 100644 index 0000000..66f850b Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xhdpi/ic_launcher_round.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher.png b/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher.png new file mode 100644 index 0000000..b0586fe Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher_foreground.png b/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher_foreground.png new file mode 100644 index 0000000..2b9bed0 Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher_foreground.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher_round.png b/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher_round.png new file mode 100644 index 0000000..4e7f4ea Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xxhdpi/ic_launcher_round.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png b/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png new file mode 100644 index 0000000..3562990 Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher_foreground.png b/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher_foreground.png new file mode 100644 index 0000000..da35f2f Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher_foreground.png differ diff --git a/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher_round.png b/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher_round.png new file mode 100644 index 0000000..5e9741c Binary files /dev/null and b/android-buyer/app/src/main/res/mipmap-xxxhdpi/ic_launcher_round.png differ diff --git a/android-buyer/app/src/main/res/values-en/strings.xml b/android-buyer/app/src/main/res/values-en/strings.xml new file mode 100644 index 0000000..cf2819e --- /dev/null +++ b/android-buyer/app/src/main/res/values-en/strings.xml @@ -0,0 +1,160 @@ + + + + Baozi + + + Baozi + History + Settings + + + Ready. Tell me what you want to do + Please connect Shizuku first + Try these commands + Tell Baozi what you want to do… + Executing Step %1$d + Send + Stop + + + Order Burger + Order a tasty burger nearby for me + Post to Weibo + Post my latest photo to Weibo + Watch Bilibili + I want to watch trending videos on Bilibili + Order Takeout + Order a pork trotter rice on Meituan + Listen to Music + Open NetEase Cloud Music and play daily recommendations + Send Message + Send a message to my recent contact saying I\'m busy + + + Execution History + %1$d records + No execution records + Records will appear here after executing tasks + %1$d steps + Delete + Delete Record + Are you sure you want to delete this record? + Execution Timeline + Task Instruction + + + Completed + Failed + Stopped + Running + Status + Steps + Duration + + + Settings + Configure API and app options + + + Appearance + Theme Mode + Light Mode + Dark Mode + Follow System + Select Theme + + + API Configuration + API Provider + Custom + Model + API Key + Set (%1$s) + Not set + Enter your API Key + Show + Hide + + + Custom Models + Add Custom Model + Add other supported models + Enter model name or ID + e.g. gpt-4-vision + Select Model + + + Help + Shizuku Guide + Learn how to install and configure Shizuku + Overlay Permission + Learn why overlay permission is needed + + + About + Version + VLM-based Android automation tool + + + Shizuku Connected + Shizuku Not Connected + Device control available + Please start Shizuku and grant permission + + + Download Shizuku + Download Shizuku from Google Play or GitHub\n\n• Google Play: Search \"Shizuku\"\n• GitHub: github.com/RikkaApps/Shizuku + Start Shizuku + Open Shizuku and choose a startup method:\n\n• Wireless debugging (recommended): Requires Android 11+, enable wireless debugging in developer options\n• Connect to computer: Start via ADB command + Authorize Baozi + Find Baozi Autopilot in Shizuku\'s \"App Management\" and click authorize + Start Using + After authorization, return to Baozi. The status card will show \"Shizuku Connected\" + + + Why is overlay permission needed? + Baozi needs to display an overlay during task execution to: + Show current execution progress + Provide a stop button to interrupt tasks anytime + Display status information above other apps + How to enable? + 1. You\'ll be prompted when executing a task\n2. Or go to: Settings > Apps > Baozi > Overlay permission\n3. Enable \"Allow display over other apps\" + Privacy & Security + The overlay is only shown during task execution and does not collect any personal information. It automatically disappears after the task is complete. + + + Welcome to Baozi + Baozi is an intelligent automation assistant\nthat helps you operate your phone to complete various tasks + AI Powered + Based on advanced vision-language models,\nBaozi can understand screen content and make intelligent decisions + Easy to Use + Just describe what you want to do in natural language,\nand Baozi will automatically complete it for you + Safe & Secure + Automatically stops when encountering sensitive pages\n(like payments, passwords) to protect your account + Skip + Next + Get Started + + + Confirm + Cancel + Close + Add + Delete + Back + Got it + + + Please enter an instruction + Please enter API Key + Please grant overlay permission + Shizuku permission granted + Please start Shizuku App first + Shizuku version too old + Task complete + User stopped + Detected sensitive page (payment/password), safely stopped + Reached maximum steps limit + Error: %1$s + diff --git a/android-buyer/app/src/main/res/values/colors.xml b/android-buyer/app/src/main/res/values/colors.xml new file mode 100644 index 0000000..1a08ea4 --- /dev/null +++ b/android-buyer/app/src/main/res/values/colors.xml @@ -0,0 +1,29 @@ + + + + #EFB471 + #EFB471 + + + #D67744 + #B85E2E + #E89060 + #EFB773 + #D69B52 + #F5CB94 + + + #1A1A1A + #252525 + #2A2A2A + + + #FFFFFF + #B0B0B0 + #666666 + + + #4CAF50 + #F44336 + #FF9800 + diff --git a/android-buyer/app/src/main/res/values/strings.xml b/android-buyer/app/src/main/res/values/strings.xml new file mode 100644 index 0000000..4d04d64 --- /dev/null +++ b/android-buyer/app/src/main/res/values/strings.xml @@ -0,0 +1,160 @@ + + + + 肉包 + + + 肉包 + 记录 + 设置 + + + 准备就绪，告诉我你想做什么 + 请先连接 Shizuku + 试试这些指令 + 告诉肉包你想做什么… + 正在执行 Step %1$d + 发送 + 停止 + + + 点汉堡 + 帮我点个附近好吃的汉堡 + 发微博 + 帮我把最后一张照片发送到微博 + 看B站 + 我要看B站热门的视频 + 点外卖 + 帮我在美团点一份猪脚饭 + 听音乐 + 打开网易云音乐播放每日推荐 + 发消息 + 帮我给最近联系人发一条消息说在忙 + + + 执行记录 + 共 %1$d 条记录 + 暂无执行记录 + 执行任务后记录会显示在这里 + %1$d 步 + 删除 + 删除记录 + 确定要删除这条执行记录吗？ + 执行时间线 + 任务指令 + + + 已完成 + 失败 + 已停止 + 执行中 + 状态 + 步骤数 + 耗时 + + + 设置 + 配置 API 和应用选项 + + + 外观 + 主题模式 + 浅色模式 + 深色模式 + 跟随系统 + 选择主题 + + + API 配置 + API 服务商 + 自定义 + 模型 + API Key + 已设置 (%1$s) + 未设置 + 请输入您的 API Key + 显示 + 隐藏 + + + 自定义模型 + 添加自定义模型 + 添加其他支持的模型 + 输入模型名称或 ID + 例如: gpt-4-vision + 选择模型 + + + 帮助 + Shizuku 使用指南 + 了解如何安装和配置 Shizuku + 悬浮窗权限说明 + 了解为什么需要悬浮窗权限 + + + 关于 + 版本 + 基于视觉语言模型的 Android 自动化工具 + + + Shizuku 已连接 + Shizuku 未连接 + 设备控制功能可用 + 请启动 Shizuku 并授权 + + + 下载 Shizuku + 从 Google Play 或 GitHub 下载 Shizuku 应用\n\n• Google Play: 搜索 \"Shizuku\"\n• GitHub: github.com/RikkaApps/Shizuku + 启动 Shizuku + 打开 Shizuku 应用，根据您的设备选择启动方式：\n\n• 无线调试（推荐）：需要 Android 11+，在开发者选项中开启无线调试\n• 连接电脑：通过 ADB 命令启动 + 授权肉包 + 在 Shizuku 的「应用管理」中找到肉包 Autopilot，点击授权按钮 + 开始使用 + 授权完成后，返回肉包应用，状态卡片会显示「Shizuku 已连接」 + + + 为什么需要悬浮窗权限？ + 肉包在执行任务时需要显示悬浮窗来： + 显示当前执行进度 + 提供停止按钮，随时中断任务 + 在其他应用上方显示状态信息 + 如何开启？ + 1. 点击执行任务时会自动提示\n2. 或前往：设置 > 应用 > 肉包 > 悬浮窗权限\n3. 开启「允许显示在其他应用上层」 + 隐私安全 + 悬浮窗仅在任务执行期间显示，不会收集任何个人信息。任务完成后悬浮窗会自动消失。 + + + 欢迎使用肉包 + 肉包是一个智能自动化助手，\n可以帮你操作手机完成各种任务 + AI 驱动 + 基于先进的视觉语言模型，\n肉包能够理解屏幕内容并做出智能决策 + 简单易用 + 只需用自然语言描述你想做的事，\n肉包会自动帮你完成 + 安全可靠 + 遇到敏感页面（如支付、密码）会自动停止，\n保护你的账户安全 + 跳过 + 下一步 + 开始使用 + + + 确定 + 取消 + 关闭 + 添加 + 删除 + 返回 + 知道了 + + + 请输入指令 + 请输入 API Key + 请授予悬浮窗权限 + Shizuku 权限已获取 + 请先启动 Shizuku App + Shizuku 版本过低 + 任务完成 + 用户停止 + 检测到敏感页面（支付/密码），已安全停止 + 达到最大步数限制 + 错误: %1$s + diff --git a/android-buyer/app/src/main/res/values/themes.xml b/android-buyer/app/src/main/res/values/themes.xml new file mode 100644 index 0000000..27136d0 --- /dev/null +++ b/android-buyer/app/src/main/res/values/themes.xml @@ -0,0 +1,29 @@ + + + + + + + + diff --git a/android-buyer/app/src/main/res/xml/file_paths.xml b/android-buyer/app/src/main/res/xml/file_paths.xml new file mode 100644 index 0000000..a485be3 --- /dev/null +++ b/android-buyer/app/src/main/res/xml/file_paths.xml @@ -0,0 +1,5 @@ + + + + + diff --git a/android-buyer/app/src/main/res/xml/network_security_config.xml b/android-buyer/app/src/main/res/xml/network_security_config.xml new file mode 100644 index 0000000..9d0b208 --- /dev/null +++ b/android-buyer/app/src/main/res/xml/network_security_config.xml @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/android-buyer/build.gradle.kts b/android-buyer/build.gradle.kts new file mode 100644 index 0000000..74f28a6 --- /dev/null +++ b/android-buyer/build.gradle.kts @@ -0,0 +1,5 @@ +// Top-level build file +plugins { + id("com.android.application") version "8.2.0" apply false + id("org.jetbrains.kotlin.android") version "1.9.20" apply false +} diff --git a/android-buyer/docs/demo/demo.gif b/android-buyer/docs/demo/demo.gif new file mode 100644 index 0000000..f09e51a Binary files /dev/null and b/android-buyer/docs/demo/demo.gif differ diff --git a/android-buyer/docs/images/logo.png b/android-buyer/docs/images/logo.png new file mode 100644 index 0000000..51a5442 Binary files /dev/null and b/android-buyer/docs/images/logo.png differ diff --git a/android-buyer/docs/screenshots/1.jpg b/android-buyer/docs/screenshots/1.jpg new file mode 100644 index 0000000..3cd9942 Binary files /dev/null and b/android-buyer/docs/screenshots/1.jpg differ diff --git a/android-buyer/docs/screenshots/2.jpg b/android-buyer/docs/screenshots/2.jpg new file mode 100644 index 0000000..e9dbc75 Binary files /dev/null and b/android-buyer/docs/screenshots/2.jpg differ diff --git a/android-buyer/docs/screenshots/3.jpg b/android-buyer/docs/screenshots/3.jpg new file mode 100644 index 0000000..0dc9d29 Binary files /dev/null and b/android-buyer/docs/screenshots/3.jpg differ diff --git a/android-buyer/docs/screenshots/4.jpg b/android-buyer/docs/screenshots/4.jpg new file mode 100644 index 0000000..d595137 Binary files /dev/null and b/android-buyer/docs/screenshots/4.jpg differ diff --git a/android-buyer/gradle.properties b/android-buyer/gradle.properties new file mode 100644 index 0000000..f0a2e55 --- /dev/null +++ b/android-buyer/gradle.properties @@ -0,0 +1,4 @@ +org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8 +android.useAndroidX=true +kotlin.code.style=official +android.nonTransitiveRClass=true diff --git a/android-buyer/gradle/wrapper/gradle-wrapper.jar b/android-buyer/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000..033e24c Binary files /dev/null and b/android-buyer/gradle/wrapper/gradle-wrapper.jar differ diff --git a/android-buyer/gradle/wrapper/gradle-wrapper.properties b/android-buyer/gradle/wrapper/gradle-wrapper.properties new file mode 100644 index 0000000..407710f --- /dev/null +++ b/android-buyer/gradle/wrapper/gradle-wrapper.properties @@ -0,0 +1,8 @@ +distributionBase=GRADLE_USER_HOME +distributionPath=wrapper/dists +distributionSha256Sum=38f66cd6eef217b4c35855bb11ea4e9fbc53594ccccb5fb82dfd317ef8c2c5a3 +distributionUrl=https\://services.gradle.org/distributions/gradle-8.2-bin.zip +networkTimeout=120000 +validateDistributionUrl=false +zipStoreBase=GRADLE_USER_HOME +zipStorePath=wrapper/dists diff --git a/android-buyer/gradlew b/android-buyer/gradlew new file mode 100755 index 0000000..fcb6fca --- /dev/null +++ b/android-buyer/gradlew @@ -0,0 +1,248 @@ +#!/bin/sh + +# +# Copyright © 2015-2021 the original authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +############################################################################## +# +# Gradle start up script for POSIX generated by Gradle. +# +# Important for running: +# +# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is +# noncompliant, but you have some other compliant shell such as ksh or +# bash, then to run this script, type that shell name before the whole +# command line, like: +# +# ksh Gradle +# +# Busybox and similar reduced shells will NOT work, because this script +# requires all of these POSIX shell features: +# * functions; +# * expansions «$var», «${var}», «${var:-default}», «${var+SET}», +# «${var#prefix}», «${var%suffix}», and «$( cmd )»; +# * compound commands having a testable exit status, especially «case»; +# * various built-in commands including «command», «set», and «ulimit». +# +# Important for patching: +# +# (2) This script targets any POSIX shell, so it avoids extensions provided +# by Bash, Ksh, etc; in particular arrays are avoided. +# +# The "traditional" practice of packing multiple parameters into a +# space-separated string is a well documented source of bugs and security +# problems, so this is (mostly) avoided, by progressively accumulating +# options in "$@", and eventually passing that to Java. +# +# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS, +# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly; +# see the in-line comments for details. +# +# There are tweaks for specific operating systems such as AIX, CygWin, +# Darwin, MinGW, and NonStop. +# +# (3) This script is generated from the Groovy template +# https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt +# within the Gradle project. +# +# You can find Gradle at https://github.com/gradle/gradle/. +# +############################################################################## + +# Attempt to set APP_HOME + +# Resolve links: $0 may be a link +app_path=$0 + +# Need this for daisy-chained symlinks. +while + APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path + [ -h "$app_path" ] +do + ls=$( ls -ld "$app_path" ) + link=${ls#*' -> '} + case $link in #( + /*) app_path=$link ;; #( + *) app_path=$APP_HOME$link ;; + esac +done + +# This is normally unused +# shellcheck disable=SC2034 +APP_BASE_NAME=${0##*/} +APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD=maximum + +warn () { + echo "$*" +} >&2 + +die () { + echo + echo "$*" + echo + exit 1 +} >&2 + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "$( uname )" in #( + CYGWIN* ) cygwin=true ;; #( + Darwin* ) darwin=true ;; #( + MSYS* | MINGW* ) msys=true ;; #( + NONSTOP* ) nonstop=true ;; +esac + +CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar + + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD=$JAVA_HOME/jre/sh/java + else + JAVACMD=$JAVA_HOME/bin/java + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD=java + if ! command -v java >/dev/null 2>&1 + then + die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +fi + +# Increase the maximum file descriptors if we can. +if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then + case $MAX_FD in #( + max*) + # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked. + # shellcheck disable=SC3045 + MAX_FD=$( ulimit -H -n ) || + warn "Could not query maximum file descriptor limit" + esac + case $MAX_FD in #( + '' | soft) :;; #( + *) + # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked. + # shellcheck disable=SC3045 + ulimit -n "$MAX_FD" || + warn "Could not set maximum file descriptor limit to $MAX_FD" + esac +fi + +# Collect all arguments for the java command, stacking in reverse order: +# * args from the command line +# * the main class name +# * -classpath +# * -D...appname settings +# * --module-path (only if needed) +# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables. + +# For Cygwin or MSYS, switch paths to Windows format before running java +if "$cygwin" || "$msys" ; then + APP_HOME=$( cygpath --path --mixed "$APP_HOME" ) + CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" ) + + JAVACMD=$( cygpath --unix "$JAVACMD" ) + + # Now convert the arguments - kludge to limit ourselves to /bin/sh + for arg do + if + case $arg in #( + -*) false ;; # don't mess with options #( + /?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath + [ -e "$t" ] ;; #( + *) false ;; + esac + then + arg=$( cygpath --path --ignore --mixed "$arg" ) + fi + # Roll the args list around exactly as many times as the number of + # args, so each arg winds up back in the position where it started, but + # possibly modified. + # + # NB: a `for` loop captures its iteration list before it begins, so + # changing the positional parameters here affects neither the number of + # iterations, nor the values presented in `arg`. + shift # remove old arg + set -- "$@" "$arg" # push replacement arg + done +fi + + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' + +# Collect all arguments for the java command; +# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of +# shell script including quotes and variable substitutions, so put them in +# double quotes to make sure that they get re-expanded; and +# * put everything else in single quotes, so that it's not re-expanded. + +set -- \ + "-Dorg.gradle.appname=$APP_BASE_NAME" \ + -classpath "$CLASSPATH" \ + org.gradle.wrapper.GradleWrapperMain \ + "$@" + +# Stop when "xargs" is not available. +if ! command -v xargs >/dev/null 2>&1 +then + die "xargs is not available" +fi + +# Use "xargs" to parse quoted args. +# +# With -n1 it outputs one arg per line, with the quotes and backslashes removed. +# +# In Bash we could simply go: +# +# readarray ARGS < <( xargs -n1 <<<"$var" ) && +# set -- "${ARGS[@]}" "$@" +# +# but POSIX shell has neither arrays nor command substitution, so instead we +# post-process each arg (as a line of input to sed) to backslash-escape any +# character that might be a shell metacharacter, then use eval to reverse +# that process (while maintaining the separation between arguments), and wrap +# the whole thing up as a single "set" statement. +# +# This will of course break if any of these variables contains a newline or +# an unmatched quote. +# + +eval "set -- $( + printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" | + xargs -n1 | + sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' | + tr '\n' ' ' + )" '"$@"' + +exec "$JAVACMD" "$@" diff --git a/android-buyer/gradlew.bat b/android-buyer/gradlew.bat new file mode 100644 index 0000000..93e3f59 --- /dev/null +++ b/android-buyer/gradlew.bat @@ -0,0 +1,92 @@ +@rem +@rem Copyright 2015 the original author or authors. +@rem +@rem Licensed under the Apache License, Version 2.0 (the "License"); +@rem you may not use this file except in compliance with the License. +@rem You may obtain a copy of the License at +@rem +@rem https://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. +@rem + +@if "%DEBUG%"=="" @echo off +@rem ########################################################################## +@rem +@rem Gradle startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables with windows NT shell +if "%OS%"=="Windows_NT" setlocal + +set DIRNAME=%~dp0 +if "%DIRNAME%"=="" set DIRNAME=. +@rem This is normally unused +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Resolve any "." and ".." in APP_HOME to make it shorter. +for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if %ERRORLEVEL% equ 0 goto execute + +echo. +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto execute + +echo. +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:execute +@rem Setup the command line + +set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar + + +@rem Execute Gradle +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %* + +:end +@rem End local scope for the variables with windows NT shell +if %ERRORLEVEL% equ 0 goto mainEnd + +:fail +rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of +rem the _cmd.exe /c_ return code! +set EXIT_CODE=%ERRORLEVEL% +if %EXIT_CODE% equ 0 set EXIT_CODE=1 +if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE% +exit /b %EXIT_CODE% + +:mainEnd +if "%OS%"=="Windows_NT" endlocal + +:omega diff --git a/android-buyer/settings.gradle.kts b/android-buyer/settings.gradle.kts new file mode 100644 index 0000000..488c65b --- /dev/null +++ b/android-buyer/settings.gradle.kts @@ -0,0 +1,19 @@ +pluginManagement { + repositories { + google() + mavenCentral() + gradlePluginPortal() + } +} + +dependencyResolutionManagement { + repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS) + repositories { + google() + mavenCentral() + maven { url = uri("https://jitpack.io") } + } +} + +rootProject.name = "AutoPilot" +include(":app") diff --git a/docs/00-ai-start-here.md b/docs/00-ai-start-here.md index 532cd9d..d9066dc 100644 --- a/docs/00-ai-start-here.md +++ b/docs/00-ai-start-here.md @@ -44,14 +44,14 @@ 2. 读取 `agent-context.json`、`current-state.md` 和当前任务文件。 3. 仓库已启用 Git 时查看最近五条提交；当前未启用时，以 `current-state.md` 为准。 4. 运行 `./init.ps1`（Windows）或 `./init.sh`（WSL/Linux）。 -5. 如果脚本因 `T-001` 尚未完成而主动失败，只能领取 `T-001`；不要宣称基线可运行。 +5. 如果脚本失败，先记录并修复基线；不要在失败状态领取依赖它的后续任务。 6. 有真实基线后先跑 smoke；基线失败时先修基线。 7. 领取编号最小、依赖已完成的一个 `TODO` 任务并改成 `DOING`。 8. 完成实现、验证、任务执行记录和当前状态同步后再结束。 ## 当前阶段与优先路径 -当前处于“文档基线完成、最高风险验证尚未开始”阶段。 +当前处于“Android 可运行基线完成、设备和 workflow 验证准备”阶段。 严格按以下顺序推进： @@ -88,6 +88,18 @@ Roubao、拼多多页面、Android 包名、上游分支和依赖版本必须从 ## 当前验证命令 -当前没有生产代码，`init.ps1` 和 `init.sh` 会主动报错。`T-001` 完成后必须用真实 -命令替换本文，例如 Android 至少应有可复现的 Gradle 构建命令；后端建立后再追加 -后端测试和启动命令。 +Windows 默认验证： + +```powershell +.\init.ps1 +``` + +连接并授权真机后执行安装和冷启动： + +```powershell +$env:RUN_START_COMMAND = "1" +.\init.ps1 +``` + +脚本执行 `test assembleDebug`；上游当前没有单元测试源，因此 `test` 的测试任务为 +`NO-SOURCE`，但编译与 APK 打包必须成功。后端建立后再追加后端测试和启动命令。 diff --git a/docs/03-tech-stack.md b/docs/03-tech-stack.md index 1b6a92c..7785a24 100644 --- a/docs/03-tech-stack.md +++ b/docs/03-tech-stack.md @@ -6,10 +6,10 @@ | 维度 | 选型 | 状态 | 说明 | | --- | --- | --- | --- | -| Android 基础 | Roubao `main` 的 Kotlin 原生 Android 代码，应用版本 1.4.2 | 上游已核实，待接入 | 上游为 `Turbo1123/roubao`，MIT；接入时固定具体 commit。 | -| Android IDE/JDK | Android Studio Hedgehog 2023.1.1 或更高；JDK 17 | 上游已核实 | 首次导入不接受 IDE 自动升级 AGP、Gradle 或 Kotlin。 | -| Android 构建链 | Gradle 8.2；AGP 8.2.0；Kotlin 1.9.20；JVM target 17 | 上游已核实 | 使用仓库内 Gradle Wrapper，不依赖全局 Gradle。 | -| Android SDK | compileSdk/targetSdk 34；minSdk 26；SDK Build Tools 34.0.0 | 上游已核实 | 支持 Android 8.0+；真机自动化优先 Android 11+。 | +| Android 基础 | Roubao `main` 的 Kotlin 原生 Android 代码，应用版本 1.4.2 | 已接入并构建 | 固定 `main@c8a6d7f...`，来源与调整见 `android-buyer/UPSTREAM.md`。 | +| Android IDE/JDK | Android Studio Hedgehog 2023.1.1 或更高；JDK 17 | JDK/CLI 已验证，IDE 可选安装 | 命令行构建不依赖 Android Studio；首次用 IDE 导入时不接受自动升级。 | +| Android 构建链 | Gradle 8.2；AGP 8.2.0；Kotlin 1.9.20；JVM target 17 | 已验证 | 已补齐上游缺失的 `gradlew.bat`，不依赖全局 Gradle。 | +| Android SDK | compileSdk/targetSdk 34；minSdk 26；SDK Build Tools 34.0.0 | 已验证 | 支持 Android 8.0+；本机使用 Command-line Tools 22.0。 | | Android UI | Jetpack Compose + Material 3；Compose Compiler 1.5.5 | 上游已核实 | Compose BOM 为 2023.10.01。 | | Android 自动化 | 项目目标以 `AccessibilityService` 为主，Shizuku 为兼容/增强路径 | 目标已定，待实现 | 上游 `main` 使用 Shizuku 13.1.5；无障碍实现位于独立开发分支，不能把它误认为主分支现状。 | | 第一层任务源 | UTF-8 四行蝦皮订单文本 + 同订单号 JPEG | 首份样例已核实，待实现 | 使用 Debug/测试专用 `TaskSource`；原始数据和生成物不得提交。 | @@ -27,7 +27,7 @@ | VLM 接入 | 应用内统一适配器，优先兼容 OpenAI 风格多模态接口 | 接口已定，供应商待定 | 模型输出必须符合本项目 JSON Schema。 | | 通知 | MVP 不使用推送 | 已定 | 点击“获取任务”调用原子 claim API；V2 再评估厂商推送/WebSocket。 | | 后端测试 | 标准库 `testing` + `httptest` | MVP 已定 | 覆盖状态机、权限、幂等、SQLite 事务和输入校验。 | -| Android 测试 | JUnit + 现有上游测试工具；真实设备 smoke | 待源码核实 | UI 自动化核心必须在目标设备验证。 | +| Android 测试 | Gradle `test` + 真实设备 smoke | 基线已验证 | 上游没有 `test/androidTest` 源；T-003 建立 workflow 测试。 | | 部署 | 单机局域网 Go 服务；容器化后置 | MVP 已定 | Android 测试机必须能通过 HTTPS 或受控测试网络访问。 | ## Roubao 上游版本基线 @@ -51,15 +51,13 @@ | 自动化依赖 | Shizuku API/Provider 13.1.5 | | 无障碍开发分支 | `roubao2.0+AccessibilityService`，核实时 commit `5b114c0a9476c359b27cfe994743fc7beb0a3554` | -上述值来自上游 README、`build.gradle.kts`、`app/build.gradle.kts` 和 -`gradle/wrapper/gradle-wrapper.properties`。它们是接入前的远端核实结果，不代表 -本仓库已经构建通过。`T-001` 接入时必须再次确认远端 commit，并把最终采用的 commit -作为可复现基线。 +上述版本已在 2026-07-25 使用 JDK 17.0.13、SDK 34 和 Build Tools 34.0.0 完成 +Windows Debug 构建。上游无障碍分支落后于 `main` 的修复和 1.4.2 能力，因此不整体 +切换分支；后续只选择性移植无障碍服务。 -上游启用了 Google Services 4.4.2、Firebase Crashlytics Gradle Plugin 3.0.2 和 -Firebase BOM 33.7.0，但 `google-services.json` 被 `.gitignore` 排除。干净构建必须 -提供本项目自己的 Firebase 配置，或者在不需要遥测时移除相关插件和依赖；不得提交 -真实 Firebase 配置。 +上游 Firebase 配置依赖未提交的 `google-services.json`，且第一层不需要遥测。 +接入基线已移除 Google Services、Firebase Analytics/Crashlytics 及云端崩溃上报 +设置，保留本地 `CrashHandler`，从而无需伪造或提交 Firebase 凭证。 ## 关键决策 @@ -113,7 +111,7 @@ RabbitMQ、NATS 和 gRPC 为主，不符合最小 Gin + SQLite 边界。 ## 计划目录 ```text -android-buyer/ # Kotlin Android App +android-buyer/ # 已接入的 Roubao Kotlin Android App backend-api/ cmd/api/ # 进程入口，只负责装配和生命周期 internal/domain/ # 实体、状态机和确定性规则 @@ -132,20 +130,21 @@ docs/ ## 构建与运行命令 -当前没有源码，以下是计划形状而非可运行证据： - -| 用途 | 计划命令 | 当前状态 | +| 用途 | 命令 | 当前状态 | | --- | --- | --- | -| Android 构建 | `android-buyer\gradlew.bat assembleDebug` | 待 `T-001` | -| Android 单元测试 | `android-buyer\gradlew.bat test` | 待 `T-001` | +| Android 标准验证 | `.\init.ps1` | 已验证 | +| Android 构建 | `android-buyer\gradlew.bat assembleDebug --no-daemon` | 已验证 | +| Android 测试任务 | `android-buyer\gradlew.bat test --no-daemon` | 已验证；无测试源 | +| Android 安装/启动 | `$env:RUN_START_COMMAND="1"; .\init.ps1` | 已在 Android 16 真机验证 | | 后端依赖 | `go mod download`（在 `backend-api/`） | 待 `T-201` | | 后端测试 | `go test ./...`（在 `backend-api/`） | 待 `T-201` | | 后端构建 | `go build -o bin/cmroubao-api.exe ./cmd/api` | 待 `T-201` | | 后端启动 | `go run ./cmd/api` | 待 `T-201` | | 数据迁移 | `goose -dir migrations sqlite3 ./var/cmroubao.db up` | 待 `T-201` | -任何命令只有实际执行成功后才能写入 `init.ps1`、`init.sh` 和 -`current-state.md` 的“当前可运行内容”。 +`gradlew installDebug` 在当前 Android 16 设备上由旧版 AGP/ddmlib 返回 `-99`， +但 SDK Platform Tools 37.0.0 的 `adb install -r -t` 成功。标准脚本因此使用 SDK +内新版 ADB 安装，避免 PATH 中旧版 ADB 1.0.32。 ## 依赖纪律 diff --git a/docs/05-coding-rules.md b/docs/05-coding-rules.md index 94e0281..93e47b5 100644 --- a/docs/05-coding-rules.md +++ b/docs/05-coding-rules.md @@ -94,5 +94,6 @@ - [ ] 当前任务文件记录命令、结果、环境和未验证项。 - [ ] `current-state.md` 与真实目录、命令和 blocker 一致。 -当前尚无真实命令。`T-001` 后必须在这里列出 Android 构建和测试命令；后端建立后 -再列出后端命令。不能把 `docs/03-tech-stack.md` 中的计划命令当作已通过证据。 +当前 Android 标准验证命令是根目录 `.\init.ps1`；它执行 Gradle `test` 和 +`assembleDebug`。设置 `RUN_START_COMMAND=1` 时使用 SDK 内新版 ADB 安装并启动。 +后端建立后再在这里列出后端命令。 diff --git a/docs/current-state.md b/docs/current-state.md index d8295de..7ae80d8 100644 --- a/docs/current-state.md +++ b/docs/current-state.md @@ -5,23 +5,23 @@ ## 当前快照 - 日期：2026-07-25 -- 阶段：Harness 文档基线完成，T-001 已开始，Android 源码尚未接入 +- 阶段：T-001 Android 可运行基线完成，准备执行 T-002 设备就绪基线 - Git：已初始化，当前分支为 `main`，文档和初始化脚本纳入首个基线提交 -- 生产代码：无 -- Android：Roubao 源码尚未接入；上游仓库、MIT 许可证、分支和构建版本已远端核实 +- 生产代码：`android-buyer/` 已接入 Roubao Android 源码 +- Android：固定 `main@c8a6d7f03422eb01744b01f3ee77bf7757741f7e`；MIT 许可证已保留 - 后端：已决定使用 Go 1.23.0 + Gin 1.11.0；Go Blueprint v0.10.11 骨架尚未接入 -- 本机 Android 工具：JDK 17.0.13 和 ADB 已有；Android Studio、Android SDK 34 - 未检测到，`ANDROID_HOME`/`ANDROID_SDK_ROOT` 未设置 -- 测试：无 +- 本机 Android 工具：JDK 17.0.13、Command-line Tools 22.0、SDK 34、 + Build Tools 34.0.0、Platform Tools/ADB 37.0.0；用户级 SDK 环境变量已设置 +- Android Studio：未安装；`winget` 静默安装卡住后已终止，不阻塞命令行构建 +- 测试：Gradle `test` 成功；上游没有 `test/androidTest` 测试源 - 版本控制内数据：只有 `deepseek总结.txt` 背景摘要和本套项目文档 - 本地私有样本：仓库根目录存在一组未跟踪、已本地排除的同名蝦皮文本/JPEG； 已核实 UTF-8 四行字段格式和图片可解码，真实内容未纳入 Git -- 标准启动路径：未配置；`init.ps1`/`init.sh` 会主动失败 -- 标准验证路径：文档结构和链接检查；无代码构建 -- 当前 blocker：Roubao 尚未导入，`main` 与无障碍开发分支尚未完成代码差异评估； - Android Studio/SDK 34 未安装或未被检测到，测试设备状态未知；拼多多测试版本/账号 - 未建立基线；蝦皮样本正式私有目录、更多图片类型和一单多 SKU 规则未确认；VLM - 供应商和测试凭证未确认 +- 标准启动路径：`$env:RUN_START_COMMAND="1"; .\init.ps1` +- 标准验证路径：`.\init.ps1` +- 当前 blocker：T-002 尚未完成 Shizuku 服务/授权、目标无障碍服务和拼多多登录状态 + 检查；蝦皮样本正式私有目录、更多图片类型和一单多 SKU 规则未确认；VLM 供应商和 + 测试凭证未确认 ## 当前目录 @@ -29,33 +29,29 @@ | --- | --- | --- | | `AGENTS.md` | 已有 | agent 权威入口 | | `docs/` | 已有 | Harness Coding 文档 | -| `docs/tasks/T-001.md` | TODO | 当前第一个可领取任务 | +| `docs/tasks/T-001.md` | DONE | Android 可构建、可安装、可启动基线 | | `deepseek总结.txt` | 已有 | 历史讨论摘要，不是正式需求权威 | -| `android-buyer/` | 待建 | Roubao Android 代码目标目录 | +| `android-buyer/` | 已有 | Roubao `main` 固定 commit 的 Android 基线 | | `backend-api/` | 待建 | Go-Gin、管理 Web 和数据目标目录 | -| `init.ps1` / `init.sh` | 占位 | T-001 后写入真实命令 | +| `init.ps1` / `init.sh` | 已验证 | Android 构建入口；可选真机安装启动 | ## 任务摘要 -- 已完成：Harness 文档初始化（本次文档工作，不作为产品开发任务编号）。 -- 正在进行：`T-001 初始化 Git 并接入 Roubao Android 基线`；已完成 Git 初始化和 - 上游远端核实，源码导入及构建尚未完成。 -- 下一个可领取任务：无；完成 T-001 后才能领取依赖它的任务。 +- 已完成：T-001 初始化 Git 并接入 Roubao Android 基线。 +- 正在进行：无。 +- 下一个可领取任务：T-002 建立测试设备基线和就绪检查。 ## 当前可运行内容 -没有生产代码或已验证构建命令。以下检查仅验证文档文件： - ```powershell -Get-ChildItem -Recurse -File -Get-Content docs/agent-context.json -Raw | ConvertFrom-Json | Out-Null +.\init.ps1 + +$env:RUN_START_COMMAND = "1" +.\init.ps1 ``` -`docs/03-tech-stack.md` 中的 Gradle、Go 构建和启动命令是计划形状，尚不能作为 -当前可运行内容。 - -已核实的 Roubao 远端版本事实记录在 `docs/03-tech-stack.md`；“已核实”只表示查看了 -上游配置，不表示本机已经具备 Android SDK 或完成过 APK 构建。 +2026-07-25 已在 PKG110、Android 16/API 36 上用第二条命令完成 Debug APK 安装和 +冷启动；画面停在预期的 Shizuku 使用指南，进程存活且 logcat 无崩溃。 ## 维护规则 diff --git a/docs/tasks/T-001.md b/docs/tasks/T-001.md index e020bb5..61b644d 100644 --- a/docs/tasks/T-001.md +++ b/docs/tasks/T-001.md @@ -3,17 +3,19 @@ id: T-001 title: 初始化 Git 并接入 Roubao Android 基线 phase: 0 deps: [] -status: DOING +status: DONE created: 2026-07-25 -context_ref: null +context_ref: 285606639bc13b6db4bb8969eb607612fa56d87a work_branch: main write_paths: + - AGENTS.md - docs/tasks/T-001.md - android-buyer/** - .gitignore - README.md - init.ps1 - init.sh + - .gitattributes - docs/00-ai-start-here.md - docs/02-requirements.md - docs/03-tech-stack.md @@ -27,9 +29,9 @@ write_paths: ## 问题 / 背景 -当前仓库只有 Harness 文档，不是 Git 仓库，也没有 Roubao 源码、Gradle Wrapper 或 +任务启动时仓库只有 Harness 文档，没有 Roubao 源码、Windows Gradle Wrapper 或 可验证的 Android 命令。所有自动化、VLM 和后台任务都依赖一个可复现的 Android -基线；在基线完成前不能开始拼多多业务修改。 +基线。 预期上游已经从真实仓库完成远端核实： @@ -68,17 +70,17 @@ AccessibilityService 位于独立开发分支；接入前要先比较分支， ## 验收要点 - [x] 当前目录已成为可正常查看状态和提交的 Git 仓库。 -- [ ] `android-buyer/` 包含可追溯到明确 URL 和 commit 的 Roubao 源码。 -- [ ] 上游许可证文件已保留，并确认允许当前预期的二次开发方式。 -- [ ] Windows 上使用仓库内 Gradle Wrapper 成功构建 Debug APK。 -- [ ] 上游已有 Android 单元测试已执行；没有测试时明确记录。 -- [ ] 有测试设备时完成安装和启动 smoke，并记录设备/Android 版本；无设备时任务 +- [x] `android-buyer/` 包含可追溯到明确 URL 和 commit 的 Roubao 源码。 +- [x] 上游许可证文件已保留，并确认允许当前预期的二次开发方式。 +- [x] Windows 上使用仓库内 Gradle Wrapper 成功构建 Debug APK。 +- [x] 上游已有 Android 单元测试已执行；没有测试时明确记录。 +- [x] 有测试设备时完成安装和启动 smoke，并记录设备/Android 版本；无设备时任务 不能把设备 smoke 写成已通过。 -- [ ] 未加入真实 API key、签名密码、设备 token 或本地 SDK 绝对路径。 -- [ ] 已处理缺失的 `google-services.json`：提供本地忽略配置，或明确移除不需要的 +- [x] 未加入真实 API key、签名密码、设备 token 或本地 SDK 绝对路径。 +- [x] 已处理缺失的 `google-services.json`：提供本地忽略配置，或明确移除不需要的 Firebase 插件和依赖。 -- [ ] `03-tech-stack.md` 和 `current-state.md` 不再把已核实信息标为待定。 -- [ ] `init.ps1`/`init.sh` 与文档中的当前命令一致，或因明确 blocker 保持主动失败。 +- [x] `03-tech-stack.md` 和 `current-state.md` 不再把已核实信息标为待定。 +- [x] `init.ps1`/`init.sh` 与文档中的当前命令一致，或因明确 blocker 保持主动失败。 建议验证命令形状，最终以接入工程为准： @@ -108,4 +110,31 @@ Set-Location android-buyer - 用户确认第一层使用本机蝦皮订单文本和同订单号参考图作为真实样本；已将输入契约 和后续 T-004 同步到需求、架构、编码规则及路线图。 - 本次只建立文档与脚本提交；Gradle 构建、测试、APK 安装和设备 smoke 均未运行。 -- T-001 保持 `DOING`，剩余 blocker 见 `docs/current-state.md`。 +- 本阶段结束时 T-001 仍保持 `DOING`，当时的 blocker 见 `docs/current-state.md` 历史。 + +### 2026-07-25：Android 可运行基线 + +- 选择 `main@c8a6d7f03422eb01744b01f3ee77bf7757741f7e` 作为导入基线。无障碍分支 + 落后 `main` 的修复和 1.4.2 能力，后续只选择性移植，不整体切换。 +- `android-buyer/UPSTREAM.md` 记录来源和调整；上游 MIT `LICENSE` 已保留。 +- 安装 JDK 17 对应的 Android SDK 34、Build Tools 34.0.0、Command-line Tools + 22.0 和 Platform Tools 37.0.0；没有把本机 SDK 绝对路径提交到仓库。 +- 新版 Command-line Tools 与 AGP 8.2 会输出 SDK XML v4/v3 兼容警告，但任务和 APK + 构建均成功；这是非阻塞工具版本提示，不作为通过证据本身。 +- 上游缺少 `gradlew.bat`，已用 Gradle 8.2 Wrapper 任务生成；Wrapper 固定官方 + SHA-256、120 秒网络超时并关闭生成阶段 URL 探测。 +- 移除需要私有 `google-services.json` 的 Firebase Analytics/Crashlytics 和云端 + 上报设置；保留本地 `CrashHandler`。 +- `.\gradlew.bat tasks --no-daemon`：成功。 +- `.\gradlew.bat test --no-daemon`：成功；Debug/Release 单元测试均为 `NO-SOURCE`， + 上游没有 `src/test` 或 `src/androidTest`。 +- `.\gradlew.bat assembleDebug --no-daemon`：成功；生成 `app-debug.apk`，当次 + SHA-256 为 `745C8E23B5C1575E9C71F6FF583BEEA3800759E9C611AD0C45D9062A856F68C4`。 +- `gradlew installDebug` 在 Android 16 上由 AGP 8.2/ddmlib 返回 `-99`；SDK ADB + 37.0.0 执行 `adb install -r -t` 成功，标准脚本采用该可靠路径。 +- `$env:RUN_START_COMMAND="1"; .\init.ps1`：成功完成测试任务、APK 构建、安装和 + 冷启动。 +- 真机 smoke：OnePlus PKG110，Android 16/API 36；MainActivity 冷启动约 0.9 秒， + 进程存活、前台窗口正确、logcat 无崩溃，画面正常停在 Shizuku 使用指南。 +- Android Studio `winget` 静默安装卡住并已终止；命令行构建不依赖 IDE，因此不作为 + T-001 blocker。 diff --git a/docs/tasks/T-004.md b/docs/tasks/T-004.md index 1268776..cf5d576 100644 --- a/docs/tasks/T-004.md +++ b/docs/tasks/T-004.md @@ -66,6 +66,6 @@ JPEG。正式私有目录、额外图片类型和一单多 SKU 规则仍未固 ## 执行记录 -尚未开始，等待 T-001 完成。前置检查已确认一组同名文本/JPEG 存在，文本字段结构和 -图片解码有效；未记录或提交实际订单号、店铺名、商品内容及图片。实现时仍需确定正式 -私有目录和一单多 SKU 规则。 +尚未开始，依赖 T-001 已满足。前置检查已确认一组同名文本/JPEG 存在，文本字段结构 +和图片解码有效；未记录或提交实际订单号、店铺名、商品内容及图片。实现时仍需确定 +正式私有目录和一单多 SKU 规则。 diff --git a/init.ps1 b/init.ps1 index 8c43757..4e4fe89 100644 --- a/init.ps1 +++ b/init.ps1 @@ -3,31 +3,68 @@ $ErrorActionPreference = "Stop" Set-Location -Path $PSScriptRoot -# T-001 接入真实工程后替换。未配置时主动失败，防止把占位命令当成有效基线。 -$InstallCmd = "__REPLACE_AFTER_T001__" -$VerifyCmd = "__REPLACE_AFTER_T001__" -$StartCmd = "__REPLACE_AFTER_T001__" - -foreach ($entry in @{ - InstallCmd = $InstallCmd - VerifyCmd = $VerifyCmd - StartCmd = $StartCmd -}.GetEnumerator()) { - if ($entry.Value -like "__REPLACE_*") { - [Console]::Error.WriteLine( - "项目尚未完成 T-001；请配置 $($entry.Key)，并同步 docs/03-tech-stack.md 和 docs/current-state.md。" - ) - exit 2 - } +$sdkRoot = if ($env:ANDROID_SDK_ROOT) { + $env:ANDROID_SDK_ROOT +} elseif ($env:ANDROID_HOME) { + $env:ANDROID_HOME +} else { + Join-Path $env:LOCALAPPDATA "Android\Sdk" } -Write-Host "==> 当前目录: $($PWD.Path)" -Write-Host "==> 同步依赖" -Invoke-Expression $InstallCmd -Write-Host "==> 运行基础验证" -Invoke-Expression $VerifyCmd -Write-Host "==> 启动命令: $StartCmd" +$gradle = Join-Path $PSScriptRoot "android-buyer\gradlew.bat" +$adb = Join-Path $sdkRoot "platform-tools\adb.exe" +$apk = Join-Path $PSScriptRoot "android-buyer\app\build\outputs\apk\debug\app-debug.apk" + +if (-not (Test-Path -LiteralPath $gradle)) { + throw "缺少 Android Gradle Wrapper: $gradle" +} +if (-not (Test-Path -LiteralPath (Join-Path $sdkRoot "platforms\android-34"))) { + throw "缺少 Android SDK 34，请设置 ANDROID_SDK_ROOT 或 ANDROID_HOME。" +} +if (-not (Test-Path -LiteralPath (Join-Path $sdkRoot "build-tools\34.0.0"))) { + throw "缺少 Android SDK Build Tools 34.0.0。" +} +if (-not (Get-Command java -ErrorAction SilentlyContinue)) { + throw "缺少 JDK 17，java 不在 PATH 中。" +} +$javaVersion = (& java -version 2>&1) -join "`n" +if ($javaVersion -notmatch 'version "17(?:\.|")') { + throw "需要 JDK 17，当前 java -version 输出: $javaVersion" +} + +$env:ANDROID_HOME = $sdkRoot +$env:ANDROID_SDK_ROOT = $sdkRoot + +Write-Host "==> 验证 Android 工程" +Push-Location (Join-Path $PSScriptRoot "android-buyer") +try { + & $gradle test assembleDebug --no-daemon + if ($LASTEXITCODE -ne 0) { + throw "Android 构建失败，退出码: $LASTEXITCODE" + } +} finally { + Pop-Location +} + +Write-Host "==> Debug APK: $apk" if ($env:RUN_START_COMMAND -eq "1") { - Invoke-Expression $StartCmd + if (-not (Test-Path -LiteralPath $adb)) { + throw "缺少 SDK Platform Tools: $adb" + } + + $devices = & $adb devices + if (($devices -join "`n") -notmatch "\sdevice\s*$") { + throw "没有已授权的 Android 设备。" + } + + Write-Host "==> 安装并启动 Debug APK" + & $adb install -r -t $apk + if ($LASTEXITCODE -ne 0) { + throw "APK 安装失败，退出码: $LASTEXITCODE" + } + & $adb shell am start -W -n "com.roubao.autopilot/.MainActivity" + if ($LASTEXITCODE -ne 0) { + throw "App 启动失败，退出码: $LASTEXITCODE" + } } diff --git a/init.sh b/init.sh old mode 100644 new mode 100755 index 5ca019f..876bda2 --- a/init.sh +++ b/init.sh @@ -3,26 +3,56 @@ set -euo pipefail ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -cd "$ROOT_DIR" +ANDROID_DIR="$ROOT_DIR/android-buyer" +SDK_ROOT="${ANDROID_SDK_ROOT:-${ANDROID_HOME:-$HOME/Android/Sdk}}" +GRADLE="$ANDROID_DIR/gradlew" +ADB="$SDK_ROOT/platform-tools/adb" +APK="$ANDROID_DIR/app/build/outputs/apk/debug/app-debug.apk" -# T-001 接入真实工程后替换。 -INSTALL_CMD=(__REPLACE_AFTER_T001__) -VERIFY_CMD=(__REPLACE_AFTER_T001__) -START_CMD=(__REPLACE_AFTER_T001__) +if [[ ! -x "$GRADLE" ]]; then + echo "ERROR: Android Gradle Wrapper is missing or not executable: $GRADLE" >&2 + exit 2 +fi +if [[ ! -d "$SDK_ROOT/platforms/android-34" ]]; then + echo "ERROR: Android SDK 34 is missing; set ANDROID_SDK_ROOT or ANDROID_HOME." >&2 + exit 2 +fi +if [[ ! -d "$SDK_ROOT/build-tools/34.0.0" ]]; then + echo "ERROR: Android SDK Build Tools 34.0.0 is missing." >&2 + exit 2 +fi +if ! command -v java >/dev/null 2>&1; then + echo "ERROR: JDK 17 is missing from PATH." >&2 + exit 2 +fi +JAVA_VERSION="$(java -version 2>&1 | head -n 1)" +if [[ "$JAVA_VERSION" != *'version "17.'* ]]; then + echo "ERROR: JDK 17 is required; found: $JAVA_VERSION" >&2 + exit 2 +fi -for value in "${INSTALL_CMD[0]}" "${VERIFY_CMD[0]}" "${START_CMD[0]}"; do - if [[ "$value" == __REPLACE_* ]]; then - echo "ERROR: 项目尚未完成 T-001，请配置 init.sh 并同步技术栈和当前状态文档。" - exit 2 - fi -done +export ANDROID_HOME="$SDK_ROOT" +export ANDROID_SDK_ROOT="$SDK_ROOT" -"${INSTALL_CMD[@]}" -"${VERIFY_CMD[@]}" -printf '启动命令:' -printf ' %q' "${START_CMD[@]}" -printf '\n' +echo "==> Verifying Android project" +( + cd "$ANDROID_DIR" + "$GRADLE" test assembleDebug --no-daemon +) + +echo "==> Debug APK: $APK" if [[ "${RUN_START_COMMAND:-0}" == "1" ]]; then - exec "${START_CMD[@]}" + if [[ ! -x "$ADB" ]]; then + echo "ERROR: Android Platform Tools are missing: $ADB" >&2 + exit 2 + fi + if ! "$ADB" devices | grep -qE $'\tdevice$'; then + echo "ERROR: No authorized Android device is connected." >&2 + exit 2 + fi + + echo "==> Installing and starting Debug APK" + "$ADB" install -r -t "$APK" + "$ADB" shell am start -W -n "com.roubao.autopilot/.MainActivity" fi diff --git a/progress.md b/progress.md index a317760..e72d4d2 100644 --- a/progress.md +++ b/progress.md @@ -38,3 +38,10 @@ 和参考图，规范化为与未来 HTTP 接口一致的探针任务。 - 影响：新增 T-004；私有订单资料和生成物不得进入 Git。首份样例已确认 UTF-8 四行 标签和同名 JPEG，Android workflow 只依赖统一 `TaskSource`。 + +## 2026-07-25 Android 可运行基线 + +- 类型：阶段完成 +- 内容：完成 T-001；导入 Roubao `main@c8a6d7f`，移除不可复现的 Firebase 私有 + 配置依赖，补齐 Windows Gradle Wrapper，并在 Android 16 真机完成构建、安装和启动。 +- 影响：Phase 0 的设备、workflow 和蝦皮样本任务可以开始；无障碍分支只选择性移植。