mirror of
https://github.com/hansjone/oclaw.git
synced 2026-10-10 01:20:46 +08:00
重构主控编排与运行时预热链路,统一工作区提示词/专家调度协议并补齐 wiki 记忆注入与写回闭环。
同时收敛启动与运维脚本默认行为(含 wiki worker)、更新 Admin 可观测性与相关测试,降低首轮时延并提高运行稳定性。 Made-with: Cursor
This commit is contained in:
parent
4a23b715a2
commit
dbbe3add6a
14438 changed files with 2693620 additions and 2546 deletions
294
openclaw/apps/android/README.md
Normal file
294
openclaw/apps/android/README.md
Normal file
|
|
@ -0,0 +1,294 @@
|
|||
## OpenClaw Android App
|
||||
|
||||
Status: **extremely alpha**. The app is actively being rebuilt from the ground up.
|
||||
|
||||
### Rebuild Checklist
|
||||
|
||||
- [x] New 4-step onboarding flow
|
||||
- [x] Connect tab with `Setup Code` + `Manual` modes
|
||||
- [x] Encrypted persistence for gateway setup/auth state
|
||||
- [x] Chat UI restyled
|
||||
- [x] Settings UI restyled and de-duplicated (gateway controls moved to Connect)
|
||||
- [x] QR code scanning in onboarding
|
||||
- [x] Performance improvements
|
||||
- [x] Streaming support in chat UI
|
||||
- [x] Request camera/location and other permissions in onboarding/settings flow
|
||||
- [x] Push notifications for gateway/chat status updates
|
||||
- [x] Security hardening (biometric lock, token handling, safer defaults)
|
||||
- [x] Voice tab full functionality
|
||||
- [x] Screen tab full functionality
|
||||
- [ ] Full end-to-end QA and release hardening
|
||||
|
||||
## Open in Android Studio
|
||||
|
||||
- Open the folder `apps/android`.
|
||||
|
||||
## Build / Run
|
||||
|
||||
```bash
|
||||
cd apps/android
|
||||
./gradlew :app:assemblePlayDebug
|
||||
./gradlew :app:installPlayDebug
|
||||
./gradlew :app:testPlayDebugUnitTest
|
||||
cd ../..
|
||||
bun run android:bundle:release
|
||||
```
|
||||
|
||||
Third-party debug flavor:
|
||||
|
||||
```bash
|
||||
cd apps/android
|
||||
./gradlew :app:assembleThirdPartyDebug
|
||||
./gradlew :app:installThirdPartyDebug
|
||||
./gradlew :app:testThirdPartyDebugUnitTest
|
||||
```
|
||||
|
||||
`bun run android:bundle:release` auto-bumps Android `versionName`/`versionCode` in `apps/android/app/build.gradle.kts`, then builds two signed release bundles:
|
||||
|
||||
- Play build: `apps/android/build/release-bundles/openclaw-<version>-play-release.aab`
|
||||
- Third-party build: `apps/android/build/release-bundles/openclaw-<version>-third-party-release.aab`
|
||||
|
||||
Flavor-specific direct Gradle tasks:
|
||||
|
||||
```bash
|
||||
cd apps/android
|
||||
./gradlew :app:bundlePlayRelease
|
||||
./gradlew :app:bundleThirdPartyRelease
|
||||
```
|
||||
|
||||
## Kotlin Lint + Format
|
||||
|
||||
```bash
|
||||
pnpm android:lint
|
||||
pnpm android:format
|
||||
```
|
||||
|
||||
Android framework/resource lint (separate pass):
|
||||
|
||||
```bash
|
||||
pnpm android:lint:android
|
||||
```
|
||||
|
||||
Direct Gradle tasks:
|
||||
|
||||
```bash
|
||||
cd apps/android
|
||||
./gradlew :app:ktlintCheck :benchmark:ktlintCheck
|
||||
./gradlew :app:ktlintFormat :benchmark:ktlintFormat
|
||||
./gradlew :app:lintDebug
|
||||
```
|
||||
|
||||
`gradlew` auto-detects the Android SDK at `~/Library/Android/sdk` (macOS default) if `ANDROID_SDK_ROOT` / `ANDROID_HOME` are unset.
|
||||
|
||||
## Macrobenchmark (Startup + Frame Timing)
|
||||
|
||||
```bash
|
||||
cd apps/android
|
||||
./gradlew :benchmark:connectedDebugAndroidTest
|
||||
```
|
||||
|
||||
Reports are written under:
|
||||
|
||||
- `apps/android/benchmark/build/reports/androidTests/connected/`
|
||||
|
||||
## Perf CLI (low-noise)
|
||||
|
||||
Deterministic startup measurement + hotspot extraction with compact CLI output:
|
||||
|
||||
```bash
|
||||
cd apps/android
|
||||
./scripts/perf-startup-benchmark.sh
|
||||
./scripts/perf-startup-hotspots.sh
|
||||
```
|
||||
|
||||
Benchmark script behavior:
|
||||
|
||||
- Runs only `StartupMacrobenchmark#coldStartup` (10 iterations).
|
||||
- Prints median/min/max/COV in one line.
|
||||
- Writes timestamped snapshot JSON to `apps/android/benchmark/results/`.
|
||||
- Auto-compares with previous local snapshot (or pass explicit baseline: `--baseline <old-benchmarkData.json>`).
|
||||
|
||||
Hotspot script behavior:
|
||||
|
||||
- Ensures debug app installed, captures startup `simpleperf` data for `.MainActivity`.
|
||||
- Prints top DSOs, top symbols, and key app-path clues (Compose/MainActivity/WebView).
|
||||
- Writes raw `perf.data` path for deeper follow-up if needed.
|
||||
|
||||
## Run on a Real Android Phone (USB)
|
||||
|
||||
1) On phone, enable **Developer options** + **USB debugging**.
|
||||
2) Connect by USB and accept the debugging trust prompt on phone.
|
||||
3) Verify ADB can see the device:
|
||||
|
||||
```bash
|
||||
adb devices -l
|
||||
```
|
||||
|
||||
4) Install + launch debug build:
|
||||
|
||||
```bash
|
||||
pnpm android:install
|
||||
pnpm android:run
|
||||
```
|
||||
|
||||
If `adb devices -l` shows `unauthorized`, re-plug and accept the trust prompt again.
|
||||
|
||||
### USB-only gateway testing (no LAN dependency)
|
||||
|
||||
Use `adb reverse` so Android `localhost:18789` tunnels to your laptop `localhost:18789`.
|
||||
|
||||
Terminal A (gateway):
|
||||
|
||||
```bash
|
||||
pnpm openclaw gateway --port 18789 --verbose
|
||||
```
|
||||
|
||||
Terminal B (USB tunnel):
|
||||
|
||||
```bash
|
||||
adb reverse tcp:18789 tcp:18789
|
||||
```
|
||||
|
||||
Then in app **Connect → Manual**:
|
||||
|
||||
- Host: `127.0.0.1`
|
||||
- Port: `18789`
|
||||
- TLS: off
|
||||
|
||||
## Hot Reload / Fast Iteration
|
||||
|
||||
This app is native Kotlin + Jetpack Compose.
|
||||
|
||||
- For Compose UI edits: use Android Studio **Live Edit** on a debug build (works on physical devices; project `minSdk=31` already meets API requirement).
|
||||
- For many non-structural code/resource changes: use Android Studio **Apply Changes**.
|
||||
- For structural/native/manifest/Gradle changes: do full reinstall (`pnpm android:run`).
|
||||
- Canvas web content already supports live reload when loaded from Gateway `__openclaw__/canvas/` (see `docs/platforms/android.md`).
|
||||
|
||||
## Connect / Pair
|
||||
|
||||
1) Start the gateway (on your main machine):
|
||||
|
||||
```bash
|
||||
pnpm openclaw gateway --port 18789 --verbose
|
||||
```
|
||||
|
||||
2) In the Android app:
|
||||
|
||||
- Open the **Connect** tab.
|
||||
- Use **Setup Code** or **Manual** mode to connect.
|
||||
|
||||
3) Approve pairing (on the gateway machine):
|
||||
|
||||
```bash
|
||||
openclaw devices list
|
||||
openclaw devices approve <requestId>
|
||||
```
|
||||
|
||||
More details: `docs/platforms/android.md`.
|
||||
|
||||
## Permissions
|
||||
|
||||
- Discovery:
|
||||
- Android 13+ (`API 33+`): `NEARBY_WIFI_DEVICES`
|
||||
- Android 12 and below: `ACCESS_FINE_LOCATION` (required for NSD scanning)
|
||||
- Foreground service notification (Android 13+): `POST_NOTIFICATIONS`
|
||||
- Camera:
|
||||
- `CAMERA` for `camera.snap` and `camera.clip`
|
||||
- `RECORD_AUDIO` for `camera.clip` when `includeAudio=true`
|
||||
|
||||
## Google Play Restricted Permissions
|
||||
|
||||
As of March 19, 2026, these manifest permissions are the main Google Play policy risk for this app:
|
||||
|
||||
- `READ_SMS`
|
||||
- `SEND_SMS`
|
||||
- `READ_CALL_LOG`
|
||||
|
||||
Why these matter:
|
||||
|
||||
- Google Play treats SMS and Call Log access as highly restricted. In most cases, Play only allows them for the default SMS app, default Phone app, default Assistant, or a narrow policy exception.
|
||||
- Review usually involves a `Permissions Declaration Form`, policy justification, and demo video evidence in Play Console.
|
||||
- If we want a Play-safe build, these should be the first permissions removed behind a dedicated product flavor / variant.
|
||||
|
||||
Current OpenClaw Android implication:
|
||||
|
||||
- APK / sideload build can keep SMS and Call Log features.
|
||||
- Google Play build should exclude SMS send/search and Call Log search unless the product is intentionally positioned and approved as a default-handler exception case.
|
||||
- The repo now ships this split as Android product flavors:
|
||||
- `play`: removes `READ_SMS`, `SEND_SMS`, and `READ_CALL_LOG`, and hides SMS / Call Log surfaces in onboarding, settings, and advertised node capabilities.
|
||||
- `thirdParty`: keeps the full permission set and the existing SMS / Call Log functionality.
|
||||
|
||||
Policy links:
|
||||
|
||||
- [Google Play SMS and Call Log policy](https://support.google.com/googleplay/android-developer/answer/10208820?hl=en)
|
||||
- [Google Play sensitive permissions policy hub](https://support.google.com/googleplay/android-developer/answer/16558241)
|
||||
- [Android default handlers guide](https://developer.android.com/guide/topics/permissions/default-handlers)
|
||||
|
||||
Other Play-restricted surfaces to watch if added later:
|
||||
|
||||
- `ACCESS_BACKGROUND_LOCATION`
|
||||
- `MANAGE_EXTERNAL_STORAGE`
|
||||
- `QUERY_ALL_PACKAGES`
|
||||
- `REQUEST_INSTALL_PACKAGES`
|
||||
- `AccessibilityService`
|
||||
|
||||
Reference links:
|
||||
|
||||
- [Background location policy](https://support.google.com/googleplay/android-developer/answer/9799150)
|
||||
- [AccessibilityService policy](https://support.google.com/googleplay/android-developer/answer/10964491?hl=en-GB)
|
||||
- [Photo and Video Permissions policy](https://support.google.com/googleplay/android-developer/answer/14594990)
|
||||
|
||||
## Integration Capability Test (Preconditioned)
|
||||
|
||||
This suite assumes setup is already done manually. It does **not** install/run/pair automatically.
|
||||
|
||||
Pre-req checklist:
|
||||
|
||||
1) Gateway is running and reachable from the Android app.
|
||||
2) Android app is connected to that gateway and `openclaw nodes status` shows it as paired + connected.
|
||||
3) App stays unlocked and in foreground for the whole run.
|
||||
4) Open the app **Screen** tab and keep it active during the run (canvas/A2UI commands require the canvas WebView attached there).
|
||||
5) Grant runtime permissions for capabilities you expect to pass (camera/mic/location/notification listener/location, etc.).
|
||||
6) No interactive system dialogs should be pending before test start.
|
||||
7) Canvas host is enabled and reachable from the device (do not run gateway with `OPENCLAW_SKIP_CANVAS_HOST=1`; startup logs should include `canvas host mounted at .../__openclaw__/`).
|
||||
8) Local operator test client pairing is approved. If first run fails with `pairing required`, approve latest pending device pairing request, then rerun:
|
||||
9) For A2UI checks, keep the app on **Screen** tab; the node now auto-refreshes canvas capability once on first A2UI reachability failure (TTL-safe retry).
|
||||
|
||||
```bash
|
||||
openclaw devices list
|
||||
openclaw devices approve --latest
|
||||
```
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
pnpm android:test:integration
|
||||
```
|
||||
|
||||
Optional overrides:
|
||||
|
||||
- `OPENCLAW_ANDROID_GATEWAY_URL=ws://...` (default: from your local OpenClaw config)
|
||||
- `OPENCLAW_ANDROID_GATEWAY_TOKEN=...`
|
||||
- `OPENCLAW_ANDROID_GATEWAY_PASSWORD=...`
|
||||
- `OPENCLAW_ANDROID_NODE_ID=...` or `OPENCLAW_ANDROID_NODE_NAME=...`
|
||||
|
||||
What it does:
|
||||
|
||||
- Reads `node.describe` command list from the selected Android node.
|
||||
- Invokes advertised non-interactive commands.
|
||||
- Skips `screen.record` in this suite (Android requires interactive per-invocation screen-capture consent).
|
||||
- Asserts command contracts (success or expected deterministic error for safe-invalid calls like `sms.send` and `notifications.actions`).
|
||||
|
||||
Common failure quick-fixes:
|
||||
|
||||
- `pairing required` before tests start:
|
||||
- approve pending device pairing (`openclaw devices approve --latest`) and rerun.
|
||||
- `A2UI host not reachable` / `A2UI_HOST_NOT_CONFIGURED`:
|
||||
- ensure gateway canvas host is running and reachable, keep the app on the **Screen** tab. The app will auto-refresh canvas capability once; if it still fails, reconnect app and rerun.
|
||||
- `NODE_BACKGROUND_UNAVAILABLE: canvas unavailable`:
|
||||
- app is not effectively ready for canvas commands; keep app foregrounded and **Screen** tab active.
|
||||
|
||||
## Contributions
|
||||
|
||||
This Android app is currently being rebuilt.
|
||||
Maintainer: @obviyus. For issues/questions/contributions, please open an issue or reach out on Discord.
|
||||
93
openclaw/apps/android/THIRD_PARTY_LICENSES/MANROPE_OFL.txt
Normal file
93
openclaw/apps/android/THIRD_PARTY_LICENSES/MANROPE_OFL.txt
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
Copyright 2018 The Manrope Project Authors (https://github.com/sharanda/manrope)
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
http://scripts.sil.org/OFL
|
||||
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font creation
|
||||
efforts of academic and linguistic communities, and to provide a free and
|
||||
open framework in which fonts may be shared and improved in partnership
|
||||
with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply
|
||||
to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as
|
||||
distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to a
|
||||
new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
||||
redistribute, and sell modified and unmodified copies of the Font
|
||||
Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components,
|
||||
in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the corresponding
|
||||
Copyright Holder. This restriction only applies to the primary font name as
|
||||
presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created
|
||||
using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
290
openclaw/apps/android/app/build.gradle.kts
Normal file
290
openclaw/apps/android/app/build.gradle.kts
Normal file
|
|
@ -0,0 +1,290 @@
|
|||
import com.android.build.api.variant.impl.VariantOutputImpl
|
||||
|
||||
val dnsjavaInetAddressResolverService = "META-INF/services/java.net.spi.InetAddressResolverProvider"
|
||||
|
||||
val androidStoreFile = providers.gradleProperty("OPENCLAW_ANDROID_STORE_FILE").orNull?.takeIf { it.isNotBlank() }
|
||||
val androidStorePassword = providers.gradleProperty("OPENCLAW_ANDROID_STORE_PASSWORD").orNull?.takeIf { it.isNotBlank() }
|
||||
val androidKeyAlias = providers.gradleProperty("OPENCLAW_ANDROID_KEY_ALIAS").orNull?.takeIf { it.isNotBlank() }
|
||||
val androidKeyPassword = providers.gradleProperty("OPENCLAW_ANDROID_KEY_PASSWORD").orNull?.takeIf { it.isNotBlank() }
|
||||
val resolvedAndroidStoreFile =
|
||||
androidStoreFile?.let { storeFilePath ->
|
||||
if (storeFilePath.startsWith("~/")) {
|
||||
"${System.getProperty("user.home")}/${storeFilePath.removePrefix("~/")}"
|
||||
} else {
|
||||
storeFilePath
|
||||
}
|
||||
}
|
||||
|
||||
val hasAndroidReleaseSigning =
|
||||
listOf(resolvedAndroidStoreFile, androidStorePassword, androidKeyAlias, androidKeyPassword).all { it != null }
|
||||
|
||||
val wantsAndroidReleaseBuild =
|
||||
gradle.startParameter.taskNames.any { taskName ->
|
||||
taskName.contains("Release", ignoreCase = true) ||
|
||||
Regex("""(^|:)(bundle|assemble)$""").containsMatchIn(taskName)
|
||||
}
|
||||
|
||||
if (wantsAndroidReleaseBuild && !hasAndroidReleaseSigning) {
|
||||
error(
|
||||
"Missing Android release signing properties. Set OPENCLAW_ANDROID_STORE_FILE, " +
|
||||
"OPENCLAW_ANDROID_STORE_PASSWORD, OPENCLAW_ANDROID_KEY_ALIAS, and " +
|
||||
"OPENCLAW_ANDROID_KEY_PASSWORD in ~/.gradle/gradle.properties.",
|
||||
)
|
||||
}
|
||||
|
||||
plugins {
|
||||
id("com.android.application")
|
||||
id("org.jlleitschuh.gradle.ktlint")
|
||||
id("org.jetbrains.kotlin.plugin.compose")
|
||||
id("org.jetbrains.kotlin.plugin.serialization")
|
||||
}
|
||||
|
||||
android {
|
||||
namespace = "ai.openclaw.app"
|
||||
compileSdk = 36
|
||||
|
||||
// Release signing is local-only; keep the keystore path and passwords out of the repo.
|
||||
signingConfigs {
|
||||
if (hasAndroidReleaseSigning) {
|
||||
create("release") {
|
||||
storeFile = project.file(checkNotNull(resolvedAndroidStoreFile))
|
||||
storePassword = checkNotNull(androidStorePassword)
|
||||
keyAlias = checkNotNull(androidKeyAlias)
|
||||
keyPassword = checkNotNull(androidKeyPassword)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
sourceSets {
|
||||
getByName("main") {
|
||||
assets.directories.add("../../shared/OpenClawKit/Sources/OpenClawKit/Resources")
|
||||
}
|
||||
}
|
||||
|
||||
defaultConfig {
|
||||
applicationId = "ai.openclaw.app"
|
||||
minSdk = 31
|
||||
targetSdk = 36
|
||||
versionCode = 2026042000
|
||||
versionName = "2026.4.20"
|
||||
ndk {
|
||||
// Support all major ABIs — native libs are tiny (~47 KB per ABI)
|
||||
abiFilters += listOf("armeabi-v7a", "arm64-v8a", "x86", "x86_64")
|
||||
}
|
||||
}
|
||||
|
||||
flavorDimensions += "store"
|
||||
|
||||
productFlavors {
|
||||
create("play") {
|
||||
dimension = "store"
|
||||
buildConfigField("boolean", "OPENCLAW_ENABLE_SMS", "false")
|
||||
buildConfigField("boolean", "OPENCLAW_ENABLE_CALL_LOG", "false")
|
||||
}
|
||||
create("thirdParty") {
|
||||
dimension = "store"
|
||||
buildConfigField("boolean", "OPENCLAW_ENABLE_SMS", "true")
|
||||
buildConfigField("boolean", "OPENCLAW_ENABLE_CALL_LOG", "true")
|
||||
}
|
||||
}
|
||||
|
||||
buildTypes {
|
||||
release {
|
||||
if (hasAndroidReleaseSigning) {
|
||||
signingConfig = signingConfigs.getByName("release")
|
||||
}
|
||||
isMinifyEnabled = true
|
||||
isShrinkResources = true
|
||||
ndk {
|
||||
debugSymbolLevel = "SYMBOL_TABLE"
|
||||
}
|
||||
proguardFiles(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro")
|
||||
}
|
||||
debug {
|
||||
isMinifyEnabled = false
|
||||
}
|
||||
}
|
||||
|
||||
buildFeatures {
|
||||
compose = true
|
||||
buildConfig = true
|
||||
}
|
||||
|
||||
compileOptions {
|
||||
sourceCompatibility = JavaVersion.VERSION_17
|
||||
targetCompatibility = JavaVersion.VERSION_17
|
||||
}
|
||||
|
||||
packaging {
|
||||
resources {
|
||||
excludes +=
|
||||
setOf(
|
||||
"/META-INF/{AL2.0,LGPL2.1}",
|
||||
"/META-INF/*.version",
|
||||
"/META-INF/LICENSE*.txt",
|
||||
"DebugProbesKt.bin",
|
||||
"kotlin-tooling-metadata.json",
|
||||
"org/bouncycastle/pqc/crypto/picnic/lowmcL1.bin.properties",
|
||||
"org/bouncycastle/pqc/crypto/picnic/lowmcL3.bin.properties",
|
||||
"org/bouncycastle/pqc/crypto/picnic/lowmcL5.bin.properties",
|
||||
"org/bouncycastle/x509/CertPathReviewerMessages*.properties",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
lint {
|
||||
disable +=
|
||||
setOf(
|
||||
"AndroidGradlePluginVersion",
|
||||
"GradleDependency",
|
||||
"IconLauncherShape",
|
||||
"NewerVersionAvailable",
|
||||
)
|
||||
warningsAsErrors = true
|
||||
}
|
||||
|
||||
testOptions {
|
||||
unitTests.isIncludeAndroidResources = true
|
||||
}
|
||||
}
|
||||
|
||||
androidComponents {
|
||||
onVariants { variant ->
|
||||
variant.outputs
|
||||
.filterIsInstance<VariantOutputImpl>()
|
||||
.forEach { output ->
|
||||
val versionName = output.versionName.orNull ?: "0"
|
||||
val buildType = variant.buildType
|
||||
val flavorName = variant.flavorName?.takeIf { it.isNotBlank() }
|
||||
val outputFileName =
|
||||
if (flavorName == null) {
|
||||
"openclaw-$versionName-$buildType.apk"
|
||||
} else {
|
||||
"openclaw-$versionName-$flavorName-$buildType.apk"
|
||||
}
|
||||
output.outputFileName = outputFileName
|
||||
}
|
||||
}
|
||||
}
|
||||
kotlin {
|
||||
compilerOptions {
|
||||
jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17)
|
||||
allWarningsAsErrors.set(true)
|
||||
}
|
||||
}
|
||||
|
||||
ktlint {
|
||||
android.set(true)
|
||||
ignoreFailures.set(false)
|
||||
filter {
|
||||
exclude("**/build/**")
|
||||
}
|
||||
}
|
||||
|
||||
dependencies {
|
||||
val composeBom = platform("androidx.compose:compose-bom:2026.02.00")
|
||||
implementation(composeBom)
|
||||
androidTestImplementation(composeBom)
|
||||
|
||||
implementation("androidx.core:core-ktx:1.17.0")
|
||||
implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.10.0")
|
||||
implementation("androidx.activity:activity-compose:1.12.2")
|
||||
implementation("androidx.webkit:webkit:1.15.0")
|
||||
|
||||
implementation("androidx.compose.ui:ui")
|
||||
implementation("androidx.compose.ui:ui-tooling-preview")
|
||||
implementation("androidx.compose.material3:material3")
|
||||
// material-icons-extended pulled in full icon set (~20 MB DEX). Only ~18 icons used.
|
||||
// R8 will tree-shake unused icons when minify is enabled on release builds.
|
||||
implementation("androidx.compose.material:material-icons-extended")
|
||||
|
||||
debugImplementation("androidx.compose.ui:ui-tooling")
|
||||
|
||||
// Material Components (XML theme + resources)
|
||||
implementation("com.google.android.material:material:1.13.0")
|
||||
|
||||
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.10.2")
|
||||
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.10.0")
|
||||
|
||||
implementation("androidx.security:security-crypto:1.1.0")
|
||||
implementation("androidx.exifinterface:exifinterface:1.4.2")
|
||||
implementation("com.squareup.okhttp3:okhttp:5.3.2")
|
||||
implementation("org.bouncycastle:bcprov-jdk18on:1.83")
|
||||
implementation("org.commonmark:commonmark:0.27.1")
|
||||
implementation("org.commonmark:commonmark-ext-autolink:0.27.1")
|
||||
implementation("org.commonmark:commonmark-ext-gfm-strikethrough:0.27.1")
|
||||
implementation("org.commonmark:commonmark-ext-gfm-tables:0.27.1")
|
||||
implementation("org.commonmark:commonmark-ext-task-list-items:0.27.1")
|
||||
|
||||
// CameraX (for node.invoke camera.* parity)
|
||||
implementation("androidx.camera:camera-core:1.5.2")
|
||||
implementation("androidx.camera:camera-camera2:1.5.2")
|
||||
implementation("androidx.camera:camera-lifecycle:1.5.2")
|
||||
implementation("androidx.camera:camera-video:1.5.2")
|
||||
implementation("com.google.android.gms:play-services-code-scanner:16.1.0")
|
||||
|
||||
// Unicast DNS-SD (Wide-Area Bonjour) for tailnet discovery domains.
|
||||
implementation("dnsjava:dnsjava:3.6.4")
|
||||
|
||||
testImplementation("junit:junit:4.13.2")
|
||||
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.10.2")
|
||||
testImplementation("io.kotest:kotest-runner-junit5-jvm:6.1.3")
|
||||
testImplementation("io.kotest:kotest-assertions-core-jvm:6.1.3")
|
||||
testImplementation("com.squareup.okhttp3:mockwebserver:5.3.2")
|
||||
testImplementation("org.robolectric:robolectric:4.16.1")
|
||||
testRuntimeOnly("org.junit.vintage:junit-vintage-engine:6.0.2")
|
||||
}
|
||||
|
||||
tasks.withType<Test>().configureEach {
|
||||
useJUnitPlatform()
|
||||
}
|
||||
|
||||
androidComponents {
|
||||
onVariants(selector().withBuildType("release")) { variant ->
|
||||
val variantName = variant.name
|
||||
val variantNameCapitalized = variantName.replaceFirstChar(Char::titlecase)
|
||||
val stripTaskName = "strip${variantNameCapitalized}DnsjavaServiceDescriptor"
|
||||
val mergeTaskName = "merge${variantNameCapitalized}JavaResource"
|
||||
val minifyTaskName = "minify${variantNameCapitalized}WithR8"
|
||||
val mergedJar =
|
||||
layout.buildDirectory.file(
|
||||
"intermediates/merged_java_res/$variantName/$mergeTaskName/base.jar",
|
||||
)
|
||||
|
||||
val stripTask =
|
||||
tasks.register(stripTaskName) {
|
||||
inputs.file(mergedJar)
|
||||
outputs.file(mergedJar)
|
||||
|
||||
doLast {
|
||||
val jarFile = mergedJar.get().asFile
|
||||
if (!jarFile.exists()) {
|
||||
return@doLast
|
||||
}
|
||||
|
||||
val unpackDir = temporaryDir.resolve("merged-java-res")
|
||||
delete(unpackDir)
|
||||
copy {
|
||||
from(zipTree(jarFile))
|
||||
into(unpackDir)
|
||||
exclude(dnsjavaInetAddressResolverService)
|
||||
}
|
||||
delete(jarFile)
|
||||
ant.invokeMethod(
|
||||
"zip",
|
||||
mapOf(
|
||||
"destfile" to jarFile.absolutePath,
|
||||
"basedir" to unpackDir.absolutePath,
|
||||
),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
tasks.matching { it.name == mergeTaskName }.configureEach {
|
||||
finalizedBy(stripTask)
|
||||
}
|
||||
tasks.matching { it.name == minifyTaskName }.configureEach {
|
||||
dependsOn(stripTask)
|
||||
}
|
||||
}
|
||||
}
|
||||
8
openclaw/apps/android/app/proguard-rules.pro
vendored
Normal file
8
openclaw/apps/android/app/proguard-rules.pro
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
-dontwarn org.bouncycastle.**
|
||||
-dontwarn okhttp3.**
|
||||
-dontwarn okio.**
|
||||
-dontwarn com.sun.jna.**
|
||||
-dontwarn javax.naming.**
|
||||
-dontwarn lombok.Generated
|
||||
-dontwarn org.slf4j.impl.StaticLoggerBinder
|
||||
-dontwarn sun.net.spi.nameservice.NameServiceDescriptor
|
||||
92
openclaw/apps/android/app/src/main/AndroidManifest.xml
Normal file
92
openclaw/apps/android/app/src/main/AndroidManifest.xml
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
|
||||
<uses-permission android:name="android.permission.INTERNET" />
|
||||
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
|
||||
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
|
||||
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_DATA_SYNC" />
|
||||
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
|
||||
<uses-permission
|
||||
android:name="android.permission.NEARBY_WIFI_DEVICES"
|
||||
android:usesPermissionFlags="neverForLocation" />
|
||||
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
|
||||
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />
|
||||
<uses-permission android:name="android.permission.CAMERA" />
|
||||
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
||||
<uses-permission android:name="android.permission.SEND_SMS" />
|
||||
<uses-permission android:name="android.permission.READ_SMS" />
|
||||
<uses-permission android:name="android.permission.READ_MEDIA_IMAGES" />
|
||||
<uses-permission android:name="android.permission.READ_MEDIA_VISUAL_USER_SELECTED" />
|
||||
<uses-permission
|
||||
android:name="android.permission.READ_EXTERNAL_STORAGE"
|
||||
android:maxSdkVersion="32" />
|
||||
<uses-permission android:name="android.permission.READ_CONTACTS" />
|
||||
<uses-permission android:name="android.permission.WRITE_CONTACTS" />
|
||||
<uses-permission android:name="android.permission.READ_CALL_LOG" />
|
||||
<uses-permission android:name="android.permission.READ_CALENDAR" />
|
||||
<uses-permission android:name="android.permission.WRITE_CALENDAR" />
|
||||
<uses-permission android:name="android.permission.ACTIVITY_RECOGNITION" />
|
||||
<uses-feature
|
||||
android:name="android.hardware.camera"
|
||||
android:required="false" />
|
||||
<uses-feature
|
||||
android:name="android.hardware.telephony"
|
||||
android:required="false" />
|
||||
|
||||
<queries>
|
||||
<intent>
|
||||
<action android:name="android.intent.action.MAIN" />
|
||||
<category android:name="android.intent.category.LAUNCHER" />
|
||||
</intent>
|
||||
</queries>
|
||||
|
||||
<application
|
||||
android:name=".NodeApp"
|
||||
android:allowBackup="true"
|
||||
android:dataExtractionRules="@xml/data_extraction_rules"
|
||||
android:fullBackupContent="@xml/backup_rules"
|
||||
android:icon="@mipmap/ic_launcher"
|
||||
android:roundIcon="@mipmap/ic_launcher_round"
|
||||
android:label="@string/app_name"
|
||||
android:supportsRtl="true"
|
||||
android:networkSecurityConfig="@xml/network_security_config"
|
||||
android:theme="@style/Theme.OpenClawNode">
|
||||
<service
|
||||
android:name=".NodeForegroundService"
|
||||
android:exported="false"
|
||||
android:foregroundServiceType="dataSync" />
|
||||
<service
|
||||
android:name=".node.DeviceNotificationListenerService"
|
||||
android:label="@string/app_name"
|
||||
android:permission="android.permission.BIND_NOTIFICATION_LISTENER_SERVICE"
|
||||
android:exported="false">
|
||||
<intent-filter>
|
||||
<action android:name="android.service.notification.NotificationListenerService" />
|
||||
</intent-filter>
|
||||
</service>
|
||||
<provider
|
||||
android:name="androidx.core.content.FileProvider"
|
||||
android:authorities="${applicationId}.fileprovider"
|
||||
android:exported="false"
|
||||
android:grantUriPermissions="true">
|
||||
<meta-data
|
||||
android:name="android.support.FILE_PROVIDER_PATHS"
|
||||
android:resource="@xml/file_paths" />
|
||||
</provider>
|
||||
<activity
|
||||
android:name=".MainActivity"
|
||||
android:exported="true"
|
||||
android:windowSoftInputMode="adjustResize"
|
||||
android:configChanges="orientation|screenSize|screenLayout|smallestScreenSize|uiMode|density|keyboard|keyboardHidden|navigation">
|
||||
<meta-data
|
||||
android:name="android.app.shortcuts"
|
||||
android:resource="@xml/shortcuts" />
|
||||
<intent-filter>
|
||||
<action android:name="android.intent.action.MAIN" />
|
||||
<category android:name="android.intent.category.LAUNCHER" />
|
||||
</intent-filter>
|
||||
<intent-filter>
|
||||
<action android:name="android.intent.action.ASSIST" />
|
||||
<category android:name="android.intent.category.DEFAULT" />
|
||||
</intent-filter>
|
||||
</activity>
|
||||
</application>
|
||||
</manifest>
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
import android.content.Intent
|
||||
|
||||
const val actionAskOpenClaw = "ai.openclaw.app.action.ASK_OPENCLAW"
|
||||
const val extraAssistantPrompt = "prompt"
|
||||
|
||||
enum class HomeDestination {
|
||||
Connect,
|
||||
Chat,
|
||||
Voice,
|
||||
Screen,
|
||||
Settings,
|
||||
}
|
||||
|
||||
data class AssistantLaunchRequest(
|
||||
val source: String,
|
||||
val prompt: String?,
|
||||
val autoSend: Boolean,
|
||||
)
|
||||
|
||||
fun parseAssistantLaunchIntent(intent: Intent?): AssistantLaunchRequest? {
|
||||
val action = intent?.action ?: return null
|
||||
return when (action) {
|
||||
Intent.ACTION_ASSIST ->
|
||||
AssistantLaunchRequest(
|
||||
source = "assist",
|
||||
prompt = null,
|
||||
autoSend = false,
|
||||
)
|
||||
|
||||
actionAskOpenClaw -> {
|
||||
val prompt = intent.getStringExtra(extraAssistantPrompt)?.trim()?.ifEmpty { null }
|
||||
AssistantLaunchRequest(
|
||||
source = "app_action",
|
||||
prompt = prompt,
|
||||
autoSend = prompt != null,
|
||||
)
|
||||
}
|
||||
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
enum class CameraHudKind {
|
||||
Photo,
|
||||
Recording,
|
||||
Success,
|
||||
Error,
|
||||
}
|
||||
|
||||
data class CameraHudState(
|
||||
val token: Long,
|
||||
val kind: CameraHudKind,
|
||||
val message: String,
|
||||
)
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
import android.content.Context
|
||||
import android.os.Build
|
||||
import android.provider.Settings
|
||||
|
||||
object DeviceNames {
|
||||
fun bestDefaultNodeName(context: Context): String {
|
||||
val deviceName =
|
||||
runCatching {
|
||||
Settings.Global.getString(context.contentResolver, "device_name")
|
||||
}
|
||||
.getOrNull()
|
||||
?.trim()
|
||||
.orEmpty()
|
||||
|
||||
if (deviceName.isNotEmpty()) return deviceName
|
||||
|
||||
val model =
|
||||
listOfNotNull(Build.MANUFACTURER?.takeIf { it.isNotBlank() }, Build.MODEL?.takeIf { it.isNotBlank() })
|
||||
.joinToString(" ")
|
||||
.trim()
|
||||
|
||||
return model.ifEmpty { "Android Node" }
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
enum class LocationMode(val rawValue: String) {
|
||||
Off("off"),
|
||||
WhileUsing("whileUsing"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
fun fromRawValue(raw: String?): LocationMode {
|
||||
val normalized = raw?.trim()?.lowercase()
|
||||
if (normalized == "always") return WhileUsing
|
||||
return entries.firstOrNull { it.rawValue.lowercase() == normalized } ?: Off
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,85 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
import android.os.Bundle
|
||||
import android.view.WindowManager
|
||||
import androidx.activity.ComponentActivity
|
||||
import androidx.activity.compose.setContent
|
||||
import androidx.activity.viewModels
|
||||
import androidx.core.view.WindowCompat
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.lifecycle.Lifecycle
|
||||
import androidx.lifecycle.lifecycleScope
|
||||
import androidx.lifecycle.repeatOnLifecycle
|
||||
import ai.openclaw.app.ui.RootScreen
|
||||
import ai.openclaw.app.ui.OpenClawTheme
|
||||
import kotlinx.coroutines.launch
|
||||
|
||||
class MainActivity : ComponentActivity() {
|
||||
private val viewModel: MainViewModel by viewModels()
|
||||
private lateinit var permissionRequester: PermissionRequester
|
||||
private var didAttachRuntimeUi = false
|
||||
private var didStartNodeService = false
|
||||
|
||||
override fun onCreate(savedInstanceState: Bundle?) {
|
||||
super.onCreate(savedInstanceState)
|
||||
handleAssistantIntent(intent)
|
||||
WindowCompat.setDecorFitsSystemWindows(window, false)
|
||||
permissionRequester = PermissionRequester(this)
|
||||
|
||||
lifecycleScope.launch {
|
||||
repeatOnLifecycle(Lifecycle.State.STARTED) {
|
||||
viewModel.preventSleep.collect { enabled ->
|
||||
if (enabled) {
|
||||
window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
|
||||
} else {
|
||||
window.clearFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
lifecycleScope.launch {
|
||||
repeatOnLifecycle(Lifecycle.State.STARTED) {
|
||||
viewModel.runtimeInitialized.collect { ready ->
|
||||
if (!ready || didAttachRuntimeUi) return@collect
|
||||
viewModel.attachRuntimeUi(owner = this@MainActivity, permissionRequester = permissionRequester)
|
||||
didAttachRuntimeUi = true
|
||||
if (!didStartNodeService) {
|
||||
NodeForegroundService.start(this@MainActivity)
|
||||
didStartNodeService = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
setContent {
|
||||
OpenClawTheme {
|
||||
Surface(modifier = Modifier) {
|
||||
RootScreen(viewModel = viewModel)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override fun onStart() {
|
||||
super.onStart()
|
||||
viewModel.setForeground(true)
|
||||
}
|
||||
|
||||
override fun onStop() {
|
||||
viewModel.setForeground(false)
|
||||
super.onStop()
|
||||
}
|
||||
|
||||
override fun onNewIntent(intent: android.content.Intent) {
|
||||
super.onNewIntent(intent)
|
||||
setIntent(intent)
|
||||
handleAssistantIntent(intent)
|
||||
}
|
||||
|
||||
private fun handleAssistantIntent(intent: android.content.Intent?) {
|
||||
val request = parseAssistantLaunchIntent(intent) ?: return
|
||||
viewModel.handleAssistantLaunch(request)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,385 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
import android.app.Application
|
||||
import androidx.lifecycle.AndroidViewModel
|
||||
import androidx.lifecycle.LifecycleOwner
|
||||
import androidx.lifecycle.viewModelScope
|
||||
import ai.openclaw.app.chat.ChatMessage
|
||||
import ai.openclaw.app.chat.ChatPendingToolCall
|
||||
import ai.openclaw.app.chat.ChatSessionEntry
|
||||
import ai.openclaw.app.chat.OutgoingAttachment
|
||||
import ai.openclaw.app.gateway.GatewayEndpoint
|
||||
import ai.openclaw.app.node.CameraCaptureManager
|
||||
import ai.openclaw.app.node.CanvasController
|
||||
import ai.openclaw.app.node.SmsManager
|
||||
import ai.openclaw.app.voice.VoiceConversationEntry
|
||||
import kotlinx.coroutines.ExperimentalCoroutinesApi
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.SharingStarted
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.flatMapLatest
|
||||
import kotlinx.coroutines.flow.flowOf
|
||||
import kotlinx.coroutines.flow.stateIn
|
||||
|
||||
@OptIn(ExperimentalCoroutinesApi::class)
|
||||
class MainViewModel(app: Application) : AndroidViewModel(app) {
|
||||
private val nodeApp = app as NodeApp
|
||||
private val prefs = nodeApp.prefs
|
||||
private val runtimeRef = MutableStateFlow<NodeRuntime?>(null)
|
||||
private var foreground = true
|
||||
private val _requestedHomeDestination = MutableStateFlow<HomeDestination?>(null)
|
||||
val requestedHomeDestination: StateFlow<HomeDestination?> = _requestedHomeDestination
|
||||
private val _chatDraft = MutableStateFlow<String?>(null)
|
||||
val chatDraft: StateFlow<String?> = _chatDraft
|
||||
private val _pendingAssistantAutoSend = MutableStateFlow<String?>(null)
|
||||
val pendingAssistantAutoSend: StateFlow<String?> = _pendingAssistantAutoSend
|
||||
|
||||
private fun ensureRuntime(): NodeRuntime {
|
||||
runtimeRef.value?.let { return it }
|
||||
val runtime = nodeApp.ensureRuntime()
|
||||
runtime.setForeground(foreground)
|
||||
runtimeRef.value = runtime
|
||||
return runtime
|
||||
}
|
||||
|
||||
private fun <T> runtimeState(
|
||||
initial: T,
|
||||
selector: (NodeRuntime) -> StateFlow<T>,
|
||||
): StateFlow<T> =
|
||||
runtimeRef
|
||||
.flatMapLatest { runtime -> runtime?.let(selector) ?: flowOf(initial) }
|
||||
.stateIn(viewModelScope, SharingStarted.Eagerly, initial)
|
||||
|
||||
val runtimeInitialized: StateFlow<Boolean> =
|
||||
runtimeRef
|
||||
.flatMapLatest { runtime -> flowOf(runtime != null) }
|
||||
.stateIn(viewModelScope, SharingStarted.Eagerly, false)
|
||||
|
||||
val canvasCurrentUrl: StateFlow<String?> = runtimeState(initial = null) { it.canvas.currentUrl }
|
||||
val canvasA2uiHydrated: StateFlow<Boolean> = runtimeState(initial = false) { it.canvasA2uiHydrated }
|
||||
val canvasRehydratePending: StateFlow<Boolean> = runtimeState(initial = false) { it.canvasRehydratePending }
|
||||
val canvasRehydrateErrorText: StateFlow<String?> = runtimeState(initial = null) { it.canvasRehydrateErrorText }
|
||||
|
||||
val gateways: StateFlow<List<GatewayEndpoint>> = runtimeState(initial = emptyList()) { it.gateways }
|
||||
val discoveryStatusText: StateFlow<String> = runtimeState(initial = "Searching…") { it.discoveryStatusText }
|
||||
val notificationForwardingEnabled: StateFlow<Boolean> = prefs.notificationForwardingEnabled
|
||||
val notificationForwardingMode: StateFlow<NotificationPackageFilterMode> =
|
||||
prefs.notificationForwardingMode
|
||||
val notificationForwardingPackages: StateFlow<Set<String>> = prefs.notificationForwardingPackages
|
||||
val notificationForwardingQuietHoursEnabled: StateFlow<Boolean> =
|
||||
prefs.notificationForwardingQuietHoursEnabled
|
||||
val notificationForwardingQuietStart: StateFlow<String> = prefs.notificationForwardingQuietStart
|
||||
val notificationForwardingQuietEnd: StateFlow<String> = prefs.notificationForwardingQuietEnd
|
||||
val notificationForwardingMaxEventsPerMinute: StateFlow<Int> =
|
||||
prefs.notificationForwardingMaxEventsPerMinute
|
||||
val notificationForwardingSessionKey: StateFlow<String?> = prefs.notificationForwardingSessionKey
|
||||
|
||||
val isConnected: StateFlow<Boolean> = runtimeState(initial = false) { it.isConnected }
|
||||
val isNodeConnected: StateFlow<Boolean> = runtimeState(initial = false) { it.nodeConnected }
|
||||
val statusText: StateFlow<String> = runtimeState(initial = "Offline") { it.statusText }
|
||||
val serverName: StateFlow<String?> = runtimeState(initial = null) { it.serverName }
|
||||
val remoteAddress: StateFlow<String?> = runtimeState(initial = null) { it.remoteAddress }
|
||||
val pendingGatewayTrust: StateFlow<NodeRuntime.GatewayTrustPrompt?> = runtimeState(initial = null) { it.pendingGatewayTrust }
|
||||
val seamColorArgb: StateFlow<Long> = runtimeState(initial = 0xFF0EA5E9) { it.seamColorArgb }
|
||||
val mainSessionKey: StateFlow<String> = runtimeState(initial = "main") { it.mainSessionKey }
|
||||
|
||||
val cameraHud: StateFlow<CameraHudState?> = runtimeState(initial = null) { it.cameraHud }
|
||||
val cameraFlashToken: StateFlow<Long> = runtimeState(initial = 0L) { it.cameraFlashToken }
|
||||
|
||||
val instanceId: StateFlow<String> = prefs.instanceId
|
||||
val displayName: StateFlow<String> = prefs.displayName
|
||||
val cameraEnabled: StateFlow<Boolean> = prefs.cameraEnabled
|
||||
val locationMode: StateFlow<LocationMode> = prefs.locationMode
|
||||
val locationPreciseEnabled: StateFlow<Boolean> = prefs.locationPreciseEnabled
|
||||
val preventSleep: StateFlow<Boolean> = prefs.preventSleep
|
||||
val manualEnabled: StateFlow<Boolean> = prefs.manualEnabled
|
||||
val manualHost: StateFlow<String> = prefs.manualHost
|
||||
val manualPort: StateFlow<Int> = prefs.manualPort
|
||||
val manualTls: StateFlow<Boolean> = prefs.manualTls
|
||||
val gatewayToken: StateFlow<String> = prefs.gatewayToken
|
||||
val gatewayBootstrapToken: StateFlow<String> = prefs.gatewayBootstrapToken
|
||||
val onboardingCompleted: StateFlow<Boolean> = prefs.onboardingCompleted
|
||||
val canvasDebugStatusEnabled: StateFlow<Boolean> = prefs.canvasDebugStatusEnabled
|
||||
val speakerEnabled: StateFlow<Boolean> = prefs.speakerEnabled
|
||||
val micEnabled: StateFlow<Boolean> = prefs.talkEnabled
|
||||
|
||||
val micCooldown: StateFlow<Boolean> = runtimeState(initial = false) { it.micCooldown }
|
||||
val micStatusText: StateFlow<String> = runtimeState(initial = "Mic off") { it.micStatusText }
|
||||
val micLiveTranscript: StateFlow<String?> = runtimeState(initial = null) { it.micLiveTranscript }
|
||||
val micIsListening: StateFlow<Boolean> = runtimeState(initial = false) { it.micIsListening }
|
||||
val micQueuedMessages: StateFlow<List<String>> = runtimeState(initial = emptyList()) { it.micQueuedMessages }
|
||||
val micConversation: StateFlow<List<VoiceConversationEntry>> = runtimeState(initial = emptyList()) { it.micConversation }
|
||||
val micInputLevel: StateFlow<Float> = runtimeState(initial = 0f) { it.micInputLevel }
|
||||
val micIsSending: StateFlow<Boolean> = runtimeState(initial = false) { it.micIsSending }
|
||||
|
||||
val chatSessionKey: StateFlow<String> = runtimeState(initial = "main") { it.chatSessionKey }
|
||||
val chatSessionId: StateFlow<String?> = runtimeState(initial = null) { it.chatSessionId }
|
||||
val chatMessages: StateFlow<List<ChatMessage>> = runtimeState(initial = emptyList()) { it.chatMessages }
|
||||
val chatError: StateFlow<String?> = runtimeState(initial = null) { it.chatError }
|
||||
val chatHealthOk: StateFlow<Boolean> = runtimeState(initial = false) { it.chatHealthOk }
|
||||
val chatThinkingLevel: StateFlow<String> = runtimeState(initial = "off") { it.chatThinkingLevel }
|
||||
val chatStreamingAssistantText: StateFlow<String?> = runtimeState(initial = null) { it.chatStreamingAssistantText }
|
||||
val chatPendingToolCalls: StateFlow<List<ChatPendingToolCall>> = runtimeState(initial = emptyList()) { it.chatPendingToolCalls }
|
||||
val chatSessions: StateFlow<List<ChatSessionEntry>> = runtimeState(initial = emptyList()) { it.chatSessions }
|
||||
val pendingRunCount: StateFlow<Int> = runtimeState(initial = 0) { it.pendingRunCount }
|
||||
|
||||
init {
|
||||
if (prefs.onboardingCompleted.value) {
|
||||
ensureRuntime()
|
||||
}
|
||||
}
|
||||
|
||||
val canvas: CanvasController
|
||||
get() = ensureRuntime().canvas
|
||||
|
||||
val camera: CameraCaptureManager
|
||||
get() = ensureRuntime().camera
|
||||
|
||||
val sms: SmsManager
|
||||
get() = ensureRuntime().sms
|
||||
|
||||
fun attachRuntimeUi(owner: LifecycleOwner, permissionRequester: PermissionRequester) {
|
||||
val runtime = runtimeRef.value ?: return
|
||||
runtime.camera.attachLifecycleOwner(owner)
|
||||
runtime.camera.attachPermissionRequester(permissionRequester)
|
||||
runtime.sms.attachPermissionRequester(permissionRequester)
|
||||
}
|
||||
|
||||
fun setForeground(value: Boolean) {
|
||||
foreground = value
|
||||
val runtime =
|
||||
if (value && prefs.onboardingCompleted.value) {
|
||||
ensureRuntime()
|
||||
} else {
|
||||
runtimeRef.value
|
||||
}
|
||||
runtime?.setForeground(value)
|
||||
}
|
||||
|
||||
fun setDisplayName(value: String) {
|
||||
prefs.setDisplayName(value)
|
||||
}
|
||||
|
||||
fun setCameraEnabled(value: Boolean) {
|
||||
prefs.setCameraEnabled(value)
|
||||
}
|
||||
|
||||
fun setLocationMode(mode: LocationMode) {
|
||||
prefs.setLocationMode(mode)
|
||||
}
|
||||
|
||||
fun setLocationPreciseEnabled(value: Boolean) {
|
||||
prefs.setLocationPreciseEnabled(value)
|
||||
}
|
||||
|
||||
fun setPreventSleep(value: Boolean) {
|
||||
prefs.setPreventSleep(value)
|
||||
}
|
||||
|
||||
fun setManualEnabled(value: Boolean) {
|
||||
prefs.setManualEnabled(value)
|
||||
}
|
||||
|
||||
fun setManualHost(value: String) {
|
||||
prefs.setManualHost(value)
|
||||
}
|
||||
|
||||
fun setManualPort(value: Int) {
|
||||
prefs.setManualPort(value)
|
||||
}
|
||||
|
||||
fun setManualTls(value: Boolean) {
|
||||
prefs.setManualTls(value)
|
||||
}
|
||||
|
||||
fun setGatewayToken(value: String) {
|
||||
prefs.setGatewayToken(value)
|
||||
}
|
||||
|
||||
fun setGatewayBootstrapToken(value: String) {
|
||||
prefs.setGatewayBootstrapToken(value)
|
||||
}
|
||||
|
||||
fun setGatewayPassword(value: String) {
|
||||
prefs.setGatewayPassword(value)
|
||||
}
|
||||
|
||||
fun resetGatewaySetupAuth() {
|
||||
ensureRuntime().resetGatewaySetupAuth()
|
||||
}
|
||||
|
||||
fun setOnboardingCompleted(value: Boolean) {
|
||||
if (value) {
|
||||
ensureRuntime()
|
||||
}
|
||||
prefs.setOnboardingCompleted(value)
|
||||
}
|
||||
|
||||
fun setCanvasDebugStatusEnabled(value: Boolean) {
|
||||
prefs.setCanvasDebugStatusEnabled(value)
|
||||
}
|
||||
|
||||
fun setNotificationForwardingEnabled(value: Boolean) {
|
||||
ensureRuntime().setNotificationForwardingEnabled(value)
|
||||
}
|
||||
|
||||
fun setNotificationForwardingMode(mode: NotificationPackageFilterMode) {
|
||||
ensureRuntime().setNotificationForwardingMode(mode)
|
||||
}
|
||||
|
||||
fun setNotificationForwardingPackagesCsv(csv: String) {
|
||||
val packages =
|
||||
csv
|
||||
.split(',')
|
||||
.map { it.trim() }
|
||||
.filter { it.isNotEmpty() }
|
||||
ensureRuntime().setNotificationForwardingPackages(packages)
|
||||
}
|
||||
|
||||
fun setNotificationForwardingQuietHours(
|
||||
enabled: Boolean,
|
||||
start: String,
|
||||
end: String,
|
||||
): Boolean {
|
||||
return ensureRuntime().setNotificationForwardingQuietHours(enabled = enabled, start = start, end = end)
|
||||
}
|
||||
|
||||
fun setNotificationForwardingMaxEventsPerMinute(value: Int) {
|
||||
ensureRuntime().setNotificationForwardingMaxEventsPerMinute(value)
|
||||
}
|
||||
|
||||
fun setNotificationForwardingSessionKey(value: String?) {
|
||||
ensureRuntime().setNotificationForwardingSessionKey(value)
|
||||
}
|
||||
|
||||
fun setVoiceScreenActive(active: Boolean) {
|
||||
ensureRuntime().setVoiceScreenActive(active)
|
||||
}
|
||||
|
||||
fun handleAssistantLaunch(request: AssistantLaunchRequest) {
|
||||
_requestedHomeDestination.value = HomeDestination.Chat
|
||||
if (request.autoSend) {
|
||||
_pendingAssistantAutoSend.value = request.prompt
|
||||
_chatDraft.value = null
|
||||
return
|
||||
}
|
||||
_pendingAssistantAutoSend.value = null
|
||||
_chatDraft.value = request.prompt
|
||||
}
|
||||
|
||||
fun clearRequestedHomeDestination() {
|
||||
_requestedHomeDestination.value = null
|
||||
}
|
||||
|
||||
fun clearChatDraft() {
|
||||
_chatDraft.value = null
|
||||
}
|
||||
|
||||
fun clearPendingAssistantAutoSend() {
|
||||
_pendingAssistantAutoSend.value = null
|
||||
}
|
||||
|
||||
fun setMicEnabled(enabled: Boolean) {
|
||||
ensureRuntime().setMicEnabled(enabled)
|
||||
}
|
||||
|
||||
fun setSpeakerEnabled(enabled: Boolean) {
|
||||
ensureRuntime().setSpeakerEnabled(enabled)
|
||||
}
|
||||
|
||||
fun refreshGatewayConnection() {
|
||||
ensureRuntime().refreshGatewayConnection()
|
||||
}
|
||||
|
||||
fun connect(endpoint: GatewayEndpoint) {
|
||||
ensureRuntime().connect(endpoint)
|
||||
}
|
||||
|
||||
fun connect(
|
||||
endpoint: GatewayEndpoint,
|
||||
token: String?,
|
||||
bootstrapToken: String?,
|
||||
password: String?,
|
||||
) {
|
||||
ensureRuntime().connect(
|
||||
endpoint,
|
||||
NodeRuntime.GatewayConnectAuth(
|
||||
token = token,
|
||||
bootstrapToken = bootstrapToken,
|
||||
password = password,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
fun connectManual() {
|
||||
ensureRuntime().connectManual()
|
||||
}
|
||||
|
||||
fun disconnect() {
|
||||
runtimeRef.value?.disconnect()
|
||||
}
|
||||
|
||||
fun acceptGatewayTrustPrompt() {
|
||||
runtimeRef.value?.acceptGatewayTrustPrompt()
|
||||
}
|
||||
|
||||
fun declineGatewayTrustPrompt() {
|
||||
runtimeRef.value?.declineGatewayTrustPrompt()
|
||||
}
|
||||
|
||||
fun handleCanvasA2UIActionFromWebView(payloadJson: String) {
|
||||
ensureRuntime().handleCanvasA2UIActionFromWebView(payloadJson)
|
||||
}
|
||||
|
||||
fun isTrustedCanvasActionUrl(rawUrl: String?): Boolean {
|
||||
return ensureRuntime().isTrustedCanvasActionUrl(rawUrl)
|
||||
}
|
||||
|
||||
fun requestCanvasRehydrate(source: String = "screen_tab") {
|
||||
ensureRuntime().requestCanvasRehydrate(source = source, force = true)
|
||||
}
|
||||
|
||||
fun refreshHomeCanvasOverviewIfConnected() {
|
||||
ensureRuntime().refreshHomeCanvasOverviewIfConnected()
|
||||
}
|
||||
|
||||
fun loadChat(sessionKey: String) {
|
||||
ensureRuntime().loadChat(sessionKey)
|
||||
}
|
||||
|
||||
fun refreshChat() {
|
||||
ensureRuntime().refreshChat()
|
||||
}
|
||||
|
||||
fun refreshChatSessions(limit: Int? = null) {
|
||||
ensureRuntime().refreshChatSessions(limit = limit)
|
||||
}
|
||||
|
||||
fun setChatThinkingLevel(level: String) {
|
||||
ensureRuntime().setChatThinkingLevel(level)
|
||||
}
|
||||
|
||||
fun switchChatSession(sessionKey: String) {
|
||||
ensureRuntime().switchChatSession(sessionKey)
|
||||
}
|
||||
|
||||
fun abortChat() {
|
||||
ensureRuntime().abortChat()
|
||||
}
|
||||
|
||||
fun sendChat(message: String, thinking: String, attachments: List<OutgoingAttachment>) {
|
||||
ensureRuntime().sendChat(message = message, thinking = thinking, attachments = attachments)
|
||||
}
|
||||
|
||||
suspend fun sendChatAwaitAcceptance(
|
||||
message: String,
|
||||
thinking: String,
|
||||
attachments: List<OutgoingAttachment>,
|
||||
): Boolean {
|
||||
return ensureRuntime().sendChatAwaitAcceptance(
|
||||
message = message,
|
||||
thinking = thinking,
|
||||
attachments = attachments,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
import android.app.Application
|
||||
import android.os.StrictMode
|
||||
|
||||
class NodeApp : Application() {
|
||||
val prefs: SecurePrefs by lazy { SecurePrefs(this) }
|
||||
|
||||
@Volatile private var runtimeInstance: NodeRuntime? = null
|
||||
|
||||
fun ensureRuntime(): NodeRuntime {
|
||||
runtimeInstance?.let { return it }
|
||||
return synchronized(this) {
|
||||
runtimeInstance ?: NodeRuntime(this, prefs).also { runtimeInstance = it }
|
||||
}
|
||||
}
|
||||
|
||||
fun peekRuntime(): NodeRuntime? = runtimeInstance
|
||||
|
||||
override fun onCreate() {
|
||||
super.onCreate()
|
||||
if (BuildConfig.DEBUG) {
|
||||
StrictMode.setThreadPolicy(
|
||||
StrictMode.ThreadPolicy.Builder()
|
||||
.detectAll()
|
||||
.penaltyLog()
|
||||
.build(),
|
||||
)
|
||||
StrictMode.setVmPolicy(
|
||||
StrictMode.VmPolicy.Builder()
|
||||
.detectAll()
|
||||
.penaltyLog()
|
||||
.build(),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,162 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
import android.app.Notification
|
||||
import android.app.NotificationChannel
|
||||
import android.app.NotificationManager
|
||||
import android.app.Service
|
||||
import android.app.PendingIntent
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.content.pm.ServiceInfo
|
||||
import androidx.core.app.NotificationCompat
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.SupervisorJob
|
||||
import kotlinx.coroutines.cancel
|
||||
import kotlinx.coroutines.flow.combine
|
||||
import kotlinx.coroutines.launch
|
||||
|
||||
class NodeForegroundService : Service() {
|
||||
private val scope: CoroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Main)
|
||||
private var notificationJob: Job? = null
|
||||
private var didStartForeground = false
|
||||
|
||||
override fun onCreate() {
|
||||
super.onCreate()
|
||||
ensureChannel()
|
||||
val initial = buildNotification(title = "OpenClaw Node", text = "Starting…")
|
||||
startForegroundWithTypes(notification = initial)
|
||||
|
||||
val runtime = (application as NodeApp).peekRuntime()
|
||||
if (runtime == null) {
|
||||
stopSelf()
|
||||
return
|
||||
}
|
||||
notificationJob =
|
||||
scope.launch {
|
||||
combine(
|
||||
runtime.statusText,
|
||||
runtime.serverName,
|
||||
runtime.isConnected,
|
||||
runtime.micEnabled,
|
||||
runtime.micIsListening,
|
||||
) { status, server, connected, micEnabled, micListening ->
|
||||
Quint(status, server, connected, micEnabled, micListening)
|
||||
}.collect { (status, server, connected, micEnabled, micListening) ->
|
||||
val title = if (connected) "OpenClaw Node · Connected" else "OpenClaw Node"
|
||||
val micSuffix =
|
||||
if (micEnabled) {
|
||||
if (micListening) " · Mic: Listening" else " · Mic: Pending"
|
||||
} else {
|
||||
""
|
||||
}
|
||||
val text = (server?.let { "$status · $it" } ?: status) + micSuffix
|
||||
|
||||
startForegroundWithTypes(
|
||||
notification = buildNotification(title = title, text = text),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
|
||||
when (intent?.action) {
|
||||
ACTION_STOP -> {
|
||||
(application as NodeApp).peekRuntime()?.disconnect()
|
||||
stopSelf()
|
||||
return START_NOT_STICKY
|
||||
}
|
||||
}
|
||||
// Keep running; connection is managed by NodeRuntime (auto-reconnect + manual).
|
||||
return START_STICKY
|
||||
}
|
||||
|
||||
override fun onDestroy() {
|
||||
notificationJob?.cancel()
|
||||
scope.cancel()
|
||||
super.onDestroy()
|
||||
}
|
||||
|
||||
override fun onBind(intent: Intent?) = null
|
||||
|
||||
private fun ensureChannel() {
|
||||
val mgr = getSystemService(NotificationManager::class.java)
|
||||
val channel =
|
||||
NotificationChannel(
|
||||
CHANNEL_ID,
|
||||
"Connection",
|
||||
NotificationManager.IMPORTANCE_LOW,
|
||||
).apply {
|
||||
description = "OpenClaw node connection status"
|
||||
setShowBadge(false)
|
||||
}
|
||||
mgr.createNotificationChannel(channel)
|
||||
}
|
||||
|
||||
private fun buildNotification(title: String, text: String): Notification {
|
||||
val launchIntent = Intent(this, MainActivity::class.java).apply {
|
||||
flags = Intent.FLAG_ACTIVITY_SINGLE_TOP or Intent.FLAG_ACTIVITY_CLEAR_TOP
|
||||
}
|
||||
val launchPending =
|
||||
PendingIntent.getActivity(
|
||||
this,
|
||||
1,
|
||||
launchIntent,
|
||||
PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE,
|
||||
)
|
||||
|
||||
val stopIntent = Intent(this, NodeForegroundService::class.java).setAction(ACTION_STOP)
|
||||
val stopPending =
|
||||
PendingIntent.getService(
|
||||
this,
|
||||
2,
|
||||
stopIntent,
|
||||
PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE,
|
||||
)
|
||||
|
||||
return NotificationCompat.Builder(this, CHANNEL_ID)
|
||||
.setSmallIcon(R.mipmap.ic_launcher)
|
||||
.setContentTitle(title)
|
||||
.setContentText(text)
|
||||
.setContentIntent(launchPending)
|
||||
.setOngoing(true)
|
||||
.setOnlyAlertOnce(true)
|
||||
.setForegroundServiceBehavior(NotificationCompat.FOREGROUND_SERVICE_IMMEDIATE)
|
||||
.addAction(0, "Disconnect", stopPending)
|
||||
.build()
|
||||
}
|
||||
|
||||
private fun updateNotification(notification: Notification) {
|
||||
val mgr = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager
|
||||
mgr.notify(NOTIFICATION_ID, notification)
|
||||
}
|
||||
|
||||
private fun startForegroundWithTypes(notification: Notification) {
|
||||
if (didStartForeground) {
|
||||
updateNotification(notification)
|
||||
return
|
||||
}
|
||||
startForeground(NOTIFICATION_ID, notification, ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC)
|
||||
didStartForeground = true
|
||||
}
|
||||
|
||||
companion object {
|
||||
private const val CHANNEL_ID = "connection"
|
||||
private const val NOTIFICATION_ID = 1
|
||||
|
||||
private const val ACTION_STOP = "ai.openclaw.app.action.STOP"
|
||||
|
||||
fun start(context: Context) {
|
||||
val intent = Intent(context, NodeForegroundService::class.java)
|
||||
context.startForegroundService(intent)
|
||||
}
|
||||
|
||||
fun stop(context: Context) {
|
||||
val intent = Intent(context, NodeForegroundService::class.java).setAction(ACTION_STOP)
|
||||
context.startService(intent)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private data class Quint<A, B, C, D, E>(val first: A, val second: B, val third: C, val fourth: D, val fifth: E)
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,102 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
import java.time.Instant
|
||||
import java.time.ZoneId
|
||||
|
||||
enum class NotificationPackageFilterMode(val rawValue: String) {
|
||||
Allowlist("allowlist"),
|
||||
Blocklist("blocklist"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
fun fromRawValue(raw: String?): NotificationPackageFilterMode {
|
||||
return entries.firstOrNull { it.rawValue == raw?.trim()?.lowercase() } ?: Blocklist
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
internal data class NotificationForwardingPolicy(
|
||||
val enabled: Boolean,
|
||||
val mode: NotificationPackageFilterMode,
|
||||
val packages: Set<String>,
|
||||
val quietHoursEnabled: Boolean,
|
||||
val quietStart: String,
|
||||
val quietEnd: String,
|
||||
val maxEventsPerMinute: Int,
|
||||
val sessionKey: String?,
|
||||
)
|
||||
|
||||
internal fun NotificationForwardingPolicy.allowsPackage(packageName: String): Boolean {
|
||||
val normalized = packageName.trim()
|
||||
if (normalized.isEmpty()) {
|
||||
return false
|
||||
}
|
||||
return when (mode) {
|
||||
NotificationPackageFilterMode.Allowlist -> packages.contains(normalized)
|
||||
NotificationPackageFilterMode.Blocklist -> !packages.contains(normalized)
|
||||
}
|
||||
}
|
||||
|
||||
internal fun NotificationForwardingPolicy.isWithinQuietHours(
|
||||
nowEpochMs: Long,
|
||||
zoneId: ZoneId = ZoneId.systemDefault(),
|
||||
): Boolean {
|
||||
if (!quietHoursEnabled) {
|
||||
return false
|
||||
}
|
||||
val startMinutes = parseLocalHourMinute(quietStart) ?: return false
|
||||
val endMinutes = parseLocalHourMinute(quietEnd) ?: return false
|
||||
if (startMinutes == endMinutes) {
|
||||
return true
|
||||
}
|
||||
val now =
|
||||
Instant.ofEpochMilli(nowEpochMs)
|
||||
.atZone(zoneId)
|
||||
.toLocalTime()
|
||||
val nowMinutes = now.hour * 60 + now.minute
|
||||
return if (startMinutes < endMinutes) {
|
||||
nowMinutes in startMinutes until endMinutes
|
||||
} else {
|
||||
nowMinutes >= startMinutes || nowMinutes < endMinutes
|
||||
}
|
||||
}
|
||||
|
||||
private val localHourMinuteRegex = Regex("""^([01]\d|2[0-3]):([0-5]\d)$""")
|
||||
|
||||
internal fun normalizeLocalHourMinute(raw: String): String? {
|
||||
val trimmed = raw.trim()
|
||||
val match = localHourMinuteRegex.matchEntire(trimmed) ?: return null
|
||||
return "${match.groupValues[1]}:${match.groupValues[2]}"
|
||||
}
|
||||
|
||||
internal fun parseLocalHourMinute(raw: String): Int? {
|
||||
val normalized = normalizeLocalHourMinute(raw) ?: return null
|
||||
val parts = normalized.split(':')
|
||||
val hour = parts[0].toInt()
|
||||
val minute = parts[1].toInt()
|
||||
return hour * 60 + minute
|
||||
}
|
||||
|
||||
internal class NotificationBurstLimiter {
|
||||
private val lock = Any()
|
||||
private var windowStartMs: Long = -1L
|
||||
private var eventsInWindow: Int = 0
|
||||
|
||||
fun allow(nowEpochMs: Long, maxEventsPerMinute: Int): Boolean {
|
||||
if (maxEventsPerMinute <= 0) {
|
||||
return false
|
||||
}
|
||||
val currentWindow = nowEpochMs - (nowEpochMs % 60_000L)
|
||||
synchronized(lock) {
|
||||
if (currentWindow != windowStartMs) {
|
||||
windowStartMs = currentWindow
|
||||
eventsInWindow = 0
|
||||
}
|
||||
if (eventsInWindow >= maxEventsPerMinute) {
|
||||
return false
|
||||
}
|
||||
eventsInWindow += 1
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,200 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
import android.content.pm.PackageManager
|
||||
import android.content.Intent
|
||||
import android.Manifest
|
||||
import android.net.Uri
|
||||
import android.os.Handler
|
||||
import android.os.Looper
|
||||
import android.provider.Settings
|
||||
import androidx.appcompat.app.AlertDialog
|
||||
import androidx.activity.ComponentActivity
|
||||
import androidx.activity.result.ActivityResultLauncher
|
||||
import androidx.activity.result.contract.ActivityResultContracts
|
||||
import androidx.core.content.ContextCompat
|
||||
import androidx.core.app.ActivityCompat
|
||||
import androidx.lifecycle.Lifecycle
|
||||
import androidx.lifecycle.LifecycleEventObserver
|
||||
import kotlinx.coroutines.CompletableDeferred
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.sync.Mutex
|
||||
import kotlinx.coroutines.sync.withLock
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
import java.util.concurrent.atomic.AtomicBoolean
|
||||
import kotlin.coroutines.resume
|
||||
|
||||
class PermissionRequester(private val activity: ComponentActivity) {
|
||||
private val mutex = Mutex()
|
||||
private var pending: CompletableDeferred<Map<String, Boolean>>? = null
|
||||
private val mainHandler = Handler(Looper.getMainLooper())
|
||||
|
||||
private val launcher: ActivityResultLauncher<Array<String>> =
|
||||
activity.registerForActivityResult(ActivityResultContracts.RequestMultiplePermissions()) { result ->
|
||||
val p = pending
|
||||
pending = null
|
||||
p?.complete(result)
|
||||
}
|
||||
|
||||
suspend fun requestIfMissing(
|
||||
permissions: List<String>,
|
||||
timeoutMs: Long = 20_000,
|
||||
): Map<String, Boolean> =
|
||||
mutex.withLock {
|
||||
val missing =
|
||||
permissions.filter { perm ->
|
||||
ContextCompat.checkSelfPermission(activity, perm) != PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
if (missing.isEmpty()) {
|
||||
return permissions.associateWith { true }
|
||||
}
|
||||
|
||||
val needsRationale =
|
||||
missing.any { ActivityCompat.shouldShowRequestPermissionRationale(activity, it) }
|
||||
if (needsRationale) {
|
||||
val proceed = showRationaleDialog(missing)
|
||||
if (!proceed) {
|
||||
return permissions.associateWith { perm ->
|
||||
ContextCompat.checkSelfPermission(activity, perm) == PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
val deferred = CompletableDeferred<Map<String, Boolean>>()
|
||||
pending = deferred
|
||||
withContext(Dispatchers.Main) {
|
||||
launcher.launch(missing.toTypedArray())
|
||||
}
|
||||
|
||||
val result =
|
||||
withContext(Dispatchers.Default) {
|
||||
kotlinx.coroutines.withTimeout(timeoutMs) { deferred.await() }
|
||||
}
|
||||
|
||||
// Merge: if something was already granted, treat it as granted even if launcher omitted it.
|
||||
val merged =
|
||||
permissions.associateWith { perm ->
|
||||
val nowGranted =
|
||||
ContextCompat.checkSelfPermission(activity, perm) == PackageManager.PERMISSION_GRANTED
|
||||
result[perm] == true || nowGranted
|
||||
}
|
||||
|
||||
val denied =
|
||||
merged.filterValues { !it }.keys.filter {
|
||||
!ActivityCompat.shouldShowRequestPermissionRationale(activity, it)
|
||||
}
|
||||
if (denied.isNotEmpty()) {
|
||||
showSettingsDialog(denied)
|
||||
}
|
||||
|
||||
return merged
|
||||
}
|
||||
|
||||
private suspend fun showRationaleDialog(permissions: List<String>): Boolean =
|
||||
withContext(Dispatchers.Main) {
|
||||
if (activity.isFinishing || activity.isDestroyed) {
|
||||
return@withContext false
|
||||
}
|
||||
suspendCancellableCoroutine { cont ->
|
||||
val lifecycle = activity.lifecycle
|
||||
var dialog: AlertDialog? = null
|
||||
var observer: LifecycleEventObserver? = null
|
||||
val finished = AtomicBoolean(false)
|
||||
val removeObserver = {
|
||||
observer?.let(lifecycle::removeObserver)
|
||||
observer = null
|
||||
}
|
||||
fun finish(result: Boolean?) {
|
||||
if (!finished.compareAndSet(false, true)) return
|
||||
removeObserver()
|
||||
dialog?.dismiss()
|
||||
if (result != null) {
|
||||
cont.resume(result)
|
||||
}
|
||||
}
|
||||
val actualObserver =
|
||||
LifecycleEventObserver { _, event ->
|
||||
if (event != Lifecycle.Event.ON_DESTROY) return@LifecycleEventObserver
|
||||
finish(false)
|
||||
}
|
||||
observer = actualObserver
|
||||
lifecycle.addObserver(actualObserver)
|
||||
cont.invokeOnCancellation {
|
||||
mainHandler.post {
|
||||
finish(null)
|
||||
}
|
||||
}
|
||||
dialog =
|
||||
AlertDialog.Builder(activity)
|
||||
.setTitle("Permission required")
|
||||
.setMessage(buildRationaleMessage(permissions))
|
||||
.setPositiveButton("Continue") { _, _ -> finish(true) }
|
||||
.setNegativeButton("Not now") { _, _ -> finish(false) }
|
||||
.setOnCancelListener { finish(false) }
|
||||
.show()
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun showSettingsDialog(permissions: List<String>) =
|
||||
withContext(Dispatchers.Main) {
|
||||
if (activity.isFinishing || activity.isDestroyed) return@withContext
|
||||
val lifecycle = activity.lifecycle
|
||||
var dialog: AlertDialog? = null
|
||||
var observer: LifecycleEventObserver? = null
|
||||
val removeObserver = {
|
||||
observer?.let(lifecycle::removeObserver)
|
||||
observer = null
|
||||
}
|
||||
val actualObserver =
|
||||
LifecycleEventObserver { _, event ->
|
||||
if (event != Lifecycle.Event.ON_DESTROY) return@LifecycleEventObserver
|
||||
removeObserver()
|
||||
dialog?.dismiss()
|
||||
}
|
||||
observer = actualObserver
|
||||
lifecycle.addObserver(actualObserver)
|
||||
dialog =
|
||||
AlertDialog.Builder(activity)
|
||||
.setTitle("Enable permission in Settings")
|
||||
.setMessage(buildSettingsMessage(permissions))
|
||||
.setPositiveButton("Open Settings") { _, _ ->
|
||||
if (activity.isFinishing || activity.isDestroyed) return@setPositiveButton
|
||||
val intent =
|
||||
Intent(
|
||||
Settings.ACTION_APPLICATION_DETAILS_SETTINGS,
|
||||
Uri.fromParts("package", activity.packageName, null),
|
||||
)
|
||||
activity.startActivity(intent)
|
||||
}
|
||||
.setNegativeButton("Cancel", null)
|
||||
.setOnDismissListener { removeObserver() }
|
||||
.show()
|
||||
}
|
||||
|
||||
private fun buildRationaleMessage(permissions: List<String>): String {
|
||||
val labels = permissions.map { permissionLabel(it) }
|
||||
return "OpenClaw needs ${labels.joinToString(", ")} permissions to continue."
|
||||
}
|
||||
|
||||
private fun buildSettingsMessage(permissions: List<String>): String {
|
||||
val labels = permissions.map { permissionLabel(it) }
|
||||
return "Please enable ${labels.joinToString(", ")} in Android Settings to continue."
|
||||
}
|
||||
|
||||
private fun permissionLabel(permission: String): String =
|
||||
when (permission) {
|
||||
Manifest.permission.CAMERA -> "Camera"
|
||||
Manifest.permission.RECORD_AUDIO -> "Microphone"
|
||||
Manifest.permission.SEND_SMS -> "Send SMS"
|
||||
Manifest.permission.READ_SMS -> "Read SMS"
|
||||
Manifest.permission.READ_CONTACTS -> "Read Contacts"
|
||||
Manifest.permission.WRITE_CONTACTS -> "Write Contacts"
|
||||
Manifest.permission.READ_CALENDAR -> "Read Calendar"
|
||||
Manifest.permission.WRITE_CALENDAR -> "Write Calendar"
|
||||
Manifest.permission.READ_CALL_LOG -> "Read Call Log"
|
||||
Manifest.permission.ACTIVITY_RECOGNITION -> "Motion Activity"
|
||||
Manifest.permission.READ_MEDIA_IMAGES -> "Photos"
|
||||
Manifest.permission.READ_EXTERNAL_STORAGE -> "Photos"
|
||||
else -> permission
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,553 @@
|
|||
@file:Suppress("DEPRECATION")
|
||||
|
||||
package ai.openclaw.app
|
||||
|
||||
import android.content.Context
|
||||
import android.content.SharedPreferences
|
||||
import androidx.core.content.edit
|
||||
import androidx.security.crypto.EncryptedSharedPreferences
|
||||
import androidx.security.crypto.MasterKey
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
import kotlinx.serialization.json.JsonNull
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import java.util.UUID
|
||||
|
||||
class SecurePrefs(
|
||||
context: Context,
|
||||
private val securePrefsOverride: SharedPreferences? = null,
|
||||
) {
|
||||
companion object {
|
||||
val defaultWakeWords: List<String> = listOf("openclaw", "claude")
|
||||
private const val displayNameKey = "node.displayName"
|
||||
private const val locationModeKey = "location.enabledMode"
|
||||
private const val voiceWakeModeKey = "voiceWake.mode"
|
||||
private const val plainPrefsName = "openclaw.node"
|
||||
private const val securePrefsName = "openclaw.node.secure"
|
||||
private const val notificationsForwardingEnabledKey = "notifications.forwarding.enabled"
|
||||
private const val defaultNotificationForwardingEnabled = false
|
||||
private const val notificationsForwardingModeKey = "notifications.forwarding.mode"
|
||||
private const val notificationsForwardingPackagesKey = "notifications.forwarding.packages"
|
||||
private const val notificationsForwardingQuietHoursEnabledKey =
|
||||
"notifications.forwarding.quietHoursEnabled"
|
||||
private const val notificationsForwardingQuietStartKey = "notifications.forwarding.quietStart"
|
||||
private const val notificationsForwardingQuietEndKey = "notifications.forwarding.quietEnd"
|
||||
private const val notificationsForwardingMaxEventsPerMinuteKey =
|
||||
"notifications.forwarding.maxEventsPerMinute"
|
||||
private const val notificationsForwardingSessionKeyKey = "notifications.forwarding.sessionKey"
|
||||
}
|
||||
|
||||
private val appContext = context.applicationContext
|
||||
private val json = Json { ignoreUnknownKeys = true }
|
||||
private val plainPrefs: SharedPreferences =
|
||||
appContext.getSharedPreferences(plainPrefsName, Context.MODE_PRIVATE)
|
||||
|
||||
private val masterKey by lazy {
|
||||
MasterKey.Builder(appContext)
|
||||
.setKeyScheme(MasterKey.KeyScheme.AES256_GCM)
|
||||
.build()
|
||||
}
|
||||
private val securePrefs: SharedPreferences by lazy { securePrefsOverride ?: createSecurePrefs(appContext, securePrefsName) }
|
||||
|
||||
private val _instanceId = MutableStateFlow(loadOrCreateInstanceId())
|
||||
val instanceId: StateFlow<String> = _instanceId
|
||||
|
||||
private val _displayName =
|
||||
MutableStateFlow(loadOrMigrateDisplayName(context = context))
|
||||
val displayName: StateFlow<String> = _displayName
|
||||
|
||||
private val _cameraEnabled = MutableStateFlow(plainPrefs.getBoolean("camera.enabled", true))
|
||||
val cameraEnabled: StateFlow<Boolean> = _cameraEnabled
|
||||
|
||||
private val _locationMode = MutableStateFlow(loadLocationMode())
|
||||
val locationMode: StateFlow<LocationMode> = _locationMode
|
||||
|
||||
private val _locationPreciseEnabled =
|
||||
MutableStateFlow(plainPrefs.getBoolean("location.preciseEnabled", true))
|
||||
val locationPreciseEnabled: StateFlow<Boolean> = _locationPreciseEnabled
|
||||
|
||||
private val _preventSleep = MutableStateFlow(plainPrefs.getBoolean("screen.preventSleep", true))
|
||||
val preventSleep: StateFlow<Boolean> = _preventSleep
|
||||
|
||||
private val _manualEnabled =
|
||||
MutableStateFlow(plainPrefs.getBoolean("gateway.manual.enabled", false))
|
||||
val manualEnabled: StateFlow<Boolean> = _manualEnabled
|
||||
|
||||
private val _manualHost =
|
||||
MutableStateFlow(plainPrefs.getString("gateway.manual.host", "") ?: "")
|
||||
val manualHost: StateFlow<String> = _manualHost
|
||||
|
||||
private val _manualPort =
|
||||
MutableStateFlow(plainPrefs.getInt("gateway.manual.port", 18789))
|
||||
val manualPort: StateFlow<Int> = _manualPort
|
||||
|
||||
private val _manualTls =
|
||||
MutableStateFlow(plainPrefs.getBoolean("gateway.manual.tls", true))
|
||||
val manualTls: StateFlow<Boolean> = _manualTls
|
||||
|
||||
private val _gatewayToken = MutableStateFlow("")
|
||||
val gatewayToken: StateFlow<String> = _gatewayToken
|
||||
|
||||
private val _gatewayBootstrapToken = MutableStateFlow("")
|
||||
val gatewayBootstrapToken: StateFlow<String> = _gatewayBootstrapToken
|
||||
|
||||
private val _onboardingCompleted =
|
||||
MutableStateFlow(plainPrefs.getBoolean("onboarding.completed", false))
|
||||
val onboardingCompleted: StateFlow<Boolean> = _onboardingCompleted
|
||||
|
||||
private val _lastDiscoveredStableId =
|
||||
MutableStateFlow(
|
||||
plainPrefs.getString("gateway.lastDiscoveredStableID", "") ?: "",
|
||||
)
|
||||
val lastDiscoveredStableId: StateFlow<String> = _lastDiscoveredStableId
|
||||
|
||||
private val _canvasDebugStatusEnabled =
|
||||
MutableStateFlow(plainPrefs.getBoolean("canvas.debugStatusEnabled", false))
|
||||
val canvasDebugStatusEnabled: StateFlow<Boolean> = _canvasDebugStatusEnabled
|
||||
|
||||
private val _notificationForwardingEnabled =
|
||||
MutableStateFlow(plainPrefs.getBoolean(notificationsForwardingEnabledKey, defaultNotificationForwardingEnabled))
|
||||
val notificationForwardingEnabled: StateFlow<Boolean> = _notificationForwardingEnabled
|
||||
|
||||
private val _notificationForwardingMode =
|
||||
MutableStateFlow(
|
||||
NotificationPackageFilterMode.fromRawValue(
|
||||
plainPrefs.getString(notificationsForwardingModeKey, null),
|
||||
),
|
||||
)
|
||||
val notificationForwardingMode: StateFlow<NotificationPackageFilterMode> = _notificationForwardingMode
|
||||
|
||||
private val _notificationForwardingPackages = MutableStateFlow(loadNotificationForwardingPackages())
|
||||
val notificationForwardingPackages: StateFlow<Set<String>> = _notificationForwardingPackages
|
||||
|
||||
private val storedQuietStart =
|
||||
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietStartKey, "22:00").orEmpty())
|
||||
?: "22:00"
|
||||
private val storedQuietEnd =
|
||||
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietEndKey, "07:00").orEmpty())
|
||||
?: "07:00"
|
||||
private val storedQuietHoursEnabled =
|
||||
plainPrefs.getBoolean(notificationsForwardingQuietHoursEnabledKey, false) &&
|
||||
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietStartKey, "22:00").orEmpty()) != null &&
|
||||
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietEndKey, "07:00").orEmpty()) != null
|
||||
|
||||
private val _notificationForwardingQuietHoursEnabled =
|
||||
MutableStateFlow(storedQuietHoursEnabled)
|
||||
val notificationForwardingQuietHoursEnabled: StateFlow<Boolean> = _notificationForwardingQuietHoursEnabled
|
||||
|
||||
private val _notificationForwardingQuietStart = MutableStateFlow(storedQuietStart)
|
||||
val notificationForwardingQuietStart: StateFlow<String> = _notificationForwardingQuietStart
|
||||
|
||||
private val _notificationForwardingQuietEnd = MutableStateFlow(storedQuietEnd)
|
||||
val notificationForwardingQuietEnd: StateFlow<String> = _notificationForwardingQuietEnd
|
||||
|
||||
private val _notificationForwardingMaxEventsPerMinute =
|
||||
MutableStateFlow(plainPrefs.getInt(notificationsForwardingMaxEventsPerMinuteKey, 20).coerceAtLeast(1))
|
||||
val notificationForwardingMaxEventsPerMinute: StateFlow<Int> = _notificationForwardingMaxEventsPerMinute
|
||||
|
||||
private val _notificationForwardingSessionKey =
|
||||
MutableStateFlow(
|
||||
plainPrefs
|
||||
.getString(notificationsForwardingSessionKeyKey, "")
|
||||
?.trim()
|
||||
?.takeIf { it.isNotEmpty() },
|
||||
)
|
||||
val notificationForwardingSessionKey: StateFlow<String?> = _notificationForwardingSessionKey
|
||||
|
||||
private val _wakeWords = MutableStateFlow(loadWakeWords())
|
||||
val wakeWords: StateFlow<List<String>> = _wakeWords
|
||||
|
||||
private val _voiceWakeMode = MutableStateFlow(loadVoiceWakeMode())
|
||||
val voiceWakeMode: StateFlow<VoiceWakeMode> = _voiceWakeMode
|
||||
|
||||
private val _talkEnabled = MutableStateFlow(plainPrefs.getBoolean("talk.enabled", false))
|
||||
val talkEnabled: StateFlow<Boolean> = _talkEnabled
|
||||
|
||||
private val _speakerEnabled = MutableStateFlow(plainPrefs.getBoolean("voice.speakerEnabled", true))
|
||||
val speakerEnabled: StateFlow<Boolean> = _speakerEnabled
|
||||
|
||||
fun setLastDiscoveredStableId(value: String) {
|
||||
val trimmed = value.trim()
|
||||
plainPrefs.edit { putString("gateway.lastDiscoveredStableID", trimmed) }
|
||||
_lastDiscoveredStableId.value = trimmed
|
||||
}
|
||||
|
||||
fun setDisplayName(value: String) {
|
||||
val trimmed = value.trim()
|
||||
plainPrefs.edit { putString(displayNameKey, trimmed) }
|
||||
_displayName.value = trimmed
|
||||
}
|
||||
|
||||
fun setCameraEnabled(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("camera.enabled", value) }
|
||||
_cameraEnabled.value = value
|
||||
}
|
||||
|
||||
fun setLocationMode(mode: LocationMode) {
|
||||
plainPrefs.edit { putString(locationModeKey, mode.rawValue) }
|
||||
_locationMode.value = mode
|
||||
}
|
||||
|
||||
fun setLocationPreciseEnabled(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("location.preciseEnabled", value) }
|
||||
_locationPreciseEnabled.value = value
|
||||
}
|
||||
|
||||
fun setPreventSleep(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("screen.preventSleep", value) }
|
||||
_preventSleep.value = value
|
||||
}
|
||||
|
||||
fun setManualEnabled(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("gateway.manual.enabled", value) }
|
||||
_manualEnabled.value = value
|
||||
}
|
||||
|
||||
fun setManualHost(value: String) {
|
||||
val trimmed = value.trim()
|
||||
plainPrefs.edit { putString("gateway.manual.host", trimmed) }
|
||||
_manualHost.value = trimmed
|
||||
}
|
||||
|
||||
fun setManualPort(value: Int) {
|
||||
plainPrefs.edit { putInt("gateway.manual.port", value) }
|
||||
_manualPort.value = value
|
||||
}
|
||||
|
||||
fun setManualTls(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("gateway.manual.tls", value) }
|
||||
_manualTls.value = value
|
||||
}
|
||||
|
||||
fun setGatewayToken(value: String) {
|
||||
val trimmed = value.trim()
|
||||
securePrefs.edit { putString("gateway.manual.token", trimmed) }
|
||||
_gatewayToken.value = trimmed
|
||||
}
|
||||
|
||||
fun setGatewayPassword(value: String) {
|
||||
saveGatewayPassword(value)
|
||||
}
|
||||
|
||||
fun setGatewayBootstrapToken(value: String) {
|
||||
saveGatewayBootstrapToken(value)
|
||||
}
|
||||
|
||||
fun setOnboardingCompleted(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("onboarding.completed", value) }
|
||||
_onboardingCompleted.value = value
|
||||
}
|
||||
|
||||
fun setCanvasDebugStatusEnabled(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("canvas.debugStatusEnabled", value) }
|
||||
_canvasDebugStatusEnabled.value = value
|
||||
}
|
||||
|
||||
internal fun getNotificationForwardingPolicy(appPackageName: String): NotificationForwardingPolicy {
|
||||
val modeRaw = plainPrefs.getString(notificationsForwardingModeKey, null)
|
||||
val mode = NotificationPackageFilterMode.fromRawValue(modeRaw)
|
||||
|
||||
val configuredPackages = loadNotificationForwardingPackages()
|
||||
val normalizedAppPackage = appPackageName.trim()
|
||||
val defaultBlockedPackages =
|
||||
if (normalizedAppPackage.isNotEmpty()) setOf(normalizedAppPackage) else emptySet()
|
||||
|
||||
val packages =
|
||||
when (mode) {
|
||||
NotificationPackageFilterMode.Allowlist -> configuredPackages
|
||||
NotificationPackageFilterMode.Blocklist -> configuredPackages + defaultBlockedPackages
|
||||
}
|
||||
|
||||
val maxEvents = plainPrefs.getInt(notificationsForwardingMaxEventsPerMinuteKey, 20)
|
||||
val quietStart =
|
||||
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietStartKey, "22:00").orEmpty())
|
||||
?: "22:00"
|
||||
val quietEnd =
|
||||
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietEndKey, "07:00").orEmpty())
|
||||
?: "07:00"
|
||||
val sessionKey =
|
||||
plainPrefs
|
||||
.getString(notificationsForwardingSessionKeyKey, "")
|
||||
?.trim()
|
||||
?.takeIf { it.isNotEmpty() }
|
||||
|
||||
val quietHoursEnabled =
|
||||
plainPrefs.getBoolean(notificationsForwardingQuietHoursEnabledKey, false) &&
|
||||
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietStartKey, "22:00").orEmpty()) != null &&
|
||||
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietEndKey, "07:00").orEmpty()) != null
|
||||
|
||||
return NotificationForwardingPolicy(
|
||||
enabled = plainPrefs.getBoolean(notificationsForwardingEnabledKey, defaultNotificationForwardingEnabled),
|
||||
mode = mode,
|
||||
packages = packages,
|
||||
quietHoursEnabled = quietHoursEnabled,
|
||||
quietStart = quietStart,
|
||||
quietEnd = quietEnd,
|
||||
maxEventsPerMinute = maxEvents.coerceAtLeast(1),
|
||||
sessionKey = sessionKey,
|
||||
)
|
||||
}
|
||||
|
||||
internal fun setNotificationForwardingEnabled(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean(notificationsForwardingEnabledKey, value) }
|
||||
_notificationForwardingEnabled.value = value
|
||||
}
|
||||
|
||||
internal fun setNotificationForwardingMode(mode: NotificationPackageFilterMode) {
|
||||
plainPrefs.edit { putString(notificationsForwardingModeKey, mode.rawValue) }
|
||||
_notificationForwardingMode.value = mode
|
||||
}
|
||||
|
||||
internal fun setNotificationForwardingPackages(packages: List<String>) {
|
||||
val sanitized =
|
||||
packages
|
||||
.asSequence()
|
||||
.map { it.trim() }
|
||||
.filter { it.isNotEmpty() }
|
||||
.toSet()
|
||||
.toList()
|
||||
.sorted()
|
||||
val encoded = JsonArray(sanitized.map { JsonPrimitive(it) }).toString()
|
||||
plainPrefs.edit { putString(notificationsForwardingPackagesKey, encoded) }
|
||||
_notificationForwardingPackages.value = sanitized.toSet()
|
||||
}
|
||||
|
||||
internal fun setNotificationForwardingQuietHours(
|
||||
enabled: Boolean,
|
||||
start: String,
|
||||
end: String,
|
||||
): Boolean {
|
||||
if (!enabled) {
|
||||
plainPrefs.edit { putBoolean(notificationsForwardingQuietHoursEnabledKey, false) }
|
||||
_notificationForwardingQuietHoursEnabled.value = false
|
||||
return true
|
||||
}
|
||||
val normalizedStart = normalizeLocalHourMinute(start) ?: return false
|
||||
val normalizedEnd = normalizeLocalHourMinute(end) ?: return false
|
||||
plainPrefs.edit {
|
||||
putBoolean(notificationsForwardingQuietHoursEnabledKey, enabled)
|
||||
putString(notificationsForwardingQuietStartKey, normalizedStart)
|
||||
putString(notificationsForwardingQuietEndKey, normalizedEnd)
|
||||
}
|
||||
_notificationForwardingQuietHoursEnabled.value = enabled
|
||||
_notificationForwardingQuietStart.value = normalizedStart
|
||||
_notificationForwardingQuietEnd.value = normalizedEnd
|
||||
return true
|
||||
}
|
||||
|
||||
internal fun setNotificationForwardingMaxEventsPerMinute(value: Int) {
|
||||
val normalized = value.coerceAtLeast(1)
|
||||
plainPrefs.edit {
|
||||
putInt(notificationsForwardingMaxEventsPerMinuteKey, normalized)
|
||||
}
|
||||
_notificationForwardingMaxEventsPerMinute.value = normalized
|
||||
}
|
||||
|
||||
internal fun setNotificationForwardingSessionKey(value: String?) {
|
||||
val normalized = value?.trim()?.takeIf { it.isNotEmpty() }
|
||||
plainPrefs.edit {
|
||||
putString(notificationsForwardingSessionKeyKey, normalized.orEmpty())
|
||||
}
|
||||
_notificationForwardingSessionKey.value = normalized
|
||||
}
|
||||
|
||||
fun loadGatewayToken(): String? {
|
||||
val manual =
|
||||
_gatewayToken.value.trim().ifEmpty {
|
||||
val stored = securePrefs.getString("gateway.manual.token", null)?.trim().orEmpty()
|
||||
if (stored.isNotEmpty()) _gatewayToken.value = stored
|
||||
stored
|
||||
}
|
||||
if (manual.isNotEmpty()) return manual
|
||||
val key = "gateway.token.${_instanceId.value}"
|
||||
val stored = securePrefs.getString(key, null)?.trim()
|
||||
return stored?.takeIf { it.isNotEmpty() }
|
||||
}
|
||||
|
||||
fun saveGatewayToken(token: String) {
|
||||
val key = "gateway.token.${_instanceId.value}"
|
||||
securePrefs.edit { putString(key, token.trim()) }
|
||||
}
|
||||
|
||||
fun loadGatewayBootstrapToken(): String? {
|
||||
val key = "gateway.bootstrapToken.${_instanceId.value}"
|
||||
val stored =
|
||||
_gatewayBootstrapToken.value.trim().ifEmpty {
|
||||
val persisted = securePrefs.getString(key, null)?.trim().orEmpty()
|
||||
if (persisted.isNotEmpty()) {
|
||||
_gatewayBootstrapToken.value = persisted
|
||||
}
|
||||
persisted
|
||||
}
|
||||
return stored.takeIf { it.isNotEmpty() }
|
||||
}
|
||||
|
||||
fun saveGatewayBootstrapToken(token: String) {
|
||||
val key = "gateway.bootstrapToken.${_instanceId.value}"
|
||||
val trimmed = token.trim()
|
||||
securePrefs.edit { putString(key, trimmed) }
|
||||
_gatewayBootstrapToken.value = trimmed
|
||||
}
|
||||
|
||||
fun loadGatewayPassword(): String? {
|
||||
val key = "gateway.password.${_instanceId.value}"
|
||||
val stored = securePrefs.getString(key, null)?.trim()
|
||||
return stored?.takeIf { it.isNotEmpty() }
|
||||
}
|
||||
|
||||
fun saveGatewayPassword(password: String) {
|
||||
val key = "gateway.password.${_instanceId.value}"
|
||||
securePrefs.edit { putString(key, password.trim()) }
|
||||
}
|
||||
|
||||
fun clearGatewaySetupAuth() {
|
||||
val instanceId = _instanceId.value
|
||||
securePrefs.edit {
|
||||
remove("gateway.manual.token")
|
||||
remove("gateway.token.$instanceId")
|
||||
remove("gateway.bootstrapToken.$instanceId")
|
||||
remove("gateway.password.$instanceId")
|
||||
}
|
||||
_gatewayToken.value = ""
|
||||
_gatewayBootstrapToken.value = ""
|
||||
}
|
||||
|
||||
fun loadGatewayTlsFingerprint(stableId: String): String? {
|
||||
val key = "gateway.tls.$stableId"
|
||||
return plainPrefs.getString(key, null)?.trim()?.takeIf { it.isNotEmpty() }
|
||||
}
|
||||
|
||||
fun saveGatewayTlsFingerprint(stableId: String, fingerprint: String) {
|
||||
val key = "gateway.tls.$stableId"
|
||||
plainPrefs.edit { putString(key, fingerprint.trim()) }
|
||||
}
|
||||
|
||||
fun getString(key: String): String? {
|
||||
return securePrefs.getString(key, null)
|
||||
}
|
||||
|
||||
fun putString(key: String, value: String) {
|
||||
securePrefs.edit { putString(key, value) }
|
||||
}
|
||||
|
||||
fun remove(key: String) {
|
||||
securePrefs.edit { remove(key) }
|
||||
}
|
||||
|
||||
private fun createSecurePrefs(context: Context, name: String): SharedPreferences {
|
||||
return EncryptedSharedPreferences.create(
|
||||
context,
|
||||
name,
|
||||
masterKey,
|
||||
EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV,
|
||||
EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM,
|
||||
)
|
||||
}
|
||||
|
||||
private fun loadOrCreateInstanceId(): String {
|
||||
val existing = plainPrefs.getString("node.instanceId", null)?.trim()
|
||||
if (!existing.isNullOrBlank()) return existing
|
||||
val fresh = UUID.randomUUID().toString()
|
||||
plainPrefs.edit { putString("node.instanceId", fresh) }
|
||||
return fresh
|
||||
}
|
||||
|
||||
private fun loadOrMigrateDisplayName(context: Context): String {
|
||||
val existing = plainPrefs.getString(displayNameKey, null)?.trim().orEmpty()
|
||||
if (existing.isNotEmpty() && existing != "Android Node") return existing
|
||||
|
||||
val candidate = DeviceNames.bestDefaultNodeName(context).trim()
|
||||
val resolved = candidate.ifEmpty { "Android Node" }
|
||||
|
||||
plainPrefs.edit { putString(displayNameKey, resolved) }
|
||||
return resolved
|
||||
}
|
||||
|
||||
fun setWakeWords(words: List<String>) {
|
||||
val sanitized = WakeWords.sanitize(words, defaultWakeWords)
|
||||
val encoded =
|
||||
JsonArray(sanitized.map { JsonPrimitive(it) }).toString()
|
||||
plainPrefs.edit { putString("voiceWake.triggerWords", encoded) }
|
||||
_wakeWords.value = sanitized
|
||||
}
|
||||
|
||||
fun setVoiceWakeMode(mode: VoiceWakeMode) {
|
||||
plainPrefs.edit { putString(voiceWakeModeKey, mode.rawValue) }
|
||||
_voiceWakeMode.value = mode
|
||||
}
|
||||
|
||||
fun setTalkEnabled(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("talk.enabled", value) }
|
||||
_talkEnabled.value = value
|
||||
}
|
||||
|
||||
fun setSpeakerEnabled(value: Boolean) {
|
||||
plainPrefs.edit { putBoolean("voice.speakerEnabled", value) }
|
||||
_speakerEnabled.value = value
|
||||
}
|
||||
|
||||
private fun loadNotificationForwardingPackages(): Set<String> {
|
||||
val raw = plainPrefs.getString(notificationsForwardingPackagesKey, null)?.trim()
|
||||
if (raw.isNullOrEmpty()) {
|
||||
return emptySet()
|
||||
}
|
||||
return try {
|
||||
val element = json.parseToJsonElement(raw)
|
||||
val array = element as? JsonArray ?: return emptySet()
|
||||
array
|
||||
.mapNotNull { item ->
|
||||
when (item) {
|
||||
is JsonNull -> null
|
||||
is JsonPrimitive -> item.content.trim().takeIf { it.isNotEmpty() }
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
.toSet()
|
||||
} catch (_: Throwable) {
|
||||
emptySet()
|
||||
}
|
||||
}
|
||||
|
||||
private fun loadVoiceWakeMode(): VoiceWakeMode {
|
||||
val raw = plainPrefs.getString(voiceWakeModeKey, null)
|
||||
val resolved = VoiceWakeMode.fromRawValue(raw)
|
||||
|
||||
// Default ON (foreground) when unset.
|
||||
if (raw.isNullOrBlank()) {
|
||||
plainPrefs.edit { putString(voiceWakeModeKey, resolved.rawValue) }
|
||||
}
|
||||
|
||||
return resolved
|
||||
}
|
||||
|
||||
private fun loadLocationMode(): LocationMode {
|
||||
val raw = plainPrefs.getString(locationModeKey, "off")
|
||||
val resolved = LocationMode.fromRawValue(raw)
|
||||
if (raw?.trim()?.lowercase() == "always") {
|
||||
plainPrefs.edit { putString(locationModeKey, resolved.rawValue) }
|
||||
}
|
||||
return resolved
|
||||
}
|
||||
|
||||
private fun loadWakeWords(): List<String> {
|
||||
val raw = plainPrefs.getString("voiceWake.triggerWords", null)?.trim()
|
||||
if (raw.isNullOrEmpty()) return defaultWakeWords
|
||||
return try {
|
||||
val element = json.parseToJsonElement(raw)
|
||||
val array = element as? JsonArray ?: return defaultWakeWords
|
||||
val decoded =
|
||||
array.mapNotNull { item ->
|
||||
when (item) {
|
||||
is JsonNull -> null
|
||||
is JsonPrimitive -> item.content.trim().takeIf { it.isNotEmpty() }
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
WakeWords.sanitize(decoded, defaultWakeWords)
|
||||
} catch (_: Throwable) {
|
||||
defaultWakeWords
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
internal fun normalizeMainKey(raw: String?): String {
|
||||
val trimmed = raw?.trim()
|
||||
return if (!trimmed.isNullOrEmpty()) trimmed else "main"
|
||||
}
|
||||
|
||||
internal fun isCanonicalMainSessionKey(raw: String?): Boolean {
|
||||
val trimmed = raw?.trim().orEmpty()
|
||||
if (trimmed.isEmpty()) return false
|
||||
if (trimmed == "global") return true
|
||||
return trimmed.startsWith("agent:")
|
||||
}
|
||||
|
||||
internal fun resolveAgentIdFromMainSessionKey(raw: String?): String? {
|
||||
val trimmed = raw?.trim().orEmpty()
|
||||
if (!trimmed.startsWith("agent:")) return null
|
||||
return trimmed.removePrefix("agent:").substringBefore(':').trim().ifEmpty { null }
|
||||
}
|
||||
|
||||
internal fun buildNodeMainSessionKey(deviceId: String, agentId: String?): String {
|
||||
val resolvedAgentId = agentId?.trim().orEmpty().ifEmpty { "main" }
|
||||
return "agent:$resolvedAgentId:node-${deviceId.take(12)}"
|
||||
}
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
enum class VoiceWakeMode(val rawValue: String) {
|
||||
Off("off"),
|
||||
Foreground("foreground"),
|
||||
Always("always"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
fun fromRawValue(raw: String?): VoiceWakeMode {
|
||||
return entries.firstOrNull { it.rawValue == raw?.trim()?.lowercase() } ?: Foreground
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
package ai.openclaw.app
|
||||
|
||||
object WakeWords {
|
||||
const val maxWords: Int = 32
|
||||
const val maxWordLength: Int = 64
|
||||
|
||||
fun parseCommaSeparated(input: String): List<String> {
|
||||
return input.split(",").map { it.trim() }.filter { it.isNotEmpty() }
|
||||
}
|
||||
|
||||
fun parseIfChanged(input: String, current: List<String>): List<String>? {
|
||||
val parsed = parseCommaSeparated(input)
|
||||
return if (parsed == current) null else parsed
|
||||
}
|
||||
|
||||
fun sanitize(words: List<String>, defaults: List<String>): List<String> {
|
||||
val cleaned =
|
||||
words.map { it.trim() }.filter { it.isNotEmpty() }.take(maxWords).map { it.take(maxWordLength) }
|
||||
return cleaned.ifEmpty { defaults }
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,641 @@
|
|||
package ai.openclaw.app.chat
|
||||
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import java.util.UUID
|
||||
import java.util.concurrent.ConcurrentHashMap
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
import kotlinx.serialization.json.JsonElement
|
||||
import kotlinx.serialization.json.JsonNull
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
|
||||
class ChatController(
|
||||
private val scope: CoroutineScope,
|
||||
private val session: GatewaySession,
|
||||
private val json: Json,
|
||||
private val supportsChatSubscribe: Boolean,
|
||||
) {
|
||||
private var appliedMainSessionKey = "main"
|
||||
private val _sessionKey = MutableStateFlow("main")
|
||||
val sessionKey: StateFlow<String> = _sessionKey.asStateFlow()
|
||||
|
||||
private val _sessionId = MutableStateFlow<String?>(null)
|
||||
val sessionId: StateFlow<String?> = _sessionId.asStateFlow()
|
||||
|
||||
private val _messages = MutableStateFlow<List<ChatMessage>>(emptyList())
|
||||
val messages: StateFlow<List<ChatMessage>> = _messages.asStateFlow()
|
||||
|
||||
private val _errorText = MutableStateFlow<String?>(null)
|
||||
val errorText: StateFlow<String?> = _errorText.asStateFlow()
|
||||
|
||||
private val _healthOk = MutableStateFlow(false)
|
||||
val healthOk: StateFlow<Boolean> = _healthOk.asStateFlow()
|
||||
|
||||
private val _thinkingLevel = MutableStateFlow("off")
|
||||
val thinkingLevel: StateFlow<String> = _thinkingLevel.asStateFlow()
|
||||
|
||||
private val _pendingRunCount = MutableStateFlow(0)
|
||||
val pendingRunCount: StateFlow<Int> = _pendingRunCount.asStateFlow()
|
||||
|
||||
private val _streamingAssistantText = MutableStateFlow<String?>(null)
|
||||
val streamingAssistantText: StateFlow<String?> = _streamingAssistantText.asStateFlow()
|
||||
|
||||
private val pendingToolCallsById = ConcurrentHashMap<String, ChatPendingToolCall>()
|
||||
private val _pendingToolCalls = MutableStateFlow<List<ChatPendingToolCall>>(emptyList())
|
||||
val pendingToolCalls: StateFlow<List<ChatPendingToolCall>> = _pendingToolCalls.asStateFlow()
|
||||
|
||||
private val _sessions = MutableStateFlow<List<ChatSessionEntry>>(emptyList())
|
||||
val sessions: StateFlow<List<ChatSessionEntry>> = _sessions.asStateFlow()
|
||||
|
||||
private val pendingRuns = mutableSetOf<String>()
|
||||
private val pendingRunTimeoutJobs = ConcurrentHashMap<String, Job>()
|
||||
private val pendingRunTimeoutMs = 120_000L
|
||||
|
||||
private var lastHealthPollAtMs: Long? = null
|
||||
|
||||
fun onDisconnected(message: String) {
|
||||
_healthOk.value = false
|
||||
// Not an error; keep connection status in the UI pill.
|
||||
_errorText.value = null
|
||||
clearPendingRuns()
|
||||
pendingToolCallsById.clear()
|
||||
publishPendingToolCalls()
|
||||
_streamingAssistantText.value = null
|
||||
_sessionId.value = null
|
||||
}
|
||||
|
||||
fun load(sessionKey: String) {
|
||||
val key = normalizeRequestedSessionKey(sessionKey)
|
||||
_sessionKey.value = key
|
||||
scope.launch { bootstrap(forceHealth = true, refreshSessions = true) }
|
||||
}
|
||||
|
||||
fun applyMainSessionKey(mainSessionKey: String) {
|
||||
val trimmed = mainSessionKey.trim()
|
||||
if (trimmed.isEmpty()) return
|
||||
val nextState =
|
||||
applyMainSessionKey(
|
||||
currentSessionKey = normalizeRequestedSessionKey(_sessionKey.value),
|
||||
appliedMainSessionKey = appliedMainSessionKey,
|
||||
nextMainSessionKey = trimmed,
|
||||
)
|
||||
appliedMainSessionKey = nextState.appliedMainSessionKey
|
||||
if (_sessionKey.value == nextState.currentSessionKey) return
|
||||
_sessionKey.value = nextState.currentSessionKey
|
||||
scope.launch { bootstrap(forceHealth = true, refreshSessions = true) }
|
||||
}
|
||||
|
||||
fun refresh() {
|
||||
scope.launch { bootstrap(forceHealth = true, refreshSessions = true) }
|
||||
}
|
||||
|
||||
fun refreshSessions(limit: Int? = null) {
|
||||
scope.launch { fetchSessions(limit = limit) }
|
||||
}
|
||||
|
||||
fun setThinkingLevel(thinkingLevel: String) {
|
||||
val normalized = normalizeThinking(thinkingLevel)
|
||||
if (normalized == _thinkingLevel.value) return
|
||||
_thinkingLevel.value = normalized
|
||||
}
|
||||
|
||||
fun switchSession(sessionKey: String) {
|
||||
val key = normalizeRequestedSessionKey(sessionKey)
|
||||
if (key.isEmpty()) return
|
||||
if (key == _sessionKey.value) return
|
||||
_sessionKey.value = key
|
||||
// Keep the thread switch path lean: history + health are needed immediately,
|
||||
// but the session list is usually unchanged and can refresh on explicit pull-to-refresh.
|
||||
scope.launch { bootstrap(forceHealth = true, refreshSessions = false) }
|
||||
}
|
||||
|
||||
private fun normalizeRequestedSessionKey(sessionKey: String): String {
|
||||
val key = sessionKey.trim()
|
||||
if (key.isEmpty()) return appliedMainSessionKey
|
||||
if (key == "main" && appliedMainSessionKey != "main") return appliedMainSessionKey
|
||||
return key
|
||||
}
|
||||
|
||||
fun sendMessage(
|
||||
message: String,
|
||||
thinkingLevel: String,
|
||||
attachments: List<OutgoingAttachment>,
|
||||
) {
|
||||
scope.launch {
|
||||
sendMessageAwaitAcceptance(
|
||||
message = message,
|
||||
thinkingLevel = thinkingLevel,
|
||||
attachments = attachments,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun sendMessageAwaitAcceptance(
|
||||
message: String,
|
||||
thinkingLevel: String,
|
||||
attachments: List<OutgoingAttachment>,
|
||||
): Boolean {
|
||||
val trimmed = message.trim()
|
||||
if (trimmed.isEmpty() && attachments.isEmpty()) return false
|
||||
if (!_healthOk.value) {
|
||||
_errorText.value = "Gateway health not OK; cannot send"
|
||||
return false
|
||||
}
|
||||
|
||||
val runId = UUID.randomUUID().toString()
|
||||
val text = if (trimmed.isEmpty() && attachments.isNotEmpty()) "See attached." else trimmed
|
||||
val sessionKey = _sessionKey.value
|
||||
val thinking = normalizeThinking(thinkingLevel)
|
||||
|
||||
// Optimistic user message.
|
||||
val userContent =
|
||||
buildList {
|
||||
add(ChatMessageContent(type = "text", text = text))
|
||||
for (att in attachments) {
|
||||
add(
|
||||
ChatMessageContent(
|
||||
type = att.type,
|
||||
mimeType = att.mimeType,
|
||||
fileName = att.fileName,
|
||||
base64 = att.base64,
|
||||
),
|
||||
)
|
||||
}
|
||||
}
|
||||
_messages.value =
|
||||
_messages.value +
|
||||
ChatMessage(
|
||||
id = UUID.randomUUID().toString(),
|
||||
role = "user",
|
||||
content = userContent,
|
||||
timestampMs = System.currentTimeMillis(),
|
||||
)
|
||||
|
||||
armPendingRunTimeout(runId)
|
||||
synchronized(pendingRuns) {
|
||||
pendingRuns.add(runId)
|
||||
_pendingRunCount.value = pendingRuns.size
|
||||
}
|
||||
|
||||
_errorText.value = null
|
||||
_streamingAssistantText.value = null
|
||||
pendingToolCallsById.clear()
|
||||
publishPendingToolCalls()
|
||||
|
||||
return try {
|
||||
val params =
|
||||
buildJsonObject {
|
||||
put("sessionKey", JsonPrimitive(sessionKey))
|
||||
put("message", JsonPrimitive(text))
|
||||
put("thinking", JsonPrimitive(thinking))
|
||||
put("timeoutMs", JsonPrimitive(30_000))
|
||||
put("idempotencyKey", JsonPrimitive(runId))
|
||||
if (attachments.isNotEmpty()) {
|
||||
put(
|
||||
"attachments",
|
||||
JsonArray(
|
||||
attachments.map { att ->
|
||||
buildJsonObject {
|
||||
put("type", JsonPrimitive(att.type))
|
||||
put("mimeType", JsonPrimitive(att.mimeType))
|
||||
put("fileName", JsonPrimitive(att.fileName))
|
||||
put("content", JsonPrimitive(att.base64))
|
||||
}
|
||||
},
|
||||
),
|
||||
)
|
||||
}
|
||||
}
|
||||
val res = session.request("chat.send", params.toString())
|
||||
val actualRunId = parseRunId(res) ?: runId
|
||||
if (actualRunId != runId) {
|
||||
clearPendingRun(runId)
|
||||
armPendingRunTimeout(actualRunId)
|
||||
synchronized(pendingRuns) {
|
||||
pendingRuns.add(actualRunId)
|
||||
_pendingRunCount.value = pendingRuns.size
|
||||
}
|
||||
}
|
||||
true
|
||||
} catch (err: Throwable) {
|
||||
clearPendingRun(runId)
|
||||
_errorText.value = err.message
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
fun abort() {
|
||||
val runIds =
|
||||
synchronized(pendingRuns) {
|
||||
pendingRuns.toList()
|
||||
}
|
||||
if (runIds.isEmpty()) return
|
||||
scope.launch {
|
||||
for (runId in runIds) {
|
||||
try {
|
||||
val params =
|
||||
buildJsonObject {
|
||||
put("sessionKey", JsonPrimitive(_sessionKey.value))
|
||||
put("runId", JsonPrimitive(runId))
|
||||
}
|
||||
session.request("chat.abort", params.toString())
|
||||
} catch (_: Throwable) {
|
||||
// best-effort
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fun handleGatewayEvent(event: String, payloadJson: String?) {
|
||||
when (event) {
|
||||
"tick" -> {
|
||||
scope.launch { pollHealthIfNeeded(force = false) }
|
||||
}
|
||||
"health" -> {
|
||||
// If we receive a health snapshot, the gateway is reachable.
|
||||
_healthOk.value = true
|
||||
}
|
||||
"seqGap" -> {
|
||||
_errorText.value = "Event stream interrupted; try refreshing."
|
||||
clearPendingRuns()
|
||||
}
|
||||
"chat" -> {
|
||||
if (payloadJson.isNullOrBlank()) return
|
||||
handleChatEvent(payloadJson)
|
||||
}
|
||||
"agent" -> {
|
||||
if (payloadJson.isNullOrBlank()) return
|
||||
handleAgentEvent(payloadJson)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun bootstrap(forceHealth: Boolean, refreshSessions: Boolean) {
|
||||
_errorText.value = null
|
||||
_healthOk.value = false
|
||||
clearPendingRuns()
|
||||
pendingToolCallsById.clear()
|
||||
publishPendingToolCalls()
|
||||
_streamingAssistantText.value = null
|
||||
_sessionId.value = null
|
||||
|
||||
val key = _sessionKey.value
|
||||
try {
|
||||
if (supportsChatSubscribe) {
|
||||
session.sendNodeEvent("chat.subscribe", """{"sessionKey":"$key"}""")
|
||||
}
|
||||
|
||||
val historyJson = session.request("chat.history", """{"sessionKey":"$key"}""")
|
||||
val history = parseHistory(historyJson, sessionKey = key, previousMessages = _messages.value)
|
||||
_messages.value = history.messages
|
||||
_sessionId.value = history.sessionId
|
||||
history.thinkingLevel?.trim()?.takeIf { it.isNotEmpty() }?.let { _thinkingLevel.value = it }
|
||||
|
||||
pollHealthIfNeeded(force = forceHealth)
|
||||
if (refreshSessions) {
|
||||
fetchSessions(limit = 50)
|
||||
}
|
||||
} catch (err: Throwable) {
|
||||
_errorText.value = err.message
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun fetchSessions(limit: Int?) {
|
||||
try {
|
||||
val params =
|
||||
buildJsonObject {
|
||||
put("includeGlobal", JsonPrimitive(true))
|
||||
put("includeUnknown", JsonPrimitive(false))
|
||||
if (limit != null && limit > 0) put("limit", JsonPrimitive(limit))
|
||||
}
|
||||
val res = session.request("sessions.list", params.toString())
|
||||
_sessions.value = parseSessions(res)
|
||||
} catch (_: Throwable) {
|
||||
// best-effort
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun pollHealthIfNeeded(force: Boolean) {
|
||||
val now = System.currentTimeMillis()
|
||||
val last = lastHealthPollAtMs
|
||||
if (!force && last != null && now - last < 10_000) return
|
||||
lastHealthPollAtMs = now
|
||||
try {
|
||||
session.request("health", null)
|
||||
_healthOk.value = true
|
||||
} catch (_: Throwable) {
|
||||
_healthOk.value = false
|
||||
}
|
||||
}
|
||||
|
||||
private fun handleChatEvent(payloadJson: String) {
|
||||
val payload = json.parseToJsonElement(payloadJson).asObjectOrNull() ?: return
|
||||
val sessionKey = payload["sessionKey"].asStringOrNull()?.trim()
|
||||
if (!sessionKey.isNullOrEmpty() && sessionKey != _sessionKey.value) return
|
||||
|
||||
val runId = payload["runId"].asStringOrNull()
|
||||
val isPending =
|
||||
if (runId != null) synchronized(pendingRuns) { pendingRuns.contains(runId) } else true
|
||||
|
||||
val state = payload["state"].asStringOrNull()
|
||||
when (state) {
|
||||
"delta" -> {
|
||||
// Only show streaming text for runs we initiated
|
||||
if (!isPending) return
|
||||
val text = parseAssistantDeltaText(payload)
|
||||
if (!text.isNullOrEmpty()) {
|
||||
_streamingAssistantText.value = text
|
||||
}
|
||||
}
|
||||
"final", "aborted", "error" -> {
|
||||
if (state == "error") {
|
||||
_errorText.value = payload["errorMessage"].asStringOrNull() ?: "Chat failed"
|
||||
}
|
||||
if (runId != null) clearPendingRun(runId) else clearPendingRuns()
|
||||
pendingToolCallsById.clear()
|
||||
publishPendingToolCalls()
|
||||
_streamingAssistantText.value = null
|
||||
scope.launch {
|
||||
try {
|
||||
val historyJson =
|
||||
session.request("chat.history", """{"sessionKey":"${_sessionKey.value}"}""")
|
||||
val history = parseHistory(historyJson, sessionKey = _sessionKey.value, previousMessages = _messages.value)
|
||||
_messages.value = history.messages
|
||||
_sessionId.value = history.sessionId
|
||||
history.thinkingLevel?.trim()?.takeIf { it.isNotEmpty() }?.let { _thinkingLevel.value = it }
|
||||
} catch (_: Throwable) {
|
||||
// best-effort
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun handleAgentEvent(payloadJson: String) {
|
||||
val payload = json.parseToJsonElement(payloadJson).asObjectOrNull() ?: return
|
||||
val sessionKey = payload["sessionKey"].asStringOrNull()?.trim()
|
||||
if (!sessionKey.isNullOrEmpty() && sessionKey != _sessionKey.value) return
|
||||
|
||||
val stream = payload["stream"].asStringOrNull()
|
||||
val data = payload["data"].asObjectOrNull()
|
||||
|
||||
when (stream) {
|
||||
"assistant" -> {
|
||||
val text = data?.get("text")?.asStringOrNull()
|
||||
if (!text.isNullOrEmpty()) {
|
||||
_streamingAssistantText.value = text
|
||||
}
|
||||
}
|
||||
"tool" -> {
|
||||
val phase = data?.get("phase")?.asStringOrNull()
|
||||
val name = data?.get("name")?.asStringOrNull()
|
||||
val toolCallId = data?.get("toolCallId")?.asStringOrNull()
|
||||
if (phase.isNullOrEmpty() || name.isNullOrEmpty() || toolCallId.isNullOrEmpty()) return
|
||||
|
||||
val ts = payload["ts"].asLongOrNull() ?: System.currentTimeMillis()
|
||||
if (phase == "start") {
|
||||
val args = data?.get("args").asObjectOrNull()
|
||||
pendingToolCallsById[toolCallId] =
|
||||
ChatPendingToolCall(
|
||||
toolCallId = toolCallId,
|
||||
name = name,
|
||||
args = args,
|
||||
startedAtMs = ts,
|
||||
isError = null,
|
||||
)
|
||||
publishPendingToolCalls()
|
||||
} else if (phase == "result") {
|
||||
pendingToolCallsById.remove(toolCallId)
|
||||
publishPendingToolCalls()
|
||||
}
|
||||
}
|
||||
"error" -> {
|
||||
_errorText.value = "Event stream interrupted; try refreshing."
|
||||
clearPendingRuns()
|
||||
pendingToolCallsById.clear()
|
||||
publishPendingToolCalls()
|
||||
_streamingAssistantText.value = null
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseAssistantDeltaText(payload: JsonObject): String? {
|
||||
val message = payload["message"].asObjectOrNull() ?: return null
|
||||
if (message["role"].asStringOrNull() != "assistant") return null
|
||||
val content = message["content"].asArrayOrNull() ?: return null
|
||||
for (item in content) {
|
||||
val obj = item.asObjectOrNull() ?: continue
|
||||
if (obj["type"].asStringOrNull() != "text") continue
|
||||
val text = obj["text"].asStringOrNull()
|
||||
if (!text.isNullOrEmpty()) {
|
||||
return text
|
||||
}
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun publishPendingToolCalls() {
|
||||
_pendingToolCalls.value =
|
||||
pendingToolCallsById.values.sortedBy { it.startedAtMs }
|
||||
}
|
||||
|
||||
private fun armPendingRunTimeout(runId: String) {
|
||||
pendingRunTimeoutJobs[runId]?.cancel()
|
||||
pendingRunTimeoutJobs[runId] =
|
||||
scope.launch {
|
||||
delay(pendingRunTimeoutMs)
|
||||
val stillPending =
|
||||
synchronized(pendingRuns) {
|
||||
pendingRuns.contains(runId)
|
||||
}
|
||||
if (!stillPending) return@launch
|
||||
clearPendingRun(runId)
|
||||
_errorText.value = "Timed out waiting for a reply; try again or refresh."
|
||||
}
|
||||
}
|
||||
|
||||
private fun clearPendingRun(runId: String) {
|
||||
pendingRunTimeoutJobs.remove(runId)?.cancel()
|
||||
synchronized(pendingRuns) {
|
||||
pendingRuns.remove(runId)
|
||||
_pendingRunCount.value = pendingRuns.size
|
||||
}
|
||||
}
|
||||
|
||||
private fun clearPendingRuns() {
|
||||
for ((_, job) in pendingRunTimeoutJobs) {
|
||||
job.cancel()
|
||||
}
|
||||
pendingRunTimeoutJobs.clear()
|
||||
synchronized(pendingRuns) {
|
||||
pendingRuns.clear()
|
||||
_pendingRunCount.value = 0
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseHistory(
|
||||
historyJson: String,
|
||||
sessionKey: String,
|
||||
previousMessages: List<ChatMessage>,
|
||||
): ChatHistory {
|
||||
val root = json.parseToJsonElement(historyJson).asObjectOrNull() ?: return ChatHistory(sessionKey, null, null, emptyList())
|
||||
val sid = root["sessionId"].asStringOrNull()
|
||||
val thinkingLevel = root["thinkingLevel"].asStringOrNull()
|
||||
val array = root["messages"].asArrayOrNull() ?: JsonArray(emptyList())
|
||||
|
||||
val messages =
|
||||
array.mapNotNull { item ->
|
||||
val obj = item.asObjectOrNull() ?: return@mapNotNull null
|
||||
val role = obj["role"].asStringOrNull() ?: return@mapNotNull null
|
||||
val content = obj["content"].asArrayOrNull()?.mapNotNull(::parseMessageContent) ?: emptyList()
|
||||
val ts = obj["timestamp"].asLongOrNull()
|
||||
ChatMessage(
|
||||
id = UUID.randomUUID().toString(),
|
||||
role = role,
|
||||
content = content,
|
||||
timestampMs = ts,
|
||||
)
|
||||
}
|
||||
|
||||
return ChatHistory(
|
||||
sessionKey = sessionKey,
|
||||
sessionId = sid,
|
||||
thinkingLevel = thinkingLevel,
|
||||
messages = reconcileMessageIds(previous = previousMessages, incoming = messages),
|
||||
)
|
||||
}
|
||||
|
||||
private fun parseMessageContent(el: JsonElement): ChatMessageContent? {
|
||||
val obj = el.asObjectOrNull() ?: return null
|
||||
val type = obj["type"].asStringOrNull() ?: "text"
|
||||
return if (type == "text") {
|
||||
ChatMessageContent(type = "text", text = obj["text"].asStringOrNull())
|
||||
} else {
|
||||
ChatMessageContent(
|
||||
type = type,
|
||||
mimeType = obj["mimeType"].asStringOrNull(),
|
||||
fileName = obj["fileName"].asStringOrNull(),
|
||||
base64 = obj["content"].asStringOrNull(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseSessions(jsonString: String): List<ChatSessionEntry> {
|
||||
val root = json.parseToJsonElement(jsonString).asObjectOrNull() ?: return emptyList()
|
||||
val sessions = root["sessions"].asArrayOrNull() ?: return emptyList()
|
||||
return sessions.mapNotNull { item ->
|
||||
val obj = item.asObjectOrNull() ?: return@mapNotNull null
|
||||
val key = obj["key"].asStringOrNull()?.trim().orEmpty()
|
||||
if (key.isEmpty()) return@mapNotNull null
|
||||
val updatedAt = obj["updatedAt"].asLongOrNull()
|
||||
val displayName = obj["displayName"].asStringOrNull()?.trim()
|
||||
ChatSessionEntry(key = key, updatedAtMs = updatedAt, displayName = displayName)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseRunId(resJson: String): String? {
|
||||
return try {
|
||||
json.parseToJsonElement(resJson).asObjectOrNull()?.get("runId").asStringOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun normalizeThinking(raw: String): String {
|
||||
return when (raw.trim().lowercase()) {
|
||||
"low" -> "low"
|
||||
"medium" -> "medium"
|
||||
"high" -> "high"
|
||||
else -> "off"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
internal data class MainSessionState(
|
||||
val currentSessionKey: String,
|
||||
val appliedMainSessionKey: String,
|
||||
)
|
||||
|
||||
internal fun applyMainSessionKey(
|
||||
currentSessionKey: String,
|
||||
appliedMainSessionKey: String,
|
||||
nextMainSessionKey: String,
|
||||
): MainSessionState {
|
||||
if (currentSessionKey == appliedMainSessionKey) {
|
||||
return MainSessionState(
|
||||
currentSessionKey = nextMainSessionKey,
|
||||
appliedMainSessionKey = nextMainSessionKey,
|
||||
)
|
||||
}
|
||||
return MainSessionState(
|
||||
currentSessionKey = currentSessionKey,
|
||||
appliedMainSessionKey = nextMainSessionKey,
|
||||
)
|
||||
}
|
||||
|
||||
internal fun reconcileMessageIds(previous: List<ChatMessage>, incoming: List<ChatMessage>): List<ChatMessage> {
|
||||
if (previous.isEmpty() || incoming.isEmpty()) return incoming
|
||||
|
||||
val idsByKey = LinkedHashMap<String, ArrayDeque<String>>()
|
||||
for (message in previous) {
|
||||
val key = messageIdentityKey(message) ?: continue
|
||||
idsByKey.getOrPut(key) { ArrayDeque() }.addLast(message.id)
|
||||
}
|
||||
|
||||
return incoming.map { message ->
|
||||
val key = messageIdentityKey(message) ?: return@map message
|
||||
val ids = idsByKey[key] ?: return@map message
|
||||
val reusedId = ids.removeFirstOrNull() ?: return@map message
|
||||
if (ids.isEmpty()) {
|
||||
idsByKey.remove(key)
|
||||
}
|
||||
if (reusedId == message.id) return@map message
|
||||
message.copy(id = reusedId)
|
||||
}
|
||||
}
|
||||
|
||||
internal fun messageIdentityKey(message: ChatMessage): String? {
|
||||
val role = message.role.trim().lowercase()
|
||||
if (role.isEmpty()) return null
|
||||
|
||||
val timestamp = message.timestampMs?.toString().orEmpty()
|
||||
val contentFingerprint =
|
||||
message.content.joinToString(separator = "\u001E") { part ->
|
||||
listOf(
|
||||
part.type.trim().lowercase(),
|
||||
part.text?.trim().orEmpty(),
|
||||
part.mimeType?.trim()?.lowercase().orEmpty(),
|
||||
part.fileName?.trim().orEmpty(),
|
||||
part.base64?.hashCode()?.toString().orEmpty(),
|
||||
).joinToString(separator = "\u001F")
|
||||
}
|
||||
|
||||
if (timestamp.isEmpty() && contentFingerprint.isEmpty()) return null
|
||||
return listOf(role, timestamp, contentFingerprint).joinToString(separator = "|")
|
||||
}
|
||||
|
||||
private fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject
|
||||
|
||||
private fun JsonElement?.asArrayOrNull(): JsonArray? = this as? JsonArray
|
||||
|
||||
private fun JsonElement?.asStringOrNull(): String? =
|
||||
when (this) {
|
||||
is JsonNull -> null
|
||||
is JsonPrimitive -> content
|
||||
else -> null
|
||||
}
|
||||
|
||||
private fun JsonElement?.asLongOrNull(): Long? =
|
||||
when (this) {
|
||||
is JsonPrimitive -> content.toLongOrNull()
|
||||
else -> null
|
||||
}
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
package ai.openclaw.app.chat
|
||||
|
||||
data class ChatMessage(
|
||||
val id: String,
|
||||
val role: String,
|
||||
val content: List<ChatMessageContent>,
|
||||
val timestampMs: Long?,
|
||||
)
|
||||
|
||||
data class ChatMessageContent(
|
||||
val type: String = "text",
|
||||
val text: String? = null,
|
||||
val mimeType: String? = null,
|
||||
val fileName: String? = null,
|
||||
val base64: String? = null,
|
||||
)
|
||||
|
||||
data class ChatPendingToolCall(
|
||||
val toolCallId: String,
|
||||
val name: String,
|
||||
val args: kotlinx.serialization.json.JsonObject? = null,
|
||||
val startedAtMs: Long,
|
||||
val isError: Boolean? = null,
|
||||
)
|
||||
|
||||
data class ChatSessionEntry(
|
||||
val key: String,
|
||||
val updatedAtMs: Long?,
|
||||
val displayName: String? = null,
|
||||
)
|
||||
|
||||
data class ChatHistory(
|
||||
val sessionKey: String,
|
||||
val sessionId: String?,
|
||||
val thinkingLevel: String?,
|
||||
val messages: List<ChatMessage>,
|
||||
)
|
||||
|
||||
data class OutgoingAttachment(
|
||||
val type: String,
|
||||
val mimeType: String,
|
||||
val fileName: String,
|
||||
val base64: String,
|
||||
)
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
object BonjourEscapes {
|
||||
fun decode(input: String): String {
|
||||
if (input.isEmpty()) return input
|
||||
|
||||
val bytes = mutableListOf<Byte>()
|
||||
var i = 0
|
||||
while (i < input.length) {
|
||||
if (input[i] == '\\' && i + 3 < input.length) {
|
||||
val d0 = input[i + 1]
|
||||
val d1 = input[i + 2]
|
||||
val d2 = input[i + 3]
|
||||
if (d0.isDigit() && d1.isDigit() && d2.isDigit()) {
|
||||
val value =
|
||||
((d0.code - '0'.code) * 100) + ((d1.code - '0'.code) * 10) + (d2.code - '0'.code)
|
||||
if (value in 0..255) {
|
||||
bytes.add(value.toByte())
|
||||
i += 4
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
val codePoint = Character.codePointAt(input, i)
|
||||
val charBytes = String(Character.toChars(codePoint)).toByteArray(Charsets.UTF_8)
|
||||
for (b in charBytes) {
|
||||
bytes.add(b)
|
||||
}
|
||||
i += Character.charCount(codePoint)
|
||||
}
|
||||
|
||||
return String(bytes.toByteArray(), Charsets.UTF_8)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
internal object DeviceAuthPayload {
|
||||
fun buildV3(
|
||||
deviceId: String,
|
||||
clientId: String,
|
||||
clientMode: String,
|
||||
role: String,
|
||||
scopes: List<String>,
|
||||
signedAtMs: Long,
|
||||
token: String?,
|
||||
nonce: String,
|
||||
platform: String?,
|
||||
deviceFamily: String?,
|
||||
): String {
|
||||
val scopeString = scopes.joinToString(",")
|
||||
val authToken = token.orEmpty()
|
||||
val platformNorm = normalizeMetadataField(platform)
|
||||
val deviceFamilyNorm = normalizeMetadataField(deviceFamily)
|
||||
return listOf(
|
||||
"v3",
|
||||
deviceId,
|
||||
clientId,
|
||||
clientMode,
|
||||
role,
|
||||
scopeString,
|
||||
signedAtMs.toString(),
|
||||
authToken,
|
||||
nonce,
|
||||
platformNorm,
|
||||
deviceFamilyNorm,
|
||||
).joinToString("|")
|
||||
}
|
||||
|
||||
internal fun normalizeMetadataField(value: String?): String {
|
||||
val trimmed = value?.trim().orEmpty()
|
||||
if (trimmed.isEmpty()) {
|
||||
return ""
|
||||
}
|
||||
// Keep cross-runtime normalization deterministic (TS/Swift/Kotlin):
|
||||
// lowercase ASCII A-Z only for auth payload metadata fields.
|
||||
val out = StringBuilder(trimmed.length)
|
||||
for (ch in trimmed) {
|
||||
if (ch in 'A'..'Z') {
|
||||
out.append((ch.code + 32).toChar())
|
||||
} else {
|
||||
out.append(ch)
|
||||
}
|
||||
}
|
||||
return out.toString()
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,92 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
import ai.openclaw.app.SecurePrefs
|
||||
import kotlinx.serialization.Serializable
|
||||
import kotlinx.serialization.encodeToString
|
||||
import kotlinx.serialization.json.Json
|
||||
|
||||
data class DeviceAuthEntry(
|
||||
val token: String,
|
||||
val role: String,
|
||||
val scopes: List<String>,
|
||||
val updatedAtMs: Long,
|
||||
)
|
||||
|
||||
@Serializable
|
||||
private data class PersistedDeviceAuthMetadata(
|
||||
val scopes: List<String> = emptyList(),
|
||||
val updatedAtMs: Long = 0L,
|
||||
)
|
||||
|
||||
interface DeviceAuthTokenStore {
|
||||
fun loadEntry(deviceId: String, role: String): DeviceAuthEntry?
|
||||
fun loadToken(deviceId: String, role: String): String? = loadEntry(deviceId, role)?.token
|
||||
fun saveToken(deviceId: String, role: String, token: String, scopes: List<String> = emptyList())
|
||||
fun clearToken(deviceId: String, role: String)
|
||||
}
|
||||
|
||||
class DeviceAuthStore(private val prefs: SecurePrefs) : DeviceAuthTokenStore {
|
||||
private val json = Json { ignoreUnknownKeys = true }
|
||||
|
||||
override fun loadEntry(deviceId: String, role: String): DeviceAuthEntry? {
|
||||
val key = tokenKey(deviceId, role)
|
||||
val token = prefs.getString(key)?.trim()?.takeIf { it.isNotEmpty() } ?: return null
|
||||
val normalizedRole = normalizeRole(role)
|
||||
val metadata =
|
||||
prefs.getString(metadataKey(deviceId, role))
|
||||
?.let { raw ->
|
||||
runCatching { json.decodeFromString<PersistedDeviceAuthMetadata>(raw) }.getOrNull()
|
||||
}
|
||||
return DeviceAuthEntry(
|
||||
token = token,
|
||||
role = normalizedRole,
|
||||
scopes = metadata?.scopes ?: emptyList(),
|
||||
updatedAtMs = metadata?.updatedAtMs ?: 0L,
|
||||
)
|
||||
}
|
||||
|
||||
override fun saveToken(deviceId: String, role: String, token: String, scopes: List<String>) {
|
||||
val normalizedScopes = normalizeScopes(scopes)
|
||||
val key = tokenKey(deviceId, role)
|
||||
prefs.putString(key, token.trim())
|
||||
prefs.putString(
|
||||
metadataKey(deviceId, role),
|
||||
json.encodeToString(
|
||||
PersistedDeviceAuthMetadata(
|
||||
scopes = normalizedScopes,
|
||||
updatedAtMs = System.currentTimeMillis(),
|
||||
),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
override fun clearToken(deviceId: String, role: String) {
|
||||
val key = tokenKey(deviceId, role)
|
||||
prefs.remove(key)
|
||||
prefs.remove(metadataKey(deviceId, role))
|
||||
}
|
||||
|
||||
private fun tokenKey(deviceId: String, role: String): String {
|
||||
val normalizedDevice = normalizeDeviceId(deviceId)
|
||||
val normalizedRole = normalizeRole(role)
|
||||
return "gateway.deviceToken.$normalizedDevice.$normalizedRole"
|
||||
}
|
||||
|
||||
private fun metadataKey(deviceId: String, role: String): String {
|
||||
val normalizedDevice = normalizeDeviceId(deviceId)
|
||||
val normalizedRole = normalizeRole(role)
|
||||
return "gateway.deviceTokenMeta.$normalizedDevice.$normalizedRole"
|
||||
}
|
||||
|
||||
private fun normalizeDeviceId(deviceId: String): String = deviceId.trim().lowercase()
|
||||
|
||||
private fun normalizeRole(role: String): String = role.trim().lowercase()
|
||||
|
||||
private fun normalizeScopes(scopes: List<String>): List<String> {
|
||||
return scopes
|
||||
.map { it.trim() }
|
||||
.filter { it.isNotEmpty() }
|
||||
.distinct()
|
||||
.sorted()
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,174 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
import android.content.Context
|
||||
import android.util.Base64
|
||||
import java.io.File
|
||||
import java.security.MessageDigest
|
||||
import kotlinx.serialization.Serializable
|
||||
import kotlinx.serialization.json.Json
|
||||
|
||||
@Serializable
|
||||
data class DeviceIdentity(
|
||||
val deviceId: String,
|
||||
val publicKeyRawBase64: String,
|
||||
val privateKeyPkcs8Base64: String,
|
||||
val createdAtMs: Long,
|
||||
)
|
||||
|
||||
class DeviceIdentityStore(context: Context) {
|
||||
private val json = Json { ignoreUnknownKeys = true }
|
||||
private val identityFile = File(context.filesDir, "openclaw/identity/device.json")
|
||||
@Volatile private var cachedIdentity: DeviceIdentity? = null
|
||||
|
||||
@Synchronized
|
||||
fun loadOrCreate(): DeviceIdentity {
|
||||
cachedIdentity?.let { return it }
|
||||
val existing = load()
|
||||
if (existing != null) {
|
||||
val derived = deriveDeviceId(existing.publicKeyRawBase64)
|
||||
if (derived != null && derived != existing.deviceId) {
|
||||
val updated = existing.copy(deviceId = derived)
|
||||
save(updated)
|
||||
cachedIdentity = updated
|
||||
return updated
|
||||
}
|
||||
cachedIdentity = existing
|
||||
return existing
|
||||
}
|
||||
val fresh = generate()
|
||||
save(fresh)
|
||||
cachedIdentity = fresh
|
||||
return fresh
|
||||
}
|
||||
|
||||
fun signPayload(payload: String, identity: DeviceIdentity): String? {
|
||||
return try {
|
||||
// Use BC lightweight API directly — JCA provider registration is broken by R8
|
||||
val privateKeyBytes = Base64.decode(identity.privateKeyPkcs8Base64, Base64.DEFAULT)
|
||||
val pkInfo = org.bouncycastle.asn1.pkcs.PrivateKeyInfo.getInstance(privateKeyBytes)
|
||||
val parsed = pkInfo.parsePrivateKey()
|
||||
val rawPrivate = org.bouncycastle.asn1.DEROctetString.getInstance(parsed).octets
|
||||
val privateKey = org.bouncycastle.crypto.params.Ed25519PrivateKeyParameters(rawPrivate, 0)
|
||||
val signer = org.bouncycastle.crypto.signers.Ed25519Signer()
|
||||
signer.init(true, privateKey)
|
||||
val payloadBytes = payload.toByteArray(Charsets.UTF_8)
|
||||
signer.update(payloadBytes, 0, payloadBytes.size)
|
||||
base64UrlEncode(signer.generateSignature())
|
||||
} catch (e: Throwable) {
|
||||
android.util.Log.e("DeviceAuth", "signPayload FAILED: ${e.javaClass.simpleName}: ${e.message}", e)
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
fun verifySelfSignature(payload: String, signatureBase64Url: String, identity: DeviceIdentity): Boolean {
|
||||
return try {
|
||||
val rawPublicKey = Base64.decode(identity.publicKeyRawBase64, Base64.DEFAULT)
|
||||
val pubKey = org.bouncycastle.crypto.params.Ed25519PublicKeyParameters(rawPublicKey, 0)
|
||||
val sigBytes = base64UrlDecode(signatureBase64Url)
|
||||
val verifier = org.bouncycastle.crypto.signers.Ed25519Signer()
|
||||
verifier.init(false, pubKey)
|
||||
val payloadBytes = payload.toByteArray(Charsets.UTF_8)
|
||||
verifier.update(payloadBytes, 0, payloadBytes.size)
|
||||
verifier.verifySignature(sigBytes)
|
||||
} catch (e: Throwable) {
|
||||
android.util.Log.e("DeviceAuth", "self-verify exception: ${e.message}", e)
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
private fun base64UrlDecode(input: String): ByteArray {
|
||||
val normalized = input.replace('-', '+').replace('_', '/')
|
||||
val padded = normalized + "=".repeat((4 - normalized.length % 4) % 4)
|
||||
return Base64.decode(padded, Base64.DEFAULT)
|
||||
}
|
||||
|
||||
fun publicKeyBase64Url(identity: DeviceIdentity): String? {
|
||||
return try {
|
||||
val raw = Base64.decode(identity.publicKeyRawBase64, Base64.DEFAULT)
|
||||
base64UrlEncode(raw)
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun load(): DeviceIdentity? {
|
||||
return readIdentity(identityFile)
|
||||
}
|
||||
|
||||
private fun readIdentity(file: File): DeviceIdentity? {
|
||||
return try {
|
||||
if (!file.exists()) return null
|
||||
val raw = file.readText(Charsets.UTF_8)
|
||||
val decoded = json.decodeFromString(DeviceIdentity.serializer(), raw)
|
||||
if (decoded.deviceId.isBlank() ||
|
||||
decoded.publicKeyRawBase64.isBlank() ||
|
||||
decoded.privateKeyPkcs8Base64.isBlank()
|
||||
) {
|
||||
null
|
||||
} else {
|
||||
decoded
|
||||
}
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun save(identity: DeviceIdentity) {
|
||||
try {
|
||||
identityFile.parentFile?.mkdirs()
|
||||
val encoded = json.encodeToString(DeviceIdentity.serializer(), identity)
|
||||
identityFile.writeText(encoded, Charsets.UTF_8)
|
||||
} catch (_: Throwable) {
|
||||
// best-effort only
|
||||
}
|
||||
}
|
||||
|
||||
private fun generate(): DeviceIdentity {
|
||||
// Use BC lightweight API directly to avoid JCA provider issues with R8
|
||||
val kpGen = org.bouncycastle.crypto.generators.Ed25519KeyPairGenerator()
|
||||
kpGen.init(org.bouncycastle.crypto.params.Ed25519KeyGenerationParameters(java.security.SecureRandom()))
|
||||
val kp = kpGen.generateKeyPair()
|
||||
val pubKey = kp.public as org.bouncycastle.crypto.params.Ed25519PublicKeyParameters
|
||||
val privKey = kp.private as org.bouncycastle.crypto.params.Ed25519PrivateKeyParameters
|
||||
val rawPublic = pubKey.encoded // 32 bytes
|
||||
val deviceId = sha256Hex(rawPublic)
|
||||
// Encode private key as PKCS8 for storage
|
||||
val privKeyInfo = org.bouncycastle.crypto.util.PrivateKeyInfoFactory.createPrivateKeyInfo(privKey)
|
||||
val pkcs8Bytes = privKeyInfo.encoded
|
||||
return DeviceIdentity(
|
||||
deviceId = deviceId,
|
||||
publicKeyRawBase64 = Base64.encodeToString(rawPublic, Base64.NO_WRAP),
|
||||
privateKeyPkcs8Base64 = Base64.encodeToString(pkcs8Bytes, Base64.NO_WRAP),
|
||||
createdAtMs = System.currentTimeMillis(),
|
||||
)
|
||||
}
|
||||
|
||||
private fun deriveDeviceId(publicKeyRawBase64: String): String? {
|
||||
return try {
|
||||
val raw = Base64.decode(publicKeyRawBase64, Base64.DEFAULT)
|
||||
sha256Hex(raw)
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun sha256Hex(data: ByteArray): String {
|
||||
val digest = MessageDigest.getInstance("SHA-256").digest(data)
|
||||
val out = CharArray(digest.size * 2)
|
||||
var i = 0
|
||||
for (byte in digest) {
|
||||
val v = byte.toInt() and 0xff
|
||||
out[i++] = HEX[v ushr 4]
|
||||
out[i++] = HEX[v and 0x0f]
|
||||
}
|
||||
return String(out)
|
||||
}
|
||||
|
||||
private fun base64UrlEncode(data: ByteArray): String {
|
||||
return Base64.encodeToString(data, Base64.URL_SAFE or Base64.NO_WRAP or Base64.NO_PADDING)
|
||||
}
|
||||
|
||||
companion object {
|
||||
private val HEX = "0123456789abcdef".toCharArray()
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,522 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
import android.content.Context
|
||||
import android.net.ConnectivityManager
|
||||
import android.net.DnsResolver
|
||||
import android.net.NetworkCapabilities
|
||||
import android.net.nsd.NsdManager
|
||||
import android.net.nsd.NsdServiceInfo
|
||||
import android.os.CancellationSignal
|
||||
import android.util.Log
|
||||
import java.io.IOException
|
||||
import java.net.InetSocketAddress
|
||||
import java.nio.ByteBuffer
|
||||
import java.nio.charset.CodingErrorAction
|
||||
import java.time.Duration
|
||||
import java.util.concurrent.ConcurrentHashMap
|
||||
import java.util.concurrent.Executor
|
||||
import java.util.concurrent.Executors
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
import org.xbill.DNS.AAAARecord
|
||||
import org.xbill.DNS.ARecord
|
||||
import org.xbill.DNS.DClass
|
||||
import org.xbill.DNS.ExtendedResolver
|
||||
import org.xbill.DNS.Message
|
||||
import org.xbill.DNS.Name
|
||||
import org.xbill.DNS.PTRRecord
|
||||
import org.xbill.DNS.Record
|
||||
import org.xbill.DNS.Rcode
|
||||
import org.xbill.DNS.Resolver
|
||||
import org.xbill.DNS.SRVRecord
|
||||
import org.xbill.DNS.Section
|
||||
import org.xbill.DNS.SimpleResolver
|
||||
import org.xbill.DNS.TextParseException
|
||||
import org.xbill.DNS.TXTRecord
|
||||
import org.xbill.DNS.Type
|
||||
import kotlin.coroutines.resume
|
||||
import kotlin.coroutines.resumeWithException
|
||||
|
||||
@Suppress("DEPRECATION")
|
||||
class GatewayDiscovery(
|
||||
context: Context,
|
||||
private val scope: CoroutineScope,
|
||||
) {
|
||||
private val nsd = context.getSystemService(NsdManager::class.java)
|
||||
private val connectivity = context.getSystemService(ConnectivityManager::class.java)
|
||||
private val dns = DnsResolver.getInstance()
|
||||
private val serviceType = "_openclaw-gw._tcp."
|
||||
private val wideAreaDomain = System.getenv("OPENCLAW_WIDE_AREA_DOMAIN")
|
||||
private val logTag = "OpenClaw/GatewayDiscovery"
|
||||
|
||||
private val localById = ConcurrentHashMap<String, GatewayEndpoint>()
|
||||
private val unicastById = ConcurrentHashMap<String, GatewayEndpoint>()
|
||||
private val _gateways = MutableStateFlow<List<GatewayEndpoint>>(emptyList())
|
||||
val gateways: StateFlow<List<GatewayEndpoint>> = _gateways.asStateFlow()
|
||||
|
||||
private val _statusText = MutableStateFlow("Searching…")
|
||||
val statusText: StateFlow<String> = _statusText.asStateFlow()
|
||||
|
||||
private var unicastJob: Job? = null
|
||||
private val dnsExecutor: Executor = Executors.newCachedThreadPool()
|
||||
|
||||
@Volatile private var lastWideAreaRcode: Int? = null
|
||||
@Volatile private var lastWideAreaCount: Int = 0
|
||||
|
||||
private val discoveryListener =
|
||||
object : NsdManager.DiscoveryListener {
|
||||
override fun onStartDiscoveryFailed(serviceType: String, errorCode: Int) {}
|
||||
override fun onStopDiscoveryFailed(serviceType: String, errorCode: Int) {}
|
||||
override fun onDiscoveryStarted(serviceType: String) {}
|
||||
override fun onDiscoveryStopped(serviceType: String) {}
|
||||
|
||||
override fun onServiceFound(serviceInfo: NsdServiceInfo) {
|
||||
if (serviceInfo.serviceType != this@GatewayDiscovery.serviceType) return
|
||||
resolve(serviceInfo)
|
||||
}
|
||||
|
||||
override fun onServiceLost(serviceInfo: NsdServiceInfo) {
|
||||
val serviceName = BonjourEscapes.decode(serviceInfo.serviceName)
|
||||
val id = stableId(serviceName, "local.")
|
||||
localById.remove(id)
|
||||
publish()
|
||||
}
|
||||
}
|
||||
|
||||
init {
|
||||
startLocalDiscovery()
|
||||
if (!wideAreaDomain.isNullOrBlank()) {
|
||||
startUnicastDiscovery(wideAreaDomain)
|
||||
}
|
||||
}
|
||||
|
||||
private fun startLocalDiscovery() {
|
||||
try {
|
||||
nsd.discoverServices(serviceType, NsdManager.PROTOCOL_DNS_SD, discoveryListener)
|
||||
} catch (_: Throwable) {
|
||||
// ignore (best-effort)
|
||||
}
|
||||
}
|
||||
|
||||
private fun stopLocalDiscovery() {
|
||||
try {
|
||||
nsd.stopServiceDiscovery(discoveryListener)
|
||||
} catch (_: Throwable) {
|
||||
// ignore (best-effort)
|
||||
}
|
||||
}
|
||||
|
||||
private fun startUnicastDiscovery(domain: String) {
|
||||
unicastJob =
|
||||
scope.launch(Dispatchers.IO) {
|
||||
while (true) {
|
||||
try {
|
||||
refreshUnicast(domain)
|
||||
} catch (_: Throwable) {
|
||||
// ignore (best-effort)
|
||||
}
|
||||
delay(5000)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun resolve(serviceInfo: NsdServiceInfo) {
|
||||
nsd.resolveService(
|
||||
serviceInfo,
|
||||
object : NsdManager.ResolveListener {
|
||||
override fun onResolveFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {}
|
||||
|
||||
override fun onServiceResolved(resolved: NsdServiceInfo) {
|
||||
val host = resolved.host?.hostAddress ?: return
|
||||
val port = resolved.port
|
||||
if (port <= 0) return
|
||||
|
||||
val rawServiceName = resolved.serviceName
|
||||
val serviceName = BonjourEscapes.decode(rawServiceName)
|
||||
val displayName = BonjourEscapes.decode(txt(resolved, "displayName") ?: serviceName)
|
||||
val lanHost = txt(resolved, "lanHost")
|
||||
val tailnetDns = txt(resolved, "tailnetDns")
|
||||
val gatewayPort = txtInt(resolved, "gatewayPort")
|
||||
val canvasPort = txtInt(resolved, "canvasPort")
|
||||
val tlsEnabled = txtBool(resolved, "gatewayTls")
|
||||
val tlsFingerprint = txt(resolved, "gatewayTlsSha256")
|
||||
val id = stableId(serviceName, "local.")
|
||||
localById[id] =
|
||||
GatewayEndpoint(
|
||||
stableId = id,
|
||||
name = displayName,
|
||||
host = host,
|
||||
port = port,
|
||||
lanHost = lanHost,
|
||||
tailnetDns = tailnetDns,
|
||||
gatewayPort = gatewayPort,
|
||||
canvasPort = canvasPort,
|
||||
tlsEnabled = tlsEnabled,
|
||||
tlsFingerprintSha256 = tlsFingerprint,
|
||||
)
|
||||
publish()
|
||||
}
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
private fun publish() {
|
||||
_gateways.value =
|
||||
(localById.values + unicastById.values).sortedBy { it.name.lowercase() }
|
||||
_statusText.value = buildStatusText()
|
||||
}
|
||||
|
||||
private fun buildStatusText(): String {
|
||||
val localCount = localById.size
|
||||
val wideRcode = lastWideAreaRcode
|
||||
val wideCount = lastWideAreaCount
|
||||
|
||||
val wide =
|
||||
when (wideRcode) {
|
||||
null -> "Wide: ?"
|
||||
Rcode.NOERROR -> "Wide: $wideCount"
|
||||
Rcode.NXDOMAIN -> "Wide: NXDOMAIN"
|
||||
else -> "Wide: ${Rcode.string(wideRcode)}"
|
||||
}
|
||||
|
||||
return when {
|
||||
localCount == 0 && wideRcode == null -> "Searching for gateways…"
|
||||
localCount == 0 -> "$wide"
|
||||
else -> "Local: $localCount • $wide"
|
||||
}
|
||||
}
|
||||
|
||||
private fun stableId(serviceName: String, domain: String): String {
|
||||
return "${serviceType}|${domain}|${normalizeName(serviceName)}"
|
||||
}
|
||||
|
||||
private fun normalizeName(raw: String): String {
|
||||
return raw.trim().split(Regex("\\s+")).joinToString(" ")
|
||||
}
|
||||
|
||||
private fun txt(info: NsdServiceInfo, key: String): String? {
|
||||
val bytes = info.attributes[key] ?: return null
|
||||
return try {
|
||||
String(bytes, Charsets.UTF_8).trim().ifEmpty { null }
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun txtInt(info: NsdServiceInfo, key: String): Int? {
|
||||
return txt(info, key)?.toIntOrNull()
|
||||
}
|
||||
|
||||
private fun txtBool(info: NsdServiceInfo, key: String): Boolean {
|
||||
val raw = txt(info, key)?.trim()?.lowercase() ?: return false
|
||||
return raw == "1" || raw == "true" || raw == "yes"
|
||||
}
|
||||
|
||||
private suspend fun refreshUnicast(domain: String) {
|
||||
val ptrName = "${serviceType}${domain}"
|
||||
val ptrMsg = lookupUnicastMessage(ptrName, Type.PTR) ?: return
|
||||
val ptrRecords = records(ptrMsg, Section.ANSWER).mapNotNull { it as? PTRRecord }
|
||||
|
||||
val next = LinkedHashMap<String, GatewayEndpoint>()
|
||||
for (ptr in ptrRecords) {
|
||||
val instanceFqdn = ptr.target.toString()
|
||||
val srv =
|
||||
recordByName(ptrMsg, instanceFqdn, Type.SRV) as? SRVRecord
|
||||
?: run {
|
||||
val msg = lookupUnicastMessage(instanceFqdn, Type.SRV) ?: return@run null
|
||||
recordByName(msg, instanceFqdn, Type.SRV) as? SRVRecord
|
||||
}
|
||||
?: continue
|
||||
val port = srv.port
|
||||
if (port <= 0) continue
|
||||
|
||||
val targetFqdn = srv.target.toString()
|
||||
val host =
|
||||
resolveHostFromMessage(ptrMsg, targetFqdn)
|
||||
?: resolveHostFromMessage(lookupUnicastMessage(instanceFqdn, Type.SRV), targetFqdn)
|
||||
?: resolveHostUnicast(targetFqdn)
|
||||
?: continue
|
||||
|
||||
val txtFromPtr =
|
||||
recordsByName(ptrMsg, Section.ADDITIONAL)[keyName(instanceFqdn)]
|
||||
.orEmpty()
|
||||
.mapNotNull { it as? TXTRecord }
|
||||
val txt =
|
||||
if (txtFromPtr.isNotEmpty()) {
|
||||
txtFromPtr
|
||||
} else {
|
||||
val msg = lookupUnicastMessage(instanceFqdn, Type.TXT)
|
||||
records(msg, Section.ANSWER).mapNotNull { it as? TXTRecord }
|
||||
}
|
||||
val instanceName = BonjourEscapes.decode(decodeInstanceName(instanceFqdn, domain))
|
||||
val displayName = BonjourEscapes.decode(txtValue(txt, "displayName") ?: instanceName)
|
||||
val lanHost = txtValue(txt, "lanHost")
|
||||
val tailnetDns = txtValue(txt, "tailnetDns")
|
||||
val gatewayPort = txtIntValue(txt, "gatewayPort")
|
||||
val canvasPort = txtIntValue(txt, "canvasPort")
|
||||
val tlsEnabled = txtBoolValue(txt, "gatewayTls")
|
||||
val tlsFingerprint = txtValue(txt, "gatewayTlsSha256")
|
||||
val id = stableId(instanceName, domain)
|
||||
next[id] =
|
||||
GatewayEndpoint(
|
||||
stableId = id,
|
||||
name = displayName,
|
||||
host = host,
|
||||
port = port,
|
||||
lanHost = lanHost,
|
||||
tailnetDns = tailnetDns,
|
||||
gatewayPort = gatewayPort,
|
||||
canvasPort = canvasPort,
|
||||
tlsEnabled = tlsEnabled,
|
||||
tlsFingerprintSha256 = tlsFingerprint,
|
||||
)
|
||||
}
|
||||
|
||||
unicastById.clear()
|
||||
unicastById.putAll(next)
|
||||
lastWideAreaRcode = ptrMsg.header.rcode
|
||||
lastWideAreaCount = next.size
|
||||
publish()
|
||||
|
||||
if (next.isEmpty()) {
|
||||
Log.d(
|
||||
logTag,
|
||||
"wide-area discovery: 0 results for $ptrName (rcode=${Rcode.string(ptrMsg.header.rcode)})",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun decodeInstanceName(instanceFqdn: String, domain: String): String {
|
||||
val suffix = "${serviceType}${domain}"
|
||||
val withoutSuffix =
|
||||
if (instanceFqdn.endsWith(suffix)) {
|
||||
instanceFqdn.removeSuffix(suffix)
|
||||
} else {
|
||||
instanceFqdn.substringBefore(serviceType)
|
||||
}
|
||||
return normalizeName(stripTrailingDot(withoutSuffix))
|
||||
}
|
||||
|
||||
private fun stripTrailingDot(raw: String): String {
|
||||
return raw.removeSuffix(".")
|
||||
}
|
||||
|
||||
private suspend fun lookupUnicastMessage(name: String, type: Int): Message? {
|
||||
val query =
|
||||
try {
|
||||
Message.newQuery(
|
||||
org.xbill.DNS.Record.newRecord(
|
||||
Name.fromString(name),
|
||||
type,
|
||||
DClass.IN,
|
||||
),
|
||||
)
|
||||
} catch (_: TextParseException) {
|
||||
return null
|
||||
}
|
||||
|
||||
val system = queryViaSystemDns(query)
|
||||
if (records(system, Section.ANSWER).any { it.type == type }) return system
|
||||
|
||||
val direct = createDirectResolver() ?: return system
|
||||
return try {
|
||||
val msg = direct.send(query)
|
||||
if (records(msg, Section.ANSWER).any { it.type == type }) msg else system
|
||||
} catch (_: Throwable) {
|
||||
system
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun queryViaSystemDns(query: Message): Message? {
|
||||
val network = preferredDnsNetwork()
|
||||
val bytes =
|
||||
try {
|
||||
rawQuery(network, query.toWire())
|
||||
} catch (_: Throwable) {
|
||||
return null
|
||||
}
|
||||
|
||||
return try {
|
||||
Message(bytes)
|
||||
} catch (_: IOException) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun records(msg: Message?, section: Int): List<Record> {
|
||||
return msg?.getSection(section).orEmpty()
|
||||
}
|
||||
|
||||
private fun keyName(raw: String): String {
|
||||
return raw.trim().lowercase()
|
||||
}
|
||||
|
||||
private fun recordsByName(msg: Message, section: Int): Map<String, List<Record>> {
|
||||
val next = LinkedHashMap<String, MutableList<Record>>()
|
||||
for (r in records(msg, section)) {
|
||||
val name = r.name?.toString() ?: continue
|
||||
next.getOrPut(keyName(name)) { mutableListOf() }.add(r)
|
||||
}
|
||||
return next
|
||||
}
|
||||
|
||||
private fun recordByName(msg: Message, fqdn: String, type: Int): Record? {
|
||||
val key = keyName(fqdn)
|
||||
val byNameAnswer = recordsByName(msg, Section.ANSWER)
|
||||
val fromAnswer = byNameAnswer[key].orEmpty().firstOrNull { it.type == type }
|
||||
if (fromAnswer != null) return fromAnswer
|
||||
|
||||
val byNameAdditional = recordsByName(msg, Section.ADDITIONAL)
|
||||
return byNameAdditional[key].orEmpty().firstOrNull { it.type == type }
|
||||
}
|
||||
|
||||
private fun resolveHostFromMessage(msg: Message?, hostname: String): String? {
|
||||
val m = msg ?: return null
|
||||
val key = keyName(hostname)
|
||||
val additional = recordsByName(m, Section.ADDITIONAL)[key].orEmpty()
|
||||
val a = additional.mapNotNull { it as? ARecord }.mapNotNull { it.address?.hostAddress }
|
||||
val aaaa = additional.mapNotNull { it as? AAAARecord }.mapNotNull { it.address?.hostAddress }
|
||||
return a.firstOrNull() ?: aaaa.firstOrNull()
|
||||
}
|
||||
|
||||
private fun preferredDnsNetwork(): android.net.Network? {
|
||||
val cm = connectivity ?: return null
|
||||
|
||||
// Prefer VPN (Tailscale) when present; otherwise use the active network.
|
||||
cm.allNetworks.firstOrNull { n ->
|
||||
val caps = cm.getNetworkCapabilities(n) ?: return@firstOrNull false
|
||||
caps.hasTransport(NetworkCapabilities.TRANSPORT_VPN)
|
||||
}?.let { return it }
|
||||
|
||||
return cm.activeNetwork
|
||||
}
|
||||
|
||||
private fun createDirectResolver(): Resolver? {
|
||||
val cm = connectivity ?: return null
|
||||
|
||||
val candidateNetworks =
|
||||
buildList {
|
||||
cm.allNetworks
|
||||
.firstOrNull { n ->
|
||||
val caps = cm.getNetworkCapabilities(n) ?: return@firstOrNull false
|
||||
caps.hasTransport(NetworkCapabilities.TRANSPORT_VPN)
|
||||
}?.let(::add)
|
||||
cm.activeNetwork?.let(::add)
|
||||
}.distinct()
|
||||
|
||||
val servers =
|
||||
candidateNetworks
|
||||
.asSequence()
|
||||
.flatMap { n ->
|
||||
cm.getLinkProperties(n)?.dnsServers?.asSequence() ?: emptySequence()
|
||||
}
|
||||
.distinctBy { it.hostAddress ?: it.toString() }
|
||||
.toList()
|
||||
if (servers.isEmpty()) return null
|
||||
|
||||
return try {
|
||||
val resolvers =
|
||||
servers.mapNotNull { addr ->
|
||||
try {
|
||||
SimpleResolver().apply {
|
||||
setAddress(InetSocketAddress(addr, 53))
|
||||
setTimeout(Duration.ofSeconds(3))
|
||||
}
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
if (resolvers.isEmpty()) return null
|
||||
ExtendedResolver(resolvers.toTypedArray()).apply { setTimeout(Duration.ofSeconds(3)) }
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun rawQuery(network: android.net.Network?, wireQuery: ByteArray): ByteArray =
|
||||
suspendCancellableCoroutine { cont ->
|
||||
val signal = CancellationSignal()
|
||||
cont.invokeOnCancellation { signal.cancel() }
|
||||
|
||||
dns.rawQuery(
|
||||
network,
|
||||
wireQuery,
|
||||
DnsResolver.FLAG_EMPTY,
|
||||
dnsExecutor,
|
||||
signal,
|
||||
object : DnsResolver.Callback<ByteArray> {
|
||||
override fun onAnswer(answer: ByteArray, rcode: Int) {
|
||||
cont.resume(answer)
|
||||
}
|
||||
|
||||
override fun onError(error: DnsResolver.DnsException) {
|
||||
cont.resumeWithException(error)
|
||||
}
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
private fun txtValue(records: List<TXTRecord>, key: String): String? {
|
||||
val prefix = "$key="
|
||||
for (r in records) {
|
||||
val strings: List<String> =
|
||||
try {
|
||||
r.strings.mapNotNull { it as? String }
|
||||
} catch (_: Throwable) {
|
||||
emptyList()
|
||||
}
|
||||
for (s in strings) {
|
||||
val trimmed = decodeDnsTxtString(s).trim()
|
||||
if (trimmed.startsWith(prefix)) {
|
||||
return trimmed.removePrefix(prefix).trim().ifEmpty { null }
|
||||
}
|
||||
}
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun txtIntValue(records: List<TXTRecord>, key: String): Int? {
|
||||
return txtValue(records, key)?.toIntOrNull()
|
||||
}
|
||||
|
||||
private fun txtBoolValue(records: List<TXTRecord>, key: String): Boolean {
|
||||
val raw = txtValue(records, key)?.trim()?.lowercase() ?: return false
|
||||
return raw == "1" || raw == "true" || raw == "yes"
|
||||
}
|
||||
|
||||
private fun decodeDnsTxtString(raw: String): String {
|
||||
// dnsjava treats TXT as opaque bytes and decodes as ISO-8859-1 to preserve bytes.
|
||||
// Our TXT payload is UTF-8 (written by the gateway), so re-decode when possible.
|
||||
val bytes = raw.toByteArray(Charsets.ISO_8859_1)
|
||||
val decoder =
|
||||
Charsets.UTF_8
|
||||
.newDecoder()
|
||||
.onMalformedInput(CodingErrorAction.REPORT)
|
||||
.onUnmappableCharacter(CodingErrorAction.REPORT)
|
||||
return try {
|
||||
decoder.decode(ByteBuffer.wrap(bytes)).toString()
|
||||
} catch (_: Throwable) {
|
||||
raw
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun resolveHostUnicast(hostname: String): String? {
|
||||
val a =
|
||||
records(lookupUnicastMessage(hostname, Type.A), Section.ANSWER)
|
||||
.mapNotNull { it as? ARecord }
|
||||
.mapNotNull { it.address?.hostAddress }
|
||||
val aaaa =
|
||||
records(lookupUnicastMessage(hostname, Type.AAAA), Section.ANSWER)
|
||||
.mapNotNull { it as? AAAARecord }
|
||||
.mapNotNull { it.address?.hostAddress }
|
||||
|
||||
return a.firstOrNull() ?: aaaa.firstOrNull()
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
data class GatewayEndpoint(
|
||||
val stableId: String,
|
||||
val name: String,
|
||||
val host: String,
|
||||
val port: Int,
|
||||
val lanHost: String? = null,
|
||||
val tailnetDns: String? = null,
|
||||
val gatewayPort: Int? = null,
|
||||
val canvasPort: Int? = null,
|
||||
val tlsEnabled: Boolean = false,
|
||||
val tlsFingerprintSha256: String? = null,
|
||||
) {
|
||||
companion object {
|
||||
fun manual(host: String, port: Int): GatewayEndpoint =
|
||||
GatewayEndpoint(
|
||||
stableId = "manual|${host.lowercase()}|$port",
|
||||
name = "$host:$port",
|
||||
host = host,
|
||||
port = port,
|
||||
tlsEnabled = false,
|
||||
tlsFingerprintSha256 = null,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,124 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
import android.os.Build
|
||||
import java.net.InetAddress
|
||||
import java.util.Locale
|
||||
|
||||
internal fun isLoopbackGatewayHost(
|
||||
rawHost: String?,
|
||||
allowEmulatorBridgeAlias: Boolean = isAndroidEmulatorRuntime(),
|
||||
): Boolean {
|
||||
var host =
|
||||
rawHost
|
||||
?.trim()
|
||||
?.lowercase(Locale.US)
|
||||
?.trim('[', ']')
|
||||
.orEmpty()
|
||||
if (host.endsWith(".")) {
|
||||
host = host.dropLast(1)
|
||||
}
|
||||
val zoneIndex = host.indexOf('%')
|
||||
if (zoneIndex >= 0) return false
|
||||
if (host.isEmpty()) return false
|
||||
if (host == "localhost") return true
|
||||
if (allowEmulatorBridgeAlias && host == "10.0.2.2") return true
|
||||
|
||||
parseIpv4Address(host)?.let { ipv4 ->
|
||||
return ipv4.first() == 127.toByte()
|
||||
}
|
||||
if (!host.contains(':') || !host.all(::isIpv6LiteralChar)) return false
|
||||
|
||||
val address = runCatching { InetAddress.getByName(host) }.getOrNull()?.address ?: return false
|
||||
if (address.size == 4) {
|
||||
return address[0] == 127.toByte()
|
||||
}
|
||||
if (address.size != 16) return false
|
||||
// `::1` is 15 zero bytes followed by `0x01`.
|
||||
val isIpv6Loopback = address.copyOfRange(0, 15).all { it == 0.toByte() } && address[15] == 1.toByte()
|
||||
if (isIpv6Loopback) return true
|
||||
|
||||
val isMappedIpv4 =
|
||||
address.copyOfRange(0, 10).all { it == 0.toByte() } &&
|
||||
address[10] == 0xFF.toByte() &&
|
||||
address[11] == 0xFF.toByte()
|
||||
return isMappedIpv4 && address[12] == 127.toByte()
|
||||
}
|
||||
|
||||
internal fun isPrivateLanGatewayHost(
|
||||
rawHost: String?,
|
||||
allowEmulatorBridgeAlias: Boolean = isAndroidEmulatorRuntime(),
|
||||
): Boolean {
|
||||
var host =
|
||||
rawHost
|
||||
?.trim()
|
||||
?.lowercase(Locale.US)
|
||||
?.trim('[', ']')
|
||||
.orEmpty()
|
||||
if (host.endsWith(".")) {
|
||||
host = host.dropLast(1)
|
||||
}
|
||||
val zoneIndex = host.indexOf('%')
|
||||
if (zoneIndex >= 0) {
|
||||
host = host.substring(0, zoneIndex)
|
||||
}
|
||||
if (host.isEmpty()) return false
|
||||
if (isLoopbackGatewayHost(host, allowEmulatorBridgeAlias = allowEmulatorBridgeAlias)) return true
|
||||
if (host.endsWith(".local")) return true
|
||||
if (!host.contains('.') && !host.contains(':')) return true
|
||||
|
||||
parseIpv4Address(host)?.let { ipv4 ->
|
||||
val first = ipv4[0].toInt() and 0xff
|
||||
val second = ipv4[1].toInt() and 0xff
|
||||
return when {
|
||||
first == 10 -> true
|
||||
first == 172 && second in 16..31 -> true
|
||||
first == 192 && second == 168 -> true
|
||||
first == 169 && second == 254 -> true
|
||||
else -> false
|
||||
}
|
||||
}
|
||||
if (!host.contains(':') || !host.all(::isIpv6LiteralChar)) return false
|
||||
|
||||
val address = runCatching { InetAddress.getByName(host) }.getOrNull() ?: return false
|
||||
return when {
|
||||
address.isLinkLocalAddress -> true
|
||||
address.isSiteLocalAddress -> true
|
||||
else -> {
|
||||
val bytes = address.address
|
||||
bytes.size == 16 && (bytes[0].toInt() and 0xfe) == 0xfc
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun isAndroidEmulatorRuntime(): Boolean {
|
||||
val fingerprint = Build.FINGERPRINT?.lowercase(Locale.US).orEmpty()
|
||||
val model = Build.MODEL?.lowercase(Locale.US).orEmpty()
|
||||
val manufacturer = Build.MANUFACTURER?.lowercase(Locale.US).orEmpty()
|
||||
val brand = Build.BRAND?.lowercase(Locale.US).orEmpty()
|
||||
val device = Build.DEVICE?.lowercase(Locale.US).orEmpty()
|
||||
val product = Build.PRODUCT?.lowercase(Locale.US).orEmpty()
|
||||
|
||||
return fingerprint.contains("generic") ||
|
||||
fingerprint.contains("robolectric") ||
|
||||
model.contains("emulator") ||
|
||||
model.contains("sdk_gphone") ||
|
||||
manufacturer.contains("genymotion") ||
|
||||
(brand.contains("generic") && device.contains("generic")) ||
|
||||
product.contains("sdk_gphone") ||
|
||||
product.contains("emulator") ||
|
||||
product.contains("simulator")
|
||||
}
|
||||
|
||||
private fun parseIpv4Address(host: String): ByteArray? {
|
||||
val parts = host.split('.')
|
||||
if (parts.size != 4) return null
|
||||
val bytes = ByteArray(4)
|
||||
for ((index, part) in parts.withIndex()) {
|
||||
val value = part.toIntOrNull() ?: return null
|
||||
if (value !in 0..255) return null
|
||||
bytes[index] = value.toByte()
|
||||
}
|
||||
return bytes
|
||||
}
|
||||
|
||||
private fun isIpv6LiteralChar(char: Char): Boolean = char in '0'..'9' || char in 'a'..'f' || char == ':' || char == '.'
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
const val GATEWAY_PROTOCOL_VERSION = 3
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,185 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
import android.annotation.SuppressLint
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
import java.io.EOFException
|
||||
import java.net.ConnectException
|
||||
import java.net.InetSocketAddress
|
||||
import java.net.SocketTimeoutException
|
||||
import java.net.UnknownHostException
|
||||
import java.security.MessageDigest
|
||||
import java.security.SecureRandom
|
||||
import java.security.cert.CertificateException
|
||||
import java.security.cert.X509Certificate
|
||||
import java.util.Locale
|
||||
import javax.net.ssl.HttpsURLConnection
|
||||
import javax.net.ssl.HostnameVerifier
|
||||
import javax.net.ssl.SSLContext
|
||||
import javax.net.ssl.SSLException
|
||||
import javax.net.ssl.SSLParameters
|
||||
import javax.net.ssl.SSLSocketFactory
|
||||
import javax.net.ssl.SNIHostName
|
||||
import javax.net.ssl.SSLSocket
|
||||
import javax.net.ssl.TrustManagerFactory
|
||||
import javax.net.ssl.X509TrustManager
|
||||
|
||||
data class GatewayTlsParams(
|
||||
val required: Boolean,
|
||||
val expectedFingerprint: String?,
|
||||
val allowTOFU: Boolean,
|
||||
val stableId: String,
|
||||
)
|
||||
|
||||
data class GatewayTlsConfig(
|
||||
val sslSocketFactory: SSLSocketFactory,
|
||||
val trustManager: X509TrustManager,
|
||||
val hostnameVerifier: HostnameVerifier,
|
||||
)
|
||||
|
||||
enum class GatewayTlsProbeFailure {
|
||||
TLS_UNAVAILABLE,
|
||||
ENDPOINT_UNREACHABLE,
|
||||
}
|
||||
|
||||
data class GatewayTlsProbeResult(
|
||||
val fingerprintSha256: String? = null,
|
||||
val failure: GatewayTlsProbeFailure? = null,
|
||||
)
|
||||
|
||||
fun buildGatewayTlsConfig(
|
||||
params: GatewayTlsParams?,
|
||||
onStore: ((String) -> Unit)? = null,
|
||||
): GatewayTlsConfig? {
|
||||
if (params == null) return null
|
||||
val expected = params.expectedFingerprint?.let(::normalizeFingerprint)
|
||||
val defaultTrust = defaultTrustManager()
|
||||
@SuppressLint("CustomX509TrustManager")
|
||||
val trustManager =
|
||||
object : X509TrustManager {
|
||||
override fun checkClientTrusted(chain: Array<X509Certificate>, authType: String) {
|
||||
defaultTrust.checkClientTrusted(chain, authType)
|
||||
}
|
||||
|
||||
override fun checkServerTrusted(chain: Array<X509Certificate>, authType: String) {
|
||||
if (chain.isEmpty()) throw CertificateException("empty certificate chain")
|
||||
val fingerprint = sha256Hex(chain[0].encoded)
|
||||
if (expected != null) {
|
||||
if (fingerprint != expected) {
|
||||
throw CertificateException("gateway TLS fingerprint mismatch")
|
||||
}
|
||||
return
|
||||
}
|
||||
if (params.allowTOFU) {
|
||||
onStore?.invoke(fingerprint)
|
||||
return
|
||||
}
|
||||
defaultTrust.checkServerTrusted(chain, authType)
|
||||
}
|
||||
|
||||
override fun getAcceptedIssuers(): Array<X509Certificate> = defaultTrust.acceptedIssuers
|
||||
}
|
||||
|
||||
val context = SSLContext.getInstance("TLS")
|
||||
context.init(null, arrayOf(trustManager), SecureRandom())
|
||||
val verifier =
|
||||
if (expected != null || params.allowTOFU) {
|
||||
// When pinning, we intentionally ignore hostname mismatch (service discovery often yields IPs).
|
||||
HostnameVerifier { _, _ -> true }
|
||||
} else {
|
||||
HttpsURLConnection.getDefaultHostnameVerifier()
|
||||
}
|
||||
return GatewayTlsConfig(
|
||||
sslSocketFactory = context.socketFactory,
|
||||
trustManager = trustManager,
|
||||
hostnameVerifier = verifier,
|
||||
)
|
||||
}
|
||||
|
||||
suspend fun probeGatewayTlsFingerprint(
|
||||
host: String,
|
||||
port: Int,
|
||||
timeoutMs: Int = 3_000,
|
||||
): GatewayTlsProbeResult {
|
||||
val trimmedHost = host.trim()
|
||||
if (trimmedHost.isEmpty()) return GatewayTlsProbeResult(failure = GatewayTlsProbeFailure.ENDPOINT_UNREACHABLE)
|
||||
if (port !in 1..65535) return GatewayTlsProbeResult(failure = GatewayTlsProbeFailure.ENDPOINT_UNREACHABLE)
|
||||
|
||||
return withContext(Dispatchers.IO) {
|
||||
val trustAll =
|
||||
@SuppressLint("CustomX509TrustManager", "TrustAllX509TrustManager")
|
||||
object : X509TrustManager {
|
||||
@SuppressLint("TrustAllX509TrustManager")
|
||||
override fun checkClientTrusted(chain: Array<X509Certificate>, authType: String) {}
|
||||
@SuppressLint("TrustAllX509TrustManager")
|
||||
override fun checkServerTrusted(chain: Array<X509Certificate>, authType: String) {}
|
||||
override fun getAcceptedIssuers(): Array<X509Certificate> = emptyArray()
|
||||
}
|
||||
|
||||
val context = SSLContext.getInstance("TLS")
|
||||
context.init(null, arrayOf(trustAll), SecureRandom())
|
||||
|
||||
val socket = (context.socketFactory.createSocket() as SSLSocket)
|
||||
try {
|
||||
socket.soTimeout = timeoutMs
|
||||
socket.connect(InetSocketAddress(trimmedHost, port), timeoutMs)
|
||||
|
||||
// Best-effort SNI for hostnames (avoid crashing on IP literals).
|
||||
try {
|
||||
if (trimmedHost.any { it.isLetter() }) {
|
||||
val params = SSLParameters()
|
||||
params.serverNames = listOf(SNIHostName(trimmedHost))
|
||||
socket.sslParameters = params
|
||||
}
|
||||
} catch (_: Throwable) {
|
||||
// ignore
|
||||
}
|
||||
|
||||
socket.startHandshake()
|
||||
val cert =
|
||||
socket.session.peerCertificates.firstOrNull() as? X509Certificate
|
||||
?: return@withContext GatewayTlsProbeResult(failure = GatewayTlsProbeFailure.TLS_UNAVAILABLE)
|
||||
GatewayTlsProbeResult(fingerprintSha256 = sha256Hex(cert.encoded))
|
||||
} catch (err: Throwable) {
|
||||
val failure =
|
||||
when (err) {
|
||||
is SSLException,
|
||||
is EOFException -> GatewayTlsProbeFailure.TLS_UNAVAILABLE
|
||||
is ConnectException,
|
||||
is SocketTimeoutException,
|
||||
is UnknownHostException -> GatewayTlsProbeFailure.ENDPOINT_UNREACHABLE
|
||||
else -> GatewayTlsProbeFailure.ENDPOINT_UNREACHABLE
|
||||
}
|
||||
GatewayTlsProbeResult(failure = failure)
|
||||
} finally {
|
||||
try {
|
||||
socket.close()
|
||||
} catch (_: Throwable) {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun defaultTrustManager(): X509TrustManager {
|
||||
val factory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
|
||||
factory.init(null as java.security.KeyStore?)
|
||||
val trust =
|
||||
factory.trustManagers.firstOrNull { it is X509TrustManager } as? X509TrustManager
|
||||
return trust ?: throw IllegalStateException("No default X509TrustManager found")
|
||||
}
|
||||
|
||||
private fun sha256Hex(data: ByteArray): String {
|
||||
val digest = MessageDigest.getInstance("SHA-256").digest(data)
|
||||
val out = StringBuilder(digest.size * 2)
|
||||
for (byte in digest) {
|
||||
out.append(String.format(Locale.US, "%02x", byte))
|
||||
}
|
||||
return out.toString()
|
||||
}
|
||||
|
||||
private fun normalizeFingerprint(raw: String): String {
|
||||
val stripped = raw.trim()
|
||||
.replace(Regex("^sha-?256\\s*:?\\s*", RegexOption.IGNORE_CASE), "")
|
||||
return stripped.lowercase(Locale.US).filter { it in '0'..'9' || it in 'a'..'f' }
|
||||
}
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
package ai.openclaw.app.gateway
|
||||
|
||||
data class ParsedInvokeError(
|
||||
val code: String,
|
||||
val message: String,
|
||||
val hadExplicitCode: Boolean,
|
||||
) {
|
||||
val prefixedMessage: String
|
||||
get() = "$code: $message"
|
||||
}
|
||||
|
||||
fun parseInvokeErrorMessage(raw: String): ParsedInvokeError {
|
||||
val trimmed = raw.trim()
|
||||
if (trimmed.isEmpty()) {
|
||||
return ParsedInvokeError(code = "UNAVAILABLE", message = "error", hadExplicitCode = false)
|
||||
}
|
||||
|
||||
val parts = trimmed.split(":", limit = 2)
|
||||
if (parts.size == 2) {
|
||||
val code = parts[0].trim()
|
||||
val rest = parts[1].trim()
|
||||
if (code.isNotEmpty() && code.all { it.isUpperCase() || it == '_' }) {
|
||||
return ParsedInvokeError(
|
||||
code = code,
|
||||
message = rest.ifEmpty { trimmed },
|
||||
hadExplicitCode = true,
|
||||
)
|
||||
}
|
||||
}
|
||||
return ParsedInvokeError(code = "UNAVAILABLE", message = trimmed, hadExplicitCode = false)
|
||||
}
|
||||
|
||||
fun parseInvokeErrorFromThrowable(
|
||||
err: Throwable,
|
||||
fallbackMessage: String = "error",
|
||||
): ParsedInvokeError {
|
||||
val raw = err.message?.trim().takeIf { !it.isNullOrEmpty() } ?: fallbackMessage
|
||||
return parseInvokeErrorMessage(raw)
|
||||
}
|
||||
|
|
@ -0,0 +1,153 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
|
||||
class A2UIHandler(
|
||||
private val canvas: CanvasController,
|
||||
private val json: Json,
|
||||
private val getNodeCanvasHostUrl: () -> String?,
|
||||
private val getOperatorCanvasHostUrl: () -> String?,
|
||||
) {
|
||||
fun isTrustedCanvasActionUrl(rawUrl: String?): Boolean {
|
||||
return CanvasActionTrust.isTrustedCanvasActionUrl(
|
||||
rawUrl = rawUrl,
|
||||
trustedA2uiUrls = listOfNotNull(resolveA2uiHostUrl()),
|
||||
)
|
||||
}
|
||||
|
||||
fun resolveA2uiHostUrl(): String? {
|
||||
val nodeRaw = getNodeCanvasHostUrl()?.trim().orEmpty()
|
||||
val operatorRaw = getOperatorCanvasHostUrl()?.trim().orEmpty()
|
||||
val raw = if (nodeRaw.isNotBlank()) nodeRaw else operatorRaw
|
||||
if (raw.isBlank()) return null
|
||||
val base = raw.trimEnd('/')
|
||||
return "${base}/__openclaw__/a2ui/?platform=android"
|
||||
}
|
||||
|
||||
suspend fun ensureA2uiReady(a2uiUrl: String): Boolean {
|
||||
try {
|
||||
val already = canvas.eval(a2uiReadyCheckJS)
|
||||
if (already == "true") return true
|
||||
} catch (_: Throwable) {
|
||||
// ignore
|
||||
}
|
||||
|
||||
canvas.navigate(a2uiUrl)
|
||||
repeat(50) {
|
||||
try {
|
||||
val ready = canvas.eval(a2uiReadyCheckJS)
|
||||
if (ready == "true") return true
|
||||
} catch (_: Throwable) {
|
||||
// ignore
|
||||
}
|
||||
delay(120)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
fun decodeA2uiMessages(command: String, paramsJson: String?): String {
|
||||
val raw = paramsJson?.trim().orEmpty()
|
||||
if (raw.isBlank()) throw IllegalArgumentException("INVALID_REQUEST: paramsJSON required")
|
||||
|
||||
val obj =
|
||||
json.parseToJsonElement(raw) as? JsonObject
|
||||
?: throw IllegalArgumentException("INVALID_REQUEST: expected object params")
|
||||
|
||||
val jsonlField = (obj["jsonl"] as? JsonPrimitive)?.content?.trim().orEmpty()
|
||||
val hasMessagesArray = obj["messages"] is JsonArray
|
||||
|
||||
if (command == "canvas.a2ui.pushJSONL" || (!hasMessagesArray && jsonlField.isNotBlank())) {
|
||||
val jsonl = jsonlField
|
||||
if (jsonl.isBlank()) throw IllegalArgumentException("INVALID_REQUEST: jsonl required")
|
||||
val messages =
|
||||
jsonl
|
||||
.lineSequence()
|
||||
.map { it.trim() }
|
||||
.filter { it.isNotBlank() }
|
||||
.mapIndexed { idx, line ->
|
||||
val el = json.parseToJsonElement(line)
|
||||
val msg =
|
||||
el as? JsonObject
|
||||
?: throw IllegalArgumentException("A2UI JSONL line ${idx + 1}: expected a JSON object")
|
||||
validateA2uiV0_8(msg, idx + 1)
|
||||
msg
|
||||
}
|
||||
.toList()
|
||||
return JsonArray(messages).toString()
|
||||
}
|
||||
|
||||
val arr = obj["messages"] as? JsonArray ?: throw IllegalArgumentException("INVALID_REQUEST: messages[] required")
|
||||
val out =
|
||||
arr.mapIndexed { idx, el ->
|
||||
val msg =
|
||||
el as? JsonObject
|
||||
?: throw IllegalArgumentException("A2UI messages[${idx}]: expected a JSON object")
|
||||
validateA2uiV0_8(msg, idx + 1)
|
||||
msg
|
||||
}
|
||||
return JsonArray(out).toString()
|
||||
}
|
||||
|
||||
private fun validateA2uiV0_8(msg: JsonObject, lineNumber: Int) {
|
||||
if (msg.containsKey("createSurface")) {
|
||||
throw IllegalArgumentException(
|
||||
"A2UI JSONL line $lineNumber: looks like A2UI v0.9 (`createSurface`). Canvas supports v0.8 messages only.",
|
||||
)
|
||||
}
|
||||
val allowed = setOf("beginRendering", "surfaceUpdate", "dataModelUpdate", "deleteSurface")
|
||||
val matched = msg.keys.filter { allowed.contains(it) }
|
||||
if (matched.size != 1) {
|
||||
val found = msg.keys.sorted().joinToString(", ")
|
||||
throw IllegalArgumentException(
|
||||
"A2UI JSONL line $lineNumber: expected exactly one of ${allowed.sorted().joinToString(", ")}; found: $found",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
const val a2uiReadyCheckJS: String =
|
||||
"""
|
||||
(() => {
|
||||
try {
|
||||
const host = globalThis.openclawA2UI;
|
||||
return !!host && typeof host.applyMessages === 'function';
|
||||
} catch (_) {
|
||||
return false;
|
||||
}
|
||||
})()
|
||||
"""
|
||||
|
||||
const val a2uiResetJS: String =
|
||||
"""
|
||||
(() => {
|
||||
try {
|
||||
const host = globalThis.openclawA2UI;
|
||||
if (!host) return { ok: false, error: "missing openclawA2UI" };
|
||||
return host.reset();
|
||||
} catch (e) {
|
||||
return { ok: false, error: String(e?.message ?? e) };
|
||||
}
|
||||
})()
|
||||
"""
|
||||
|
||||
fun a2uiApplyMessagesJS(messagesJson: String): String {
|
||||
return """
|
||||
(() => {
|
||||
try {
|
||||
const host = globalThis.openclawA2UI;
|
||||
if (!host) return { ok: false, error: "missing openclawA2UI" };
|
||||
const messages = $messagesJson;
|
||||
return host.applyMessages(messages);
|
||||
} catch (e) {
|
||||
return { ok: false, error: String(e?.message ?? e) };
|
||||
}
|
||||
})()
|
||||
""".trimIndent()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,384 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.content.ContentResolver
|
||||
import android.content.ContentUris
|
||||
import android.content.ContentValues
|
||||
import android.content.Context
|
||||
import android.provider.CalendarContract
|
||||
import androidx.core.content.ContextCompat
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import java.time.Instant
|
||||
import java.time.temporal.ChronoUnit
|
||||
import java.util.TimeZone
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonArray
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
private const val DEFAULT_CALENDAR_LIMIT = 50
|
||||
|
||||
internal data class CalendarEventsRequest(
|
||||
val startMs: Long,
|
||||
val endMs: Long,
|
||||
val limit: Int,
|
||||
)
|
||||
|
||||
internal data class CalendarAddRequest(
|
||||
val title: String,
|
||||
val startMs: Long,
|
||||
val endMs: Long,
|
||||
val isAllDay: Boolean,
|
||||
val location: String?,
|
||||
val notes: String?,
|
||||
val calendarId: Long?,
|
||||
val calendarTitle: String?,
|
||||
)
|
||||
|
||||
internal data class CalendarEventRecord(
|
||||
val identifier: String,
|
||||
val title: String,
|
||||
val startISO: String,
|
||||
val endISO: String,
|
||||
val isAllDay: Boolean,
|
||||
val location: String?,
|
||||
val calendarTitle: String?,
|
||||
)
|
||||
|
||||
internal interface CalendarDataSource {
|
||||
fun hasReadPermission(context: Context): Boolean
|
||||
|
||||
fun hasWritePermission(context: Context): Boolean
|
||||
|
||||
fun events(context: Context, request: CalendarEventsRequest): List<CalendarEventRecord>
|
||||
|
||||
fun add(context: Context, request: CalendarAddRequest): CalendarEventRecord
|
||||
}
|
||||
|
||||
private object SystemCalendarDataSource : CalendarDataSource {
|
||||
override fun hasReadPermission(context: Context): Boolean {
|
||||
return ContextCompat.checkSelfPermission(context, Manifest.permission.READ_CALENDAR) ==
|
||||
android.content.pm.PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
|
||||
override fun hasWritePermission(context: Context): Boolean {
|
||||
return ContextCompat.checkSelfPermission(context, Manifest.permission.WRITE_CALENDAR) ==
|
||||
android.content.pm.PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
|
||||
override fun events(context: Context, request: CalendarEventsRequest): List<CalendarEventRecord> {
|
||||
val resolver = context.contentResolver
|
||||
val builder = CalendarContract.Instances.CONTENT_URI.buildUpon()
|
||||
ContentUris.appendId(builder, request.startMs)
|
||||
ContentUris.appendId(builder, request.endMs)
|
||||
val projection =
|
||||
arrayOf(
|
||||
CalendarContract.Instances.EVENT_ID,
|
||||
CalendarContract.Instances.TITLE,
|
||||
CalendarContract.Instances.BEGIN,
|
||||
CalendarContract.Instances.END,
|
||||
CalendarContract.Instances.ALL_DAY,
|
||||
CalendarContract.Instances.EVENT_LOCATION,
|
||||
CalendarContract.Instances.CALENDAR_DISPLAY_NAME,
|
||||
)
|
||||
val sortOrder = "${CalendarContract.Instances.BEGIN} ASC LIMIT ${request.limit}"
|
||||
resolver.query(builder.build(), projection, null, null, sortOrder).use { cursor ->
|
||||
if (cursor == null) return emptyList()
|
||||
val out = mutableListOf<CalendarEventRecord>()
|
||||
while (cursor.moveToNext() && out.size < request.limit) {
|
||||
val id = cursor.getLong(0)
|
||||
val title = cursor.getString(1)?.trim().orEmpty().ifEmpty { "(untitled)" }
|
||||
val beginMs = cursor.getLong(2)
|
||||
val endMs = cursor.getLong(3)
|
||||
val isAllDay = cursor.getInt(4) == 1
|
||||
val location = cursor.getString(5)?.trim()?.ifEmpty { null }
|
||||
val calendarTitle = cursor.getString(6)?.trim()?.ifEmpty { null }
|
||||
out +=
|
||||
CalendarEventRecord(
|
||||
identifier = id.toString(),
|
||||
title = title,
|
||||
startISO = Instant.ofEpochMilli(beginMs).toString(),
|
||||
endISO = Instant.ofEpochMilli(endMs).toString(),
|
||||
isAllDay = isAllDay,
|
||||
location = location,
|
||||
calendarTitle = calendarTitle,
|
||||
)
|
||||
}
|
||||
return out
|
||||
}
|
||||
}
|
||||
|
||||
override fun add(context: Context, request: CalendarAddRequest): CalendarEventRecord {
|
||||
val resolver = context.contentResolver
|
||||
val resolvedCalendarId = resolveCalendarId(resolver, request.calendarId, request.calendarTitle)
|
||||
val values =
|
||||
ContentValues().apply {
|
||||
put(CalendarContract.Events.CALENDAR_ID, resolvedCalendarId)
|
||||
put(CalendarContract.Events.TITLE, request.title)
|
||||
put(CalendarContract.Events.DTSTART, request.startMs)
|
||||
put(CalendarContract.Events.DTEND, request.endMs)
|
||||
put(CalendarContract.Events.ALL_DAY, if (request.isAllDay) 1 else 0)
|
||||
put(CalendarContract.Events.EVENT_TIMEZONE, TimeZone.getDefault().id)
|
||||
request.location?.let { put(CalendarContract.Events.EVENT_LOCATION, it) }
|
||||
request.notes?.let { put(CalendarContract.Events.DESCRIPTION, it) }
|
||||
}
|
||||
val uri = resolver.insert(CalendarContract.Events.CONTENT_URI, values)
|
||||
?: throw IllegalStateException("calendar insert failed")
|
||||
val eventId = uri.lastPathSegment?.toLongOrNull()
|
||||
?: throw IllegalStateException("calendar insert failed")
|
||||
return loadEventById(resolver, eventId)
|
||||
?: throw IllegalStateException("calendar insert failed")
|
||||
}
|
||||
|
||||
private fun resolveCalendarId(
|
||||
resolver: ContentResolver,
|
||||
calendarId: Long?,
|
||||
calendarTitle: String?,
|
||||
): Long {
|
||||
if (calendarId != null) {
|
||||
if (calendarExists(resolver, calendarId)) return calendarId
|
||||
throw IllegalArgumentException("CALENDAR_NOT_FOUND: no calendar id $calendarId")
|
||||
}
|
||||
if (!calendarTitle.isNullOrEmpty()) {
|
||||
findCalendarByTitle(resolver, calendarTitle)?.let { return it }
|
||||
throw IllegalArgumentException("CALENDAR_NOT_FOUND: no calendar named $calendarTitle")
|
||||
}
|
||||
findDefaultCalendarId(resolver)?.let { return it }
|
||||
throw IllegalArgumentException("CALENDAR_NOT_FOUND: no default calendar")
|
||||
}
|
||||
|
||||
private fun calendarExists(resolver: ContentResolver, id: Long): Boolean {
|
||||
val projection = arrayOf(CalendarContract.Calendars._ID)
|
||||
resolver.query(
|
||||
CalendarContract.Calendars.CONTENT_URI,
|
||||
projection,
|
||||
"${CalendarContract.Calendars._ID}=?",
|
||||
arrayOf(id.toString()),
|
||||
null,
|
||||
).use { cursor ->
|
||||
return cursor != null && cursor.moveToFirst()
|
||||
}
|
||||
}
|
||||
|
||||
private fun findCalendarByTitle(resolver: ContentResolver, title: String): Long? {
|
||||
val projection = arrayOf(CalendarContract.Calendars._ID)
|
||||
resolver.query(
|
||||
CalendarContract.Calendars.CONTENT_URI,
|
||||
projection,
|
||||
"${CalendarContract.Calendars.CALENDAR_DISPLAY_NAME}=?",
|
||||
arrayOf(title),
|
||||
"${CalendarContract.Calendars.IS_PRIMARY} DESC",
|
||||
).use { cursor ->
|
||||
if (cursor == null || !cursor.moveToFirst()) return null
|
||||
return cursor.getLong(0)
|
||||
}
|
||||
}
|
||||
|
||||
private fun findDefaultCalendarId(resolver: ContentResolver): Long? {
|
||||
val projection = arrayOf(CalendarContract.Calendars._ID)
|
||||
resolver.query(
|
||||
CalendarContract.Calendars.CONTENT_URI,
|
||||
projection,
|
||||
"${CalendarContract.Calendars.VISIBLE}=1",
|
||||
null,
|
||||
"${CalendarContract.Calendars.IS_PRIMARY} DESC, ${CalendarContract.Calendars._ID} ASC",
|
||||
).use { cursor ->
|
||||
if (cursor == null || !cursor.moveToFirst()) return null
|
||||
return cursor.getLong(0)
|
||||
}
|
||||
}
|
||||
|
||||
private fun loadEventById(
|
||||
resolver: ContentResolver,
|
||||
eventId: Long,
|
||||
): CalendarEventRecord? {
|
||||
val projection =
|
||||
arrayOf(
|
||||
CalendarContract.Events._ID,
|
||||
CalendarContract.Events.TITLE,
|
||||
CalendarContract.Events.DTSTART,
|
||||
CalendarContract.Events.DTEND,
|
||||
CalendarContract.Events.ALL_DAY,
|
||||
CalendarContract.Events.EVENT_LOCATION,
|
||||
CalendarContract.Events.CALENDAR_DISPLAY_NAME,
|
||||
)
|
||||
resolver.query(
|
||||
CalendarContract.Events.CONTENT_URI,
|
||||
projection,
|
||||
"${CalendarContract.Events._ID}=?",
|
||||
arrayOf(eventId.toString()),
|
||||
null,
|
||||
).use { cursor ->
|
||||
if (cursor == null || !cursor.moveToFirst()) return null
|
||||
return CalendarEventRecord(
|
||||
identifier = cursor.getLong(0).toString(),
|
||||
title = cursor.getString(1)?.trim().orEmpty().ifEmpty { "(untitled)" },
|
||||
startISO = Instant.ofEpochMilli(cursor.getLong(2)).toString(),
|
||||
endISO = Instant.ofEpochMilli(cursor.getLong(3)).toString(),
|
||||
isAllDay = cursor.getInt(4) == 1,
|
||||
location = cursor.getString(5)?.trim()?.ifEmpty { null },
|
||||
calendarTitle = cursor.getString(6)?.trim()?.ifEmpty { null },
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
class CalendarHandler private constructor(
|
||||
private val appContext: Context,
|
||||
private val dataSource: CalendarDataSource,
|
||||
) {
|
||||
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemCalendarDataSource)
|
||||
|
||||
fun handleCalendarEvents(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!dataSource.hasReadPermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "CALENDAR_PERMISSION_REQUIRED",
|
||||
message = "CALENDAR_PERMISSION_REQUIRED: grant Calendar permission",
|
||||
)
|
||||
}
|
||||
val request =
|
||||
parseEventsRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
return try {
|
||||
val events = dataSource.events(appContext, request)
|
||||
GatewaySession.InvokeResult.ok(
|
||||
buildJsonObject {
|
||||
put(
|
||||
"events",
|
||||
buildJsonArray { events.forEach { add(eventJson(it)) } },
|
||||
)
|
||||
}.toString(),
|
||||
)
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "CALENDAR_UNAVAILABLE",
|
||||
message = "CALENDAR_UNAVAILABLE: ${err.message ?: "calendar query failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
fun handleCalendarAdd(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!dataSource.hasWritePermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "CALENDAR_PERMISSION_REQUIRED",
|
||||
message = "CALENDAR_PERMISSION_REQUIRED: grant Calendar permission",
|
||||
)
|
||||
}
|
||||
val request =
|
||||
parseAddRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
if (request.title.isEmpty()) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "CALENDAR_INVALID",
|
||||
message = "CALENDAR_INVALID: title required",
|
||||
)
|
||||
}
|
||||
if (request.endMs <= request.startMs) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "CALENDAR_INVALID",
|
||||
message = "CALENDAR_INVALID: endISO must be after startISO",
|
||||
)
|
||||
}
|
||||
return try {
|
||||
val event = dataSource.add(appContext, request)
|
||||
GatewaySession.InvokeResult.ok(
|
||||
buildJsonObject {
|
||||
put("event", eventJson(event))
|
||||
}.toString(),
|
||||
)
|
||||
} catch (err: IllegalArgumentException) {
|
||||
val msg = err.message ?: "CALENDAR_INVALID: invalid request"
|
||||
val code = if (msg.startsWith("CALENDAR_NOT_FOUND")) "CALENDAR_NOT_FOUND" else "CALENDAR_INVALID"
|
||||
GatewaySession.InvokeResult.error(code = code, message = msg)
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "CALENDAR_UNAVAILABLE",
|
||||
message = "CALENDAR_UNAVAILABLE: ${err.message ?: "calendar add failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseEventsRequest(paramsJson: String?): CalendarEventsRequest? {
|
||||
if (paramsJson.isNullOrBlank()) {
|
||||
val start = Instant.now()
|
||||
val end = start.plus(7, ChronoUnit.DAYS)
|
||||
return CalendarEventsRequest(startMs = start.toEpochMilli(), endMs = end.toEpochMilli(), limit = DEFAULT_CALENDAR_LIMIT)
|
||||
}
|
||||
val params =
|
||||
try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
val start = parseISO((params["startISO"] as? JsonPrimitive)?.content)
|
||||
val end = parseISO((params["endISO"] as? JsonPrimitive)?.content)
|
||||
val resolvedStart = start ?: Instant.now()
|
||||
val resolvedEnd = end ?: resolvedStart.plus(7, ChronoUnit.DAYS)
|
||||
val limit = ((params["limit"] as? JsonPrimitive)?.content?.toIntOrNull() ?: DEFAULT_CALENDAR_LIMIT).coerceIn(1, 500)
|
||||
return CalendarEventsRequest(
|
||||
startMs = resolvedStart.toEpochMilli(),
|
||||
endMs = resolvedEnd.toEpochMilli(),
|
||||
limit = limit,
|
||||
)
|
||||
}
|
||||
|
||||
private fun parseAddRequest(paramsJson: String?): CalendarAddRequest? {
|
||||
val params =
|
||||
try {
|
||||
paramsJson?.let { Json.parseToJsonElement(it).asObjectOrNull() }
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
val start = parseISO((params["startISO"] as? JsonPrimitive)?.content)
|
||||
?: return null
|
||||
val end = parseISO((params["endISO"] as? JsonPrimitive)?.content)
|
||||
?: return null
|
||||
return CalendarAddRequest(
|
||||
title = (params["title"] as? JsonPrimitive)?.content?.trim().orEmpty(),
|
||||
startMs = start.toEpochMilli(),
|
||||
endMs = end.toEpochMilli(),
|
||||
isAllDay = (params["isAllDay"] as? JsonPrimitive)?.content?.toBooleanStrictOrNull() ?: false,
|
||||
location = (params["location"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
notes = (params["notes"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
calendarId = (params["calendarId"] as? JsonPrimitive)?.content?.toLongOrNull(),
|
||||
calendarTitle = (params["calendarTitle"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
)
|
||||
}
|
||||
|
||||
private fun parseISO(raw: String?): Instant? {
|
||||
val value = raw?.trim().orEmpty()
|
||||
if (value.isEmpty()) return null
|
||||
return try {
|
||||
Instant.parse(value)
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun eventJson(event: CalendarEventRecord): JsonObject {
|
||||
return buildJsonObject {
|
||||
put("identifier", JsonPrimitive(event.identifier))
|
||||
put("title", JsonPrimitive(event.title))
|
||||
put("startISO", JsonPrimitive(event.startISO))
|
||||
put("endISO", JsonPrimitive(event.endISO))
|
||||
put("isAllDay", JsonPrimitive(event.isAllDay))
|
||||
event.location?.let { put("location", JsonPrimitive(it)) }
|
||||
event.calendarTitle?.let { put("calendarTitle", JsonPrimitive(it)) }
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
internal fun forTesting(
|
||||
appContext: Context,
|
||||
dataSource: CalendarDataSource,
|
||||
): CalendarHandler = CalendarHandler(appContext = appContext, dataSource = dataSource)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,247 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.content.Context
|
||||
import android.provider.CallLog
|
||||
import androidx.core.content.ContextCompat
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.buildJsonArray
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
private const val DEFAULT_CALL_LOG_LIMIT = 25
|
||||
|
||||
internal data class CallLogRecord(
|
||||
val number: String?,
|
||||
val cachedName: String?,
|
||||
val date: Long,
|
||||
val duration: Long,
|
||||
val type: Int,
|
||||
)
|
||||
|
||||
internal data class CallLogSearchRequest(
|
||||
val limit: Int, // Number of records to return
|
||||
val offset: Int, // Offset value
|
||||
val cachedName: String?, // Search by contact name
|
||||
val number: String?, // Search by phone number
|
||||
val date: Long?, // Search by time (timestamp, deprecated, use dateStart/dateEnd)
|
||||
val dateStart: Long?, // Query start time (timestamp)
|
||||
val dateEnd: Long?, // Query end time (timestamp)
|
||||
val duration: Long?, // Search by duration (seconds)
|
||||
val type: Int?, // Search by call log type
|
||||
)
|
||||
|
||||
internal interface CallLogDataSource {
|
||||
fun hasReadPermission(context: Context): Boolean
|
||||
|
||||
fun search(context: Context, request: CallLogSearchRequest): List<CallLogRecord>
|
||||
}
|
||||
|
||||
private object SystemCallLogDataSource : CallLogDataSource {
|
||||
override fun hasReadPermission(context: Context): Boolean {
|
||||
return ContextCompat.checkSelfPermission(
|
||||
context,
|
||||
Manifest.permission.READ_CALL_LOG
|
||||
) == android.content.pm.PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
|
||||
override fun search(context: Context, request: CallLogSearchRequest): List<CallLogRecord> {
|
||||
val resolver = context.contentResolver
|
||||
val projection = arrayOf(
|
||||
CallLog.Calls.NUMBER,
|
||||
CallLog.Calls.CACHED_NAME,
|
||||
CallLog.Calls.DATE,
|
||||
CallLog.Calls.DURATION,
|
||||
CallLog.Calls.TYPE,
|
||||
)
|
||||
|
||||
// Build selection and selectionArgs for filtering
|
||||
val selections = mutableListOf<String>()
|
||||
val selectionArgs = mutableListOf<String>()
|
||||
|
||||
request.cachedName?.let {
|
||||
selections.add("${CallLog.Calls.CACHED_NAME} LIKE ?")
|
||||
selectionArgs.add("%$it%")
|
||||
}
|
||||
|
||||
request.number?.let {
|
||||
selections.add("${CallLog.Calls.NUMBER} LIKE ?")
|
||||
selectionArgs.add("%$it%")
|
||||
}
|
||||
|
||||
// Support time range query
|
||||
if (request.dateStart != null && request.dateEnd != null) {
|
||||
selections.add("${CallLog.Calls.DATE} >= ? AND ${CallLog.Calls.DATE} <= ?")
|
||||
selectionArgs.add(request.dateStart.toString())
|
||||
selectionArgs.add(request.dateEnd.toString())
|
||||
} else if (request.dateStart != null) {
|
||||
selections.add("${CallLog.Calls.DATE} >= ?")
|
||||
selectionArgs.add(request.dateStart.toString())
|
||||
} else if (request.dateEnd != null) {
|
||||
selections.add("${CallLog.Calls.DATE} <= ?")
|
||||
selectionArgs.add(request.dateEnd.toString())
|
||||
} else if (request.date != null) {
|
||||
// Compatible with the old date parameter (exact match)
|
||||
selections.add("${CallLog.Calls.DATE} = ?")
|
||||
selectionArgs.add(request.date.toString())
|
||||
}
|
||||
|
||||
request.duration?.let {
|
||||
selections.add("${CallLog.Calls.DURATION} = ?")
|
||||
selectionArgs.add(it.toString())
|
||||
}
|
||||
|
||||
request.type?.let {
|
||||
selections.add("${CallLog.Calls.TYPE} = ?")
|
||||
selectionArgs.add(it.toString())
|
||||
}
|
||||
|
||||
val selection = if (selections.isNotEmpty()) selections.joinToString(" AND ") else null
|
||||
val selectionArgsArray = if (selectionArgs.isNotEmpty()) selectionArgs.toTypedArray() else null
|
||||
|
||||
val sortOrder = "${CallLog.Calls.DATE} DESC"
|
||||
|
||||
resolver.query(
|
||||
CallLog.Calls.CONTENT_URI,
|
||||
projection,
|
||||
selection,
|
||||
selectionArgsArray,
|
||||
sortOrder,
|
||||
).use { cursor ->
|
||||
if (cursor == null) return emptyList()
|
||||
|
||||
val numberIndex = cursor.getColumnIndex(CallLog.Calls.NUMBER)
|
||||
val cachedNameIndex = cursor.getColumnIndex(CallLog.Calls.CACHED_NAME)
|
||||
val dateIndex = cursor.getColumnIndex(CallLog.Calls.DATE)
|
||||
val durationIndex = cursor.getColumnIndex(CallLog.Calls.DURATION)
|
||||
val typeIndex = cursor.getColumnIndex(CallLog.Calls.TYPE)
|
||||
|
||||
// Skip offset rows
|
||||
if (request.offset > 0 && cursor.moveToPosition(request.offset - 1)) {
|
||||
// Successfully moved to offset position
|
||||
}
|
||||
|
||||
val out = mutableListOf<CallLogRecord>()
|
||||
var count = 0
|
||||
while (cursor.moveToNext() && count < request.limit) {
|
||||
out += CallLogRecord(
|
||||
number = cursor.getString(numberIndex),
|
||||
cachedName = cursor.getString(cachedNameIndex),
|
||||
date = cursor.getLong(dateIndex),
|
||||
duration = cursor.getLong(durationIndex),
|
||||
type = cursor.getInt(typeIndex),
|
||||
)
|
||||
count++
|
||||
}
|
||||
return out
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
class CallLogHandler private constructor(
|
||||
private val appContext: Context,
|
||||
private val dataSource: CallLogDataSource,
|
||||
) {
|
||||
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemCallLogDataSource)
|
||||
|
||||
fun handleCallLogSearch(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!dataSource.hasReadPermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "CALL_LOG_PERMISSION_REQUIRED",
|
||||
message = "CALL_LOG_PERMISSION_REQUIRED: grant Call Log permission",
|
||||
)
|
||||
}
|
||||
|
||||
val request = parseSearchRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
|
||||
return try {
|
||||
val callLogs = dataSource.search(appContext, request)
|
||||
GatewaySession.InvokeResult.ok(
|
||||
buildJsonObject {
|
||||
put(
|
||||
"callLogs",
|
||||
buildJsonArray {
|
||||
callLogs.forEach { add(callLogJson(it)) }
|
||||
},
|
||||
)
|
||||
}.toString(),
|
||||
)
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "CALL_LOG_UNAVAILABLE",
|
||||
message = "CALL_LOG_UNAVAILABLE: ${err.message ?: "call log query failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseSearchRequest(paramsJson: String?): CallLogSearchRequest? {
|
||||
if (paramsJson.isNullOrBlank()) {
|
||||
return CallLogSearchRequest(
|
||||
limit = DEFAULT_CALL_LOG_LIMIT,
|
||||
offset = 0,
|
||||
cachedName = null,
|
||||
number = null,
|
||||
date = null,
|
||||
dateStart = null,
|
||||
dateEnd = null,
|
||||
duration = null,
|
||||
type = null,
|
||||
)
|
||||
}
|
||||
|
||||
val params = try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
|
||||
val limit = ((params["limit"] as? JsonPrimitive)?.content?.toIntOrNull() ?: DEFAULT_CALL_LOG_LIMIT)
|
||||
.coerceIn(1, 200)
|
||||
val offset = ((params["offset"] as? JsonPrimitive)?.content?.toIntOrNull() ?: 0)
|
||||
.coerceAtLeast(0)
|
||||
val cachedName = (params["cachedName"] as? JsonPrimitive)?.content?.takeIf { it.isNotBlank() }
|
||||
val number = (params["number"] as? JsonPrimitive)?.content?.takeIf { it.isNotBlank() }
|
||||
val date = (params["date"] as? JsonPrimitive)?.content?.toLongOrNull()
|
||||
val dateStart = (params["dateStart"] as? JsonPrimitive)?.content?.toLongOrNull()
|
||||
val dateEnd = (params["dateEnd"] as? JsonPrimitive)?.content?.toLongOrNull()
|
||||
val duration = (params["duration"] as? JsonPrimitive)?.content?.toLongOrNull()
|
||||
val type = (params["type"] as? JsonPrimitive)?.content?.toIntOrNull()
|
||||
|
||||
return CallLogSearchRequest(
|
||||
limit = limit,
|
||||
offset = offset,
|
||||
cachedName = cachedName,
|
||||
number = number,
|
||||
date = date,
|
||||
dateStart = dateStart,
|
||||
dateEnd = dateEnd,
|
||||
duration = duration,
|
||||
type = type,
|
||||
)
|
||||
}
|
||||
|
||||
private fun callLogJson(callLog: CallLogRecord): JsonObject {
|
||||
return buildJsonObject {
|
||||
put("number", JsonPrimitive(callLog.number))
|
||||
put("cachedName", JsonPrimitive(callLog.cachedName))
|
||||
put("date", JsonPrimitive(callLog.date))
|
||||
put("duration", JsonPrimitive(callLog.duration))
|
||||
put("type", JsonPrimitive(callLog.type))
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
internal fun forTesting(
|
||||
appContext: Context,
|
||||
dataSource: CallLogDataSource,
|
||||
): CallLogHandler = CallLogHandler(appContext = appContext, dataSource = dataSource)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,425 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.annotation.SuppressLint
|
||||
import android.content.Context
|
||||
import android.graphics.Bitmap
|
||||
import android.graphics.BitmapFactory
|
||||
import android.graphics.Matrix
|
||||
import android.content.pm.PackageManager
|
||||
import android.hardware.camera2.CameraCharacteristics
|
||||
import android.util.Base64
|
||||
import androidx.camera.camera2.interop.Camera2CameraInfo
|
||||
import androidx.camera.core.CameraInfo
|
||||
import androidx.exifinterface.media.ExifInterface
|
||||
import androidx.lifecycle.LifecycleOwner
|
||||
import androidx.camera.core.CameraSelector
|
||||
import androidx.camera.core.ImageCapture
|
||||
import androidx.camera.core.ImageCaptureException
|
||||
import androidx.camera.lifecycle.ProcessCameraProvider
|
||||
import androidx.camera.video.FileOutputOptions
|
||||
import androidx.camera.video.FallbackStrategy
|
||||
import androidx.camera.video.Quality
|
||||
import androidx.camera.video.QualitySelector
|
||||
import androidx.camera.video.Recorder
|
||||
import androidx.camera.video.Recording
|
||||
import androidx.camera.video.VideoCapture
|
||||
import androidx.camera.video.VideoRecordEvent
|
||||
import androidx.core.content.ContextCompat
|
||||
import androidx.core.content.ContextCompat.checkSelfPermission
|
||||
import androidx.core.graphics.scale
|
||||
import ai.openclaw.app.PermissionRequester
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
import kotlinx.coroutines.withTimeout
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import java.io.ByteArrayOutputStream
|
||||
import java.io.File
|
||||
import java.util.concurrent.Executor
|
||||
import kotlin.math.roundToInt
|
||||
import kotlin.coroutines.resume
|
||||
import kotlin.coroutines.resumeWithException
|
||||
|
||||
class CameraCaptureManager(private val context: Context) {
|
||||
data class Payload(val payloadJson: String)
|
||||
data class FilePayload(val file: File, val durationMs: Long, val hasAudio: Boolean)
|
||||
data class CameraDeviceInfo(
|
||||
val id: String,
|
||||
val name: String,
|
||||
val position: String,
|
||||
val deviceType: String,
|
||||
)
|
||||
|
||||
@Volatile private var lifecycleOwner: LifecycleOwner? = null
|
||||
@Volatile private var permissionRequester: PermissionRequester? = null
|
||||
|
||||
fun attachLifecycleOwner(owner: LifecycleOwner) {
|
||||
lifecycleOwner = owner
|
||||
}
|
||||
|
||||
fun attachPermissionRequester(requester: PermissionRequester) {
|
||||
permissionRequester = requester
|
||||
}
|
||||
|
||||
suspend fun listDevices(): List<CameraDeviceInfo> =
|
||||
withContext(Dispatchers.Main) {
|
||||
val provider = context.cameraProvider()
|
||||
provider.availableCameraInfos
|
||||
.mapNotNull { info -> cameraDeviceInfoOrNull(info) }
|
||||
.sortedBy { it.id }
|
||||
}
|
||||
|
||||
private suspend fun ensureCameraPermission() {
|
||||
val granted = checkSelfPermission(context, Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED
|
||||
if (granted) return
|
||||
|
||||
val requester = permissionRequester
|
||||
?: throw IllegalStateException("CAMERA_PERMISSION_REQUIRED: grant Camera permission")
|
||||
val results = requester.requestIfMissing(listOf(Manifest.permission.CAMERA))
|
||||
if (results[Manifest.permission.CAMERA] != true) {
|
||||
throw IllegalStateException("CAMERA_PERMISSION_REQUIRED: grant Camera permission")
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun ensureMicPermission() {
|
||||
val granted = checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED
|
||||
if (granted) return
|
||||
|
||||
val requester = permissionRequester
|
||||
?: throw IllegalStateException("MIC_PERMISSION_REQUIRED: grant Microphone permission")
|
||||
val results = requester.requestIfMissing(listOf(Manifest.permission.RECORD_AUDIO))
|
||||
if (results[Manifest.permission.RECORD_AUDIO] != true) {
|
||||
throw IllegalStateException("MIC_PERMISSION_REQUIRED: grant Microphone permission")
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun snap(paramsJson: String?): Payload =
|
||||
withContext(Dispatchers.Main) {
|
||||
ensureCameraPermission()
|
||||
val owner = lifecycleOwner ?: throw IllegalStateException("UNAVAILABLE: camera not ready")
|
||||
val params = parseJsonParamsObject(paramsJson)
|
||||
val facing = parseFacing(params) ?: "front"
|
||||
val quality = (parseQuality(params) ?: 0.95).coerceIn(0.1, 1.0)
|
||||
val maxWidth = parseMaxWidth(params) ?: 1600
|
||||
val deviceId = parseDeviceId(params)
|
||||
|
||||
val provider = context.cameraProvider()
|
||||
val capture = ImageCapture.Builder().build()
|
||||
val selector = resolveCameraSelector(provider, facing, deviceId)
|
||||
|
||||
provider.unbindAll()
|
||||
provider.bindToLifecycle(owner, selector, capture)
|
||||
|
||||
val (bytes, orientation) = capture.takeJpegWithExif(context.mainExecutor())
|
||||
val decoded = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
|
||||
?: throw IllegalStateException("UNAVAILABLE: failed to decode captured image")
|
||||
val rotated = rotateBitmapByExif(decoded, orientation)
|
||||
val scaled =
|
||||
if (maxWidth > 0 && rotated.width > maxWidth) {
|
||||
val h =
|
||||
(rotated.height.toDouble() * (maxWidth.toDouble() / rotated.width.toDouble()))
|
||||
.toInt()
|
||||
.coerceAtLeast(1)
|
||||
val s = rotated.scale(maxWidth, h)
|
||||
if (s !== rotated) rotated.recycle()
|
||||
s
|
||||
} else {
|
||||
rotated
|
||||
}
|
||||
|
||||
try {
|
||||
val maxPayloadBytes = 5 * 1024 * 1024
|
||||
// Base64 inflates payloads by ~4/3; cap encoded bytes so the payload stays under 5MB (API limit).
|
||||
val maxEncodedBytes = (maxPayloadBytes / 4) * 3
|
||||
val result =
|
||||
JpegSizeLimiter.compressToLimit(
|
||||
initialWidth = scaled.width,
|
||||
initialHeight = scaled.height,
|
||||
startQuality = (quality * 100.0).roundToInt().coerceIn(10, 100),
|
||||
maxBytes = maxEncodedBytes,
|
||||
encode = { width, height, q ->
|
||||
val bitmap =
|
||||
if (width == scaled.width && height == scaled.height) {
|
||||
scaled
|
||||
} else {
|
||||
scaled.scale(width, height)
|
||||
}
|
||||
val out = ByteArrayOutputStream()
|
||||
if (!bitmap.compress(Bitmap.CompressFormat.JPEG, q, out)) {
|
||||
if (bitmap !== scaled) bitmap.recycle()
|
||||
throw IllegalStateException("UNAVAILABLE: failed to encode JPEG")
|
||||
}
|
||||
if (bitmap !== scaled) {
|
||||
bitmap.recycle()
|
||||
}
|
||||
out.toByteArray()
|
||||
},
|
||||
)
|
||||
val base64 = Base64.encodeToString(result.bytes, Base64.NO_WRAP)
|
||||
Payload(
|
||||
"""{"format":"jpg","base64":"$base64","width":${result.width},"height":${result.height}}""",
|
||||
)
|
||||
} finally {
|
||||
scaled.recycle()
|
||||
}
|
||||
}
|
||||
|
||||
@SuppressLint("MissingPermission")
|
||||
suspend fun clip(paramsJson: String?): FilePayload =
|
||||
withContext(Dispatchers.Main) {
|
||||
ensureCameraPermission()
|
||||
val owner = lifecycleOwner ?: throw IllegalStateException("UNAVAILABLE: camera not ready")
|
||||
val params = parseJsonParamsObject(paramsJson)
|
||||
val facing = parseFacing(params) ?: "front"
|
||||
val durationMs = (parseDurationMs(params) ?: 3_000).coerceIn(200, 60_000)
|
||||
val includeAudio = parseIncludeAudio(params) ?: true
|
||||
val deviceId = parseDeviceId(params)
|
||||
if (includeAudio) ensureMicPermission()
|
||||
|
||||
android.util.Log.w("CameraCaptureManager", "clip: start facing=$facing duration=$durationMs audio=$includeAudio deviceId=${deviceId ?: "-"}")
|
||||
|
||||
val provider = context.cameraProvider()
|
||||
android.util.Log.w("CameraCaptureManager", "clip: got camera provider")
|
||||
|
||||
// Use LOWEST quality for smallest files over WebSocket
|
||||
val recorder = Recorder.Builder()
|
||||
.setQualitySelector(
|
||||
QualitySelector.from(Quality.LOWEST, FallbackStrategy.lowerQualityOrHigherThan(Quality.LOWEST))
|
||||
)
|
||||
.build()
|
||||
val videoCapture = VideoCapture.withOutput(recorder)
|
||||
val selector = resolveCameraSelector(provider, facing, deviceId)
|
||||
|
||||
// CameraX requires a Preview use case for the camera to start producing frames;
|
||||
// without it, the encoder may get no data (ERROR_NO_VALID_DATA).
|
||||
val preview = androidx.camera.core.Preview.Builder().build()
|
||||
// Provide a dummy SurfaceTexture so the preview pipeline activates
|
||||
val surfaceTexture = android.graphics.SurfaceTexture(0)
|
||||
surfaceTexture.setDefaultBufferSize(640, 480)
|
||||
preview.setSurfaceProvider { request ->
|
||||
val surface = android.view.Surface(surfaceTexture)
|
||||
request.provideSurface(surface, context.mainExecutor()) { result ->
|
||||
surface.release()
|
||||
surfaceTexture.release()
|
||||
}
|
||||
}
|
||||
|
||||
provider.unbindAll()
|
||||
android.util.Log.w("CameraCaptureManager", "clip: binding preview + videoCapture to lifecycle")
|
||||
val camera = provider.bindToLifecycle(owner, selector, preview, videoCapture)
|
||||
android.util.Log.w("CameraCaptureManager", "clip: bound, cameraInfo=${camera.cameraInfo}")
|
||||
|
||||
// Give camera pipeline time to initialize before recording
|
||||
android.util.Log.w("CameraCaptureManager", "clip: warming up camera 1.5s...")
|
||||
kotlinx.coroutines.delay(1_500)
|
||||
|
||||
val file = File.createTempFile("openclaw-clip-", ".mp4")
|
||||
val outputOptions = FileOutputOptions.Builder(file).build()
|
||||
|
||||
val finalized = kotlinx.coroutines.CompletableDeferred<VideoRecordEvent.Finalize>()
|
||||
android.util.Log.w("CameraCaptureManager", "clip: starting recording to ${file.absolutePath}")
|
||||
val recording: Recording =
|
||||
videoCapture.output
|
||||
.prepareRecording(context, outputOptions)
|
||||
.apply {
|
||||
if (includeAudio) withAudioEnabled()
|
||||
}
|
||||
.start(context.mainExecutor()) { event ->
|
||||
android.util.Log.w("CameraCaptureManager", "clip: event ${event.javaClass.simpleName}")
|
||||
if (event is VideoRecordEvent.Status) {
|
||||
android.util.Log.w("CameraCaptureManager", "clip: recording status update")
|
||||
}
|
||||
if (event is VideoRecordEvent.Finalize) {
|
||||
android.util.Log.w("CameraCaptureManager", "clip: finalize hasError=${event.hasError()} error=${event.error} cause=${event.cause}")
|
||||
finalized.complete(event)
|
||||
}
|
||||
}
|
||||
|
||||
android.util.Log.w("CameraCaptureManager", "clip: recording started, delaying ${durationMs}ms")
|
||||
try {
|
||||
kotlinx.coroutines.delay(durationMs.toLong())
|
||||
} finally {
|
||||
android.util.Log.w("CameraCaptureManager", "clip: stopping recording")
|
||||
recording.stop()
|
||||
}
|
||||
|
||||
val finalizeEvent =
|
||||
try {
|
||||
withTimeout(15_000) { finalized.await() }
|
||||
} catch (err: Throwable) {
|
||||
android.util.Log.e("CameraCaptureManager", "clip: finalize timed out", err)
|
||||
withContext(Dispatchers.IO) { file.delete() }
|
||||
provider.unbindAll()
|
||||
throw IllegalStateException("UNAVAILABLE: camera clip finalize timed out")
|
||||
}
|
||||
if (finalizeEvent.hasError()) {
|
||||
android.util.Log.e("CameraCaptureManager", "clip: FAILED error=${finalizeEvent.error}, cause=${finalizeEvent.cause}", finalizeEvent.cause)
|
||||
// Check file size for debugging
|
||||
val fileSize = withContext(Dispatchers.IO) { if (file.exists()) file.length() else -1 }
|
||||
android.util.Log.e("CameraCaptureManager", "clip: file exists=${file.exists()} size=$fileSize")
|
||||
withContext(Dispatchers.IO) { file.delete() }
|
||||
provider.unbindAll()
|
||||
throw IllegalStateException("UNAVAILABLE: camera clip failed (error=${finalizeEvent.error})")
|
||||
}
|
||||
|
||||
val fileSize = withContext(Dispatchers.IO) { file.length() }
|
||||
android.util.Log.w("CameraCaptureManager", "clip: SUCCESS file size=$fileSize")
|
||||
|
||||
provider.unbindAll()
|
||||
|
||||
FilePayload(file = file, durationMs = durationMs.toLong(), hasAudio = includeAudio)
|
||||
}
|
||||
|
||||
private fun rotateBitmapByExif(bitmap: Bitmap, orientation: Int): Bitmap {
|
||||
val matrix = Matrix()
|
||||
when (orientation) {
|
||||
ExifInterface.ORIENTATION_ROTATE_90 -> matrix.postRotate(90f)
|
||||
ExifInterface.ORIENTATION_ROTATE_180 -> matrix.postRotate(180f)
|
||||
ExifInterface.ORIENTATION_ROTATE_270 -> matrix.postRotate(270f)
|
||||
ExifInterface.ORIENTATION_FLIP_HORIZONTAL -> matrix.postScale(-1f, 1f)
|
||||
ExifInterface.ORIENTATION_FLIP_VERTICAL -> matrix.postScale(1f, -1f)
|
||||
ExifInterface.ORIENTATION_TRANSPOSE -> {
|
||||
matrix.postRotate(90f)
|
||||
matrix.postScale(-1f, 1f)
|
||||
}
|
||||
ExifInterface.ORIENTATION_TRANSVERSE -> {
|
||||
matrix.postRotate(-90f)
|
||||
matrix.postScale(-1f, 1f)
|
||||
}
|
||||
else -> return bitmap
|
||||
}
|
||||
val rotated = Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, true)
|
||||
if (rotated !== bitmap) {
|
||||
bitmap.recycle()
|
||||
}
|
||||
return rotated
|
||||
}
|
||||
|
||||
private fun parseFacing(params: JsonObject?): String? {
|
||||
val value = parseJsonString(params, "facing")?.trim()?.lowercase() ?: return null
|
||||
return when (value) {
|
||||
"front", "back" -> value
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseQuality(params: JsonObject?): Double? =
|
||||
parseJsonDouble(params, "quality")
|
||||
|
||||
private fun parseMaxWidth(params: JsonObject?): Int? =
|
||||
parseJsonInt(params, "maxWidth")
|
||||
?.takeIf { it > 0 }
|
||||
|
||||
private fun parseDurationMs(params: JsonObject?): Int? =
|
||||
parseJsonInt(params, "durationMs")
|
||||
|
||||
private fun parseDeviceId(params: JsonObject?): String? =
|
||||
parseJsonString(params, "deviceId")
|
||||
?.trim()
|
||||
?.takeIf { it.isNotEmpty() }
|
||||
|
||||
private fun parseIncludeAudio(params: JsonObject?): Boolean? = parseJsonBooleanFlag(params, "includeAudio")
|
||||
|
||||
private fun Context.mainExecutor(): Executor = ContextCompat.getMainExecutor(this)
|
||||
|
||||
private fun resolveCameraSelector(
|
||||
provider: ProcessCameraProvider,
|
||||
facing: String,
|
||||
deviceId: String?,
|
||||
): CameraSelector {
|
||||
if (deviceId.isNullOrEmpty()) {
|
||||
return if (facing == "front") CameraSelector.DEFAULT_FRONT_CAMERA else CameraSelector.DEFAULT_BACK_CAMERA
|
||||
}
|
||||
val availableIds = provider.availableCameraInfos.mapNotNull { cameraIdOrNull(it) }.toSet()
|
||||
if (!availableIds.contains(deviceId)) {
|
||||
throw IllegalStateException("INVALID_REQUEST: unknown camera deviceId '$deviceId'")
|
||||
}
|
||||
return CameraSelector.Builder()
|
||||
.addCameraFilter { infos -> infos.filter { cameraIdOrNull(it) == deviceId } }
|
||||
.build()
|
||||
}
|
||||
|
||||
@SuppressLint("UnsafeOptInUsageError")
|
||||
private fun cameraDeviceInfoOrNull(info: CameraInfo): CameraDeviceInfo? {
|
||||
val cameraId = cameraIdOrNull(info) ?: return null
|
||||
val lensFacing =
|
||||
runCatching {
|
||||
Camera2CameraInfo.from(info).getCameraCharacteristic(CameraCharacteristics.LENS_FACING)
|
||||
}.getOrNull()
|
||||
val position =
|
||||
when (lensFacing) {
|
||||
CameraCharacteristics.LENS_FACING_FRONT -> "front"
|
||||
CameraCharacteristics.LENS_FACING_BACK -> "back"
|
||||
CameraCharacteristics.LENS_FACING_EXTERNAL -> "external"
|
||||
else -> "unspecified"
|
||||
}
|
||||
val deviceType =
|
||||
if (lensFacing == CameraCharacteristics.LENS_FACING_EXTERNAL) "external" else "builtIn"
|
||||
val name =
|
||||
when (position) {
|
||||
"front" -> "Front Camera"
|
||||
"back" -> "Back Camera"
|
||||
"external" -> "External Camera"
|
||||
else -> "Camera $cameraId"
|
||||
}
|
||||
return CameraDeviceInfo(
|
||||
id = cameraId,
|
||||
name = name,
|
||||
position = position,
|
||||
deviceType = deviceType,
|
||||
)
|
||||
}
|
||||
|
||||
@SuppressLint("UnsafeOptInUsageError")
|
||||
private fun cameraIdOrNull(info: CameraInfo): String? =
|
||||
runCatching { Camera2CameraInfo.from(info).cameraId }.getOrNull()
|
||||
}
|
||||
|
||||
private suspend fun Context.cameraProvider(): ProcessCameraProvider =
|
||||
suspendCancellableCoroutine { cont ->
|
||||
val future = ProcessCameraProvider.getInstance(this)
|
||||
future.addListener(
|
||||
{
|
||||
try {
|
||||
cont.resume(future.get())
|
||||
} catch (e: Exception) {
|
||||
cont.resumeWithException(e)
|
||||
}
|
||||
},
|
||||
ContextCompat.getMainExecutor(this),
|
||||
)
|
||||
}
|
||||
|
||||
/** Returns (jpegBytes, exifOrientation) so caller can rotate the decoded bitmap. */
|
||||
private suspend fun ImageCapture.takeJpegWithExif(executor: Executor): Pair<ByteArray, Int> =
|
||||
suspendCancellableCoroutine { cont ->
|
||||
val file = File.createTempFile("openclaw-snap-", ".jpg")
|
||||
val options = ImageCapture.OutputFileOptions.Builder(file).build()
|
||||
takePicture(
|
||||
options,
|
||||
executor,
|
||||
object : ImageCapture.OnImageSavedCallback {
|
||||
override fun onError(exception: ImageCaptureException) {
|
||||
file.delete()
|
||||
cont.resumeWithException(exception)
|
||||
}
|
||||
|
||||
override fun onImageSaved(outputFileResults: ImageCapture.OutputFileResults) {
|
||||
try {
|
||||
val exif = ExifInterface(file.absolutePath)
|
||||
val orientation = exif.getAttributeInt(
|
||||
ExifInterface.TAG_ORIENTATION,
|
||||
ExifInterface.ORIENTATION_NORMAL,
|
||||
)
|
||||
val bytes = file.readBytes()
|
||||
cont.resume(Pair(bytes, orientation))
|
||||
} catch (e: Exception) {
|
||||
cont.resumeWithException(e)
|
||||
} finally {
|
||||
file.delete()
|
||||
}
|
||||
}
|
||||
},
|
||||
)
|
||||
}
|
||||
|
|
@ -0,0 +1,177 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.content.Context
|
||||
import ai.openclaw.app.CameraHudKind
|
||||
import ai.openclaw.app.BuildConfig
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonArray
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.contentOrNull
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
internal const val CAMERA_CLIP_MAX_RAW_BYTES: Long = 18L * 1024L * 1024L
|
||||
|
||||
internal fun isCameraClipWithinPayloadLimit(rawBytes: Long): Boolean =
|
||||
rawBytes in 0L..CAMERA_CLIP_MAX_RAW_BYTES
|
||||
|
||||
class CameraHandler(
|
||||
private val appContext: Context,
|
||||
private val camera: CameraCaptureManager,
|
||||
private val externalAudioCaptureActive: MutableStateFlow<Boolean>,
|
||||
private val showCameraHud: (message: String, kind: CameraHudKind, autoHideMs: Long?) -> Unit,
|
||||
private val triggerCameraFlash: () -> Unit,
|
||||
private val invokeErrorFromThrowable: (err: Throwable) -> Pair<String, String>,
|
||||
) {
|
||||
suspend fun handleList(_paramsJson: String?): GatewaySession.InvokeResult {
|
||||
return try {
|
||||
val devices = camera.listDevices()
|
||||
val payload =
|
||||
buildJsonObject {
|
||||
put(
|
||||
"devices",
|
||||
buildJsonArray {
|
||||
devices.forEach { device ->
|
||||
add(
|
||||
buildJsonObject {
|
||||
put("id", JsonPrimitive(device.id))
|
||||
put("name", JsonPrimitive(device.name))
|
||||
put("position", JsonPrimitive(device.position))
|
||||
put("deviceType", JsonPrimitive(device.deviceType))
|
||||
},
|
||||
)
|
||||
}
|
||||
},
|
||||
)
|
||||
}.toString()
|
||||
GatewaySession.InvokeResult.ok(payload)
|
||||
} catch (err: Throwable) {
|
||||
val (code, message) = invokeErrorFromThrowable(err)
|
||||
GatewaySession.InvokeResult.error(code = code, message = message)
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun handleSnap(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
val logFile = if (BuildConfig.DEBUG) java.io.File(appContext.cacheDir, "camera_debug.log") else null
|
||||
fun camLog(msg: String) {
|
||||
if (!BuildConfig.DEBUG) return
|
||||
val ts = java.text.SimpleDateFormat("HH:mm:ss.SSS", java.util.Locale.US).format(java.util.Date())
|
||||
logFile?.appendText("[$ts] $msg\n")
|
||||
android.util.Log.w("openclaw", "camera.snap: $msg")
|
||||
}
|
||||
try {
|
||||
logFile?.writeText("") // clear
|
||||
camLog("starting, params=$paramsJson")
|
||||
camLog("calling showCameraHud")
|
||||
showCameraHud("Taking photo…", CameraHudKind.Photo, null)
|
||||
camLog("calling triggerCameraFlash")
|
||||
triggerCameraFlash()
|
||||
val res =
|
||||
try {
|
||||
camLog("calling camera.snap()")
|
||||
val r = camera.snap(paramsJson)
|
||||
camLog("success, payload size=${r.payloadJson.length}")
|
||||
r
|
||||
} catch (err: Throwable) {
|
||||
camLog("inner error: ${err::class.java.simpleName}: ${err.message}")
|
||||
camLog("stack: ${err.stackTraceToString().take(2000)}")
|
||||
val (code, message) = invokeErrorFromThrowable(err)
|
||||
showCameraHud(message, CameraHudKind.Error, 2200)
|
||||
return GatewaySession.InvokeResult.error(code = code, message = message)
|
||||
}
|
||||
camLog("returning result")
|
||||
showCameraHud("Photo captured", CameraHudKind.Success, 1600)
|
||||
return GatewaySession.InvokeResult.ok(res.payloadJson)
|
||||
} catch (err: Throwable) {
|
||||
camLog("outer error: ${err::class.java.simpleName}: ${err.message}")
|
||||
camLog("stack: ${err.stackTraceToString().take(2000)}")
|
||||
return GatewaySession.InvokeResult.error(code = "UNAVAILABLE", message = err.message ?: "camera snap failed")
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun handleClip(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
val clipLogFile = if (BuildConfig.DEBUG) java.io.File(appContext.cacheDir, "camera_debug.log") else null
|
||||
fun clipLog(msg: String) {
|
||||
if (!BuildConfig.DEBUG) return
|
||||
val ts = java.text.SimpleDateFormat("HH:mm:ss.SSS", java.util.Locale.US).format(java.util.Date())
|
||||
clipLogFile?.appendText("[CLIP $ts] $msg\n")
|
||||
android.util.Log.w("openclaw", "camera.clip: $msg")
|
||||
}
|
||||
val includeAudio = parseIncludeAudio(paramsJson) ?: true
|
||||
if (includeAudio) externalAudioCaptureActive.value = true
|
||||
try {
|
||||
clipLogFile?.writeText("") // clear
|
||||
clipLog("starting, params=$paramsJson includeAudio=$includeAudio")
|
||||
clipLog("calling showCameraHud")
|
||||
showCameraHud("Recording…", CameraHudKind.Recording, null)
|
||||
val filePayload =
|
||||
try {
|
||||
clipLog("calling camera.clip()")
|
||||
val r = camera.clip(paramsJson)
|
||||
clipLog("success, file size=${r.file.length()}")
|
||||
r
|
||||
} catch (err: Throwable) {
|
||||
clipLog("inner error: ${err::class.java.simpleName}: ${err.message}")
|
||||
clipLog("stack: ${err.stackTraceToString().take(2000)}")
|
||||
val (code, message) = invokeErrorFromThrowable(err)
|
||||
showCameraHud(message, CameraHudKind.Error, 2400)
|
||||
return GatewaySession.InvokeResult.error(code = code, message = message)
|
||||
}
|
||||
val rawBytes = filePayload.file.length()
|
||||
if (!isCameraClipWithinPayloadLimit(rawBytes)) {
|
||||
clipLog("payload too large: bytes=$rawBytes max=$CAMERA_CLIP_MAX_RAW_BYTES")
|
||||
withContext(Dispatchers.IO) { filePayload.file.delete() }
|
||||
showCameraHud("Clip too large", CameraHudKind.Error, 2400)
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "PAYLOAD_TOO_LARGE",
|
||||
message =
|
||||
"PAYLOAD_TOO_LARGE: camera clip is $rawBytes bytes; max is $CAMERA_CLIP_MAX_RAW_BYTES bytes. Reduce durationMs and retry.",
|
||||
)
|
||||
}
|
||||
|
||||
val bytes = withContext(Dispatchers.IO) {
|
||||
try {
|
||||
filePayload.file.readBytes()
|
||||
} finally {
|
||||
filePayload.file.delete()
|
||||
}
|
||||
}
|
||||
val base64 = android.util.Base64.encodeToString(bytes, android.util.Base64.NO_WRAP)
|
||||
clipLog("returning base64 payload")
|
||||
showCameraHud("Clip captured", CameraHudKind.Success, 1800)
|
||||
return GatewaySession.InvokeResult.ok(
|
||||
"""{"format":"mp4","base64":"$base64","durationMs":${filePayload.durationMs},"hasAudio":${filePayload.hasAudio}}"""
|
||||
)
|
||||
} catch (err: Throwable) {
|
||||
clipLog("outer error: ${err::class.java.simpleName}: ${err.message}")
|
||||
clipLog("stack: ${err.stackTraceToString().take(2000)}")
|
||||
return GatewaySession.InvokeResult.error(code = "UNAVAILABLE", message = err.message ?: "camera clip failed")
|
||||
} finally {
|
||||
if (includeAudio) externalAudioCaptureActive.value = false
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseIncludeAudio(paramsJson: String?): Boolean? {
|
||||
if (paramsJson.isNullOrBlank()) return null
|
||||
val root =
|
||||
try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
val value =
|
||||
(root["includeAudio"] as? JsonPrimitive)
|
||||
?.contentOrNull
|
||||
?.trim()
|
||||
?.lowercase()
|
||||
return when (value) {
|
||||
"true" -> true
|
||||
"false" -> false
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import java.net.URI
|
||||
|
||||
object CanvasActionTrust {
|
||||
const val scaffoldAssetUrl: String = "file:///android_asset/CanvasScaffold/scaffold.html"
|
||||
|
||||
fun isTrustedCanvasActionUrl(rawUrl: String?, trustedA2uiUrls: List<String>): Boolean {
|
||||
val candidate = rawUrl?.trim().orEmpty()
|
||||
if (candidate.isEmpty()) return false
|
||||
if (candidate == scaffoldAssetUrl) return true
|
||||
|
||||
val candidateUri = parseUri(candidate) ?: return false
|
||||
if (candidateUri.scheme.equals("file", ignoreCase = true)) {
|
||||
return false
|
||||
}
|
||||
val normalizedCandidate = normalizeTrustedRemoteA2uiUri(candidateUri) ?: return false
|
||||
|
||||
return trustedA2uiUrls.any { trusted ->
|
||||
matchesTrustedRemoteA2uiUrlExact(normalizedCandidate, trusted)
|
||||
}
|
||||
}
|
||||
|
||||
private fun matchesTrustedRemoteA2uiUrlExact(candidateUri: URI, trustedUrl: String): Boolean {
|
||||
val trustedUri = parseUri(trustedUrl) ?: return false
|
||||
val normalizedTrusted = normalizeTrustedRemoteA2uiUri(trustedUri) ?: return false
|
||||
return candidateUri == normalizedTrusted
|
||||
}
|
||||
|
||||
private fun normalizeTrustedRemoteA2uiUri(uri: URI): URI? {
|
||||
// Keep Android trust normalization aligned with iOS ScreenController:
|
||||
// exact remote URL match, scheme/host normalized, fragment ignored.
|
||||
val scheme = uri.scheme?.lowercase() ?: return null
|
||||
if (scheme != "http" && scheme != "https") return null
|
||||
|
||||
val host = uri.host?.trim()?.takeIf { it.isNotEmpty() }?.lowercase() ?: return null
|
||||
|
||||
return try {
|
||||
URI(scheme, uri.userInfo, host, uri.port, uri.rawPath, uri.rawQuery, null)
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseUri(raw: String): URI? =
|
||||
try {
|
||||
URI(raw)
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,310 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.graphics.Bitmap
|
||||
import android.graphics.Canvas
|
||||
import android.os.Looper
|
||||
import android.util.Log
|
||||
import android.webkit.WebView
|
||||
import androidx.core.graphics.createBitmap
|
||||
import androidx.core.graphics.scale
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
import java.io.ByteArrayOutputStream
|
||||
import android.util.Base64
|
||||
import org.json.JSONObject
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonElement
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import ai.openclaw.app.BuildConfig
|
||||
import kotlin.coroutines.resume
|
||||
|
||||
class CanvasController {
|
||||
enum class SnapshotFormat(val rawValue: String) {
|
||||
Png("png"),
|
||||
Jpeg("jpeg"),
|
||||
}
|
||||
|
||||
@Volatile private var webView: WebView? = null
|
||||
@Volatile private var url: String? = null
|
||||
@Volatile private var debugStatusEnabled: Boolean = false
|
||||
@Volatile private var debugStatusTitle: String? = null
|
||||
@Volatile private var debugStatusSubtitle: String? = null
|
||||
@Volatile private var homeCanvasStateJson: String? = null
|
||||
private val _currentUrl = MutableStateFlow<String?>(null)
|
||||
val currentUrl: StateFlow<String?> = _currentUrl.asStateFlow()
|
||||
|
||||
private val scaffoldAssetUrl = "file:///android_asset/CanvasScaffold/scaffold.html"
|
||||
|
||||
private fun clampJpegQuality(quality: Double?): Int {
|
||||
val q = (quality ?: 0.82).coerceIn(0.1, 1.0)
|
||||
return (q * 100.0).toInt().coerceIn(1, 100)
|
||||
}
|
||||
|
||||
private fun Bitmap.scaleForMaxWidth(maxWidth: Int?): Bitmap {
|
||||
if (maxWidth == null || maxWidth <= 0 || width <= maxWidth) {
|
||||
return this
|
||||
}
|
||||
val scaledHeight = (height.toDouble() * (maxWidth.toDouble() / width.toDouble())).toInt().coerceAtLeast(1)
|
||||
return scale(maxWidth, scaledHeight)
|
||||
}
|
||||
|
||||
fun attach(webView: WebView) {
|
||||
this.webView = webView
|
||||
reload()
|
||||
applyDebugStatus()
|
||||
applyHomeCanvasState()
|
||||
}
|
||||
|
||||
fun detach(webView: WebView) {
|
||||
if (this.webView === webView) {
|
||||
this.webView = null
|
||||
}
|
||||
}
|
||||
|
||||
fun navigate(url: String) {
|
||||
val trimmed = url.trim()
|
||||
this.url = if (trimmed.isBlank() || trimmed == "/") null else trimmed
|
||||
_currentUrl.value = this.url
|
||||
reload()
|
||||
}
|
||||
|
||||
fun currentUrl(): String? = url
|
||||
|
||||
fun isDefaultCanvas(): Boolean = url == null
|
||||
|
||||
fun setDebugStatusEnabled(enabled: Boolean) {
|
||||
debugStatusEnabled = enabled
|
||||
applyDebugStatus()
|
||||
}
|
||||
|
||||
fun setDebugStatus(title: String?, subtitle: String?) {
|
||||
debugStatusTitle = title
|
||||
debugStatusSubtitle = subtitle
|
||||
applyDebugStatus()
|
||||
}
|
||||
|
||||
fun onPageFinished() {
|
||||
applyDebugStatus()
|
||||
applyHomeCanvasState()
|
||||
}
|
||||
|
||||
fun updateHomeCanvasState(json: String?) {
|
||||
homeCanvasStateJson = json
|
||||
applyHomeCanvasState()
|
||||
}
|
||||
|
||||
private inline fun withWebViewOnMain(crossinline block: (WebView) -> Unit) {
|
||||
val wv = webView ?: return
|
||||
if (Looper.myLooper() == Looper.getMainLooper()) {
|
||||
block(wv)
|
||||
} else {
|
||||
wv.post { block(wv) }
|
||||
}
|
||||
}
|
||||
|
||||
private fun reload() {
|
||||
val currentUrl = url
|
||||
withWebViewOnMain { wv ->
|
||||
if (currentUrl == null) {
|
||||
if (BuildConfig.DEBUG) {
|
||||
Log.d("OpenClawCanvas", "load scaffold: $scaffoldAssetUrl")
|
||||
}
|
||||
wv.loadUrl(scaffoldAssetUrl)
|
||||
} else {
|
||||
if (BuildConfig.DEBUG) {
|
||||
Log.d("OpenClawCanvas", "load url: $currentUrl")
|
||||
}
|
||||
wv.loadUrl(currentUrl)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun applyDebugStatus() {
|
||||
val enabled = debugStatusEnabled
|
||||
val title = debugStatusTitle
|
||||
val subtitle = debugStatusSubtitle
|
||||
withWebViewOnMain { wv ->
|
||||
val titleJs = title?.let { JSONObject.quote(it) } ?: "null"
|
||||
val subtitleJs = subtitle?.let { JSONObject.quote(it) } ?: "null"
|
||||
val js = """
|
||||
(() => {
|
||||
try {
|
||||
const api = globalThis.__openclaw;
|
||||
if (!api) return;
|
||||
if (typeof api.setDebugStatusEnabled === 'function') {
|
||||
api.setDebugStatusEnabled(${if (enabled) "true" else "false"});
|
||||
}
|
||||
if (!${if (enabled) "true" else "false"}) return;
|
||||
if (typeof api.setStatus === 'function') {
|
||||
api.setStatus($titleJs, $subtitleJs);
|
||||
}
|
||||
} catch (_) {}
|
||||
})();
|
||||
""".trimIndent()
|
||||
wv.evaluateJavascript(js, null)
|
||||
}
|
||||
}
|
||||
|
||||
private fun applyHomeCanvasState() {
|
||||
val payload = homeCanvasStateJson ?: "null"
|
||||
withWebViewOnMain { wv ->
|
||||
val js = """
|
||||
(() => {
|
||||
try {
|
||||
const api = globalThis.__openclaw;
|
||||
if (!api || typeof api.renderHome !== 'function') return;
|
||||
api.renderHome($payload);
|
||||
} catch (_) {}
|
||||
})();
|
||||
""".trimIndent()
|
||||
wv.evaluateJavascript(js, null)
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun eval(javaScript: String): String =
|
||||
withContext(Dispatchers.Main) {
|
||||
val wv = webView ?: throw IllegalStateException("no webview")
|
||||
suspendCancellableCoroutine { cont ->
|
||||
wv.evaluateJavascript(javaScript) { result ->
|
||||
cont.resume(result ?: "")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun snapshotPngBase64(maxWidth: Int?): String =
|
||||
withContext(Dispatchers.Main) {
|
||||
val wv = webView ?: throw IllegalStateException("no webview")
|
||||
val bmp = wv.captureBitmap()
|
||||
try {
|
||||
val scaled = bmp.scaleForMaxWidth(maxWidth)
|
||||
try {
|
||||
val out = ByteArrayOutputStream()
|
||||
scaled.compress(Bitmap.CompressFormat.PNG, 100, out)
|
||||
Base64.encodeToString(out.toByteArray(), Base64.NO_WRAP)
|
||||
} finally {
|
||||
if (scaled !== bmp) scaled.recycle()
|
||||
}
|
||||
} finally {
|
||||
bmp.recycle()
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun snapshotBase64(format: SnapshotFormat, quality: Double?, maxWidth: Int?): String =
|
||||
withContext(Dispatchers.Main) {
|
||||
val wv = webView ?: throw IllegalStateException("no webview")
|
||||
val bmp = wv.captureBitmap()
|
||||
try {
|
||||
val scaled = bmp.scaleForMaxWidth(maxWidth)
|
||||
try {
|
||||
val out = ByteArrayOutputStream()
|
||||
val (compressFormat, compressQuality) =
|
||||
when (format) {
|
||||
SnapshotFormat.Png -> Bitmap.CompressFormat.PNG to 100
|
||||
SnapshotFormat.Jpeg -> Bitmap.CompressFormat.JPEG to clampJpegQuality(quality)
|
||||
}
|
||||
scaled.compress(compressFormat, compressQuality, out)
|
||||
Base64.encodeToString(out.toByteArray(), Base64.NO_WRAP)
|
||||
} finally {
|
||||
if (scaled !== bmp) scaled.recycle()
|
||||
}
|
||||
} finally {
|
||||
bmp.recycle()
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun WebView.captureBitmap(): Bitmap =
|
||||
suspendCancellableCoroutine { cont ->
|
||||
val width = width.coerceAtLeast(1)
|
||||
val height = height.coerceAtLeast(1)
|
||||
val bitmap = createBitmap(width, height, Bitmap.Config.ARGB_8888)
|
||||
|
||||
// WebView isn't supported by PixelCopy.request(...) directly; draw() is the most reliable
|
||||
// cross-version snapshot for this lightweight "canvas" use-case.
|
||||
draw(Canvas(bitmap))
|
||||
cont.resume(bitmap)
|
||||
}
|
||||
|
||||
companion object {
|
||||
data class SnapshotParams(val format: SnapshotFormat, val quality: Double?, val maxWidth: Int?)
|
||||
|
||||
fun parseNavigateUrl(paramsJson: String?): String {
|
||||
val obj = parseParamsObject(paramsJson) ?: return ""
|
||||
return obj.string("url").trim()
|
||||
}
|
||||
|
||||
fun parseEvalJs(paramsJson: String?): String? {
|
||||
val obj = parseParamsObject(paramsJson) ?: return null
|
||||
val js = obj.string("javaScript").trim()
|
||||
return js.takeIf { it.isNotBlank() }
|
||||
}
|
||||
|
||||
fun parseSnapshotMaxWidth(paramsJson: String?): Int? {
|
||||
val obj = parseParamsObject(paramsJson) ?: return null
|
||||
if (!obj.containsKey("maxWidth")) return null
|
||||
val width = obj.int("maxWidth") ?: 0
|
||||
return width.takeIf { it > 0 }
|
||||
}
|
||||
|
||||
fun parseSnapshotFormat(paramsJson: String?): SnapshotFormat {
|
||||
val obj = parseParamsObject(paramsJson) ?: return SnapshotFormat.Jpeg
|
||||
val raw = obj.string("format").trim().lowercase()
|
||||
return when (raw) {
|
||||
"png" -> SnapshotFormat.Png
|
||||
"jpeg", "jpg" -> SnapshotFormat.Jpeg
|
||||
"" -> SnapshotFormat.Jpeg
|
||||
else -> SnapshotFormat.Jpeg
|
||||
}
|
||||
}
|
||||
|
||||
fun parseSnapshotQuality(paramsJson: String?): Double? {
|
||||
val obj = parseParamsObject(paramsJson) ?: return null
|
||||
if (!obj.containsKey("quality")) return null
|
||||
val q = obj.double("quality") ?: Double.NaN
|
||||
if (!q.isFinite()) return null
|
||||
return q.coerceIn(0.1, 1.0)
|
||||
}
|
||||
|
||||
fun parseSnapshotParams(paramsJson: String?): SnapshotParams {
|
||||
return SnapshotParams(
|
||||
format = parseSnapshotFormat(paramsJson),
|
||||
quality = parseSnapshotQuality(paramsJson),
|
||||
maxWidth = parseSnapshotMaxWidth(paramsJson),
|
||||
)
|
||||
}
|
||||
|
||||
private val json = Json { ignoreUnknownKeys = true }
|
||||
|
||||
private fun parseParamsObject(paramsJson: String?): JsonObject? {
|
||||
val raw = paramsJson?.trim().orEmpty()
|
||||
if (raw.isEmpty()) return null
|
||||
return try {
|
||||
json.parseToJsonElement(raw).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject
|
||||
|
||||
private fun JsonObject.string(key: String): String {
|
||||
val prim = this[key] as? JsonPrimitive ?: return ""
|
||||
val raw = prim.content
|
||||
return raw.takeIf { it != "null" }.orEmpty()
|
||||
}
|
||||
|
||||
private fun JsonObject.int(key: String): Int? {
|
||||
val prim = this[key] as? JsonPrimitive ?: return null
|
||||
return prim.content.toIntOrNull()
|
||||
}
|
||||
|
||||
private fun JsonObject.double(key: String): Double? {
|
||||
val prim = this[key] as? JsonPrimitive ?: return null
|
||||
return prim.content.toDoubleOrNull()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.os.Build
|
||||
import ai.openclaw.app.BuildConfig
|
||||
import ai.openclaw.app.SecurePrefs
|
||||
import ai.openclaw.app.gateway.GatewayClientInfo
|
||||
import ai.openclaw.app.gateway.GatewayConnectOptions
|
||||
import ai.openclaw.app.gateway.GatewayEndpoint
|
||||
import ai.openclaw.app.gateway.GatewayTlsParams
|
||||
import ai.openclaw.app.gateway.isPrivateLanGatewayHost
|
||||
import ai.openclaw.app.LocationMode
|
||||
import ai.openclaw.app.VoiceWakeMode
|
||||
|
||||
class ConnectionManager(
|
||||
private val prefs: SecurePrefs,
|
||||
private val cameraEnabled: () -> Boolean,
|
||||
private val locationMode: () -> LocationMode,
|
||||
private val voiceWakeMode: () -> VoiceWakeMode,
|
||||
private val motionActivityAvailable: () -> Boolean,
|
||||
private val motionPedometerAvailable: () -> Boolean,
|
||||
private val sendSmsAvailable: () -> Boolean,
|
||||
private val readSmsAvailable: () -> Boolean,
|
||||
private val smsSearchPossible: () -> Boolean,
|
||||
private val callLogAvailable: () -> Boolean,
|
||||
private val hasRecordAudioPermission: () -> Boolean,
|
||||
private val manualTls: () -> Boolean,
|
||||
) {
|
||||
companion object {
|
||||
internal fun resolveTlsParamsForEndpoint(
|
||||
endpoint: GatewayEndpoint,
|
||||
storedFingerprint: String?,
|
||||
manualTlsEnabled: Boolean,
|
||||
): GatewayTlsParams? {
|
||||
val stableId = endpoint.stableId
|
||||
val stored = storedFingerprint?.trim().takeIf { !it.isNullOrEmpty() }
|
||||
val isManual = stableId.startsWith("manual|")
|
||||
val cleartextAllowedHost = isPrivateLanGatewayHost(endpoint.host)
|
||||
|
||||
if (isManual) {
|
||||
if (!manualTlsEnabled && cleartextAllowedHost) return null
|
||||
if (!stored.isNullOrBlank()) {
|
||||
return GatewayTlsParams(
|
||||
required = true,
|
||||
expectedFingerprint = stored,
|
||||
allowTOFU = false,
|
||||
stableId = stableId,
|
||||
)
|
||||
}
|
||||
return GatewayTlsParams(
|
||||
required = true,
|
||||
expectedFingerprint = null,
|
||||
allowTOFU = false,
|
||||
stableId = stableId,
|
||||
)
|
||||
}
|
||||
|
||||
// Prefer stored pins. Never let discovery-provided TXT override a stored fingerprint.
|
||||
if (!stored.isNullOrBlank()) {
|
||||
return GatewayTlsParams(
|
||||
required = true,
|
||||
expectedFingerprint = stored,
|
||||
allowTOFU = false,
|
||||
stableId = stableId,
|
||||
)
|
||||
}
|
||||
|
||||
val hinted = endpoint.tlsEnabled || !endpoint.tlsFingerprintSha256.isNullOrBlank()
|
||||
if (hinted) {
|
||||
// TXT is unauthenticated. Do not treat the advertised fingerprint as authoritative.
|
||||
return GatewayTlsParams(
|
||||
required = true,
|
||||
expectedFingerprint = null,
|
||||
allowTOFU = false,
|
||||
stableId = stableId,
|
||||
)
|
||||
}
|
||||
|
||||
if (!cleartextAllowedHost) {
|
||||
return GatewayTlsParams(
|
||||
required = true,
|
||||
expectedFingerprint = null,
|
||||
allowTOFU = false,
|
||||
stableId = stableId,
|
||||
)
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
private fun runtimeFlags(): NodeRuntimeFlags =
|
||||
NodeRuntimeFlags(
|
||||
cameraEnabled = cameraEnabled(),
|
||||
locationEnabled = locationMode() != LocationMode.Off,
|
||||
sendSmsAvailable = sendSmsAvailable(),
|
||||
readSmsAvailable = readSmsAvailable(),
|
||||
smsSearchPossible = smsSearchPossible(),
|
||||
callLogAvailable = callLogAvailable(),
|
||||
voiceWakeEnabled = voiceWakeMode() != VoiceWakeMode.Off && hasRecordAudioPermission(),
|
||||
motionActivityAvailable = motionActivityAvailable(),
|
||||
motionPedometerAvailable = motionPedometerAvailable(),
|
||||
debugBuild = BuildConfig.DEBUG,
|
||||
)
|
||||
|
||||
fun buildInvokeCommands(): List<String> = InvokeCommandRegistry.advertisedCommands(runtimeFlags())
|
||||
|
||||
fun buildCapabilities(): List<String> = InvokeCommandRegistry.advertisedCapabilities(runtimeFlags())
|
||||
|
||||
fun resolvedVersionName(): String {
|
||||
val versionName = BuildConfig.VERSION_NAME.trim().ifEmpty { "dev" }
|
||||
return if (BuildConfig.DEBUG && !versionName.contains("dev", ignoreCase = true)) {
|
||||
"$versionName-dev"
|
||||
} else {
|
||||
versionName
|
||||
}
|
||||
}
|
||||
|
||||
fun resolveModelIdentifier(): String? {
|
||||
return listOfNotNull(Build.MANUFACTURER, Build.MODEL)
|
||||
.joinToString(" ")
|
||||
.trim()
|
||||
.ifEmpty { null }
|
||||
}
|
||||
|
||||
fun buildUserAgent(): String {
|
||||
val version = resolvedVersionName()
|
||||
val release = Build.VERSION.RELEASE?.trim().orEmpty()
|
||||
val releaseLabel = if (release.isEmpty()) "unknown" else release
|
||||
return "OpenClawAndroid/$version (Android $releaseLabel; SDK ${Build.VERSION.SDK_INT})"
|
||||
}
|
||||
|
||||
fun buildClientInfo(clientId: String, clientMode: String): GatewayClientInfo {
|
||||
return GatewayClientInfo(
|
||||
id = clientId,
|
||||
displayName = prefs.displayName.value,
|
||||
version = resolvedVersionName(),
|
||||
platform = "android",
|
||||
mode = clientMode,
|
||||
instanceId = prefs.instanceId.value,
|
||||
deviceFamily = "Android",
|
||||
modelIdentifier = resolveModelIdentifier(),
|
||||
)
|
||||
}
|
||||
|
||||
fun buildNodeConnectOptions(): GatewayConnectOptions {
|
||||
return GatewayConnectOptions(
|
||||
role = "node",
|
||||
scopes = emptyList(),
|
||||
caps = buildCapabilities(),
|
||||
commands = buildInvokeCommands(),
|
||||
permissions = emptyMap(),
|
||||
client = buildClientInfo(clientId = "openclaw-android", clientMode = "node"),
|
||||
userAgent = buildUserAgent(),
|
||||
)
|
||||
}
|
||||
|
||||
fun buildOperatorConnectOptions(): GatewayConnectOptions {
|
||||
return GatewayConnectOptions(
|
||||
role = "operator",
|
||||
scopes = listOf("operator.read", "operator.write", "operator.talk.secrets"),
|
||||
caps = emptyList(),
|
||||
commands = emptyList(),
|
||||
permissions = emptyMap(),
|
||||
client = buildClientInfo(clientId = "openclaw-android", clientMode = "ui"),
|
||||
userAgent = buildUserAgent(),
|
||||
)
|
||||
}
|
||||
|
||||
fun resolveTlsParams(endpoint: GatewayEndpoint): GatewayTlsParams? {
|
||||
val stored = prefs.loadGatewayTlsFingerprint(endpoint.stableId)
|
||||
return resolveTlsParamsForEndpoint(endpoint, storedFingerprint = stored, manualTlsEnabled = manualTls())
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,433 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.content.ContentProviderOperation
|
||||
import android.content.ContentResolver
|
||||
import android.content.ContentValues
|
||||
import android.content.Context
|
||||
import android.provider.ContactsContract
|
||||
import androidx.core.content.ContextCompat
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonArray
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
private const val DEFAULT_CONTACTS_LIMIT = 25
|
||||
|
||||
internal data class ContactRecord(
|
||||
val identifier: String,
|
||||
val displayName: String,
|
||||
val givenName: String,
|
||||
val familyName: String,
|
||||
val organizationName: String,
|
||||
val phoneNumbers: List<String>,
|
||||
val emails: List<String>,
|
||||
)
|
||||
|
||||
internal data class ContactsSearchRequest(
|
||||
val query: String?,
|
||||
val limit: Int,
|
||||
)
|
||||
|
||||
internal data class ContactsAddRequest(
|
||||
val givenName: String?,
|
||||
val familyName: String?,
|
||||
val organizationName: String?,
|
||||
val displayName: String?,
|
||||
val phoneNumbers: List<String>,
|
||||
val emails: List<String>,
|
||||
)
|
||||
|
||||
internal interface ContactsDataSource {
|
||||
fun hasReadPermission(context: Context): Boolean
|
||||
|
||||
fun hasWritePermission(context: Context): Boolean
|
||||
|
||||
fun search(context: Context, request: ContactsSearchRequest): List<ContactRecord>
|
||||
|
||||
fun add(context: Context, request: ContactsAddRequest): ContactRecord
|
||||
}
|
||||
|
||||
private object SystemContactsDataSource : ContactsDataSource {
|
||||
override fun hasReadPermission(context: Context): Boolean {
|
||||
return ContextCompat.checkSelfPermission(context, Manifest.permission.READ_CONTACTS) ==
|
||||
android.content.pm.PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
|
||||
override fun hasWritePermission(context: Context): Boolean {
|
||||
return ContextCompat.checkSelfPermission(context, Manifest.permission.WRITE_CONTACTS) ==
|
||||
android.content.pm.PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
|
||||
override fun search(context: Context, request: ContactsSearchRequest): List<ContactRecord> {
|
||||
val resolver = context.contentResolver
|
||||
val projection =
|
||||
arrayOf(
|
||||
ContactsContract.Contacts._ID,
|
||||
ContactsContract.Contacts.DISPLAY_NAME_PRIMARY,
|
||||
)
|
||||
val selection: String?
|
||||
val selectionArgs: Array<String>?
|
||||
if (request.query.isNullOrBlank()) {
|
||||
selection = null
|
||||
selectionArgs = null
|
||||
} else {
|
||||
selection = "${ContactsContract.Contacts.DISPLAY_NAME_PRIMARY} LIKE ? ESCAPE '\\'"
|
||||
selectionArgs = arrayOf("%${escapeLikePattern(request.query)}%")
|
||||
}
|
||||
val sortOrder = "${ContactsContract.Contacts.DISPLAY_NAME_PRIMARY} COLLATE NOCASE ASC LIMIT ${request.limit}"
|
||||
resolver.query(
|
||||
ContactsContract.Contacts.CONTENT_URI,
|
||||
projection,
|
||||
selection,
|
||||
selectionArgs,
|
||||
sortOrder,
|
||||
).use { cursor ->
|
||||
if (cursor == null) return emptyList()
|
||||
val idIndex = cursor.getColumnIndexOrThrow(ContactsContract.Contacts._ID)
|
||||
val displayNameIndex = cursor.getColumnIndexOrThrow(ContactsContract.Contacts.DISPLAY_NAME_PRIMARY)
|
||||
val out = mutableListOf<ContactRecord>()
|
||||
while (cursor.moveToNext() && out.size < request.limit) {
|
||||
val contactId = cursor.getLong(idIndex)
|
||||
val displayName = cursor.getString(displayNameIndex).orEmpty()
|
||||
out += loadContactRecord(resolver, contactId, fallbackDisplayName = displayName)
|
||||
}
|
||||
return out
|
||||
}
|
||||
}
|
||||
|
||||
override fun add(context: Context, request: ContactsAddRequest): ContactRecord {
|
||||
val resolver = context.contentResolver
|
||||
val operations = ArrayList<ContentProviderOperation>()
|
||||
operations +=
|
||||
ContentProviderOperation.newInsert(ContactsContract.RawContacts.CONTENT_URI)
|
||||
.withValue(ContactsContract.RawContacts.ACCOUNT_TYPE, null)
|
||||
.withValue(ContactsContract.RawContacts.ACCOUNT_NAME, null)
|
||||
.build()
|
||||
if (!request.givenName.isNullOrEmpty() || !request.familyName.isNullOrEmpty() || !request.displayName.isNullOrEmpty()) {
|
||||
operations +=
|
||||
ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
|
||||
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0)
|
||||
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.StructuredName.CONTENT_ITEM_TYPE)
|
||||
.withValue(ContactsContract.CommonDataKinds.StructuredName.GIVEN_NAME, request.givenName)
|
||||
.withValue(ContactsContract.CommonDataKinds.StructuredName.FAMILY_NAME, request.familyName)
|
||||
.withValue(ContactsContract.CommonDataKinds.StructuredName.DISPLAY_NAME, request.displayName)
|
||||
.build()
|
||||
}
|
||||
if (!request.organizationName.isNullOrEmpty()) {
|
||||
operations +=
|
||||
ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
|
||||
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0)
|
||||
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.Organization.CONTENT_ITEM_TYPE)
|
||||
.withValue(ContactsContract.CommonDataKinds.Organization.COMPANY, request.organizationName)
|
||||
.build()
|
||||
}
|
||||
request.phoneNumbers.forEach { number ->
|
||||
operations +=
|
||||
ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
|
||||
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0)
|
||||
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.Phone.CONTENT_ITEM_TYPE)
|
||||
.withValue(ContactsContract.CommonDataKinds.Phone.NUMBER, number)
|
||||
.withValue(ContactsContract.CommonDataKinds.Phone.TYPE, ContactsContract.CommonDataKinds.Phone.TYPE_MOBILE)
|
||||
.build()
|
||||
}
|
||||
request.emails.forEach { email ->
|
||||
operations +=
|
||||
ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
|
||||
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0)
|
||||
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.Email.CONTENT_ITEM_TYPE)
|
||||
.withValue(ContactsContract.CommonDataKinds.Email.ADDRESS, email)
|
||||
.withValue(ContactsContract.CommonDataKinds.Email.TYPE, ContactsContract.CommonDataKinds.Email.TYPE_HOME)
|
||||
.build()
|
||||
}
|
||||
|
||||
val results = resolver.applyBatch(ContactsContract.AUTHORITY, operations)
|
||||
val rawContactUri = results.firstOrNull()?.uri
|
||||
?: throw IllegalStateException("contact insert failed")
|
||||
val rawContactId = rawContactUri.lastPathSegment?.toLongOrNull()
|
||||
?: throw IllegalStateException("contact insert failed")
|
||||
val contactId = resolveContactIdForRawContact(resolver, rawContactId)
|
||||
?: throw IllegalStateException("contact insert failed")
|
||||
return loadContactRecord(
|
||||
resolver = resolver,
|
||||
contactId = contactId,
|
||||
fallbackDisplayName = request.displayName.orEmpty(),
|
||||
)
|
||||
}
|
||||
|
||||
private fun resolveContactIdForRawContact(resolver: ContentResolver, rawContactId: Long): Long? {
|
||||
val projection = arrayOf(ContactsContract.RawContacts.CONTACT_ID)
|
||||
resolver.query(
|
||||
ContactsContract.RawContacts.CONTENT_URI,
|
||||
projection,
|
||||
"${ContactsContract.RawContacts._ID}=?",
|
||||
arrayOf(rawContactId.toString()),
|
||||
null,
|
||||
).use { cursor ->
|
||||
if (cursor == null || !cursor.moveToFirst()) return null
|
||||
val index = cursor.getColumnIndexOrThrow(ContactsContract.RawContacts.CONTACT_ID)
|
||||
return cursor.getLong(index)
|
||||
}
|
||||
}
|
||||
|
||||
private fun loadContactRecord(
|
||||
resolver: ContentResolver,
|
||||
contactId: Long,
|
||||
fallbackDisplayName: String,
|
||||
): ContactRecord {
|
||||
val nameRow = loadNameRow(resolver, contactId)
|
||||
val organization = loadOrganization(resolver, contactId)
|
||||
val phones = loadPhones(resolver, contactId)
|
||||
val emails = loadEmails(resolver, contactId)
|
||||
val displayName =
|
||||
when {
|
||||
!nameRow.displayName.isNullOrEmpty() -> nameRow.displayName
|
||||
!fallbackDisplayName.isNullOrEmpty() -> fallbackDisplayName
|
||||
else -> listOfNotNull(nameRow.givenName, nameRow.familyName).joinToString(" ").trim()
|
||||
}.ifEmpty { "(unnamed)" }
|
||||
return ContactRecord(
|
||||
identifier = contactId.toString(),
|
||||
displayName = displayName,
|
||||
givenName = nameRow.givenName.orEmpty(),
|
||||
familyName = nameRow.familyName.orEmpty(),
|
||||
organizationName = organization.orEmpty(),
|
||||
phoneNumbers = phones,
|
||||
emails = emails,
|
||||
)
|
||||
}
|
||||
|
||||
private data class NameRow(
|
||||
val givenName: String?,
|
||||
val familyName: String?,
|
||||
val displayName: String?,
|
||||
)
|
||||
|
||||
private fun loadNameRow(resolver: ContentResolver, contactId: Long): NameRow {
|
||||
val projection =
|
||||
arrayOf(
|
||||
ContactsContract.CommonDataKinds.StructuredName.GIVEN_NAME,
|
||||
ContactsContract.CommonDataKinds.StructuredName.FAMILY_NAME,
|
||||
ContactsContract.CommonDataKinds.StructuredName.DISPLAY_NAME,
|
||||
)
|
||||
resolver.query(
|
||||
ContactsContract.Data.CONTENT_URI,
|
||||
projection,
|
||||
"${ContactsContract.Data.CONTACT_ID}=? AND ${ContactsContract.Data.MIMETYPE}=?",
|
||||
arrayOf(
|
||||
contactId.toString(),
|
||||
ContactsContract.CommonDataKinds.StructuredName.CONTENT_ITEM_TYPE,
|
||||
),
|
||||
null,
|
||||
).use { cursor ->
|
||||
if (cursor == null || !cursor.moveToFirst()) {
|
||||
return NameRow(givenName = null, familyName = null, displayName = null)
|
||||
}
|
||||
val given = cursor.getString(0)?.trim()?.ifEmpty { null }
|
||||
val family = cursor.getString(1)?.trim()?.ifEmpty { null }
|
||||
val display = cursor.getString(2)?.trim()?.ifEmpty { null }
|
||||
return NameRow(givenName = given, familyName = family, displayName = display)
|
||||
}
|
||||
}
|
||||
|
||||
private fun loadOrganization(resolver: ContentResolver, contactId: Long): String? {
|
||||
val projection = arrayOf(ContactsContract.CommonDataKinds.Organization.COMPANY)
|
||||
resolver.query(
|
||||
ContactsContract.Data.CONTENT_URI,
|
||||
projection,
|
||||
"${ContactsContract.Data.CONTACT_ID}=? AND ${ContactsContract.Data.MIMETYPE}=?",
|
||||
arrayOf(contactId.toString(), ContactsContract.CommonDataKinds.Organization.CONTENT_ITEM_TYPE),
|
||||
null,
|
||||
).use { cursor ->
|
||||
if (cursor == null || !cursor.moveToFirst()) return null
|
||||
return cursor.getString(0)?.trim()?.ifEmpty { null }
|
||||
}
|
||||
}
|
||||
|
||||
private fun escapeLikePattern(pattern: String): String =
|
||||
pattern.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
|
||||
private fun loadPhones(resolver: ContentResolver, contactId: Long): List<String> {
|
||||
return queryContactValues(
|
||||
resolver = resolver,
|
||||
contentUri = ContactsContract.CommonDataKinds.Phone.CONTENT_URI,
|
||||
valueColumn = ContactsContract.CommonDataKinds.Phone.NUMBER,
|
||||
contactIdColumn = ContactsContract.CommonDataKinds.Phone.CONTACT_ID,
|
||||
contactId = contactId,
|
||||
)
|
||||
}
|
||||
|
||||
private fun loadEmails(resolver: ContentResolver, contactId: Long): List<String> {
|
||||
return queryContactValues(
|
||||
resolver = resolver,
|
||||
contentUri = ContactsContract.CommonDataKinds.Email.CONTENT_URI,
|
||||
valueColumn = ContactsContract.CommonDataKinds.Email.ADDRESS,
|
||||
contactIdColumn = ContactsContract.CommonDataKinds.Email.CONTACT_ID,
|
||||
contactId = contactId,
|
||||
)
|
||||
}
|
||||
|
||||
private fun queryContactValues(
|
||||
resolver: ContentResolver,
|
||||
contentUri: android.net.Uri,
|
||||
valueColumn: String,
|
||||
contactIdColumn: String,
|
||||
contactId: Long,
|
||||
): List<String> {
|
||||
val projection = arrayOf(valueColumn)
|
||||
resolver.query(
|
||||
contentUri,
|
||||
projection,
|
||||
"$contactIdColumn=?",
|
||||
arrayOf(contactId.toString()),
|
||||
null,
|
||||
).use { cursor ->
|
||||
if (cursor == null) return emptyList()
|
||||
val out = LinkedHashSet<String>()
|
||||
while (cursor.moveToNext()) {
|
||||
val value = cursor.getString(0)?.trim().orEmpty()
|
||||
if (value.isNotEmpty()) out += value
|
||||
}
|
||||
return out.toList()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
class ContactsHandler private constructor(
|
||||
private val appContext: Context,
|
||||
private val dataSource: ContactsDataSource,
|
||||
) {
|
||||
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemContactsDataSource)
|
||||
|
||||
fun handleContactsSearch(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!dataSource.hasReadPermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "CONTACTS_PERMISSION_REQUIRED",
|
||||
message = "CONTACTS_PERMISSION_REQUIRED: grant Contacts permission",
|
||||
)
|
||||
}
|
||||
val request =
|
||||
parseSearchRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
return try {
|
||||
val contacts = dataSource.search(appContext, request)
|
||||
GatewaySession.InvokeResult.ok(
|
||||
buildJsonObject {
|
||||
put(
|
||||
"contacts",
|
||||
buildJsonArray {
|
||||
contacts.forEach { add(contactJson(it)) }
|
||||
},
|
||||
)
|
||||
}.toString(),
|
||||
)
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "CONTACTS_UNAVAILABLE",
|
||||
message = "CONTACTS_UNAVAILABLE: ${err.message ?: "contacts query failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
fun handleContactsAdd(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!dataSource.hasWritePermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "CONTACTS_PERMISSION_REQUIRED",
|
||||
message = "CONTACTS_PERMISSION_REQUIRED: grant Contacts permission",
|
||||
)
|
||||
}
|
||||
val request =
|
||||
parseAddRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
val hasName =
|
||||
!(request.givenName.isNullOrEmpty() && request.familyName.isNullOrEmpty() && request.displayName.isNullOrEmpty())
|
||||
val hasOrg = !request.organizationName.isNullOrEmpty()
|
||||
val hasDetails = request.phoneNumbers.isNotEmpty() || request.emails.isNotEmpty()
|
||||
if (!hasName && !hasOrg && !hasDetails) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "CONTACTS_INVALID",
|
||||
message = "CONTACTS_INVALID: include a name, organization, phone, or email",
|
||||
)
|
||||
}
|
||||
return try {
|
||||
val contact = dataSource.add(appContext, request)
|
||||
GatewaySession.InvokeResult.ok(
|
||||
buildJsonObject {
|
||||
put("contact", contactJson(contact))
|
||||
}.toString(),
|
||||
)
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "CONTACTS_UNAVAILABLE",
|
||||
message = "CONTACTS_UNAVAILABLE: ${err.message ?: "contact add failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseSearchRequest(paramsJson: String?): ContactsSearchRequest? {
|
||||
if (paramsJson.isNullOrBlank()) {
|
||||
return ContactsSearchRequest(query = null, limit = DEFAULT_CONTACTS_LIMIT)
|
||||
}
|
||||
val params =
|
||||
try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
val query = (params["query"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null }
|
||||
val limit = ((params["limit"] as? JsonPrimitive)?.content?.toIntOrNull() ?: DEFAULT_CONTACTS_LIMIT).coerceIn(1, 200)
|
||||
return ContactsSearchRequest(query = query, limit = limit)
|
||||
}
|
||||
|
||||
private fun parseAddRequest(paramsJson: String?): ContactsAddRequest? {
|
||||
val params =
|
||||
try {
|
||||
paramsJson?.let { Json.parseToJsonElement(it).asObjectOrNull() }
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
return ContactsAddRequest(
|
||||
givenName = (params["givenName"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
familyName = (params["familyName"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
organizationName = (params["organizationName"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
displayName = (params["displayName"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
phoneNumbers = stringArray(params["phoneNumbers"] as? JsonArray),
|
||||
emails = stringArray(params["emails"] as? JsonArray).map { it.lowercase() },
|
||||
)
|
||||
}
|
||||
|
||||
private fun stringArray(array: JsonArray?): List<String> {
|
||||
if (array == null) return emptyList()
|
||||
return array.mapNotNull { element ->
|
||||
(element as? JsonPrimitive)?.content?.trim()?.ifEmpty { null }
|
||||
}
|
||||
}
|
||||
|
||||
private fun contactJson(contact: ContactRecord): JsonObject {
|
||||
return buildJsonObject {
|
||||
put("identifier", JsonPrimitive(contact.identifier))
|
||||
put("displayName", JsonPrimitive(contact.displayName))
|
||||
put("givenName", JsonPrimitive(contact.givenName))
|
||||
put("familyName", JsonPrimitive(contact.familyName))
|
||||
put("organizationName", JsonPrimitive(contact.organizationName))
|
||||
put("phoneNumbers", buildJsonArray { contact.phoneNumbers.forEach { add(JsonPrimitive(it)) } })
|
||||
put("emails", buildJsonArray { contact.emails.forEach { add(JsonPrimitive(it)) } })
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
internal fun forTesting(
|
||||
appContext: Context,
|
||||
dataSource: ContactsDataSource,
|
||||
): ContactsHandler = ContactsHandler(appContext = appContext, dataSource = dataSource)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,118 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.content.Context
|
||||
import ai.openclaw.app.BuildConfig
|
||||
import ai.openclaw.app.gateway.DeviceIdentityStore
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
|
||||
class DebugHandler(
|
||||
private val appContext: Context,
|
||||
private val identityStore: DeviceIdentityStore,
|
||||
) {
|
||||
|
||||
fun handleEd25519(): GatewaySession.InvokeResult {
|
||||
if (!BuildConfig.DEBUG) {
|
||||
return GatewaySession.InvokeResult.error(code = "UNAVAILABLE", message = "debug commands are disabled in release builds")
|
||||
}
|
||||
// Self-test Ed25519 signing and return diagnostic info
|
||||
try {
|
||||
val identity = identityStore.loadOrCreate()
|
||||
val testPayload = "test|${identity.deviceId}|${System.currentTimeMillis()}"
|
||||
val results = mutableListOf<String>()
|
||||
results.add("deviceId: ${identity.deviceId}")
|
||||
results.add("publicKeyRawBase64: ${identity.publicKeyRawBase64.take(20)}...")
|
||||
results.add("privateKeyPkcs8Base64: ${identity.privateKeyPkcs8Base64.take(20)}...")
|
||||
|
||||
// Test publicKeyBase64Url
|
||||
val pubKeyUrl = identityStore.publicKeyBase64Url(identity)
|
||||
results.add("publicKeyBase64Url: ${pubKeyUrl ?: "NULL (FAILED)"}")
|
||||
|
||||
// Test signing
|
||||
val signature = identityStore.signPayload(testPayload, identity)
|
||||
results.add("signPayload: ${if (signature != null) "${signature.take(20)}... (OK)" else "NULL (FAILED)"}")
|
||||
|
||||
// Test self-verify
|
||||
if (signature != null) {
|
||||
val verifyOk = identityStore.verifySelfSignature(testPayload, signature, identity)
|
||||
results.add("verifySelfSignature: $verifyOk")
|
||||
}
|
||||
|
||||
// Check available providers
|
||||
val providers = java.security.Security.getProviders()
|
||||
val ed25519Providers = providers.filter { p ->
|
||||
p.services.any { s -> s.algorithm.contains("Ed25519", ignoreCase = true) }
|
||||
}
|
||||
results.add("Ed25519 providers: ${ed25519Providers.map { "${it.name} v${it.version}" }}")
|
||||
results.add("Provider order: ${providers.take(5).map { it.name }}")
|
||||
|
||||
// Test KeyFactory directly
|
||||
try {
|
||||
val kf = java.security.KeyFactory.getInstance("Ed25519")
|
||||
results.add("KeyFactory.Ed25519: ${kf.provider.name} (OK)")
|
||||
} catch (e: Throwable) {
|
||||
results.add("KeyFactory.Ed25519: FAILED - ${e.javaClass.simpleName}: ${e.message}")
|
||||
}
|
||||
|
||||
// Test Signature directly
|
||||
try {
|
||||
val sig = java.security.Signature.getInstance("Ed25519")
|
||||
results.add("Signature.Ed25519: ${sig.provider.name} (OK)")
|
||||
} catch (e: Throwable) {
|
||||
results.add("Signature.Ed25519: FAILED - ${e.javaClass.simpleName}: ${e.message}")
|
||||
}
|
||||
|
||||
val diagnostics = results.joinToString("\n")
|
||||
return GatewaySession.InvokeResult.ok("""{"diagnostics":${JsonPrimitive(diagnostics)}}""")
|
||||
} catch (e: Throwable) {
|
||||
return GatewaySession.InvokeResult.error(code = "ED25519_TEST_FAILED", message = "${e.javaClass.simpleName}: ${e.message}\n${e.stackTraceToString().take(500)}")
|
||||
}
|
||||
}
|
||||
|
||||
fun handleLogs(): GatewaySession.InvokeResult {
|
||||
if (!BuildConfig.DEBUG) {
|
||||
return GatewaySession.InvokeResult.error(code = "UNAVAILABLE", message = "debug commands are disabled in release builds")
|
||||
}
|
||||
val pid = android.os.Process.myPid()
|
||||
val rt = Runtime.getRuntime()
|
||||
val info = "v6 pid=$pid thread=${Thread.currentThread().name} free=${rt.freeMemory()/1024}K total=${rt.totalMemory()/1024}K max=${rt.maxMemory()/1024}K uptime=${android.os.SystemClock.elapsedRealtime()/1000}s sdk=${android.os.Build.VERSION.SDK_INT} device=${android.os.Build.MODEL}\n"
|
||||
// Run logcat on current dispatcher thread (no withContext) with file redirect
|
||||
val logResult = try {
|
||||
val tmpFile = java.io.File(appContext.cacheDir, "debug_logs.txt")
|
||||
if (tmpFile.exists()) tmpFile.delete()
|
||||
val pb = ProcessBuilder("logcat", "-d", "-t", "200", "--pid=$pid")
|
||||
pb.redirectOutput(tmpFile)
|
||||
pb.redirectErrorStream(true)
|
||||
val proc = pb.start()
|
||||
val finished = proc.waitFor(4, java.util.concurrent.TimeUnit.SECONDS)
|
||||
if (!finished) proc.destroyForcibly()
|
||||
val raw = if (tmpFile.exists() && tmpFile.length() > 0) {
|
||||
tmpFile.readText().take(128000)
|
||||
} else {
|
||||
"(no output, finished=$finished, exists=${tmpFile.exists()})"
|
||||
}
|
||||
tmpFile.delete()
|
||||
val spamPatterns = listOf("setRequestedFrameRate", "I View :", "BLASTBufferQueue", "VRI[Pop-Up",
|
||||
"InsetsController:", "VRI[MainActivity", "InsetsSource:", "handleResized", "ProfileInstaller",
|
||||
"I VRI[", "onStateChanged: host=", "D StrictMode:", "E StrictMode:", "ImeFocusController",
|
||||
"InputTransport", "IncorrectContextUseViolation")
|
||||
val sb = StringBuilder()
|
||||
for (line in raw.lineSequence()) {
|
||||
if (line.isBlank()) continue
|
||||
if (spamPatterns.any { line.contains(it) }) continue
|
||||
if (sb.length + line.length > 16000) { sb.append("\n(truncated)"); break }
|
||||
if (sb.isNotEmpty()) sb.append('\n')
|
||||
sb.append(line)
|
||||
}
|
||||
sb.toString().ifEmpty { "(all ${raw.lines().size} lines filtered as spam)" }
|
||||
} catch (e: Throwable) {
|
||||
"(logcat error: ${e::class.java.simpleName}: ${e.message})"
|
||||
}
|
||||
// Also include camera debug log if it exists
|
||||
val camLogFile = java.io.File(appContext.cacheDir, "camera_debug.log")
|
||||
val camLog = if (camLogFile.exists() && camLogFile.length() > 0) {
|
||||
"\n--- camera_debug.log ---\n" + camLogFile.readText().take(4000)
|
||||
} else ""
|
||||
return GatewaySession.InvokeResult.ok("""{"logs":${JsonPrimitive(info + logResult + camLog)}}""")
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,453 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import ai.openclaw.app.BuildConfig
|
||||
import android.Manifest
|
||||
import android.app.ActivityManager
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.content.IntentFilter
|
||||
import android.content.pm.PackageManager
|
||||
import android.net.ConnectivityManager
|
||||
import android.net.NetworkCapabilities
|
||||
import android.os.BatteryManager
|
||||
import android.os.Build
|
||||
import android.os.Environment
|
||||
import android.os.PowerManager
|
||||
import android.os.StatFs
|
||||
import android.os.SystemClock
|
||||
import androidx.core.content.ContextCompat
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import java.util.Locale
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonArray
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
class DeviceHandler(
|
||||
private val appContext: Context,
|
||||
private val smsEnabled: Boolean = BuildConfig.OPENCLAW_ENABLE_SMS,
|
||||
private val callLogEnabled: Boolean = BuildConfig.OPENCLAW_ENABLE_CALL_LOG,
|
||||
) {
|
||||
companion object {
|
||||
internal fun hasAnySmsCapability(
|
||||
smsEnabled: Boolean,
|
||||
telephonyAvailable: Boolean,
|
||||
smsSendGranted: Boolean,
|
||||
smsReadGranted: Boolean,
|
||||
): Boolean {
|
||||
return smsEnabled && telephonyAvailable && (smsSendGranted || smsReadGranted)
|
||||
}
|
||||
|
||||
internal fun isSmsPromptable(
|
||||
smsEnabled: Boolean,
|
||||
telephonyAvailable: Boolean,
|
||||
smsSendGranted: Boolean,
|
||||
smsReadGranted: Boolean,
|
||||
): Boolean {
|
||||
return smsEnabled && telephonyAvailable && (!smsSendGranted || !smsReadGranted)
|
||||
}
|
||||
}
|
||||
private data class BatterySnapshot(
|
||||
val status: Int,
|
||||
val plugged: Int,
|
||||
val levelFraction: Double?,
|
||||
val temperatureC: Double?,
|
||||
)
|
||||
|
||||
fun handleDeviceStatus(_paramsJson: String?): GatewaySession.InvokeResult {
|
||||
return GatewaySession.InvokeResult.ok(statusPayloadJson())
|
||||
}
|
||||
|
||||
fun handleDeviceInfo(_paramsJson: String?): GatewaySession.InvokeResult {
|
||||
return GatewaySession.InvokeResult.ok(infoPayloadJson())
|
||||
}
|
||||
|
||||
fun handleDevicePermissions(_paramsJson: String?): GatewaySession.InvokeResult {
|
||||
return GatewaySession.InvokeResult.ok(permissionsPayloadJson())
|
||||
}
|
||||
|
||||
fun handleDeviceHealth(_paramsJson: String?): GatewaySession.InvokeResult {
|
||||
return GatewaySession.InvokeResult.ok(healthPayloadJson())
|
||||
}
|
||||
|
||||
private fun statusPayloadJson(): String {
|
||||
val battery = readBatterySnapshot()
|
||||
val powerManager = appContext.getSystemService(PowerManager::class.java)
|
||||
val storage = StatFs(Environment.getDataDirectory().absolutePath)
|
||||
val totalBytes = storage.totalBytes
|
||||
val freeBytes = storage.availableBytes
|
||||
val usedBytes = (totalBytes - freeBytes).coerceAtLeast(0L)
|
||||
val connectivity = appContext.getSystemService(ConnectivityManager::class.java)
|
||||
val activeNetwork = connectivity?.activeNetwork
|
||||
val caps = activeNetwork?.let { connectivity.getNetworkCapabilities(it) }
|
||||
val uptimeSeconds = SystemClock.elapsedRealtime() / 1_000.0
|
||||
|
||||
return buildJsonObject {
|
||||
put(
|
||||
"battery",
|
||||
buildJsonObject {
|
||||
battery.levelFraction?.let { put("level", JsonPrimitive(it)) }
|
||||
put("state", JsonPrimitive(mapBatteryState(battery.status)))
|
||||
put("lowPowerModeEnabled", JsonPrimitive(powerManager?.isPowerSaveMode == true))
|
||||
},
|
||||
)
|
||||
put(
|
||||
"thermal",
|
||||
buildJsonObject {
|
||||
put("state", JsonPrimitive(mapThermalState(powerManager)))
|
||||
},
|
||||
)
|
||||
put(
|
||||
"storage",
|
||||
buildJsonObject {
|
||||
put("totalBytes", JsonPrimitive(totalBytes))
|
||||
put("freeBytes", JsonPrimitive(freeBytes))
|
||||
put("usedBytes", JsonPrimitive(usedBytes))
|
||||
},
|
||||
)
|
||||
put(
|
||||
"network",
|
||||
buildJsonObject {
|
||||
put("status", JsonPrimitive(mapNetworkStatus(caps)))
|
||||
put(
|
||||
"isExpensive",
|
||||
JsonPrimitive(
|
||||
caps?.hasCapability(NetworkCapabilities.NET_CAPABILITY_NOT_METERED)?.not() ?: false,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"isConstrained",
|
||||
JsonPrimitive(
|
||||
caps?.hasCapability(NetworkCapabilities.NET_CAPABILITY_NOT_RESTRICTED)?.not() ?: false,
|
||||
),
|
||||
)
|
||||
put("interfaces", networkInterfacesJson(caps))
|
||||
},
|
||||
)
|
||||
put("uptimeSeconds", JsonPrimitive(uptimeSeconds))
|
||||
}.toString()
|
||||
}
|
||||
|
||||
private fun infoPayloadJson(): String {
|
||||
val model = Build.MODEL?.trim().orEmpty()
|
||||
val manufacturer = Build.MANUFACTURER?.trim().orEmpty()
|
||||
val modelIdentifier = Build.DEVICE?.trim().orEmpty()
|
||||
val systemVersion = Build.VERSION.RELEASE?.trim().orEmpty()
|
||||
val locale = Locale.getDefault().toLanguageTag().trim()
|
||||
val appVersion = BuildConfig.VERSION_NAME.trim()
|
||||
val appBuild = BuildConfig.VERSION_CODE.toString()
|
||||
|
||||
return buildJsonObject {
|
||||
put("deviceName", JsonPrimitive(model.ifEmpty { "Android" }))
|
||||
put("modelIdentifier", JsonPrimitive(modelIdentifier.ifEmpty { listOf(manufacturer, model).filter { it.isNotEmpty() }.joinToString(" ") }))
|
||||
put("systemName", JsonPrimitive("Android"))
|
||||
put("systemVersion", JsonPrimitive(systemVersion.ifEmpty { Build.VERSION.SDK_INT.toString() }))
|
||||
put("appVersion", JsonPrimitive(appVersion.ifEmpty { "dev" }))
|
||||
put("appBuild", JsonPrimitive(appBuild.ifEmpty { "0" }))
|
||||
put("locale", JsonPrimitive(locale.ifEmpty { Locale.getDefault().toString() }))
|
||||
}.toString()
|
||||
}
|
||||
|
||||
private fun permissionsPayloadJson(): String {
|
||||
val canSendSms = appContext.packageManager.hasSystemFeature(PackageManager.FEATURE_TELEPHONY)
|
||||
val smsSendGranted = hasPermission(Manifest.permission.SEND_SMS)
|
||||
val smsReadGranted = hasPermission(Manifest.permission.READ_SMS)
|
||||
val notificationAccess = DeviceNotificationListenerService.isAccessEnabled(appContext)
|
||||
val photosGranted =
|
||||
if (Build.VERSION.SDK_INT >= 33) {
|
||||
hasPermission(Manifest.permission.READ_MEDIA_IMAGES)
|
||||
} else {
|
||||
hasPermission(Manifest.permission.READ_EXTERNAL_STORAGE)
|
||||
}
|
||||
val motionGranted = hasPermission(Manifest.permission.ACTIVITY_RECOGNITION)
|
||||
val notificationsGranted =
|
||||
if (Build.VERSION.SDK_INT >= 33) {
|
||||
hasPermission(Manifest.permission.POST_NOTIFICATIONS)
|
||||
} else {
|
||||
true
|
||||
}
|
||||
return buildJsonObject {
|
||||
put(
|
||||
"permissions",
|
||||
buildJsonObject {
|
||||
put(
|
||||
"camera",
|
||||
permissionStateJson(
|
||||
granted = hasPermission(Manifest.permission.CAMERA),
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"microphone",
|
||||
permissionStateJson(
|
||||
granted = hasPermission(Manifest.permission.RECORD_AUDIO),
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"location",
|
||||
permissionStateJson(
|
||||
granted =
|
||||
hasPermission(Manifest.permission.ACCESS_FINE_LOCATION) ||
|
||||
hasPermission(Manifest.permission.ACCESS_COARSE_LOCATION),
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"sms",
|
||||
buildJsonObject {
|
||||
put(
|
||||
"status",
|
||||
JsonPrimitive(
|
||||
if (hasAnySmsCapability(smsEnabled, canSendSms, smsSendGranted, smsReadGranted)) "granted" else "denied",
|
||||
),
|
||||
)
|
||||
put("promptable", JsonPrimitive(isSmsPromptable(smsEnabled, canSendSms, smsSendGranted, smsReadGranted)))
|
||||
put(
|
||||
"capabilities",
|
||||
buildJsonObject {
|
||||
put(
|
||||
"send",
|
||||
permissionStateJson(
|
||||
granted = smsEnabled && smsSendGranted && canSendSms,
|
||||
promptableWhenDenied = smsEnabled && canSendSms,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"read",
|
||||
permissionStateJson(
|
||||
granted = smsEnabled && smsReadGranted && canSendSms,
|
||||
promptableWhenDenied = smsEnabled && canSendSms,
|
||||
),
|
||||
)
|
||||
},
|
||||
)
|
||||
},
|
||||
)
|
||||
put(
|
||||
"notificationListener",
|
||||
permissionStateJson(
|
||||
granted = notificationAccess,
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"notifications",
|
||||
permissionStateJson(
|
||||
granted = notificationsGranted,
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"photos",
|
||||
permissionStateJson(
|
||||
granted = photosGranted,
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"contacts",
|
||||
permissionStateJson(
|
||||
granted = hasPermission(Manifest.permission.READ_CONTACTS),
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"calendar",
|
||||
permissionStateJson(
|
||||
granted = hasPermission(Manifest.permission.READ_CALENDAR),
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"callLog",
|
||||
permissionStateJson(
|
||||
granted = callLogEnabled && hasPermission(Manifest.permission.READ_CALL_LOG),
|
||||
promptableWhenDenied = callLogEnabled,
|
||||
),
|
||||
)
|
||||
put(
|
||||
"motion",
|
||||
permissionStateJson(
|
||||
granted = motionGranted,
|
||||
promptableWhenDenied = true,
|
||||
),
|
||||
)
|
||||
},
|
||||
)
|
||||
}.toString()
|
||||
}
|
||||
|
||||
private fun healthPayloadJson(): String {
|
||||
val battery = readBatterySnapshot()
|
||||
val batteryManager = appContext.getSystemService(BatteryManager::class.java)
|
||||
val currentNowUa = batteryManager?.getLongProperty(BatteryManager.BATTERY_PROPERTY_CURRENT_NOW)
|
||||
val currentNowMa =
|
||||
if (currentNowUa == null || currentNowUa == Long.MIN_VALUE) {
|
||||
null
|
||||
} else {
|
||||
currentNowUa.toDouble() / 1_000.0
|
||||
}
|
||||
|
||||
val powerManager = appContext.getSystemService(PowerManager::class.java)
|
||||
val activityManager = appContext.getSystemService(ActivityManager::class.java)
|
||||
val memoryInfo = ActivityManager.MemoryInfo()
|
||||
activityManager?.getMemoryInfo(memoryInfo)
|
||||
val totalRamBytes = memoryInfo.totalMem.coerceAtLeast(0L)
|
||||
val availableRamBytes = memoryInfo.availMem.coerceAtLeast(0L)
|
||||
val usedRamBytes = (totalRamBytes - availableRamBytes).coerceAtLeast(0L)
|
||||
val lowMemory = memoryInfo.lowMemory
|
||||
val memoryPressure = mapMemoryPressure(totalRamBytes, availableRamBytes, lowMemory)
|
||||
|
||||
return buildJsonObject {
|
||||
put(
|
||||
"memory",
|
||||
buildJsonObject {
|
||||
put("pressure", JsonPrimitive(memoryPressure))
|
||||
put("totalRamBytes", JsonPrimitive(totalRamBytes))
|
||||
put("availableRamBytes", JsonPrimitive(availableRamBytes))
|
||||
put("usedRamBytes", JsonPrimitive(usedRamBytes))
|
||||
put("thresholdBytes", JsonPrimitive(memoryInfo.threshold.coerceAtLeast(0L)))
|
||||
put("lowMemory", JsonPrimitive(lowMemory))
|
||||
},
|
||||
)
|
||||
put(
|
||||
"battery",
|
||||
buildJsonObject {
|
||||
put("state", JsonPrimitive(mapBatteryState(battery.status)))
|
||||
put("chargingType", JsonPrimitive(mapChargingType(battery.plugged)))
|
||||
battery.temperatureC?.let { put("temperatureC", JsonPrimitive(it)) }
|
||||
currentNowMa?.let { put("currentMa", JsonPrimitive(it)) }
|
||||
},
|
||||
)
|
||||
put(
|
||||
"power",
|
||||
buildJsonObject {
|
||||
put("dozeModeEnabled", JsonPrimitive(powerManager?.isDeviceIdleMode == true))
|
||||
put("lowPowerModeEnabled", JsonPrimitive(powerManager?.isPowerSaveMode == true))
|
||||
},
|
||||
)
|
||||
put(
|
||||
"system",
|
||||
buildJsonObject {
|
||||
Build.VERSION.SECURITY_PATCH
|
||||
?.trim()
|
||||
?.takeIf { it.isNotEmpty() }
|
||||
?.let { put("securityPatchLevel", JsonPrimitive(it)) }
|
||||
},
|
||||
)
|
||||
}.toString()
|
||||
}
|
||||
|
||||
private fun readBatterySnapshot(): BatterySnapshot {
|
||||
val intent = appContext.registerReceiver(null, IntentFilter(Intent.ACTION_BATTERY_CHANGED))
|
||||
val status =
|
||||
intent?.getIntExtra(BatteryManager.EXTRA_STATUS, BatteryManager.BATTERY_STATUS_UNKNOWN)
|
||||
?: BatteryManager.BATTERY_STATUS_UNKNOWN
|
||||
val plugged = intent?.getIntExtra(BatteryManager.EXTRA_PLUGGED, 0) ?: 0
|
||||
val temperatureC =
|
||||
intent
|
||||
?.getIntExtra(BatteryManager.EXTRA_TEMPERATURE, Int.MIN_VALUE)
|
||||
?.takeIf { it != Int.MIN_VALUE }
|
||||
?.toDouble()
|
||||
?.div(10.0)
|
||||
return BatterySnapshot(
|
||||
status = status,
|
||||
plugged = plugged,
|
||||
levelFraction = batteryLevelFraction(intent),
|
||||
temperatureC = temperatureC,
|
||||
)
|
||||
}
|
||||
|
||||
private fun batteryLevelFraction(intent: Intent?): Double? {
|
||||
val rawLevel = intent?.getIntExtra(BatteryManager.EXTRA_LEVEL, -1) ?: -1
|
||||
val rawScale = intent?.getIntExtra(BatteryManager.EXTRA_SCALE, -1) ?: -1
|
||||
if (rawLevel < 0 || rawScale <= 0) return null
|
||||
return rawLevel.toDouble() / rawScale.toDouble()
|
||||
}
|
||||
|
||||
private fun mapBatteryState(status: Int): String {
|
||||
return when (status) {
|
||||
BatteryManager.BATTERY_STATUS_CHARGING -> "charging"
|
||||
BatteryManager.BATTERY_STATUS_FULL -> "full"
|
||||
BatteryManager.BATTERY_STATUS_DISCHARGING, BatteryManager.BATTERY_STATUS_NOT_CHARGING -> "unplugged"
|
||||
else -> "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
private fun mapChargingType(plugged: Int): String {
|
||||
return when (plugged) {
|
||||
BatteryManager.BATTERY_PLUGGED_AC -> "ac"
|
||||
BatteryManager.BATTERY_PLUGGED_USB -> "usb"
|
||||
BatteryManager.BATTERY_PLUGGED_WIRELESS -> "wireless"
|
||||
BatteryManager.BATTERY_PLUGGED_DOCK -> "dock"
|
||||
else -> "none"
|
||||
}
|
||||
}
|
||||
|
||||
private fun mapThermalState(powerManager: PowerManager?): String {
|
||||
val thermal = powerManager?.currentThermalStatus ?: return "nominal"
|
||||
return when (thermal) {
|
||||
PowerManager.THERMAL_STATUS_NONE, PowerManager.THERMAL_STATUS_LIGHT -> "nominal"
|
||||
PowerManager.THERMAL_STATUS_MODERATE -> "fair"
|
||||
PowerManager.THERMAL_STATUS_SEVERE -> "serious"
|
||||
PowerManager.THERMAL_STATUS_CRITICAL,
|
||||
PowerManager.THERMAL_STATUS_EMERGENCY,
|
||||
PowerManager.THERMAL_STATUS_SHUTDOWN -> "critical"
|
||||
else -> "nominal"
|
||||
}
|
||||
}
|
||||
|
||||
private fun mapNetworkStatus(caps: NetworkCapabilities?): String {
|
||||
if (caps == null) return "unsatisfied"
|
||||
return when {
|
||||
caps.hasCapability(NetworkCapabilities.NET_CAPABILITY_VALIDATED) -> "satisfied"
|
||||
caps.hasCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET) -> "requiresConnection"
|
||||
else -> "unsatisfied"
|
||||
}
|
||||
}
|
||||
|
||||
private fun permissionStateJson(granted: Boolean, promptableWhenDenied: Boolean) =
|
||||
buildJsonObject {
|
||||
put("status", JsonPrimitive(if (granted) "granted" else "denied"))
|
||||
put("promptable", JsonPrimitive(!granted && promptableWhenDenied))
|
||||
}
|
||||
|
||||
private fun hasPermission(permission: String): Boolean {
|
||||
return (
|
||||
ContextCompat.checkSelfPermission(appContext, permission) == PackageManager.PERMISSION_GRANTED
|
||||
)
|
||||
}
|
||||
|
||||
private fun mapMemoryPressure(totalBytes: Long, availableBytes: Long, lowMemory: Boolean): String {
|
||||
if (totalBytes <= 0L) return if (lowMemory) "critical" else "unknown"
|
||||
if (lowMemory) return "critical"
|
||||
val freeRatio = availableBytes.toDouble() / totalBytes.toDouble()
|
||||
return when {
|
||||
freeRatio <= 0.05 -> "critical"
|
||||
freeRatio <= 0.15 -> "high"
|
||||
freeRatio <= 0.30 -> "moderate"
|
||||
else -> "normal"
|
||||
}
|
||||
}
|
||||
|
||||
private fun networkInterfacesJson(caps: NetworkCapabilities?) =
|
||||
buildJsonArray {
|
||||
if (caps == null) return@buildJsonArray
|
||||
var hasKnownTransport = false
|
||||
if (caps.hasTransport(NetworkCapabilities.TRANSPORT_WIFI)) {
|
||||
hasKnownTransport = true
|
||||
add(JsonPrimitive("wifi"))
|
||||
}
|
||||
if (caps.hasTransport(NetworkCapabilities.TRANSPORT_CELLULAR)) {
|
||||
hasKnownTransport = true
|
||||
add(JsonPrimitive("cellular"))
|
||||
}
|
||||
if (caps.hasTransport(NetworkCapabilities.TRANSPORT_ETHERNET)) {
|
||||
hasKnownTransport = true
|
||||
add(JsonPrimitive("wired"))
|
||||
}
|
||||
if (!hasKnownTransport) add(JsonPrimitive("other"))
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,473 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.app.Notification
|
||||
import android.app.NotificationManager
|
||||
import android.app.RemoteInput
|
||||
import android.content.ComponentName
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.service.notification.NotificationListenerService
|
||||
import android.service.notification.StatusBarNotification
|
||||
import ai.openclaw.app.NotificationBurstLimiter
|
||||
import ai.openclaw.app.SecurePrefs
|
||||
import ai.openclaw.app.allowsPackage
|
||||
import ai.openclaw.app.isWithinQuietHours
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
private const val MAX_NOTIFICATION_TEXT_CHARS = 512
|
||||
private const val NOTIFICATIONS_CHANGED_EVENT = "notifications.changed"
|
||||
|
||||
internal fun sanitizeNotificationText(value: CharSequence?): String? {
|
||||
val normalized = value?.toString()?.trim().orEmpty()
|
||||
return normalized.take(MAX_NOTIFICATION_TEXT_CHARS).ifEmpty { null }
|
||||
}
|
||||
|
||||
data class DeviceNotificationEntry(
|
||||
val key: String,
|
||||
val packageName: String,
|
||||
val title: String?,
|
||||
val text: String?,
|
||||
val subText: String?,
|
||||
val category: String?,
|
||||
val channelId: String?,
|
||||
val postTimeMs: Long,
|
||||
val isOngoing: Boolean,
|
||||
val isClearable: Boolean,
|
||||
)
|
||||
|
||||
internal fun DeviceNotificationEntry.toJsonObject(): JsonObject {
|
||||
return buildJsonObject {
|
||||
put("key", JsonPrimitive(key))
|
||||
put("packageName", JsonPrimitive(packageName))
|
||||
put("postTimeMs", JsonPrimitive(postTimeMs))
|
||||
put("isOngoing", JsonPrimitive(isOngoing))
|
||||
put("isClearable", JsonPrimitive(isClearable))
|
||||
title?.let { put("title", JsonPrimitive(it)) }
|
||||
text?.let { put("text", JsonPrimitive(it)) }
|
||||
subText?.let { put("subText", JsonPrimitive(it)) }
|
||||
category?.let { put("category", JsonPrimitive(it)) }
|
||||
channelId?.let { put("channelId", JsonPrimitive(it)) }
|
||||
}
|
||||
}
|
||||
|
||||
data class DeviceNotificationSnapshot(
|
||||
val enabled: Boolean,
|
||||
val connected: Boolean,
|
||||
val notifications: List<DeviceNotificationEntry>,
|
||||
)
|
||||
|
||||
enum class NotificationActionKind {
|
||||
Open,
|
||||
Dismiss,
|
||||
Reply,
|
||||
}
|
||||
|
||||
data class NotificationActionRequest(
|
||||
val key: String,
|
||||
val kind: NotificationActionKind,
|
||||
val replyText: String? = null,
|
||||
)
|
||||
|
||||
data class NotificationActionResult(
|
||||
val ok: Boolean,
|
||||
val code: String? = null,
|
||||
val message: String? = null,
|
||||
)
|
||||
|
||||
internal fun actionRequiresClearableNotification(kind: NotificationActionKind): Boolean {
|
||||
return kind == NotificationActionKind.Dismiss
|
||||
}
|
||||
|
||||
private object DeviceNotificationStore {
|
||||
private val lock = Any()
|
||||
private var connected = false
|
||||
private val byKey = LinkedHashMap<String, DeviceNotificationEntry>()
|
||||
|
||||
fun replace(entries: List<DeviceNotificationEntry>) {
|
||||
synchronized(lock) {
|
||||
byKey.clear()
|
||||
for (entry in entries) {
|
||||
byKey[entry.key] = entry
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fun upsert(entry: DeviceNotificationEntry) {
|
||||
synchronized(lock) {
|
||||
byKey[entry.key] = entry
|
||||
}
|
||||
}
|
||||
|
||||
fun remove(key: String) {
|
||||
synchronized(lock) {
|
||||
byKey.remove(key)
|
||||
}
|
||||
}
|
||||
|
||||
fun setConnected(value: Boolean) {
|
||||
synchronized(lock) {
|
||||
connected = value
|
||||
if (!value) {
|
||||
byKey.clear()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fun snapshot(enabled: Boolean): DeviceNotificationSnapshot {
|
||||
val (isConnected, entries) =
|
||||
synchronized(lock) {
|
||||
connected to byKey.values.sortedByDescending { it.postTimeMs }
|
||||
}
|
||||
return DeviceNotificationSnapshot(
|
||||
enabled = enabled,
|
||||
connected = isConnected,
|
||||
notifications = entries,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
class DeviceNotificationListenerService : NotificationListenerService() {
|
||||
private val securePrefs by lazy { SecurePrefs(applicationContext) }
|
||||
private val forwardingLimiter = NotificationBurstLimiter()
|
||||
|
||||
override fun onListenerConnected() {
|
||||
super.onListenerConnected()
|
||||
activeService = this
|
||||
DeviceNotificationStore.setConnected(true)
|
||||
refreshActiveNotifications()
|
||||
}
|
||||
|
||||
override fun onListenerDisconnected() {
|
||||
if (activeService === this) {
|
||||
activeService = null
|
||||
}
|
||||
DeviceNotificationStore.setConnected(false)
|
||||
super.onListenerDisconnected()
|
||||
}
|
||||
|
||||
override fun onDestroy() {
|
||||
if (activeService === this) {
|
||||
activeService = null
|
||||
}
|
||||
super.onDestroy()
|
||||
}
|
||||
|
||||
override fun onNotificationPosted(sbn: StatusBarNotification?) {
|
||||
super.onNotificationPosted(sbn)
|
||||
val entry = sbn?.toEntry() ?: return
|
||||
DeviceNotificationStore.upsert(entry)
|
||||
rememberRecentPackage(entry.packageName)
|
||||
if (entry.packageName == packageName) {
|
||||
return
|
||||
}
|
||||
val payload = notificationChangedPayload(entry) ?: return
|
||||
emitNotificationsChanged(payload)
|
||||
}
|
||||
|
||||
override fun onNotificationRemoved(sbn: StatusBarNotification?) {
|
||||
super.onNotificationRemoved(sbn)
|
||||
val removed = sbn ?: return
|
||||
val key = removed.key.trim()
|
||||
if (key.isEmpty()) {
|
||||
return
|
||||
}
|
||||
DeviceNotificationStore.remove(key)
|
||||
rememberRecentPackage(removed.packageName)
|
||||
if (removed.packageName == packageName) {
|
||||
return
|
||||
}
|
||||
val packageName = removed.packageName.trim()
|
||||
val payload =
|
||||
notificationChangedPayload(
|
||||
entry = null,
|
||||
change = "removed",
|
||||
key = key,
|
||||
packageName = packageName,
|
||||
postTimeMs = removed.postTime,
|
||||
isOngoing = removed.isOngoing,
|
||||
isClearable = removed.isClearable,
|
||||
) ?: return
|
||||
emitNotificationsChanged(payload)
|
||||
}
|
||||
|
||||
private fun notificationChangedPayload(entry: DeviceNotificationEntry): String? {
|
||||
return notificationChangedPayload(
|
||||
entry = entry,
|
||||
change = "posted",
|
||||
key = entry.key,
|
||||
packageName = entry.packageName,
|
||||
postTimeMs = entry.postTimeMs,
|
||||
isOngoing = entry.isOngoing,
|
||||
isClearable = entry.isClearable,
|
||||
)
|
||||
}
|
||||
|
||||
private fun notificationChangedPayload(
|
||||
entry: DeviceNotificationEntry?,
|
||||
change: String,
|
||||
key: String,
|
||||
packageName: String,
|
||||
postTimeMs: Long,
|
||||
isOngoing: Boolean,
|
||||
isClearable: Boolean,
|
||||
): String? {
|
||||
val normalizedPackage = packageName.trim()
|
||||
if (normalizedPackage.isEmpty()) {
|
||||
return null
|
||||
}
|
||||
val policy = securePrefs.getNotificationForwardingPolicy(appPackageName = this.packageName)
|
||||
if (!policy.enabled) {
|
||||
return null
|
||||
}
|
||||
if (!policy.allowsPackage(normalizedPackage)) {
|
||||
return null
|
||||
}
|
||||
val nowEpochMs = System.currentTimeMillis()
|
||||
if (policy.isWithinQuietHours(nowEpochMs = nowEpochMs)) {
|
||||
return null
|
||||
}
|
||||
if (!forwardingLimiter.allow(nowEpochMs, policy.maxEventsPerMinute)) {
|
||||
return null
|
||||
}
|
||||
return buildJsonObject {
|
||||
put("change", JsonPrimitive(change))
|
||||
put("key", JsonPrimitive(key))
|
||||
put("packageName", JsonPrimitive(normalizedPackage))
|
||||
put("postTimeMs", JsonPrimitive(postTimeMs))
|
||||
put("isOngoing", JsonPrimitive(isOngoing))
|
||||
put("isClearable", JsonPrimitive(isClearable))
|
||||
policy.sessionKey?.let { put("sessionKey", JsonPrimitive(it)) }
|
||||
entry?.title?.let { put("title", JsonPrimitive(it)) }
|
||||
entry?.text?.let { put("text", JsonPrimitive(it)) }
|
||||
entry?.subText?.let { put("subText", JsonPrimitive(it)) }
|
||||
entry?.category?.let { put("category", JsonPrimitive(it)) }
|
||||
entry?.channelId?.let { put("channelId", JsonPrimitive(it)) }
|
||||
}.toString()
|
||||
}
|
||||
|
||||
private fun refreshActiveNotifications() {
|
||||
val entries =
|
||||
runCatching {
|
||||
activeNotifications
|
||||
?.mapNotNull { it.toEntry() }
|
||||
?: emptyList()
|
||||
}.getOrElse { emptyList() }
|
||||
DeviceNotificationStore.replace(entries)
|
||||
}
|
||||
|
||||
private fun StatusBarNotification.toEntry(): DeviceNotificationEntry {
|
||||
val extras = notification.extras
|
||||
val keyValue = key.takeIf { it.isNotBlank() } ?: "$packageName:$id:$postTime"
|
||||
val title = sanitizeNotificationText(extras?.getCharSequence(Notification.EXTRA_TITLE))
|
||||
val body =
|
||||
sanitizeNotificationText(extras?.getCharSequence(Notification.EXTRA_BIG_TEXT))
|
||||
?: sanitizeNotificationText(extras?.getCharSequence(Notification.EXTRA_TEXT))
|
||||
val subText = sanitizeNotificationText(extras?.getCharSequence(Notification.EXTRA_SUB_TEXT))
|
||||
return DeviceNotificationEntry(
|
||||
key = keyValue,
|
||||
packageName = packageName,
|
||||
title = title,
|
||||
text = body,
|
||||
subText = subText,
|
||||
category = notification.category?.trim()?.ifEmpty { null },
|
||||
channelId = notification.channelId?.trim()?.ifEmpty { null },
|
||||
postTimeMs = postTime,
|
||||
isOngoing = isOngoing,
|
||||
isClearable = isClearable,
|
||||
)
|
||||
}
|
||||
|
||||
companion object {
|
||||
private const val recentPackagesPref = "notifications.forwarding.recentPackages"
|
||||
private const val legacyRecentPackagesPref = "notifications.recentPackages"
|
||||
private const val recentPackagesLimit = 64
|
||||
@Volatile private var activeService: DeviceNotificationListenerService? = null
|
||||
@Volatile private var nodeEventSink: ((event: String, payloadJson: String?) -> Unit)? = null
|
||||
|
||||
private fun serviceComponent(context: Context): ComponentName {
|
||||
return ComponentName(context, DeviceNotificationListenerService::class.java)
|
||||
}
|
||||
|
||||
fun setNodeEventSink(sink: ((event: String, payloadJson: String?) -> Unit)?) {
|
||||
nodeEventSink = sink
|
||||
}
|
||||
|
||||
private fun recentPackagesPrefs(context: Context) =
|
||||
context.applicationContext.getSharedPreferences("openclaw.secure", Context.MODE_PRIVATE)
|
||||
|
||||
private fun migrateLegacyRecentPackagesIfNeeded(context: Context) {
|
||||
val prefs = recentPackagesPrefs(context)
|
||||
val hasNew = prefs.contains(recentPackagesPref)
|
||||
val legacy = prefs.getString(legacyRecentPackagesPref, null)?.trim().orEmpty()
|
||||
if (!hasNew && legacy.isNotEmpty()) {
|
||||
prefs.edit().putString(recentPackagesPref, legacy).remove(legacyRecentPackagesPref).apply()
|
||||
} else if (hasNew && prefs.contains(legacyRecentPackagesPref)) {
|
||||
prefs.edit().remove(legacyRecentPackagesPref).apply()
|
||||
}
|
||||
}
|
||||
|
||||
fun recentPackages(context: Context): List<String> {
|
||||
migrateLegacyRecentPackagesIfNeeded(context)
|
||||
val prefs = recentPackagesPrefs(context)
|
||||
val stored = prefs.getString(recentPackagesPref, null).orEmpty()
|
||||
return stored
|
||||
.split(',')
|
||||
.map { it.trim() }
|
||||
.filter { it.isNotEmpty() }
|
||||
.distinct()
|
||||
}
|
||||
|
||||
fun isAccessEnabled(context: Context): Boolean {
|
||||
val manager = context.getSystemService(NotificationManager::class.java) ?: return false
|
||||
return manager.isNotificationListenerAccessGranted(serviceComponent(context))
|
||||
}
|
||||
|
||||
fun snapshot(context: Context, enabled: Boolean = isAccessEnabled(context)): DeviceNotificationSnapshot {
|
||||
return DeviceNotificationStore.snapshot(enabled = enabled)
|
||||
}
|
||||
|
||||
fun requestServiceRebind(context: Context) {
|
||||
runCatching {
|
||||
NotificationListenerService.requestRebind(serviceComponent(context))
|
||||
}
|
||||
}
|
||||
|
||||
fun executeAction(context: Context, request: NotificationActionRequest): NotificationActionResult {
|
||||
if (!isAccessEnabled(context)) {
|
||||
return NotificationActionResult(
|
||||
ok = false,
|
||||
code = "NOTIFICATIONS_DISABLED",
|
||||
message = "NOTIFICATIONS_DISABLED: enable notification access in system Settings",
|
||||
)
|
||||
}
|
||||
val service = activeService
|
||||
?: return NotificationActionResult(
|
||||
ok = false,
|
||||
code = "NOTIFICATIONS_UNAVAILABLE",
|
||||
message = "NOTIFICATIONS_UNAVAILABLE: notification listener not connected",
|
||||
)
|
||||
return service.executeActionInternal(request)
|
||||
}
|
||||
|
||||
private fun emitNotificationsChanged(payloadJson: String) {
|
||||
runCatching {
|
||||
nodeEventSink?.invoke(NOTIFICATIONS_CHANGED_EVENT, payloadJson)
|
||||
}
|
||||
}
|
||||
|
||||
private fun rememberRecentPackage(packageName: String?) {
|
||||
val service = activeService ?: return
|
||||
val normalized = packageName?.trim().orEmpty()
|
||||
if (normalized.isEmpty() || normalized == service.packageName) return
|
||||
migrateLegacyRecentPackagesIfNeeded(service.applicationContext)
|
||||
val prefs = recentPackagesPrefs(service.applicationContext)
|
||||
val existing = prefs.getString(recentPackagesPref, null).orEmpty()
|
||||
.split(',')
|
||||
.map { it.trim() }
|
||||
.filter { it.isNotEmpty() && it != normalized }
|
||||
.take(recentPackagesLimit - 1)
|
||||
val updated = listOf(normalized) + existing
|
||||
prefs.edit().putString(recentPackagesPref, updated.joinToString(",")).apply()
|
||||
}
|
||||
}
|
||||
|
||||
private fun executeActionInternal(request: NotificationActionRequest): NotificationActionResult {
|
||||
val sbn =
|
||||
activeNotifications
|
||||
?.firstOrNull { it.key == request.key }
|
||||
?: return NotificationActionResult(
|
||||
ok = false,
|
||||
code = "NOTIFICATION_NOT_FOUND",
|
||||
message = "NOTIFICATION_NOT_FOUND: notification key not found",
|
||||
)
|
||||
if (actionRequiresClearableNotification(request.kind) && !sbn.isClearable) {
|
||||
return NotificationActionResult(
|
||||
ok = false,
|
||||
code = "NOTIFICATION_NOT_CLEARABLE",
|
||||
message = "NOTIFICATION_NOT_CLEARABLE: notification is ongoing or protected",
|
||||
)
|
||||
}
|
||||
|
||||
return when (request.kind) {
|
||||
NotificationActionKind.Open -> {
|
||||
val pendingIntent = sbn.notification.contentIntent
|
||||
?: return NotificationActionResult(
|
||||
ok = false,
|
||||
code = "ACTION_UNAVAILABLE",
|
||||
message = "ACTION_UNAVAILABLE: notification has no open action",
|
||||
)
|
||||
runCatching {
|
||||
pendingIntent.send()
|
||||
}.fold(
|
||||
onSuccess = { NotificationActionResult(ok = true) },
|
||||
onFailure = { err ->
|
||||
NotificationActionResult(
|
||||
ok = false,
|
||||
code = "ACTION_FAILED",
|
||||
message = "ACTION_FAILED: ${err.message ?: "open failed"}",
|
||||
)
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
NotificationActionKind.Dismiss -> {
|
||||
runCatching {
|
||||
cancelNotification(sbn.key)
|
||||
DeviceNotificationStore.remove(sbn.key)
|
||||
}.fold(
|
||||
onSuccess = { NotificationActionResult(ok = true) },
|
||||
onFailure = { err ->
|
||||
NotificationActionResult(
|
||||
ok = false,
|
||||
code = "ACTION_FAILED",
|
||||
message = "ACTION_FAILED: ${err.message ?: "dismiss failed"}",
|
||||
)
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
NotificationActionKind.Reply -> {
|
||||
val replyText = request.replyText?.trim().orEmpty()
|
||||
if (replyText.isEmpty()) {
|
||||
return NotificationActionResult(
|
||||
ok = false,
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: replyText required for reply action",
|
||||
)
|
||||
}
|
||||
val action =
|
||||
sbn.notification.actions
|
||||
?.firstOrNull { candidate ->
|
||||
candidate.actionIntent != null && !candidate.remoteInputs.isNullOrEmpty()
|
||||
}
|
||||
?: return NotificationActionResult(
|
||||
ok = false,
|
||||
code = "ACTION_UNAVAILABLE",
|
||||
message = "ACTION_UNAVAILABLE: notification has no reply action",
|
||||
)
|
||||
val remoteInputs = action.remoteInputs ?: emptyArray()
|
||||
val fillInIntent = Intent()
|
||||
val replyBundle = android.os.Bundle()
|
||||
for (remoteInput in remoteInputs) {
|
||||
replyBundle.putCharSequence(remoteInput.resultKey, replyText)
|
||||
}
|
||||
RemoteInput.addResultsToIntent(remoteInputs, fillInIntent, replyBundle)
|
||||
runCatching {
|
||||
action.actionIntent.send(this, 0, fillInIntent)
|
||||
}.fold(
|
||||
onSuccess = { NotificationActionResult(ok = true) },
|
||||
onFailure = { err ->
|
||||
NotificationActionResult(
|
||||
ok = false,
|
||||
code = "ACTION_FAILED",
|
||||
message = "ACTION_FAILED: ${err.message ?: "reply failed"}",
|
||||
)
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import ai.openclaw.app.SecurePrefs
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
|
||||
class GatewayEventHandler(
|
||||
private val scope: CoroutineScope,
|
||||
private val prefs: SecurePrefs,
|
||||
private val json: Json,
|
||||
private val operatorSession: GatewaySession,
|
||||
private val isConnected: () -> Boolean,
|
||||
) {
|
||||
private var suppressWakeWordsSync = false
|
||||
private var wakeWordsSyncJob: Job? = null
|
||||
|
||||
fun applyWakeWordsFromGateway(words: List<String>) {
|
||||
suppressWakeWordsSync = true
|
||||
prefs.setWakeWords(words)
|
||||
suppressWakeWordsSync = false
|
||||
}
|
||||
|
||||
fun scheduleWakeWordsSyncIfNeeded() {
|
||||
if (suppressWakeWordsSync) return
|
||||
if (!isConnected()) return
|
||||
|
||||
val snapshot = prefs.wakeWords.value
|
||||
wakeWordsSyncJob?.cancel()
|
||||
wakeWordsSyncJob =
|
||||
scope.launch {
|
||||
delay(650)
|
||||
val jsonList = snapshot.joinToString(separator = ",") { it.toJsonString() }
|
||||
val params = """{"triggers":[$jsonList]}"""
|
||||
try {
|
||||
operatorSession.request("voicewake.set", params)
|
||||
} catch (_: Throwable) {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun refreshWakeWordsFromGateway() {
|
||||
if (!isConnected()) return
|
||||
try {
|
||||
val res = operatorSession.request("voicewake.get", "{}")
|
||||
val payload = json.parseToJsonElement(res).asObjectOrNull() ?: return
|
||||
val array = payload["triggers"] as? JsonArray ?: return
|
||||
val triggers = array.mapNotNull { it.asStringOrNull() }
|
||||
applyWakeWordsFromGateway(triggers)
|
||||
} catch (_: Throwable) {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
|
||||
fun handleVoiceWakeChangedEvent(payloadJson: String?) {
|
||||
if (payloadJson.isNullOrBlank()) return
|
||||
try {
|
||||
val payload = json.parseToJsonElement(payloadJson).asObjectOrNull() ?: return
|
||||
val array = payload["triggers"] as? JsonArray ?: return
|
||||
val triggers = array.mapNotNull { it.asStringOrNull() }
|
||||
applyWakeWordsFromGateway(triggers)
|
||||
} catch (_: Throwable) {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,258 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import ai.openclaw.app.protocol.OpenClawCalendarCommand
|
||||
import ai.openclaw.app.protocol.OpenClawCanvasA2UICommand
|
||||
import ai.openclaw.app.protocol.OpenClawCanvasCommand
|
||||
import ai.openclaw.app.protocol.OpenClawCameraCommand
|
||||
import ai.openclaw.app.protocol.OpenClawCapability
|
||||
import ai.openclaw.app.protocol.OpenClawCallLogCommand
|
||||
import ai.openclaw.app.protocol.OpenClawContactsCommand
|
||||
import ai.openclaw.app.protocol.OpenClawDeviceCommand
|
||||
import ai.openclaw.app.protocol.OpenClawLocationCommand
|
||||
import ai.openclaw.app.protocol.OpenClawMotionCommand
|
||||
import ai.openclaw.app.protocol.OpenClawNotificationsCommand
|
||||
import ai.openclaw.app.protocol.OpenClawPhotosCommand
|
||||
import ai.openclaw.app.protocol.OpenClawSmsCommand
|
||||
import ai.openclaw.app.protocol.OpenClawSystemCommand
|
||||
|
||||
data class NodeRuntimeFlags(
|
||||
val cameraEnabled: Boolean,
|
||||
val locationEnabled: Boolean,
|
||||
val sendSmsAvailable: Boolean,
|
||||
val readSmsAvailable: Boolean,
|
||||
val smsSearchPossible: Boolean,
|
||||
val callLogAvailable: Boolean,
|
||||
val voiceWakeEnabled: Boolean,
|
||||
val motionActivityAvailable: Boolean,
|
||||
val motionPedometerAvailable: Boolean,
|
||||
val debugBuild: Boolean,
|
||||
)
|
||||
|
||||
enum class InvokeCommandAvailability {
|
||||
Always,
|
||||
CameraEnabled,
|
||||
LocationEnabled,
|
||||
SendSmsAvailable,
|
||||
ReadSmsAvailable,
|
||||
RequestableSmsSearchAvailable,
|
||||
CallLogAvailable,
|
||||
MotionActivityAvailable,
|
||||
MotionPedometerAvailable,
|
||||
DebugBuild,
|
||||
}
|
||||
|
||||
enum class NodeCapabilityAvailability {
|
||||
Always,
|
||||
CameraEnabled,
|
||||
LocationEnabled,
|
||||
SmsAvailable,
|
||||
CallLogAvailable,
|
||||
VoiceWakeEnabled,
|
||||
MotionAvailable,
|
||||
}
|
||||
|
||||
data class NodeCapabilitySpec(
|
||||
val name: String,
|
||||
val availability: NodeCapabilityAvailability = NodeCapabilityAvailability.Always,
|
||||
)
|
||||
|
||||
data class InvokeCommandSpec(
|
||||
val name: String,
|
||||
val requiresForeground: Boolean = false,
|
||||
val availability: InvokeCommandAvailability = InvokeCommandAvailability.Always,
|
||||
)
|
||||
|
||||
object InvokeCommandRegistry {
|
||||
val capabilityManifest: List<NodeCapabilitySpec> =
|
||||
listOf(
|
||||
NodeCapabilitySpec(name = OpenClawCapability.Canvas.rawValue),
|
||||
NodeCapabilitySpec(name = OpenClawCapability.Device.rawValue),
|
||||
NodeCapabilitySpec(name = OpenClawCapability.Notifications.rawValue),
|
||||
NodeCapabilitySpec(name = OpenClawCapability.System.rawValue),
|
||||
NodeCapabilitySpec(
|
||||
name = OpenClawCapability.Camera.rawValue,
|
||||
availability = NodeCapabilityAvailability.CameraEnabled,
|
||||
),
|
||||
NodeCapabilitySpec(
|
||||
name = OpenClawCapability.Sms.rawValue,
|
||||
availability = NodeCapabilityAvailability.SmsAvailable,
|
||||
),
|
||||
NodeCapabilitySpec(
|
||||
name = OpenClawCapability.VoiceWake.rawValue,
|
||||
availability = NodeCapabilityAvailability.VoiceWakeEnabled,
|
||||
),
|
||||
NodeCapabilitySpec(
|
||||
name = OpenClawCapability.Location.rawValue,
|
||||
availability = NodeCapabilityAvailability.LocationEnabled,
|
||||
),
|
||||
NodeCapabilitySpec(name = OpenClawCapability.Photos.rawValue),
|
||||
NodeCapabilitySpec(name = OpenClawCapability.Contacts.rawValue),
|
||||
NodeCapabilitySpec(name = OpenClawCapability.Calendar.rawValue),
|
||||
NodeCapabilitySpec(
|
||||
name = OpenClawCapability.Motion.rawValue,
|
||||
availability = NodeCapabilityAvailability.MotionAvailable,
|
||||
),
|
||||
NodeCapabilitySpec(
|
||||
name = OpenClawCapability.CallLog.rawValue,
|
||||
availability = NodeCapabilityAvailability.CallLogAvailable,
|
||||
),
|
||||
)
|
||||
|
||||
val all: List<InvokeCommandSpec> =
|
||||
listOf(
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCanvasCommand.Present.rawValue,
|
||||
requiresForeground = true,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCanvasCommand.Hide.rawValue,
|
||||
requiresForeground = true,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCanvasCommand.Navigate.rawValue,
|
||||
requiresForeground = true,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCanvasCommand.Eval.rawValue,
|
||||
requiresForeground = true,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCanvasCommand.Snapshot.rawValue,
|
||||
requiresForeground = true,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCanvasA2UICommand.Push.rawValue,
|
||||
requiresForeground = true,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCanvasA2UICommand.PushJSONL.rawValue,
|
||||
requiresForeground = true,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCanvasA2UICommand.Reset.rawValue,
|
||||
requiresForeground = true,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawSystemCommand.Notify.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCameraCommand.List.rawValue,
|
||||
requiresForeground = true,
|
||||
availability = InvokeCommandAvailability.CameraEnabled,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCameraCommand.Snap.rawValue,
|
||||
requiresForeground = true,
|
||||
availability = InvokeCommandAvailability.CameraEnabled,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCameraCommand.Clip.rawValue,
|
||||
requiresForeground = true,
|
||||
availability = InvokeCommandAvailability.CameraEnabled,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawLocationCommand.Get.rawValue,
|
||||
availability = InvokeCommandAvailability.LocationEnabled,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawDeviceCommand.Status.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawDeviceCommand.Info.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawDeviceCommand.Permissions.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawDeviceCommand.Health.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawNotificationsCommand.List.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawNotificationsCommand.Actions.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawPhotosCommand.Latest.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawContactsCommand.Search.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawContactsCommand.Add.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCalendarCommand.Events.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCalendarCommand.Add.rawValue,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawMotionCommand.Activity.rawValue,
|
||||
availability = InvokeCommandAvailability.MotionActivityAvailable,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawMotionCommand.Pedometer.rawValue,
|
||||
availability = InvokeCommandAvailability.MotionPedometerAvailable,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawSmsCommand.Send.rawValue,
|
||||
availability = InvokeCommandAvailability.SendSmsAvailable,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawSmsCommand.Search.rawValue,
|
||||
availability = InvokeCommandAvailability.RequestableSmsSearchAvailable,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = OpenClawCallLogCommand.Search.rawValue,
|
||||
availability = InvokeCommandAvailability.CallLogAvailable,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = "debug.logs",
|
||||
availability = InvokeCommandAvailability.DebugBuild,
|
||||
),
|
||||
InvokeCommandSpec(
|
||||
name = "debug.ed25519",
|
||||
availability = InvokeCommandAvailability.DebugBuild,
|
||||
),
|
||||
)
|
||||
|
||||
private val byNameInternal: Map<String, InvokeCommandSpec> = all.associateBy { it.name }
|
||||
|
||||
fun find(command: String): InvokeCommandSpec? = byNameInternal[command]
|
||||
|
||||
fun advertisedCapabilities(flags: NodeRuntimeFlags): List<String> {
|
||||
return capabilityManifest
|
||||
.filter { spec ->
|
||||
when (spec.availability) {
|
||||
NodeCapabilityAvailability.Always -> true
|
||||
NodeCapabilityAvailability.CameraEnabled -> flags.cameraEnabled
|
||||
NodeCapabilityAvailability.LocationEnabled -> flags.locationEnabled
|
||||
NodeCapabilityAvailability.SmsAvailable -> flags.sendSmsAvailable || flags.readSmsAvailable
|
||||
NodeCapabilityAvailability.CallLogAvailable -> flags.callLogAvailable
|
||||
NodeCapabilityAvailability.VoiceWakeEnabled -> flags.voiceWakeEnabled
|
||||
NodeCapabilityAvailability.MotionAvailable -> flags.motionActivityAvailable || flags.motionPedometerAvailable
|
||||
}
|
||||
}
|
||||
.map { it.name }
|
||||
}
|
||||
|
||||
fun advertisedCommands(flags: NodeRuntimeFlags): List<String> {
|
||||
return all
|
||||
.filter { spec ->
|
||||
when (spec.availability) {
|
||||
InvokeCommandAvailability.Always -> true
|
||||
InvokeCommandAvailability.CameraEnabled -> flags.cameraEnabled
|
||||
InvokeCommandAvailability.LocationEnabled -> flags.locationEnabled
|
||||
InvokeCommandAvailability.SendSmsAvailable -> flags.sendSmsAvailable
|
||||
InvokeCommandAvailability.ReadSmsAvailable -> flags.readSmsAvailable
|
||||
InvokeCommandAvailability.RequestableSmsSearchAvailable -> flags.smsSearchPossible
|
||||
InvokeCommandAvailability.CallLogAvailable -> flags.callLogAvailable
|
||||
InvokeCommandAvailability.MotionActivityAvailable -> flags.motionActivityAvailable
|
||||
InvokeCommandAvailability.MotionPedometerAvailable -> flags.motionPedometerAvailable
|
||||
InvokeCommandAvailability.DebugBuild -> flags.debugBuild
|
||||
}
|
||||
}
|
||||
.map { it.name }
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,338 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import ai.openclaw.app.protocol.OpenClawCalendarCommand
|
||||
import ai.openclaw.app.protocol.OpenClawCanvasA2UICommand
|
||||
import ai.openclaw.app.protocol.OpenClawCanvasCommand
|
||||
import ai.openclaw.app.protocol.OpenClawCameraCommand
|
||||
import ai.openclaw.app.protocol.OpenClawCallLogCommand
|
||||
import ai.openclaw.app.protocol.OpenClawContactsCommand
|
||||
import ai.openclaw.app.protocol.OpenClawDeviceCommand
|
||||
import ai.openclaw.app.protocol.OpenClawLocationCommand
|
||||
import ai.openclaw.app.protocol.OpenClawMotionCommand
|
||||
import ai.openclaw.app.protocol.OpenClawNotificationsCommand
|
||||
import ai.openclaw.app.protocol.OpenClawSmsCommand
|
||||
import ai.openclaw.app.protocol.OpenClawSystemCommand
|
||||
|
||||
internal enum class SmsSearchAvailabilityReason {
|
||||
Available,
|
||||
PermissionRequired,
|
||||
Unavailable,
|
||||
}
|
||||
|
||||
internal fun classifySmsSearchAvailability(
|
||||
readSmsAvailable: Boolean,
|
||||
smsFeatureEnabled: Boolean,
|
||||
smsTelephonyAvailable: Boolean,
|
||||
): SmsSearchAvailabilityReason {
|
||||
if (readSmsAvailable) return SmsSearchAvailabilityReason.Available
|
||||
if (!smsFeatureEnabled || !smsTelephonyAvailable) return SmsSearchAvailabilityReason.Unavailable
|
||||
return SmsSearchAvailabilityReason.PermissionRequired
|
||||
}
|
||||
|
||||
internal fun smsSearchAvailabilityError(
|
||||
readSmsAvailable: Boolean,
|
||||
smsFeatureEnabled: Boolean,
|
||||
smsTelephonyAvailable: Boolean,
|
||||
): GatewaySession.InvokeResult? {
|
||||
return when (
|
||||
classifySmsSearchAvailability(
|
||||
readSmsAvailable = readSmsAvailable,
|
||||
smsFeatureEnabled = smsFeatureEnabled,
|
||||
smsTelephonyAvailable = smsTelephonyAvailable,
|
||||
)
|
||||
) {
|
||||
SmsSearchAvailabilityReason.Available,
|
||||
SmsSearchAvailabilityReason.PermissionRequired -> null
|
||||
SmsSearchAvailabilityReason.Unavailable ->
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "SMS_UNAVAILABLE",
|
||||
message = "SMS_UNAVAILABLE: SMS not available on this device",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
class InvokeDispatcher(
|
||||
private val canvas: CanvasController,
|
||||
private val cameraHandler: CameraHandler,
|
||||
private val locationHandler: LocationHandler,
|
||||
private val deviceHandler: DeviceHandler,
|
||||
private val notificationsHandler: NotificationsHandler,
|
||||
private val systemHandler: SystemHandler,
|
||||
private val photosHandler: PhotosHandler,
|
||||
private val contactsHandler: ContactsHandler,
|
||||
private val calendarHandler: CalendarHandler,
|
||||
private val motionHandler: MotionHandler,
|
||||
private val smsHandler: SmsHandler,
|
||||
private val a2uiHandler: A2UIHandler,
|
||||
private val debugHandler: DebugHandler,
|
||||
private val callLogHandler: CallLogHandler,
|
||||
private val isForeground: () -> Boolean,
|
||||
private val cameraEnabled: () -> Boolean,
|
||||
private val locationEnabled: () -> Boolean,
|
||||
private val sendSmsAvailable: () -> Boolean,
|
||||
private val readSmsAvailable: () -> Boolean,
|
||||
private val smsFeatureEnabled: () -> Boolean,
|
||||
private val smsTelephonyAvailable: () -> Boolean,
|
||||
private val callLogAvailable: () -> Boolean,
|
||||
private val debugBuild: () -> Boolean,
|
||||
private val refreshNodeCanvasCapability: suspend () -> Boolean,
|
||||
private val onCanvasA2uiPush: () -> Unit,
|
||||
private val onCanvasA2uiReset: () -> Unit,
|
||||
private val motionActivityAvailable: () -> Boolean,
|
||||
private val motionPedometerAvailable: () -> Boolean,
|
||||
) {
|
||||
suspend fun handleInvoke(command: String, paramsJson: String?): GatewaySession.InvokeResult {
|
||||
val spec =
|
||||
InvokeCommandRegistry.find(command)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: unknown command",
|
||||
)
|
||||
if (spec.requiresForeground && !isForeground()) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "NODE_BACKGROUND_UNAVAILABLE",
|
||||
message = "NODE_BACKGROUND_UNAVAILABLE: canvas/camera/screen commands require foreground",
|
||||
)
|
||||
}
|
||||
availabilityError(spec.availability)?.let { return it }
|
||||
|
||||
return when (command) {
|
||||
// Canvas commands
|
||||
OpenClawCanvasCommand.Present.rawValue -> {
|
||||
val url = CanvasController.parseNavigateUrl(paramsJson)
|
||||
canvas.navigate(url)
|
||||
GatewaySession.InvokeResult.ok(null)
|
||||
}
|
||||
OpenClawCanvasCommand.Hide.rawValue -> GatewaySession.InvokeResult.ok(null)
|
||||
OpenClawCanvasCommand.Navigate.rawValue -> {
|
||||
val url = CanvasController.parseNavigateUrl(paramsJson)
|
||||
canvas.navigate(url)
|
||||
GatewaySession.InvokeResult.ok(null)
|
||||
}
|
||||
OpenClawCanvasCommand.Eval.rawValue -> {
|
||||
val js =
|
||||
CanvasController.parseEvalJs(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: javaScript required",
|
||||
)
|
||||
withCanvasAvailable {
|
||||
val result = canvas.eval(js)
|
||||
GatewaySession.InvokeResult.ok("""{"result":${result.toJsonString()}}""")
|
||||
}
|
||||
}
|
||||
OpenClawCanvasCommand.Snapshot.rawValue -> {
|
||||
val snapshotParams = CanvasController.parseSnapshotParams(paramsJson)
|
||||
withCanvasAvailable {
|
||||
val base64 =
|
||||
canvas.snapshotBase64(
|
||||
format = snapshotParams.format,
|
||||
quality = snapshotParams.quality,
|
||||
maxWidth = snapshotParams.maxWidth,
|
||||
)
|
||||
GatewaySession.InvokeResult.ok("""{"format":"${snapshotParams.format.rawValue}","base64":"$base64"}""")
|
||||
}
|
||||
}
|
||||
|
||||
// A2UI commands
|
||||
OpenClawCanvasA2UICommand.Reset.rawValue ->
|
||||
withReadyA2ui {
|
||||
withCanvasAvailable {
|
||||
val res = canvas.eval(A2UIHandler.a2uiResetJS)
|
||||
onCanvasA2uiReset()
|
||||
GatewaySession.InvokeResult.ok(res)
|
||||
}
|
||||
}
|
||||
OpenClawCanvasA2UICommand.Push.rawValue, OpenClawCanvasA2UICommand.PushJSONL.rawValue -> {
|
||||
val messages =
|
||||
try {
|
||||
a2uiHandler.decodeA2uiMessages(command, paramsJson)
|
||||
} catch (err: Throwable) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = err.message ?: "invalid A2UI payload"
|
||||
)
|
||||
}
|
||||
withReadyA2ui {
|
||||
withCanvasAvailable {
|
||||
val js = A2UIHandler.a2uiApplyMessagesJS(messages)
|
||||
val res = canvas.eval(js)
|
||||
onCanvasA2uiPush()
|
||||
GatewaySession.InvokeResult.ok(res)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Camera commands
|
||||
OpenClawCameraCommand.List.rawValue -> cameraHandler.handleList(paramsJson)
|
||||
OpenClawCameraCommand.Snap.rawValue -> cameraHandler.handleSnap(paramsJson)
|
||||
OpenClawCameraCommand.Clip.rawValue -> cameraHandler.handleClip(paramsJson)
|
||||
|
||||
// Location command
|
||||
OpenClawLocationCommand.Get.rawValue -> locationHandler.handleLocationGet(paramsJson)
|
||||
|
||||
// Device commands
|
||||
OpenClawDeviceCommand.Status.rawValue -> deviceHandler.handleDeviceStatus(paramsJson)
|
||||
OpenClawDeviceCommand.Info.rawValue -> deviceHandler.handleDeviceInfo(paramsJson)
|
||||
OpenClawDeviceCommand.Permissions.rawValue -> deviceHandler.handleDevicePermissions(paramsJson)
|
||||
OpenClawDeviceCommand.Health.rawValue -> deviceHandler.handleDeviceHealth(paramsJson)
|
||||
|
||||
// Notifications command
|
||||
OpenClawNotificationsCommand.List.rawValue -> notificationsHandler.handleNotificationsList(paramsJson)
|
||||
OpenClawNotificationsCommand.Actions.rawValue -> notificationsHandler.handleNotificationsActions(paramsJson)
|
||||
|
||||
// System command
|
||||
OpenClawSystemCommand.Notify.rawValue -> systemHandler.handleSystemNotify(paramsJson)
|
||||
|
||||
// Photos command
|
||||
ai.openclaw.app.protocol.OpenClawPhotosCommand.Latest.rawValue -> photosHandler.handlePhotosLatest(
|
||||
paramsJson,
|
||||
)
|
||||
|
||||
// Contacts command
|
||||
OpenClawContactsCommand.Search.rawValue -> contactsHandler.handleContactsSearch(paramsJson)
|
||||
OpenClawContactsCommand.Add.rawValue -> contactsHandler.handleContactsAdd(paramsJson)
|
||||
|
||||
// Calendar command
|
||||
OpenClawCalendarCommand.Events.rawValue -> calendarHandler.handleCalendarEvents(paramsJson)
|
||||
OpenClawCalendarCommand.Add.rawValue -> calendarHandler.handleCalendarAdd(paramsJson)
|
||||
|
||||
// Motion command
|
||||
OpenClawMotionCommand.Activity.rawValue -> motionHandler.handleMotionActivity(paramsJson)
|
||||
OpenClawMotionCommand.Pedometer.rawValue -> motionHandler.handleMotionPedometer(paramsJson)
|
||||
|
||||
// SMS command
|
||||
OpenClawSmsCommand.Send.rawValue -> smsHandler.handleSmsSend(paramsJson)
|
||||
OpenClawSmsCommand.Search.rawValue -> smsHandler.handleSmsSearch(paramsJson)
|
||||
|
||||
// CallLog command
|
||||
OpenClawCallLogCommand.Search.rawValue -> callLogHandler.handleCallLogSearch(paramsJson)
|
||||
|
||||
// Debug commands
|
||||
"debug.ed25519" -> debugHandler.handleEd25519()
|
||||
"debug.logs" -> debugHandler.handleLogs()
|
||||
else -> GatewaySession.InvokeResult.error(code = "INVALID_REQUEST", message = "INVALID_REQUEST: unknown command")
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun withReadyA2ui(
|
||||
block: suspend () -> GatewaySession.InvokeResult,
|
||||
): GatewaySession.InvokeResult {
|
||||
var a2uiUrl = a2uiHandler.resolveA2uiHostUrl()
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "A2UI_HOST_NOT_CONFIGURED",
|
||||
message = "A2UI_HOST_NOT_CONFIGURED: gateway did not advertise canvas host",
|
||||
)
|
||||
val readyOnFirstCheck = a2uiHandler.ensureA2uiReady(a2uiUrl)
|
||||
if (!readyOnFirstCheck) {
|
||||
if (!refreshNodeCanvasCapability()) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "A2UI_HOST_UNAVAILABLE",
|
||||
message = "A2UI_HOST_UNAVAILABLE: A2UI host not reachable",
|
||||
)
|
||||
}
|
||||
a2uiUrl = a2uiHandler.resolveA2uiHostUrl()
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "A2UI_HOST_NOT_CONFIGURED",
|
||||
message = "A2UI_HOST_NOT_CONFIGURED: gateway did not advertise canvas host",
|
||||
)
|
||||
if (!a2uiHandler.ensureA2uiReady(a2uiUrl)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "A2UI_HOST_UNAVAILABLE",
|
||||
message = "A2UI_HOST_UNAVAILABLE: A2UI host not reachable",
|
||||
)
|
||||
}
|
||||
}
|
||||
return block()
|
||||
}
|
||||
|
||||
private suspend fun withCanvasAvailable(
|
||||
block: suspend () -> GatewaySession.InvokeResult,
|
||||
): GatewaySession.InvokeResult {
|
||||
return try {
|
||||
block()
|
||||
} catch (_: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "NODE_BACKGROUND_UNAVAILABLE",
|
||||
message = "NODE_BACKGROUND_UNAVAILABLE: canvas unavailable",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun availabilityError(availability: InvokeCommandAvailability): GatewaySession.InvokeResult? {
|
||||
return when (availability) {
|
||||
InvokeCommandAvailability.Always -> null
|
||||
InvokeCommandAvailability.CameraEnabled ->
|
||||
if (cameraEnabled()) {
|
||||
null
|
||||
} else {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "CAMERA_DISABLED",
|
||||
message = "CAMERA_DISABLED: enable Camera in Settings",
|
||||
)
|
||||
}
|
||||
InvokeCommandAvailability.LocationEnabled ->
|
||||
if (locationEnabled()) {
|
||||
null
|
||||
} else {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "LOCATION_DISABLED",
|
||||
message = "LOCATION_DISABLED: enable Location in Settings",
|
||||
)
|
||||
}
|
||||
InvokeCommandAvailability.MotionActivityAvailable ->
|
||||
if (motionActivityAvailable()) {
|
||||
null
|
||||
} else {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "MOTION_UNAVAILABLE",
|
||||
message = "MOTION_UNAVAILABLE: accelerometer not available",
|
||||
)
|
||||
}
|
||||
InvokeCommandAvailability.MotionPedometerAvailable ->
|
||||
if (motionPedometerAvailable()) {
|
||||
null
|
||||
} else {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "PEDOMETER_UNAVAILABLE",
|
||||
message = "PEDOMETER_UNAVAILABLE: step counter not available",
|
||||
)
|
||||
}
|
||||
InvokeCommandAvailability.SendSmsAvailable ->
|
||||
if (sendSmsAvailable()) {
|
||||
null
|
||||
} else {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "SMS_UNAVAILABLE",
|
||||
message = "SMS_UNAVAILABLE: SMS not available on this device",
|
||||
)
|
||||
}
|
||||
InvokeCommandAvailability.ReadSmsAvailable,
|
||||
InvokeCommandAvailability.RequestableSmsSearchAvailable ->
|
||||
smsSearchAvailabilityError(
|
||||
readSmsAvailable = readSmsAvailable(),
|
||||
smsFeatureEnabled = smsFeatureEnabled(),
|
||||
smsTelephonyAvailable = smsTelephonyAvailable(),
|
||||
)
|
||||
InvokeCommandAvailability.CallLogAvailable ->
|
||||
if (callLogAvailable()) {
|
||||
null
|
||||
} else {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "CALL_LOG_UNAVAILABLE",
|
||||
message = "CALL_LOG_UNAVAILABLE: call log not available on this build",
|
||||
)
|
||||
}
|
||||
InvokeCommandAvailability.DebugBuild ->
|
||||
if (debugBuild()) {
|
||||
null
|
||||
} else {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: unknown command",
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import kotlin.math.max
|
||||
import kotlin.math.min
|
||||
import kotlin.math.roundToInt
|
||||
|
||||
internal data class JpegSizeLimiterResult(
|
||||
val bytes: ByteArray,
|
||||
val width: Int,
|
||||
val height: Int,
|
||||
val quality: Int,
|
||||
)
|
||||
|
||||
internal object JpegSizeLimiter {
|
||||
fun compressToLimit(
|
||||
initialWidth: Int,
|
||||
initialHeight: Int,
|
||||
startQuality: Int,
|
||||
maxBytes: Int,
|
||||
minQuality: Int = 20,
|
||||
minSize: Int = 256,
|
||||
scaleStep: Double = 0.85,
|
||||
maxScaleAttempts: Int = 6,
|
||||
maxQualityAttempts: Int = 6,
|
||||
encode: (width: Int, height: Int, quality: Int) -> ByteArray,
|
||||
): JpegSizeLimiterResult {
|
||||
require(initialWidth > 0 && initialHeight > 0) { "Invalid image size" }
|
||||
require(maxBytes > 0) { "Invalid maxBytes" }
|
||||
|
||||
var width = initialWidth
|
||||
var height = initialHeight
|
||||
val clampedStartQuality = startQuality.coerceIn(minQuality, 100)
|
||||
var best = JpegSizeLimiterResult(bytes = encode(width, height, clampedStartQuality), width = width, height = height, quality = clampedStartQuality)
|
||||
if (best.bytes.size <= maxBytes) return best
|
||||
|
||||
repeat(maxScaleAttempts) {
|
||||
var quality = clampedStartQuality
|
||||
repeat(maxQualityAttempts) {
|
||||
val bytes = encode(width, height, quality)
|
||||
best = JpegSizeLimiterResult(bytes = bytes, width = width, height = height, quality = quality)
|
||||
if (bytes.size <= maxBytes) return best
|
||||
if (quality <= minQuality) return@repeat
|
||||
quality = max(minQuality, (quality * 0.75).roundToInt())
|
||||
}
|
||||
|
||||
val minScale = (minSize.toDouble() / min(width, height).toDouble()).coerceAtMost(1.0)
|
||||
val nextScale = max(scaleStep, minScale)
|
||||
val nextWidth = max(minSize, (width * nextScale).roundToInt())
|
||||
val nextHeight = max(minSize, (height * nextScale).roundToInt())
|
||||
if (nextWidth == width && nextHeight == height) return@repeat
|
||||
width = min(nextWidth, width)
|
||||
height = min(nextHeight, height)
|
||||
}
|
||||
|
||||
if (best.bytes.size > maxBytes) {
|
||||
throw IllegalStateException("CAMERA_TOO_LARGE: ${best.bytes.size} bytes > $maxBytes bytes")
|
||||
}
|
||||
|
||||
return best
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,112 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.content.Context
|
||||
import android.content.pm.PackageManager
|
||||
import android.location.Location
|
||||
import android.location.LocationManager
|
||||
import android.os.CancellationSignal
|
||||
import androidx.core.content.ContextCompat
|
||||
import java.time.Instant
|
||||
import java.time.format.DateTimeFormatter
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.coroutines.withTimeout
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
|
||||
class LocationCaptureManager(private val context: Context) {
|
||||
data class Payload(val payloadJson: String)
|
||||
|
||||
suspend fun getLocation(
|
||||
desiredProviders: List<String>,
|
||||
maxAgeMs: Long?,
|
||||
timeoutMs: Long,
|
||||
isPrecise: Boolean,
|
||||
): Payload =
|
||||
withContext(Dispatchers.Main) {
|
||||
val manager = context.getSystemService(Context.LOCATION_SERVICE) as LocationManager
|
||||
if (!manager.isProviderEnabled(LocationManager.GPS_PROVIDER) &&
|
||||
!manager.isProviderEnabled(LocationManager.NETWORK_PROVIDER)
|
||||
) {
|
||||
throw IllegalStateException("LOCATION_UNAVAILABLE: no location providers enabled")
|
||||
}
|
||||
|
||||
val cached = bestLastKnown(manager, desiredProviders, maxAgeMs)
|
||||
val location =
|
||||
cached ?: requestCurrent(manager, desiredProviders, timeoutMs)
|
||||
|
||||
val timestamp = DateTimeFormatter.ISO_INSTANT.format(Instant.ofEpochMilli(location.time))
|
||||
val source = location.provider
|
||||
val altitudeMeters = if (location.hasAltitude()) location.altitude else null
|
||||
val speedMps = if (location.hasSpeed()) location.speed.toDouble() else null
|
||||
val headingDeg = if (location.hasBearing()) location.bearing.toDouble() else null
|
||||
Payload(
|
||||
buildString {
|
||||
append("{\"lat\":")
|
||||
append(location.latitude)
|
||||
append(",\"lon\":")
|
||||
append(location.longitude)
|
||||
append(",\"accuracyMeters\":")
|
||||
append(location.accuracy.toDouble())
|
||||
if (altitudeMeters != null) append(",\"altitudeMeters\":").append(altitudeMeters)
|
||||
if (speedMps != null) append(",\"speedMps\":").append(speedMps)
|
||||
if (headingDeg != null) append(",\"headingDeg\":").append(headingDeg)
|
||||
append(",\"timestamp\":\"").append(timestamp).append('"')
|
||||
append(",\"isPrecise\":").append(isPrecise)
|
||||
append(",\"source\":\"").append(source).append('"')
|
||||
append('}')
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
private fun bestLastKnown(
|
||||
manager: LocationManager,
|
||||
providers: List<String>,
|
||||
maxAgeMs: Long?,
|
||||
): Location? {
|
||||
val fineOk =
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_FINE_LOCATION) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
val coarseOk =
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_COARSE_LOCATION) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
if (!fineOk && !coarseOk) {
|
||||
throw IllegalStateException("LOCATION_PERMISSION_REQUIRED: grant Location permission")
|
||||
}
|
||||
val now = System.currentTimeMillis()
|
||||
val candidates =
|
||||
providers.mapNotNull { provider -> manager.getLastKnownLocation(provider) }
|
||||
val freshest = candidates.maxByOrNull { it.time } ?: return null
|
||||
if (maxAgeMs != null && now - freshest.time > maxAgeMs) return null
|
||||
return freshest
|
||||
}
|
||||
|
||||
private suspend fun requestCurrent(
|
||||
manager: LocationManager,
|
||||
providers: List<String>,
|
||||
timeoutMs: Long,
|
||||
): Location {
|
||||
val fineOk =
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_FINE_LOCATION) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
val coarseOk =
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_COARSE_LOCATION) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
if (!fineOk && !coarseOk) {
|
||||
throw IllegalStateException("LOCATION_PERMISSION_REQUIRED: grant Location permission")
|
||||
}
|
||||
val resolved =
|
||||
providers.firstOrNull { manager.isProviderEnabled(it) }
|
||||
?: throw IllegalStateException("LOCATION_UNAVAILABLE: no providers available")
|
||||
val location = withTimeout(timeoutMs.coerceAtLeast(1)) {
|
||||
suspendCancellableCoroutine<Location?> { cont ->
|
||||
val signal = CancellationSignal()
|
||||
cont.invokeOnCancellation { signal.cancel() }
|
||||
manager.getCurrentLocation(resolved, signal, context.mainExecutor) { location ->
|
||||
cont.resume(location) { _, _, _ -> }
|
||||
}
|
||||
}
|
||||
}
|
||||
return location ?: throw IllegalStateException("LOCATION_UNAVAILABLE: no fix")
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,158 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.content.Context
|
||||
import android.content.pm.PackageManager
|
||||
import android.location.LocationManager
|
||||
import androidx.core.content.ContextCompat
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.coroutines.TimeoutCancellationException
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
|
||||
internal interface LocationDataSource {
|
||||
fun hasFinePermission(context: Context): Boolean
|
||||
|
||||
fun hasCoarsePermission(context: Context): Boolean
|
||||
|
||||
suspend fun fetchLocation(
|
||||
desiredProviders: List<String>,
|
||||
maxAgeMs: Long?,
|
||||
timeoutMs: Long,
|
||||
isPrecise: Boolean,
|
||||
): LocationCaptureManager.Payload
|
||||
}
|
||||
|
||||
private class DefaultLocationDataSource(
|
||||
private val capture: LocationCaptureManager,
|
||||
) : LocationDataSource {
|
||||
override fun hasFinePermission(context: Context): Boolean =
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_FINE_LOCATION) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
|
||||
override fun hasCoarsePermission(context: Context): Boolean =
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_COARSE_LOCATION) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
|
||||
override suspend fun fetchLocation(
|
||||
desiredProviders: List<String>,
|
||||
maxAgeMs: Long?,
|
||||
timeoutMs: Long,
|
||||
isPrecise: Boolean,
|
||||
): LocationCaptureManager.Payload =
|
||||
capture.getLocation(
|
||||
desiredProviders = desiredProviders,
|
||||
maxAgeMs = maxAgeMs,
|
||||
timeoutMs = timeoutMs,
|
||||
isPrecise = isPrecise,
|
||||
)
|
||||
}
|
||||
|
||||
class LocationHandler private constructor(
|
||||
private val appContext: Context,
|
||||
private val dataSource: LocationDataSource,
|
||||
private val json: Json,
|
||||
private val isForeground: () -> Boolean,
|
||||
private val locationPreciseEnabled: () -> Boolean,
|
||||
) {
|
||||
constructor(
|
||||
appContext: Context,
|
||||
location: LocationCaptureManager,
|
||||
json: Json,
|
||||
isForeground: () -> Boolean,
|
||||
locationPreciseEnabled: () -> Boolean,
|
||||
) : this(
|
||||
appContext = appContext,
|
||||
dataSource = DefaultLocationDataSource(location),
|
||||
json = json,
|
||||
isForeground = isForeground,
|
||||
locationPreciseEnabled = locationPreciseEnabled,
|
||||
)
|
||||
|
||||
fun hasFineLocationPermission(): Boolean = dataSource.hasFinePermission(appContext)
|
||||
|
||||
fun hasCoarseLocationPermission(): Boolean = dataSource.hasCoarsePermission(appContext)
|
||||
|
||||
companion object {
|
||||
internal fun forTesting(
|
||||
appContext: Context,
|
||||
dataSource: LocationDataSource,
|
||||
json: Json = Json { ignoreUnknownKeys = true },
|
||||
isForeground: () -> Boolean = { true },
|
||||
locationPreciseEnabled: () -> Boolean = { true },
|
||||
): LocationHandler =
|
||||
LocationHandler(
|
||||
appContext = appContext,
|
||||
dataSource = dataSource,
|
||||
json = json,
|
||||
isForeground = isForeground,
|
||||
locationPreciseEnabled = locationPreciseEnabled,
|
||||
)
|
||||
}
|
||||
|
||||
suspend fun handleLocationGet(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!isForeground()) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "LOCATION_BACKGROUND_UNAVAILABLE",
|
||||
message = "LOCATION_BACKGROUND_UNAVAILABLE: location requires OpenClaw to stay open",
|
||||
)
|
||||
}
|
||||
if (!dataSource.hasFinePermission(appContext) && !dataSource.hasCoarsePermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "LOCATION_PERMISSION_REQUIRED",
|
||||
message = "LOCATION_PERMISSION_REQUIRED: grant Location permission",
|
||||
)
|
||||
}
|
||||
val (maxAgeMs, timeoutMs, desiredAccuracy) = parseLocationParams(paramsJson)
|
||||
val preciseEnabled = locationPreciseEnabled()
|
||||
val accuracy =
|
||||
when (desiredAccuracy) {
|
||||
"precise" -> if (preciseEnabled && dataSource.hasFinePermission(appContext)) "precise" else "balanced"
|
||||
"coarse" -> "coarse"
|
||||
else -> if (preciseEnabled && dataSource.hasFinePermission(appContext)) "precise" else "balanced"
|
||||
}
|
||||
val providers =
|
||||
when (accuracy) {
|
||||
"precise" -> listOf(LocationManager.GPS_PROVIDER, LocationManager.NETWORK_PROVIDER)
|
||||
"coarse" -> listOf(LocationManager.NETWORK_PROVIDER, LocationManager.GPS_PROVIDER)
|
||||
else -> listOf(LocationManager.NETWORK_PROVIDER, LocationManager.GPS_PROVIDER)
|
||||
}
|
||||
try {
|
||||
val payload =
|
||||
dataSource.fetchLocation(
|
||||
desiredProviders = providers,
|
||||
maxAgeMs = maxAgeMs,
|
||||
timeoutMs = timeoutMs,
|
||||
isPrecise = accuracy == "precise",
|
||||
)
|
||||
return GatewaySession.InvokeResult.ok(payload.payloadJson)
|
||||
} catch (err: TimeoutCancellationException) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "LOCATION_TIMEOUT",
|
||||
message = "LOCATION_TIMEOUT: no fix in time",
|
||||
)
|
||||
} catch (err: Throwable) {
|
||||
val message = err.message ?: "LOCATION_UNAVAILABLE: no fix"
|
||||
return GatewaySession.InvokeResult.error(code = "LOCATION_UNAVAILABLE", message = message)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseLocationParams(paramsJson: String?): Triple<Long?, Long, String?> {
|
||||
if (paramsJson.isNullOrBlank()) {
|
||||
return Triple(null, 10_000L, null)
|
||||
}
|
||||
val root =
|
||||
try {
|
||||
json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
val maxAgeMs = (root?.get("maxAgeMs") as? JsonPrimitive)?.content?.toLongOrNull()
|
||||
val timeoutMs =
|
||||
(root?.get("timeoutMs") as? JsonPrimitive)?.content?.toLongOrNull()?.coerceIn(1_000L, 60_000L)
|
||||
?: 10_000L
|
||||
val desiredAccuracy =
|
||||
(root?.get("desiredAccuracy") as? JsonPrimitive)?.content?.trim()?.lowercase()
|
||||
return Triple(maxAgeMs, timeoutMs, desiredAccuracy)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,371 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.content.Context
|
||||
import android.hardware.Sensor
|
||||
import android.hardware.SensorEvent
|
||||
import android.hardware.SensorEventListener
|
||||
import android.hardware.SensorManager
|
||||
import android.os.SystemClock
|
||||
import androidx.core.content.ContextCompat
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import java.time.Instant
|
||||
import kotlinx.coroutines.InternalCoroutinesApi
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
import kotlinx.coroutines.withTimeoutOrNull
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonArray
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.put
|
||||
import kotlin.math.abs
|
||||
import kotlin.math.max
|
||||
import kotlin.math.sqrt
|
||||
|
||||
private const val ACCELEROMETER_SAMPLE_TARGET = 20
|
||||
private const val ACCELEROMETER_SAMPLE_TIMEOUT_MS = 6_000L
|
||||
|
||||
internal data class MotionActivityRequest(
|
||||
val startISO: String?,
|
||||
val endISO: String?,
|
||||
val limit: Int,
|
||||
)
|
||||
|
||||
internal data class MotionPedometerRequest(
|
||||
val startISO: String?,
|
||||
val endISO: String?,
|
||||
)
|
||||
|
||||
internal data class MotionActivityRecord(
|
||||
val startISO: String,
|
||||
val endISO: String,
|
||||
val confidence: String,
|
||||
val isWalking: Boolean,
|
||||
val isRunning: Boolean,
|
||||
val isCycling: Boolean,
|
||||
val isAutomotive: Boolean,
|
||||
val isStationary: Boolean,
|
||||
val isUnknown: Boolean,
|
||||
)
|
||||
|
||||
internal data class PedometerRecord(
|
||||
val startISO: String,
|
||||
val endISO: String,
|
||||
val steps: Int?,
|
||||
val distanceMeters: Double?,
|
||||
val floorsAscended: Int?,
|
||||
val floorsDescended: Int?,
|
||||
)
|
||||
|
||||
internal interface MotionDataSource {
|
||||
fun isActivityAvailable(context: Context): Boolean
|
||||
|
||||
fun isPedometerAvailable(context: Context): Boolean
|
||||
|
||||
fun isAvailable(context: Context): Boolean = isActivityAvailable(context) || isPedometerAvailable(context)
|
||||
|
||||
fun hasPermission(context: Context): Boolean
|
||||
|
||||
suspend fun activity(context: Context, request: MotionActivityRequest): MotionActivityRecord
|
||||
|
||||
suspend fun pedometer(context: Context, request: MotionPedometerRequest): PedometerRecord
|
||||
}
|
||||
|
||||
private object SystemMotionDataSource : MotionDataSource {
|
||||
override fun isActivityAvailable(context: Context): Boolean {
|
||||
val sensorManager = context.getSystemService(SensorManager::class.java)
|
||||
return sensorManager?.getDefaultSensor(Sensor.TYPE_ACCELEROMETER) != null
|
||||
}
|
||||
|
||||
override fun isPedometerAvailable(context: Context): Boolean {
|
||||
val sensorManager = context.getSystemService(SensorManager::class.java)
|
||||
return sensorManager?.getDefaultSensor(Sensor.TYPE_STEP_COUNTER) != null
|
||||
}
|
||||
|
||||
override fun hasPermission(context: Context): Boolean {
|
||||
return ContextCompat.checkSelfPermission(context, Manifest.permission.ACTIVITY_RECOGNITION) ==
|
||||
android.content.pm.PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
|
||||
override suspend fun activity(context: Context, request: MotionActivityRequest): MotionActivityRecord {
|
||||
if (!request.startISO.isNullOrBlank() || !request.endISO.isNullOrBlank()) {
|
||||
throw IllegalArgumentException("MOTION_RANGE_UNAVAILABLE: historical activity range not supported on Android")
|
||||
}
|
||||
val sensorManager = context.getSystemService(SensorManager::class.java)
|
||||
?: throw IllegalStateException("MOTION_UNAVAILABLE: sensor manager unavailable")
|
||||
val accelerometer = sensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER)
|
||||
?: throw IllegalStateException("MOTION_UNAVAILABLE: accelerometer not available")
|
||||
|
||||
val sample = readAccelerometerSample(sensorManager, accelerometer)
|
||||
?: throw IllegalStateException("MOTION_UNAVAILABLE: no accelerometer sample")
|
||||
val end = Instant.now()
|
||||
val start = end.minusSeconds(2)
|
||||
val classification = classifyActivity(sample.averageDelta)
|
||||
return MotionActivityRecord(
|
||||
startISO = start.toString(),
|
||||
endISO = end.toString(),
|
||||
confidence = classifyConfidence(sample.samples, sample.averageDelta),
|
||||
isWalking = classification == "walking",
|
||||
isRunning = classification == "running",
|
||||
isCycling = false,
|
||||
isAutomotive = false,
|
||||
isStationary = classification == "stationary",
|
||||
isUnknown = classification == "unknown",
|
||||
)
|
||||
}
|
||||
|
||||
override suspend fun pedometer(context: Context, request: MotionPedometerRequest): PedometerRecord {
|
||||
if (!request.startISO.isNullOrBlank() || !request.endISO.isNullOrBlank()) {
|
||||
throw IllegalArgumentException("PEDOMETER_RANGE_UNAVAILABLE: historical pedometer range not supported on Android")
|
||||
}
|
||||
val sensorManager = context.getSystemService(SensorManager::class.java)
|
||||
?: throw IllegalStateException("PEDOMETER_UNAVAILABLE: sensor manager unavailable")
|
||||
val stepCounter = sensorManager.getDefaultSensor(Sensor.TYPE_STEP_COUNTER)
|
||||
?: throw IllegalStateException("PEDOMETER_UNAVAILABLE: step counting not supported")
|
||||
|
||||
val steps = readStepCounter(sensorManager, stepCounter)
|
||||
?: throw IllegalStateException("PEDOMETER_UNAVAILABLE: no step counter sample")
|
||||
val bootMs = System.currentTimeMillis() - SystemClock.elapsedRealtime()
|
||||
return PedometerRecord(
|
||||
startISO = Instant.ofEpochMilli(max(0L, bootMs)).toString(),
|
||||
endISO = Instant.now().toString(),
|
||||
steps = steps,
|
||||
distanceMeters = null,
|
||||
floorsAscended = null,
|
||||
floorsDescended = null,
|
||||
)
|
||||
}
|
||||
|
||||
private data class AccelerometerSample(
|
||||
val samples: Int,
|
||||
val averageDelta: Double,
|
||||
)
|
||||
|
||||
@OptIn(InternalCoroutinesApi::class)
|
||||
private suspend fun readStepCounter(sensorManager: SensorManager, sensor: Sensor): Int? {
|
||||
val sample =
|
||||
withTimeoutOrNull(1200L) {
|
||||
suspendCancellableCoroutine<Float?> { cont ->
|
||||
val listener =
|
||||
object : SensorEventListener {
|
||||
override fun onSensorChanged(event: SensorEvent?) {
|
||||
val value = event?.values?.firstOrNull()
|
||||
val token = cont.tryResume(value) ?: return
|
||||
cont.completeResume(token)
|
||||
sensorManager.unregisterListener(this)
|
||||
}
|
||||
|
||||
override fun onAccuracyChanged(sensor: Sensor?, accuracy: Int) = Unit
|
||||
}
|
||||
val registered = sensorManager.registerListener(listener, sensor, SensorManager.SENSOR_DELAY_NORMAL)
|
||||
if (!registered) {
|
||||
sensorManager.unregisterListener(listener)
|
||||
cont.resume(null) { _, _, _ -> }
|
||||
return@suspendCancellableCoroutine
|
||||
}
|
||||
cont.invokeOnCancellation { sensorManager.unregisterListener(listener) }
|
||||
}
|
||||
}
|
||||
return sample?.toInt()?.takeIf { it >= 0 }
|
||||
}
|
||||
|
||||
@OptIn(InternalCoroutinesApi::class)
|
||||
private suspend fun readAccelerometerSample(
|
||||
sensorManager: SensorManager,
|
||||
sensor: Sensor,
|
||||
): AccelerometerSample? {
|
||||
val sample =
|
||||
withTimeoutOrNull(ACCELEROMETER_SAMPLE_TIMEOUT_MS) {
|
||||
suspendCancellableCoroutine<AccelerometerSample?> { cont ->
|
||||
var count = 0
|
||||
var sumDelta = 0.0
|
||||
val listener =
|
||||
object : SensorEventListener {
|
||||
override fun onSensorChanged(event: SensorEvent?) {
|
||||
val values = event?.values ?: return
|
||||
if (values.size < 3) return
|
||||
val magnitude =
|
||||
sqrt(
|
||||
values[0] * values[0] +
|
||||
values[1] * values[1] +
|
||||
values[2] * values[2],
|
||||
).toDouble()
|
||||
sumDelta += abs(magnitude - SensorManager.GRAVITY_EARTH.toDouble())
|
||||
count += 1
|
||||
if (count >= ACCELEROMETER_SAMPLE_TARGET) {
|
||||
val result = AccelerometerSample(
|
||||
samples = count,
|
||||
averageDelta = sumDelta / count,
|
||||
)
|
||||
val token = cont.tryResume(result) ?: return
|
||||
cont.completeResume(token)
|
||||
sensorManager.unregisterListener(this)
|
||||
}
|
||||
}
|
||||
|
||||
override fun onAccuracyChanged(sensor: Sensor?, accuracy: Int) = Unit
|
||||
}
|
||||
val registered = sensorManager.registerListener(listener, sensor, SensorManager.SENSOR_DELAY_NORMAL)
|
||||
if (!registered) {
|
||||
cont.resume(null) { _, _, _ -> }
|
||||
return@suspendCancellableCoroutine
|
||||
}
|
||||
cont.invokeOnCancellation { sensorManager.unregisterListener(listener) }
|
||||
}
|
||||
}
|
||||
return sample
|
||||
}
|
||||
|
||||
private fun classifyActivity(averageDelta: Double): String {
|
||||
return when {
|
||||
averageDelta <= 0.55 -> "stationary"
|
||||
averageDelta <= 1.80 -> "walking"
|
||||
else -> "running"
|
||||
}
|
||||
}
|
||||
|
||||
private fun classifyConfidence(samples: Int, averageDelta: Double): String {
|
||||
if (samples < 6) return "low"
|
||||
if (samples >= 14 && averageDelta > 0.4) return "high"
|
||||
return "medium"
|
||||
}
|
||||
}
|
||||
|
||||
class MotionHandler private constructor(
|
||||
private val appContext: Context,
|
||||
private val dataSource: MotionDataSource,
|
||||
) {
|
||||
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemMotionDataSource)
|
||||
|
||||
suspend fun handleMotionActivity(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!dataSource.hasPermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "MOTION_PERMISSION_REQUIRED",
|
||||
message = "MOTION_PERMISSION_REQUIRED: grant Motion permission",
|
||||
)
|
||||
}
|
||||
val request =
|
||||
parseActivityRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
return try {
|
||||
val activity = dataSource.activity(appContext, request)
|
||||
GatewaySession.InvokeResult.ok(
|
||||
buildJsonObject {
|
||||
put(
|
||||
"activities",
|
||||
buildJsonArray {
|
||||
add(
|
||||
buildJsonObject {
|
||||
put("startISO", JsonPrimitive(activity.startISO))
|
||||
put("endISO", JsonPrimitive(activity.endISO))
|
||||
put("confidence", JsonPrimitive(activity.confidence))
|
||||
put("isWalking", JsonPrimitive(activity.isWalking))
|
||||
put("isRunning", JsonPrimitive(activity.isRunning))
|
||||
put("isCycling", JsonPrimitive(activity.isCycling))
|
||||
put("isAutomotive", JsonPrimitive(activity.isAutomotive))
|
||||
put("isStationary", JsonPrimitive(activity.isStationary))
|
||||
put("isUnknown", JsonPrimitive(activity.isUnknown))
|
||||
},
|
||||
)
|
||||
},
|
||||
)
|
||||
}.toString(),
|
||||
)
|
||||
} catch (err: IllegalArgumentException) {
|
||||
GatewaySession.InvokeResult.error(code = "MOTION_UNAVAILABLE", message = err.message ?: "MOTION_UNAVAILABLE")
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "MOTION_UNAVAILABLE",
|
||||
message = "MOTION_UNAVAILABLE: ${err.message ?: "motion activity failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun handleMotionPedometer(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!dataSource.hasPermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "MOTION_PERMISSION_REQUIRED",
|
||||
message = "MOTION_PERMISSION_REQUIRED: grant Motion permission",
|
||||
)
|
||||
}
|
||||
val request =
|
||||
parsePedometerRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
return try {
|
||||
val payload = dataSource.pedometer(appContext, request)
|
||||
GatewaySession.InvokeResult.ok(
|
||||
buildJsonObject {
|
||||
put("startISO", JsonPrimitive(payload.startISO))
|
||||
put("endISO", JsonPrimitive(payload.endISO))
|
||||
payload.steps?.let { put("steps", JsonPrimitive(it)) }
|
||||
payload.distanceMeters?.let { put("distanceMeters", JsonPrimitive(it)) }
|
||||
payload.floorsAscended?.let { put("floorsAscended", JsonPrimitive(it)) }
|
||||
payload.floorsDescended?.let { put("floorsDescended", JsonPrimitive(it)) }
|
||||
}.toString(),
|
||||
)
|
||||
} catch (err: IllegalArgumentException) {
|
||||
GatewaySession.InvokeResult.error(code = "MOTION_UNAVAILABLE", message = err.message ?: "MOTION_UNAVAILABLE")
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "MOTION_UNAVAILABLE",
|
||||
message = "MOTION_UNAVAILABLE: ${err.message ?: "pedometer query failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
fun isAvailable(): Boolean = dataSource.isAvailable(appContext)
|
||||
|
||||
fun isActivityAvailable(): Boolean = dataSource.isActivityAvailable(appContext)
|
||||
|
||||
fun isPedometerAvailable(): Boolean = dataSource.isPedometerAvailable(appContext)
|
||||
|
||||
private fun parseActivityRequest(paramsJson: String?): MotionActivityRequest? {
|
||||
if (paramsJson.isNullOrBlank()) {
|
||||
return MotionActivityRequest(startISO = null, endISO = null, limit = 200)
|
||||
}
|
||||
val params =
|
||||
try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
val limit = ((params["limit"] as? JsonPrimitive)?.content?.toIntOrNull() ?: 200).coerceIn(1, 1000)
|
||||
return MotionActivityRequest(
|
||||
startISO = (params["startISO"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
endISO = (params["endISO"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
limit = limit,
|
||||
)
|
||||
}
|
||||
|
||||
private fun parsePedometerRequest(paramsJson: String?): MotionPedometerRequest? {
|
||||
if (paramsJson.isNullOrBlank()) {
|
||||
return MotionPedometerRequest(startISO = null, endISO = null)
|
||||
}
|
||||
val params =
|
||||
try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
return MotionPedometerRequest(
|
||||
startISO = (params["startISO"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
endISO = (params["endISO"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
|
||||
)
|
||||
}
|
||||
|
||||
companion object {
|
||||
fun isMotionCapabilityAvailable(context: Context): Boolean = SystemMotionDataSource.isAvailable(context)
|
||||
|
||||
internal fun forTesting(
|
||||
appContext: Context,
|
||||
dataSource: MotionDataSource,
|
||||
): MotionHandler = MotionHandler(appContext = appContext, dataSource = dataSource)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,84 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import ai.openclaw.app.gateway.parseInvokeErrorFromThrowable
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonElement
|
||||
import kotlinx.serialization.json.JsonNull
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.contentOrNull
|
||||
|
||||
const val DEFAULT_SEAM_COLOR_ARGB: Long = 0xFF4F7A9A
|
||||
|
||||
data class Quad<A, B, C, D>(val first: A, val second: B, val third: C, val fourth: D)
|
||||
|
||||
fun String.toJsonString(): String {
|
||||
val escaped =
|
||||
this.replace("\\", "\\\\")
|
||||
.replace("\"", "\\\"")
|
||||
.replace("\n", "\\n")
|
||||
.replace("\r", "\\r")
|
||||
return "\"$escaped\""
|
||||
}
|
||||
|
||||
fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject
|
||||
|
||||
fun parseJsonParamsObject(paramsJson: String?): JsonObject? {
|
||||
if (paramsJson.isNullOrBlank()) return null
|
||||
return try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
fun readJsonPrimitive(params: JsonObject?, key: String): JsonPrimitive? = params?.get(key) as? JsonPrimitive
|
||||
|
||||
fun parseJsonInt(params: JsonObject?, key: String): Int? =
|
||||
readJsonPrimitive(params, key)?.contentOrNull?.toIntOrNull()
|
||||
|
||||
fun parseJsonDouble(params: JsonObject?, key: String): Double? =
|
||||
readJsonPrimitive(params, key)?.contentOrNull?.toDoubleOrNull()
|
||||
|
||||
fun parseJsonString(params: JsonObject?, key: String): String? =
|
||||
readJsonPrimitive(params, key)?.contentOrNull
|
||||
|
||||
fun parseJsonBooleanFlag(params: JsonObject?, key: String): Boolean? {
|
||||
val value = readJsonPrimitive(params, key)?.contentOrNull?.trim()?.lowercase() ?: return null
|
||||
return when (value) {
|
||||
"true" -> true
|
||||
"false" -> false
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
|
||||
fun JsonElement?.asStringOrNull(): String? =
|
||||
when (this) {
|
||||
is JsonNull -> null
|
||||
is JsonPrimitive -> content
|
||||
else -> null
|
||||
}
|
||||
|
||||
fun parseHexColorArgb(raw: String?): Long? {
|
||||
val trimmed = raw?.trim().orEmpty()
|
||||
if (trimmed.isEmpty()) return null
|
||||
val hex = if (trimmed.startsWith("#")) trimmed.drop(1) else trimmed
|
||||
if (hex.length != 6) return null
|
||||
val rgb = hex.toLongOrNull(16) ?: return null
|
||||
return 0xFF000000L or rgb
|
||||
}
|
||||
|
||||
fun invokeErrorFromThrowable(err: Throwable): Pair<String, String> {
|
||||
val parsed = parseInvokeErrorFromThrowable(err, fallbackMessage = "UNAVAILABLE: error")
|
||||
val message = if (parsed.hadExplicitCode) parsed.prefixedMessage else parsed.message
|
||||
return parsed.code to message
|
||||
}
|
||||
|
||||
fun normalizeMainKey(raw: String?): String? {
|
||||
val trimmed = raw?.trim().orEmpty()
|
||||
return if (trimmed.isEmpty()) null else trimmed
|
||||
}
|
||||
|
||||
fun isCanonicalMainSessionKey(key: String): Boolean {
|
||||
return key == "main"
|
||||
}
|
||||
|
|
@ -0,0 +1,161 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.content.Context
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.contentOrNull
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
internal interface NotificationsStateProvider {
|
||||
fun readSnapshot(context: Context): DeviceNotificationSnapshot
|
||||
|
||||
fun requestServiceRebind(context: Context)
|
||||
|
||||
fun executeAction(context: Context, request: NotificationActionRequest): NotificationActionResult
|
||||
}
|
||||
|
||||
private object SystemNotificationsStateProvider : NotificationsStateProvider {
|
||||
override fun readSnapshot(context: Context): DeviceNotificationSnapshot {
|
||||
val enabled = DeviceNotificationListenerService.isAccessEnabled(context)
|
||||
if (!enabled) {
|
||||
return DeviceNotificationSnapshot(
|
||||
enabled = false,
|
||||
connected = false,
|
||||
notifications = emptyList(),
|
||||
)
|
||||
}
|
||||
return DeviceNotificationListenerService.snapshot(context, enabled = true)
|
||||
}
|
||||
|
||||
override fun requestServiceRebind(context: Context) {
|
||||
DeviceNotificationListenerService.requestServiceRebind(context)
|
||||
}
|
||||
|
||||
override fun executeAction(context: Context, request: NotificationActionRequest): NotificationActionResult {
|
||||
return DeviceNotificationListenerService.executeAction(context, request)
|
||||
}
|
||||
}
|
||||
|
||||
class NotificationsHandler private constructor(
|
||||
private val appContext: Context,
|
||||
private val stateProvider: NotificationsStateProvider,
|
||||
) {
|
||||
constructor(appContext: Context) : this(appContext = appContext, stateProvider = SystemNotificationsStateProvider)
|
||||
|
||||
suspend fun handleNotificationsList(_paramsJson: String?): GatewaySession.InvokeResult {
|
||||
val snapshot = readSnapshotWithRebind()
|
||||
return GatewaySession.InvokeResult.ok(snapshotPayloadJson(snapshot))
|
||||
}
|
||||
|
||||
suspend fun handleNotificationsActions(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
readSnapshotWithRebind()
|
||||
|
||||
val params = parseParamsObject(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
val key =
|
||||
readString(params, "key")
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: key required",
|
||||
)
|
||||
val actionRaw =
|
||||
readString(params, "action")?.lowercase()
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: action required (open|dismiss|reply)",
|
||||
)
|
||||
val action =
|
||||
when (actionRaw) {
|
||||
"open" -> NotificationActionKind.Open
|
||||
"dismiss" -> NotificationActionKind.Dismiss
|
||||
"reply" -> NotificationActionKind.Reply
|
||||
else ->
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: action must be open|dismiss|reply",
|
||||
)
|
||||
}
|
||||
val replyText = readString(params, "replyText")
|
||||
if (action == NotificationActionKind.Reply && replyText.isNullOrBlank()) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: replyText required for reply action",
|
||||
)
|
||||
}
|
||||
|
||||
val result =
|
||||
stateProvider.executeAction(
|
||||
appContext,
|
||||
NotificationActionRequest(
|
||||
key = key,
|
||||
kind = action,
|
||||
replyText = replyText,
|
||||
),
|
||||
)
|
||||
if (!result.ok) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = result.code ?: "UNAVAILABLE",
|
||||
message = result.message ?: "notification action failed",
|
||||
)
|
||||
}
|
||||
|
||||
val payload =
|
||||
buildJsonObject {
|
||||
put("ok", JsonPrimitive(true))
|
||||
put("key", JsonPrimitive(key))
|
||||
put("action", JsonPrimitive(actionRaw))
|
||||
}.toString()
|
||||
return GatewaySession.InvokeResult.ok(payload)
|
||||
}
|
||||
|
||||
private fun readSnapshotWithRebind(): DeviceNotificationSnapshot {
|
||||
val snapshot = stateProvider.readSnapshot(appContext)
|
||||
if (snapshot.enabled && !snapshot.connected) {
|
||||
stateProvider.requestServiceRebind(appContext)
|
||||
}
|
||||
return snapshot
|
||||
}
|
||||
|
||||
private fun snapshotPayloadJson(snapshot: DeviceNotificationSnapshot): String {
|
||||
return buildJsonObject {
|
||||
put("enabled", JsonPrimitive(snapshot.enabled))
|
||||
put("connected", JsonPrimitive(snapshot.connected))
|
||||
put("count", JsonPrimitive(snapshot.notifications.size))
|
||||
put(
|
||||
"notifications",
|
||||
JsonArray(
|
||||
snapshot.notifications.map { entry -> entry.toJsonObject() },
|
||||
),
|
||||
)
|
||||
}.toString()
|
||||
}
|
||||
|
||||
private fun parseParamsObject(paramsJson: String?): JsonObject? {
|
||||
if (paramsJson.isNullOrBlank()) return null
|
||||
return try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun readString(params: JsonObject, key: String): String? =
|
||||
(params[key] as? JsonPrimitive)
|
||||
?.contentOrNull
|
||||
?.trim()
|
||||
?.takeIf { it.isNotEmpty() }
|
||||
|
||||
companion object {
|
||||
internal fun forTesting(
|
||||
appContext: Context,
|
||||
stateProvider: NotificationsStateProvider,
|
||||
): NotificationsHandler = NotificationsHandler(appContext = appContext, stateProvider = stateProvider)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,303 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.content.ContentResolver
|
||||
import android.content.ContentUris
|
||||
import android.content.Context
|
||||
import android.graphics.Bitmap
|
||||
import android.graphics.BitmapFactory
|
||||
import android.net.Uri
|
||||
import android.os.Build
|
||||
import android.os.Bundle
|
||||
import android.provider.MediaStore
|
||||
import androidx.core.content.ContextCompat
|
||||
import androidx.core.graphics.scale
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import java.io.ByteArrayOutputStream
|
||||
import java.time.Instant
|
||||
import kotlin.math.max
|
||||
import kotlin.math.roundToInt
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonArray
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
private const val DEFAULT_PHOTOS_LIMIT = 1
|
||||
private const val DEFAULT_PHOTOS_MAX_WIDTH = 1600
|
||||
private const val DEFAULT_PHOTOS_QUALITY = 0.85
|
||||
private const val MAX_TOTAL_BASE64_CHARS = 340 * 1024
|
||||
private const val MAX_PER_PHOTO_BASE64_CHARS = 300 * 1024
|
||||
|
||||
internal data class PhotosLatestRequest(
|
||||
val limit: Int,
|
||||
val maxWidth: Int,
|
||||
val quality: Double,
|
||||
)
|
||||
|
||||
internal data class EncodedPhotoPayload(
|
||||
val format: String,
|
||||
val base64: String,
|
||||
val width: Int,
|
||||
val height: Int,
|
||||
val createdAt: String?,
|
||||
)
|
||||
|
||||
internal interface PhotosDataSource {
|
||||
fun hasPermission(context: Context): Boolean
|
||||
|
||||
fun latest(context: Context, request: PhotosLatestRequest): List<EncodedPhotoPayload>
|
||||
}
|
||||
|
||||
private object SystemPhotosDataSource : PhotosDataSource {
|
||||
override fun hasPermission(context: Context): Boolean {
|
||||
val permission =
|
||||
if (Build.VERSION.SDK_INT >= 33) {
|
||||
Manifest.permission.READ_MEDIA_IMAGES
|
||||
} else {
|
||||
Manifest.permission.READ_EXTERNAL_STORAGE
|
||||
}
|
||||
return ContextCompat.checkSelfPermission(context, permission) == android.content.pm.PackageManager.PERMISSION_GRANTED
|
||||
}
|
||||
|
||||
override fun latest(context: Context, request: PhotosLatestRequest): List<EncodedPhotoPayload> {
|
||||
val resolver = context.contentResolver
|
||||
val rows = queryLatestRows(resolver, request.limit)
|
||||
if (rows.isEmpty()) return emptyList()
|
||||
|
||||
var remainingBudget = MAX_TOTAL_BASE64_CHARS
|
||||
val out = mutableListOf<EncodedPhotoPayload>()
|
||||
for (row in rows) {
|
||||
if (remainingBudget <= 0) break
|
||||
val bitmap = decodeScaledBitmap(resolver, row.uri, request.maxWidth) ?: continue
|
||||
try {
|
||||
val encoded = encodeJpegUnderBudget(bitmap, request.quality, MAX_PER_PHOTO_BASE64_CHARS)
|
||||
if (encoded == null) continue
|
||||
if (encoded.base64.length > remainingBudget) break
|
||||
remainingBudget -= encoded.base64.length
|
||||
out +=
|
||||
EncodedPhotoPayload(
|
||||
format = "jpeg",
|
||||
base64 = encoded.base64,
|
||||
width = encoded.width,
|
||||
height = encoded.height,
|
||||
createdAt = row.createdAtMs?.let { Instant.ofEpochMilli(it).toString() },
|
||||
)
|
||||
} finally {
|
||||
bitmap.recycle()
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
private data class PhotoRow(
|
||||
val uri: Uri,
|
||||
val createdAtMs: Long?,
|
||||
)
|
||||
|
||||
private data class EncodedJpeg(
|
||||
val base64: String,
|
||||
val width: Int,
|
||||
val height: Int,
|
||||
)
|
||||
|
||||
private fun queryLatestRows(resolver: ContentResolver, limit: Int): List<PhotoRow> {
|
||||
val projection =
|
||||
arrayOf(
|
||||
MediaStore.Images.Media._ID,
|
||||
MediaStore.Images.Media.DATE_TAKEN,
|
||||
MediaStore.Images.Media.DATE_ADDED,
|
||||
)
|
||||
val sortOrder =
|
||||
"${MediaStore.Images.Media.DATE_TAKEN} DESC, ${MediaStore.Images.Media.DATE_ADDED} DESC"
|
||||
val args =
|
||||
Bundle().apply {
|
||||
putString(ContentResolver.QUERY_ARG_SQL_SORT_ORDER, sortOrder)
|
||||
putInt(ContentResolver.QUERY_ARG_LIMIT, limit)
|
||||
}
|
||||
|
||||
resolver.query(
|
||||
MediaStore.Images.Media.EXTERNAL_CONTENT_URI,
|
||||
projection,
|
||||
args,
|
||||
null,
|
||||
).use { cursor ->
|
||||
if (cursor == null) return emptyList()
|
||||
val idIndex = cursor.getColumnIndexOrThrow(MediaStore.Images.Media._ID)
|
||||
val takenIndex = cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATE_TAKEN)
|
||||
val addedIndex = cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATE_ADDED)
|
||||
val rows = mutableListOf<PhotoRow>()
|
||||
while (cursor.moveToNext()) {
|
||||
val id = cursor.getLong(idIndex)
|
||||
val takenMs = cursor.getLong(takenIndex).takeIf { it > 0L }
|
||||
val addedMs = cursor.getLong(addedIndex).takeIf { it > 0L }?.times(1000L)
|
||||
rows +=
|
||||
PhotoRow(
|
||||
uri = ContentUris.withAppendedId(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, id),
|
||||
createdAtMs = takenMs ?: addedMs,
|
||||
)
|
||||
}
|
||||
return rows
|
||||
}
|
||||
}
|
||||
|
||||
private fun decodeScaledBitmap(
|
||||
resolver: ContentResolver,
|
||||
uri: Uri,
|
||||
maxWidth: Int,
|
||||
): Bitmap? {
|
||||
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
|
||||
resolver.openInputStream(uri).use { input ->
|
||||
if (input == null) return null
|
||||
BitmapFactory.decodeStream(input, null, bounds)
|
||||
}
|
||||
if (bounds.outWidth <= 0 || bounds.outHeight <= 0) return null
|
||||
|
||||
val inSampleSize = computeInSampleSize(bounds.outWidth, maxWidth)
|
||||
val decodeOptions = BitmapFactory.Options().apply { this.inSampleSize = inSampleSize }
|
||||
val decoded =
|
||||
resolver.openInputStream(uri).use { input ->
|
||||
if (input == null) return null
|
||||
BitmapFactory.decodeStream(input, null, decodeOptions)
|
||||
} ?: return null
|
||||
|
||||
if (decoded.width <= maxWidth) return decoded
|
||||
val targetHeight = max(1, ((decoded.height.toDouble() * maxWidth) / decoded.width).roundToInt())
|
||||
return try {
|
||||
decoded.scale(maxWidth, targetHeight, true)
|
||||
} finally {
|
||||
decoded.recycle()
|
||||
}
|
||||
}
|
||||
|
||||
private fun computeInSampleSize(width: Int, maxWidth: Int): Int {
|
||||
var sample = 1
|
||||
var candidate = width
|
||||
while (candidate > maxWidth && sample < 64) {
|
||||
sample *= 2
|
||||
candidate = width / sample
|
||||
}
|
||||
return sample
|
||||
}
|
||||
|
||||
private fun encodeJpegUnderBudget(
|
||||
bitmap: Bitmap,
|
||||
quality: Double,
|
||||
maxBase64Chars: Int,
|
||||
): EncodedJpeg? {
|
||||
var working = bitmap
|
||||
try {
|
||||
var jpegQuality = (quality.coerceIn(0.1, 1.0) * 100.0).roundToInt().coerceIn(10, 100)
|
||||
repeat(10) {
|
||||
val out = ByteArrayOutputStream()
|
||||
val ok = working.compress(Bitmap.CompressFormat.JPEG, jpegQuality, out)
|
||||
if (!ok) return null
|
||||
val bytes = out.toByteArray()
|
||||
val base64 = android.util.Base64.encodeToString(bytes, android.util.Base64.NO_WRAP)
|
||||
if (base64.length <= maxBase64Chars) {
|
||||
return EncodedJpeg(
|
||||
base64 = base64,
|
||||
width = working.width,
|
||||
height = working.height,
|
||||
)
|
||||
}
|
||||
if (jpegQuality > 35) {
|
||||
jpegQuality = max(25, jpegQuality - 15)
|
||||
return@repeat
|
||||
}
|
||||
val nextWidth = max(240, (working.width * 0.75f).roundToInt())
|
||||
if (nextWidth >= working.width) return null
|
||||
val nextHeight = max(1, ((working.height.toDouble() * nextWidth) / working.width).roundToInt())
|
||||
val previous = working
|
||||
working = working.scale(nextWidth, nextHeight, true)
|
||||
if (previous !== bitmap) previous.recycle()
|
||||
}
|
||||
return null
|
||||
} finally {
|
||||
if (working !== bitmap) working.recycle()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
class PhotosHandler private constructor(
|
||||
private val appContext: Context,
|
||||
private val dataSource: PhotosDataSource,
|
||||
) {
|
||||
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemPhotosDataSource)
|
||||
|
||||
fun handlePhotosLatest(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
if (!dataSource.hasPermission(appContext)) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "PHOTOS_PERMISSION_REQUIRED",
|
||||
message = "PHOTOS_PERMISSION_REQUIRED: grant Photos permission",
|
||||
)
|
||||
}
|
||||
val request =
|
||||
parseRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object",
|
||||
)
|
||||
return try {
|
||||
val photos = dataSource.latest(appContext, request)
|
||||
val payload =
|
||||
buildJsonObject {
|
||||
put(
|
||||
"photos",
|
||||
buildJsonArray {
|
||||
photos.forEach { photo ->
|
||||
add(
|
||||
buildJsonObject {
|
||||
put("format", JsonPrimitive(photo.format))
|
||||
put("base64", JsonPrimitive(photo.base64))
|
||||
put("width", JsonPrimitive(photo.width))
|
||||
put("height", JsonPrimitive(photo.height))
|
||||
photo.createdAt?.let { put("createdAt", JsonPrimitive(it)) }
|
||||
},
|
||||
)
|
||||
}
|
||||
},
|
||||
)
|
||||
}.toString()
|
||||
GatewaySession.InvokeResult.ok(payload)
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "PHOTOS_UNAVAILABLE",
|
||||
message = "PHOTOS_UNAVAILABLE: ${err.message ?: "photo fetch failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseRequest(paramsJson: String?): PhotosLatestRequest? {
|
||||
if (paramsJson.isNullOrBlank()) {
|
||||
return PhotosLatestRequest(
|
||||
limit = DEFAULT_PHOTOS_LIMIT,
|
||||
maxWidth = DEFAULT_PHOTOS_MAX_WIDTH,
|
||||
quality = DEFAULT_PHOTOS_QUALITY,
|
||||
)
|
||||
}
|
||||
val params =
|
||||
try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return null
|
||||
|
||||
val limitRaw = (params["limit"] as? JsonPrimitive)?.content?.toIntOrNull()
|
||||
val maxWidthRaw = (params["maxWidth"] as? JsonPrimitive)?.content?.toIntOrNull()
|
||||
val qualityRaw = (params["quality"] as? JsonPrimitive)?.content?.toDoubleOrNull()
|
||||
|
||||
val limit = (limitRaw ?: DEFAULT_PHOTOS_LIMIT).coerceIn(1, 20)
|
||||
val maxWidth = (maxWidthRaw ?: DEFAULT_PHOTOS_MAX_WIDTH).coerceIn(240, 4096)
|
||||
val quality = (qualityRaw ?: DEFAULT_PHOTOS_QUALITY).coerceIn(0.1, 1.0)
|
||||
return PhotosLatestRequest(limit = limit, maxWidth = maxWidth, quality = quality)
|
||||
}
|
||||
|
||||
companion object {
|
||||
internal fun forTesting(
|
||||
appContext: Context,
|
||||
dataSource: PhotosDataSource,
|
||||
): PhotosHandler = PhotosHandler(appContext = appContext, dataSource = dataSource)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
|
||||
class SmsHandler(
|
||||
private val sms: SmsManager,
|
||||
) {
|
||||
suspend fun handleSmsSend(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
val res = sms.send(paramsJson)
|
||||
if (res.ok) {
|
||||
return GatewaySession.InvokeResult.ok(res.payloadJson)
|
||||
}
|
||||
return errorResult(res.error, defaultCode = "SMS_SEND_FAILED")
|
||||
}
|
||||
|
||||
suspend fun handleSmsSearch(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
val res = sms.search(paramsJson)
|
||||
if (res.ok) {
|
||||
return GatewaySession.InvokeResult.ok(res.payloadJson)
|
||||
}
|
||||
return errorResult(res.error, defaultCode = "SMS_SEARCH_FAILED")
|
||||
}
|
||||
|
||||
private fun errorResult(error: String?, defaultCode: String): GatewaySession.InvokeResult {
|
||||
val rawMessage = error ?: defaultCode
|
||||
val idx = rawMessage.indexOf(':')
|
||||
val code = if (idx > 0) rawMessage.substring(0, idx).trim() else defaultCode
|
||||
val message =
|
||||
if (idx > 0 && code == rawMessage.substring(0, idx).trim()) {
|
||||
rawMessage.substring(idx + 1).trim().ifEmpty { rawMessage }
|
||||
} else {
|
||||
rawMessage
|
||||
}
|
||||
return GatewaySession.InvokeResult.error(code = code, message = message)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,172 @@
|
|||
package ai.openclaw.app.node
|
||||
|
||||
import android.Manifest
|
||||
import android.app.NotificationChannel
|
||||
import android.app.NotificationManager
|
||||
import android.content.Context
|
||||
import android.content.pm.PackageManager
|
||||
import android.os.Build
|
||||
import androidx.core.app.NotificationCompat
|
||||
import androidx.core.app.NotificationManagerCompat
|
||||
import androidx.core.content.ContextCompat
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.contentOrNull
|
||||
|
||||
private const val NOTIFICATION_CHANNEL_BASE_ID = "openclaw.system.notify"
|
||||
|
||||
internal data class SystemNotifyRequest(
|
||||
val title: String,
|
||||
val body: String,
|
||||
val sound: String?,
|
||||
val priority: String?,
|
||||
)
|
||||
|
||||
internal interface SystemNotificationPoster {
|
||||
fun isAuthorized(): Boolean
|
||||
|
||||
fun post(request: SystemNotifyRequest)
|
||||
}
|
||||
|
||||
private class AndroidSystemNotificationPoster(
|
||||
private val appContext: Context,
|
||||
) : SystemNotificationPoster {
|
||||
override fun isAuthorized(): Boolean {
|
||||
if (Build.VERSION.SDK_INT >= 33) {
|
||||
val granted =
|
||||
ContextCompat.checkSelfPermission(appContext, Manifest.permission.POST_NOTIFICATIONS) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
if (!granted) return false
|
||||
}
|
||||
return NotificationManagerCompat.from(appContext).areNotificationsEnabled()
|
||||
}
|
||||
|
||||
override fun post(request: SystemNotifyRequest) {
|
||||
val channelId = ensureChannel(request.priority)
|
||||
val silent = isSilentSound(request.sound)
|
||||
val notification =
|
||||
NotificationCompat.Builder(appContext, channelId)
|
||||
.setSmallIcon(android.R.drawable.ic_dialog_info)
|
||||
.setContentTitle(request.title)
|
||||
.setContentText(request.body)
|
||||
.setPriority(compatPriority(request.priority))
|
||||
.setAutoCancel(true)
|
||||
.setOnlyAlertOnce(true)
|
||||
.setSilent(silent)
|
||||
.build()
|
||||
if (
|
||||
Build.VERSION.SDK_INT >= 33 &&
|
||||
ContextCompat.checkSelfPermission(appContext, Manifest.permission.POST_NOTIFICATIONS) !=
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
) {
|
||||
throw SecurityException("notifications permission missing")
|
||||
}
|
||||
NotificationManagerCompat.from(appContext).notify((System.currentTimeMillis() and 0x7FFFFFFF).toInt(), notification)
|
||||
}
|
||||
|
||||
private fun ensureChannel(priority: String?): String {
|
||||
val normalizedPriority = priority.orEmpty().trim().lowercase()
|
||||
val (suffix, importance, name) =
|
||||
when (normalizedPriority) {
|
||||
"passive" -> Triple("passive", NotificationManager.IMPORTANCE_LOW, "OpenClaw Passive")
|
||||
"timesensitive" -> Triple("timesensitive", NotificationManager.IMPORTANCE_HIGH, "OpenClaw Time Sensitive")
|
||||
else -> Triple("active", NotificationManager.IMPORTANCE_DEFAULT, "OpenClaw Active")
|
||||
}
|
||||
val channelId = "$NOTIFICATION_CHANNEL_BASE_ID.$suffix"
|
||||
val manager = appContext.getSystemService(NotificationManager::class.java)
|
||||
val existing = manager.getNotificationChannel(channelId)
|
||||
if (existing == null) {
|
||||
manager.createNotificationChannel(NotificationChannel(channelId, name, importance))
|
||||
}
|
||||
return channelId
|
||||
}
|
||||
|
||||
private fun compatPriority(priority: String?): Int {
|
||||
return when (priority.orEmpty().trim().lowercase()) {
|
||||
"passive" -> NotificationCompat.PRIORITY_LOW
|
||||
"timesensitive" -> NotificationCompat.PRIORITY_HIGH
|
||||
else -> NotificationCompat.PRIORITY_DEFAULT
|
||||
}
|
||||
}
|
||||
|
||||
private fun isSilentSound(sound: String?): Boolean {
|
||||
val normalized = sound?.trim()?.lowercase() ?: return false
|
||||
return normalized in setOf("none", "silent", "off", "false", "0")
|
||||
}
|
||||
}
|
||||
|
||||
class SystemHandler private constructor(
|
||||
private val poster: SystemNotificationPoster,
|
||||
) {
|
||||
constructor(appContext: Context) : this(poster = AndroidSystemNotificationPoster(appContext))
|
||||
|
||||
fun handleSystemNotify(paramsJson: String?): GatewaySession.InvokeResult {
|
||||
val params =
|
||||
parseNotifyRequest(paramsJson)
|
||||
?: return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: expected JSON object with title/body",
|
||||
)
|
||||
if (params.title.isEmpty() && params.body.isEmpty()) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "INVALID_REQUEST",
|
||||
message = "INVALID_REQUEST: empty notification",
|
||||
)
|
||||
}
|
||||
if (!poster.isAuthorized()) {
|
||||
return GatewaySession.InvokeResult.error(
|
||||
code = "NOT_AUTHORIZED",
|
||||
message = "NOT_AUTHORIZED: notifications",
|
||||
)
|
||||
}
|
||||
return try {
|
||||
poster.post(params)
|
||||
GatewaySession.InvokeResult.ok(null)
|
||||
} catch (_: SecurityException) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "NOT_AUTHORIZED",
|
||||
message = "NOT_AUTHORIZED: notifications",
|
||||
)
|
||||
} catch (err: Throwable) {
|
||||
GatewaySession.InvokeResult.error(
|
||||
code = "UNAVAILABLE",
|
||||
message = "NOTIFICATION_FAILED: ${err.message ?: "notification post failed"}",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun parseNotifyRequest(paramsJson: String?): SystemNotifyRequest? {
|
||||
val params = parseParamsObject(paramsJson) ?: return null
|
||||
val rawTitle =
|
||||
(params["title"] as? JsonPrimitive)
|
||||
?.contentOrNull
|
||||
?: return null
|
||||
val rawBody =
|
||||
(params["body"] as? JsonPrimitive)
|
||||
?.contentOrNull
|
||||
?: return null
|
||||
val sound = (params["sound"] as? JsonPrimitive)?.contentOrNull
|
||||
val priority = (params["priority"] as? JsonPrimitive)?.contentOrNull
|
||||
return SystemNotifyRequest(
|
||||
title = rawTitle.trim(),
|
||||
body = rawBody.trim(),
|
||||
sound = sound?.trim()?.ifEmpty { null },
|
||||
priority = priority?.trim()?.ifEmpty { null },
|
||||
)
|
||||
}
|
||||
|
||||
private fun parseParamsObject(paramsJson: String?): JsonObject? {
|
||||
if (paramsJson.isNullOrBlank()) return null
|
||||
return try {
|
||||
Json.parseToJsonElement(paramsJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
internal fun forTesting(poster: SystemNotificationPoster): SystemHandler = SystemHandler(poster)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
package ai.openclaw.app.protocol
|
||||
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
|
||||
object OpenClawCanvasA2UIAction {
|
||||
fun extractActionName(userAction: JsonObject): String? {
|
||||
val name =
|
||||
(userAction["name"] as? JsonPrimitive)
|
||||
?.content
|
||||
?.trim()
|
||||
.orEmpty()
|
||||
if (name.isNotEmpty()) return name
|
||||
val action =
|
||||
(userAction["action"] as? JsonPrimitive)
|
||||
?.content
|
||||
?.trim()
|
||||
.orEmpty()
|
||||
return action.ifEmpty { null }
|
||||
}
|
||||
|
||||
fun sanitizeTagValue(value: String): String {
|
||||
val trimmed = value.trim().ifEmpty { "-" }
|
||||
val normalized = trimmed.replace(" ", "_")
|
||||
val out = StringBuilder(normalized.length)
|
||||
for (c in normalized) {
|
||||
val ok =
|
||||
c.isLetterOrDigit() ||
|
||||
c == '_' ||
|
||||
c == '-' ||
|
||||
c == '.' ||
|
||||
c == ':'
|
||||
out.append(if (ok) c else '_')
|
||||
}
|
||||
return out.toString()
|
||||
}
|
||||
|
||||
fun formatAgentMessage(
|
||||
actionName: String,
|
||||
sessionKey: String,
|
||||
surfaceId: String,
|
||||
sourceComponentId: String,
|
||||
host: String,
|
||||
instanceId: String,
|
||||
contextJson: String?,
|
||||
): String {
|
||||
val ctxSuffix = contextJson?.takeIf { it.isNotBlank() }?.let { " ctx=$it" }.orEmpty()
|
||||
return listOf(
|
||||
"CANVAS_A2UI",
|
||||
"action=${sanitizeTagValue(actionName)}",
|
||||
"session=${sanitizeTagValue(sessionKey)}",
|
||||
"surface=${sanitizeTagValue(surfaceId)}",
|
||||
"component=${sanitizeTagValue(sourceComponentId)}",
|
||||
"host=${sanitizeTagValue(host)}",
|
||||
"instance=${sanitizeTagValue(instanceId)}$ctxSuffix",
|
||||
"default=update_canvas",
|
||||
).joinToString(separator = " ")
|
||||
}
|
||||
|
||||
fun jsDispatchA2UIActionStatus(actionId: String, ok: Boolean, error: String?): String {
|
||||
val err = jsonStringLiteral(error ?: "")
|
||||
val okLiteral = if (ok) "true" else "false"
|
||||
val idLiteral = jsonStringLiteral(actionId)
|
||||
return "window.dispatchEvent(new CustomEvent('openclaw:a2ui-action-status', { detail: { id: ${idLiteral}, ok: ${okLiteral}, error: ${err} } }));"
|
||||
}
|
||||
|
||||
private fun jsonStringLiteral(raw: String): String =
|
||||
JsonPrimitive(raw).toString().replace("\u2028", "\\u2028").replace("\u2029", "\\u2029")
|
||||
}
|
||||
|
|
@ -0,0 +1,150 @@
|
|||
package ai.openclaw.app.protocol
|
||||
|
||||
enum class OpenClawCapability(val rawValue: String) {
|
||||
Canvas("canvas"),
|
||||
Camera("camera"),
|
||||
Sms("sms"),
|
||||
VoiceWake("voiceWake"),
|
||||
Location("location"),
|
||||
Device("device"),
|
||||
Notifications("notifications"),
|
||||
System("system"),
|
||||
Photos("photos"),
|
||||
Contacts("contacts"),
|
||||
Calendar("calendar"),
|
||||
Motion("motion"),
|
||||
CallLog("callLog"),
|
||||
}
|
||||
|
||||
enum class OpenClawCanvasCommand(val rawValue: String) {
|
||||
Present("canvas.present"),
|
||||
Hide("canvas.hide"),
|
||||
Navigate("canvas.navigate"),
|
||||
Eval("canvas.eval"),
|
||||
Snapshot("canvas.snapshot"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "canvas."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawCanvasA2UICommand(val rawValue: String) {
|
||||
Push("canvas.a2ui.push"),
|
||||
PushJSONL("canvas.a2ui.pushJSONL"),
|
||||
Reset("canvas.a2ui.reset"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "canvas.a2ui."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawCameraCommand(val rawValue: String) {
|
||||
List("camera.list"),
|
||||
Snap("camera.snap"),
|
||||
Clip("camera.clip"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "camera."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawSmsCommand(val rawValue: String) {
|
||||
Send("sms.send"),
|
||||
Search("sms.search"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "sms."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawLocationCommand(val rawValue: String) {
|
||||
Get("location.get"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "location."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawDeviceCommand(val rawValue: String) {
|
||||
Status("device.status"),
|
||||
Info("device.info"),
|
||||
Permissions("device.permissions"),
|
||||
Health("device.health"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "device."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawNotificationsCommand(val rawValue: String) {
|
||||
List("notifications.list"),
|
||||
Actions("notifications.actions"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "notifications."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawSystemCommand(val rawValue: String) {
|
||||
Notify("system.notify"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "system."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawPhotosCommand(val rawValue: String) {
|
||||
Latest("photos.latest"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "photos."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawContactsCommand(val rawValue: String) {
|
||||
Search("contacts.search"),
|
||||
Add("contacts.add"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "contacts."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawCalendarCommand(val rawValue: String) {
|
||||
Events("calendar.events"),
|
||||
Add("calendar.add"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "calendar."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawMotionCommand(val rawValue: String) {
|
||||
Activity("motion.activity"),
|
||||
Pedometer("motion.pedometer"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "motion."
|
||||
}
|
||||
}
|
||||
|
||||
enum class OpenClawCallLogCommand(val rawValue: String) {
|
||||
Search("callLog.search"),
|
||||
;
|
||||
|
||||
companion object {
|
||||
const val NamespacePrefix: String = "callLog."
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,222 @@
|
|||
package ai.openclaw.app.tools
|
||||
|
||||
import android.content.Context
|
||||
import kotlinx.serialization.Serializable
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
import kotlinx.serialization.json.JsonElement
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.contentOrNull
|
||||
|
||||
@Serializable
|
||||
private data class ToolDisplayActionSpec(
|
||||
val label: String? = null,
|
||||
val detailKeys: List<String>? = null,
|
||||
)
|
||||
|
||||
@Serializable
|
||||
private data class ToolDisplaySpec(
|
||||
val emoji: String? = null,
|
||||
val title: String? = null,
|
||||
val label: String? = null,
|
||||
val detailKeys: List<String>? = null,
|
||||
val actions: Map<String, ToolDisplayActionSpec>? = null,
|
||||
)
|
||||
|
||||
@Serializable
|
||||
private data class ToolDisplayConfig(
|
||||
val version: Int? = null,
|
||||
val fallback: ToolDisplaySpec? = null,
|
||||
val tools: Map<String, ToolDisplaySpec>? = null,
|
||||
)
|
||||
|
||||
data class ToolDisplaySummary(
|
||||
val name: String,
|
||||
val emoji: String,
|
||||
val title: String,
|
||||
val label: String,
|
||||
val verb: String?,
|
||||
val detail: String?,
|
||||
) {
|
||||
val detailLine: String?
|
||||
get() {
|
||||
val parts = mutableListOf<String>()
|
||||
if (!verb.isNullOrBlank()) parts.add(verb)
|
||||
if (!detail.isNullOrBlank()) parts.add(detail)
|
||||
return if (parts.isEmpty()) null else parts.joinToString(" · ")
|
||||
}
|
||||
|
||||
val summaryLine: String
|
||||
get() = if (detailLine != null) "${emoji} ${label}: ${detailLine}" else "${emoji} ${label}"
|
||||
}
|
||||
|
||||
object ToolDisplayRegistry {
|
||||
private const val CONFIG_ASSET = "tool-display.json"
|
||||
|
||||
private val json = Json { ignoreUnknownKeys = true }
|
||||
@Volatile private var cachedConfig: ToolDisplayConfig? = null
|
||||
|
||||
fun resolve(
|
||||
context: Context,
|
||||
name: String?,
|
||||
args: JsonObject?,
|
||||
meta: String? = null,
|
||||
): ToolDisplaySummary {
|
||||
val trimmedName = name?.trim().orEmpty().ifEmpty { "tool" }
|
||||
val key = trimmedName.lowercase()
|
||||
val config = loadConfig(context)
|
||||
val spec = config.tools?.get(key)
|
||||
val fallback = config.fallback
|
||||
|
||||
val emoji = spec?.emoji ?: fallback?.emoji ?: "🧩"
|
||||
val title = spec?.title ?: titleFromName(trimmedName)
|
||||
val label = spec?.label ?: trimmedName
|
||||
|
||||
val actionRaw = args?.get("action")?.asStringOrNull()?.trim()
|
||||
val action = actionRaw?.takeIf { it.isNotEmpty() }
|
||||
val actionSpec = action?.let { spec?.actions?.get(it) }
|
||||
val verb = normalizeVerb(actionSpec?.label ?: action)
|
||||
|
||||
var detail: String? = null
|
||||
if (key == "read") {
|
||||
detail = readDetail(args)
|
||||
} else if (key == "write" || key == "edit" || key == "attach") {
|
||||
detail = pathDetail(args)
|
||||
}
|
||||
|
||||
val detailKeys = actionSpec?.detailKeys ?: spec?.detailKeys ?: fallback?.detailKeys ?: emptyList()
|
||||
if (detail == null) {
|
||||
detail = firstValue(args, detailKeys)
|
||||
}
|
||||
|
||||
if (detail == null) {
|
||||
detail = meta
|
||||
}
|
||||
|
||||
if (detail != null) {
|
||||
detail = shortenHomeInString(detail)
|
||||
}
|
||||
|
||||
return ToolDisplaySummary(
|
||||
name = trimmedName,
|
||||
emoji = emoji,
|
||||
title = title,
|
||||
label = label,
|
||||
verb = verb,
|
||||
detail = detail,
|
||||
)
|
||||
}
|
||||
|
||||
private fun loadConfig(context: Context): ToolDisplayConfig {
|
||||
val existing = cachedConfig
|
||||
if (existing != null) return existing
|
||||
return try {
|
||||
val jsonString = context.assets.open(CONFIG_ASSET).bufferedReader().use { it.readText() }
|
||||
val decoded = json.decodeFromString(ToolDisplayConfig.serializer(), jsonString)
|
||||
cachedConfig = decoded
|
||||
decoded
|
||||
} catch (_: Throwable) {
|
||||
val fallback = ToolDisplayConfig()
|
||||
cachedConfig = fallback
|
||||
fallback
|
||||
}
|
||||
}
|
||||
|
||||
private fun titleFromName(name: String): String {
|
||||
val cleaned = name.replace("_", " ").trim()
|
||||
if (cleaned.isEmpty()) return "Tool"
|
||||
return cleaned
|
||||
.split(Regex("\\s+"))
|
||||
.joinToString(" ") { part ->
|
||||
val upper = part.uppercase()
|
||||
if (part.length <= 2 && part == upper) part
|
||||
else upper.firstOrNull()?.toString().orEmpty() + part.lowercase().drop(1)
|
||||
}
|
||||
}
|
||||
|
||||
private fun normalizeVerb(value: String?): String? {
|
||||
val trimmed = value?.trim().orEmpty()
|
||||
if (trimmed.isEmpty()) return null
|
||||
return trimmed.replace("_", " ")
|
||||
}
|
||||
|
||||
private fun readDetail(args: JsonObject?): String? {
|
||||
val path = args?.get("path")?.asStringOrNull() ?: return null
|
||||
val offset = args["offset"].asNumberOrNull()
|
||||
val limit = args["limit"].asNumberOrNull()
|
||||
return if (offset != null && limit != null) {
|
||||
val end = offset + limit
|
||||
"${path}:${offset.toInt()}-${end.toInt()}"
|
||||
} else {
|
||||
path
|
||||
}
|
||||
}
|
||||
|
||||
private fun pathDetail(args: JsonObject?): String? {
|
||||
return args?.get("path")?.asStringOrNull()
|
||||
}
|
||||
|
||||
private fun firstValue(args: JsonObject?, keys: List<String>): String? {
|
||||
for (key in keys) {
|
||||
val value = valueForPath(args, key)
|
||||
val rendered = renderValue(value)
|
||||
if (!rendered.isNullOrBlank()) return rendered
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun valueForPath(args: JsonObject?, path: String): JsonElement? {
|
||||
var current: JsonElement? = args
|
||||
for (segment in path.split(".")) {
|
||||
if (segment.isBlank()) return null
|
||||
val obj = current as? JsonObject ?: return null
|
||||
current = obj[segment]
|
||||
}
|
||||
return current
|
||||
}
|
||||
|
||||
private fun renderValue(value: JsonElement?): String? {
|
||||
if (value == null) return null
|
||||
if (value is JsonPrimitive) {
|
||||
if (value.isString) {
|
||||
val trimmed = value.contentOrNull?.trim().orEmpty()
|
||||
if (trimmed.isEmpty()) return null
|
||||
val firstLine = trimmed.lineSequence().firstOrNull()?.trim().orEmpty()
|
||||
if (firstLine.isEmpty()) return null
|
||||
return if (firstLine.length > 160) "${firstLine.take(157)}…" else firstLine
|
||||
}
|
||||
val raw = value.contentOrNull?.trim().orEmpty()
|
||||
raw.toBooleanStrictOrNull()?.let { return it.toString() }
|
||||
raw.toLongOrNull()?.let { return it.toString() }
|
||||
raw.toDoubleOrNull()?.let { return it.toString() }
|
||||
}
|
||||
if (value is JsonArray) {
|
||||
val items = value.mapNotNull { renderValue(it) }
|
||||
if (items.isEmpty()) return null
|
||||
val preview = items.take(3).joinToString(", ")
|
||||
return if (items.size > 3) "${preview}…" else preview
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun shortenHomeInString(value: String): String {
|
||||
val home = System.getProperty("user.home")?.takeIf { it.isNotBlank() }
|
||||
?: System.getenv("HOME")?.takeIf { it.isNotBlank() }
|
||||
if (home.isNullOrEmpty()) return value
|
||||
return value.replace(home, "~")
|
||||
.replace(Regex("/Users/[^/]+"), "~")
|
||||
.replace(Regex("/home/[^/]+"), "~")
|
||||
}
|
||||
|
||||
private fun JsonElement?.asStringOrNull(): String? {
|
||||
val primitive = this as? JsonPrimitive ?: return null
|
||||
return if (primitive.isString) primitive.contentOrNull else primitive.toString()
|
||||
}
|
||||
|
||||
private fun JsonElement?.asNumberOrNull(): Double? {
|
||||
val primitive = this as? JsonPrimitive ?: return null
|
||||
val raw = primitive.contentOrNull ?: return null
|
||||
return raw.toDoubleOrNull()
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import androidx.compose.foundation.background
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableFloatStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.draw.alpha
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import kotlinx.coroutines.delay
|
||||
|
||||
@Composable
|
||||
fun CameraFlashOverlay(
|
||||
token: Long,
|
||||
modifier: Modifier = Modifier,
|
||||
) {
|
||||
Box(modifier = modifier.fillMaxSize()) {
|
||||
CameraFlash(token = token)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun CameraFlash(token: Long) {
|
||||
var alpha by remember { mutableFloatStateOf(0f) }
|
||||
LaunchedEffect(token) {
|
||||
if (token == 0L) return@LaunchedEffect
|
||||
alpha = 0.85f
|
||||
delay(110)
|
||||
alpha = 0f
|
||||
}
|
||||
|
||||
Box(
|
||||
modifier =
|
||||
Modifier
|
||||
.fillMaxSize()
|
||||
.alpha(alpha)
|
||||
.background(Color.White),
|
||||
)
|
||||
}
|
||||
|
|
@ -0,0 +1,175 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import android.annotation.SuppressLint
|
||||
import android.util.Log
|
||||
import android.view.View
|
||||
import android.webkit.ConsoleMessage
|
||||
import android.webkit.JavascriptInterface
|
||||
import android.webkit.WebChromeClient
|
||||
import android.webkit.WebResourceError
|
||||
import android.webkit.WebResourceRequest
|
||||
import android.webkit.WebResourceResponse
|
||||
import android.webkit.WebSettings
|
||||
import android.webkit.WebView
|
||||
import android.webkit.WebViewClient
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.DisposableEffect
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.platform.LocalContext
|
||||
import androidx.compose.ui.viewinterop.AndroidView
|
||||
import androidx.webkit.WebSettingsCompat
|
||||
import androidx.webkit.WebViewFeature
|
||||
import ai.openclaw.app.MainViewModel
|
||||
import java.util.concurrent.atomic.AtomicReference
|
||||
|
||||
@SuppressLint("SetJavaScriptEnabled")
|
||||
@Composable
|
||||
fun CanvasScreen(viewModel: MainViewModel, visible: Boolean, modifier: Modifier = Modifier) {
|
||||
val context = LocalContext.current
|
||||
val isDebuggable = (context.applicationInfo.flags and android.content.pm.ApplicationInfo.FLAG_DEBUGGABLE) != 0
|
||||
val webViewRef = remember { mutableStateOf<WebView?>(null) }
|
||||
val currentPageUrlRef = remember { AtomicReference<String?>(null) }
|
||||
|
||||
DisposableEffect(viewModel) {
|
||||
onDispose {
|
||||
val webView = webViewRef.value ?: return@onDispose
|
||||
viewModel.canvas.detach(webView)
|
||||
webView.removeJavascriptInterface(CanvasA2UIActionBridge.interfaceName)
|
||||
webView.stopLoading()
|
||||
webView.destroy()
|
||||
webViewRef.value = null
|
||||
}
|
||||
}
|
||||
|
||||
AndroidView(
|
||||
modifier = modifier,
|
||||
factory = {
|
||||
WebView(context).apply {
|
||||
visibility = if (visible) View.VISIBLE else View.INVISIBLE
|
||||
settings.javaScriptEnabled = true
|
||||
settings.domStorageEnabled = true
|
||||
settings.mixedContentMode = WebSettings.MIXED_CONTENT_COMPATIBILITY_MODE
|
||||
settings.useWideViewPort = false
|
||||
settings.loadWithOverviewMode = false
|
||||
settings.builtInZoomControls = false
|
||||
settings.displayZoomControls = false
|
||||
settings.setSupportZoom(false)
|
||||
// targetSdk 33+ ignores Force Dark APIs, so only opt out through the supported
|
||||
// algorithmic darkening flag when this WebView implementation exposes it.
|
||||
if (WebViewFeature.isFeatureSupported(WebViewFeature.ALGORITHMIC_DARKENING)) {
|
||||
WebSettingsCompat.setAlgorithmicDarkeningAllowed(settings, false)
|
||||
}
|
||||
if (isDebuggable) {
|
||||
Log.d("OpenClawWebView", "userAgent: ${settings.userAgentString}")
|
||||
}
|
||||
isScrollContainer = true
|
||||
overScrollMode = View.OVER_SCROLL_IF_CONTENT_SCROLLS
|
||||
isVerticalScrollBarEnabled = true
|
||||
isHorizontalScrollBarEnabled = true
|
||||
webViewClient =
|
||||
object : WebViewClient() {
|
||||
override fun onPageStarted(
|
||||
view: WebView,
|
||||
url: String?,
|
||||
favicon: android.graphics.Bitmap?,
|
||||
) {
|
||||
currentPageUrlRef.set(url)
|
||||
}
|
||||
|
||||
override fun onReceivedError(
|
||||
view: WebView,
|
||||
request: WebResourceRequest,
|
||||
error: WebResourceError,
|
||||
) {
|
||||
if (!isDebuggable || !request.isForMainFrame) return
|
||||
Log.e("OpenClawWebView", "onReceivedError: ${error.errorCode} ${error.description} ${request.url}")
|
||||
}
|
||||
|
||||
override fun onReceivedHttpError(
|
||||
view: WebView,
|
||||
request: WebResourceRequest,
|
||||
errorResponse: WebResourceResponse,
|
||||
) {
|
||||
if (!isDebuggable || !request.isForMainFrame) return
|
||||
Log.e(
|
||||
"OpenClawWebView",
|
||||
"onReceivedHttpError: ${errorResponse.statusCode} ${errorResponse.reasonPhrase} ${request.url}",
|
||||
)
|
||||
}
|
||||
|
||||
override fun onPageFinished(view: WebView, url: String?) {
|
||||
currentPageUrlRef.set(url)
|
||||
if (isDebuggable) {
|
||||
Log.d("OpenClawWebView", "onPageFinished: $url")
|
||||
}
|
||||
viewModel.canvas.onPageFinished()
|
||||
}
|
||||
|
||||
override fun onRenderProcessGone(
|
||||
view: WebView,
|
||||
detail: android.webkit.RenderProcessGoneDetail,
|
||||
): Boolean {
|
||||
if (isDebuggable) {
|
||||
Log.e(
|
||||
"OpenClawWebView",
|
||||
"onRenderProcessGone didCrash=${detail.didCrash()} priorityAtExit=${detail.rendererPriorityAtExit()}",
|
||||
)
|
||||
}
|
||||
return true
|
||||
}
|
||||
}
|
||||
webChromeClient =
|
||||
object : WebChromeClient() {
|
||||
override fun onConsoleMessage(consoleMessage: ConsoleMessage?): Boolean {
|
||||
if (!isDebuggable) return false
|
||||
val msg = consoleMessage ?: return false
|
||||
Log.d(
|
||||
"OpenClawWebView",
|
||||
"console ${msg.messageLevel()} @ ${msg.sourceId()}:${msg.lineNumber()} ${msg.message()}",
|
||||
)
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
val bridge =
|
||||
CanvasA2UIActionBridge(
|
||||
isTrustedPage = { viewModel.isTrustedCanvasActionUrl(currentPageUrlRef.get()) },
|
||||
) { payload ->
|
||||
viewModel.handleCanvasA2UIActionFromWebView(payload)
|
||||
}
|
||||
addJavascriptInterface(bridge, CanvasA2UIActionBridge.interfaceName)
|
||||
viewModel.canvas.attach(this)
|
||||
webViewRef.value = this
|
||||
}
|
||||
},
|
||||
update = { webView ->
|
||||
webView.visibility = if (visible) View.VISIBLE else View.INVISIBLE
|
||||
if (visible) {
|
||||
webView.resumeTimers()
|
||||
webView.onResume()
|
||||
} else {
|
||||
webView.onPause()
|
||||
webView.pauseTimers()
|
||||
}
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
internal class CanvasA2UIActionBridge(
|
||||
private val isTrustedPage: () -> Boolean,
|
||||
private val onMessage: (String) -> Unit,
|
||||
) {
|
||||
@JavascriptInterface
|
||||
fun postMessage(payload: String?) {
|
||||
val msg = payload?.trim().orEmpty()
|
||||
if (msg.isEmpty()) return
|
||||
if (!isTrustedPage()) return
|
||||
onMessage(msg)
|
||||
}
|
||||
|
||||
companion object {
|
||||
const val interfaceName: String = "openclawCanvasA2UIAction"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import androidx.compose.runtime.Composable
|
||||
import ai.openclaw.app.MainViewModel
|
||||
import ai.openclaw.app.ui.chat.ChatSheetContent
|
||||
|
||||
@Composable
|
||||
fun ChatSheet(viewModel: MainViewModel) {
|
||||
ChatSheetContent(viewModel = viewModel)
|
||||
}
|
||||
|
|
@ -0,0 +1,655 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import androidx.compose.foundation.BorderStroke
|
||||
import androidx.compose.animation.AnimatedVisibility
|
||||
import androidx.compose.foundation.background
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.PaddingValues
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.height
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.layout.width
|
||||
import androidx.compose.foundation.rememberScrollState
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.foundation.text.KeyboardOptions
|
||||
import androidx.compose.foundation.verticalScroll
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.filled.Cloud
|
||||
import androidx.compose.material.icons.filled.ContentCopy
|
||||
import androidx.compose.material.icons.filled.ExpandLess
|
||||
import androidx.compose.material.icons.filled.ExpandMore
|
||||
import androidx.compose.material.icons.filled.Link
|
||||
import androidx.compose.material.icons.filled.PowerSettingsNew
|
||||
import androidx.compose.material3.AlertDialog
|
||||
import androidx.compose.material3.Button
|
||||
import androidx.compose.material3.ButtonDefaults
|
||||
import androidx.compose.material3.HorizontalDivider
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.OutlinedTextField
|
||||
import androidx.compose.material3.OutlinedTextFieldDefaults
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.material3.Switch
|
||||
import androidx.compose.material3.SwitchDefaults
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.material3.TextButton
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.collectAsState
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.saveable.rememberSaveable
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.text.font.FontFamily
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.text.input.KeyboardType
|
||||
import androidx.compose.ui.platform.LocalContext
|
||||
import androidx.compose.ui.unit.dp
|
||||
import ai.openclaw.app.MainViewModel
|
||||
import ai.openclaw.app.gateway.GatewayEndpoint
|
||||
import ai.openclaw.app.ui.mobileCardSurface
|
||||
|
||||
private enum class ConnectInputMode {
|
||||
SetupCode,
|
||||
Manual,
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun ConnectTabScreen(viewModel: MainViewModel) {
|
||||
val context = LocalContext.current
|
||||
val statusText by viewModel.statusText.collectAsState()
|
||||
val isConnected by viewModel.isConnected.collectAsState()
|
||||
val remoteAddress by viewModel.remoteAddress.collectAsState()
|
||||
val manualHost by viewModel.manualHost.collectAsState()
|
||||
val manualPort by viewModel.manualPort.collectAsState()
|
||||
val manualTls by viewModel.manualTls.collectAsState()
|
||||
val manualEnabled by viewModel.manualEnabled.collectAsState()
|
||||
val gatewayToken by viewModel.gatewayToken.collectAsState()
|
||||
val gatewayBootstrapToken by viewModel.gatewayBootstrapToken.collectAsState()
|
||||
val pendingTrust by viewModel.pendingGatewayTrust.collectAsState()
|
||||
|
||||
var advancedOpen by rememberSaveable { mutableStateOf(false) }
|
||||
var inputMode by
|
||||
remember(manualEnabled, manualHost, gatewayToken) {
|
||||
mutableStateOf(
|
||||
if (manualEnabled || manualHost.isNotBlank() || gatewayToken.trim().isNotEmpty()) {
|
||||
ConnectInputMode.Manual
|
||||
} else {
|
||||
ConnectInputMode.SetupCode
|
||||
},
|
||||
)
|
||||
}
|
||||
var setupCode by rememberSaveable { mutableStateOf("") }
|
||||
var manualHostInput by rememberSaveable { mutableStateOf(manualHost.ifBlank { "10.0.2.2" }) }
|
||||
var manualPortInput by rememberSaveable { mutableStateOf(manualPort.toString()) }
|
||||
var manualTlsInput by rememberSaveable { mutableStateOf(manualTls) }
|
||||
var passwordInput by rememberSaveable { mutableStateOf("") }
|
||||
var validationText by rememberSaveable { mutableStateOf<String?>(null) }
|
||||
|
||||
if (pendingTrust != null) {
|
||||
val prompt = pendingTrust!!
|
||||
AlertDialog(
|
||||
onDismissRequest = { viewModel.declineGatewayTrustPrompt() },
|
||||
containerColor = mobileCardSurface,
|
||||
title = { Text("Trust this gateway?", style = mobileHeadline, color = mobileText) },
|
||||
text = {
|
||||
Text(
|
||||
"First-time TLS connection.\n\nVerify this SHA-256 fingerprint before trusting:\n${prompt.fingerprintSha256}",
|
||||
style = mobileCallout,
|
||||
color = mobileText,
|
||||
)
|
||||
},
|
||||
confirmButton = {
|
||||
TextButton(
|
||||
onClick = { viewModel.acceptGatewayTrustPrompt() },
|
||||
colors = ButtonDefaults.textButtonColors(contentColor = mobileAccent),
|
||||
) {
|
||||
Text("Trust and continue")
|
||||
}
|
||||
},
|
||||
dismissButton = {
|
||||
TextButton(
|
||||
onClick = { viewModel.declineGatewayTrustPrompt() },
|
||||
colors = ButtonDefaults.textButtonColors(contentColor = mobileTextSecondary),
|
||||
) {
|
||||
Text("Cancel")
|
||||
}
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
val setupResolvedEndpoint = remember(setupCode) { decodeGatewaySetupCode(setupCode)?.url?.let { parseGatewayEndpoint(it)?.displayUrl } }
|
||||
val manualResolvedEndpoint = remember(manualHostInput, manualPortInput, manualTlsInput) {
|
||||
composeGatewayManualUrl(manualHostInput, manualPortInput, manualTlsInput)?.let { parseGatewayEndpoint(it)?.displayUrl }
|
||||
}
|
||||
|
||||
val activeEndpoint =
|
||||
remember(isConnected, remoteAddress, setupResolvedEndpoint, manualResolvedEndpoint, inputMode) {
|
||||
when {
|
||||
isConnected && !remoteAddress.isNullOrBlank() -> remoteAddress!!
|
||||
inputMode == ConnectInputMode.SetupCode -> setupResolvedEndpoint ?: "Not set"
|
||||
else -> manualResolvedEndpoint ?: "Not set"
|
||||
}
|
||||
}
|
||||
|
||||
val showDiagnostics = !isConnected && gatewayStatusHasDiagnostics(statusText)
|
||||
val pairingRequired = !isConnected && gatewayStatusLooksLikePairing(statusText)
|
||||
val statusLabel = gatewayStatusForDisplay(statusText)
|
||||
|
||||
PairingAutoRetryEffect(enabled = pairingRequired) {
|
||||
viewModel.refreshGatewayConnection()
|
||||
}
|
||||
|
||||
Column(
|
||||
modifier = Modifier.verticalScroll(rememberScrollState()).padding(horizontal = 20.dp, vertical = 16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(14.dp),
|
||||
) {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(6.dp)) {
|
||||
Text("Gateway Connection", style = mobileTitle1, color = mobileText)
|
||||
Text(
|
||||
if (isConnected) "Your gateway is active and ready." else "Connect to your gateway to get started.",
|
||||
style = mobileCallout,
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
}
|
||||
|
||||
// Status cards in a unified card group
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
color = mobileCardSurface,
|
||||
border = BorderStroke(1.dp, mobileBorder),
|
||||
) {
|
||||
Column {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 12.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(12.dp),
|
||||
) {
|
||||
Surface(
|
||||
shape = RoundedCornerShape(10.dp),
|
||||
color = mobileAccentSoft,
|
||||
) {
|
||||
Icon(
|
||||
imageVector = Icons.Default.Link,
|
||||
contentDescription = null,
|
||||
modifier = Modifier.padding(8.dp).size(18.dp),
|
||||
tint = mobileAccent,
|
||||
)
|
||||
}
|
||||
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
|
||||
Text("Endpoint", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
Text(activeEndpoint, style = mobileBody.copy(fontFamily = FontFamily.Monospace), color = mobileText)
|
||||
}
|
||||
}
|
||||
HorizontalDivider(color = mobileBorder)
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 12.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(12.dp),
|
||||
) {
|
||||
Surface(
|
||||
shape = RoundedCornerShape(10.dp),
|
||||
color = if (isConnected) mobileSuccessSoft else mobileSurface,
|
||||
) {
|
||||
Icon(
|
||||
imageVector = Icons.Default.Cloud,
|
||||
contentDescription = null,
|
||||
modifier = Modifier.padding(8.dp).size(18.dp),
|
||||
tint = if (isConnected) mobileSuccess else mobileTextTertiary,
|
||||
)
|
||||
}
|
||||
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
|
||||
Text("Status", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
Text(statusText, style = mobileBody, color = if (isConnected) mobileSuccess else mobileText)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (isConnected) {
|
||||
// Outlined secondary button when connected — don't scream "danger"
|
||||
Button(
|
||||
onClick = {
|
||||
viewModel.disconnect()
|
||||
validationText = null
|
||||
},
|
||||
modifier = Modifier.fillMaxWidth().height(48.dp),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors =
|
||||
ButtonDefaults.buttonColors(
|
||||
containerColor = mobileCardSurface,
|
||||
contentColor = mobileDanger,
|
||||
),
|
||||
border = BorderStroke(1.dp, mobileDanger.copy(alpha = 0.4f)),
|
||||
) {
|
||||
Icon(Icons.Default.PowerSettingsNew, contentDescription = null, modifier = Modifier.size(18.dp))
|
||||
Spacer(modifier = Modifier.width(8.dp))
|
||||
Text("Disconnect", style = mobileHeadline.copy(fontWeight = FontWeight.SemiBold))
|
||||
}
|
||||
} else {
|
||||
Button(
|
||||
onClick = {
|
||||
if (statusText.contains("operator offline", ignoreCase = true)) {
|
||||
validationText = null
|
||||
viewModel.refreshGatewayConnection()
|
||||
return@Button
|
||||
}
|
||||
|
||||
val config =
|
||||
resolveGatewayConnectConfig(
|
||||
useSetupCode = inputMode == ConnectInputMode.SetupCode,
|
||||
setupCode = setupCode,
|
||||
savedManualHost = manualHost,
|
||||
savedManualPort = manualPort.toString(),
|
||||
savedManualTls = manualTls,
|
||||
manualHostInput = manualHostInput,
|
||||
manualPortInput = manualPortInput,
|
||||
manualTlsInput = manualTlsInput,
|
||||
fallbackBootstrapToken = gatewayBootstrapToken,
|
||||
fallbackToken = gatewayToken,
|
||||
fallbackPassword = passwordInput,
|
||||
)
|
||||
|
||||
if (config == null) {
|
||||
validationText =
|
||||
if (inputMode == ConnectInputMode.SetupCode) {
|
||||
val parsedSetup = decodeGatewaySetupCode(setupCode)
|
||||
if (parsedSetup == null) {
|
||||
"Paste a valid setup code to connect."
|
||||
} else {
|
||||
val parsedGateway = parseGatewayEndpointResult(parsedSetup.url)
|
||||
gatewayEndpointValidationMessage(
|
||||
parsedGateway.error ?: GatewayEndpointValidationError.INVALID_URL,
|
||||
GatewayEndpointInputSource.SETUP_CODE,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
val manualUrl = composeGatewayManualUrl(manualHostInput, manualPortInput, manualTlsInput)
|
||||
val parsedGateway = manualUrl?.let(::parseGatewayEndpointResult)
|
||||
gatewayEndpointValidationMessage(
|
||||
parsedGateway?.error ?: GatewayEndpointValidationError.INVALID_URL,
|
||||
GatewayEndpointInputSource.MANUAL,
|
||||
)
|
||||
}
|
||||
return@Button
|
||||
}
|
||||
|
||||
validationText = null
|
||||
if (inputMode == ConnectInputMode.SetupCode) {
|
||||
viewModel.resetGatewaySetupAuth()
|
||||
}
|
||||
viewModel.setManualEnabled(true)
|
||||
viewModel.setManualHost(config.host)
|
||||
viewModel.setManualPort(config.port)
|
||||
viewModel.setManualTls(config.tls)
|
||||
viewModel.setGatewayBootstrapToken(config.bootstrapToken)
|
||||
if (config.token.isNotBlank()) {
|
||||
viewModel.setGatewayToken(config.token)
|
||||
} else if (config.bootstrapToken.isNotBlank()) {
|
||||
viewModel.setGatewayToken("")
|
||||
}
|
||||
viewModel.setGatewayPassword(config.password)
|
||||
viewModel.connect(
|
||||
GatewayEndpoint.manual(host = config.host, port = config.port),
|
||||
token = config.token.ifEmpty { null },
|
||||
bootstrapToken = config.bootstrapToken.ifEmpty { null },
|
||||
password = config.password.ifEmpty { null },
|
||||
)
|
||||
},
|
||||
modifier = Modifier.fillMaxWidth().height(52.dp),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors =
|
||||
ButtonDefaults.buttonColors(
|
||||
containerColor = mobileAccent,
|
||||
contentColor = Color.White,
|
||||
),
|
||||
) {
|
||||
Text("Connect Gateway", style = mobileHeadline.copy(fontWeight = FontWeight.Bold))
|
||||
}
|
||||
}
|
||||
|
||||
if (showDiagnostics) {
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
color = mobileWarningSoft,
|
||||
border = BorderStroke(1.dp, mobileWarning.copy(alpha = 0.25f)),
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 14.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(10.dp),
|
||||
) {
|
||||
Text(if (pairingRequired) "Pairing required" else "Last gateway error", style = mobileHeadline, color = mobileWarning)
|
||||
Text(statusLabel, style = mobileBody.copy(fontFamily = FontFamily.Monospace), color = mobileText)
|
||||
if (pairingRequired) {
|
||||
Text(
|
||||
"Approve this phone on the gateway. OpenClaw retries automatically while this screen stays open.",
|
||||
style = mobileCallout,
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
CommandBlock("openclaw devices list")
|
||||
CommandBlock("openclaw devices approve <requestId>")
|
||||
}
|
||||
Text("OpenClaw Android ${openClawAndroidVersionLabel()}", style = mobileCaption1, color = mobileTextSecondary)
|
||||
Button(
|
||||
onClick = {
|
||||
copyGatewayDiagnosticsReport(
|
||||
context = context,
|
||||
screen = "connect tab",
|
||||
gatewayAddress = activeEndpoint,
|
||||
statusText = statusLabel,
|
||||
)
|
||||
},
|
||||
modifier = Modifier.fillMaxWidth().height(46.dp),
|
||||
shape = RoundedCornerShape(12.dp),
|
||||
colors =
|
||||
ButtonDefaults.buttonColors(
|
||||
containerColor = mobileCardSurface,
|
||||
contentColor = mobileWarning,
|
||||
),
|
||||
border = BorderStroke(1.dp, mobileWarning.copy(alpha = 0.3f)),
|
||||
) {
|
||||
Icon(Icons.Default.ContentCopy, contentDescription = null, modifier = Modifier.size(18.dp))
|
||||
Spacer(modifier = Modifier.width(8.dp))
|
||||
Text("Copy Report for Claw", style = mobileCallout.copy(fontWeight = FontWeight.Bold))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
color = mobileSurface,
|
||||
border = BorderStroke(1.dp, mobileBorder),
|
||||
onClick = { advancedOpen = !advancedOpen },
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 12.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.SpaceBetween,
|
||||
) {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
|
||||
Text("Advanced controls", style = mobileHeadline, color = mobileText)
|
||||
Text("Setup code, endpoint, TLS, token, password, onboarding.", style = mobileCaption1, color = mobileTextSecondary)
|
||||
}
|
||||
Icon(
|
||||
imageVector = if (advancedOpen) Icons.Default.ExpandLess else Icons.Default.ExpandMore,
|
||||
contentDescription = if (advancedOpen) "Collapse advanced controls" else "Expand advanced controls",
|
||||
tint = mobileTextSecondary,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
AnimatedVisibility(visible = advancedOpen) {
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
color = mobileCardSurface,
|
||||
border = BorderStroke(1.dp, mobileBorder),
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 14.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(12.dp),
|
||||
) {
|
||||
Text("Connection method", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
MethodChip(
|
||||
label = "Setup Code",
|
||||
active = inputMode == ConnectInputMode.SetupCode,
|
||||
onClick = { inputMode = ConnectInputMode.SetupCode },
|
||||
)
|
||||
MethodChip(
|
||||
label = "Manual",
|
||||
active = inputMode == ConnectInputMode.Manual,
|
||||
onClick = { inputMode = ConnectInputMode.Manual },
|
||||
)
|
||||
}
|
||||
|
||||
Text("Run these on the gateway host:", style = mobileCallout, color = mobileTextSecondary)
|
||||
CommandBlock("openclaw qr --setup-code-only")
|
||||
CommandBlock("openclaw qr --json")
|
||||
Text(
|
||||
"For Tailscale or public hosts, use wss:// or Tailscale Serve. Private LAN ws:// remains supported.",
|
||||
style = mobileCaption1,
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
|
||||
if (inputMode == ConnectInputMode.SetupCode) {
|
||||
Text("Setup Code", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
OutlinedTextField(
|
||||
value = setupCode,
|
||||
onValueChange = {
|
||||
setupCode = it
|
||||
validationText = null
|
||||
},
|
||||
placeholder = { Text("Paste setup code", style = mobileBody, color = mobileTextTertiary) },
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
minLines = 3,
|
||||
maxLines = 5,
|
||||
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Ascii),
|
||||
textStyle = mobileBody.copy(fontFamily = FontFamily.Monospace, color = mobileText),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = outlinedColors(),
|
||||
)
|
||||
if (!setupResolvedEndpoint.isNullOrBlank()) {
|
||||
EndpointPreview(endpoint = setupResolvedEndpoint)
|
||||
}
|
||||
} else {
|
||||
Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
QuickFillChip(
|
||||
label = "Android Emulator",
|
||||
onClick = {
|
||||
manualHostInput = "10.0.2.2"
|
||||
manualPortInput = "18789"
|
||||
manualTlsInput = false
|
||||
validationText = null
|
||||
},
|
||||
)
|
||||
QuickFillChip(
|
||||
label = "Localhost",
|
||||
onClick = {
|
||||
manualHostInput = "127.0.0.1"
|
||||
manualPortInput = "18789"
|
||||
manualTlsInput = false
|
||||
validationText = null
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
Text("Host", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
OutlinedTextField(
|
||||
value = manualHostInput,
|
||||
onValueChange = {
|
||||
manualHostInput = it
|
||||
validationText = null
|
||||
},
|
||||
placeholder = { Text("10.0.2.2", style = mobileBody, color = mobileTextTertiary) },
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
singleLine = true,
|
||||
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Uri),
|
||||
textStyle = mobileBody.copy(color = mobileText),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = outlinedColors(),
|
||||
)
|
||||
|
||||
Text(
|
||||
if (manualTlsInput) "Port (optional, defaults to 443)" else "Port",
|
||||
style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold),
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
OutlinedTextField(
|
||||
value = manualPortInput,
|
||||
onValueChange = {
|
||||
manualPortInput = it
|
||||
validationText = null
|
||||
},
|
||||
placeholder = { Text(if (manualTlsInput) "443" else "18789", style = mobileBody, color = mobileTextTertiary) },
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
singleLine = true,
|
||||
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Number),
|
||||
textStyle = mobileBody.copy(fontFamily = FontFamily.Monospace, color = mobileText),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = outlinedColors(),
|
||||
)
|
||||
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.SpaceBetween,
|
||||
) {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
|
||||
Text("Use TLS", style = mobileHeadline, color = mobileText)
|
||||
Text(
|
||||
"Turn this on for Tailscale or public hosts. Private LAN ws:// remains supported.",
|
||||
style = mobileCallout,
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
}
|
||||
Switch(
|
||||
checked = manualTlsInput,
|
||||
onCheckedChange = {
|
||||
manualTlsInput = it
|
||||
validationText = null
|
||||
},
|
||||
colors =
|
||||
SwitchDefaults.colors(
|
||||
checkedTrackColor = mobileAccent,
|
||||
uncheckedTrackColor = mobileBorderStrong,
|
||||
checkedThumbColor = Color.White,
|
||||
uncheckedThumbColor = Color.White,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
Text("Token (optional)", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
OutlinedTextField(
|
||||
value = gatewayToken,
|
||||
onValueChange = { viewModel.setGatewayToken(it) },
|
||||
placeholder = { Text("token", style = mobileBody, color = mobileTextTertiary) },
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
singleLine = true,
|
||||
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Ascii),
|
||||
textStyle = mobileBody.copy(color = mobileText),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = outlinedColors(),
|
||||
)
|
||||
|
||||
Text("Password (optional)", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
OutlinedTextField(
|
||||
value = passwordInput,
|
||||
onValueChange = { passwordInput = it },
|
||||
placeholder = { Text("password", style = mobileBody, color = mobileTextTertiary) },
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
singleLine = true,
|
||||
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Ascii),
|
||||
textStyle = mobileBody.copy(color = mobileText),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = outlinedColors(),
|
||||
)
|
||||
|
||||
if (!manualResolvedEndpoint.isNullOrBlank()) {
|
||||
EndpointPreview(endpoint = manualResolvedEndpoint)
|
||||
}
|
||||
}
|
||||
|
||||
HorizontalDivider(color = mobileBorder)
|
||||
|
||||
TextButton(onClick = { viewModel.setOnboardingCompleted(false) }) {
|
||||
Text("Run onboarding again", style = mobileCallout.copy(fontWeight = FontWeight.SemiBold), color = mobileAccent)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!validationText.isNullOrBlank()) {
|
||||
Text(validationText!!, style = mobileCaption1, color = mobileWarning)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun MethodChip(label: String, active: Boolean, onClick: () -> Unit) {
|
||||
Button(
|
||||
onClick = onClick,
|
||||
modifier = Modifier.height(40.dp),
|
||||
shape = RoundedCornerShape(12.dp),
|
||||
contentPadding = PaddingValues(horizontal = 12.dp, vertical = 8.dp),
|
||||
colors =
|
||||
ButtonDefaults.buttonColors(
|
||||
containerColor = if (active) mobileAccent else mobileSurface,
|
||||
contentColor = if (active) Color.White else mobileText,
|
||||
),
|
||||
border = BorderStroke(1.dp, if (active) mobileAccentBorderStrong else mobileBorderStrong),
|
||||
) {
|
||||
Text(label, style = mobileCaption1.copy(fontWeight = FontWeight.Bold))
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun QuickFillChip(label: String, onClick: () -> Unit) {
|
||||
Button(
|
||||
onClick = onClick,
|
||||
shape = RoundedCornerShape(999.dp),
|
||||
contentPadding = PaddingValues(horizontal = 12.dp, vertical = 6.dp),
|
||||
colors =
|
||||
ButtonDefaults.buttonColors(
|
||||
containerColor = mobileAccentSoft,
|
||||
contentColor = mobileAccent,
|
||||
),
|
||||
elevation = null,
|
||||
) {
|
||||
Text(label, style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold))
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun CommandBlock(command: String) {
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(12.dp),
|
||||
color = mobileCodeBg,
|
||||
border = BorderStroke(1.dp, mobileCodeBorder),
|
||||
) {
|
||||
Row(modifier = Modifier.fillMaxWidth(), verticalAlignment = Alignment.CenterVertically) {
|
||||
Box(modifier = Modifier.width(3.dp).height(42.dp).background(mobileCodeAccent))
|
||||
Text(
|
||||
text = command,
|
||||
modifier = Modifier.padding(horizontal = 12.dp, vertical = 10.dp),
|
||||
style = mobileCallout.copy(fontFamily = FontFamily.Monospace),
|
||||
color = mobileCodeText,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun EndpointPreview(endpoint: String) {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(4.dp)) {
|
||||
HorizontalDivider(color = mobileBorder)
|
||||
Text("Resolved endpoint", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
Text(endpoint, style = mobileCallout.copy(fontFamily = FontFamily.Monospace), color = mobileText)
|
||||
HorizontalDivider(color = mobileBorder)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun outlinedColors() =
|
||||
OutlinedTextFieldDefaults.colors(
|
||||
focusedContainerColor = mobileSurface,
|
||||
unfocusedContainerColor = mobileSurface,
|
||||
focusedBorderColor = mobileAccent,
|
||||
unfocusedBorderColor = mobileBorder,
|
||||
focusedTextColor = mobileText,
|
||||
unfocusedTextColor = mobileText,
|
||||
cursorColor = mobileAccent,
|
||||
)
|
||||
|
|
@ -0,0 +1,271 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import ai.openclaw.app.gateway.isPrivateLanGatewayHost
|
||||
import java.util.Base64
|
||||
import java.util.Locale
|
||||
import java.net.URI
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.contentOrNull
|
||||
import kotlinx.serialization.json.jsonObject
|
||||
|
||||
internal data class GatewayEndpointConfig(
|
||||
val host: String,
|
||||
val port: Int,
|
||||
val tls: Boolean,
|
||||
val displayUrl: String,
|
||||
)
|
||||
|
||||
internal data class GatewaySetupCode(
|
||||
val url: String,
|
||||
val bootstrapToken: String?,
|
||||
val token: String?,
|
||||
val password: String?,
|
||||
)
|
||||
|
||||
internal data class GatewayConnectConfig(
|
||||
val host: String,
|
||||
val port: Int,
|
||||
val tls: Boolean,
|
||||
val bootstrapToken: String,
|
||||
val token: String,
|
||||
val password: String,
|
||||
)
|
||||
|
||||
internal enum class GatewayEndpointValidationError {
|
||||
INVALID_URL,
|
||||
INSECURE_REMOTE_URL,
|
||||
}
|
||||
|
||||
internal enum class GatewayEndpointInputSource {
|
||||
SETUP_CODE,
|
||||
MANUAL,
|
||||
QR_SCAN,
|
||||
}
|
||||
|
||||
internal data class GatewayEndpointParseResult(
|
||||
val config: GatewayEndpointConfig? = null,
|
||||
val error: GatewayEndpointValidationError? = null,
|
||||
)
|
||||
|
||||
internal data class GatewayScannedSetupCodeResult(
|
||||
val setupCode: String? = null,
|
||||
val error: GatewayEndpointValidationError? = null,
|
||||
)
|
||||
|
||||
private val gatewaySetupJson = Json { ignoreUnknownKeys = true }
|
||||
private const val remoteGatewaySecurityRule =
|
||||
"Tailscale and public mobile nodes require wss:// or Tailscale Serve. ws:// is allowed for private LAN, localhost, and the Android emulator."
|
||||
private const val remoteGatewaySecurityFix =
|
||||
"Use a private LAN host/address, or enable Tailscale Serve / expose a wss:// gateway URL."
|
||||
|
||||
internal fun resolveGatewayConnectConfig(
|
||||
useSetupCode: Boolean,
|
||||
setupCode: String,
|
||||
savedManualHost: String,
|
||||
savedManualPort: String,
|
||||
savedManualTls: Boolean,
|
||||
manualHostInput: String,
|
||||
manualPortInput: String,
|
||||
manualTlsInput: Boolean,
|
||||
fallbackBootstrapToken: String,
|
||||
fallbackToken: String,
|
||||
fallbackPassword: String,
|
||||
): GatewayConnectConfig? {
|
||||
if (useSetupCode) {
|
||||
val setup = decodeGatewaySetupCode(setupCode) ?: return null
|
||||
val parsed = parseGatewayEndpointResult(setup.url).config ?: return null
|
||||
val setupBootstrapToken = setup.bootstrapToken?.trim().orEmpty()
|
||||
val sharedToken =
|
||||
when {
|
||||
!setup.token.isNullOrBlank() -> setup.token.trim()
|
||||
setupBootstrapToken.isNotEmpty() -> ""
|
||||
else -> fallbackToken.trim()
|
||||
}
|
||||
val sharedPassword =
|
||||
when {
|
||||
!setup.password.isNullOrBlank() -> setup.password.trim()
|
||||
setupBootstrapToken.isNotEmpty() -> ""
|
||||
else -> fallbackPassword.trim()
|
||||
}
|
||||
return GatewayConnectConfig(
|
||||
host = parsed.host,
|
||||
port = parsed.port,
|
||||
tls = parsed.tls,
|
||||
bootstrapToken = setupBootstrapToken,
|
||||
token = sharedToken,
|
||||
password = sharedPassword,
|
||||
)
|
||||
}
|
||||
|
||||
val manualUrl = composeGatewayManualUrl(manualHostInput, manualPortInput, manualTlsInput) ?: return null
|
||||
val parsed = parseGatewayEndpointResult(manualUrl).config ?: return null
|
||||
val savedManualEndpoint =
|
||||
composeGatewayManualUrl(savedManualHost, savedManualPort, savedManualTls)
|
||||
?.let { parseGatewayEndpointResult(it).config }
|
||||
val preserveBootstrapToken =
|
||||
savedManualEndpoint != null &&
|
||||
savedManualEndpoint.host == parsed.host &&
|
||||
savedManualEndpoint.port == parsed.port &&
|
||||
savedManualEndpoint.tls == parsed.tls &&
|
||||
fallbackToken.isBlank() &&
|
||||
fallbackPassword.isBlank()
|
||||
return GatewayConnectConfig(
|
||||
host = parsed.host,
|
||||
port = parsed.port,
|
||||
tls = parsed.tls,
|
||||
bootstrapToken = if (preserveBootstrapToken) fallbackBootstrapToken.trim() else "",
|
||||
token = fallbackToken.trim(),
|
||||
password = fallbackPassword.trim(),
|
||||
)
|
||||
}
|
||||
|
||||
internal fun parseGatewayEndpoint(rawInput: String): GatewayEndpointConfig? {
|
||||
return parseGatewayEndpointResult(rawInput).config
|
||||
}
|
||||
|
||||
internal fun parseGatewayEndpointResult(rawInput: String): GatewayEndpointParseResult {
|
||||
val raw = rawInput.trim()
|
||||
if (raw.isEmpty()) return GatewayEndpointParseResult(error = GatewayEndpointValidationError.INVALID_URL)
|
||||
|
||||
val normalized = if (raw.contains("://")) raw else "https://$raw"
|
||||
val uri =
|
||||
runCatching { URI(normalized) }.getOrNull()
|
||||
?: return GatewayEndpointParseResult(error = GatewayEndpointValidationError.INVALID_URL)
|
||||
val host = uri.host?.trim()?.trim('[', ']').orEmpty()
|
||||
if (host.isEmpty()) return GatewayEndpointParseResult(error = GatewayEndpointValidationError.INVALID_URL)
|
||||
|
||||
val scheme = uri.scheme?.trim()?.lowercase(Locale.US).orEmpty()
|
||||
val tls =
|
||||
when (scheme) {
|
||||
"ws", "http" -> false
|
||||
"wss", "https" -> true
|
||||
else -> true
|
||||
}
|
||||
if (!tls && !isPrivateLanGatewayHost(host)) {
|
||||
return GatewayEndpointParseResult(error = GatewayEndpointValidationError.INSECURE_REMOTE_URL)
|
||||
}
|
||||
val defaultPort =
|
||||
when (scheme) {
|
||||
"wss", "https" -> 443
|
||||
"ws", "http" -> 18789
|
||||
else -> 443
|
||||
}
|
||||
val displayPort =
|
||||
when (scheme) {
|
||||
"wss", "https" -> 443
|
||||
"ws", "http" -> 80
|
||||
else -> 443
|
||||
}
|
||||
val port = uri.port.takeIf { it in 1..65535 } ?: defaultPort
|
||||
val displayHost = if (host.contains(":")) "[$host]" else host
|
||||
val displayUrl =
|
||||
if (port == displayPort && defaultPort == displayPort) {
|
||||
"${if (tls) "https" else "http"}://$displayHost"
|
||||
} else {
|
||||
"${if (tls) "https" else "http"}://$displayHost:$port"
|
||||
}
|
||||
|
||||
return GatewayEndpointParseResult(
|
||||
config = GatewayEndpointConfig(host = host, port = port, tls = tls, displayUrl = displayUrl),
|
||||
)
|
||||
}
|
||||
|
||||
internal fun decodeGatewaySetupCode(rawInput: String): GatewaySetupCode? {
|
||||
val trimmed = rawInput.trim()
|
||||
if (trimmed.isEmpty()) return null
|
||||
|
||||
val padded =
|
||||
trimmed
|
||||
.replace('-', '+')
|
||||
.replace('_', '/')
|
||||
.let { normalized ->
|
||||
val remainder = normalized.length % 4
|
||||
if (remainder == 0) normalized else normalized + "=".repeat(4 - remainder)
|
||||
}
|
||||
|
||||
return try {
|
||||
val decoded = String(Base64.getDecoder().decode(padded), Charsets.UTF_8)
|
||||
val obj = parseJsonObject(decoded) ?: return null
|
||||
val url = jsonField(obj, "url").orEmpty()
|
||||
if (url.isEmpty()) return null
|
||||
val bootstrapToken = jsonField(obj, "bootstrapToken")
|
||||
val token = jsonField(obj, "token")
|
||||
val password = jsonField(obj, "password")
|
||||
GatewaySetupCode(url = url, bootstrapToken = bootstrapToken, token = token, password = password)
|
||||
} catch (_: IllegalArgumentException) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
internal fun resolveScannedSetupCode(rawInput: String): String? {
|
||||
return resolveScannedSetupCodeResult(rawInput).setupCode
|
||||
}
|
||||
|
||||
internal fun resolveScannedSetupCodeResult(rawInput: String): GatewayScannedSetupCodeResult {
|
||||
val setupCode =
|
||||
resolveSetupCodeCandidate(rawInput)
|
||||
?: return GatewayScannedSetupCodeResult(error = GatewayEndpointValidationError.INVALID_URL)
|
||||
val decoded =
|
||||
decodeGatewaySetupCode(setupCode)
|
||||
?: return GatewayScannedSetupCodeResult(error = GatewayEndpointValidationError.INVALID_URL)
|
||||
val parsed = parseGatewayEndpointResult(decoded.url)
|
||||
if (parsed.config == null) {
|
||||
return GatewayScannedSetupCodeResult(error = parsed.error)
|
||||
}
|
||||
return GatewayScannedSetupCodeResult(setupCode = setupCode)
|
||||
}
|
||||
|
||||
internal fun gatewayEndpointValidationMessage(
|
||||
error: GatewayEndpointValidationError,
|
||||
source: GatewayEndpointInputSource,
|
||||
): String {
|
||||
return when (error) {
|
||||
GatewayEndpointValidationError.INSECURE_REMOTE_URL ->
|
||||
when (source) {
|
||||
GatewayEndpointInputSource.SETUP_CODE ->
|
||||
"Setup code points to an insecure remote gateway. $remoteGatewaySecurityRule $remoteGatewaySecurityFix"
|
||||
GatewayEndpointInputSource.QR_SCAN ->
|
||||
"QR code points to an insecure remote gateway. $remoteGatewaySecurityRule $remoteGatewaySecurityFix"
|
||||
GatewayEndpointInputSource.MANUAL ->
|
||||
"$remoteGatewaySecurityRule $remoteGatewaySecurityFix"
|
||||
}
|
||||
GatewayEndpointValidationError.INVALID_URL ->
|
||||
when (source) {
|
||||
GatewayEndpointInputSource.SETUP_CODE -> "Setup code has invalid gateway URL."
|
||||
GatewayEndpointInputSource.QR_SCAN -> "QR code did not contain a valid setup code."
|
||||
GatewayEndpointInputSource.MANUAL -> "Enter a valid manual endpoint to connect."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
internal fun composeGatewayManualUrl(hostInput: String, portInput: String, tls: Boolean): String? {
|
||||
val host = hostInput.trim()
|
||||
if (host.isEmpty()) return null
|
||||
val portTrimmed = portInput.trim()
|
||||
val port = if (portTrimmed.isEmpty()) {
|
||||
if (tls) 443 else return null
|
||||
} else {
|
||||
portTrimmed.toIntOrNull() ?: return null
|
||||
}
|
||||
if (port !in 1..65535) return null
|
||||
val scheme = if (tls) "https" else "http"
|
||||
return "$scheme://$host:$port"
|
||||
}
|
||||
|
||||
private fun parseJsonObject(input: String): JsonObject? {
|
||||
return runCatching { gatewaySetupJson.parseToJsonElement(input).jsonObject }.getOrNull()
|
||||
}
|
||||
|
||||
private fun resolveSetupCodeCandidate(rawInput: String): String? {
|
||||
val trimmed = rawInput.trim()
|
||||
if (trimmed.isEmpty()) return null
|
||||
val qrSetupCode = parseJsonObject(trimmed)?.let { jsonField(it, "setupCode") }
|
||||
return qrSetupCode ?: trimmed
|
||||
}
|
||||
|
||||
private fun jsonField(obj: JsonObject, key: String): String? {
|
||||
val value = (obj[key] as? JsonPrimitive)?.contentOrNull?.trim().orEmpty()
|
||||
return value.ifEmpty { null }
|
||||
}
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import android.content.ClipData
|
||||
import android.content.ClipboardManager
|
||||
import android.content.Context
|
||||
import android.os.Build
|
||||
import android.widget.Toast
|
||||
import ai.openclaw.app.BuildConfig
|
||||
|
||||
internal fun openClawAndroidVersionLabel(): String {
|
||||
val versionName = BuildConfig.VERSION_NAME.trim().ifEmpty { "dev" }
|
||||
return if (BuildConfig.DEBUG && !versionName.contains("dev", ignoreCase = true)) {
|
||||
"$versionName-dev"
|
||||
} else {
|
||||
versionName
|
||||
}
|
||||
}
|
||||
|
||||
internal fun gatewayStatusForDisplay(statusText: String): String {
|
||||
return statusText.trim().ifEmpty { "Offline" }
|
||||
}
|
||||
|
||||
internal fun gatewayStatusHasDiagnostics(statusText: String): Boolean {
|
||||
val lower = gatewayStatusForDisplay(statusText).lowercase()
|
||||
return lower != "offline" && !lower.contains("connecting")
|
||||
}
|
||||
|
||||
internal fun gatewayStatusLooksLikePairing(statusText: String): Boolean {
|
||||
val lower = gatewayStatusForDisplay(statusText).lowercase()
|
||||
return lower.contains("pair") || lower.contains("approve")
|
||||
}
|
||||
|
||||
internal fun buildGatewayDiagnosticsReport(
|
||||
screen: String,
|
||||
gatewayAddress: String,
|
||||
statusText: String,
|
||||
): String {
|
||||
val device =
|
||||
listOfNotNull(Build.MANUFACTURER, Build.MODEL)
|
||||
.joinToString(" ")
|
||||
.trim()
|
||||
.ifEmpty { "Android" }
|
||||
val androidVersion = Build.VERSION.RELEASE?.trim().orEmpty().ifEmpty { Build.VERSION.SDK_INT.toString() }
|
||||
val endpoint = gatewayAddress.trim().ifEmpty { "unknown" }
|
||||
val status = gatewayStatusForDisplay(statusText)
|
||||
return """
|
||||
Help diagnose this OpenClaw Android gateway connection failure.
|
||||
|
||||
Please:
|
||||
- pick one route only: same machine, same LAN, Tailscale, or public URL
|
||||
- classify this as pairing/auth, TLS trust, wrong advertised route, wrong address/port, or gateway down
|
||||
- remember: Tailscale/public mobile routes require wss:// or Tailscale Serve; private LAN ws:// is still allowed
|
||||
- quote the exact app status/error below
|
||||
- tell me whether `openclaw devices list` should show a pending pairing request
|
||||
- if more signal is needed, ask for `openclaw qr --json`, `openclaw devices list`, and `openclaw nodes status`
|
||||
- give the next exact command or tap
|
||||
|
||||
Debug info:
|
||||
- screen: $screen
|
||||
- app version: ${openClawAndroidVersionLabel()}
|
||||
- device: $device
|
||||
- android: $androidVersion (SDK ${Build.VERSION.SDK_INT})
|
||||
- gateway address: $endpoint
|
||||
- status/error: $status
|
||||
""".trimIndent()
|
||||
}
|
||||
|
||||
internal fun copyGatewayDiagnosticsReport(
|
||||
context: Context,
|
||||
screen: String,
|
||||
gatewayAddress: String,
|
||||
statusText: String,
|
||||
) {
|
||||
val clipboard = context.getSystemService(ClipboardManager::class.java) ?: return
|
||||
val report = buildGatewayDiagnosticsReport(screen = screen, gatewayAddress = gatewayAddress, statusText = statusText)
|
||||
clipboard.setPrimaryClip(ClipData.newPlainText("OpenClaw gateway diagnostics", report))
|
||||
Toast.makeText(context, "Copied gateway diagnostics", Toast.LENGTH_SHORT).show()
|
||||
}
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.DisposableEffect
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.lifecycle.Lifecycle
|
||||
import androidx.lifecycle.LifecycleEventObserver
|
||||
import androidx.lifecycle.compose.LocalLifecycleOwner
|
||||
import kotlinx.coroutines.delay
|
||||
|
||||
internal const val PAIRING_AUTO_RETRY_MS = 6_000L
|
||||
|
||||
@Composable
|
||||
internal fun PairingAutoRetryEffect(enabled: Boolean, onRetry: () -> Unit) {
|
||||
val lifecycleOwner = LocalLifecycleOwner.current
|
||||
var lifecycleStarted by
|
||||
remember(lifecycleOwner) {
|
||||
mutableStateOf(lifecycleOwner.lifecycle.currentState.isAtLeast(Lifecycle.State.STARTED))
|
||||
}
|
||||
|
||||
DisposableEffect(lifecycleOwner) {
|
||||
val observer =
|
||||
LifecycleEventObserver { _, _ ->
|
||||
lifecycleStarted = lifecycleOwner.lifecycle.currentState.isAtLeast(Lifecycle.State.STARTED)
|
||||
}
|
||||
lifecycleOwner.lifecycle.addObserver(observer)
|
||||
onDispose {
|
||||
lifecycleOwner.lifecycle.removeObserver(observer)
|
||||
}
|
||||
}
|
||||
|
||||
LaunchedEffect(enabled, lifecycleStarted) {
|
||||
if (!enabled || !lifecycleStarted) {
|
||||
return@LaunchedEffect
|
||||
}
|
||||
while (true) {
|
||||
delay(PAIRING_AUTO_RETRY_MS)
|
||||
onRetry()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,232 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.staticCompositionLocalOf
|
||||
import androidx.compose.ui.graphics.Brush
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.text.TextStyle
|
||||
import androidx.compose.ui.text.font.Font
|
||||
import androidx.compose.ui.text.font.FontFamily
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.unit.sp
|
||||
import ai.openclaw.app.R
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// MobileColors – semantic color tokens with light + dark variants
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
internal data class MobileColors(
|
||||
val surface: Color,
|
||||
val surfaceStrong: Color,
|
||||
val cardSurface: Color,
|
||||
val border: Color,
|
||||
val borderStrong: Color,
|
||||
val text: Color,
|
||||
val textSecondary: Color,
|
||||
val textTertiary: Color,
|
||||
val accent: Color,
|
||||
val accentSoft: Color,
|
||||
val accentBorderStrong: Color,
|
||||
val success: Color,
|
||||
val successSoft: Color,
|
||||
val warning: Color,
|
||||
val warningSoft: Color,
|
||||
val danger: Color,
|
||||
val dangerSoft: Color,
|
||||
val codeBg: Color,
|
||||
val codeText: Color,
|
||||
val codeBorder: Color,
|
||||
val codeAccent: Color,
|
||||
val chipBorderConnected: Color,
|
||||
val chipBorderConnecting: Color,
|
||||
val chipBorderWarning: Color,
|
||||
val chipBorderError: Color,
|
||||
)
|
||||
|
||||
internal fun lightMobileColors() =
|
||||
MobileColors(
|
||||
surface = Color(0xFFF6F7FA),
|
||||
surfaceStrong = Color(0xFFECEEF3),
|
||||
cardSurface = Color(0xFFFFFFFF),
|
||||
border = Color(0xFFE5E7EC),
|
||||
borderStrong = Color(0xFFD6DAE2),
|
||||
text = Color(0xFF17181C),
|
||||
textSecondary = Color(0xFF5D6472),
|
||||
textTertiary = Color(0xFF99A0AE),
|
||||
accent = Color(0xFF1D5DD8),
|
||||
accentSoft = Color(0xFFECF3FF),
|
||||
accentBorderStrong = Color(0xFF184DAF),
|
||||
success = Color(0xFF2F8C5A),
|
||||
successSoft = Color(0xFFEEF9F3),
|
||||
warning = Color(0xFFC8841A),
|
||||
warningSoft = Color(0xFFFFF8EC),
|
||||
danger = Color(0xFFD04B4B),
|
||||
dangerSoft = Color(0xFFFFF2F2),
|
||||
codeBg = Color(0xFF15171B),
|
||||
codeText = Color(0xFFE8EAEE),
|
||||
codeBorder = Color(0xFF2B2E35),
|
||||
codeAccent = Color(0xFF3FC97A),
|
||||
chipBorderConnected = Color(0xFFCFEBD8),
|
||||
chipBorderConnecting = Color(0xFFD5E2FA),
|
||||
chipBorderWarning = Color(0xFFEED8B8),
|
||||
chipBorderError = Color(0xFFF3C8C8),
|
||||
)
|
||||
|
||||
internal fun darkMobileColors() =
|
||||
MobileColors(
|
||||
surface = Color(0xFF1A1C20),
|
||||
surfaceStrong = Color(0xFF24262B),
|
||||
cardSurface = Color(0xFF1E2024),
|
||||
border = Color(0xFF2E3038),
|
||||
borderStrong = Color(0xFF3A3D46),
|
||||
text = Color(0xFFE4E5EA),
|
||||
textSecondary = Color(0xFFA0A6B4),
|
||||
textTertiary = Color(0xFF6B7280),
|
||||
accent = Color(0xFF6EA8FF),
|
||||
accentSoft = Color(0xFF1A2A44),
|
||||
accentBorderStrong = Color(0xFF5B93E8),
|
||||
success = Color(0xFF5FBB85),
|
||||
successSoft = Color(0xFF152E22),
|
||||
warning = Color(0xFFE8A844),
|
||||
warningSoft = Color(0xFF2E2212),
|
||||
danger = Color(0xFFE87070),
|
||||
dangerSoft = Color(0xFF2E1616),
|
||||
codeBg = Color(0xFF111317),
|
||||
codeText = Color(0xFFE8EAEE),
|
||||
codeBorder = Color(0xFF2B2E35),
|
||||
codeAccent = Color(0xFF3FC97A),
|
||||
chipBorderConnected = Color(0xFF1E4A30),
|
||||
chipBorderConnecting = Color(0xFF1E3358),
|
||||
chipBorderWarning = Color(0xFF3E3018),
|
||||
chipBorderError = Color(0xFF3E1E1E),
|
||||
)
|
||||
|
||||
internal val LocalMobileColors = staticCompositionLocalOf { lightMobileColors() }
|
||||
|
||||
internal object MobileColorsAccessor {
|
||||
val current: MobileColors
|
||||
@Composable get() = LocalMobileColors.current
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Backward-compatible top-level accessors (composable getters)
|
||||
// ---------------------------------------------------------------------------
|
||||
// These allow existing call sites to keep using `mobileSurface`, `mobileText`, etc.
|
||||
// without converting every file at once. Each resolves to the themed value.
|
||||
|
||||
internal val mobileSurface: Color @Composable get() = LocalMobileColors.current.surface
|
||||
internal val mobileSurfaceStrong: Color @Composable get() = LocalMobileColors.current.surfaceStrong
|
||||
internal val mobileCardSurface: Color @Composable get() = LocalMobileColors.current.cardSurface
|
||||
internal val mobileBorder: Color @Composable get() = LocalMobileColors.current.border
|
||||
internal val mobileBorderStrong: Color @Composable get() = LocalMobileColors.current.borderStrong
|
||||
internal val mobileText: Color @Composable get() = LocalMobileColors.current.text
|
||||
internal val mobileTextSecondary: Color @Composable get() = LocalMobileColors.current.textSecondary
|
||||
internal val mobileTextTertiary: Color @Composable get() = LocalMobileColors.current.textTertiary
|
||||
internal val mobileAccent: Color @Composable get() = LocalMobileColors.current.accent
|
||||
internal val mobileAccentSoft: Color @Composable get() = LocalMobileColors.current.accentSoft
|
||||
internal val mobileAccentBorderStrong: Color @Composable get() = LocalMobileColors.current.accentBorderStrong
|
||||
internal val mobileSuccess: Color @Composable get() = LocalMobileColors.current.success
|
||||
internal val mobileSuccessSoft: Color @Composable get() = LocalMobileColors.current.successSoft
|
||||
internal val mobileWarning: Color @Composable get() = LocalMobileColors.current.warning
|
||||
internal val mobileWarningSoft: Color @Composable get() = LocalMobileColors.current.warningSoft
|
||||
internal val mobileDanger: Color @Composable get() = LocalMobileColors.current.danger
|
||||
internal val mobileDangerSoft: Color @Composable get() = LocalMobileColors.current.dangerSoft
|
||||
internal val mobileCodeBg: Color @Composable get() = LocalMobileColors.current.codeBg
|
||||
internal val mobileCodeText: Color @Composable get() = LocalMobileColors.current.codeText
|
||||
internal val mobileCodeBorder: Color @Composable get() = LocalMobileColors.current.codeBorder
|
||||
internal val mobileCodeAccent: Color @Composable get() = LocalMobileColors.current.codeAccent
|
||||
|
||||
// Background gradient – light fades white→gray, dark fades near-black→dark-gray
|
||||
internal val mobileBackgroundGradient: Brush
|
||||
@Composable get() {
|
||||
val colors = LocalMobileColors.current
|
||||
return Brush.verticalGradient(
|
||||
listOf(
|
||||
colors.surface,
|
||||
colors.surfaceStrong,
|
||||
colors.surfaceStrong,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Typography tokens (theme-independent)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
internal val mobileFontFamily =
|
||||
FontFamily(
|
||||
Font(resId = R.font.manrope_400_regular, weight = FontWeight.Normal),
|
||||
Font(resId = R.font.manrope_500_medium, weight = FontWeight.Medium),
|
||||
Font(resId = R.font.manrope_600_semibold, weight = FontWeight.SemiBold),
|
||||
Font(resId = R.font.manrope_700_bold, weight = FontWeight.Bold),
|
||||
)
|
||||
|
||||
internal val mobileDisplay =
|
||||
TextStyle(
|
||||
fontFamily = mobileFontFamily,
|
||||
fontWeight = FontWeight.Bold,
|
||||
fontSize = 34.sp,
|
||||
lineHeight = 40.sp,
|
||||
letterSpacing = (-0.8).sp,
|
||||
)
|
||||
|
||||
internal val mobileTitle1 =
|
||||
TextStyle(
|
||||
fontFamily = mobileFontFamily,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
fontSize = 24.sp,
|
||||
lineHeight = 30.sp,
|
||||
letterSpacing = (-0.5).sp,
|
||||
)
|
||||
|
||||
internal val mobileTitle2 =
|
||||
TextStyle(
|
||||
fontFamily = mobileFontFamily,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
fontSize = 20.sp,
|
||||
lineHeight = 26.sp,
|
||||
letterSpacing = (-0.3).sp,
|
||||
)
|
||||
|
||||
internal val mobileHeadline =
|
||||
TextStyle(
|
||||
fontFamily = mobileFontFamily,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
fontSize = 16.sp,
|
||||
lineHeight = 22.sp,
|
||||
letterSpacing = (-0.1).sp,
|
||||
)
|
||||
|
||||
internal val mobileBody =
|
||||
TextStyle(
|
||||
fontFamily = mobileFontFamily,
|
||||
fontWeight = FontWeight.Medium,
|
||||
fontSize = 15.sp,
|
||||
lineHeight = 22.sp,
|
||||
)
|
||||
|
||||
internal val mobileCallout =
|
||||
TextStyle(
|
||||
fontFamily = mobileFontFamily,
|
||||
fontWeight = FontWeight.Medium,
|
||||
fontSize = 14.sp,
|
||||
lineHeight = 20.sp,
|
||||
)
|
||||
|
||||
internal val mobileCaption1 =
|
||||
TextStyle(
|
||||
fontFamily = mobileFontFamily,
|
||||
fontWeight = FontWeight.Medium,
|
||||
fontSize = 12.sp,
|
||||
lineHeight = 16.sp,
|
||||
letterSpacing = 0.2.sp,
|
||||
)
|
||||
|
||||
internal val mobileCaption2 =
|
||||
TextStyle(
|
||||
fontFamily = mobileFontFamily,
|
||||
fontWeight = FontWeight.Medium,
|
||||
fontSize = 11.sp,
|
||||
lineHeight = 14.sp,
|
||||
letterSpacing = 0.4.sp,
|
||||
)
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,49 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import android.app.Activity
|
||||
import androidx.compose.foundation.isSystemInDarkTheme
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.dynamicDarkColorScheme
|
||||
import androidx.compose.material3.dynamicLightColorScheme
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.CompositionLocalProvider
|
||||
import androidx.compose.runtime.SideEffect
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.platform.LocalContext
|
||||
import androidx.compose.ui.platform.LocalView
|
||||
import androidx.core.view.WindowCompat
|
||||
|
||||
@Composable
|
||||
fun OpenClawTheme(content: @Composable () -> Unit) {
|
||||
val context = LocalContext.current
|
||||
val isDark = isSystemInDarkTheme()
|
||||
val colorScheme = if (isDark) dynamicDarkColorScheme(context) else dynamicLightColorScheme(context)
|
||||
val mobileColors = if (isDark) darkMobileColors() else lightMobileColors()
|
||||
|
||||
val view = LocalView.current
|
||||
if (!view.isInEditMode) {
|
||||
SideEffect {
|
||||
val window = (view.context as Activity).window
|
||||
WindowCompat.getInsetsController(window, window.decorView)
|
||||
.isAppearanceLightStatusBars = !isDark
|
||||
}
|
||||
}
|
||||
|
||||
CompositionLocalProvider(LocalMobileColors provides mobileColors) {
|
||||
MaterialTheme(colorScheme = colorScheme, content = content)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun overlayContainerColor(): Color {
|
||||
val scheme = MaterialTheme.colorScheme
|
||||
val isDark = isSystemInDarkTheme()
|
||||
val base = if (isDark) scheme.surfaceContainerLow else scheme.surfaceContainerHigh
|
||||
// Light mode: background stays dark (canvas), so clamp overlays away from pure-white glare.
|
||||
return if (isDark) base else base.copy(alpha = 0.88f)
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun overlayIconColor(): Color {
|
||||
return MaterialTheme.colorScheme.onSurfaceVariant
|
||||
}
|
||||
|
|
@ -0,0 +1,350 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import androidx.compose.foundation.background
|
||||
import androidx.compose.foundation.BorderStroke
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.WindowInsets
|
||||
import androidx.compose.foundation.layout.WindowInsetsSides
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.heightIn
|
||||
import androidx.compose.foundation.layout.ime
|
||||
import androidx.compose.foundation.layout.navigationBars
|
||||
import androidx.compose.foundation.layout.only
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.safeDrawing
|
||||
import androidx.compose.foundation.layout.windowInsetsPadding
|
||||
import androidx.compose.foundation.layout.consumeWindowInsets
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.automirrored.filled.ScreenShare
|
||||
import androidx.compose.material.icons.filled.ChatBubble
|
||||
import androidx.compose.material.icons.filled.CheckCircle
|
||||
import androidx.compose.material.icons.filled.RecordVoiceOver
|
||||
import androidx.compose.material.icons.filled.Settings
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.Scaffold
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.collectAsState
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.saveable.rememberSaveable
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.draw.alpha
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.zIndex
|
||||
import androidx.compose.ui.graphics.vector.ImageVector
|
||||
import androidx.compose.ui.platform.LocalDensity
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.unit.dp
|
||||
import ai.openclaw.app.HomeDestination
|
||||
import ai.openclaw.app.MainViewModel
|
||||
|
||||
private enum class HomeTab(
|
||||
val label: String,
|
||||
val icon: ImageVector,
|
||||
) {
|
||||
Connect(label = "Connect", icon = Icons.Default.CheckCircle),
|
||||
Chat(label = "Chat", icon = Icons.Default.ChatBubble),
|
||||
Voice(label = "Voice", icon = Icons.Default.RecordVoiceOver),
|
||||
Screen(label = "Screen", icon = Icons.AutoMirrored.Filled.ScreenShare),
|
||||
Settings(label = "Settings", icon = Icons.Default.Settings),
|
||||
}
|
||||
|
||||
private enum class StatusVisual {
|
||||
Connected,
|
||||
Connecting,
|
||||
Warning,
|
||||
Error,
|
||||
Offline,
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun PostOnboardingTabs(viewModel: MainViewModel, modifier: Modifier = Modifier) {
|
||||
var activeTab by rememberSaveable { mutableStateOf(HomeTab.Connect) }
|
||||
var chatTabStarted by rememberSaveable { mutableStateOf(false) }
|
||||
var screenTabStarted by rememberSaveable { mutableStateOf(false) }
|
||||
val requestedHomeDestination by viewModel.requestedHomeDestination.collectAsState()
|
||||
|
||||
LaunchedEffect(requestedHomeDestination) {
|
||||
val destination = requestedHomeDestination ?: return@LaunchedEffect
|
||||
activeTab =
|
||||
when (destination) {
|
||||
HomeDestination.Connect -> HomeTab.Connect
|
||||
HomeDestination.Chat -> HomeTab.Chat
|
||||
HomeDestination.Voice -> HomeTab.Voice
|
||||
HomeDestination.Screen -> HomeTab.Screen
|
||||
HomeDestination.Settings -> HomeTab.Settings
|
||||
}
|
||||
viewModel.clearRequestedHomeDestination()
|
||||
}
|
||||
|
||||
// Stop TTS when user navigates away from voice tab, and lazily keep the Chat/Screen tabs
|
||||
// alive after the first visit so repeated tab switches do not rebuild their UI trees.
|
||||
LaunchedEffect(activeTab) {
|
||||
viewModel.setVoiceScreenActive(activeTab == HomeTab.Voice)
|
||||
if (activeTab == HomeTab.Chat) {
|
||||
chatTabStarted = true
|
||||
}
|
||||
if (activeTab == HomeTab.Screen) {
|
||||
screenTabStarted = true
|
||||
}
|
||||
}
|
||||
|
||||
val statusText by viewModel.statusText.collectAsState()
|
||||
val isConnected by viewModel.isConnected.collectAsState()
|
||||
|
||||
val statusVisual =
|
||||
remember(statusText, isConnected) {
|
||||
val lower = statusText.lowercase()
|
||||
when {
|
||||
isConnected -> StatusVisual.Connected
|
||||
lower.contains("connecting") || lower.contains("reconnecting") -> StatusVisual.Connecting
|
||||
lower.contains("pairing") || lower.contains("approval") || lower.contains("auth") -> StatusVisual.Warning
|
||||
lower.contains("error") || lower.contains("failed") -> StatusVisual.Error
|
||||
else -> StatusVisual.Offline
|
||||
}
|
||||
}
|
||||
|
||||
val density = LocalDensity.current
|
||||
val imeVisible = WindowInsets.ime.getBottom(density) > 0
|
||||
val hideBottomTabBar = activeTab == HomeTab.Chat && imeVisible
|
||||
|
||||
Scaffold(
|
||||
modifier = modifier,
|
||||
containerColor = Color.Transparent,
|
||||
contentWindowInsets = WindowInsets(0, 0, 0, 0),
|
||||
topBar = {
|
||||
TopStatusBar(
|
||||
statusText = statusText,
|
||||
statusVisual = statusVisual,
|
||||
)
|
||||
},
|
||||
bottomBar = {
|
||||
if (!hideBottomTabBar) {
|
||||
BottomTabBar(
|
||||
activeTab = activeTab,
|
||||
onSelect = { activeTab = it },
|
||||
)
|
||||
}
|
||||
},
|
||||
) { innerPadding ->
|
||||
Box(
|
||||
modifier =
|
||||
Modifier
|
||||
.fillMaxSize()
|
||||
.padding(innerPadding)
|
||||
.consumeWindowInsets(innerPadding)
|
||||
.background(mobileBackgroundGradient),
|
||||
) {
|
||||
if (chatTabStarted) {
|
||||
Box(
|
||||
modifier =
|
||||
Modifier
|
||||
.matchParentSize()
|
||||
.alpha(if (activeTab == HomeTab.Chat) 1f else 0f)
|
||||
.zIndex(if (activeTab == HomeTab.Chat) 1f else 0f),
|
||||
) {
|
||||
ChatSheet(viewModel = viewModel)
|
||||
}
|
||||
}
|
||||
|
||||
if (screenTabStarted) {
|
||||
ScreenTabScreen(
|
||||
viewModel = viewModel,
|
||||
visible = activeTab == HomeTab.Screen,
|
||||
modifier =
|
||||
Modifier
|
||||
.matchParentSize()
|
||||
.alpha(if (activeTab == HomeTab.Screen) 1f else 0f)
|
||||
.zIndex(if (activeTab == HomeTab.Screen) 1f else 0f),
|
||||
)
|
||||
}
|
||||
|
||||
when (activeTab) {
|
||||
HomeTab.Connect -> ConnectTabScreen(viewModel = viewModel)
|
||||
HomeTab.Chat -> if (!chatTabStarted) ChatSheet(viewModel = viewModel)
|
||||
HomeTab.Voice -> VoiceTabScreen(viewModel = viewModel)
|
||||
HomeTab.Screen -> Unit
|
||||
HomeTab.Settings -> SettingsSheet(viewModel = viewModel)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ScreenTabScreen(viewModel: MainViewModel, visible: Boolean, modifier: Modifier = Modifier) {
|
||||
val isConnected by viewModel.isConnected.collectAsState()
|
||||
var refreshedForCurrentConnection by rememberSaveable(isConnected) { mutableStateOf(false) }
|
||||
|
||||
LaunchedEffect(isConnected, visible, refreshedForCurrentConnection) {
|
||||
if (visible && isConnected && !refreshedForCurrentConnection) {
|
||||
viewModel.refreshHomeCanvasOverviewIfConnected()
|
||||
refreshedForCurrentConnection = true
|
||||
}
|
||||
}
|
||||
|
||||
Box(modifier = modifier.fillMaxSize()) {
|
||||
CanvasScreen(viewModel = viewModel, visible = visible, modifier = Modifier.fillMaxSize())
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun TopStatusBar(
|
||||
statusText: String,
|
||||
statusVisual: StatusVisual,
|
||||
) {
|
||||
val safeInsets = WindowInsets.safeDrawing.only(WindowInsetsSides.Top + WindowInsetsSides.Horizontal)
|
||||
|
||||
val (chipBg, chipDot, chipText, chipBorder) =
|
||||
when (statusVisual) {
|
||||
StatusVisual.Connected ->
|
||||
listOf(
|
||||
mobileSuccessSoft,
|
||||
mobileSuccess,
|
||||
mobileSuccess,
|
||||
LocalMobileColors.current.chipBorderConnected,
|
||||
)
|
||||
StatusVisual.Connecting ->
|
||||
listOf(
|
||||
mobileAccentSoft,
|
||||
mobileAccent,
|
||||
mobileAccent,
|
||||
LocalMobileColors.current.chipBorderConnecting,
|
||||
)
|
||||
StatusVisual.Warning ->
|
||||
listOf(
|
||||
mobileWarningSoft,
|
||||
mobileWarning,
|
||||
mobileWarning,
|
||||
LocalMobileColors.current.chipBorderWarning,
|
||||
)
|
||||
StatusVisual.Error ->
|
||||
listOf(
|
||||
mobileDangerSoft,
|
||||
mobileDanger,
|
||||
mobileDanger,
|
||||
LocalMobileColors.current.chipBorderError,
|
||||
)
|
||||
StatusVisual.Offline ->
|
||||
listOf(
|
||||
mobileSurface,
|
||||
mobileTextTertiary,
|
||||
mobileTextSecondary,
|
||||
mobileBorder,
|
||||
)
|
||||
}
|
||||
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth().windowInsetsPadding(safeInsets),
|
||||
color = Color.Transparent,
|
||||
shadowElevation = 0.dp,
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth().padding(horizontal = 18.dp, vertical = 12.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.SpaceBetween,
|
||||
) {
|
||||
Text(
|
||||
text = "OpenClaw",
|
||||
style = mobileTitle2,
|
||||
color = mobileText,
|
||||
)
|
||||
Surface(
|
||||
shape = RoundedCornerShape(999.dp),
|
||||
color = chipBg,
|
||||
border = androidx.compose.foundation.BorderStroke(1.dp, chipBorder),
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.padding(horizontal = 10.dp, vertical = 5.dp),
|
||||
horizontalArrangement = Arrangement.spacedBy(6.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
) {
|
||||
Surface(
|
||||
modifier = Modifier.padding(top = 1.dp),
|
||||
color = chipDot,
|
||||
shape = RoundedCornerShape(999.dp),
|
||||
) {
|
||||
Box(modifier = Modifier.padding(4.dp))
|
||||
}
|
||||
Text(
|
||||
text = statusText.trim().ifEmpty { "Offline" },
|
||||
style = mobileCaption1,
|
||||
color = chipText,
|
||||
maxLines = 1,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun BottomTabBar(
|
||||
activeTab: HomeTab,
|
||||
onSelect: (HomeTab) -> Unit,
|
||||
) {
|
||||
val safeInsets = WindowInsets.navigationBars.only(WindowInsetsSides.Bottom + WindowInsetsSides.Horizontal)
|
||||
|
||||
Box(
|
||||
modifier =
|
||||
Modifier
|
||||
.fillMaxWidth(),
|
||||
) {
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
color = mobileCardSurface.copy(alpha = 0.97f),
|
||||
shape = RoundedCornerShape(topStart = 24.dp, topEnd = 24.dp),
|
||||
border = BorderStroke(1.dp, mobileBorder),
|
||||
shadowElevation = 6.dp,
|
||||
) {
|
||||
Row(
|
||||
modifier =
|
||||
Modifier
|
||||
.fillMaxWidth()
|
||||
.windowInsetsPadding(safeInsets)
|
||||
.padding(horizontal = 10.dp, vertical = 10.dp),
|
||||
horizontalArrangement = Arrangement.spacedBy(6.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
) {
|
||||
HomeTab.entries.forEach { tab ->
|
||||
val active = tab == activeTab
|
||||
Surface(
|
||||
onClick = { onSelect(tab) },
|
||||
modifier = Modifier.weight(1f).heightIn(min = 58.dp),
|
||||
shape = RoundedCornerShape(16.dp),
|
||||
color = if (active) mobileAccentSoft else Color.Transparent,
|
||||
border = if (active) BorderStroke(1.dp, LocalMobileColors.current.chipBorderConnecting) else null,
|
||||
shadowElevation = 0.dp,
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.fillMaxWidth().padding(horizontal = 6.dp, vertical = 7.dp),
|
||||
horizontalAlignment = Alignment.CenterHorizontally,
|
||||
verticalArrangement = Arrangement.spacedBy(2.dp),
|
||||
) {
|
||||
Icon(
|
||||
imageVector = tab.icon,
|
||||
contentDescription = tab.label,
|
||||
tint = if (active) mobileAccent else mobileTextTertiary,
|
||||
)
|
||||
Text(
|
||||
text = tab.label,
|
||||
color = if (active) mobileAccent else mobileTextSecondary,
|
||||
style = mobileCaption2.copy(fontWeight = if (active) FontWeight.Bold else FontWeight.Medium),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.collectAsState
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.ui.Modifier
|
||||
import ai.openclaw.app.MainViewModel
|
||||
|
||||
@Composable
|
||||
fun RootScreen(viewModel: MainViewModel) {
|
||||
val onboardingCompleted by viewModel.onboardingCompleted.collectAsState()
|
||||
|
||||
if (!onboardingCompleted) {
|
||||
OnboardingFlow(viewModel = viewModel, modifier = Modifier.fillMaxSize())
|
||||
return
|
||||
}
|
||||
|
||||
PostOnboardingTabs(viewModel = viewModel, modifier = Modifier.fillMaxSize())
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,134 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import androidx.compose.animation.core.LinearEasing
|
||||
import androidx.compose.animation.core.RepeatMode
|
||||
import androidx.compose.animation.core.animateFloat
|
||||
import androidx.compose.animation.core.infiniteRepeatable
|
||||
import androidx.compose.animation.core.rememberInfiniteTransition
|
||||
import androidx.compose.animation.core.tween
|
||||
import androidx.compose.foundation.Canvas
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.shape.CircleShape
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.graphics.Brush
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.graphics.drawscope.Stroke
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.unit.dp
|
||||
|
||||
@Composable
|
||||
fun TalkOrbOverlay(
|
||||
seamColor: Color,
|
||||
statusText: String,
|
||||
isListening: Boolean,
|
||||
isSpeaking: Boolean,
|
||||
modifier: Modifier = Modifier,
|
||||
) {
|
||||
val transition = rememberInfiniteTransition(label = "talk-orb")
|
||||
val t by
|
||||
transition.animateFloat(
|
||||
initialValue = 0f,
|
||||
targetValue = 1f,
|
||||
animationSpec =
|
||||
infiniteRepeatable(
|
||||
animation = tween(durationMillis = 1500, easing = LinearEasing),
|
||||
repeatMode = RepeatMode.Restart,
|
||||
),
|
||||
label = "pulse",
|
||||
)
|
||||
|
||||
val trimmed = statusText.trim()
|
||||
val showStatus = trimmed.isNotEmpty() && trimmed != "Off"
|
||||
val phase =
|
||||
when {
|
||||
isSpeaking -> "Speaking"
|
||||
isListening -> "Listening"
|
||||
else -> "Thinking"
|
||||
}
|
||||
|
||||
Column(
|
||||
modifier = modifier.padding(24.dp),
|
||||
horizontalAlignment = Alignment.CenterHorizontally,
|
||||
verticalArrangement = Arrangement.spacedBy(12.dp),
|
||||
) {
|
||||
Box(contentAlignment = Alignment.Center) {
|
||||
Canvas(modifier = Modifier.size(360.dp)) {
|
||||
val center = this.center
|
||||
val baseRadius = size.minDimension * 0.30f
|
||||
|
||||
val ring1 = 1.05f + (t * 0.25f)
|
||||
val ring2 = 1.20f + (t * 0.55f)
|
||||
val ringAlpha1 = (1f - t) * 0.34f
|
||||
val ringAlpha2 = (1f - t) * 0.22f
|
||||
|
||||
drawCircle(
|
||||
color = seamColor.copy(alpha = ringAlpha1),
|
||||
radius = baseRadius * ring1,
|
||||
center = center,
|
||||
style = Stroke(width = 3.dp.toPx()),
|
||||
)
|
||||
drawCircle(
|
||||
color = seamColor.copy(alpha = ringAlpha2),
|
||||
radius = baseRadius * ring2,
|
||||
center = center,
|
||||
style = Stroke(width = 3.dp.toPx()),
|
||||
)
|
||||
|
||||
drawCircle(
|
||||
brush =
|
||||
Brush.radialGradient(
|
||||
colors =
|
||||
listOf(
|
||||
seamColor.copy(alpha = 0.92f),
|
||||
seamColor.copy(alpha = 0.40f),
|
||||
Color.Black.copy(alpha = 0.56f),
|
||||
),
|
||||
center = center,
|
||||
radius = baseRadius * 1.35f,
|
||||
),
|
||||
radius = baseRadius,
|
||||
center = center,
|
||||
)
|
||||
|
||||
drawCircle(
|
||||
color = seamColor.copy(alpha = 0.34f),
|
||||
radius = baseRadius,
|
||||
center = center,
|
||||
style = Stroke(width = 1.dp.toPx()),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if (showStatus) {
|
||||
Surface(
|
||||
color = Color.Black.copy(alpha = 0.40f),
|
||||
shape = CircleShape,
|
||||
) {
|
||||
Text(
|
||||
text = trimmed,
|
||||
modifier = Modifier.padding(horizontal = 14.dp, vertical = 8.dp),
|
||||
color = Color.White.copy(alpha = 0.92f),
|
||||
style = MaterialTheme.typography.labelLarge,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
Text(
|
||||
text = phase,
|
||||
color = Color.White.copy(alpha = 0.80f),
|
||||
style = MaterialTheme.typography.labelLarge,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,448 @@
|
|||
package ai.openclaw.app.ui
|
||||
|
||||
import android.Manifest
|
||||
import android.app.Activity
|
||||
import android.content.Context
|
||||
import android.content.ContextWrapper
|
||||
import android.content.Intent
|
||||
import android.content.pm.PackageManager
|
||||
import android.net.Uri
|
||||
import android.provider.Settings
|
||||
import androidx.activity.compose.rememberLauncherForActivityResult
|
||||
import androidx.activity.result.contract.ActivityResultContracts
|
||||
import androidx.compose.foundation.BorderStroke
|
||||
import androidx.compose.foundation.background
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.PaddingValues
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
import androidx.compose.foundation.layout.WindowInsets
|
||||
import androidx.compose.foundation.layout.WindowInsetsSides
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.height
|
||||
import androidx.compose.foundation.layout.imePadding
|
||||
import androidx.compose.foundation.layout.only
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.safeDrawing
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.layout.windowInsetsPadding
|
||||
import androidx.compose.foundation.lazy.LazyColumn
|
||||
import androidx.compose.foundation.lazy.items
|
||||
import androidx.compose.foundation.lazy.rememberLazyListState
|
||||
import androidx.compose.foundation.shape.CircleShape
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.filled.Mic
|
||||
import androidx.compose.material.icons.filled.MicOff
|
||||
import androidx.compose.material.icons.automirrored.filled.VolumeOff
|
||||
import androidx.compose.material.icons.automirrored.filled.VolumeUp
|
||||
import androidx.compose.material3.Button
|
||||
import androidx.compose.material3.ButtonDefaults
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.IconButton
|
||||
import androidx.compose.material3.IconButtonDefaults
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.DisposableEffect
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.collectAsState
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.draw.alpha
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.platform.LocalContext
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.text.style.TextAlign
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.compose.ui.unit.sp
|
||||
import androidx.core.app.ActivityCompat
|
||||
import androidx.core.content.ContextCompat
|
||||
import androidx.lifecycle.Lifecycle
|
||||
import androidx.lifecycle.LifecycleEventObserver
|
||||
import androidx.lifecycle.compose.LocalLifecycleOwner
|
||||
import ai.openclaw.app.MainViewModel
|
||||
import ai.openclaw.app.voice.VoiceConversationEntry
|
||||
import ai.openclaw.app.voice.VoiceConversationRole
|
||||
import kotlin.math.max
|
||||
|
||||
@Composable
|
||||
fun VoiceTabScreen(viewModel: MainViewModel) {
|
||||
val context = LocalContext.current
|
||||
val lifecycleOwner = LocalLifecycleOwner.current
|
||||
val activity = remember(context) { context.findActivity() }
|
||||
val listState = rememberLazyListState()
|
||||
|
||||
val gatewayStatus by viewModel.statusText.collectAsState()
|
||||
val micEnabled by viewModel.micEnabled.collectAsState()
|
||||
val micCooldown by viewModel.micCooldown.collectAsState()
|
||||
val speakerEnabled by viewModel.speakerEnabled.collectAsState()
|
||||
val micStatusText by viewModel.micStatusText.collectAsState()
|
||||
val micLiveTranscript by viewModel.micLiveTranscript.collectAsState()
|
||||
val micQueuedMessages by viewModel.micQueuedMessages.collectAsState()
|
||||
val micConversation by viewModel.micConversation.collectAsState()
|
||||
val micInputLevel by viewModel.micInputLevel.collectAsState()
|
||||
val micIsSending by viewModel.micIsSending.collectAsState()
|
||||
|
||||
val hasStreamingAssistant = micConversation.any { it.role == VoiceConversationRole.Assistant && it.isStreaming }
|
||||
val showThinkingBubble = micIsSending && !hasStreamingAssistant
|
||||
|
||||
var hasMicPermission by remember { mutableStateOf(context.hasRecordAudioPermission()) }
|
||||
var pendingMicEnable by remember { mutableStateOf(false) }
|
||||
|
||||
DisposableEffect(lifecycleOwner, context) {
|
||||
val observer =
|
||||
LifecycleEventObserver { _, event ->
|
||||
if (event == Lifecycle.Event.ON_RESUME) {
|
||||
hasMicPermission = context.hasRecordAudioPermission()
|
||||
}
|
||||
}
|
||||
lifecycleOwner.lifecycle.addObserver(observer)
|
||||
onDispose {
|
||||
lifecycleOwner.lifecycle.removeObserver(observer)
|
||||
// Stop TTS when leaving the voice screen
|
||||
viewModel.setVoiceScreenActive(false)
|
||||
}
|
||||
}
|
||||
|
||||
val requestMicPermission =
|
||||
rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
|
||||
hasMicPermission = granted
|
||||
if (granted && pendingMicEnable) {
|
||||
viewModel.setMicEnabled(true)
|
||||
}
|
||||
pendingMicEnable = false
|
||||
}
|
||||
|
||||
LaunchedEffect(micConversation.size, showThinkingBubble) {
|
||||
val total = micConversation.size + if (showThinkingBubble) 1 else 0
|
||||
if (total > 0) {
|
||||
listState.animateScrollToItem(total - 1)
|
||||
}
|
||||
}
|
||||
|
||||
Column(
|
||||
modifier =
|
||||
Modifier
|
||||
.fillMaxSize()
|
||||
.background(mobileBackgroundGradient)
|
||||
.imePadding()
|
||||
.windowInsetsPadding(WindowInsets.safeDrawing.only(WindowInsetsSides.Bottom))
|
||||
.padding(horizontal = 20.dp, vertical = 14.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(10.dp),
|
||||
) {
|
||||
LazyColumn(
|
||||
state = listState,
|
||||
modifier = Modifier.fillMaxWidth().weight(1f),
|
||||
contentPadding = PaddingValues(vertical = 4.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(10.dp),
|
||||
) {
|
||||
if (micConversation.isEmpty() && !showThinkingBubble) {
|
||||
item {
|
||||
Box(
|
||||
modifier = Modifier.fillParentMaxHeight().fillMaxWidth(),
|
||||
contentAlignment = Alignment.Center,
|
||||
) {
|
||||
Column(
|
||||
horizontalAlignment = Alignment.CenterHorizontally,
|
||||
verticalArrangement = Arrangement.spacedBy(10.dp),
|
||||
) {
|
||||
Icon(
|
||||
imageVector = Icons.Default.Mic,
|
||||
contentDescription = null,
|
||||
modifier = Modifier.size(48.dp),
|
||||
tint = mobileTextTertiary,
|
||||
)
|
||||
Text(
|
||||
"Tap the mic to start",
|
||||
style = mobileHeadline,
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
Text(
|
||||
"Each pause sends a turn automatically.",
|
||||
style = mobileCallout,
|
||||
color = mobileTextTertiary,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
items(items = micConversation, key = { it.id }) { entry ->
|
||||
VoiceTurnBubble(entry = entry)
|
||||
}
|
||||
|
||||
if (showThinkingBubble) {
|
||||
item {
|
||||
VoiceThinkingBubble()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Column(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalAlignment = Alignment.CenterHorizontally,
|
||||
verticalArrangement = Arrangement.spacedBy(6.dp),
|
||||
) {
|
||||
if (!micLiveTranscript.isNullOrBlank()) {
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
color = mobileAccentSoft,
|
||||
border = BorderStroke(1.dp, mobileAccent.copy(alpha = 0.2f)),
|
||||
) {
|
||||
Text(
|
||||
micLiveTranscript!!.trim(),
|
||||
modifier = Modifier.padding(horizontal = 12.dp, vertical = 10.dp),
|
||||
style = mobileCallout,
|
||||
color = mobileText,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// Mic button with input-reactive ring + speaker toggle
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalArrangement = Arrangement.Center,
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
) {
|
||||
// Speaker toggle
|
||||
Column(horizontalAlignment = Alignment.CenterHorizontally, verticalArrangement = Arrangement.spacedBy(4.dp)) {
|
||||
IconButton(
|
||||
onClick = { viewModel.setSpeakerEnabled(!speakerEnabled) },
|
||||
modifier = Modifier.size(48.dp),
|
||||
colors =
|
||||
IconButtonDefaults.iconButtonColors(
|
||||
containerColor = if (speakerEnabled) mobileSurface else mobileDangerSoft,
|
||||
),
|
||||
) {
|
||||
Icon(
|
||||
imageVector = if (speakerEnabled) Icons.AutoMirrored.Filled.VolumeUp else Icons.AutoMirrored.Filled.VolumeOff,
|
||||
contentDescription = if (speakerEnabled) "Mute speaker" else "Unmute speaker",
|
||||
modifier = Modifier.size(22.dp),
|
||||
tint = if (speakerEnabled) mobileTextSecondary else mobileDanger,
|
||||
)
|
||||
}
|
||||
Text(
|
||||
if (speakerEnabled) "Speaker" else "Muted",
|
||||
style = mobileCaption2,
|
||||
color = if (speakerEnabled) mobileTextTertiary else mobileDanger,
|
||||
)
|
||||
}
|
||||
|
||||
// Ring size = 68dp base + up to 22dp driven by mic input level.
|
||||
// The outer Box is fixed at 90dp (max ring size) so the ring never shifts the button.
|
||||
Box(
|
||||
modifier = Modifier.padding(horizontal = 16.dp).size(90.dp),
|
||||
contentAlignment = Alignment.Center,
|
||||
) {
|
||||
if (micEnabled) {
|
||||
val ringLevel = micInputLevel.coerceIn(0f, 1f)
|
||||
val ringSize = 68.dp + (22.dp * max(ringLevel, 0.05f))
|
||||
Box(
|
||||
modifier =
|
||||
Modifier
|
||||
.size(ringSize)
|
||||
.background(mobileAccent.copy(alpha = 0.12f + 0.14f * ringLevel), CircleShape),
|
||||
)
|
||||
}
|
||||
Button(
|
||||
onClick = {
|
||||
if (micCooldown) return@Button
|
||||
if (micEnabled) {
|
||||
viewModel.setMicEnabled(false)
|
||||
return@Button
|
||||
}
|
||||
if (hasMicPermission) {
|
||||
viewModel.setMicEnabled(true)
|
||||
} else {
|
||||
pendingMicEnable = true
|
||||
requestMicPermission.launch(Manifest.permission.RECORD_AUDIO)
|
||||
}
|
||||
},
|
||||
enabled = !micCooldown,
|
||||
shape = CircleShape,
|
||||
contentPadding = PaddingValues(0.dp),
|
||||
modifier = Modifier.size(60.dp),
|
||||
colors =
|
||||
ButtonDefaults.buttonColors(
|
||||
containerColor = if (micCooldown) mobileTextSecondary else if (micEnabled) mobileDanger else mobileAccent,
|
||||
contentColor = Color.White,
|
||||
disabledContainerColor = mobileTextSecondary,
|
||||
disabledContentColor = Color.White.copy(alpha = 0.5f),
|
||||
),
|
||||
) {
|
||||
Icon(
|
||||
imageVector = if (micEnabled) Icons.Default.MicOff else Icons.Default.Mic,
|
||||
contentDescription = if (micEnabled) "Turn microphone off" else "Turn microphone on",
|
||||
modifier = Modifier.size(24.dp),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// Invisible spacer to balance the row (matches speaker column width)
|
||||
Column(horizontalAlignment = Alignment.CenterHorizontally) {
|
||||
Box(modifier = Modifier.size(48.dp))
|
||||
Spacer(modifier = Modifier.height(4.dp))
|
||||
Text("", style = mobileCaption2)
|
||||
}
|
||||
}
|
||||
|
||||
// Status + labels
|
||||
val queueCount = micQueuedMessages.size
|
||||
val stateText =
|
||||
when {
|
||||
queueCount > 0 -> "$queueCount queued"
|
||||
micIsSending -> "Sending"
|
||||
micCooldown -> "Cooldown"
|
||||
micEnabled -> "Listening"
|
||||
else -> "Mic off"
|
||||
}
|
||||
val stateColor =
|
||||
when {
|
||||
micEnabled -> mobileSuccess
|
||||
micIsSending -> mobileAccent
|
||||
else -> mobileTextSecondary
|
||||
}
|
||||
Surface(
|
||||
shape = RoundedCornerShape(999.dp),
|
||||
color = if (micEnabled) mobileSuccessSoft else mobileSurface,
|
||||
border = BorderStroke(1.dp, if (micEnabled) mobileSuccess.copy(alpha = 0.3f) else mobileBorder),
|
||||
) {
|
||||
Text(
|
||||
"$gatewayStatus · $stateText",
|
||||
style = mobileCallout.copy(fontWeight = FontWeight.SemiBold),
|
||||
color = stateColor,
|
||||
modifier = Modifier.padding(horizontal = 14.dp, vertical = 6.dp),
|
||||
)
|
||||
}
|
||||
|
||||
if (!hasMicPermission) {
|
||||
val showRationale =
|
||||
if (activity == null) {
|
||||
false
|
||||
} else {
|
||||
ActivityCompat.shouldShowRequestPermissionRationale(activity, Manifest.permission.RECORD_AUDIO)
|
||||
}
|
||||
Text(
|
||||
if (showRationale) {
|
||||
"Microphone permission is required for voice mode."
|
||||
} else {
|
||||
"Microphone blocked. Open app settings to enable it."
|
||||
},
|
||||
style = mobileCaption1,
|
||||
color = mobileWarning,
|
||||
textAlign = TextAlign.Center,
|
||||
)
|
||||
Button(
|
||||
onClick = { openAppSettings(context) },
|
||||
shape = RoundedCornerShape(12.dp),
|
||||
colors = ButtonDefaults.buttonColors(containerColor = mobileSurfaceStrong, contentColor = mobileText),
|
||||
) {
|
||||
Text("Open settings", style = mobileCallout.copy(fontWeight = FontWeight.SemiBold))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun VoiceTurnBubble(entry: VoiceConversationEntry) {
|
||||
val isUser = entry.role == VoiceConversationRole.User
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalArrangement = if (isUser) Arrangement.End else Arrangement.Start,
|
||||
) {
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(0.90f),
|
||||
shape = RoundedCornerShape(12.dp),
|
||||
color = if (isUser) mobileAccentSoft else mobileCardSurface,
|
||||
border = BorderStroke(1.dp, if (isUser) mobileAccent else mobileBorderStrong),
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.fillMaxWidth().padding(horizontal = 11.dp, vertical = 8.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(3.dp),
|
||||
) {
|
||||
Text(
|
||||
if (isUser) "You" else "OpenClaw",
|
||||
style = mobileCaption2.copy(fontWeight = FontWeight.SemiBold, letterSpacing = 0.6.sp),
|
||||
color = if (isUser) mobileAccent else mobileTextSecondary,
|
||||
)
|
||||
Text(
|
||||
if (entry.isStreaming && entry.text.isBlank()) "Listening response…" else entry.text,
|
||||
style = mobileCallout,
|
||||
color = mobileText,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun VoiceThinkingBubble() {
|
||||
Row(modifier = Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.Start) {
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(0.68f),
|
||||
shape = RoundedCornerShape(12.dp),
|
||||
color = mobileCardSurface,
|
||||
border = BorderStroke(1.dp, mobileBorderStrong),
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.padding(horizontal = 11.dp, vertical = 8.dp),
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
) {
|
||||
ThinkingDots(color = mobileTextSecondary)
|
||||
Text("OpenClaw is thinking…", style = mobileCallout, color = mobileTextSecondary)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ThinkingDots(color: Color) {
|
||||
Row(horizontalArrangement = Arrangement.spacedBy(5.dp), verticalAlignment = Alignment.CenterVertically) {
|
||||
ThinkingDot(alpha = 0.38f, color = color)
|
||||
ThinkingDot(alpha = 0.62f, color = color)
|
||||
ThinkingDot(alpha = 0.90f, color = color)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ThinkingDot(alpha: Float, color: Color) {
|
||||
Surface(
|
||||
modifier = Modifier.size(6.dp).alpha(alpha),
|
||||
shape = CircleShape,
|
||||
color = color,
|
||||
) {}
|
||||
}
|
||||
|
||||
private fun Context.hasRecordAudioPermission(): Boolean {
|
||||
return (
|
||||
ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
)
|
||||
}
|
||||
|
||||
private fun Context.findActivity(): Activity? =
|
||||
when (this) {
|
||||
is Activity -> this
|
||||
is ContextWrapper -> baseContext.findActivity()
|
||||
else -> null
|
||||
}
|
||||
|
||||
private fun openAppSettings(context: Context) {
|
||||
val intent =
|
||||
Intent(
|
||||
Settings.ACTION_APPLICATION_DETAILS_SETTINGS,
|
||||
Uri.fromParts("package", context.packageName, null),
|
||||
)
|
||||
context.startActivity(intent)
|
||||
}
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
package ai.openclaw.app.ui.chat
|
||||
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.graphics.ImageBitmap
|
||||
import androidx.compose.ui.graphics.asImageBitmap
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
|
||||
internal data class Base64ImageState(
|
||||
val image: ImageBitmap?,
|
||||
val failed: Boolean,
|
||||
)
|
||||
|
||||
@Composable
|
||||
internal fun rememberBase64ImageState(base64: String): Base64ImageState {
|
||||
var image by remember(base64) { mutableStateOf<ImageBitmap?>(null) }
|
||||
var failed by remember(base64) { mutableStateOf(false) }
|
||||
|
||||
LaunchedEffect(base64) {
|
||||
failed = false
|
||||
image =
|
||||
withContext(Dispatchers.Default) {
|
||||
try {
|
||||
val bitmap = decodeBase64Bitmap(base64) ?: return@withContext null
|
||||
bitmap.asImageBitmap()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
if (image == null) failed = true
|
||||
}
|
||||
|
||||
return Base64ImageState(image = image, failed = failed)
|
||||
}
|
||||
|
|
@ -0,0 +1,393 @@
|
|||
package ai.openclaw.app.ui.chat
|
||||
|
||||
import androidx.compose.foundation.BorderStroke
|
||||
import androidx.compose.foundation.horizontalScroll
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.PaddingValues
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.height
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.layout.width
|
||||
import androidx.compose.foundation.rememberScrollState
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.automirrored.filled.Send
|
||||
import androidx.compose.material.icons.filled.ArrowDropDown
|
||||
import androidx.compose.material.icons.filled.AttachFile
|
||||
import androidx.compose.material.icons.filled.Refresh
|
||||
import androidx.compose.material.icons.filled.Stop
|
||||
import androidx.compose.material3.Button
|
||||
import androidx.compose.material3.ButtonDefaults
|
||||
import androidx.compose.material3.CircularProgressIndicator
|
||||
import androidx.compose.material3.DropdownMenu
|
||||
import androidx.compose.material3.DropdownMenuItem
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.OutlinedTextField
|
||||
import androidx.compose.material3.OutlinedTextFieldDefaults
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.saveable.rememberSaveable
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.text.style.TextOverflow
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.compose.ui.unit.sp
|
||||
import ai.openclaw.app.ui.mobileAccent
|
||||
import ai.openclaw.app.ui.mobileAccentBorderStrong
|
||||
import ai.openclaw.app.ui.mobileAccentSoft
|
||||
import ai.openclaw.app.ui.mobileBorder
|
||||
import ai.openclaw.app.ui.mobileBorderStrong
|
||||
import ai.openclaw.app.ui.mobileCallout
|
||||
import ai.openclaw.app.ui.mobileCaption1
|
||||
import ai.openclaw.app.ui.mobileCardSurface
|
||||
import ai.openclaw.app.ui.mobileHeadline
|
||||
import ai.openclaw.app.ui.mobileSurface
|
||||
import ai.openclaw.app.ui.mobileText
|
||||
import ai.openclaw.app.ui.mobileTextSecondary
|
||||
import ai.openclaw.app.ui.mobileTextTertiary
|
||||
|
||||
internal data class DraftApplication(
|
||||
val input: String,
|
||||
val lastAppliedDraft: String?,
|
||||
val consumed: Boolean,
|
||||
)
|
||||
|
||||
internal fun applyDraftText(
|
||||
draftText: String?,
|
||||
currentInput: String,
|
||||
lastAppliedDraft: String?,
|
||||
): DraftApplication {
|
||||
val draft =
|
||||
draftText?.trim()?.ifEmpty { null } ?: return DraftApplication(
|
||||
input = currentInput,
|
||||
lastAppliedDraft = null,
|
||||
consumed = false,
|
||||
)
|
||||
if (draft == lastAppliedDraft) {
|
||||
return DraftApplication(
|
||||
input = currentInput,
|
||||
lastAppliedDraft = lastAppliedDraft,
|
||||
consumed = false,
|
||||
)
|
||||
}
|
||||
return DraftApplication(
|
||||
input = draft,
|
||||
lastAppliedDraft = draft,
|
||||
consumed = true,
|
||||
)
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun ChatComposer(
|
||||
draftText: String?,
|
||||
healthOk: Boolean,
|
||||
thinkingLevel: String,
|
||||
pendingRunCount: Int,
|
||||
attachments: List<PendingImageAttachment>,
|
||||
onDraftApplied: () -> Unit,
|
||||
onPickImages: () -> Unit,
|
||||
onRemoveAttachment: (id: String) -> Unit,
|
||||
onSetThinkingLevel: (level: String) -> Unit,
|
||||
onRefresh: () -> Unit,
|
||||
onAbort: () -> Unit,
|
||||
onSend: (text: String) -> Unit,
|
||||
) {
|
||||
var input by rememberSaveable { mutableStateOf("") }
|
||||
var lastAppliedDraft by rememberSaveable { mutableStateOf<String?>(null) }
|
||||
var showThinkingMenu by remember { mutableStateOf(false) }
|
||||
|
||||
LaunchedEffect(draftText) {
|
||||
val next = applyDraftText(draftText = draftText, currentInput = input, lastAppliedDraft = lastAppliedDraft)
|
||||
input = next.input
|
||||
lastAppliedDraft = next.lastAppliedDraft
|
||||
if (next.consumed) {
|
||||
onDraftApplied()
|
||||
}
|
||||
}
|
||||
|
||||
val canSend = pendingRunCount == 0 && (input.trim().isNotEmpty() || attachments.isNotEmpty()) && healthOk
|
||||
val sendBusy = pendingRunCount > 0
|
||||
|
||||
Column(modifier = Modifier.fillMaxWidth(), verticalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
if (attachments.isNotEmpty()) {
|
||||
AttachmentsStrip(attachments = attachments, onRemoveAttachment = onRemoveAttachment)
|
||||
}
|
||||
|
||||
OutlinedTextField(
|
||||
value = input,
|
||||
onValueChange = { input = it },
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
placeholder = { Text("Type a message…", style = mobileBodyStyle(), color = mobileTextTertiary) },
|
||||
minLines = 2,
|
||||
maxLines = 5,
|
||||
textStyle = mobileBodyStyle().copy(color = mobileText),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = chatTextFieldColors(),
|
||||
)
|
||||
|
||||
if (!healthOk) {
|
||||
Text(
|
||||
text = "Gateway is offline. Connect first in the Connect tab.",
|
||||
style = mobileCallout,
|
||||
color = ai.openclaw.app.ui.mobileWarning,
|
||||
)
|
||||
}
|
||||
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
) {
|
||||
Box {
|
||||
Surface(
|
||||
onClick = { showThinkingMenu = true },
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
color = mobileCardSurface,
|
||||
border = BorderStroke(1.dp, mobileBorderStrong),
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.padding(horizontal = 10.dp, vertical = 10.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
) {
|
||||
Text(
|
||||
text = thinkingLabel(thinkingLevel),
|
||||
style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold),
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
Icon(Icons.Default.ArrowDropDown, contentDescription = "Select thinking level", modifier = Modifier.size(18.dp), tint = mobileTextTertiary)
|
||||
}
|
||||
}
|
||||
|
||||
DropdownMenu(
|
||||
expanded = showThinkingMenu,
|
||||
onDismissRequest = { showThinkingMenu = false },
|
||||
shape = RoundedCornerShape(16.dp),
|
||||
containerColor = mobileCardSurface,
|
||||
tonalElevation = 0.dp,
|
||||
shadowElevation = 8.dp,
|
||||
border = BorderStroke(1.dp, mobileBorder),
|
||||
) {
|
||||
ThinkingMenuItem("off", thinkingLevel, onSetThinkingLevel) { showThinkingMenu = false }
|
||||
ThinkingMenuItem("low", thinkingLevel, onSetThinkingLevel) { showThinkingMenu = false }
|
||||
ThinkingMenuItem("medium", thinkingLevel, onSetThinkingLevel) { showThinkingMenu = false }
|
||||
ThinkingMenuItem("high", thinkingLevel, onSetThinkingLevel) { showThinkingMenu = false }
|
||||
}
|
||||
}
|
||||
|
||||
SecondaryActionButton(
|
||||
label = "Attach",
|
||||
icon = Icons.Default.AttachFile,
|
||||
enabled = true,
|
||||
compact = true,
|
||||
onClick = onPickImages,
|
||||
)
|
||||
|
||||
SecondaryActionButton(
|
||||
label = "Refresh",
|
||||
icon = Icons.Default.Refresh,
|
||||
enabled = true,
|
||||
compact = true,
|
||||
onClick = onRefresh,
|
||||
)
|
||||
|
||||
SecondaryActionButton(
|
||||
label = "Abort",
|
||||
icon = Icons.Default.Stop,
|
||||
enabled = pendingRunCount > 0,
|
||||
compact = true,
|
||||
onClick = onAbort,
|
||||
)
|
||||
|
||||
Spacer(modifier = Modifier.weight(1f))
|
||||
|
||||
Button(
|
||||
onClick = {
|
||||
val text = input
|
||||
input = ""
|
||||
onSend(text)
|
||||
},
|
||||
enabled = canSend,
|
||||
modifier = Modifier.height(44.dp),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
contentPadding = PaddingValues(horizontal = 20.dp),
|
||||
colors =
|
||||
ButtonDefaults.buttonColors(
|
||||
containerColor = mobileAccent,
|
||||
contentColor = Color.White,
|
||||
disabledContainerColor = mobileBorderStrong,
|
||||
disabledContentColor = mobileTextTertiary,
|
||||
),
|
||||
border = BorderStroke(1.dp, if (canSend) mobileAccentBorderStrong else mobileBorderStrong),
|
||||
) {
|
||||
if (sendBusy) {
|
||||
CircularProgressIndicator(modifier = Modifier.size(16.dp), strokeWidth = 2.dp, color = Color.White)
|
||||
} else {
|
||||
Icon(Icons.AutoMirrored.Filled.Send, contentDescription = null, modifier = Modifier.size(16.dp))
|
||||
}
|
||||
Spacer(modifier = Modifier.width(6.dp))
|
||||
Text(
|
||||
text = "Send",
|
||||
style = mobileHeadline.copy(fontWeight = FontWeight.Bold),
|
||||
maxLines = 1,
|
||||
overflow = TextOverflow.Ellipsis,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun SecondaryActionButton(
|
||||
label: String,
|
||||
icon: androidx.compose.ui.graphics.vector.ImageVector,
|
||||
enabled: Boolean,
|
||||
compact: Boolean = false,
|
||||
onClick: () -> Unit,
|
||||
) {
|
||||
Button(
|
||||
onClick = onClick,
|
||||
enabled = enabled,
|
||||
modifier = if (compact) Modifier.size(44.dp) else Modifier.height(44.dp),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors =
|
||||
ButtonDefaults.buttonColors(
|
||||
containerColor = mobileCardSurface,
|
||||
contentColor = mobileTextSecondary,
|
||||
disabledContainerColor = mobileCardSurface,
|
||||
disabledContentColor = mobileTextTertiary,
|
||||
),
|
||||
border = BorderStroke(1.dp, mobileBorderStrong),
|
||||
contentPadding = if (compact) PaddingValues(0.dp) else ButtonDefaults.ContentPadding,
|
||||
) {
|
||||
Icon(icon, contentDescription = label, modifier = Modifier.size(14.dp))
|
||||
if (!compact) {
|
||||
Spacer(modifier = Modifier.width(5.dp))
|
||||
Text(
|
||||
text = label,
|
||||
style = mobileCallout.copy(fontWeight = FontWeight.SemiBold),
|
||||
color = if (enabled) mobileTextSecondary else mobileTextTertiary,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ThinkingMenuItem(
|
||||
value: String,
|
||||
current: String,
|
||||
onSet: (String) -> Unit,
|
||||
onDismiss: () -> Unit,
|
||||
) {
|
||||
DropdownMenuItem(
|
||||
text = { Text(thinkingLabel(value), style = mobileCallout, color = mobileText) },
|
||||
onClick = {
|
||||
onSet(value)
|
||||
onDismiss()
|
||||
},
|
||||
trailingIcon = {
|
||||
if (value == current.trim().lowercase()) {
|
||||
Text("✓", style = mobileCallout, color = mobileAccent)
|
||||
} else {
|
||||
Spacer(modifier = Modifier.width(10.dp))
|
||||
}
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
private fun thinkingLabel(raw: String): String {
|
||||
return when (raw.trim().lowercase()) {
|
||||
"low" -> "Low"
|
||||
"medium" -> "Medium"
|
||||
"high" -> "High"
|
||||
else -> "Off"
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun AttachmentsStrip(
|
||||
attachments: List<PendingImageAttachment>,
|
||||
onRemoveAttachment: (id: String) -> Unit,
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth().horizontalScroll(rememberScrollState()),
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
) {
|
||||
for (att in attachments) {
|
||||
AttachmentChip(
|
||||
fileName = att.fileName,
|
||||
onRemove = { onRemoveAttachment(att.id) },
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun AttachmentChip(fileName: String, onRemove: () -> Unit) {
|
||||
Surface(
|
||||
shape = RoundedCornerShape(999.dp),
|
||||
color = mobileAccentSoft,
|
||||
border = BorderStroke(1.dp, mobileBorderStrong),
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.padding(horizontal = 10.dp, vertical = 6.dp),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
) {
|
||||
Text(
|
||||
text = fileName,
|
||||
style = mobileCaption1,
|
||||
color = mobileText,
|
||||
maxLines = 1,
|
||||
overflow = TextOverflow.Ellipsis,
|
||||
)
|
||||
Surface(
|
||||
onClick = onRemove,
|
||||
shape = RoundedCornerShape(999.dp),
|
||||
color = mobileCardSurface,
|
||||
border = BorderStroke(1.dp, mobileBorderStrong),
|
||||
) {
|
||||
Text(
|
||||
text = "×",
|
||||
style = mobileCaption1.copy(fontWeight = FontWeight.Bold),
|
||||
color = mobileTextSecondary,
|
||||
modifier = Modifier.padding(horizontal = 8.dp, vertical = 2.dp),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun chatTextFieldColors() =
|
||||
OutlinedTextFieldDefaults.colors(
|
||||
focusedContainerColor = mobileSurface,
|
||||
unfocusedContainerColor = mobileSurface,
|
||||
focusedBorderColor = mobileAccent,
|
||||
unfocusedBorderColor = mobileBorder,
|
||||
focusedTextColor = mobileText,
|
||||
unfocusedTextColor = mobileText,
|
||||
cursorColor = mobileAccent,
|
||||
)
|
||||
|
||||
@Composable
|
||||
private fun mobileBodyStyle() =
|
||||
MaterialTheme.typography.bodyMedium.copy(
|
||||
fontFamily = ai.openclaw.app.ui.mobileFontFamily,
|
||||
fontWeight = FontWeight.Medium,
|
||||
fontSize = 15.sp,
|
||||
lineHeight = 22.sp,
|
||||
)
|
||||
|
|
@ -0,0 +1,150 @@
|
|||
package ai.openclaw.app.ui.chat
|
||||
|
||||
import android.content.ContentResolver
|
||||
import android.graphics.Bitmap
|
||||
import android.graphics.BitmapFactory
|
||||
import android.net.Uri
|
||||
import android.util.Base64
|
||||
import android.util.LruCache
|
||||
import androidx.core.graphics.scale
|
||||
import ai.openclaw.app.node.JpegSizeLimiter
|
||||
import java.io.ByteArrayOutputStream
|
||||
import kotlin.math.max
|
||||
import kotlin.math.roundToInt
|
||||
|
||||
private const val CHAT_ATTACHMENT_MAX_WIDTH = 1600
|
||||
private const val CHAT_ATTACHMENT_MAX_BASE64_CHARS = 300 * 1024
|
||||
private const val CHAT_ATTACHMENT_START_QUALITY = 85
|
||||
private const val CHAT_DECODE_MAX_DIMENSION = 1600
|
||||
private const val CHAT_IMAGE_CACHE_BYTES = 16 * 1024 * 1024
|
||||
|
||||
private val decodedBitmapCache =
|
||||
object : LruCache<String, Bitmap>(CHAT_IMAGE_CACHE_BYTES) {
|
||||
override fun sizeOf(key: String, value: Bitmap): Int = value.byteCount.coerceAtLeast(1)
|
||||
}
|
||||
|
||||
internal fun loadSizedImageAttachment(resolver: ContentResolver, uri: Uri): PendingImageAttachment {
|
||||
val fileName = normalizeAttachmentFileName((uri.lastPathSegment ?: "image").substringAfterLast('/'))
|
||||
val bitmap = decodeScaledBitmap(resolver, uri, maxDimension = CHAT_ATTACHMENT_MAX_WIDTH)
|
||||
if (bitmap == null) {
|
||||
throw IllegalStateException("unsupported attachment")
|
||||
}
|
||||
val maxBytes = (CHAT_ATTACHMENT_MAX_BASE64_CHARS / 4) * 3
|
||||
val encoded =
|
||||
JpegSizeLimiter.compressToLimit(
|
||||
initialWidth = bitmap.width,
|
||||
initialHeight = bitmap.height,
|
||||
startQuality = CHAT_ATTACHMENT_START_QUALITY,
|
||||
maxBytes = maxBytes,
|
||||
minSize = 240,
|
||||
encode = { width, height, quality ->
|
||||
val working =
|
||||
if (width == bitmap.width && height == bitmap.height) {
|
||||
bitmap
|
||||
} else {
|
||||
bitmap.scale(width, height, true)
|
||||
}
|
||||
try {
|
||||
val out = ByteArrayOutputStream()
|
||||
if (!working.compress(Bitmap.CompressFormat.JPEG, quality, out)) {
|
||||
throw IllegalStateException("attachment encode failed")
|
||||
}
|
||||
out.toByteArray()
|
||||
} finally {
|
||||
if (working !== bitmap) {
|
||||
working.recycle()
|
||||
}
|
||||
}
|
||||
},
|
||||
)
|
||||
val base64 = Base64.encodeToString(encoded.bytes, Base64.NO_WRAP)
|
||||
return PendingImageAttachment(
|
||||
id = uri.toString() + "#" + System.currentTimeMillis().toString(),
|
||||
fileName = fileName,
|
||||
mimeType = "image/jpeg",
|
||||
base64 = base64,
|
||||
)
|
||||
}
|
||||
|
||||
internal fun decodeBase64Bitmap(base64: String, maxDimension: Int = CHAT_DECODE_MAX_DIMENSION): Bitmap? {
|
||||
val cacheKey = "$maxDimension:${base64.length}:${base64.hashCode()}"
|
||||
decodedBitmapCache.get(cacheKey)?.let { return it }
|
||||
|
||||
val bytes = Base64.decode(base64, Base64.DEFAULT)
|
||||
if (bytes.isEmpty()) return null
|
||||
|
||||
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
|
||||
BitmapFactory.decodeByteArray(bytes, 0, bytes.size, bounds)
|
||||
if (bounds.outWidth <= 0 || bounds.outHeight <= 0) return null
|
||||
|
||||
val bitmap =
|
||||
BitmapFactory.decodeByteArray(
|
||||
bytes,
|
||||
0,
|
||||
bytes.size,
|
||||
BitmapFactory.Options().apply {
|
||||
inSampleSize = computeInSampleSize(bounds.outWidth, bounds.outHeight, maxDimension)
|
||||
inPreferredConfig = Bitmap.Config.RGB_565
|
||||
},
|
||||
) ?: return null
|
||||
|
||||
decodedBitmapCache.put(cacheKey, bitmap)
|
||||
return bitmap
|
||||
}
|
||||
|
||||
internal fun computeInSampleSize(width: Int, height: Int, maxDimension: Int): Int {
|
||||
if (width <= 0 || height <= 0 || maxDimension <= 0) return 1
|
||||
|
||||
var sample = 1
|
||||
var longestEdge = max(width, height)
|
||||
while (longestEdge > maxDimension && sample < 64) {
|
||||
sample *= 2
|
||||
longestEdge = max(width / sample, height / sample)
|
||||
}
|
||||
return sample.coerceAtLeast(1)
|
||||
}
|
||||
|
||||
internal fun normalizeAttachmentFileName(raw: String): String {
|
||||
val trimmed = raw.trim()
|
||||
if (trimmed.isEmpty()) return "image.jpg"
|
||||
val stem = trimmed.substringBeforeLast('.', missingDelimiterValue = trimmed).ifEmpty { "image" }
|
||||
return "$stem.jpg"
|
||||
}
|
||||
|
||||
private fun decodeScaledBitmap(
|
||||
resolver: ContentResolver,
|
||||
uri: Uri,
|
||||
maxDimension: Int,
|
||||
): Bitmap? {
|
||||
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
|
||||
resolver.openInputStream(uri).use { input ->
|
||||
if (input == null) return null
|
||||
BitmapFactory.decodeStream(input, null, bounds)
|
||||
}
|
||||
if (bounds.outWidth <= 0 || bounds.outHeight <= 0) return null
|
||||
|
||||
val decoded =
|
||||
resolver.openInputStream(uri).use { input ->
|
||||
if (input == null) return null
|
||||
BitmapFactory.decodeStream(
|
||||
input,
|
||||
null,
|
||||
BitmapFactory.Options().apply {
|
||||
inSampleSize = computeInSampleSize(bounds.outWidth, bounds.outHeight, maxDimension)
|
||||
inPreferredConfig = Bitmap.Config.ARGB_8888
|
||||
},
|
||||
)
|
||||
} ?: return null
|
||||
|
||||
val longestEdge = max(decoded.width, decoded.height)
|
||||
if (longestEdge <= maxDimension) return decoded
|
||||
|
||||
val scale = maxDimension.toDouble() / longestEdge.toDouble()
|
||||
val targetWidth = max(1, (decoded.width * scale).roundToInt())
|
||||
val targetHeight = max(1, (decoded.height * scale).roundToInt())
|
||||
val scaled = decoded.scale(targetWidth, targetHeight, true)
|
||||
if (scaled !== decoded) {
|
||||
decoded.recycle()
|
||||
}
|
||||
return scaled
|
||||
}
|
||||
|
|
@ -0,0 +1,571 @@
|
|||
package ai.openclaw.app.ui.chat
|
||||
|
||||
import androidx.compose.foundation.Image
|
||||
import androidx.compose.foundation.background
|
||||
import androidx.compose.foundation.border
|
||||
import androidx.compose.foundation.horizontalScroll
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.IntrinsicSize
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.fillMaxHeight
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.height
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.width
|
||||
import androidx.compose.foundation.rememberScrollState
|
||||
import androidx.compose.foundation.text.selection.SelectionContainer
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.layout.ContentScale
|
||||
import androidx.compose.ui.text.AnnotatedString
|
||||
import androidx.compose.ui.text.SpanStyle
|
||||
import androidx.compose.ui.text.TextStyle
|
||||
import androidx.compose.ui.text.buildAnnotatedString
|
||||
import androidx.compose.ui.text.font.FontFamily
|
||||
import androidx.compose.ui.text.font.FontStyle
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.text.style.TextDecoration
|
||||
import androidx.compose.ui.text.withStyle
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.compose.ui.unit.sp
|
||||
import ai.openclaw.app.ui.mobileAccent
|
||||
import ai.openclaw.app.ui.mobileCallout
|
||||
import ai.openclaw.app.ui.mobileCaption1
|
||||
import ai.openclaw.app.ui.mobileCodeBg
|
||||
import ai.openclaw.app.ui.mobileCodeText
|
||||
import ai.openclaw.app.ui.mobileTextSecondary
|
||||
import org.commonmark.Extension
|
||||
import org.commonmark.ext.autolink.AutolinkExtension
|
||||
import org.commonmark.ext.gfm.strikethrough.Strikethrough
|
||||
import org.commonmark.ext.gfm.strikethrough.StrikethroughExtension
|
||||
import org.commonmark.ext.gfm.tables.TableBlock
|
||||
import org.commonmark.ext.gfm.tables.TableBody
|
||||
import org.commonmark.ext.gfm.tables.TableCell
|
||||
import org.commonmark.ext.gfm.tables.TableHead
|
||||
import org.commonmark.ext.gfm.tables.TableRow
|
||||
import org.commonmark.ext.gfm.tables.TablesExtension
|
||||
import org.commonmark.ext.task.list.items.TaskListItemMarker
|
||||
import org.commonmark.ext.task.list.items.TaskListItemsExtension
|
||||
import org.commonmark.node.BlockQuote
|
||||
import org.commonmark.node.BulletList
|
||||
import org.commonmark.node.Code
|
||||
import org.commonmark.node.Document
|
||||
import org.commonmark.node.Emphasis
|
||||
import org.commonmark.node.FencedCodeBlock
|
||||
import org.commonmark.node.Heading
|
||||
import org.commonmark.node.HardLineBreak
|
||||
import org.commonmark.node.HtmlBlock
|
||||
import org.commonmark.node.HtmlInline
|
||||
import org.commonmark.node.Image as MarkdownImage
|
||||
import org.commonmark.node.IndentedCodeBlock
|
||||
import org.commonmark.node.Link
|
||||
import org.commonmark.node.ListItem
|
||||
import org.commonmark.node.Node
|
||||
import org.commonmark.node.OrderedList
|
||||
import org.commonmark.node.Paragraph
|
||||
import org.commonmark.node.SoftLineBreak
|
||||
import org.commonmark.node.StrongEmphasis
|
||||
import org.commonmark.node.Text as MarkdownTextNode
|
||||
import org.commonmark.node.ThematicBreak
|
||||
import org.commonmark.parser.Parser
|
||||
|
||||
private const val LIST_INDENT_DP = 14
|
||||
private val dataImageRegex = Regex("^data:image/([a-zA-Z0-9+.-]+);base64,([A-Za-z0-9+/=\\n\\r]+)$")
|
||||
|
||||
private val markdownParser: Parser by lazy {
|
||||
val extensions: List<Extension> =
|
||||
listOf(
|
||||
AutolinkExtension.create(),
|
||||
StrikethroughExtension.create(),
|
||||
TablesExtension.create(),
|
||||
TaskListItemsExtension.create(),
|
||||
)
|
||||
Parser.builder()
|
||||
.extensions(extensions)
|
||||
.build()
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun ChatMarkdown(text: String, textColor: Color) {
|
||||
val document = remember(text) { markdownParser.parse(text) as Document }
|
||||
val inlineStyles = InlineStyles(inlineCodeBg = mobileCodeBg, inlineCodeColor = mobileCodeText, linkColor = mobileAccent, baseCallout = mobileCallout)
|
||||
|
||||
Column(verticalArrangement = Arrangement.spacedBy(10.dp)) {
|
||||
RenderMarkdownBlocks(
|
||||
start = document.firstChild,
|
||||
textColor = textColor,
|
||||
inlineStyles = inlineStyles,
|
||||
listDepth = 0,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun RenderMarkdownBlocks(
|
||||
start: Node?,
|
||||
textColor: Color,
|
||||
inlineStyles: InlineStyles,
|
||||
listDepth: Int,
|
||||
) {
|
||||
var node = start
|
||||
while (node != null) {
|
||||
val current = node
|
||||
when (current) {
|
||||
is Paragraph -> {
|
||||
RenderParagraph(current, textColor = textColor, inlineStyles = inlineStyles)
|
||||
}
|
||||
is Heading -> {
|
||||
val headingText = remember(current) { buildInlineMarkdown(current.firstChild, inlineStyles) }
|
||||
Text(
|
||||
text = headingText,
|
||||
style = headingStyle(current.level, inlineStyles.baseCallout),
|
||||
color = textColor,
|
||||
)
|
||||
}
|
||||
is FencedCodeBlock -> {
|
||||
SelectionContainer(modifier = Modifier.fillMaxWidth()) {
|
||||
ChatCodeBlock(code = current.literal.orEmpty(), language = current.info?.trim()?.ifEmpty { null })
|
||||
}
|
||||
}
|
||||
is IndentedCodeBlock -> {
|
||||
SelectionContainer(modifier = Modifier.fillMaxWidth()) {
|
||||
ChatCodeBlock(code = current.literal.orEmpty(), language = null)
|
||||
}
|
||||
}
|
||||
is BlockQuote -> {
|
||||
Row(
|
||||
modifier = Modifier
|
||||
.fillMaxWidth()
|
||||
.height(IntrinsicSize.Min)
|
||||
.padding(vertical = 2.dp),
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
verticalAlignment = Alignment.Top,
|
||||
) {
|
||||
Box(
|
||||
modifier = Modifier
|
||||
.width(2.dp)
|
||||
.fillMaxHeight()
|
||||
.background(mobileTextSecondary.copy(alpha = 0.35f)),
|
||||
)
|
||||
Column(
|
||||
modifier = Modifier.weight(1f),
|
||||
verticalArrangement = Arrangement.spacedBy(8.dp),
|
||||
) {
|
||||
RenderMarkdownBlocks(
|
||||
start = current.firstChild,
|
||||
textColor = textColor,
|
||||
inlineStyles = inlineStyles,
|
||||
listDepth = listDepth,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
is BulletList -> {
|
||||
RenderBulletList(
|
||||
list = current,
|
||||
textColor = textColor,
|
||||
inlineStyles = inlineStyles,
|
||||
listDepth = listDepth,
|
||||
)
|
||||
}
|
||||
is OrderedList -> {
|
||||
RenderOrderedList(
|
||||
list = current,
|
||||
textColor = textColor,
|
||||
inlineStyles = inlineStyles,
|
||||
listDepth = listDepth,
|
||||
)
|
||||
}
|
||||
is TableBlock -> {
|
||||
RenderTableBlock(
|
||||
table = current,
|
||||
textColor = textColor,
|
||||
inlineStyles = inlineStyles,
|
||||
)
|
||||
}
|
||||
is ThematicBreak -> {
|
||||
Box(
|
||||
modifier = Modifier
|
||||
.fillMaxWidth()
|
||||
.height(1.dp)
|
||||
.background(mobileTextSecondary.copy(alpha = 0.25f)),
|
||||
)
|
||||
}
|
||||
is HtmlBlock -> {
|
||||
val literal = current.literal.orEmpty().trim()
|
||||
if (literal.isNotEmpty()) {
|
||||
Text(
|
||||
text = literal,
|
||||
style = mobileCallout.copy(fontFamily = FontFamily.Monospace),
|
||||
color = textColor,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
node = current.next
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun RenderParagraph(
|
||||
paragraph: Paragraph,
|
||||
textColor: Color,
|
||||
inlineStyles: InlineStyles,
|
||||
) {
|
||||
val standaloneImage = remember(paragraph) { standaloneDataImage(paragraph) }
|
||||
if (standaloneImage != null) {
|
||||
InlineBase64Image(base64 = standaloneImage.base64, mimeType = standaloneImage.mimeType)
|
||||
return
|
||||
}
|
||||
|
||||
val annotated = remember(paragraph) { buildInlineMarkdown(paragraph.firstChild, inlineStyles) }
|
||||
if (annotated.text.trimEnd().isEmpty()) {
|
||||
return
|
||||
}
|
||||
|
||||
Text(
|
||||
text = annotated,
|
||||
style = inlineStyles.baseCallout,
|
||||
color = textColor,
|
||||
)
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun RenderBulletList(
|
||||
list: BulletList,
|
||||
textColor: Color,
|
||||
inlineStyles: InlineStyles,
|
||||
listDepth: Int,
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.padding(start = (LIST_INDENT_DP * listDepth).dp),
|
||||
verticalArrangement = Arrangement.spacedBy(6.dp),
|
||||
) {
|
||||
var item = list.firstChild
|
||||
while (item != null) {
|
||||
if (item is ListItem) {
|
||||
RenderListItem(
|
||||
item = item,
|
||||
markerText = "•",
|
||||
textColor = textColor,
|
||||
inlineStyles = inlineStyles,
|
||||
listDepth = listDepth,
|
||||
)
|
||||
}
|
||||
item = item.next
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun RenderOrderedList(
|
||||
list: OrderedList,
|
||||
textColor: Color,
|
||||
inlineStyles: InlineStyles,
|
||||
listDepth: Int,
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.padding(start = (LIST_INDENT_DP * listDepth).dp),
|
||||
verticalArrangement = Arrangement.spacedBy(6.dp),
|
||||
) {
|
||||
var index = list.markerStartNumber ?: 1
|
||||
var item = list.firstChild
|
||||
while (item != null) {
|
||||
if (item is ListItem) {
|
||||
RenderListItem(
|
||||
item = item,
|
||||
markerText = "$index.",
|
||||
textColor = textColor,
|
||||
inlineStyles = inlineStyles,
|
||||
listDepth = listDepth,
|
||||
)
|
||||
index += 1
|
||||
}
|
||||
item = item.next
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun RenderListItem(
|
||||
item: ListItem,
|
||||
markerText: String,
|
||||
textColor: Color,
|
||||
inlineStyles: InlineStyles,
|
||||
listDepth: Int,
|
||||
) {
|
||||
var contentStart = item.firstChild
|
||||
var marker = markerText
|
||||
val task = contentStart as? TaskListItemMarker
|
||||
if (task != null) {
|
||||
marker = if (task.isChecked) "☑" else "☐"
|
||||
contentStart = task.next
|
||||
}
|
||||
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
verticalAlignment = Alignment.Top,
|
||||
) {
|
||||
Text(
|
||||
text = marker,
|
||||
style = inlineStyles.baseCallout.copy(fontWeight = FontWeight.SemiBold),
|
||||
color = textColor,
|
||||
modifier = Modifier.width(24.dp),
|
||||
)
|
||||
|
||||
Column(
|
||||
modifier = Modifier.weight(1f),
|
||||
verticalArrangement = Arrangement.spacedBy(4.dp),
|
||||
) {
|
||||
RenderMarkdownBlocks(
|
||||
start = contentStart,
|
||||
textColor = textColor,
|
||||
inlineStyles = inlineStyles,
|
||||
listDepth = listDepth + 1,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun RenderTableBlock(
|
||||
table: TableBlock,
|
||||
textColor: Color,
|
||||
inlineStyles: InlineStyles,
|
||||
) {
|
||||
val rows = remember(table) { buildTableRows(table, inlineStyles) }
|
||||
if (rows.isEmpty()) return
|
||||
|
||||
val maxCols = rows.maxOf { row -> row.cells.size }.coerceAtLeast(1)
|
||||
val scrollState = rememberScrollState()
|
||||
|
||||
Column(
|
||||
modifier = Modifier
|
||||
.fillMaxWidth()
|
||||
.horizontalScroll(scrollState)
|
||||
.border(1.dp, mobileTextSecondary.copy(alpha = 0.25f)),
|
||||
) {
|
||||
for (row in rows) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
) {
|
||||
for (index in 0 until maxCols) {
|
||||
val cell = row.cells.getOrNull(index) ?: AnnotatedString("")
|
||||
Text(
|
||||
text = cell,
|
||||
style = if (row.isHeader) mobileCaption1.copy(fontWeight = FontWeight.SemiBold) else inlineStyles.baseCallout,
|
||||
color = textColor,
|
||||
modifier = Modifier
|
||||
.border(1.dp, mobileTextSecondary.copy(alpha = 0.22f))
|
||||
.padding(horizontal = 8.dp, vertical = 6.dp)
|
||||
.width(160.dp),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun buildTableRows(table: TableBlock, inlineStyles: InlineStyles): List<TableRenderRow> {
|
||||
val rows = mutableListOf<TableRenderRow>()
|
||||
var child = table.firstChild
|
||||
while (child != null) {
|
||||
when (child) {
|
||||
is TableHead -> rows.addAll(readTableSection(child, isHeader = true, inlineStyles = inlineStyles))
|
||||
is TableBody -> rows.addAll(readTableSection(child, isHeader = false, inlineStyles = inlineStyles))
|
||||
is TableRow -> rows.add(readTableRow(child, isHeader = false, inlineStyles = inlineStyles))
|
||||
}
|
||||
child = child.next
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
private fun readTableSection(section: Node, isHeader: Boolean, inlineStyles: InlineStyles): List<TableRenderRow> {
|
||||
val rows = mutableListOf<TableRenderRow>()
|
||||
var row = section.firstChild
|
||||
while (row != null) {
|
||||
if (row is TableRow) {
|
||||
rows.add(readTableRow(row, isHeader = isHeader, inlineStyles = inlineStyles))
|
||||
}
|
||||
row = row.next
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
private fun readTableRow(row: TableRow, isHeader: Boolean, inlineStyles: InlineStyles): TableRenderRow {
|
||||
val cells = mutableListOf<AnnotatedString>()
|
||||
var cellNode = row.firstChild
|
||||
while (cellNode != null) {
|
||||
if (cellNode is TableCell) {
|
||||
cells.add(buildInlineMarkdown(cellNode.firstChild, inlineStyles))
|
||||
}
|
||||
cellNode = cellNode.next
|
||||
}
|
||||
return TableRenderRow(isHeader = isHeader, cells = cells)
|
||||
}
|
||||
|
||||
private fun buildInlineMarkdown(start: Node?, inlineStyles: InlineStyles): AnnotatedString {
|
||||
return buildAnnotatedString {
|
||||
appendInlineNode(
|
||||
node = start,
|
||||
inlineCodeBg = inlineStyles.inlineCodeBg,
|
||||
inlineCodeColor = inlineStyles.inlineCodeColor,
|
||||
linkColor = inlineStyles.linkColor,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun AnnotatedString.Builder.appendInlineNode(
|
||||
node: Node?,
|
||||
inlineCodeBg: Color,
|
||||
inlineCodeColor: Color,
|
||||
linkColor: Color,
|
||||
) {
|
||||
var current = node
|
||||
while (current != null) {
|
||||
when (current) {
|
||||
is MarkdownTextNode -> append(current.literal)
|
||||
is SoftLineBreak -> append('\n')
|
||||
is HardLineBreak -> append('\n')
|
||||
is Code -> {
|
||||
withStyle(
|
||||
SpanStyle(
|
||||
fontFamily = FontFamily.Monospace,
|
||||
background = inlineCodeBg,
|
||||
color = inlineCodeColor,
|
||||
),
|
||||
) {
|
||||
append(current.literal)
|
||||
}
|
||||
}
|
||||
is Emphasis -> {
|
||||
withStyle(SpanStyle(fontStyle = FontStyle.Italic)) {
|
||||
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
|
||||
}
|
||||
}
|
||||
is StrongEmphasis -> {
|
||||
withStyle(SpanStyle(fontWeight = FontWeight.SemiBold)) {
|
||||
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
|
||||
}
|
||||
}
|
||||
is Strikethrough -> {
|
||||
withStyle(SpanStyle(textDecoration = TextDecoration.LineThrough)) {
|
||||
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
|
||||
}
|
||||
}
|
||||
is Link -> {
|
||||
withStyle(
|
||||
SpanStyle(
|
||||
color = linkColor,
|
||||
textDecoration = TextDecoration.Underline,
|
||||
),
|
||||
) {
|
||||
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
|
||||
}
|
||||
}
|
||||
is MarkdownImage -> {
|
||||
val alt = buildPlainText(current.firstChild)
|
||||
if (alt.isNotBlank()) {
|
||||
append(alt)
|
||||
} else {
|
||||
append("image")
|
||||
}
|
||||
}
|
||||
is HtmlInline -> {
|
||||
if (!current.literal.isNullOrBlank()) {
|
||||
append(current.literal)
|
||||
}
|
||||
}
|
||||
else -> {
|
||||
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
|
||||
}
|
||||
}
|
||||
current = current.next
|
||||
}
|
||||
}
|
||||
|
||||
private fun buildPlainText(start: Node?): String {
|
||||
val sb = StringBuilder()
|
||||
var node = start
|
||||
while (node != null) {
|
||||
when (node) {
|
||||
is MarkdownTextNode -> sb.append(node.literal)
|
||||
is SoftLineBreak, is HardLineBreak -> sb.append('\n')
|
||||
else -> sb.append(buildPlainText(node.firstChild))
|
||||
}
|
||||
node = node.next
|
||||
}
|
||||
return sb.toString()
|
||||
}
|
||||
|
||||
private fun standaloneDataImage(paragraph: Paragraph): ParsedDataImage? {
|
||||
val only = paragraph.firstChild as? MarkdownImage ?: return null
|
||||
if (only.next != null) return null
|
||||
return parseDataImageDestination(only.destination)
|
||||
}
|
||||
|
||||
private fun parseDataImageDestination(destination: String?): ParsedDataImage? {
|
||||
val raw = destination?.trim().orEmpty()
|
||||
if (raw.isEmpty()) return null
|
||||
val match = dataImageRegex.matchEntire(raw) ?: return null
|
||||
val subtype = match.groupValues.getOrNull(1)?.trim()?.ifEmpty { "png" } ?: "png"
|
||||
val base64 = match.groupValues.getOrNull(2)?.replace("\n", "")?.replace("\r", "")?.trim().orEmpty()
|
||||
if (base64.isEmpty()) return null
|
||||
return ParsedDataImage(mimeType = "image/$subtype", base64 = base64)
|
||||
}
|
||||
|
||||
private fun headingStyle(level: Int, baseCallout: TextStyle): TextStyle {
|
||||
return when (level.coerceIn(1, 6)) {
|
||||
1 -> baseCallout.copy(fontSize = 22.sp, lineHeight = 28.sp, fontWeight = FontWeight.Bold)
|
||||
2 -> baseCallout.copy(fontSize = 20.sp, lineHeight = 26.sp, fontWeight = FontWeight.Bold)
|
||||
3 -> baseCallout.copy(fontSize = 18.sp, lineHeight = 24.sp, fontWeight = FontWeight.SemiBold)
|
||||
4 -> baseCallout.copy(fontSize = 16.sp, lineHeight = 22.sp, fontWeight = FontWeight.SemiBold)
|
||||
else -> baseCallout.copy(fontWeight = FontWeight.SemiBold)
|
||||
}
|
||||
}
|
||||
|
||||
private data class InlineStyles(
|
||||
val inlineCodeBg: Color,
|
||||
val inlineCodeColor: Color,
|
||||
val linkColor: Color,
|
||||
val baseCallout: TextStyle,
|
||||
)
|
||||
|
||||
private data class TableRenderRow(
|
||||
val isHeader: Boolean,
|
||||
val cells: List<AnnotatedString>,
|
||||
)
|
||||
|
||||
private data class ParsedDataImage(
|
||||
val mimeType: String,
|
||||
val base64: String,
|
||||
)
|
||||
|
||||
@Composable
|
||||
private fun InlineBase64Image(base64: String, mimeType: String?) {
|
||||
val imageState = rememberBase64ImageState(base64)
|
||||
val image = imageState.image
|
||||
|
||||
if (image != null) {
|
||||
Image(
|
||||
bitmap = image!!,
|
||||
contentDescription = mimeType ?: "image",
|
||||
contentScale = ContentScale.Fit,
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
)
|
||||
} else if (imageState.failed) {
|
||||
Text(
|
||||
text = "Image unavailable",
|
||||
modifier = Modifier.padding(vertical = 2.dp),
|
||||
style = mobileCaption1,
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,117 @@
|
|||
package ai.openclaw.app.ui.chat
|
||||
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.lazy.LazyColumn
|
||||
import androidx.compose.foundation.lazy.items
|
||||
import androidx.compose.foundation.lazy.rememberLazyListState
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.unit.dp
|
||||
import ai.openclaw.app.chat.ChatMessage
|
||||
import ai.openclaw.app.chat.ChatPendingToolCall
|
||||
import ai.openclaw.app.ui.mobileBorder
|
||||
import ai.openclaw.app.ui.mobileCallout
|
||||
import ai.openclaw.app.ui.mobileCardSurface
|
||||
import ai.openclaw.app.ui.mobileHeadline
|
||||
import ai.openclaw.app.ui.mobileText
|
||||
import ai.openclaw.app.ui.mobileTextSecondary
|
||||
|
||||
@Composable
|
||||
fun ChatMessageListCard(
|
||||
messages: List<ChatMessage>,
|
||||
pendingRunCount: Int,
|
||||
pendingToolCalls: List<ChatPendingToolCall>,
|
||||
streamingAssistantText: String?,
|
||||
healthOk: Boolean,
|
||||
modifier: Modifier = Modifier,
|
||||
) {
|
||||
val listState = rememberLazyListState()
|
||||
val displayMessages = remember(messages) { messages.asReversed() }
|
||||
val stream = streamingAssistantText?.trim()
|
||||
|
||||
// New list items/tool rows should animate into view, but token streaming should not restart
|
||||
// that animation on every delta.
|
||||
LaunchedEffect(messages.size, pendingRunCount, pendingToolCalls.size) {
|
||||
listState.animateScrollToItem(index = 0)
|
||||
}
|
||||
LaunchedEffect(stream) {
|
||||
if (!stream.isNullOrEmpty()) {
|
||||
listState.scrollToItem(index = 0)
|
||||
}
|
||||
}
|
||||
|
||||
Box(modifier = modifier.fillMaxWidth()) {
|
||||
LazyColumn(
|
||||
modifier = Modifier.fillMaxSize(),
|
||||
state = listState,
|
||||
reverseLayout = true,
|
||||
verticalArrangement = Arrangement.spacedBy(10.dp),
|
||||
contentPadding = androidx.compose.foundation.layout.PaddingValues(bottom = 8.dp),
|
||||
) {
|
||||
// With reverseLayout = true, index 0 renders at the BOTTOM.
|
||||
// So we emit newest items first: streaming → tools → typing → messages (newest→oldest).
|
||||
if (!stream.isNullOrEmpty()) {
|
||||
item(key = "stream") {
|
||||
ChatStreamingAssistantBubble(text = stream)
|
||||
}
|
||||
}
|
||||
|
||||
if (pendingToolCalls.isNotEmpty()) {
|
||||
item(key = "tools") {
|
||||
ChatPendingToolsBubble(toolCalls = pendingToolCalls)
|
||||
}
|
||||
}
|
||||
|
||||
if (pendingRunCount > 0) {
|
||||
item(key = "typing") {
|
||||
ChatTypingIndicatorBubble()
|
||||
}
|
||||
}
|
||||
|
||||
items(items = displayMessages, key = { it.id }) { message ->
|
||||
ChatMessageBubble(message = message)
|
||||
}
|
||||
}
|
||||
|
||||
if (messages.isEmpty() && pendingRunCount == 0 && pendingToolCalls.isEmpty() && streamingAssistantText.isNullOrBlank()) {
|
||||
EmptyChatHint(modifier = Modifier.align(Alignment.Center), healthOk = healthOk)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun EmptyChatHint(modifier: Modifier = Modifier, healthOk: Boolean) {
|
||||
Surface(
|
||||
modifier = modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
color = mobileCardSurface.copy(alpha = 0.9f),
|
||||
border = androidx.compose.foundation.BorderStroke(1.dp, mobileBorder),
|
||||
) {
|
||||
androidx.compose.foundation.layout.Column(
|
||||
modifier = Modifier.padding(horizontal = 12.dp, vertical = 12.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(4.dp),
|
||||
) {
|
||||
Text("No messages yet", style = mobileHeadline, color = mobileText)
|
||||
Text(
|
||||
text =
|
||||
if (healthOk) {
|
||||
"Send the first prompt to start this session."
|
||||
} else {
|
||||
"Connect gateway first, then return to chat."
|
||||
},
|
||||
style = mobileCallout,
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,302 @@
|
|||
package ai.openclaw.app.ui.chat
|
||||
|
||||
import androidx.compose.foundation.BorderStroke
|
||||
import androidx.compose.foundation.Image
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.shape.CircleShape
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.draw.alpha
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.layout.ContentScale
|
||||
import androidx.compose.ui.platform.LocalContext
|
||||
import androidx.compose.ui.text.font.FontFamily
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.compose.ui.unit.sp
|
||||
import ai.openclaw.app.chat.ChatMessage
|
||||
import ai.openclaw.app.chat.ChatMessageContent
|
||||
import ai.openclaw.app.chat.ChatPendingToolCall
|
||||
import ai.openclaw.app.tools.ToolDisplayRegistry
|
||||
import ai.openclaw.app.ui.mobileAccent
|
||||
import ai.openclaw.app.ui.mobileAccentSoft
|
||||
import ai.openclaw.app.ui.mobileBorder
|
||||
import ai.openclaw.app.ui.mobileBorderStrong
|
||||
import ai.openclaw.app.ui.mobileCallout
|
||||
import ai.openclaw.app.ui.mobileCaption1
|
||||
import ai.openclaw.app.ui.mobileCaption2
|
||||
import ai.openclaw.app.ui.mobileCardSurface
|
||||
import ai.openclaw.app.ui.mobileCodeBg
|
||||
import ai.openclaw.app.ui.mobileCodeBorder
|
||||
import ai.openclaw.app.ui.mobileCodeText
|
||||
import ai.openclaw.app.ui.mobileHeadline
|
||||
import ai.openclaw.app.ui.mobileText
|
||||
import ai.openclaw.app.ui.mobileTextSecondary
|
||||
import ai.openclaw.app.ui.mobileWarning
|
||||
import ai.openclaw.app.ui.mobileWarningSoft
|
||||
import java.util.Locale
|
||||
|
||||
private data class ChatBubbleStyle(
|
||||
val alignEnd: Boolean,
|
||||
val containerColor: Color,
|
||||
val borderColor: Color,
|
||||
val roleColor: Color,
|
||||
)
|
||||
|
||||
@Composable
|
||||
fun ChatMessageBubble(message: ChatMessage) {
|
||||
val role = message.role.trim().lowercase(Locale.US)
|
||||
val style = bubbleStyle(role)
|
||||
|
||||
// Filter to only displayable content parts (text with content, or base64 images).
|
||||
val displayableContent =
|
||||
message.content.filter { part ->
|
||||
when (part.type) {
|
||||
"text" -> !part.text.isNullOrBlank()
|
||||
else -> part.base64 != null
|
||||
}
|
||||
}
|
||||
|
||||
if (displayableContent.isEmpty()) return
|
||||
|
||||
ChatBubbleContainer(style = style, roleLabel = roleLabel(role)) {
|
||||
ChatMessageBody(content = displayableContent, textColor = mobileText)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ChatBubbleContainer(
|
||||
style: ChatBubbleStyle,
|
||||
roleLabel: String,
|
||||
modifier: Modifier = Modifier,
|
||||
content: @Composable () -> Unit,
|
||||
) {
|
||||
Row(
|
||||
modifier = modifier.fillMaxWidth(),
|
||||
horizontalArrangement = if (style.alignEnd) Arrangement.End else Arrangement.Start,
|
||||
) {
|
||||
Surface(
|
||||
shape = RoundedCornerShape(12.dp),
|
||||
border = BorderStroke(1.dp, style.borderColor),
|
||||
color = style.containerColor,
|
||||
tonalElevation = 0.dp,
|
||||
shadowElevation = 0.dp,
|
||||
modifier = Modifier.fillMaxWidth(0.90f),
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.padding(horizontal = 11.dp, vertical = 8.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(3.dp),
|
||||
) {
|
||||
Text(
|
||||
text = roleLabel,
|
||||
style = mobileCaption2.copy(fontWeight = FontWeight.SemiBold, letterSpacing = 0.6.sp),
|
||||
color = style.roleColor,
|
||||
)
|
||||
content()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ChatMessageBody(content: List<ChatMessageContent>, textColor: Color) {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
for (part in content) {
|
||||
when (part.type) {
|
||||
"text" -> {
|
||||
val text = part.text ?: continue
|
||||
ChatMarkdown(text = text, textColor = textColor)
|
||||
}
|
||||
else -> {
|
||||
val b64 = part.base64 ?: continue
|
||||
ChatBase64Image(base64 = b64, mimeType = part.mimeType)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun ChatTypingIndicatorBubble() {
|
||||
ChatBubbleContainer(
|
||||
style = bubbleStyle("assistant"),
|
||||
roleLabel = roleLabel("assistant"),
|
||||
) {
|
||||
Row(
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
) {
|
||||
DotPulse(color = mobileTextSecondary)
|
||||
Text("Thinking...", style = mobileCallout, color = mobileTextSecondary)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun ChatPendingToolsBubble(toolCalls: List<ChatPendingToolCall>) {
|
||||
val context = LocalContext.current
|
||||
val displays =
|
||||
remember(toolCalls, context) {
|
||||
toolCalls.map { ToolDisplayRegistry.resolve(context, it.name, it.args) }
|
||||
}
|
||||
|
||||
ChatBubbleContainer(
|
||||
style = bubbleStyle("assistant"),
|
||||
roleLabel = "Tools",
|
||||
) {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(4.dp)) {
|
||||
Text("Running tools...", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
|
||||
for (display in displays.take(6)) {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
|
||||
Text(
|
||||
"${display.emoji} ${display.label}",
|
||||
style = mobileCallout,
|
||||
color = mobileTextSecondary,
|
||||
fontFamily = FontFamily.Monospace,
|
||||
)
|
||||
display.detailLine?.let { detail ->
|
||||
Text(
|
||||
detail,
|
||||
style = mobileCaption1,
|
||||
color = mobileTextSecondary,
|
||||
fontFamily = FontFamily.Monospace,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
if (toolCalls.size > 6) {
|
||||
Text(
|
||||
text = "... +${toolCalls.size - 6} more",
|
||||
style = mobileCaption1,
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun ChatStreamingAssistantBubble(text: String) {
|
||||
ChatBubbleContainer(
|
||||
style = bubbleStyle("assistant").copy(borderColor = mobileAccent),
|
||||
roleLabel = "OpenClaw · Live",
|
||||
) {
|
||||
ChatMarkdown(text = text, textColor = mobileText)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun bubbleStyle(role: String): ChatBubbleStyle {
|
||||
return when (role) {
|
||||
"user" ->
|
||||
ChatBubbleStyle(
|
||||
alignEnd = true,
|
||||
containerColor = mobileAccentSoft,
|
||||
borderColor = mobileAccent,
|
||||
roleColor = mobileAccent,
|
||||
)
|
||||
|
||||
"system" ->
|
||||
ChatBubbleStyle(
|
||||
alignEnd = false,
|
||||
containerColor = mobileWarningSoft,
|
||||
borderColor = mobileWarning.copy(alpha = 0.45f),
|
||||
roleColor = mobileWarning,
|
||||
)
|
||||
|
||||
else ->
|
||||
ChatBubbleStyle(
|
||||
alignEnd = false,
|
||||
containerColor = mobileCardSurface,
|
||||
borderColor = mobileBorderStrong,
|
||||
roleColor = mobileTextSecondary,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun roleLabel(role: String): String {
|
||||
return when (role) {
|
||||
"user" -> "You"
|
||||
"system" -> "System"
|
||||
else -> "OpenClaw"
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ChatBase64Image(base64: String, mimeType: String?) {
|
||||
val imageState = rememberBase64ImageState(base64)
|
||||
val image = imageState.image
|
||||
|
||||
if (image != null) {
|
||||
Surface(
|
||||
shape = RoundedCornerShape(10.dp),
|
||||
border = BorderStroke(1.dp, mobileBorder),
|
||||
color = mobileCardSurface,
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
) {
|
||||
Image(
|
||||
bitmap = image!!,
|
||||
contentDescription = mimeType ?: "attachment",
|
||||
contentScale = ContentScale.Fit,
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
)
|
||||
}
|
||||
} else if (imageState.failed) {
|
||||
Text("Unsupported attachment", style = mobileCaption1, color = mobileTextSecondary)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun DotPulse(color: Color) {
|
||||
Row(horizontalArrangement = Arrangement.spacedBy(5.dp), verticalAlignment = Alignment.CenterVertically) {
|
||||
PulseDot(alpha = 0.38f, color = color)
|
||||
PulseDot(alpha = 0.62f, color = color)
|
||||
PulseDot(alpha = 0.90f, color = color)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun PulseDot(alpha: Float, color: Color) {
|
||||
Surface(
|
||||
modifier = Modifier.size(6.dp).alpha(alpha),
|
||||
shape = CircleShape,
|
||||
color = color,
|
||||
) {}
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun ChatCodeBlock(code: String, language: String?) {
|
||||
Surface(
|
||||
shape = RoundedCornerShape(8.dp),
|
||||
color = mobileCodeBg,
|
||||
border = BorderStroke(1.dp, mobileCodeBorder),
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
) {
|
||||
Column(modifier = Modifier.padding(horizontal = 10.dp, vertical = 8.dp), verticalArrangement = Arrangement.spacedBy(4.dp)) {
|
||||
if (!language.isNullOrBlank()) {
|
||||
Text(
|
||||
text = language.uppercase(Locale.US),
|
||||
style = mobileCaption2.copy(letterSpacing = 0.4.sp),
|
||||
color = mobileTextSecondary,
|
||||
)
|
||||
}
|
||||
Text(
|
||||
text = code.trimEnd(),
|
||||
fontFamily = FontFamily.Monospace,
|
||||
style = mobileCallout,
|
||||
color = mobileCodeText,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,260 @@
|
|||
package ai.openclaw.app.ui.chat
|
||||
|
||||
import androidx.activity.compose.rememberLauncherForActivityResult
|
||||
import androidx.activity.result.contract.ActivityResultContracts
|
||||
import androidx.compose.foundation.BorderStroke
|
||||
import androidx.compose.foundation.horizontalScroll
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.imePadding
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.rememberScrollState
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material3.Surface
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.collectAsState
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateListOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.rememberCoroutineScope
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.platform.LocalContext
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.text.style.TextOverflow
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.compose.ui.unit.sp
|
||||
import ai.openclaw.app.MainViewModel
|
||||
import ai.openclaw.app.chat.ChatSessionEntry
|
||||
import ai.openclaw.app.chat.OutgoingAttachment
|
||||
import ai.openclaw.app.ui.mobileAccent
|
||||
import ai.openclaw.app.ui.mobileAccentBorderStrong
|
||||
import ai.openclaw.app.ui.mobileBorder
|
||||
import ai.openclaw.app.ui.mobileBorderStrong
|
||||
import ai.openclaw.app.ui.mobileCallout
|
||||
import ai.openclaw.app.ui.mobileCardSurface
|
||||
import ai.openclaw.app.ui.mobileCaption1
|
||||
import ai.openclaw.app.ui.mobileCaption2
|
||||
import ai.openclaw.app.ui.mobileDanger
|
||||
import ai.openclaw.app.ui.mobileDangerSoft
|
||||
import ai.openclaw.app.ui.mobileText
|
||||
import ai.openclaw.app.ui.mobileTextSecondary
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.withContext
|
||||
|
||||
internal fun resolvePendingAssistantAutoSend(
|
||||
pendingPrompt: String?,
|
||||
healthOk: Boolean,
|
||||
pendingRunCount: Int,
|
||||
): String? {
|
||||
val prompt = pendingPrompt?.trim()?.ifEmpty { null } ?: return null
|
||||
if (!healthOk || pendingRunCount > 0) return null
|
||||
return prompt
|
||||
}
|
||||
|
||||
internal suspend fun dispatchPendingAssistantAutoSend(
|
||||
pendingPrompt: String?,
|
||||
healthOk: Boolean,
|
||||
pendingRunCount: Int,
|
||||
dispatch: suspend (String) -> Boolean,
|
||||
): Boolean {
|
||||
val prompt =
|
||||
resolvePendingAssistantAutoSend(
|
||||
pendingPrompt = pendingPrompt,
|
||||
healthOk = healthOk,
|
||||
pendingRunCount = pendingRunCount,
|
||||
) ?: return false
|
||||
return dispatch(prompt)
|
||||
}
|
||||
|
||||
@Composable
|
||||
fun ChatSheetContent(viewModel: MainViewModel) {
|
||||
val messages by viewModel.chatMessages.collectAsState()
|
||||
val errorText by viewModel.chatError.collectAsState()
|
||||
val pendingRunCount by viewModel.pendingRunCount.collectAsState()
|
||||
val healthOk by viewModel.chatHealthOk.collectAsState()
|
||||
val sessionKey by viewModel.chatSessionKey.collectAsState()
|
||||
val mainSessionKey by viewModel.mainSessionKey.collectAsState()
|
||||
val thinkingLevel by viewModel.chatThinkingLevel.collectAsState()
|
||||
val streamingAssistantText by viewModel.chatStreamingAssistantText.collectAsState()
|
||||
val pendingToolCalls by viewModel.chatPendingToolCalls.collectAsState()
|
||||
val sessions by viewModel.chatSessions.collectAsState()
|
||||
val chatDraft by viewModel.chatDraft.collectAsState()
|
||||
val pendingAssistantAutoSend by viewModel.pendingAssistantAutoSend.collectAsState()
|
||||
|
||||
LaunchedEffect(Unit) {
|
||||
viewModel.loadChat(mainSessionKey)
|
||||
}
|
||||
|
||||
LaunchedEffect(pendingAssistantAutoSend, healthOk, pendingRunCount, thinkingLevel) {
|
||||
val accepted =
|
||||
dispatchPendingAssistantAutoSend(
|
||||
pendingPrompt = pendingAssistantAutoSend,
|
||||
healthOk = healthOk,
|
||||
pendingRunCount = pendingRunCount,
|
||||
) { prompt ->
|
||||
viewModel.sendChatAwaitAcceptance(
|
||||
message = prompt,
|
||||
thinking = thinkingLevel,
|
||||
attachments = emptyList(),
|
||||
)
|
||||
}
|
||||
if (!accepted) return@LaunchedEffect
|
||||
viewModel.clearPendingAssistantAutoSend()
|
||||
}
|
||||
|
||||
val context = LocalContext.current
|
||||
val resolver = context.contentResolver
|
||||
val scope = rememberCoroutineScope()
|
||||
|
||||
val attachments = remember { mutableStateListOf<PendingImageAttachment>() }
|
||||
|
||||
val pickImages =
|
||||
rememberLauncherForActivityResult(ActivityResultContracts.GetMultipleContents()) { uris ->
|
||||
if (uris.isNullOrEmpty()) return@rememberLauncherForActivityResult
|
||||
scope.launch(Dispatchers.IO) {
|
||||
val next =
|
||||
uris.take(8).mapNotNull { uri ->
|
||||
try {
|
||||
loadSizedImageAttachment(resolver, uri)
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
withContext(Dispatchers.Main) {
|
||||
attachments.addAll(next)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Column(
|
||||
modifier =
|
||||
Modifier
|
||||
.fillMaxSize()
|
||||
.padding(horizontal = 20.dp, vertical = 12.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(8.dp),
|
||||
) {
|
||||
ChatThreadSelector(
|
||||
sessionKey = sessionKey,
|
||||
sessions = sessions,
|
||||
mainSessionKey = mainSessionKey,
|
||||
onSelectSession = { key -> viewModel.switchChatSession(key) },
|
||||
)
|
||||
|
||||
if (!errorText.isNullOrBlank()) {
|
||||
ChatErrorRail(errorText = errorText!!)
|
||||
}
|
||||
|
||||
ChatMessageListCard(
|
||||
messages = messages,
|
||||
pendingRunCount = pendingRunCount,
|
||||
pendingToolCalls = pendingToolCalls,
|
||||
streamingAssistantText = streamingAssistantText,
|
||||
healthOk = healthOk,
|
||||
modifier = Modifier.weight(1f, fill = true),
|
||||
)
|
||||
|
||||
Row(modifier = Modifier.fillMaxWidth().imePadding()) {
|
||||
ChatComposer(
|
||||
draftText = chatDraft,
|
||||
healthOk = healthOk,
|
||||
thinkingLevel = thinkingLevel,
|
||||
pendingRunCount = pendingRunCount,
|
||||
attachments = attachments,
|
||||
onDraftApplied = viewModel::clearChatDraft,
|
||||
onPickImages = { pickImages.launch("image/*") },
|
||||
onRemoveAttachment = { id -> attachments.removeAll { it.id == id } },
|
||||
onSetThinkingLevel = { level -> viewModel.setChatThinkingLevel(level) },
|
||||
onRefresh = {
|
||||
viewModel.refreshChat()
|
||||
viewModel.refreshChatSessions(limit = 200)
|
||||
},
|
||||
onAbort = { viewModel.abortChat() },
|
||||
onSend = { text ->
|
||||
val outgoing =
|
||||
attachments.map { att ->
|
||||
OutgoingAttachment(
|
||||
type = "image",
|
||||
mimeType = att.mimeType,
|
||||
fileName = att.fileName,
|
||||
base64 = att.base64,
|
||||
)
|
||||
}
|
||||
viewModel.sendChat(message = text, thinking = thinkingLevel, attachments = outgoing)
|
||||
attachments.clear()
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ChatThreadSelector(
|
||||
sessionKey: String,
|
||||
sessions: List<ChatSessionEntry>,
|
||||
mainSessionKey: String,
|
||||
onSelectSession: (String) -> Unit,
|
||||
) {
|
||||
val sessionOptions =
|
||||
remember(sessionKey, sessions, mainSessionKey) {
|
||||
resolveSessionChoices(sessionKey, sessions, mainSessionKey = mainSessionKey)
|
||||
}
|
||||
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth().horizontalScroll(rememberScrollState()),
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
) {
|
||||
for (entry in sessionOptions) {
|
||||
val active = entry.key == sessionKey
|
||||
Surface(
|
||||
onClick = { onSelectSession(entry.key) },
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
color = if (active) mobileAccent else mobileCardSurface,
|
||||
border = BorderStroke(1.dp, if (active) mobileAccentBorderStrong else mobileBorderStrong),
|
||||
tonalElevation = 0.dp,
|
||||
shadowElevation = 0.dp,
|
||||
) {
|
||||
Text(
|
||||
text = friendlySessionName(entry.displayName ?: entry.key),
|
||||
style = mobileCaption1.copy(fontWeight = if (active) FontWeight.Bold else FontWeight.SemiBold),
|
||||
color = if (active) Color.White else mobileText,
|
||||
maxLines = 1,
|
||||
overflow = TextOverflow.Ellipsis,
|
||||
modifier = Modifier.padding(horizontal = 12.dp, vertical = 8.dp),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ChatErrorRail(errorText: String) {
|
||||
Surface(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
color = mobileDangerSoft,
|
||||
shape = RoundedCornerShape(12.dp),
|
||||
border = androidx.compose.foundation.BorderStroke(1.dp, mobileDanger),
|
||||
) {
|
||||
Column(modifier = Modifier.padding(horizontal = 10.dp, vertical = 8.dp), verticalArrangement = Arrangement.spacedBy(2.dp)) {
|
||||
Text(
|
||||
text = "CHAT ERROR",
|
||||
style = mobileCaption2.copy(letterSpacing = 0.6.sp),
|
||||
color = mobileDanger,
|
||||
)
|
||||
Text(text = errorText, style = mobileCallout, color = mobileText)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
data class PendingImageAttachment(
|
||||
val id: String,
|
||||
val fileName: String,
|
||||
val mimeType: String,
|
||||
val base64: String,
|
||||
)
|
||||
|
|
@ -0,0 +1,73 @@
|
|||
package ai.openclaw.app.ui.chat
|
||||
|
||||
import ai.openclaw.app.chat.ChatSessionEntry
|
||||
|
||||
private const val RECENT_WINDOW_MS = 24 * 60 * 60 * 1000L
|
||||
|
||||
/**
|
||||
* Derive a human-friendly label from a raw session key.
|
||||
* Examples:
|
||||
* "telegram:g-agent-main-main" -> "Main"
|
||||
* "agent:main:main" -> "Main"
|
||||
* "discord:g-server-channel" -> "Server Channel"
|
||||
* "my-custom-session" -> "My Custom Session"
|
||||
*/
|
||||
fun friendlySessionName(key: String): String {
|
||||
// Strip common prefixes like "telegram:", "agent:", "discord:" etc.
|
||||
val stripped = key.substringAfterLast(":")
|
||||
|
||||
// Remove leading "g-" prefix (gateway artifact)
|
||||
val cleaned = if (stripped.startsWith("g-")) stripped.removePrefix("g-") else stripped
|
||||
|
||||
// Split on hyphens/underscores, title-case each word, collapse "main main" -> "Main"
|
||||
val words = cleaned.split('-', '_').filter { it.isNotBlank() }.map { word ->
|
||||
word.replaceFirstChar { it.uppercaseChar() }
|
||||
}.distinct()
|
||||
|
||||
val result = words.joinToString(" ")
|
||||
return result.ifBlank { key }
|
||||
}
|
||||
|
||||
fun resolveSessionChoices(
|
||||
currentSessionKey: String,
|
||||
sessions: List<ChatSessionEntry>,
|
||||
mainSessionKey: String,
|
||||
nowMs: Long = System.currentTimeMillis(),
|
||||
): List<ChatSessionEntry> {
|
||||
val mainKey = mainSessionKey.trim().ifEmpty { "main" }
|
||||
val current = currentSessionKey.trim().let { if (it == "main" && mainKey != "main") mainKey else it }
|
||||
val aliasKey = if (mainKey == "main") null else "main"
|
||||
val cutoff = nowMs - RECENT_WINDOW_MS
|
||||
val sorted = sessions.sortedByDescending { it.updatedAtMs ?: 0L }
|
||||
val recent = mutableListOf<ChatSessionEntry>()
|
||||
val seen = mutableSetOf<String>()
|
||||
for (entry in sorted) {
|
||||
if (aliasKey != null && entry.key == aliasKey) continue
|
||||
if (!seen.add(entry.key)) continue
|
||||
if ((entry.updatedAtMs ?: 0L) < cutoff) continue
|
||||
recent.add(entry)
|
||||
}
|
||||
|
||||
val result = mutableListOf<ChatSessionEntry>()
|
||||
val included = mutableSetOf<String>()
|
||||
val mainEntry = sorted.firstOrNull { it.key == mainKey }
|
||||
if (mainEntry != null) {
|
||||
result.add(mainEntry)
|
||||
included.add(mainKey)
|
||||
} else if (current == mainKey) {
|
||||
result.add(ChatSessionEntry(key = mainKey, updatedAtMs = null))
|
||||
included.add(mainKey)
|
||||
}
|
||||
|
||||
for (entry in recent) {
|
||||
if (included.add(entry.key)) {
|
||||
result.add(entry)
|
||||
}
|
||||
}
|
||||
|
||||
if (current.isNotEmpty() && !included.contains(current)) {
|
||||
result.add(ChatSessionEntry(key = current, updatedAtMs = null))
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
|
@ -0,0 +1,695 @@
|
|||
package ai.openclaw.app.voice
|
||||
|
||||
import android.Manifest
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.content.pm.PackageManager
|
||||
import android.os.Bundle
|
||||
import android.os.Handler
|
||||
import android.os.Looper
|
||||
import android.util.Log
|
||||
import android.speech.RecognitionListener
|
||||
import android.speech.RecognizerIntent
|
||||
import android.speech.SpeechRecognizer
|
||||
import androidx.core.content.ContextCompat
|
||||
import java.util.UUID
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonArray
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
|
||||
enum class VoiceConversationRole {
|
||||
User,
|
||||
Assistant,
|
||||
}
|
||||
|
||||
data class VoiceConversationEntry(
|
||||
val id: String,
|
||||
val role: VoiceConversationRole,
|
||||
val text: String,
|
||||
val isStreaming: Boolean = false,
|
||||
)
|
||||
|
||||
class MicCaptureManager(
|
||||
private val context: Context,
|
||||
private val scope: CoroutineScope,
|
||||
/**
|
||||
* Send [message] to the gateway and return the run ID.
|
||||
* [onRunIdKnown] is called with the idempotency key *before* the network
|
||||
* round-trip so [pendingRunId] is set before any chat events can arrive.
|
||||
*/
|
||||
private val sendToGateway: suspend (message: String, onRunIdKnown: (String) -> Unit) -> String?,
|
||||
private val speakAssistantReply: suspend (String) -> Unit = {},
|
||||
) {
|
||||
companion object {
|
||||
private const val tag = "MicCapture"
|
||||
private const val speechMinSessionMs = 30_000L
|
||||
private const val speechCompleteSilenceMs = 1_500L
|
||||
private const val speechPossibleSilenceMs = 900L
|
||||
private const val transcriptIdleFlushMs = 1_600L
|
||||
private const val maxConversationEntries = 40
|
||||
private const val pendingRunTimeoutMs = 45_000L
|
||||
}
|
||||
|
||||
private val mainHandler = Handler(Looper.getMainLooper())
|
||||
private val json = Json { ignoreUnknownKeys = true }
|
||||
|
||||
private val _micEnabled = MutableStateFlow(false)
|
||||
val micEnabled: StateFlow<Boolean> = _micEnabled
|
||||
|
||||
private val _micCooldown = MutableStateFlow(false)
|
||||
val micCooldown: StateFlow<Boolean> = _micCooldown
|
||||
|
||||
private val _isListening = MutableStateFlow(false)
|
||||
val isListening: StateFlow<Boolean> = _isListening
|
||||
|
||||
private val _statusText = MutableStateFlow("Mic off")
|
||||
val statusText: StateFlow<String> = _statusText
|
||||
|
||||
private val _liveTranscript = MutableStateFlow<String?>(null)
|
||||
val liveTranscript: StateFlow<String?> = _liveTranscript
|
||||
|
||||
private val _queuedMessages = MutableStateFlow<List<String>>(emptyList())
|
||||
val queuedMessages: StateFlow<List<String>> = _queuedMessages
|
||||
|
||||
private val _conversation = MutableStateFlow<List<VoiceConversationEntry>>(emptyList())
|
||||
val conversation: StateFlow<List<VoiceConversationEntry>> = _conversation
|
||||
|
||||
private val _inputLevel = MutableStateFlow(0f)
|
||||
val inputLevel: StateFlow<Float> = _inputLevel
|
||||
|
||||
private val _isSending = MutableStateFlow(false)
|
||||
val isSending: StateFlow<Boolean> = _isSending
|
||||
|
||||
private val messageQueue = ArrayDeque<String>()
|
||||
private val messageQueueLock = Any()
|
||||
private var flushedPartialTranscript: String? = null
|
||||
private var pendingRunId: String? = null
|
||||
private var pendingAssistantEntryId: String? = null
|
||||
private var gatewayConnected = false
|
||||
|
||||
private var recognizer: SpeechRecognizer? = null
|
||||
private var restartJob: Job? = null
|
||||
private var drainJob: Job? = null
|
||||
private var transcriptFlushJob: Job? = null
|
||||
private var pendingRunTimeoutJob: Job? = null
|
||||
private var stopRequested = false
|
||||
private val ttsPauseLock = Any()
|
||||
private var ttsPauseDepth = 0
|
||||
private var resumeMicAfterTts = false
|
||||
|
||||
private fun enqueueMessage(message: String) {
|
||||
synchronized(messageQueueLock) {
|
||||
messageQueue.addLast(message)
|
||||
}
|
||||
}
|
||||
|
||||
private fun snapshotMessageQueue(): List<String> {
|
||||
return synchronized(messageQueueLock) {
|
||||
messageQueue.toList()
|
||||
}
|
||||
}
|
||||
|
||||
private fun hasQueuedMessages(): Boolean {
|
||||
return synchronized(messageQueueLock) {
|
||||
messageQueue.isNotEmpty()
|
||||
}
|
||||
}
|
||||
|
||||
private fun firstQueuedMessage(): String? {
|
||||
return synchronized(messageQueueLock) {
|
||||
messageQueue.firstOrNull()
|
||||
}
|
||||
}
|
||||
|
||||
private fun removeFirstQueuedMessage(): String? {
|
||||
return synchronized(messageQueueLock) {
|
||||
if (messageQueue.isEmpty()) null else messageQueue.removeFirst()
|
||||
}
|
||||
}
|
||||
|
||||
private fun queuedMessageCount(): Int {
|
||||
return synchronized(messageQueueLock) {
|
||||
messageQueue.size
|
||||
}
|
||||
}
|
||||
|
||||
fun setMicEnabled(enabled: Boolean) {
|
||||
if (_micEnabled.value == enabled) return
|
||||
_micEnabled.value = enabled
|
||||
if (enabled) {
|
||||
val pausedForTts =
|
||||
synchronized(ttsPauseLock) {
|
||||
if (ttsPauseDepth > 0) {
|
||||
resumeMicAfterTts = true
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
}
|
||||
if (pausedForTts) {
|
||||
_statusText.value = if (_isSending.value) "Speaking · waiting for reply" else "Speaking…"
|
||||
return
|
||||
}
|
||||
start()
|
||||
sendQueuedIfIdle()
|
||||
} else {
|
||||
// Give the recognizer time to finish processing buffered audio.
|
||||
// Cancel any prior drain to prevent duplicate sends on rapid toggle.
|
||||
drainJob?.cancel()
|
||||
_micCooldown.value = true
|
||||
drainJob = scope.launch {
|
||||
delay(2000L)
|
||||
stop()
|
||||
// Capture any partial transcript that didn't get a final result from the recognizer
|
||||
val partial = _liveTranscript.value?.trim().orEmpty()
|
||||
if (partial.isNotEmpty()) {
|
||||
queueRecognizedMessage(partial)
|
||||
}
|
||||
drainJob = null
|
||||
_micCooldown.value = false
|
||||
sendQueuedIfIdle()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun pauseForTts() {
|
||||
val shouldPause =
|
||||
synchronized(ttsPauseLock) {
|
||||
ttsPauseDepth += 1
|
||||
if (ttsPauseDepth > 1) return@synchronized false
|
||||
resumeMicAfterTts = _micEnabled.value
|
||||
val active = resumeMicAfterTts || recognizer != null || _isListening.value
|
||||
if (!active) return@synchronized false
|
||||
stopRequested = true
|
||||
restartJob?.cancel()
|
||||
restartJob = null
|
||||
transcriptFlushJob?.cancel()
|
||||
transcriptFlushJob = null
|
||||
_isListening.value = false
|
||||
_inputLevel.value = 0f
|
||||
_liveTranscript.value = null
|
||||
_statusText.value = if (_isSending.value) "Speaking · waiting for reply" else "Speaking…"
|
||||
true
|
||||
}
|
||||
if (!shouldPause) return
|
||||
withContext(Dispatchers.Main) {
|
||||
recognizer?.cancel()
|
||||
recognizer?.destroy()
|
||||
recognizer = null
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun resumeAfterTts() {
|
||||
val shouldResume =
|
||||
synchronized(ttsPauseLock) {
|
||||
if (ttsPauseDepth == 0) return@synchronized false
|
||||
ttsPauseDepth -= 1
|
||||
if (ttsPauseDepth > 0) return@synchronized false
|
||||
val resume = resumeMicAfterTts && _micEnabled.value
|
||||
resumeMicAfterTts = false
|
||||
if (!resume) {
|
||||
_statusText.value =
|
||||
when {
|
||||
_micEnabled.value && _isSending.value -> "Listening · sending queued voice"
|
||||
_micEnabled.value -> "Listening"
|
||||
_isSending.value -> "Mic off · sending…"
|
||||
else -> "Mic off"
|
||||
}
|
||||
}
|
||||
resume
|
||||
}
|
||||
if (!shouldResume) return
|
||||
stopRequested = false
|
||||
start()
|
||||
sendQueuedIfIdle()
|
||||
}
|
||||
|
||||
fun onGatewayConnectionChanged(connected: Boolean) {
|
||||
gatewayConnected = connected
|
||||
if (connected) {
|
||||
sendQueuedIfIdle()
|
||||
return
|
||||
}
|
||||
pendingRunTimeoutJob?.cancel()
|
||||
pendingRunTimeoutJob = null
|
||||
pendingRunId = null
|
||||
pendingAssistantEntryId = null
|
||||
_isSending.value = false
|
||||
if (hasQueuedMessages()) {
|
||||
_statusText.value = queuedWaitingStatus()
|
||||
}
|
||||
}
|
||||
|
||||
fun handleGatewayEvent(event: String, payloadJson: String?) {
|
||||
if (event != "chat") return
|
||||
if (payloadJson.isNullOrBlank()) return
|
||||
val payload =
|
||||
try {
|
||||
json.parseToJsonElement(payloadJson).asObjectOrNull()
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
} ?: return
|
||||
|
||||
val runId = pendingRunId ?: run { Log.d("MicCapture", "no pendingRunId — drop"); return }
|
||||
val eventRunId = payload["runId"].asStringOrNull() ?: return
|
||||
if (eventRunId != runId) { Log.d("MicCapture", "runId mismatch: event=$eventRunId pending=$runId"); return }
|
||||
|
||||
when (payload["state"].asStringOrNull()) {
|
||||
"delta" -> {
|
||||
val deltaText = parseAssistantText(payload)
|
||||
if (!deltaText.isNullOrBlank()) {
|
||||
upsertPendingAssistant(text = deltaText.trim(), isStreaming = true)
|
||||
}
|
||||
}
|
||||
"final" -> {
|
||||
val finalText = parseAssistantText(payload)?.trim().orEmpty()
|
||||
if (finalText.isNotEmpty()) {
|
||||
upsertPendingAssistant(text = finalText, isStreaming = false)
|
||||
playAssistantReplyAsync(finalText)
|
||||
} else if (pendingAssistantEntryId != null) {
|
||||
updateConversationEntry(pendingAssistantEntryId!!, text = null, isStreaming = false)
|
||||
}
|
||||
completePendingTurn()
|
||||
}
|
||||
"error" -> {
|
||||
val errorMessage = payload["errorMessage"].asStringOrNull()?.trim().orEmpty().ifEmpty { "Voice request failed" }
|
||||
upsertPendingAssistant(text = errorMessage, isStreaming = false)
|
||||
completePendingTurn()
|
||||
}
|
||||
"aborted" -> {
|
||||
upsertPendingAssistant(text = "Response aborted", isStreaming = false)
|
||||
completePendingTurn()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun start() {
|
||||
stopRequested = false
|
||||
if (!SpeechRecognizer.isRecognitionAvailable(context)) {
|
||||
_statusText.value = "Speech recognizer unavailable"
|
||||
_micEnabled.value = false
|
||||
return
|
||||
}
|
||||
if (!hasMicPermission()) {
|
||||
_statusText.value = "Microphone permission required"
|
||||
_micEnabled.value = false
|
||||
return
|
||||
}
|
||||
|
||||
mainHandler.post {
|
||||
try {
|
||||
if (recognizer == null) {
|
||||
recognizer = SpeechRecognizer.createSpeechRecognizer(context).also { it.setRecognitionListener(listener) }
|
||||
}
|
||||
startListeningSession()
|
||||
} catch (err: Throwable) {
|
||||
_statusText.value = "Start failed: ${err.message ?: err::class.simpleName}"
|
||||
_micEnabled.value = false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun stop() {
|
||||
stopRequested = true
|
||||
restartJob?.cancel()
|
||||
restartJob = null
|
||||
transcriptFlushJob?.cancel()
|
||||
transcriptFlushJob = null
|
||||
_isListening.value = false
|
||||
_statusText.value = if (_isSending.value) "Mic off · sending…" else "Mic off"
|
||||
_inputLevel.value = 0f
|
||||
mainHandler.post {
|
||||
recognizer?.cancel()
|
||||
recognizer?.destroy()
|
||||
recognizer = null
|
||||
}
|
||||
}
|
||||
|
||||
private fun startListeningSession() {
|
||||
val recognizerInstance = recognizer ?: return
|
||||
val intent =
|
||||
Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
|
||||
putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
|
||||
putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true)
|
||||
putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3)
|
||||
putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, context.packageName)
|
||||
putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS, speechMinSessionMs)
|
||||
putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_COMPLETE_SILENCE_LENGTH_MILLIS, speechCompleteSilenceMs)
|
||||
putExtra(
|
||||
RecognizerIntent.EXTRA_SPEECH_INPUT_POSSIBLY_COMPLETE_SILENCE_LENGTH_MILLIS,
|
||||
speechPossibleSilenceMs,
|
||||
)
|
||||
}
|
||||
_statusText.value =
|
||||
when {
|
||||
_isSending.value -> "Listening · sending queued voice"
|
||||
hasQueuedMessages() -> "Listening · ${queuedMessageCount()} queued"
|
||||
else -> "Listening"
|
||||
}
|
||||
_isListening.value = true
|
||||
recognizerInstance.startListening(intent)
|
||||
}
|
||||
|
||||
private fun scheduleRestart(delayMs: Long = 300L) {
|
||||
if (stopRequested) return
|
||||
if (!_micEnabled.value) return
|
||||
restartJob?.cancel()
|
||||
restartJob =
|
||||
scope.launch {
|
||||
delay(delayMs)
|
||||
mainHandler.post {
|
||||
if (stopRequested || !_micEnabled.value) return@post
|
||||
try {
|
||||
startListeningSession()
|
||||
} catch (_: Throwable) {
|
||||
// retry through onError
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun queueRecognizedMessage(text: String) {
|
||||
val message = text.trim()
|
||||
_liveTranscript.value = null
|
||||
if (message.isEmpty()) return
|
||||
appendConversation(
|
||||
role = VoiceConversationRole.User,
|
||||
text = message,
|
||||
)
|
||||
enqueueMessage(message)
|
||||
publishQueue()
|
||||
}
|
||||
|
||||
private fun scheduleTranscriptFlush(expectedText: String) {
|
||||
transcriptFlushJob?.cancel()
|
||||
transcriptFlushJob =
|
||||
scope.launch {
|
||||
delay(transcriptIdleFlushMs)
|
||||
if (!_micEnabled.value || _isSending.value) return@launch
|
||||
val current = _liveTranscript.value?.trim().orEmpty()
|
||||
if (current.isEmpty() || current != expectedText) return@launch
|
||||
flushedPartialTranscript = current
|
||||
queueRecognizedMessage(current)
|
||||
sendQueuedIfIdle()
|
||||
}
|
||||
}
|
||||
|
||||
private fun publishQueue() {
|
||||
_queuedMessages.value = snapshotMessageQueue()
|
||||
}
|
||||
|
||||
private fun sendQueuedIfIdle() {
|
||||
if (_isSending.value) return
|
||||
if (!hasQueuedMessages()) {
|
||||
if (_micEnabled.value) {
|
||||
_statusText.value = "Listening"
|
||||
} else {
|
||||
_statusText.value = "Mic off"
|
||||
}
|
||||
return
|
||||
}
|
||||
if (!gatewayConnected) {
|
||||
_statusText.value = queuedWaitingStatus()
|
||||
return
|
||||
}
|
||||
|
||||
val next = firstQueuedMessage() ?: return
|
||||
_isSending.value = true
|
||||
pendingRunTimeoutJob?.cancel()
|
||||
pendingRunTimeoutJob = null
|
||||
_statusText.value = if (_micEnabled.value) "Listening · sending queued voice" else "Sending queued voice"
|
||||
|
||||
scope.launch {
|
||||
try {
|
||||
val runId = sendToGateway(next) { earlyRunId ->
|
||||
// Called with the idempotency key before chat.send fires so that
|
||||
// pendingRunId is populated before any chat events can arrive.
|
||||
pendingRunId = earlyRunId
|
||||
}
|
||||
// Update to the real runId if the gateway returned a different one.
|
||||
if (runId != null && runId != pendingRunId) pendingRunId = runId
|
||||
if (runId == null) {
|
||||
pendingRunTimeoutJob?.cancel()
|
||||
pendingRunTimeoutJob = null
|
||||
removeFirstQueuedMessage()
|
||||
publishQueue()
|
||||
_isSending.value = false
|
||||
pendingAssistantEntryId = null
|
||||
sendQueuedIfIdle()
|
||||
} else {
|
||||
armPendingRunTimeout(runId)
|
||||
}
|
||||
} catch (err: Throwable) {
|
||||
pendingRunTimeoutJob?.cancel()
|
||||
pendingRunTimeoutJob = null
|
||||
_isSending.value = false
|
||||
pendingRunId = null
|
||||
pendingAssistantEntryId = null
|
||||
_statusText.value =
|
||||
if (!gatewayConnected) {
|
||||
queuedWaitingStatus()
|
||||
} else {
|
||||
"Send failed: ${err.message ?: err::class.simpleName}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun armPendingRunTimeout(runId: String) {
|
||||
pendingRunTimeoutJob?.cancel()
|
||||
pendingRunTimeoutJob =
|
||||
scope.launch {
|
||||
delay(pendingRunTimeoutMs)
|
||||
if (pendingRunId != runId) return@launch
|
||||
pendingRunId = null
|
||||
pendingAssistantEntryId = null
|
||||
_isSending.value = false
|
||||
_statusText.value =
|
||||
if (gatewayConnected) {
|
||||
"Voice reply timed out; retrying queued turn"
|
||||
} else {
|
||||
queuedWaitingStatus()
|
||||
}
|
||||
sendQueuedIfIdle()
|
||||
}
|
||||
}
|
||||
|
||||
private fun completePendingTurn() {
|
||||
pendingRunTimeoutJob?.cancel()
|
||||
pendingRunTimeoutJob = null
|
||||
if (removeFirstQueuedMessage() != null) {
|
||||
publishQueue()
|
||||
}
|
||||
pendingRunId = null
|
||||
pendingAssistantEntryId = null
|
||||
_isSending.value = false
|
||||
sendQueuedIfIdle()
|
||||
}
|
||||
|
||||
private fun queuedWaitingStatus(): String {
|
||||
return "${queuedMessageCount()} queued · waiting for gateway"
|
||||
}
|
||||
|
||||
private fun appendConversation(
|
||||
role: VoiceConversationRole,
|
||||
text: String,
|
||||
isStreaming: Boolean = false,
|
||||
): String {
|
||||
val id = UUID.randomUUID().toString()
|
||||
_conversation.value =
|
||||
(_conversation.value + VoiceConversationEntry(id = id, role = role, text = text, isStreaming = isStreaming))
|
||||
.takeLast(maxConversationEntries)
|
||||
return id
|
||||
}
|
||||
|
||||
private fun updateConversationEntry(id: String, text: String?, isStreaming: Boolean) {
|
||||
val current = _conversation.value
|
||||
if (current.isEmpty()) return
|
||||
|
||||
val targetIndex =
|
||||
when {
|
||||
current[current.lastIndex].id == id -> current.lastIndex
|
||||
else -> current.indexOfFirst { it.id == id }
|
||||
}
|
||||
if (targetIndex < 0) return
|
||||
|
||||
val entry = current[targetIndex]
|
||||
val updatedText = text ?: entry.text
|
||||
if (updatedText == entry.text && entry.isStreaming == isStreaming) return
|
||||
val updated = current.toMutableList()
|
||||
updated[targetIndex] = entry.copy(text = updatedText, isStreaming = isStreaming)
|
||||
_conversation.value = updated
|
||||
}
|
||||
|
||||
private fun upsertPendingAssistant(text: String, isStreaming: Boolean) {
|
||||
val currentId = pendingAssistantEntryId
|
||||
if (currentId == null) {
|
||||
pendingAssistantEntryId =
|
||||
appendConversation(
|
||||
role = VoiceConversationRole.Assistant,
|
||||
text = text,
|
||||
isStreaming = isStreaming,
|
||||
)
|
||||
return
|
||||
}
|
||||
updateConversationEntry(id = currentId, text = text, isStreaming = isStreaming)
|
||||
}
|
||||
|
||||
private fun playAssistantReplyAsync(text: String) {
|
||||
val spoken = text.trim()
|
||||
if (spoken.isEmpty()) return
|
||||
scope.launch {
|
||||
try {
|
||||
speakAssistantReply(spoken)
|
||||
} catch (err: Throwable) {
|
||||
Log.w(tag, "assistant speech failed: ${err.message ?: err::class.simpleName}")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun disableMic(status: String) {
|
||||
stopRequested = true
|
||||
restartJob?.cancel()
|
||||
restartJob = null
|
||||
transcriptFlushJob?.cancel()
|
||||
transcriptFlushJob = null
|
||||
_micEnabled.value = false
|
||||
_isListening.value = false
|
||||
_inputLevel.value = 0f
|
||||
_statusText.value = status
|
||||
mainHandler.post {
|
||||
recognizer?.cancel()
|
||||
recognizer?.destroy()
|
||||
recognizer = null
|
||||
}
|
||||
}
|
||||
|
||||
private fun hasMicPermission(): Boolean {
|
||||
return (
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) ==
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
)
|
||||
}
|
||||
|
||||
private fun parseAssistantText(payload: JsonObject): String? {
|
||||
val message = payload["message"].asObjectOrNull() ?: return null
|
||||
if (message["role"].asStringOrNull() != "assistant") return null
|
||||
val content = message["content"] as? JsonArray ?: return null
|
||||
|
||||
val parts =
|
||||
content.mapNotNull { item ->
|
||||
val obj = item.asObjectOrNull() ?: return@mapNotNull null
|
||||
if (obj["type"].asStringOrNull() != "text") return@mapNotNull null
|
||||
obj["text"].asStringOrNull()?.trim()?.takeIf { it.isNotEmpty() }
|
||||
}
|
||||
if (parts.isEmpty()) return null
|
||||
return parts.joinToString("\n")
|
||||
}
|
||||
|
||||
private val listener =
|
||||
object : RecognitionListener {
|
||||
override fun onReadyForSpeech(params: Bundle?) {
|
||||
_isListening.value = true
|
||||
}
|
||||
|
||||
override fun onBeginningOfSpeech() {}
|
||||
|
||||
override fun onRmsChanged(rmsdB: Float) {
|
||||
val level = ((rmsdB + 2f) / 12f).coerceIn(0f, 1f)
|
||||
_inputLevel.value = level
|
||||
}
|
||||
|
||||
override fun onBufferReceived(buffer: ByteArray?) {}
|
||||
|
||||
override fun onEndOfSpeech() {
|
||||
_inputLevel.value = 0f
|
||||
scheduleRestart()
|
||||
}
|
||||
|
||||
override fun onError(error: Int) {
|
||||
if (stopRequested) return
|
||||
_isListening.value = false
|
||||
_inputLevel.value = 0f
|
||||
val status =
|
||||
when (error) {
|
||||
SpeechRecognizer.ERROR_AUDIO -> "Audio error"
|
||||
SpeechRecognizer.ERROR_CLIENT -> "Client error"
|
||||
SpeechRecognizer.ERROR_NETWORK -> "Network error"
|
||||
SpeechRecognizer.ERROR_NETWORK_TIMEOUT -> "Network timeout"
|
||||
SpeechRecognizer.ERROR_NO_MATCH -> "Listening"
|
||||
SpeechRecognizer.ERROR_RECOGNIZER_BUSY -> "Recognizer busy"
|
||||
SpeechRecognizer.ERROR_SERVER -> "Server error"
|
||||
SpeechRecognizer.ERROR_SPEECH_TIMEOUT -> "Listening"
|
||||
SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS -> "Microphone permission required"
|
||||
SpeechRecognizer.ERROR_LANGUAGE_NOT_SUPPORTED -> "Language not supported on this device"
|
||||
SpeechRecognizer.ERROR_LANGUAGE_UNAVAILABLE -> "Language unavailable on this device"
|
||||
SpeechRecognizer.ERROR_SERVER_DISCONNECTED -> "Speech service disconnected"
|
||||
SpeechRecognizer.ERROR_TOO_MANY_REQUESTS -> "Speech requests limited; retrying"
|
||||
else -> "Speech error ($error)"
|
||||
}
|
||||
_statusText.value = status
|
||||
|
||||
if (
|
||||
error == SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS ||
|
||||
error == SpeechRecognizer.ERROR_LANGUAGE_NOT_SUPPORTED ||
|
||||
error == SpeechRecognizer.ERROR_LANGUAGE_UNAVAILABLE
|
||||
) {
|
||||
disableMic(status)
|
||||
return
|
||||
}
|
||||
|
||||
val restartDelayMs =
|
||||
when (error) {
|
||||
SpeechRecognizer.ERROR_NO_MATCH,
|
||||
SpeechRecognizer.ERROR_SPEECH_TIMEOUT,
|
||||
-> 1_200L
|
||||
SpeechRecognizer.ERROR_TOO_MANY_REQUESTS -> 2_500L
|
||||
else -> 600L
|
||||
}
|
||||
scheduleRestart(delayMs = restartDelayMs)
|
||||
}
|
||||
|
||||
override fun onResults(results: Bundle?) {
|
||||
transcriptFlushJob?.cancel()
|
||||
transcriptFlushJob = null
|
||||
val text = results?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).orEmpty().firstOrNull()
|
||||
if (!text.isNullOrBlank()) {
|
||||
val trimmed = text.trim()
|
||||
if (trimmed != flushedPartialTranscript) {
|
||||
queueRecognizedMessage(trimmed)
|
||||
sendQueuedIfIdle()
|
||||
} else {
|
||||
flushedPartialTranscript = null
|
||||
_liveTranscript.value = null
|
||||
}
|
||||
}
|
||||
scheduleRestart()
|
||||
}
|
||||
|
||||
override fun onPartialResults(partialResults: Bundle?) {
|
||||
val text = partialResults?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).orEmpty().firstOrNull()
|
||||
if (!text.isNullOrBlank()) {
|
||||
val trimmed = text.trim()
|
||||
_liveTranscript.value = trimmed
|
||||
scheduleTranscriptFlush(trimmed)
|
||||
}
|
||||
}
|
||||
|
||||
override fun onEvent(eventType: Int, params: Bundle?) {}
|
||||
}
|
||||
}
|
||||
|
||||
private fun kotlinx.serialization.json.JsonElement?.asObjectOrNull(): JsonObject? =
|
||||
this as? JsonObject
|
||||
|
||||
private fun kotlinx.serialization.json.JsonElement?.asStringOrNull(): String? =
|
||||
(this as? JsonPrimitive)?.takeIf { it.isString }?.content
|
||||
|
|
@ -0,0 +1,242 @@
|
|||
package ai.openclaw.app.voice
|
||||
|
||||
import android.content.Context
|
||||
import android.media.AudioAttributes
|
||||
import android.media.AudioFormat
|
||||
import android.media.AudioTrack
|
||||
import android.media.MediaPlayer
|
||||
import kotlinx.coroutines.CancellationException
|
||||
import kotlinx.coroutines.CompletableDeferred
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.withContext
|
||||
import java.io.File
|
||||
|
||||
internal class TalkAudioPlayer(
|
||||
private val context: Context,
|
||||
) {
|
||||
private val lock = Any()
|
||||
private var active: ActivePlayback? = null
|
||||
|
||||
suspend fun play(audio: TalkSpeakAudio) {
|
||||
when (val mode = resolvePlaybackMode(audio)) {
|
||||
is TalkPlaybackMode.Pcm -> playPcm(audio.bytes, mode.sampleRate)
|
||||
is TalkPlaybackMode.Compressed -> playCompressed(audio.bytes, mode.fileExtension)
|
||||
}
|
||||
}
|
||||
|
||||
fun stop() {
|
||||
synchronized(lock) {
|
||||
active?.cancel()
|
||||
active = null
|
||||
}
|
||||
}
|
||||
|
||||
internal fun resolvePlaybackMode(audio: TalkSpeakAudio): TalkPlaybackMode {
|
||||
return resolvePlaybackMode(
|
||||
outputFormat = audio.outputFormat,
|
||||
mimeType = audio.mimeType,
|
||||
fileExtension = audio.fileExtension,
|
||||
)
|
||||
}
|
||||
|
||||
companion object {
|
||||
internal fun resolvePlaybackMode(
|
||||
outputFormat: String?,
|
||||
mimeType: String?,
|
||||
fileExtension: String?,
|
||||
): TalkPlaybackMode {
|
||||
val normalizedOutputFormat = outputFormat?.trim()?.lowercase()
|
||||
if (normalizedOutputFormat != null) {
|
||||
val pcmSampleRate = parsePcmSampleRate(normalizedOutputFormat)
|
||||
if (pcmSampleRate != null) {
|
||||
return TalkPlaybackMode.Pcm(sampleRate = pcmSampleRate)
|
||||
}
|
||||
}
|
||||
val normalizedMimeType = mimeType?.trim()?.lowercase()
|
||||
val extension =
|
||||
normalizeExtension(
|
||||
fileExtension ?: inferExtension(outputFormat = normalizedOutputFormat, mimeType = normalizedMimeType),
|
||||
)
|
||||
if (extension != null) {
|
||||
return TalkPlaybackMode.Compressed(fileExtension = extension)
|
||||
}
|
||||
throw IllegalStateException("Unsupported talk audio format")
|
||||
}
|
||||
|
||||
private fun parsePcmSampleRate(outputFormat: String): Int? {
|
||||
return when (outputFormat) {
|
||||
"pcm_16000" -> 16_000
|
||||
"pcm_22050" -> 22_050
|
||||
"pcm_24000" -> 24_000
|
||||
"pcm_44100" -> 44_100
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
|
||||
private fun inferExtension(outputFormat: String?, mimeType: String?): String? {
|
||||
return when {
|
||||
outputFormat == "mp3" || outputFormat?.startsWith("mp3_") == true || mimeType == "audio/mpeg" -> ".mp3"
|
||||
outputFormat == "opus" || outputFormat?.startsWith("opus_") == true || mimeType == "audio/ogg" -> ".ogg"
|
||||
outputFormat?.endsWith("-wav") == true || mimeType == "audio/wav" -> ".wav"
|
||||
outputFormat?.endsWith("-webm") == true || mimeType == "audio/webm" -> ".webm"
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
|
||||
private fun normalizeExtension(value: String?): String? {
|
||||
val trimmed = value?.trim()?.lowercase().orEmpty()
|
||||
if (trimmed.isEmpty()) return null
|
||||
return if (trimmed.startsWith(".")) trimmed else ".$trimmed"
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun playPcm(bytes: ByteArray, sampleRate: Int) {
|
||||
withContext(Dispatchers.IO) {
|
||||
val minBufferSize =
|
||||
AudioTrack.getMinBufferSize(
|
||||
sampleRate,
|
||||
AudioFormat.CHANNEL_OUT_MONO,
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
)
|
||||
if (minBufferSize <= 0) {
|
||||
throw IllegalStateException("AudioTrack buffer unavailable")
|
||||
}
|
||||
val track =
|
||||
AudioTrack.Builder()
|
||||
.setAudioAttributes(
|
||||
AudioAttributes.Builder()
|
||||
.setUsage(AudioAttributes.USAGE_MEDIA)
|
||||
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
|
||||
.build(),
|
||||
)
|
||||
.setAudioFormat(
|
||||
AudioFormat.Builder()
|
||||
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||
.setSampleRate(sampleRate)
|
||||
.setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
|
||||
.build(),
|
||||
)
|
||||
.setTransferMode(AudioTrack.MODE_STATIC)
|
||||
.setBufferSizeInBytes(maxOf(minBufferSize, bytes.size))
|
||||
.build()
|
||||
val finished = CompletableDeferred<Unit>()
|
||||
val playback =
|
||||
ActivePlayback(
|
||||
cancel = {
|
||||
finished.completeExceptionally(CancellationException("assistant speech cancelled"))
|
||||
runCatching { track.pause() }
|
||||
runCatching { track.flush() }
|
||||
runCatching { track.stop() }
|
||||
},
|
||||
)
|
||||
register(playback)
|
||||
try {
|
||||
val written = track.write(bytes, 0, bytes.size)
|
||||
if (written != bytes.size) {
|
||||
throw IllegalStateException("AudioTrack write failed")
|
||||
}
|
||||
val totalFrames = bytes.size / 2
|
||||
track.play()
|
||||
while (track.playState == AudioTrack.PLAYSTATE_PLAYING) {
|
||||
if (track.playbackHeadPosition >= totalFrames) {
|
||||
finished.complete(Unit)
|
||||
break
|
||||
}
|
||||
delay(20)
|
||||
}
|
||||
if (!finished.isCompleted) {
|
||||
finished.complete(Unit)
|
||||
}
|
||||
finished.await()
|
||||
} finally {
|
||||
clear(playback)
|
||||
runCatching { track.pause() }
|
||||
runCatching { track.flush() }
|
||||
runCatching { track.stop() }
|
||||
track.release()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun playCompressed(bytes: ByteArray, fileExtension: String) {
|
||||
val tempFile = withContext(Dispatchers.IO) {
|
||||
File.createTempFile("talk-audio-", fileExtension, context.cacheDir).apply {
|
||||
writeBytes(bytes)
|
||||
}
|
||||
}
|
||||
try {
|
||||
val finished = CompletableDeferred<Unit>()
|
||||
val player =
|
||||
withContext(Dispatchers.Main) {
|
||||
MediaPlayer().apply {
|
||||
setAudioAttributes(
|
||||
AudioAttributes.Builder()
|
||||
.setUsage(AudioAttributes.USAGE_MEDIA)
|
||||
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
|
||||
.build(),
|
||||
)
|
||||
setDataSource(tempFile.absolutePath)
|
||||
setOnCompletionListener {
|
||||
finished.complete(Unit)
|
||||
}
|
||||
setOnErrorListener { _, what, extra ->
|
||||
finished.completeExceptionally(IllegalStateException("MediaPlayer error ($what/$extra)"))
|
||||
true
|
||||
}
|
||||
prepare()
|
||||
}
|
||||
}
|
||||
val playback =
|
||||
ActivePlayback(
|
||||
cancel = {
|
||||
finished.completeExceptionally(CancellationException("assistant speech cancelled"))
|
||||
runCatching { player.stop() }
|
||||
},
|
||||
)
|
||||
register(playback)
|
||||
try {
|
||||
withContext(Dispatchers.Main) {
|
||||
player.start()
|
||||
}
|
||||
finished.await()
|
||||
} finally {
|
||||
clear(playback)
|
||||
withContext(Dispatchers.Main) {
|
||||
runCatching { player.stop() }
|
||||
player.release()
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
withContext(Dispatchers.IO) {
|
||||
tempFile.delete()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun register(playback: ActivePlayback) {
|
||||
synchronized(lock) {
|
||||
active?.cancel()
|
||||
active = playback
|
||||
}
|
||||
}
|
||||
|
||||
private fun clear(playback: ActivePlayback) {
|
||||
synchronized(lock) {
|
||||
if (active === playback) {
|
||||
active = null
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
internal sealed interface TalkPlaybackMode {
|
||||
data class Pcm(val sampleRate: Int) : TalkPlaybackMode
|
||||
|
||||
data class Compressed(val fileExtension: String) : TalkPlaybackMode
|
||||
}
|
||||
|
||||
private class ActivePlayback(
|
||||
val cancel: () -> Unit,
|
||||
)
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
package ai.openclaw.app.voice
|
||||
|
||||
internal object TalkDefaults {
|
||||
const val defaultSilenceTimeoutMs = 700L
|
||||
}
|
||||
|
|
@ -0,0 +1,191 @@
|
|||
package ai.openclaw.app.voice
|
||||
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonElement
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
|
||||
private val directiveJson = Json { ignoreUnknownKeys = true }
|
||||
|
||||
data class TalkDirective(
|
||||
val voiceId: String? = null,
|
||||
val modelId: String? = null,
|
||||
val speed: Double? = null,
|
||||
val rateWpm: Int? = null,
|
||||
val stability: Double? = null,
|
||||
val similarity: Double? = null,
|
||||
val style: Double? = null,
|
||||
val speakerBoost: Boolean? = null,
|
||||
val seed: Long? = null,
|
||||
val normalize: String? = null,
|
||||
val language: String? = null,
|
||||
val outputFormat: String? = null,
|
||||
val latencyTier: Int? = null,
|
||||
val once: Boolean? = null,
|
||||
)
|
||||
|
||||
data class TalkDirectiveParseResult(
|
||||
val directive: TalkDirective?,
|
||||
val stripped: String,
|
||||
val unknownKeys: List<String>,
|
||||
)
|
||||
|
||||
object TalkDirectiveParser {
|
||||
fun parse(text: String): TalkDirectiveParseResult {
|
||||
val normalized = text.replace("\r\n", "\n")
|
||||
val lines = normalized.split("\n").toMutableList()
|
||||
if (lines.isEmpty()) return TalkDirectiveParseResult(null, text, emptyList())
|
||||
|
||||
val firstNonEmpty = lines.indexOfFirst { it.trim().isNotEmpty() }
|
||||
if (firstNonEmpty == -1) return TalkDirectiveParseResult(null, text, emptyList())
|
||||
|
||||
val head = lines[firstNonEmpty].trim()
|
||||
if (!head.startsWith("{") || !head.endsWith("}")) {
|
||||
return TalkDirectiveParseResult(null, text, emptyList())
|
||||
}
|
||||
|
||||
val obj = parseJsonObject(head) ?: return TalkDirectiveParseResult(null, text, emptyList())
|
||||
|
||||
val speakerBoost =
|
||||
boolValue(obj, listOf("speaker_boost", "speakerBoost"))
|
||||
?: boolValue(obj, listOf("no_speaker_boost", "noSpeakerBoost"))?.not()
|
||||
|
||||
val directive = TalkDirective(
|
||||
voiceId = stringValue(obj, listOf("voice", "voice_id", "voiceId")),
|
||||
modelId = stringValue(obj, listOf("model", "model_id", "modelId")),
|
||||
speed = doubleValue(obj, listOf("speed")),
|
||||
rateWpm = intValue(obj, listOf("rate", "wpm")),
|
||||
stability = doubleValue(obj, listOf("stability")),
|
||||
similarity = doubleValue(obj, listOf("similarity", "similarity_boost", "similarityBoost")),
|
||||
style = doubleValue(obj, listOf("style")),
|
||||
speakerBoost = speakerBoost,
|
||||
seed = longValue(obj, listOf("seed")),
|
||||
normalize = stringValue(obj, listOf("normalize", "apply_text_normalization")),
|
||||
language = stringValue(obj, listOf("lang", "language_code", "language")),
|
||||
outputFormat = stringValue(obj, listOf("output_format", "format")),
|
||||
latencyTier = intValue(obj, listOf("latency", "latency_tier", "latencyTier")),
|
||||
once = boolValue(obj, listOf("once")),
|
||||
)
|
||||
|
||||
val hasDirective = listOf(
|
||||
directive.voiceId,
|
||||
directive.modelId,
|
||||
directive.speed,
|
||||
directive.rateWpm,
|
||||
directive.stability,
|
||||
directive.similarity,
|
||||
directive.style,
|
||||
directive.speakerBoost,
|
||||
directive.seed,
|
||||
directive.normalize,
|
||||
directive.language,
|
||||
directive.outputFormat,
|
||||
directive.latencyTier,
|
||||
directive.once,
|
||||
).any { it != null }
|
||||
|
||||
if (!hasDirective) return TalkDirectiveParseResult(null, text, emptyList())
|
||||
|
||||
val knownKeys = setOf(
|
||||
"voice", "voice_id", "voiceid",
|
||||
"model", "model_id", "modelid",
|
||||
"speed", "rate", "wpm",
|
||||
"stability", "similarity", "similarity_boost", "similarityboost",
|
||||
"style",
|
||||
"speaker_boost", "speakerboost",
|
||||
"no_speaker_boost", "nospeakerboost",
|
||||
"seed",
|
||||
"normalize", "apply_text_normalization",
|
||||
"lang", "language_code", "language",
|
||||
"output_format", "format",
|
||||
"latency", "latency_tier", "latencytier",
|
||||
"once",
|
||||
)
|
||||
val unknownKeys = obj.keys.filter { !knownKeys.contains(it.lowercase()) }.sorted()
|
||||
|
||||
lines.removeAt(firstNonEmpty)
|
||||
if (firstNonEmpty < lines.size) {
|
||||
if (lines[firstNonEmpty].trim().isEmpty()) {
|
||||
lines.removeAt(firstNonEmpty)
|
||||
}
|
||||
}
|
||||
|
||||
return TalkDirectiveParseResult(directive, lines.joinToString("\n"), unknownKeys)
|
||||
}
|
||||
|
||||
private fun parseJsonObject(line: String): JsonObject? {
|
||||
return try {
|
||||
directiveJson.parseToJsonElement(line) as? JsonObject
|
||||
} catch (_: Throwable) {
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
private fun stringValue(obj: JsonObject, keys: List<String>): String? {
|
||||
for (key in keys) {
|
||||
val value = obj[key].asStringOrNull()?.trim()
|
||||
if (!value.isNullOrEmpty()) return value
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun doubleValue(obj: JsonObject, keys: List<String>): Double? {
|
||||
for (key in keys) {
|
||||
val value = obj[key].asDoubleOrNull()
|
||||
if (value != null) return value
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun intValue(obj: JsonObject, keys: List<String>): Int? {
|
||||
for (key in keys) {
|
||||
val value = obj[key].asIntOrNull()
|
||||
if (value != null) return value
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun longValue(obj: JsonObject, keys: List<String>): Long? {
|
||||
for (key in keys) {
|
||||
val value = obj[key].asLongOrNull()
|
||||
if (value != null) return value
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun boolValue(obj: JsonObject, keys: List<String>): Boolean? {
|
||||
for (key in keys) {
|
||||
val value = obj[key].asBooleanOrNull()
|
||||
if (value != null) return value
|
||||
}
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
private fun JsonElement?.asStringOrNull(): String? =
|
||||
(this as? JsonPrimitive)?.takeIf { it.isString }?.content
|
||||
|
||||
private fun JsonElement?.asDoubleOrNull(): Double? {
|
||||
val primitive = this as? JsonPrimitive ?: return null
|
||||
return primitive.content.toDoubleOrNull()
|
||||
}
|
||||
|
||||
private fun JsonElement?.asIntOrNull(): Int? {
|
||||
val primitive = this as? JsonPrimitive ?: return null
|
||||
return primitive.content.toIntOrNull()
|
||||
}
|
||||
|
||||
private fun JsonElement?.asLongOrNull(): Long? {
|
||||
val primitive = this as? JsonPrimitive ?: return null
|
||||
return primitive.content.toLongOrNull()
|
||||
}
|
||||
|
||||
private fun JsonElement?.asBooleanOrNull(): Boolean? {
|
||||
val primitive = this as? JsonPrimitive ?: return null
|
||||
val content = primitive.content.trim().lowercase()
|
||||
return when (content) {
|
||||
"true", "yes", "1" -> true
|
||||
"false", "no", "0" -> false
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
package ai.openclaw.app.voice
|
||||
|
||||
import ai.openclaw.app.normalizeMainKey
|
||||
import kotlinx.serialization.json.JsonElement
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.booleanOrNull
|
||||
import kotlinx.serialization.json.contentOrNull
|
||||
|
||||
internal data class TalkModeGatewayConfigState(
|
||||
val mainSessionKey: String,
|
||||
val interruptOnSpeech: Boolean?,
|
||||
val silenceTimeoutMs: Long,
|
||||
)
|
||||
|
||||
internal object TalkModeGatewayConfigParser {
|
||||
fun parse(config: JsonObject?): TalkModeGatewayConfigState {
|
||||
val talk = config?.get("talk").asObjectOrNull()
|
||||
val sessionCfg = config?.get("session").asObjectOrNull()
|
||||
return TalkModeGatewayConfigState(
|
||||
mainSessionKey = normalizeMainKey(sessionCfg?.get("mainKey").asStringOrNull()),
|
||||
interruptOnSpeech = talk?.get("interruptOnSpeech").asBooleanOrNull(),
|
||||
silenceTimeoutMs = resolvedSilenceTimeoutMs(talk),
|
||||
)
|
||||
}
|
||||
|
||||
fun resolvedSilenceTimeoutMs(talk: JsonObject?): Long {
|
||||
val fallback = TalkDefaults.defaultSilenceTimeoutMs
|
||||
val primitive = talk?.get("silenceTimeoutMs") as? JsonPrimitive ?: return fallback
|
||||
if (primitive.isString) return fallback
|
||||
val timeout = primitive.content.toDoubleOrNull() ?: return fallback
|
||||
if (timeout <= 0 || timeout % 1.0 != 0.0 || timeout > Long.MAX_VALUE.toDouble()) {
|
||||
return fallback
|
||||
}
|
||||
return timeout.toLong()
|
||||
}
|
||||
}
|
||||
|
||||
private fun JsonElement?.asStringOrNull(): String? =
|
||||
this?.let { element ->
|
||||
element as? JsonPrimitive
|
||||
}?.contentOrNull
|
||||
|
||||
private fun JsonElement?.asBooleanOrNull(): Boolean? {
|
||||
val primitive = this as? JsonPrimitive ?: return null
|
||||
return primitive.booleanOrNull
|
||||
}
|
||||
|
||||
private fun JsonElement?.asObjectOrNull(): JsonObject? =
|
||||
this as? JsonObject
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,143 @@
|
|||
package ai.openclaw.app.voice
|
||||
|
||||
import ai.openclaw.app.gateway.GatewaySession
|
||||
import kotlinx.serialization.Serializable
|
||||
import kotlinx.serialization.encodeToString
|
||||
import kotlinx.serialization.json.Json
|
||||
|
||||
internal data class TalkSpeakAudio(
|
||||
val bytes: ByteArray,
|
||||
val provider: String,
|
||||
val outputFormat: String?,
|
||||
val voiceCompatible: Boolean?,
|
||||
val mimeType: String?,
|
||||
val fileExtension: String?,
|
||||
)
|
||||
|
||||
internal sealed interface TalkSpeakResult {
|
||||
data class Success(val audio: TalkSpeakAudio) : TalkSpeakResult
|
||||
|
||||
data class FallbackToLocal(val message: String) : TalkSpeakResult
|
||||
|
||||
data class Failure(val message: String) : TalkSpeakResult
|
||||
}
|
||||
|
||||
internal class TalkSpeakClient(
|
||||
private val session: GatewaySession? = null,
|
||||
private val json: Json = Json { ignoreUnknownKeys = true },
|
||||
private val requestDetailed: (suspend (String, String, Long) -> GatewaySession.RpcResult)? = null,
|
||||
) {
|
||||
suspend fun synthesize(text: String, directive: TalkDirective?): TalkSpeakResult {
|
||||
val response =
|
||||
try {
|
||||
performRequest(
|
||||
method = "talk.speak",
|
||||
paramsJson = json.encodeToString(TalkSpeakRequest.from(text = text, directive = directive)),
|
||||
timeoutMs = 45_000,
|
||||
)
|
||||
} catch (err: Throwable) {
|
||||
return TalkSpeakResult.Failure(err.message ?: "talk.speak request failed")
|
||||
}
|
||||
if (!response.ok) {
|
||||
val error = response.error
|
||||
val message = error?.message ?: "talk.speak request failed"
|
||||
return if (isFallbackEligible(error)) {
|
||||
TalkSpeakResult.FallbackToLocal(message)
|
||||
} else {
|
||||
TalkSpeakResult.Failure(message)
|
||||
}
|
||||
}
|
||||
val payload =
|
||||
try {
|
||||
json.decodeFromString<TalkSpeakResponse>(response.payloadJson ?: "")
|
||||
} catch (err: Throwable) {
|
||||
return TalkSpeakResult.Failure(err.message ?: "talk.speak payload invalid")
|
||||
}
|
||||
val bytes =
|
||||
try {
|
||||
android.util.Base64.decode(payload.audioBase64, android.util.Base64.DEFAULT)
|
||||
} catch (err: Throwable) {
|
||||
return TalkSpeakResult.Failure(err.message ?: "talk.speak audio decode failed")
|
||||
}
|
||||
if (bytes.isEmpty()) {
|
||||
return TalkSpeakResult.Failure("talk.speak returned empty audio")
|
||||
}
|
||||
return TalkSpeakResult.Success(
|
||||
TalkSpeakAudio(
|
||||
bytes = bytes,
|
||||
provider = payload.provider,
|
||||
outputFormat = payload.outputFormat,
|
||||
voiceCompatible = payload.voiceCompatible,
|
||||
mimeType = payload.mimeType,
|
||||
fileExtension = payload.fileExtension,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
private fun isFallbackEligible(error: GatewaySession.ErrorShape?): Boolean {
|
||||
val reason = error?.details?.reason
|
||||
if (reason == null) return true
|
||||
return reason == "talk_unconfigured" ||
|
||||
reason == "talk_provider_unsupported" ||
|
||||
reason == "method_unavailable"
|
||||
}
|
||||
|
||||
private suspend fun performRequest(
|
||||
method: String,
|
||||
paramsJson: String,
|
||||
timeoutMs: Long,
|
||||
): GatewaySession.RpcResult {
|
||||
requestDetailed?.let { return it(method, paramsJson, timeoutMs) }
|
||||
val activeSession = session ?: throw IllegalStateException("session missing")
|
||||
return activeSession.requestDetailed(method = method, paramsJson = paramsJson, timeoutMs = timeoutMs)
|
||||
}
|
||||
}
|
||||
|
||||
@Serializable
|
||||
internal data class TalkSpeakRequest(
|
||||
val text: String,
|
||||
val voiceId: String? = null,
|
||||
val modelId: String? = null,
|
||||
val outputFormat: String? = null,
|
||||
val speed: Double? = null,
|
||||
val rateWpm: Int? = null,
|
||||
val stability: Double? = null,
|
||||
val similarity: Double? = null,
|
||||
val style: Double? = null,
|
||||
val speakerBoost: Boolean? = null,
|
||||
val seed: Long? = null,
|
||||
val normalize: String? = null,
|
||||
val language: String? = null,
|
||||
val latencyTier: Int? = null,
|
||||
) {
|
||||
companion object {
|
||||
fun from(text: String, directive: TalkDirective?): TalkSpeakRequest {
|
||||
return TalkSpeakRequest(
|
||||
text = text,
|
||||
voiceId = directive?.voiceId,
|
||||
modelId = directive?.modelId,
|
||||
outputFormat = directive?.outputFormat,
|
||||
speed = directive?.speed,
|
||||
rateWpm = directive?.rateWpm,
|
||||
stability = directive?.stability,
|
||||
similarity = directive?.similarity,
|
||||
style = directive?.style,
|
||||
speakerBoost = directive?.speakerBoost,
|
||||
seed = directive?.seed,
|
||||
normalize = directive?.normalize,
|
||||
language = directive?.language,
|
||||
latencyTier = directive?.latencyTier,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Serializable
|
||||
private data class TalkSpeakResponse(
|
||||
val audioBase64: String,
|
||||
val provider: String,
|
||||
val outputFormat: String? = null,
|
||||
val voiceCompatible: Boolean? = null,
|
||||
val mimeType: String? = null,
|
||||
val fileExtension: String? = null,
|
||||
)
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
package ai.openclaw.app.voice
|
||||
|
||||
object VoiceWakeCommandExtractor {
|
||||
fun extractCommand(text: String, triggerWords: List<String>): String? {
|
||||
val raw = text.trim()
|
||||
if (raw.isEmpty()) return null
|
||||
|
||||
val triggers =
|
||||
triggerWords
|
||||
.map { it.trim().lowercase() }
|
||||
.filter { it.isNotEmpty() }
|
||||
.distinct()
|
||||
if (triggers.isEmpty()) return null
|
||||
|
||||
val alternation = triggers.joinToString("|") { Regex.escape(it) }
|
||||
// Match: "<anything> <trigger><punct/space> <command>"
|
||||
val regex = Regex("(?i)(?:^|\\s)($alternation)\\b[\\s\\p{Punct}]*([\\s\\S]+)$")
|
||||
val match = regex.find(raw) ?: return null
|
||||
val extracted = match.groupValues.getOrNull(2)?.trim().orEmpty()
|
||||
if (extracted.isEmpty()) return null
|
||||
|
||||
val cleaned = extracted.trimStart { it.isWhitespace() || it.isPunctuation() }.trim()
|
||||
if (cleaned.isEmpty()) return null
|
||||
return cleaned
|
||||
}
|
||||
}
|
||||
|
||||
private fun Char.isPunctuation(): Boolean {
|
||||
return when (Character.getType(this)) {
|
||||
Character.CONNECTOR_PUNCTUATION.toInt(),
|
||||
Character.DASH_PUNCTUATION.toInt(),
|
||||
Character.START_PUNCTUATION.toInt(),
|
||||
Character.END_PUNCTUATION.toInt(),
|
||||
Character.INITIAL_QUOTE_PUNCTUATION.toInt(),
|
||||
Character.FINAL_QUOTE_PUNCTUATION.toInt(),
|
||||
Character.OTHER_PUNCTUATION.toInt(),
|
||||
-> true
|
||||
else -> false
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
package ai.openclaw.app.voice
|
||||
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.os.Bundle
|
||||
import android.os.Handler
|
||||
import android.os.Looper
|
||||
import android.speech.RecognitionListener
|
||||
import android.speech.RecognizerIntent
|
||||
import android.speech.SpeechRecognizer
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.launch
|
||||
|
||||
class VoiceWakeManager(
|
||||
private val context: Context,
|
||||
private val scope: CoroutineScope,
|
||||
private val onCommand: suspend (String) -> Unit,
|
||||
) {
|
||||
private val mainHandler = Handler(Looper.getMainLooper())
|
||||
|
||||
private val _isListening = MutableStateFlow(false)
|
||||
val isListening: StateFlow<Boolean> = _isListening
|
||||
|
||||
private val _statusText = MutableStateFlow("Off")
|
||||
val statusText: StateFlow<String> = _statusText
|
||||
|
||||
var triggerWords: List<String> = emptyList()
|
||||
private set
|
||||
|
||||
private var recognizer: SpeechRecognizer? = null
|
||||
private var restartJob: Job? = null
|
||||
private var lastDispatched: String? = null
|
||||
private var stopRequested = false
|
||||
|
||||
fun setTriggerWords(words: List<String>) {
|
||||
triggerWords = words
|
||||
}
|
||||
|
||||
fun start() {
|
||||
mainHandler.post {
|
||||
if (_isListening.value) return@post
|
||||
stopRequested = false
|
||||
|
||||
if (!SpeechRecognizer.isRecognitionAvailable(context)) {
|
||||
_isListening.value = false
|
||||
_statusText.value = "Speech recognizer unavailable"
|
||||
return@post
|
||||
}
|
||||
|
||||
try {
|
||||
recognizer?.destroy()
|
||||
recognizer = SpeechRecognizer.createSpeechRecognizer(context).also { it.setRecognitionListener(listener) }
|
||||
startListeningInternal()
|
||||
} catch (err: Throwable) {
|
||||
_isListening.value = false
|
||||
_statusText.value = "Start failed: ${err.message ?: err::class.simpleName}"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fun stop(statusText: String = "Off") {
|
||||
stopRequested = true
|
||||
restartJob?.cancel()
|
||||
restartJob = null
|
||||
mainHandler.post {
|
||||
_isListening.value = false
|
||||
_statusText.value = statusText
|
||||
recognizer?.cancel()
|
||||
recognizer?.destroy()
|
||||
recognizer = null
|
||||
}
|
||||
}
|
||||
|
||||
private fun startListeningInternal() {
|
||||
val r = recognizer ?: return
|
||||
val intent =
|
||||
Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
|
||||
putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
|
||||
putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true)
|
||||
putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3)
|
||||
putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, context.packageName)
|
||||
}
|
||||
|
||||
_statusText.value = "Listening"
|
||||
_isListening.value = true
|
||||
r.startListening(intent)
|
||||
}
|
||||
|
||||
private fun scheduleRestart(delayMs: Long = 350) {
|
||||
if (stopRequested) return
|
||||
restartJob?.cancel()
|
||||
restartJob =
|
||||
scope.launch {
|
||||
delay(delayMs)
|
||||
mainHandler.post {
|
||||
if (stopRequested) return@post
|
||||
try {
|
||||
recognizer?.cancel()
|
||||
startListeningInternal()
|
||||
} catch (_: Throwable) {
|
||||
// Will be picked up by onError and retry again.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun handleTranscription(text: String) {
|
||||
val command = VoiceWakeCommandExtractor.extractCommand(text, triggerWords) ?: return
|
||||
if (command == lastDispatched) return
|
||||
lastDispatched = command
|
||||
|
||||
scope.launch { onCommand(command) }
|
||||
_statusText.value = "Triggered"
|
||||
scheduleRestart(delayMs = 650)
|
||||
}
|
||||
|
||||
private val listener =
|
||||
object : RecognitionListener {
|
||||
override fun onReadyForSpeech(params: Bundle?) {
|
||||
_statusText.value = "Listening"
|
||||
}
|
||||
|
||||
override fun onBeginningOfSpeech() {}
|
||||
|
||||
override fun onRmsChanged(rmsdB: Float) {}
|
||||
|
||||
override fun onBufferReceived(buffer: ByteArray?) {}
|
||||
|
||||
override fun onEndOfSpeech() {
|
||||
scheduleRestart()
|
||||
}
|
||||
|
||||
override fun onError(error: Int) {
|
||||
if (stopRequested) return
|
||||
_isListening.value = false
|
||||
if (error == SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS) {
|
||||
_statusText.value = "Microphone permission required"
|
||||
return
|
||||
}
|
||||
|
||||
_statusText.value =
|
||||
when (error) {
|
||||
SpeechRecognizer.ERROR_AUDIO -> "Audio error"
|
||||
SpeechRecognizer.ERROR_CLIENT -> "Client error"
|
||||
SpeechRecognizer.ERROR_NETWORK -> "Network error"
|
||||
SpeechRecognizer.ERROR_NETWORK_TIMEOUT -> "Network timeout"
|
||||
SpeechRecognizer.ERROR_NO_MATCH -> "Listening"
|
||||
SpeechRecognizer.ERROR_RECOGNIZER_BUSY -> "Recognizer busy"
|
||||
SpeechRecognizer.ERROR_SERVER -> "Server error"
|
||||
SpeechRecognizer.ERROR_SPEECH_TIMEOUT -> "Listening"
|
||||
else -> "Speech error ($error)"
|
||||
}
|
||||
scheduleRestart(delayMs = 600)
|
||||
}
|
||||
|
||||
override fun onResults(results: Bundle?) {
|
||||
val list = results?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).orEmpty()
|
||||
list.firstOrNull()?.let(::handleTranscription)
|
||||
scheduleRestart()
|
||||
}
|
||||
|
||||
override fun onPartialResults(partialResults: Bundle?) {
|
||||
val list = partialResults?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).orEmpty()
|
||||
list.firstOrNull()?.let(::handleTranscription)
|
||||
}
|
||||
|
||||
override fun onEvent(eventType: Int, params: Bundle?) {}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
BIN
openclaw/apps/android/app/src/main/res/font/manrope_700_bold.ttf
Normal file
BIN
openclaw/apps/android/app/src/main/res/font/manrope_700_bold.ttf
Normal file
Binary file not shown.
|
|
@ -0,0 +1,6 @@
|
|||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<adaptive-icon xmlns:android="http://schemas.android.com/apk/res/android">
|
||||
<background android:drawable="@color/ic_launcher_background" />
|
||||
<foreground android:drawable="@mipmap/ic_launcher_foreground" />
|
||||
<monochrome android:drawable="@mipmap/ic_launcher_foreground" />
|
||||
</adaptive-icon>
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<adaptive-icon xmlns:android="http://schemas.android.com/apk/res/android">
|
||||
<background android:drawable="@color/ic_launcher_background" />
|
||||
<foreground android:drawable="@mipmap/ic_launcher_foreground" />
|
||||
<monochrome android:drawable="@mipmap/ic_launcher_foreground" />
|
||||
</adaptive-icon>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 3.3 KiB |
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue